I've heard it's meant to be a security feature, but it often seems like a security problem. If I want to write a server that uses a privileged port, not only do I have to worry about how secure my code is, I have to especially worry about whether I'm using setuid right and dropping privileges.
Answer
True. But it also means that anyone talking to you knows that you must have to root privileges to run that server. When you log in to a server on port 22 (say), you know you're talking to a process that was run by root (security problems aside), so you trust it with your password for that system, or other information you might not trust to anyone with a user account on that system.
Reference: http://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html.
Edit to elaborate on the reasoning: a lot of the most important network services - telnet (yes, it's still used - surprisingly often), SSH, many HTTP services, FTP etc. etc. - involve sending important data like passwords over the wire. In a secure setup some sort of encryption, whether inherent in the protocol (SSH) or wrapped around it (stunnel, IPSec), protects the data from being snooped on the wire, but all these protections end at the server.
In order to protect your data properly, you need to be sure that you're talking to the 'real' server. Today secure certificates are the most important way of doing this on the web (and elsewhere): you assume that only the 'real' server has access to the certificate, so if you verify that the server you're talking to has that certificate you'll trust it.
Privileged ports work in a very similar way: only root has access to privileged ports, so if you're talking to a privileged port you know you're talking to root. This isn't very useful on the modern web: what matters is the identity of the server, not its IP. In other types of networks, this isn't the case: in an academic network, for example, servers are often physically controlled by trusted staff in secure rooms, but students and staff have quite free access as users. In this situation it's often safe to assume you can always trust root, so you can log in and send private data to a privileged port safely. If ordinary users could listen on all ports, you'd need a whole extra layer to verify that a particular program was trusted with certain data.