As promised, I'm going to discuss some of the advances in PCs that really do make a difference (for ordinary home use -- if you're a power user with specialized needs, then you know better than I when the following doesn't apply.)
About all USB devices and PC ports of recent years use USB 2.0; some of the newest use the new USB 3.0 standard; the oldest use USB 1.1 or 1.0. Each successive standard is substantially faster than the previous (but the functional speed will be determined by the slowest component involved.) You'll notice the difference only if you're transferring a lot of data, e.g., uploading a lot of pictures from your camera, or using a USB external drive.
Since Intel came out with the Core Duo, most desktop PCs were at least dual-core (AMD had already been making dual core chips for a while.) Now quad-core is becoming popular. Dual-core really is very nice, as a single process won't drag your whole computer to a halt. You'd think quad-core would be even nicer, but it's harder for it to have a noticeable effect in ordinary usage. Most programs can't directly take advantage of multiple cores. Unless you're using one that can, or routinely simultaneously run things that heavily use the CPU, you may not notice.
Modern motherboards offer an interface called SATA for hard drives and optical drives that's substantially faster than the old IDE. Unless you're moving large files around, it matters less than you might think. But if you're replacing a drive and your motherboard offers SATA, there's no reason not to get it. (There was a brief time when motherboards were already offering SATA but the computers built around them were often still shipping with IDE drives.)
Your motherboard may also offer external SATA (eSATA) and even if it doesn't, an adapter can turn an internal SATA port to an external one. This lets you use an external drive at the same speeds as in internal one, a lot more pleasant. (And anything that makes backups easier makes them more likely to happen.)
Your own uses will determine how much hard-drive space you need. You can get a 1TB drive for under $80 these days; unless you're storing a lot of video or a lot of audio, that'd be somewhat hard to fill. You may want to look into a spare for backup purposes, one of those things most people recognize as a good idea, but too many ignore until it's too late.
Memory (RAM) continues to get faster and new standards support "synchronous access." But the killer feature of recent years' memory is cheapness. If you have an old computer, odds are you can max out its memory fairly cheaply, and that's probably the best bang for your buck in upgrading an old machine.
New machines ofter boast ridiculously powerful power supplies. This is mostly for marketing purposes so they can show a bigger number than last year's model. Unless you have a ridiculously overpowered CPU and video card or a big array of inefficient old hard drives, you'd be extremely hard pressed to need more than 350W. I'm currently running my desktop with its modern dual-core CPU, DDR2 memory, and a dual-head video card with a 200W power supply stripped from a 9-year old machine.
Many, maybe most modern motherboards have video chips and VGA or DVI ports built-in, obviating the need for a separate video card. Some older ones may be challenged by HD video playback, but other than that, unless you want to play new videogames, it's probably all you need.
Likewise, sound cards are a thing of the past, except for specialized needs. Your motherboard's native sound support is probably good enough.
A component that remains expensive but is one of the biggest advances of recent years is the solid-state hard drive (SSD.) Hard drives have moving parts. Moving parts mean vibration and noise (and will likely be the first parts to fail.) SSDs are very fast, and completely silent. There are several different technologies in use; the very cheapest won't be faster than a good hard drive, and I don't recommend them. These remain too expensive for me to get one, but I look forward to killing off one more noise source.
This brings me to my final point. Now that the insane drive for faster CPU speed (to boast a bigger number than last year's model) has abated, the chipmakers have been actually pursuing greater efficiency. That means less waste heat, which means you can cool them with fewer and quieter fans. It's finally possible to build an acceptable modern machine that's passively cooled, with no fans, and thus completely silent. Some day soon, we'll be able to look back on the whines, and whirs, and clicks we've put up with for decades as a bad dream.
And, to repeat myself, here are things you don't need: quad-cores, the fastest memory, the fastest video card (especially since it'll have a loud high-speed fan, when a weaker one would be passively cooled), a sound card, the biggest power supply, the biggest hard-drives (price/unit storage steadily drops until you get to the biggest, newest drives, when it rises again.)
And if you're considering upgrading, consider also that acceptable new machines can be as cheap as $200. It becomes hard to recommend upgrades other than maxing out memory or replacing a hard drive. It pains the environmentalist in me to say it, but as a frugalista, I feel obliged. On the bright side, any new machine is likely to be more efficient than the old one, but it's far from clear that the power savings during its lifetime would compensate for the environmental cost of the new machine's production. Those things are hard to compute.
Just be sure not to throw the machine out. If it's still usable and useful, look for a school that could use it. If it's not, find out where to recycle it. Here in the East Bay we drop our old stuff at the Alameda County Computer Resource Center; your community may have an equivalent.