1. The waiting period when you first boot it up is torturous. Staring at a white progress bar for over two minutes is too long. I want to start watching TV on my TV now, damnit!
2. Syncing over a wireless G network is slower than you would expect. Not really Apple TV's fault, though. Draft-N FTW. Oh, and pick a short password for your wireless network unless you want to take a half hour punching it in with the onscreen keyboard.
3. Speaking of syncing, it's all you can do regarding content on the Apple TV hard drive - you can't manually manage it. That is, on an iPod, you can add and delete songs as you wish, one at a time in iTunes, but you can only re-sync your Apple TV to change what's on the disk.
4. If you have ugly or tiny album art, browsing through your music collection sucks because of how prominent the art is in the interface.
5. Trailers have to be completely buffered before you can play them if you stream them from the iTunes store, and fast forwarding is a bit wonky on streamed videos.
Hands-On Impressions: What You Don't Already Know About AppleTV [Gizmodo]
Apple TV [Gizmodo]