The first problems that come to mind when you think about time travel probably involve the fabric of the universe or the state of your own existence. But those aren't what would make time travel just plain old irritating. Glove and Boots came up with a little list, and you'll wonder how you never considered them. Maybe blowing up the universe isn't so bad. [Glove and Boots via Laughing Squid]
The real problem with time travel is that you'll probably end up somewhere out in the middle of space by doing so. Unless your time machine is also capable of transporting you to any point in space you desire, as well. It's the thing that people never seem to talk about with time travel. The Earth is constantly rotating, while also orbiting around the sun, which is orbiting around the galactic center, which is constantly moving away from the singularity. Basically, I will never again be at the exact same point in space I'm at right now. People seem to have this assumption that because they're travelling to the same point relative to the Earth, you'd also be travelling to the same point overall. That shit just ain't happening.