Some hold with the theory that the development of an embryo shows the stages of evolution. In other words, what first develops is fishlike, and then like a small mammal, and then like a lemur or ape, and then something we would recognize as human. Very early embryos have what look like little gill slits in the beginning of their development. At about four weeks, embryos have a little tail. At around six to twelve weeks, the white blood cells dissolve the tail, and the fetus develops into an average, tail-less baby... most of the time, at least. Every now and again, we get a little extra bit of baby, in the form of a vestigial tail.
Not all things that look like tails - protrusions from the tailbone - actually are what doctors consider "true" tails. There are a number of growths or cysts that can form right on the tip of the tailbone. Some of the more unpleasant options are large tumors, elongation of the existing vertebrae, and even parasitic twin tissue. (A parasitic twin is not a fully-formed twin, but the product of another fertilized egg that somehow became fused with the embryo and never developed into a full human being.) True tails form when the white blood cells, for whatever reason, don't absorb all the tissue that formed during embryonic development. These babies carry the marks of humans earliest ancestors.
Because there are only between 20 and 30 cases of "true" vestigial tails since the late 1800s, there is some controversy about what such a tail contains. Some early accounts say that there are sometimes extra vertebrae in such tails. No modern tails have been found to have any bone tissue. They're mostly skin with fat, connective tissue, nerves, and muscle tissue. They can be just a stub, but some babies can be born with tails 13 centimeters long. The tails aren't strictly useless inert structures. Because they have muscle tissue inside, they can actually be twitched back and forth, or even contracted into curves. These days babies don't have their tails long enough to gain a lot of muscle control over them. Removing them is a simple operation, usually done not long after birth.
What remains are questions of why these tails grow in the first place. They're rare enough that researchers aren't left with many clues. Researchers have, for the most part, ruled out family history - which throws the science of the X-Files episode I got the top image from right out the window. The tails are associated with spina bifida, a dangerous condition in which the canals of the spinal cord don't entire close before birth, but they are often present without the disorder. And for some reason they're twice in common in males as they are in females. In the end, no one knows why some babies just develop tails. (Besides, gills are much more practical.)
Image: Popular Science Monthly