Teiyu Goto, a Sony engineer who had much freedom to design the original PlayStation as well as its controllers, actually explains that the PlayStations' iconic button shapes weren't just arbitrarily picked—they actually mean something.
Other game companies at the time assigned alphabet letters or colors to the buttons. We wanted something simple to remember, which is why we went with icons or symbols, and I came up with the triangle-circle-X-square combination immediately afterward. I gave each symbol a meaning and a color. The triangle refers to viewpoint; I had it represent one's head or direction and made it green. Square refers to a piece of paper; I had it represent menus or documents and made it pink. The circle and X represent 'yes' or 'no' decision-making and I made them red and blue respectively. People thought those colors were mixed up, and I had to reinforce to management that that's what I wanted.
What's also interesting, in the interview translated by 1up, is that the designer had to fight upper management, who wanted to keep the controller more similar-looking to the Super Nintendo one. Goto preferred the PlayStation controller we now know—the one with the handles—and won in the end. It's pretty nice that I now know who to pinpoint my anger towards for the fact that Sony still has the same basic controller design, a decade and a half later, when Microsoft's Xbox 360 controller is much more comfortable. [1up via Joystiq]