Until now, augmented reality has been mostly used as a way of giving you more information about the world around you, but a new iPhone app uses your surroundings to remix the music you're listening to.
"Sadly by your side" is an interdisciplinary art project encompassing an album, a book, and an iPhone app. It's written by David Cairo and produced by Fabrica, a design and communications strategy firm based in Italy. There are two different ways to listen to the album using the iPhone app, both of which require that you use the iPhone's camera. To hear the album exactly as Cairo intended it, simply hold the camera up to the $17 dollar book, it'll lock on, and playback the original version. (Or if your arms get tired, you can check out the unremixed version on SoundCloud for free.)
The more experimental route is to use the information hitting the camera sensor as the basis for a remix of the music. As you can see in the video above, the app breaks up the image the camera is seeing into a mixture of black, blue, and red, each of which controls different parameters of the music. The underlying code for each track is programmed so that the data visualization affects the harmony, melody, and rhythm of the track differently.
The work is an intriguing proof of concept for musical apps that could later be developed in the future. The effect of the visual remix is subtle because there aren't too many parameters being controlled and because the music is of the chilled out, meandering instrumental variety to begin with. It's kind of disorienting and odd to point your phone at different stuff to try to discern how it's changing the music. Sometimes you think you've figured out just what increasing the amount of a certain color will do—only to discover that you're totally wrong. [iTunes and Fabrica and SoundCloud]