In what looks and sounds like something straight out of a Star Wars movie, a robot named Shimon from Georgia Tech can now sing and compose songs with human collaborators, among many other cool upgrades.
Shimon has been around for several years, but its developers at the Georgia Tech Center for Music Technology have endowed the marimba-playing robot with some awesome new capacities, according to a press release.
Remarkably, the bot now sings, and it does so using an eerily human voice. In addition, Shimon can now play the marimba better than before, compose lyrics, and even make expressive gestures with its eyebrows.
The researchers have released a new song, called “Into Your Mind,” that showcases Shimon’s new abilities. It’s surprisingly good, featuring a jazzy, prog-like sensibility, along with trippy lyrics. Incredibly, Shimon wrote the song and the lyrics, with some help from human collaborators. This track and several others will appear on a forthcoming album that’s set to be released on Spotify this spring. Shimon will also embark upon a tour in support of the new material.
Shimon’s creator, Georgia Tech professor Gil Weinberg, said the main goal of the project isn’t to replace humans but to “create surprising and inspiring music through human-robot collaboration,” he told Gizmodo in an email.
“Our work focuses on human-robot collaboration, and we hope robots will bring what they are good at—unique and surprising algorithmic-based musical ideas, humanly impossible mechanical abilities, and so on—and humans bring what they are good at,” such as emotions and expression, he said. Together, this collaboration has the potential to “connect with humans on an emotional and expressive level,” he added.
To make the system work, Weinberg provides Shimon with a theme for a new song. The robot then assembles lyrics based around this theme by learning from a database containing the lyrics of over 50,000 songs, which include genres such as jazz, prog rock, and hip-hop.
So, if Shimon is presented with a theme like “storm,” the bot will generate a batch of associated words, such as “rain,” and then loop these words together to construct a lyric sheet. Shimon can also decide which words are more relevant than others and generate more lyrics based on that.
“There are lots of systems that use deep learning, but lyrics are different,” explained Richard Savery, a Georgia Tech PhD student who helped Shimon with its songwriting abilities, in the press release. “The way semantic meaning moves through lyrics is different. Also, rhyme and rhythm are obviously super important for lyrics, but that isn’t as present in other text generators. So, we use deep learning to generate lyrics, but it’s also combined with semantic knowledge.”
As for Shimon’s human-like voice, the Georgia Tech team collaborated with the Music Technology Group at Pompeu Fabra University in Barcelona, Spain. Georgia Tech provided this team with the lyrics and the melody, and they in turn used their deep learning-based voice synthesizer, which learns from hundreds of songs, to create Shimon’s voice, according to Weinberg.
Shimon also features some new hardware, namely a mouth, eyebrows, and updated head movements, all of which work in concert with the music. The point of these features is to connect the robot with its human bandmates during performances. Georgia Tech PhD student Lisa Zahray helped to devise Shimon’s expressive eyebrow movements.
“We have to think about his role at each time during the song and what he should be doing,” said Zahray in the press release. “We also want to make sure he’s interacting with the other musicians around him to give that feel that he’s performing with people.”
The robot also has improved dexterity, owing to the addition of new motors in his four “hands.” This upgrade has dramatically improved the way Shimon plays the marimba. Designed by PhD student Ning Yang, the hands allow for a greater range of motion and superior control. Shimon can now perform with better dynamic range, in which the soft parts are quieter and the loud parts are louder, which makes for more expressive music.
“We used to have solenoid driven mallets for Shimon that could hit as fast as seven hits per second,” Weinberg told Gizmodo. “We replaced all the actuators with brushless DC motors that can play much faster—up to 30 hits per second—faster than any human can.”
With its rapid playing ability, Shimon can create novel timbres that are otherwise impossible for human performers.
When you watch the new music video for “In Your Mind,” be sure to keep your eye on the drummer, Jason Barnes, who wears a drum-playing prosthetic limb, allowing for superhuman performance—yet another cool project from this innovative music lab.