Most of us have at one point or another had a dream in which we were unable to speak or move; to wake up from such a nightmare—and to recall what it’s like to be able to freely use your voice—feels like a liberation. Now, Meta says it’s getting closer to helping people who actually live with this paralyzing condition to communicate through the use of brainwave-decoding AI.
On Monday, the company introduced Brain2Qwerty v2, its latest effort to translate noisy brain activity into coherent text: think of it like a rudimentary form of algorithmically mediated mind-reading. While the research is still in its early stages, it offers a glimpse of a perhaps not-so-distant future in which patients suffering from anarthria, locked-in syndrome, amyotrophic lateral sclerosis (ALS), and other paralyzing neurodegenerative disorders are able to communicate via thought without the need for neuroprosthetics, which typically require extremely invasive, complex, and expensive brain surgery.
“We believe this research has the potential to make a real difference for the millions of people who suffer from brain lesions that prevent them from communicating,” Meta wrote in its announcement. The underlying code for Brain2Qwerty v2, as well as that of its predecessor, v1, have been made available online. “Our hope is that this work, done in the open, advances neuroscience to identify, diagnose, and treat neurological disorders faster than in siloes,” the company wrote, echoing a burgeoning movement within the AI industry to provide scientists with access to open source AI in the name of accelerating the pace of discovery.
How Meta trained Brain2Qwerty v2
The training for the new model, which was conducted at the Basque Center on Cognition, Brain, and Language in San Sebastián, Spain, involved nine healthy volunteers between the ages of 25 and 56 who were asked to type more than 2,500 sentences over the course of ten sessions. Throughout these sessions, their brain activity was monitored via magnetoencephelography (MEG), which measures the minuscule electric fields produced by neuronal activity in the brain. All of those typed sentences and brainscans then served as the raw training data that was fed into Brain2Qwerty.
In its most successful experiment, Brain2Qwerty v2 achieved a word accuracy—meaning more than half of the sentences that were decoded from brain activity contained no more than one word error—of 78%. In contrast, Brain2Qwerty v1 (which was released last year) achieved a score of 48% in its most successful case.
The researchers also found that the accuracy of the new system’s decoding ability increased alongside the amount of training data it was provided with, suggesting that simple scaling laws could be applied to build more capable systems in the future: “if extended training on non-invasive MEG data can eventually obviate the need for neurosurgery,” the researchers wrote in their technical paper, “it would represent a transformative shift in patient care.”
From brainwaves to LLM to communication
Brain2Qwerty v2’s unprecedented decoding accuracy was achieved in large part by leveraging the same pattern-recognition technology behind chatbots like ChatGPT and Meta’s Llama. In the first two stages of the decoding process, subjects’ brainwaves measured by MEG were translated via AI into tokens representing individual characters, at which point another AI system—called an aligner—organized the individual characters into complete words. A large language model takes over from there, turning the other AI’s jumble of characters and words into coherent sentences.
The results mark the first time an LLM has been successfully deployed to translate noisy brain activity into structured, intelligible sentences. It could also offer a valuable new model for future researchers trying to build new brain-machine interfaces, be they physical or virtual, in which multiple AI systems are used to decode brain activity in a hierarchical and cooperative fashion.
Alongside that multi-tiered AI-driven decoding system, Brain2Qwerty also relies upon a contingent of “auto-research” AI agents, whose task is to autonomously hone the decoding process in order to boost its accuracy and efficiency; think of them like worker bees continually making structural refinements to the hive, so that all the vital activity happening inside can keep going on without a hitch. The agents were trained “to iteratively change our code base to invent novel, better architectures,” the researchers wrote in the paper, producing “a substantial improvement” in word error rate (WER).
The paper also noted, however, that while the agents were helpful in identifying new optimization strategies, they were a long way from replacing human researchers wholesale: “while AI agents may serve as a powerful force multiplier, human research remains, for now, a critical part of the scientific process.”
