An experiment by University of Washington researchers is setting the stage for advances in mind reading technology. Using brain implants and sophisticated software, researchers can now predict what their subjects are seeing with startling speed and accuracy.
The ability to view a two-dimensional image on a page or computer screen, and then transform that image into something our minds can immediately recognize, is a neurological process that remains mysterious to scientists. To learn more about how our brains perform this task—and to see if computers can collect and predict what a person is seeing in real time—a research team led by University of Washington neuroscientist Rajesh Rao and neurosurgeon Jeff Ojermann demonstrated that it’s possible to decode human brain signals at nearly the speed of perception. The details of their work can be found in a new paper in PLOS Computational Biology.
The team sought the assistance of seven patients undergoing treatment for epilepsy. Medications weren’t helping alleviate their seizures, so these patients were given temporary brain implants, and electrodes were used to pinpoint the focal points of their seizures. The UW researchers saw this as an opportunity to perform their experiment. “They were going to get the electrodes no matter what,” noted Ojermann in a UW NewsBeat article. “We were just giving them additional tasks to do during their hospital stay while they are otherwise just waiting around.”
The patients were shown a random sequence of pictures—images of human faces, houses, and blank gray screens—on computer monitors in brief 400 millisecond intervals. Their specific task was to watch for an image of an upside-down house.
The face and house discrimination task. Credit: Kai J. Miller et al., 2016/PLOS Computational Biology
At the same time, the electrodes in their brain were connected to software that extracted two distinct brain signal properties, namely “event related potentials” (when massive batches of neurons simultaneously light up in response to an image) and “broadband spectral” changes (signals that linger after viewing an image).
As the images flickered on the screen, a computer sampled and digitized the incoming brain signals at a rate of 1,000 times per second. This resolution allowed the software to determine which combination of electrode locations and signals correlated best to what the patients were seeing. “We got different responses from different (electrode) locations; some were sensitive to faces and some were sensitive to houses,” Rao said.
After training the software, researchers exposed the patients to an entirely new set of pictures. Without previous exposure to these new images, the computer was able to predict with 96 percent accuracy when a test subject was seeing a house, a face, or a grey screen. And it did so at nearly the speed of perception.
This proficiency only occurred when the computer considered both event-related potentials and broadband changes, which as stated in the study, suggests “they capture different and complementary aspects of the subject’s perceptual state.”So when it comes to understanding how a person perceives a complex visual object, it’s important to consider the “global picture” of large neural networks.
While interesting, the results of the study are exceptionally limited. A true test of the system would be to see if it could learn a much larger set of images, including different categories. It’s not immediately obvious, for example, if the computer could discern if a patient was viewing the face of a human or a dog.
Once refined, however, this kind of brain decoding could be used to build communication mechanisms for “locked-in” patients who are paralyzed or have suffered a stroke. This technique could also assist with brain mapping, allowing neuroscientists to identify locations in the brain responsible for certain types of information in real time.
Top image: Kai Miller and Brian Donohue