The room looks unremarkable. White walls, wood panelled floors, natural light. It could pass for a home office. Ann Johnson sits in her wheelchair facing a large monitor, a cable trailing from a port in her skull to a nearby computer. On the screen is a digital avatar with a face resembling hers.
Johnson had not spoken in 18 years. A brainstem stroke in 2005, at age 30, had taken that from her, along with the ability to move almost any muscle in her body. She had been a math teacher in Saskatchewan, a volleyball coach, a new mother, the kind of person who gave 15-minute speeches at her own wedding. Then, one afternoon playing volleyball with friends, everything changed.
Now, in a UCSF lab in 2023, the avatar opened its mouth. “I think you are wonderful,” it said, in something close to Ann Johnson’s voice. She turned to look at her husband, sitting beside her, and smiled.
Behind that moment is a decade-long collaboration between UCSF surgeons and UC Berkeley engineers, whose work has accelerated our understanding of how brains turn abstract thought into speech, and led to the development of Speech Brain-Computer Interfaces (BCIs), like the one Ann Johnson was using to communicate to her husband.
It started with a map of sorts. In 2013, American neurosurgeon Edward Chang’s lab at UCSF began uncovering how the brain operates the muscles of the tongue, lips, jaw, and larynx by recording information directly from the surface of the cerebral cortex in epilepsy patients. That work produced the first detailed maps of the “speech motor homunculus,” a kind of atlas of the brain’s surface, showing exactly which regions of the brain control the mechanics of speech and revealed, for the first time, how the anatomical movements necessary for speech are encoded at the neural level.
This early mapping set the stage for a breakthrough in 2019, when a Berkeley–UCSF team led by Chang and Gopala Anumanchipalli, a professor of mechanical engineering at Berkeley, published a paper in Nature demonstrating that it was possible to turn brain activity directly into spoken words. Instead of having to type out letters or select words on a screen through eye tracking, paralyzed patients could now produce fluid, sentence-level speech through neural signals alone.
The next leap came in 2021, when UCSF researchers published a New England Journal of Medicine study showing they could decode speech in a person rendered mute by a stroke. By interpreting the neural signals associated with “attempted speech,” the system generated text in real time—the first demonstration of restoring communication in someone who could no longer articulate aloud.
Critical to these breakthroughs was the ability to place sensors directly onto the surface of the brain, a technique that arose from studies of epilepsy patients at UCSF who are awaiting surgery. In order to identify the source of their intractable epilepsy, doctors open the patients’ skulls and sensors are applied to the cortex to locate which parts of the brain are generating the seizures—and if they can be safely removed. The sensors are electrode grids: thin, translucent sheets roughly the size of a credit card, laid directly onto the exposed surface of the brain. Each grid is dotted with dozens of small metal contacts, each one listening—and giving a rare glimpse into higher resolution brain signals.
While the grids are applied, subjects might be asked to listen to music, or speak in multiple languages, giving live data to researchers, which is then interpreted through computational models.
This rapid progress is why many in the field see UC Berkeley and UCSF as global leaders: a rare pairing of neurosurgeons with access to patients and engineers building the machine-learning models that make real-time decoding possible.
As this research accelerates, however, so do the ethical questions. Language is “proximate to thought,” UCSF neuroscience associate professor Narayan Sankaran ’10 stressed, “And so it raises some kind of deep existential fears in people around what we call mental privacy.”
For now, the concern is strictly hypothetical. “We can’t do anything like read minds,” Sankaran says. There are “economic barriers to a mass rollout of a device that can just unwillingly snoop in on private thoughts.”
Nevertheless, he thinks robust ethical and legal norms are necessary. “You can’t just think about what we can do today. You have to kind of extrapolate and predict the future of it.”
