In a groundbreaking leap at the intersection of neuroscience and technology, a collaborative effort between researchers at UC San Francisco and UC Berkeley has resulted in an extraordinary innovation – a Brain Computer Interface (BCI) that empowers a woman debilitated by a brainstem stroke to communicate through a virtual persona.
The marvel of this achievement lies in its pioneering fusion of speech synthesis and facial expressions, all orchestrated through brain signals. With unprecedented speed and precision, this system decodes these brain signals into text at a remarkable rate of nearly 80 words per minute, shattering the limitations of existing technologies in this domain.
Dr. Edward Chang, a visionary in the field and Chair of Neurological Surgery at UCSF, has dedicated over a decade to advancing this technology. Now, with their breakthrough findings published in Nature on August 23, 2023, the team harbors hopes of obtaining FDA approval for a practical, real-world application that allows speech via brain signals in the foreseeable future.
“Our aspiration is to restore the innate, holistic way of communication that resonates most naturally with us,” shared Chang. As a member of the UCSF Weill Institute for Neuroscience and a distinguished professor, his commitment to enabling full-bodied communication stands at the forefront of this endeavor. “These strides bring us a significant step closer to transforming this vision into reality for patients.”
Previous accomplishments from Chang’s team demonstrated the possibility of converting brain signals into text for an individual who had also suffered a brainstem stroke. This current study, however, ventures further by translating brain signals into the complexity of speech and the nuanced facial expressions that accompany human interaction.
The methodology employed in this remarkable feat included implanting a slender array of 253 electrodes onto the patient’s brain, carefully positioned over regions crucial for speech production. These electrodes intercepted the brain’s intended signals to muscles governing speech and facial expressions, which had been disrupted by the stroke. A cable connected the electrodes to a bank of computers via a port on the patient’s head.
The participant dedicated weeks to collaboratively train the system’s artificial intelligence algorithms to discern her unique brain signals for speech. This training regimen involved repetitively articulating diverse phrases from a 1,024-word lexicon. The objective was for the computer to learn and recognize the distinct brain activity patterns linked to these sounds.
The researchers ingeniously bypassed word recognition and instead decoded words from phonemes, the elemental speech units analogous to letters in written words. This not only bolstered the system’s accuracy but also significantly accelerated the process.
Sean Metzger and Alex Silva, graduate students in the Bioengineering Program at UC Berkeley and UCSF, were pivotal in developing the text decoder. Metzger elucidated the significance, stating, “The accuracy, speed, and vocabulary are crucial. It’s what gives a user the potential, in time, to communicate almost as fast as we do, and to have much more naturalistic and normal conversations.”
To restore the individual’s voice, the team designed an algorithm that synthesized speech, meticulously tailored to resemble her pre-injury voice. This personalization was based on a recording of her speech during her wedding.
The crowning touch came with animating a digital avatar, a feat made possible by software that emulates facial muscle movements, courtesy of Speech Graphics. The avatar’s expressions were orchestrated in tandem with the brain signals that represented the participant’s attempts at speech. This innovative collaboration fused the avatar’s facial motions with the corresponding brain signals for a remarkably authentic emulation of facial expressions.
Kaylo Littlejohn, a graduate student on the project, explained, “We’re making up for the connections between the brain and vocal tract that have been severed by the stroke.” As the avatars mirrored emotions like joy, sorrow, and astonishment, the potential impact on restoring communication capabilities became glaringly evident.
Reference: https://www.sciencedaily.com/releases/2023/08/230823122530.htm
