Silent speech with ultrasound

For comparison, lip-reading achieves 12.5% word error rate on a 1M hour dataset. We're excited that our system approaches existing methods despite being an early investigation trained on a 50-hour dataset and done in just a month.

We place an ultrasound probe behind the chin and capture videos like this of the tongue:

And then we turn them into words. Here's a quick demo:

A few days ago, we found out that the model also generalizes to new people (as long as they speak with an American accent 🇺🇸🦅¹1Our Eastern European friends were less thrilled.). Our friends were able to walk in, pick up a probe, and start using the system right away. We did not expect this to work with so little data.

Speaking silently

Speech is roughly four times faster than typing, making it one of the fastest ways to communicate with a computer. As AI gets better at understanding natural language, we're beginning to interact with computers the way science fiction always imagined: by simply talking to them. We're already seeing the beginning of this with tools like Wispr Flow and ChatGPT voice.

But regular speech has one major limitation: most of the time, we're around other people. We're sitting next to coworkers and strangers in coffee shops or on the subway, and we don't want other people overhearing our conversations with our computers.

Just as earphones made listening private, silent speech could make speaking private.

There are several ways of detecting silent speech: EMG, radar, lip reading, etc.²2Andy Matuschak has a great brief review here and on his Patreon. What's special about ultrasound is that you can see the tongue directly and cleanly in the image — no need to infer what's going on from noisy indirect measurements like EMG or radar. And it enables invisible silent speech, where, in principle, you wouldn't even be able to tell that someone is speaking silently.

Moreover, the tongue carries so much information about speech. The tongue forms around 34 distinct phoneme classes across the 40 English phonemes,³3Some phoneme pairs like /t/ and /d/ or /s/ and /z/ differ mainly in voicing rather than tongue position, but they still produce subtle differences in the root of the tongue that ultrasound can detect. compared with only about 10–14 visually distinguishable lip shapes.

Data & Infra

We collected our own dataset of ultrasound tongue imaging with silent speech.

To train a good model, the data has to satisfy two conditions: (1) an ultrasound recording should actually show the tongue, and (2) a person should actually say the text they were supposed to say. Both were surprisingly challenging, as people got tired, mumbled, and didn't hold their probes correctly.

When setting up data collection, the first question was: should people speak silently or out loud?

We ultimately care about decoding silent speech, but audible speech allows us to easily check that people said the words properly. By looking at tongue recordings, we noticed that silent and vocalized speech show similar tongue movements. That led us to believe that we could collect vocalized ultrasound speech data, use the audio to quality-check the data, and still train a model that generalizes to silent speech.

To quality-check the vocalized data, we asked people to speak audibly, transcribed the audio, and checked whether it matched the sentence they were supposed to read. We also used a real-time ultrasound quality classifier to detect poor probe positioning or poor coupling, and had a supervisor with a real-time dashboard monitoring the sessions.

We collected 50 hours of data of people reading synthetically generated short stories out loud.

We used stories instead of isolated phrases because they are easier to read and produce more natural speech. People can stay in the flow of reading, while we still control what appears in the data: broad vocabulary, different sentence structures, and hard cases like articles and contractions.

Training

The task is simple to describe but hard to solve: given an ultrasound video of the tongue, predict the spoken text.

Instead of training everything from scratch, we started with models that already knew something useful: ResNet-18 2+1d for video and Whisper Base for decoding speech-like embeddings into text.

Whisper is a strong speech-to-text model. We found its decoder useful as it is a pretrained model that can convert speech embeddings into text. We just needed to transfer our video embeddings into the embedding space of Whisper. To do that, we trained the tongue-video encoder's embeddings to be as close as possible to the embeddings of Whisper's encoder outputs for the corresponding spoken audio.

Because our dataset was still limited, we used the smallest Whisper decoder. Early training was unstable: the model would either collapse or lean too heavily on its language priors. But after about 20,000 samples, we started seeing the kind of mistakes we wanted: "a key stick" instead of "acoustic," "heart" instead of "hard," and a few others.

These errors were more encouraging than many of the correct answers we had seen before. They suggested that the model was starting to generalize from the signal itself and learn phonetic structure, rather than just predicting likely word sequences.

By collecting more data, running more ablations, training a better model, and improving post-processing (such as generating multiple candidate outputs, using beam search, incorporating sentence-length awareness, or using an LLM as a judge), we reached a word error rate (WER) of 15.6% on our internal open-vocabulary cross-speaker validation set.⁴4Silent-speech numbers are messy because the task changes a lot across papers. Many low-WER systems use fixed command sets, visible lip video, or implanted sensors. The closest ultrasound-only cross-speaker baseline we found reports 83.8% WER on TaL. Our system reaches 15.6% WER from about 50 hours of tongue ultrasound alone.

The word error rate keeps dropping as the dataset grows — from 102% at 15k examples to 15.6% at 50k — and shows no sign of flattening out yet.

Future

This is still an early prototype, not yet a consumer product. The two biggest hardware challenges are reducing the size and weight of the ultrasound probe and replacing ultrasound gel with a more practical coupling material, such as hydrogel. We think both are solvable, making it possible for the probe to eventually become a lightweight wearable or adhesive patch.

With about 50 hours of data and a system built in a month, we already see open-vocabulary transcription that generalizes across people. The current version still struggles with non-American accents. However, more data, better models, and smaller ultrasound hardware should move this from a research demo towards something people can actually use in their daily lives.

Acknowledgements

Thanks to Evan and Angelina for supervising the data factory and keeping the data quality high, and to everyone who came in to read stories into a probe under their chin. We also thank Alex Pokras for staying up all night to film the videos, Claire Wang for helping set up the data curation pipeline, Mustafa Toumi for making many probes, Artem Brustovetskii for helping set up the infrastructure, Galen Mead for providing great thoughts on the product, demo, and model architecture, and Donald Jewkes for directing the video and much of the launch.