‘Talking face’ simulations in the brain help us work out what’s being said
The easiest way to talk to someone else is face-to-face. If you can see the movements of a person’s lips and facial muscles, you can more easily work out what they’re saying, a fact made obvious if you’re trying to have a conversation in a noisy environment. These visual cues clue our brains in on how best to interpret the signals coming from our ears.
But what happens when that’s not possible, like when you’re chatting on the phone or listening to a recorded message? New research suggests that if you’ve spoken to someone before, your brain uses memories of their face to help decode what they’re saying when they’re not in front of you. Based on previous experience, It runs a simulation of the speaker’s face to fill in any information missing from the sound stream alone.
These results contradict a classical theory about hearing – the “auditory-only model” – which suggest that the brain deciphers the spoken word using only the signals it receives from the ears. The model has been opposed before, by earlier studies which found that people are better at identifying a speaker by voice if they have briefly seen that person speaking before. Katherina von Kriegstein from University College London extended these discoveries by showing that previous experience also helps us to work out what’s being said, as well as who said it.
Face-offs
She trained 34 volunteers to identify six male speakers by voice and name. The volunteers saw videos of three of the speakers as they talked, but the other three remained faceless, represented only by a drawing of their occupation. As a further catch, half of the volunteers had a condition called prosopagnosia or face blindness, that prevents them from recognising faces, but has no effect on their ability to recognise objects in general.
After the training, the Kriegstein tested the volunteers while they lay inside a magnetic resonance imaging (MRI) scanner. They listened to short recordings of one of the six speakers and had to either work out who was speaking (“speaker recognition”) or what they were saying (“speech recognition”).
Kriegstein found that both prosopagnosics and controls were slightly better at recognising speech when they had seen the speaker’s face before. The improvement was small – between 1-2% – but that is still significant given that typical success rate for this task is greater than 90%. But of the two groups, only the controls were better at recognising speakers after seeing videos of them beforehand. They were 5% more accurate, while the prosopagnosics didn’t benefit at all.
Facial simulations
Using the fMRI scanner, Kriegstein found that these ‘face benefits’ were reflected by the strength of neural activity in two parts of the brain. The first, the superior temporal sulcus (STS) detects facial movements (among other biological motion), of the kind that we use to help us make out the words of a person speaking in front of us. The stronger their activity in the STS, the more benefit the volunteers gained from having seeing videos of the speakers in the speech recognition task.
The second area, the fusiform face area (FFA), specialises in recognising faces and is often damaged in prosopagnosics. Unlike the STS, it played more of a role in the speaker recognition task but only the controls were more accurate at identifying speakers if they had strong activity in the FFA. So two separate networks that are involved in facial processing are active even when there are no faces to process.
Kriegstein concluded that the people pick up key visual elements of a stranger’s speech after less than two minutes of watching them talk, and we use these to store ‘facial signatures’ of new speakers. The brain effectively uses these to run ‘talking face’ simulations, to better decipher any voice it hears. It’s one of the reasons why phone conversations are easier if you’ve previously met the person at the other end of the line in the flesh.
Reference: 10.1073/pnas.0710826105
Image: by Xenia
Go Further
Animals
- How can we protect grizzlies from their biggest threat—trains?How can we protect grizzlies from their biggest threat—trains?
- This ‘saber-toothed’ salmon wasn’t quite what we thoughtThis ‘saber-toothed’ salmon wasn’t quite what we thought
- Why this rhino-zebra friendship makes perfect senseWhy this rhino-zebra friendship makes perfect sense
- When did bioluminescence evolve? It’s older than we thought.When did bioluminescence evolve? It’s older than we thought.
- Soy, skim … spider. Are any of these technically milk?Soy, skim … spider. Are any of these technically milk?
Environment
- Are the Great Lakes the key to solving America’s emissions conundrum?Are the Great Lakes the key to solving America’s emissions conundrum?
- The world’s historic sites face climate change. Can Petra lead the way?The world’s historic sites face climate change. Can Petra lead the way?
- This pristine piece of the Amazon shows nature’s resilienceThis pristine piece of the Amazon shows nature’s resilience
- Listen to 30 years of climate change transformed into haunting musicListen to 30 years of climate change transformed into haunting music
History & Culture
- Meet the original members of the tortured poets departmentMeet the original members of the tortured poets department
- Séances at the White House? Why these first ladies turned to the occultSéances at the White House? Why these first ladies turned to the occult
- Gambling is everywhere now. When is that a problem?Gambling is everywhere now. When is that a problem?
- Beauty is pain—at least it was in 17th-century SpainBeauty is pain—at least it was in 17th-century Spain
Science
- Here's how astronomers found one of the rarest phenomenons in spaceHere's how astronomers found one of the rarest phenomenons in space
- Not an extrovert or introvert? There’s a word for that.Not an extrovert or introvert? There’s a word for that.
- NASA has a plan to clean up space junk—but is going green enough?NASA has a plan to clean up space junk—but is going green enough?
- Soy, skim … spider. Are any of these technically milk?Soy, skim … spider. Are any of these technically milk?
Travel
- Could Mexico's Chepe Express be the ultimate slow rail adventure?Could Mexico's Chepe Express be the ultimate slow rail adventure?
- What it's like to hike the Camino del Mayab in MexicoWhat it's like to hike the Camino del Mayab in Mexico