I don’t usually stream Netflix onto my television to probe the inner workings of my mind, but it had that effect not long ago. While I was catching an old episode of Law & Order: Criminal Intent, the actors’ voices lagged a fraction of a second behind the movement of their mouths, making me so disoriented it completely ruined the show. Soon my irritation turned to puzzlement, and some self-observation allowed me to track my frustration to a precise source. I didn’t care that the ominous soundtrack rose half a second late when Vincent D’Onofrio and Kathryn Erbe crept into the subway tunnel where they were about to find a body. I didn’t care that the show’s trademark duh-dung! sound marking a new scene was still duh-dung-ing after the scene started. It was only when people talked that I went batty. I would watch the characters speak, and then I’d switch to listening to them, and then I’d watch them speak again. I just couldn’t meld the two streams of information in my head.
Thanks to Netflix, I was confronted with one of the most crucial tricks that the human brain uses to make sense of the world: combining input from all five senses into a single, coherent experience, updated many times a second in virtually real time. Because the techniques our brains use to meld the senses are far from perfect, it turns out, we can fall prey to a variety of illusions—and to maddening confusion when Netflix delivers audio and video out of sync.
Neuroscientists have come a long way since the mid-1900s, when they launched their first efforts to map out the brain’s sensory pathways. They identified regions of the brain that became active when people saw things, other regions that became active when they heard sounds, and so on through the list of senses. The implications seemed straightforward enough. Separate systems of neurons handled information from different senses. Only after each system had produced a sophisticated representation of the world did the brain combine their perceptions into one experience of reality, like a film editor adding a soundtrack to a movie.
But later scientists began to discover that the brain sometimes broke these rules. And out of that rule-breaking came some intriguing illusions. One of the most famous such illusions is known as the McGurk Effect, named for its discoverer, Harry McGurk, a developmental psychologist at the University of Surrey in England. In the 1970s he filmed people repeatedly making the sound ga. Then he had a new audio track laid over the film so that ga was replaced with the sound ba. The new audio and video were perfectly in sync. Many people who watched the movie were sure that the speakers were actually saying da, a different syllable entirely. If they closed their eyes, they heard the correct ba. When they opened their eyes, it became da again. (If you don’t know about the McGurk Effect, you may want to experience it via this very impressive video.)
9(unknown*no*no*0)&guid=ON){kind=small}
...</p>){kind=link}