Research conducted at Columbia University, Department of Electrical Engineering
Our ability to listen to speech in noisy environments is a complex process involving the
auditory system, in which many phenomena remain unexplained. Phonemic restoration helps
us to communicate with others even if we cannot hear every word because we subconsciously
fill in missing phonemes—speech sounds— in speech when there is background noise.
Previous studies have explained the perceptual processes involved in the phonemic restoration
effect, clarifying the role of language and acoustic grouping cues. However, the relationship
between top-down and bottom-up processing in a continuous sound remains largely unknown.
Particularly sensitive to the delineation of brain areas, these processes provide an insightful look
at human behavior. In this study, we focus on this interaction by investigating the breakdown
of the phonemic restoration effect, where subjects will have to surpass noise with different
frequencies in order to correctly identify pseudowords. We played recordings of a series of
mispronounced phonemes, which were interrupted by static-sounding noise, and asked subjects
to report the word they heard while varying the intensity of noise in each trial. In the future, we
will also measure neural activity with an electroencephalogram (EEG) cap while the participants
perform this task to further understand the underlying brain mechanisms behind the processing
required for this task. By clarifying the relationship between what we hear and what we actually
perceive, this study investigates where and how the brain subconsciously interprets unclear
speech, which affects our communication abilities in noisy environments.
In my study, I focus on the interaction between these processes and signal to noise
ratios (SNRs) in order to observe how an interruption of a phoneme by a noise signal affects the
way our brains process sounds. SNR is the ratio of signal strength—in this case, our phonemic
words—to the noise level disrupting the signal. Hence, a ratio higher than 1:1 indicates more
signal than noise. The experiment we conducted created a conflict by saying the phonemes
incorrectly with different SNR levels to see where the two processes in our brain are able to
identify the error in each word.
The potential benefits of this perceptual study include a better understanding of the
phonemic restoration effect and shed light on the processes of the auditory system involved in
selectively listening to speech in a noisy environment. This better understanding in turn enables
novel therapeutic approaches for hearing impaired listeners for whom perception of speech in
noise remains very challenging.
Most of our participants were unable to hear the erroneous phoneme within the
words when the SNR was below 0.1, but the majority of subjects correctly identified a
mispronunciation of the phoneme by SNR 0.8. Moreover, while many subjects heard the error
in “identi-bication,” no one was correctly identifying the phoneme; most subjects were hearing a
similar plosive: “identi-pication.”
There were four cases which we saw most often from our participants: I. Correctly
identified pseudoword (acknowledgesent, apprekiation, etc.); II. Partially correct because
participant said a pseudoword within the same phoneme class (saying “identipication” instead
of “identibication” since they are both plosives); III. Phonemic restoration totally effective
(“miscommunication” instead of “miscommunimation”); and IV. Phonemic restoration partially
effective (“disappointshment” instead of “disappointshent” because the “m” was still heard).
Rogue data that was eliminated from our graphs included words that were mispronounced in
areas where the words weren’t even changed (itentification).
The reason for having subjects listen to single pseudowords is in preparation for our
main projects in the upcoming year which will help us identify where in the brain the phonemic
restoration effect takes place. Our data in the future will include epilepsy patients’ responses
to masked phonemes within a contextual sentence. We predict that subjects will have more
difficulty identifying the pseudoword within a sentence than identifying the pseudoword in
our current experiment, where each mispronounced word is said separately and out of context.
Also, our future results will include a visual component in which we will be showing a picture
of a word and will observe if the visually stimulating image affects the phonemic restoration.
Collecting this data involves plotting the power spectrum of each image, which can possibly
indicate more on how top-down and bottom-up processing function together in the brain.
Shweta is a senior. She says, “I interned at Columbia University, Department of Electrical Engineering, but a main goal in our lab was to bridge the gap between electrical engineering and neuroscience. We took an interdisciplinary approach to study auditory neuroscience by implementing electroencephalogram (EEG) technology and designing our experiment through Praat software. I did this internship through NYU’s Girls’ STEM Program, where I was one of the top 5 presenters who spoke about our research experience at the Global Center, across from Washington Square Park. I was also top 10 for this research paper on phonemic restoration.”