Neurophonetics

Vijay Solanki

v.solanki.1@research.gla.c.uk
University of Glasgow
Thurs 9th March 2017

What is Neurophonetics?

''Neurophonetics deals with neurogenic impairments of the motor act of speaking and of the perceptual processes of spoken language understanding, with the aim of unravelling the neural organization of speech motor control and speech perception.''

(Ziegler 2008: 491)

''Neurophonetics aims at the elucidation of the brain mechanisms underlying speech communication in our species''

(Hertrich & Ackermann 2013)

''To the extent that phonetics is a subdiscipline of linguistics, neurophonetics can be viewed as a subdiscipline of neurolinguistics''

(Ziegler 2008)

Lecture Outline

Anatomy & Physiology of the Brain
The WLG Model
Modern Advances

Where?
How?

Conclusion

Learning Outcomes

Acquire a basic understanding of the anatomy and Physiology of the human brain.
Understand the beginnings of neurolinguistics (ie. the WLG model) and how the field has progressed.
Develop a basic understanding of modern thinking behind neurolinguistics/phonetics
Understand what some of the classic neurolinguistic imaging studies can tell us about language processing
Develop knowledge of what additional anatomical structures in the brain contribute to language processing
Gain a very basic insight into the direction that the field is heading

By the end, it should be clear that the brain runs multiple functions simultaneously in order to interpret speech.

Warning slide

There will be some bloody(ish) images

Anatomy and physiology of the brain

Illustration for Ned the Neuron, published by Kizoom in 2012.

The brain is made of NEURONS and GLIAL CELLS (but glial cells are not discussed here)

NEURONS are arranged into layers which allow for the efficient transfer of information, carried as electrical impulses

NEURONAL LAYERS in different areas are arranged to maximise processing efficiency for the functions that they are required to perform and provide the basis for the larger structures of the brain

Neurons

Neurons transmit information to other neurons or muscles (via the brain stem)

FIRING: electrical ACTION POTENTIAL travels down AXON to SYNAPSE (junction)

A NEUROTRANSMITTER is released which either EXCITES or INHIBITS the post-synaptic cell

MYELIN encases important connections, both to protect vital information transfer routes and to speed up transmission of the action potential

Web source 1.

Arrows indicate the direction of conduction of action potentials in axons (red).

(a) Multipolar interneurons. Each has profusely branched dendrites, which receive signals at synapses with several hundred other neurons, and a single long axon that branches laterally and at its terminus.

(b) A motor neuron that innervates a muscle cell. Typically, motor neurons have a single long axon extending from the cell body to the effector cell. In mammalian motor neurons an insulating sheath of myelin usually covers all parts of the axon except at the nodes of Ranvier and the axon terminals.

(c) A sensory neuron in which the axon branches just after it leaves the cell body. The peripheral branch carries the nerve impulse from the receptor cell to the cell body, which is located in the dorsal root ganglion near the spinal cord; the central branch carries the impulse from the cell body to the spinal cord or brain. Both branches are structurally and functionally axons, except at their terminal portions, even though the peripheral branch conducts impulses toward, rather than away from, the cell body.

Neuronal Layers

Web source 2.

Neurons are arranged into layers specific to the functions that they are required to perform

Neurons that perform similar functions are LOCALISED to a particular region of the brain

Brain Structure

Genes to Cognition Online

www.g2conline.org

Sections to take note of and explain include:

Each of the major lobes and what their functions seem to be

Frontal = The frontal lobes are part of the cerebral cortex and are the largest of the brain's structures. They are the main site of so–called 'higher' cognitive functions. The frontal lobes contain a number of important substructures, including the prefrontal cortex, orbitofrontal cortex, motor and premotor cortices, and Broca's area. These substructures are involved in attention and thought, voluntary movement, decision–making, and language.
Parietal = The parietal cortex plays an important role in integrating information from different senses to build a coherent picture of the world. It integrates information from the ventral visual pathways (which process what things are) and dorsal visual pathways (which process where things are). This allows us to coordinate our movements in response to the objects in our environment. It contains a number of distinct reference maps of the body, near space, and distant space, which are constantly updated as we move and interact with the world. The parietal cortex processes attentional awareness of the environment, is involved in manipulating objects, and representing numbers.
Temporal = The temporal lobes contain a large number of substructures, whose functions include perception, face recognition, object recognition, memory acquisition, understanding language, and emotional reactions. Damage to the temporal lobes can result in intriguing neurological deficits called agnosias, which refer to the inability to recognize specific categories (body parts, colors, faces, music, smells).
Occipital = The occipital cortex is the primary visual area of the brain. It receives projections from the retina (via the thalamus) from where different groups of neurons separately encode different visual information such as color, orientation, and motion. Pathways from the occipital lobes reach the temporal and parietal lobes and are eventually processed consciously. Two important pathways of information originating in the occipital lobes are the dorsal and ventral streams. The dorsal stream projects to the parietal lobes and processes where objects are located. The ventral stream projects to structures in the temporal lobes and processes what objects are.

Corpus Callosum
Motor Cortex
Somatosensory Cortex
Primary Auditory Cortex
Cerebellum
Basal ganglia, thalamus, limbic system, hippocampus

The Cerebrum

Cerebrum = 2 hemispheres

Each hemisphere is divided into 4 lobes:

Frontal
Parietal
Temporal
Occipital

The cerebral cortex

Cerebral cortex = surface of cerebrum

Cortex can be roughly divided into areas of functions

e.g. language, personality, vision, audition, motor and sensory functions

Photograph by Robert Ludlow. Wellcome Trust Image Awards winner 2012.

Left & right hemispheres

Equal size/proportion, but different white/grey matter structures

Different networks of connectivity

Some functions are the same:

MOVEMENT SENSATION

Planning & execution cross sides

However, some functions are different across the two hemispheres…

Left Brain – Right Brain

WRONG

Nielsen, et al. (2013)

Hemispheric functions

Each hemisphere has specific processing strengths, eg:

Left Hemisphere	Right Hemisphere
Important for language processing, mathematical functioning	Important for processing visual and spatial information
(Hervé, et al. 2013, Jolles et al. 2015)	(Hervé, et al. 2013)

Within the cerebrum

Cortex (cerebrum surface):

Grey matter (nerve cell bodies)
Some white matter

Subcortical areas:

Grey matter
White matter fibres (eg.):

Corpus Callosum (connects right and left hemispheres)
Arcuate Fasciculus
(connects motor and sensory cortices)

The WLG Model

Phineas Gage

1823-1861, accident in 1848

Phineas Gage (1848)

Van Horn (2012)

Phineas Gage (1848)

Trauma to frontal lobe

One of first cases to highlight role of frontal lobe in:

Personality
Emotional regulation
Decision making / Problem solving

What does this suggest?

Specific brain functions are grouped in specialised regions
We say that brain function is LOCALISED

Brain Pathology and Language

Aphasia:

Loss or impairment of the ability to produce or comprehend language, due to brain damage

Various types:

Global
Broca’s/motor
Wernicke’s/jargon/anomic

Broca & Wernicke

Paul Broca (1824-1880)

Broca’s patient, ‘Tan’ – 1861:

Problem with production (only one syllable ‘tan’)

Large cyst in the left hemisphere (“mushy and deformed”)

Karl Wernicke (1848-1905)

Wernicke’s patient – 1874:

Patient who could speak but couldn’t comprehend language

Lesion at the crossroads of 2 lobes of the brain

Broca’s Area

Where?

Frontal lobe - Inferior frontal gyrus

Pars opercularis
Pars triangularis

Anwander, et al. (2007)

Broca’s Aphasia

Function

Motor language area
Expression

Aphasia

Motor / non-fluent aphasia
Good comprehension, no/impaired speech
E.G. ‘boy go store’ vs ‘The boy has gone to the store’
Slow, laboured, ungrammatical speech
“yes…ah…Monday…ah…dad and…and…ah…hospital….and ah….Wednesday….Wednesday”

Broca’s Area

Photograph of the brain of Paul Broca’s patient called “Tan”

Wernicke’s Area

Where?

Temporal/Parietal lobe

Supramaringal gyrus
Angular gyrus

Andoh, et al. (2008)

Wernicke’s Aphasias

Function

Sensory language area
Comprehension

Aphasia

Fluent / Receptive (cortical sensory) aphasia
Defect in comprehension, good spontaneous speech
Anomic aphasia - word finding difficulty

Slow, laboured, ungrammatical speech
Jargon aphasia - fluent, but unintelligible jargon

The WLG Model

Broca’s and Wernicke’s areas are connected via the ARCUATE FASCICULUS

This view was dominant for more than a century and still carries weight today

It is known as the Wernicke-Lichtheim-Gerschwind Model (WLG model), named after those that helped to develop it

Hagoort (2014)

Summary

Broca’s area

Inferior frontal gyrus (frontal lobe)
Good comprehension but impaired speech
Seat of language production(?)

Wernicke’s area

Supramaringal/angular gyrus (temporal/parietal lobe)
Good production but defect in comprehension
Seat of language comprehension(?)

Both localised to left hemisphere
They operate together as part of a larger network

Modern Advances

Where?

Classic neuroimaging evidence about language processing

Scott, et al (2000)

Normal Speech (Sp)

Spectrally Rotated Speech (RSp)

Vocoded Speech (VCo)

Rotated vocoded Speech (RVCo)

Classic neuroimaging evidence about language processing

Scott, et al (2000)

Red = Responses to sounds with phonetic information

ie. Sp, RSp & VCo

Yellow = Responses to sounds that are intelligible

ie. Sp & VCo

Classic neuroimaging evidence about language processing

Davis & Johnsrude (2003)

English sentences, distorted in a variety of ways.

Looked for correlations between blood flow and intelligibility of speech sounds

Left: intelligible speech vs noise

Right: responses to different forms of distortion

This is a good slide to introduce the notion of bottom-up and top-down processes. Remember: top-down modulation need not be language specific

Davis and Johnsrude: more detailed study: Speech in noise, Segmented, Vocoded Left: These are the areas that light up when speech is intelligible as opposed to unintelligible Right: These are the areas that light up according to the form of the distortion (acoustics)

What does this tell us? Higher order areas are recruited to improve the signal to noise ratio of the incoming speech signal if the signal is distorted. This is shown by the recruitment of more areas in the form independent condition. Those sounds that are clearly speech, do not require further processing, they can be dealt with in areas local to the primary auditory cortex.

Additional information This study also demonstrated the inclusion of the left anterior hippocampus (involved in memory formation), suggests large, interconnected networks. However, this study also highlighted the dominant role of the left hemisphere in resolving perceptual ambiguity, finding only marginal activity in the right hemisphere.

Classic neuroimaging evidence about language processing

Hickock & Poeppel (2004)

Superior Temporal Gyrus

Bilateral (both hemispheres)
Acoustic-phonetic forms

Dorsal Stream (left only)

Wernicke’s area
Broca’s area
Motor cortex
Articulatory forms

Ventral Stream (left only)

Posterior Inferior Temporal Lobe
Sound-meaning interface

Lateralisation of language functions

Left Hemisphere	Right Hemisphere
Traditionally thought to be dominant for language processing Preference for intelligible speech	Damage may spare production and comprehension, but can lead to problems with: pragmatic ability, prosody, speaker characteristics (phonagnosia), recognition of music & environmental sounds
Possibly more sensitive to phonetic form with better time resolution (limited evidence)	Possibly more sensitive to speaker characteristics with better frequency resolution (limited evidence)

Left Hemisphere

Right Hemisphere

Traditionally thought to be dominant for language processing

Preference for intelligible speech

Damage may spare production and comprehension, but can lead to problems with: pragmatic ability, prosody, speaker characteristics (phonagnosia), recognition of music & environmental sounds

Possibly more sensitive to phonetic form with better time resolution (limited evidence)

Possibly more sensitive to speaker characteristics with better frequency resolution (limited evidence)

Zatorre, et al. (2002); Scott, et al. (2009); Francis & Driscoll (2006); Friederici (2011)

Lateralisation of language functions

Spoken word recognition test, was used to establish cerebral dominance

Lateralisation (%Ss):

Spanish 100% left
English 80% left
Chinese 79% bilateral

(tone lang.)

Chinese English Spanish

Valaki et al. (2004)

Language Processing Beyond the Cortex

Cerebellum	Basal Ganglia	Thalamus	Hippocampus

Co-ordinates muscle groups to produce smooth speech & swallowing.	Controls muscles of face, larynx, tongue and pharynx	Inner chamber determines which sensory information to forward to cortex	Long-term memory, language comprehension, word-generation
Helps integrate sensory perception and motor output.	Damage can lead to lack of coordination and facial expression (e.g. Parkinson’s)	Damage can lead to deficits in memory, attention, reduced spontaneous speech	Damage (severe in Alzheimer's) can lead to word-finding difficulties
Damage can lead to slurring of speech	Also disruption to rhythm and temporal processing

Neural correlates of phonetic skill

Born with an ear for dialects?

In naïve (English) listeners, an individual’s brain structure in left auditory cortex, parietal cortex, and left inferior frontal cortex partly predicts their ability to discriminate a difficult contrast (Hindi dental vs. retroflex)
(Golestani et al., 2002, 2007)
In phoneticians, years of transcription experience correlate with size of left pars opercularis
(Golestani et al., 2011)
Phoneticians are also more likely to have multiple or split left transverse gyri in auditory cortex (thought to develop in utero)

How?

Phineas Gage - Modern Insights

Van Horn, et al. (2012)

Friederici (2011) – Web Link, Firefox & Chrome Only

Neuronal Oscillations

Kim, et al. (2014)

Neuronal Coherence

Adapted from Fries (2005)
Also see: Bastos, et al. (2015); Brunet, et al. (2015)

Neuronal Entrainment

Giraud & Poeppel (2012)

In this article, we have articulated a set of hypotheses to investigate the relation between the perception of connected speech and neurobiological mechanisms. We developed a model at anatomic (Figs. 3 and 6), physiological (Figs. 1, 2 and 4) and computational (Fig. 5) levels. At the center of the research program lies the assumption that cortical oscillations provide ways to temporally organize the incoming speech signal. The main emerging principles are that two prerequisites for constructing intelligible representations of the speech stream are phase-locking between stimulus and cortex in (at least) two discrete time domains and the hierarchical coupling of related cortical oscillations during speech processing.

Critique - Oblesser, et al. (2012)

In sum, we argue that an overly enthusiastic focus on speech envelope and concomitantly a too narrow focus on theta oscillations, or the readiness to force all slower neural oscillations into a theta straightjacket, might not get us closer to the neural mechanics of speech comprehension. Without visionary, synergistic perspectives like the one offered by Giraud and Poeppel (2012) we will not make it there either.

Gross et al. (2013)

In summary, we report a nested hierarchy of auditory oscillations at multiple frequencies that match the frequency of relevant linguistic components in continuous speech. These oscillations entrain to speech with differential hemispheric preference for high (left) and low (right) frequencies. Our results indicate that temporal edges in speech increase first the coupling between auditory oscillations across frequency bands and, second, their coupling to the speech envelope. We can only speculate about the nature of the observed phase/amplitude alignments. Most likely the alignments are caused by a combination of modulatory and evoked effects [55,56] where stimulus-driven activity is top-down modulated via ongoing oscillatory activity [30,61]. In this framework oscillatory activity is a mechanism for attentional selection and flexible gating of information from primary sensory areas. Finally, going beyond speech perception, the entrainment of hierarchically organized oscillations between speaker and listener may well have a more general role in interpersonal communication [62,63].

Neuronal Entrainment

Giraud & Poeppel (2012)

Critique - Oblesser, et al. (2012)

Gross et al. (2013)

Neuronal Entrainment

Giraud & Poeppel (2012)

Critique - Oblesser, et al. (2012)

Gross et al. (2013)

Conclusion?

“it takes the whole brain and, by extension, the whole person to participate in producing and perceiving a voice”

Sidtis & Kreiman (2011)

References

Andoh, J., Artiges, E., Pallier, C., Riviиre, D., Mangin, J.-F., Paillиre-Martinot, M.-L. & Martinot, J.-L. (2008). Priming frequencies of transcranial magnetic stimulation over Wernicke's area modulate word detection.. Cerebral cortex (New York, N.Y. : 1991) 18 (1), 210--6.

Anwander, A., Tittgemeyer, M., von Cramon, D. Y., Friederici, A. D. & Knцsche, T. R. (2007). Connectivity-Based Parcellation of Broca's Area.. Cerebral cortex (New York, N.Y. : 1991) 17 (4), 816--25.

Bastos, A. M., Vezoli, J. & Fries, P. (2015). Communication through coherence with inter-areal delays. Current Opinion in Neurobiology 31, 173--180.

Brunet, N., Vinck, M., Bosman, C. A., Singer, W. & Fries, P. (2014). Gamma or no gamma, that is the question. Trends in Cognitive Sciences 18 (10), 507--509.

Davis, M. H. & Johnsrude, I. S. (2003). Hierarchical Processing in Spoken Language Comprehension. J. Neurosci. 23 (8), 3423--3431.

Feng, S., Legault, J., Yang, L., Zhu, J., Shao, K. & Yang, Y. (2015). Differences in grammatical processing strategies for active and passive sentences: An fMRI study. Journal of Neurolinguistics 33 (0), 104--117.

Francis, A. L. & Driscoll, C. (2006). Training to use voice onset time as a cue to talker identification induces a left-ear/right-hemisphere processing advantage.. Brain and language 98 (3), 310--8.

Friederici, A. D. (2011). The brain basis of language processing: from structure to function.. Physiological reviews 91 (4), 1357--92.

Friederici, A. D. (2012). The cortical language circuit: from auditory perception to sentence comprehension. Trends in Cognitive Sciences 16 (5), 262--268.

Fries, P., et al. (2005). A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends in cognitive sciences 9 (10), 474--480.

Ghitza, O., Giraud, A.-L. & Poeppel, D. (2012). Neuronal oscillations and speech perception: critical-band temporal envelopes are the essence.. Frontiers in human neuroscience 6 (340), 340.

Giraud, A.-L. & Poeppel, D. (2012). Cortical oscillations and speech processing: emerging computational principles and operations.. Nature neuroscience 15 (4), 511--7.

Golestani, N., Molko, N., Dehaene, S., LeBihan, D. & Pallier, C. (2007). Brain structure predicts the learning of foreign speech sounds.. Cerebral cortex (New York, N.Y. : 1991) 17 (3), 575--82.

Golestani, N., Paus, T. & Zatorre, R. J. (2002). Anatomical Correlates of Learning Novel Speech Sounds. Neuron 35 (5), 997--1010.

Golestani, N., Price, C. J. & Scott, S. K. (2011). Born with an ear for dialects? Structural plasticity in the expert phonetician brain.. The Journal of neuroscience : the official journal of the Society for Neuroscience 31 (11), 4213--20.

Gross, J., Hoogenboom, N., Thut, G., Schyns, Philippe Phillippe, Panzeri, S., Belin, P., Garrod, S. & Stefano, P. (2013). Speech rhythms and multiplexed oscillatory sensory coding in the human brain.. PLoS biology 11 (12), e1001752.

Hagoort, P. (2014). Nodes and networks in the neural architecture for language: Broca's region and beyond.. Current opinion in neurobiology 28C, 136--141.

Hertrich, I. & Ackermann, H. (2013). Neurophonetics. Wiley Interdisciplinary Reviews: Cognitive Science 4 (2), 191--200.

Hervé, P.-Y., Zago, L., Petit, L., Mazoyer, B. & Tzourio-Mazoyer, N. (2013). Revisiting human hemispheric specialization with neuroimaging.. Trends in cognitive sciences 17 (2), 69--80.

Hickok, G. & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language.. Cognition 92 (1-2), 67--99.

Jolles, D., Wassermann, D., Chokhani, R., Richardson, J., Tenison, C., Bammer, R., Fuchs, L., Supekar, K. & Menon, V. (2015). Plasticity of left perisylvian white-matter tracts is associated with individual differences in math learning. Brain Structure and Function, 1--15.

Kim, J., Lee, S.-K. & Lee, B. (2014). EEG classification in a single-trial basis for vowel speech perception using multivariate empirical mode decomposition. Journal of Neural Engineering 11 (3), 36010--36021.

Mehta, R. K. & Parasuraman, R. (2013). Neuroergonomics: A Review of Applications to Physical and Cognitive Work. Frontiers in Human Neuroscience 7 (889).

Nielsen, J. A., Zielinski, B. A., Ferguson, M. A., Lainhart, J. E. & Anderson, J. S. (2013). An evaluation of the left-brain vs. right-brain hypothesis with resting state functional connectivity magnetic resonance imaging.. PloS one 8 (8), e71275.

Obleser, J., Herrmann, B. & Henry, M. J. (2012). Neural oscillations in speech: don't be enslaved by the envelope. Frontiers in Human Neuroscience 6.

Scott, S. K. (2000). Identification of a pathway for intelligible speech in the left temporal lobe. Brain 123 (12), 2400--2406.

Scott, S. K., McGettigan, C. & Eisner, F. (2009). A little more conversation, a little less action–candidate roles for the motor cortex in speech perception.. Nature reviews. Neuroscience 10 (4), 295--302.

Sidtis, D. & Kreiman, J. (2012). In the beginning was the familiar voice: personally familiar voices in the evolutionary and contemporary biology of communication. Integrative psychological & behavioral science 46 (2), 146--59.

Tsigka, S., Papadelis, C., Braun, C. & Miceli, G. (2014). Distinguishable neural correlates of verbs and nouns: A MEG study on homonyms. Neuropsychologia 54 (0), 87--97.

Valaki, C. E., Maestu, F., Simos, P. G., Zhang, W., Fernandez, A., Amo, C. M., Ortiz, T. M. & Papanicolaou, A. C. (2004). Cortical organization for receptive language functions in Chinese, English, and Spanish: a cross-linguistic MEG study. Neuropsychologia 42 (7), 967--79.

Van Horn, J. D., Irimia, A., Torgerson, C. M., Chambers, M. C., Kikinis, R. & Toga, A. W. (2012). Mapping Connectivity Damage in the Case of Phineas Gage. PLoS ONE 7 (5), e37454.

Zatorre, R. J., Belin, P. & Penhune, V. B. (2002). Structure and function of auditory cortex: music and speech. Trends in cognitive sciences 6 (1), 37--46.

Ziegler, W. (2009). Neurophonetics. In The Handbook of Clinical Linguistics (pp. 491--505). Blackwell Publishing Ltd. (ISBN: 9781444301007.)

Web Sources

rijnlandmodel.nl

thebrain.mcgill.ca