Grasping Virtual Objects Benefits Lower Aptitude Learners’ Acquisition of Foreign Language Vocabulary
Theories of embodied cognition describe language acquisition and representation as dependent on sensorimotor experiences that are collected during learning. Whereas native language words are typically acquired through sensorimotor experiences, foreign language (L2) words are often learned by reading, listening or repeating bilingual word lists. Recently, grasping referent objects has been demonstrated to serve as a useful sensorimotor strategy for L2 vocabulary learning. The effects of grasping virtual objects, however, remain unknown. In a virtual reality cave, we trained adult participants (N = 46) having low language aptitude and high language aptitude on novel L2 words under three conditions. In an audiovisual (baseline) condition, participants were presented with written and spoken L2 words. In an audiovisual observation condition, participants additionally saw virtual visual objects that corresponded to the meanings of L2 words. In an audiovisual, an observation, and a grasping condition, participants were asked to grasp the virtual objects. Participants’ word learning was assessed in free and cued recall tests administered immediately after training. Relative to baseline learning, simply viewing virtual objects during learning benefitted both groups. As expected, grasping virtual objects was found to benefit vocabulary retention in low language aptitude learners. Interestingly, this benefit was not observed in high language aptitude learners. Language learning aptitude scores correlated with vocabulary learning outcomes in both audiovisual learning conditions, but not in the sensorimotor condition, suggesting that grasping altered the typical relationship between aptitude and language learning performance. The findings are interpreted in terms of differences in the extent to which procedural and declarative memory systems are accessed in low language aptitude and high language aptitude learners during sensorimotor-based vocabulary learning. Additionally, the results suggest that simulated interactions without tangible feedback can benefit learning. This outcome expands our understanding of how physical experience extends cognition and paves the way for the application of sensorimotor enrichment strategies to virtual environments.
Educational Psychology Review, December 2023
Twelve- and Fourteen-Year-Old School Children Differentially Benefit from Sensorimotor- and Multisensory-Enriched Vocabulary Training
Both children and adults have been shown to benefit from the integration of multisensory and sensorimotor enrichment into pedagogy. For example, integrating pictures or gestures into foreign language (L2) vocabulary learning can improve learning outcomes relative to unisensory learning. However, whereas adults seem to benefit to a greater extent from sensorimotor enrichment such as the performance of gestures in contrast to multisensory enrichment with pictures, this is not the case in elementary school children. Here, we compared multisensory- and sensorimotor-enriched learning in an intermediate age group that falls between the age groups tested in previous studies (elementary school children and young adults), in an attempt to determine the developmental time point at which children’s responses to enrichment mature from a child-like pattern into an adult-like pattern. Twelve-year-old and fourteen-year-old German children were trained over 5 consecutive days on auditorily presented, concrete and abstract, Spanish vocabulary. The vocabulary was learned under picture-enriched, gesture-enriched, and non-enriched (auditory-only) conditions. The children performed vocabulary recall and translation tests at 3 days, 2 months, and 6 months post-learning. Both picture and gesture enrichment interventions were found to benefit children’s L2 learning relative to non-enriched learning up to 6 months post-training. Interestingly, gesture-enriched learning was even more beneficial than picture-enriched learning for the 14-year-olds, while the 12-year-olds benefitted equivalently from learning enriched with pictures and gestures. These findings provide evidence for opting to integrate gestures rather than pictures into L2 pedagogy starting at 14 years of age.
Educational Psychology Review, March 2022
Gesture based word (re) acquisition with a virtual agent in augmented reality: A preliminary study
Visual recognition of words learned with gestures induces motor resonance in the forearm muscles Motor cortex causally contributes to vocabulary translation following sensorimotor-enriched training
According to theories of Embodied Cognition, memory for words is related to sensorimotor experiences collected during learning. At a neural level, words encoded with self-performed gestures are represented in distributed sensorimotor networks that resonate during word recognition. Here, we ask whether muscles involved in gesture execution also resonate during word recognition. Native German speakers encoded words by reading them (baseline condition) or by reading them in tandem with picture observation, gesture observation, or gesture observation and execution. Surface electromyogram (EMG) activity from both arms was recorded during the word recognition task and responses were detected using eye-tracking. The recognition of words encoded with self-performed gestures coincided with an increase in arm muscle EMG activity compared to the recognition of words learned under other conditions. This finding suggests that sensorimotor networks resonate into the periphery and provides new evidence for a strongly embodied view of recognition memory.
Scientific Reports, August 2021
Motor cortex causally contributes to vocabulary translation following sensorimotor-enriched training
The role of the motor cortex in perceptual and cognitive functions is highly controversial. Here, we investigated the hypothesis that the motor cortex can be instrumental for translating foreign language vocabulary. Human participants of both sexes were trained on foreign language (L2) words and their native language translations over 4 consecutive days. L2 words were accompanied by complementary gestures (sensorimotor enrichment) or pictures (sensory enrichment). Following training, participants translated the auditorily presented L2 words that they had learned. During translation, repetitive transcranial magnetic stimulation was applied bilaterally to a site within the primary motor cortex (Brodmann area 4) located in the vicinity of the arm functional compartment. Responses within the stimulated motor region have previously been found to correlate with behavioral benefits of sensorimotor-enriched L2 vocabulary learning. Compared to sham stimulation, effective perturbation by repetitive transcranial magnetic stimulation slowed down the translation of sensorimotor-enriched L2 words, but not sensory-enriched L2 words. This finding suggests that sensorimotor-enriched training induced changes in L2 representations within the motor cortex, which in turn facilitated the translation of L2 words. The motor cortex may play a causal role in precipitating sensorimotor-based learning benefits, and may directly aid in remembering the native language translations of foreign language words following sensorimotor-enriched training. These findings support multisensory theories of learning while challenging reactivation-based theories.
SIGNIFICANCE STATEMENT Despite the potential for sensorimotor enrichment to serve as a powerful tool for learning in many domains, its underlying brain mechanisms remain largely unexplored. Using transcranial magnetic stimulation and a foreign language (L2) learning paradigm, we found that sensorimotor-enriched training can induce changes in L2 representations within the motor cortex, which in turn causally facilitate the translation of L2 words. The translation of recently acquired L2 words may therefore rely not only on auditory information stored in memory or on modality-independent L2 representations, but also on the sensorimotor context in which the words have been experienced.
The Journal of Neuroscience, October 2021
Visual sensory cortices causally contribute to auditory word recognition following sensorimotor-enriched vocabulary training
Despite a rise in the use of “learning by doing” pedagogical methods in praxis, little is known as to how the brain benefits from these methods. Learning by doing strategies that utilize complementary information (‘‘enrichment’’) such as gestures have been shown to optimize learning outcomes in several domains including foreign language (L2) training. Here we tested the hypothesis that behavioral benefits of gesture-based enrichment are critically supported by integrity of the biological motion visual cortices (bmSTS). Prior functional neuroimaging work has implicated the visual motion cortices in L2 translation following sensorimotor-enriched training; the current study is the first to investigate the causal relevance of these structures in learning by doing contexts. Using neuronavigated transcranial magnetic stimulation (TMS) and a gesture-enriched L2 vocabulary learning paradigm, we found that the bmSTS causally contributed to behavioral benefits of gesture-enriched learning. Visual motion cortex integrity benefitted both short- and long-term learning outcomes, as well as the learning of concrete and abstract words. These results adjudicate between opposing predictions of two neuroscientific learning theories: While reactivation-based theories predict no functional role of specialized sensory cortices in vocabulary learning outcomes, the current study supports the predictive coding theory view that these cortices precipitate sensorimotor-based learning benefits.
Cerebral Cortex, September 2020
How Can We Learn Foreign Language Vocabulary More Easily?
Have you ever tried to remember a word in a foreign language? What strategy did you use? In several studies, we examined the beneficial effects of viewing pictures and performing gestures while learning foreign language words. Both pictures and gestures helped primary school kids and adults to better remember the meanings of foreign language words compared to learning by just listening. For kids, pictures and gestures were equally helpful. For adults, gestures were more helpful than pictures. Both visual and motor brain areas helped with learning the foreign language words. Our studies suggest that learning foreign language words with pictures and gestures is helpful for learners, because pictures and gestures allow both kids and adults to experience the meanings of words through multiple senses.
Frontiers for Young Minds, Juli 2020
Positive effects of grasping virtual objects on memory for novel words in a second language
Theories of embodied cognition describe language processing and representation as inherently connected to the sensorimotor experiences collected during acquisition. While children grasp their world, collect bodily experiences and name them, in second language (L2), students learn bilingual word lists. Experimental evidence shows that embodiment by mean of gestures enhances memory for words in L2. However, no study has been conducted on the effects of grasping in L2. In a virtual scenario, we trained 46 participants on 18 two- and three-syllabic words of Vimmi, an artificial corpus created for experimental purposes. The words were assigned concrete meanings of graspable objects. Six words were learned audio-visually, by reading the words projected on the wall and by hearing them. Another 6 words were trained by observation of virtual objects. Another 6 words were learned by observation and additional grasping the virtual objects. Thereafter participants were subministered free, cued recall, and reaction time tests in order to assess the word retention and the word recognition. After 30 days, the recall tests were repeated remotely to assess the memory in the long term. The results show that grasping of virtual objects can lead to superior memory performance and to lower reaction times during recognition.
Scientific Reports, Juni 2020
Link zum Download
Motor cortex causally contributes to auditory word recognition following sensorimotor-enriched vocabulary training
Despite all methodological efforts made in the last three decades, Western instruction grounds on traditional principles. Most educational programs follow theories that are mentalistic, i.e., they separate the mind from the body. At school, learners sit, watch, listen, and write. The aim of this paper is to present embodied learning as an alternative to mentalistic education. Similarly, this paper wants to describe embodied learning from a neuroscientific perspective. After a brief historical overview, I will review studies highlighting the behavioral effectiveness of embodied instruction in second language learning, mathematics and spatial thinking. On this base, I will discuss some of the brain mechanisms driving embodied learning and describe its advantages, clearly pleading in favor of instructional practice that reunites body and mind.
arXiv:2005.08956. , 2020
Learning Foreign Language Vocabulary with Gestures and Pictures Enhances Vocabulary Memory for Several Months Post-Learning in Eight-Year-Old School Children
The integration of gestures and pictures into pedagogy has demonstrated potential for improving adults’ learning of foreign language (L2) vocabulary. However, the relative benefits of gestures and pictures on children’s L2 vocabulary learning have not been formally evaluated. In three experiments, we investigated the effects of gesture-based and picture-based learning on 8-year-old primary school children’s acquisition of novel L2 vocabulary. In each experiment, German children were trained over 5 consecutive days on auditorily presented, concrete and abstract, English vocabulary. In Experiments 1 and 2, gesture enrichment (auditorily presented L2 words accompanied with self-performed gestures) was compared with a non-enriched baseline condition. In Experiment 3, gesture enrichment was compared with picture enrichment (auditorily presented words accompanied with pictures). Children performed vocabulary recall and translation tests at 3 days, 2 months, and 6 months post-learning. Both gesture and picture enrichment enhanced children’s test performance compared with non-enriched learning. Benefits of gesture and picture enrichment persisted up to 6 months after training and occurred for both concrete and abstract words. Gesture-enriched learning was hypothesized to boost learning outcomes more than picture-enriched learning on the basis of previous findings in adults. Unexpectedly, however, we observed similar benefits of gesture and picture enrichment on children’s L2 learning. These findings suggest that both gestures and pictures enhance children’s L2 learning and that performance benefits are robust over long timescales.
Educational Psychology Review, 2020
A causal role of sensory cortices in behavioral benefits of ‘learning by doing’
Despite all methodological efforts made in the last three decades, Western instruction grounds on traditional principles. Most educational programs follow theories that are mentalistic, i.e., they separate the mind from the body. At school, learners sit, watch, listen, and write. The aim of this paper is to present embodied learning as an alternative to mentalistic education. Similarly, this paper wants to describe embodied learning from a neuroscientific perspective. After a brief historical overview, I will review studies highlighting the behavioral effectiveness of embodied instruction in second language learning, mathematics and spatial thinking. On this base, I will discuss some of the brain mechanisms driving embodied learning and describe its advantages, clearly pleading in favor of instructional practice that reunites body and mind.
Embodied Learning: Why at School the Mind Needs the Body
Despite all methodological efforts made in the last three decades, Western instruction grounds on traditional principles. Most educational programs follow theories that are mentalistic, i.e., they separate the mind from the body. At school, learners sit, watch, listen, and write. The aim of this paper is to present embodied learning as an alternative to mentalistic education. Similarly, this paper wants to describe embodied learning from a neuroscientific perspective. After a brief historical overview, I will review studies highlighting the behavioral effectiveness of embodied instruction in second language learning, mathematics and spatial thinking. On this base, I will discuss some of the brain mechanisms driving embodied learning and describe its advantages, clearly pleading in favor of instructional practice that reunites body and mind.
Frontiers in Psychology, Special Topic Neuroeducation: Translating Lab Insights into Classroom Practice, 1.10.19
Depth of encoding through observed gestures in foreign language word learning
Word learning is basic to foreign language acquisition, however time consuming and not always successful. Empirical studies have shown that traditional (visual) word learning can be enhanced by gestures. The gesture benefit has been attributed to depth of encoding. Gestures can lead to depth of encoding because they trigger semantic processing and sensorimotor enrichment of the novel word. However, the neural underpinning of depth of encoding is still unclear. Here, we combined an fMRI and a behavioral study to investigate word encoding online. In the scanner, participants encoded 30 novel words of an artificial language created for experimental purposes and their translation into the subjects’ native language. Participants encoded the words three times visually, audiovisually, and by additionally observing semantically related gestures performed by an actress. Hemodynamic activity during word encoding revealed the recruitment of cortical areas involved in stimulus processing. In this study, depth of encoding can be spelt out in terms of sensorimotor brain networks that grow larger the more sensory modalities are linked to the novel word. Word retention outside the scanner documented a positive effect of gestures in a free recall test in the short-term.
Front. Psychol. – Language Sciences, 8 January, 2019 DOI: 10.3389/fpsyg.2019.00033 Link zum Download
The impact of gestures on Formal Language learning and its neural correlates: A study proposal
This pilot study reports about the impact of gestures on learning a formal language like Python. The aim of this research-in-progress is to find out if memory performance will benefit from the coupling of gestures and words in the learning phase. Previous research has demonstrated that gestures accompanying speech have an impact on memory for verbal information. This is the first study applying the body-mind concept to formal language learning. We introduce the study design and the results of one person.
Information Systems and Neuroscience Springer, Cham, Januyary 20198, Link zum Download
Guided embodiment and potential applications of tutor systems in language instruction and rehabilitation
Intelligent tutor systems (ITSs) in mobile devices take us through learning tasks and make learning ubiquitous, autonomous and at low cost (Nye, 2015). In this paper, we describe guided embodiment as an ITS essential feature for second language learning (L2) and aphasia rehabilitation (ARe) that enhances efficiency in the learning process. In embodiment, cognitive processes, here specifically language (re)learning are grounded in actions and gestures (Dijkstra & Post, 2015; Fischer & Zwaan, 2008; Pecher & Zwaan, 2005). In order to guide users through embodiment, ITSs must track action and gesture, and give corrective feed-back to achieve the users’ goals. Therefore, sensor systems are essential to guided embodiment. In the next sections, we describe sensor systems that can be implemented in ITS for guided embodiment.
Frontiers in Psychology, Cognitive Aspects of Interactive Technology Use: From Computers to Smart Objects and Autonomous Agents, June 2018, Link zum Download
Enrichment effects of metaphoric gestures and pictures on abstract words in a second language
Laboratory research has demonstrated that multisensory enrichment promotes verbal learning in a foreign language (L2). Enrichment can be done in various ways, e.g., by adding a picture that illustrates the L2 word’s meaning or by the learner performing a gesture to the word (enactment). Most studies have tested enrichment on concrete but not on abstract words. Unlike concrete words, the representation of abstract words is deprived of sensory-motor features. This has been addressed as one of the reasons why abstract words are difficult to remember. Here, we ask whether a brief enrichment training by means of pictures and by self-performed gestures also enhances the memorability of abstract words in L2. Further, we explore which of these two enrichment strategies is more effective. Twenty young adults learned 30 novel abstract words in L2 according to three encoding conditions: 1) reading, 2) reading and pairing the novel word to a picture, and 3) reading and enacting the word by means of a gesture. We measured memory performance in free and cued recall tests, as well as in a visual recognition task. Words encoded with gestures were better remembered in the free recall in the native language (L1). When recognizing the novel words, participants made less errors for words encoded with gestures compared to words encoded with pictures. The reaction times in the recognition task did not differ across conditions. The present findings support, even if only partially, the idea that enactment promotes learning of abstract words and that it is superior to enrichment by means of pictures even after short training.
Frontiers in Psychology, December 2017, Link zum Download
Recently learned foreign abstract and concrete nouns are represented in distinct cortical networks similar to the native language
In the native language, abstract and concrete nouns are represented in distinct areas of the cerebral cortex. Currently, it is unknown whether this is also the case for abstract and concrete nouns of a foreign language. Here, we taught adult native speakers of German 45 abstract and 45 concrete nouns of a foreign language. After learning the nouns for 5 days, participants performed a vocabulary translation task during functional magnetic resonance imaging. Translating abstract nouns in contrast to concrete nouns elicited responses in regions that are also responsive to abstract nouns in the native language: the left inferior frontal gyrus and the left middle and superior temporal gyri. Concrete nouns elicited larger responses in the angular gyri bilaterally and the left parahippocampal gyrus than abstract nouns. The cluster in the left angular gyrus showed psychophysiological interaction (PPI) with the left lingual gyrus. The left parahippocampal gyrus showed PPI with the posterior cingulate cortex. Similar regions have been previously found for concrete nouns in the native language. The results reveal similarities in the cortical representation of foreign language nouns with the representation of native language nouns that already occur after 5 days of vocabulary learning. Furthermore, we showed that verbal and enriched learning methods were equally suitable to teach foreign abstract and concrete nouns.
Why Your Body Can Jog Your Mind
Philosophical tradition influences the way we think about body and mind (Rogers, 1936). We have a body to move around and a mind to think and to learn (Descartes, 1637). At school, we sit, listen, and read, but we are not allowed to move. However, cognitive science has shown that our body is tightly linked to the mind (Wilson, 2002; Pecher and Zwaan, 2005; Gärtner, 2013). In this paper, we provide evidence that better learning is achieved if the body supports the mind. We review studies showing how physical movement impacts brain functions and structures, and why physical movement is beneficial to learning. Thereafter, we explain how the body supports the mind in difficult cognitive tasks. Finally, we discuss how the body can be employed as a tool in second language learning and mathematics.
Frontiers in Educational Psychology, March 2017, Link zum Download
Virtual Reality as en embodied tool to enhance episodic memory in elderly
In the last decade, embodiment has dramatically influenced our conception of cognition. In this new frame, episodic memory and particularly memory decline have been reinterpreted. Interventions supporting memory in the aging population address the connection between mind and body. Here, we discuss the use of Virtual Reality (VR) as an innovative tool to support episodic memory in older adults.
Frontiers in Educational Psychology, November 2016, Link zum Download
Exploring the Neural Representation of Novel Words Learned through Enactment in a Word Recognition Task
Vocabulary learning in a second language is enhanced if learners enrich the learning experience with self-performed iconic gestures. This learning strategy is called enactment. Here we explore how enacted words are functionally represented in the brain and which brain regions contribute to enhance retention. After an enactment training lasting 4 days, participants performed a word recognition task in the functional Magnetic Resonance Imaging (fMRI) scanner. Data analysis suggests the participation of different and partially intertwined networks that are engaged in higher cognitive processes, i.e., enhanced attention and word recognition. Also, an experience-related network seems to map word representation. Besides core language regions, this latter network includes sensory and motor cortices, the basal ganglia, and the cerebellum. On the basis of its complexity and the involvement of the motor system, this sensorimotor network might explain superior retention for enactment.
Frontiers in Educational Psychology, Juni 2016, Link zum Download
Brief Multisensory Training Enhances Second Language Vocabulary Acquisition in Both High and Low Performers
Research in the field of vocabulary acquisition has demonstrated that enriching novel words with sensorimotor information enhances memory outcome compared to reading. However, it has been asserted that enrichment might exceed the cognitive load of low performers and therefore be detrimental to them. Here, in a brief training, thirty-two subjects learned thirty novel items of a foreign language according to three conditions: (1) reading, (2) reading and listening, (3) reading and listening and watching an actress performing a gesture semantically related to the words. Conditions (2) and (3) enriched the baseline (1) with multisensory information. Memory performance was assessed through written tests immediately after learning. Results indicate that both high and low performers benefit from sensorimotor learning. The significant interaction between group and method in one of the tests shows that low performers learn better through enrichment than by only reading the words. Implications for education are discussed.
International Journal of Learning, Teaching and Educational Research, März 2016, Link zum Download
Learning Styles and Vocabulary Acquisition in Second Language: How the Brain Learns
In recent years, foreign language education has been focussing on learning styles. However, despite the quantity of articles and practice books, websites on the topic, and investment in teacher training, there is no empirical evidence for the existence of learning styles. Furthermore, if one agrees that it is the brain that learns, there should be indicators in the brain for the existence of learning styles, anatomically, and/or functionally. This is not the case. In this paper, the validity and reliability of tests assessing learning styles are questioned. Thereafter, following on basics of cognitive neuroscience and experimental evidence it is argued that the natural way for the brain to learn words is by collecting multiple sensory and sensorimotor experiences. In fact, evidence-based literature in the domain of vocabulary acquisition demonstrates that the inclusion of multiple modalities leads to best results. Impoverished linguistic input by allowing only one modality, for example only acoustic or visual input—the so called learning style (Pashler et al., 2008) of the student—reduces the chances of acquiring words. Also, the article briefly outlines brain related factors that lead to high performance in vocabulary learning.
Frontiers in Educational Psychology, Nomberber 2015, Link zum Download
Visual and Motor Cortices Differentially Support the Translation of Foreign Language Words
At present, it is largely unclear how the human brain optimally learns foreign languages. We investigated teaching strategies that utilize complementary information (“enrichment”), such as pictures [ 1 ] or gestures [ 2 ], to optimize vocabulary learning outcome. We found that learning while performing gestures was more efficient than the common practice of learning with pictures and that both enrichment strategies were better than learning without enrichment (“verbal learning”). We tested the prediction of an influential cognitive neuroscience theory that provides explanations for the beneficial behavioral effects of enrichment: the “multisensory learning theory” [ 3, 4 ] attributes the benefits of enrichment to recruitment of brain areas specialized in processing the enrichment. To test this prediction, we asked participants to translate auditorily presented foreign words during fMRI. Multivariate pattern classification allowed us to decode from the brain activity under which enrichment condition the vocabulary had been learned. The visual-object-sensitive lateral occipital complex (LOC) represented auditory words that had been learned with pictures. The biological motion superior temporal sulcus (bmSTS) and motor areas represented auditory words that had been learned with gestures. Importantly, brain activity in these specialized visual and motor brain areas correlated with behavioral performance. The cortical activation pattern found in the present study strongly supports the multisensory learning theory [ 3, 4 ] in contrast to alternative explanations. In addition, the results highlight the importance of learning foreign language vocabulary with enrichment, particularly with self-performed gestures.
Current Biology, Januar 2015, Link zum Download
(Wenn Sie keinen Bibliothekzugang haben, kann ich gerne den Artikel zur Verfügung stellen)
Bringing back the body into the mind: gestures enhance word learning in foreign language
Foreign language education in the twenty-first century still teaches vocabulary mainly through reading and listening activities. This is due to the link between teaching practice and traditional philosophy of language, where language is considered to be an abstract phenomenon of the mind. However, a number of studies have shown that accompanying words or phrases of a foreign language with gestures leads to better memory results. In this paper, I review behavioral research on the positive effects of gestures on memory. Then I move to the factors that have been addressed as contributing to the effect, and I embed the reviewed evidence in the theoretical framework of embodiment. Finally, I argue that gestures accompanying foreign language vocabulary learning create embodied representations of those words. I conclude by advocating the use of gestures in future language education as a learning tool that enhances the mind.
Frontiers in Psychology Special Issue on “Pedagogical Psychology: Beyond the 21st Century” Link zum Download
Imitation of a Pedagogical Agent’s Gestures Enhances Memory for Words in Second Language
Pedagogical agents (PAs) are virtual characters in computer-based learning environments. PAs can train humans in various domains. Here, a PA cues subjects to learn vocabulary items through enactment, i.e., to perform an illustrative gesture while learning a word. It is well known that enactment impacts memory. Also, imitation is a natural mechanism driving learning. Combining both enactment and imitation could improve memory even more. In a within-subjects study, 44 school children learned 45 vocabulary items according to three conditions: an audio-visual baseline, an observation condition (participants watched the PA during enactment) and an imitation condition (participants imitated the PA’s gestures). We documented learning progress by cued recall tests. Over four days, we found that, compared to the baseline and to mere observation, imitation of enactment significantly enhanced memory for words in the foreign language.
Science Journal of Education, Vol. 2, Number 5, Pages 141-145, October 2014 Link zum Download
Do children accept virtual agents as foreign language trainers?
Virtual (animated software) agents can train humans in vocabulary learning. This has been successfully tested with adults and more recently also with children. However, the question of how children perceive a virtual agent training them had not been investigated. Here we invited 25 children to evaluate their perception of a virtual and a human trainer who presented written words in a foreign language on videos; both the human trainer and the virtual agent additionally performed a semantically related gesture for each word. Subjects rated the trainers for features related to gestures and for their “personalities”. Subjects found human gestures better and gave the human trainer higher sympathy scores; however, the overall difference between their perception of virtual and human trainers was not significant.
International Journal of Learning, Teaching and Educational Research, Vol. 7, Number 1, Pages 131-137, August 2014 Link zum Download
A Bizarre Virtual Trainer Outperforms a Human Trainer in Foreign Language Word Learning
In this paper we investigate the effect that a human trainer and a pedagogical virtual agent have on the memory for words in a foreign language (L2). In a recent study on L2 word learning, Bergmann and Macedonia (2013) cued participants to memorize novel words both audiovisually and by performing additional gestures. The gestures were performed by both a human and a virtual trainer. In some of the tests, the virtual agent had a greater positive influence on memory performance than the human trainer. In order to determine why the agent was a better trainer than the human, we invited 18 naïve subjects to rate the gestures performed by both trainers. Furthermore, we asked participants to evaluate their perception of the human and the agent. We hypothesized that the gestures performed by the agent would be more peculiar than those of the human and possibly attract greater attention. We also hypothesized that the agent’s personality might be more appealing than that of the human. Results show that the agent’s gestures were perceived as less natural than those of the human. The perception of both trainers as “personalities” did not differ, with the exception of a few traits for which the human trainer was considered to be better. Altogether, because of the peculiar gestures it made and because of its looks, the agent may have been perceived as bizarre. Therefore, he might have induced the bizarreness effect in the memory for words.
International Journal of Computer Science and Artificial Intelligence, Vol. 4, Number 2, Pages 24-34, June 2014 Link zum Download
Long-term effects of gestures on memory for foreign language words trained in the classroom
Language and gesture are viewed as highly interdependent systems. Besides supporting communication, gestures also have an impact on memory for verbal information compared to pure verbal encoding in native but also in foreign language learning. In a within-subject longitudinal study lasting 14 months, we tested the use of gestures in the classroom with the experimenter presenting the items to be acquired. Participants learned thirty-six words distributed across two training conditions: In the audio-visual condition subjects read, heard, and spoke the words; in the gestural condition subjects additionally accompanied the words with symbolic gestures. Memory performance was assessed through cued native- to-foreign translation tests at five time points. The results show that gestures significantly enhance vocabulary learning in quantity and over time. The findings are discussed in terms of Klimesch’s Connectivity Model (CM) of information processing. Thereafter, a code, a word, is better integrated into long-term memory if it is deep, that is, if it comprises many interconnected components.
Mind, Brain, and Education, Vol. 8, Number 2, Pages 74-88, June 2014 Link zum Download
Intelligent Virtual Agents as Language Trainers Facilitate Multilingualism
In this paper we introduce a new generation of language trainers: intelligent virtual agents (IVAs) with human appearance and the capability to teach foreign language vocabulary. We report results from studies that we have conducted with Billie, an IVA employed as a vocabulary trainer, as well as research findings on the acceptance of the agent as a trainer by adults and children. The results show that Billie can train humans as well as a human teacher can and that both adults and children accept the IVA as a trainer. The advantages of IVAs are multiple. First, their teaching methods can be based on neuropsychological research findings concerning memory and learning practice. Second, virtual teachers can provide individualized training. Third, they coach users during training, are always supportive, and motivate learners to train. Fourth, agents will reside in the user’s mobile devices and thus be at the user’s disposal everywhere and anytime. Agents in apps will make foreign language training accessible to anybody at low cost. This will enable people around the world, including physically, financially and geographically disadvantaged persons, to learn a foreign language and help to facilitate multilingualism.
Frontiers in Psychology Special Issue on “Pedagogical Psychology: Beyond the 21st Century” Link zum Download
Pronunciation in foreign language: How to train? Effects of different kinds of training in perception and production of the Italian consonant / λ/ by adult German learners
This study investigates the role of perception and sensory motor learning on speech production in L2. Compared to natural language learning, acoustic input in formal adult instruction is deprived of multiple sensory motor cues and lacks the imitation component. Consequently, it is possible that inaccurate pronunciation results from training. Inaccuracy manifests itself in the use of suppletive sounds. For the Italian phoneme /λ/ [gl] like in “paglia” (It. straw), native Germans often produce the suppletive phoneme /l/. The Motor Theory of Speech Perception provides theoretical underpinning for the interdependency between perception and production: Thereafter, speech is perceived by reference to the articulator gestures necessary to produce it. Furthermore, imitation is a mechanism driving learning, particularly language acquisition. Accordingly, we hypothesized that training with sensory motor cues together with imitation induces the development of articulatory motor programs. They enable learners to accurately discriminate and pronounce the Italian phoneme /λ/. In a between subjects experiment, we trained 49 native Germans to perceive and produce minimal pairs of syllables containing /λ/ and /l/ embedded in vocalic contexts. Participants were randomly divided into three subgroups according to the following training conditions: 1) acoustic/imitation (AI), 2) audiovisual/motor task (AVM), and 3) audiovisual/imitation (AVI). The stimuli, consisting of audio files and video clips, were presented in two training blocks totalling 408 stimuli and responses per participant. Responses in stimulus discrimination and reproduction were recorded. The results show that participants discriminated both sounds /λ/ and /l/, pre- and post-training equally well. Sound discrimination reached ceiling, independently of the training participants had received. However, training did not improve production accuracy which persisted in being inaccurate until the end of the experiment. We attribute the results in production to insufficient training, and we discuss the findings in terms of age-related resiliency in L2 learning.
Journal of Education and Training Studies, 2014, Volume 2, Issue 1, Pages 53–62, January 2014 Link zum Download
Learning a Second Language Naturally -The Voice Movement Icon Approach
Second language (L2) instruction greatly differs from natural input during native language (L1) acquisition. Whereas a child collects sensorimotor experience while learning novel words, L2 employs primarily reading, writing and listening and comprehension. We describe an alternative proposal that integrates the body into the learning process: the Voice Movement Icon (VMI) approach. A VMI consists of a word that is read and spoken in L2 and synchronously paired with an action or a gesture. A VMI is first performed by the language trainer and then imitated by the learners. Behavioral experiments demonstrate that words encoded through VMIs are easier to memorize than audio-visually encoded words and that they are better retained over time. The reasons why gestures promote language learning are manifold. First, we focus on language as an embodied phenomenon of cognition. Then we review evidence that gestures scaffold the acquisition of L1. Because VMIs reconnect language learning with the body, they can be considered as a more natural tool for language instruction than audio-visual activities.
Journal of Educational and Developmental Psychology, 2013, Volume 3, Issue 2, Pages 102–116, November 2013 Link zum Download
Three Good Reasons why Foreign Language Instructors Need Neuroscience
In the last three decades, brightly colored images of neural activity in the media have enhanced interest in brain functions. As a result, people have become more aware of the fact that cognition is not simply a black box. Instead, it consists of dynamic systems and mechanisms that respond to stimuli and which educators can exert influence on. In this paper the author shows how three areas of neuroscience are of crucial importance for L2 instruction. First a brief overview of studies is provided which show that words are not just encoded as labels for concepts, but rather are stored in brain networks together with the perceptions and actions that are relevant to them. From this point of departure the author goes on to show that sensorimotor encoding is the natural way to learn L2 vocabulary. Secondly, the concept of mirror neurons is introduced. They represent the neurobiological basis for imitation and are therefore essential for learning, particularly for the acquisition of languages. Thirdly, the author deals with brain maturation and elucidates its effects on L2-learning proficiency at different ages. These three lines of evidence suggest that knowledge of brain mechanisms is crucial for language teachers. Briefly stated, the aim of this paper is to help to correlate language instruction with neuroscience.
Journal of Studies in Education, , Volume 3, Issue 4, Pages 1-20, November 2013 Link zum Download
A Virtual Agent as Vocabulary Trainer: Iconic Gestures Help to Improve Learners’ Memory Performance
An important and often laborious task in foreign language acquisition is vocabulary learning. Research has repeatedly demonstrated that performing iconic gestures together with novel words has a beneficial effect on learning performance. Can these findings be transferred onto virtual agents applied in gesture-supported vocabulary training? We present a study investigating whether iconic gestures performed by a virtual agent and imitated by learners have an impact on verbal memory for words in a foreign language. In a within-subject design we compared participants’ memory performance achieved with the help of a virtual agent and those achieved with the help of a human trainer regarding both short-term learning effects and long-term decay effects. The overall results demonstrate improved memory scores when participants learned with a virtual agent. Especially high performers could profit from gesture-supported training with a virtual agent.
PDF Pressemitteilung JKU und Pressespiegel Link zum Download
Gestures Enhance Foreign Language Learning
Language and gesture are highly interdependent systems that reciprocally influence each other. For example, performing a gesture when learning a word or a phrase enhances its retrieval compared to pure verbal learning. Although the enhancing effects of co-speech gestures on memory are known to be robust, the underlying neural mechanisms are still unclear. Here, we summarize the results of behavioral and neuroscientific studies. They indicate that the neural representation of words consists of complex multimodal networks connecting perception and motor acts that occur during learning. In this context, gestures can reinforce the sensorimotor representation of a word or a phrase, making it resistant to decay. Also, gestures can favor embodiment of abstract words by creating it from scratch. Thus, we propose the use of gesture as a facilitating educational tool that integrates body and mind.
Body in Mind: How Gestures Empower Foreign Language Learning
It has previously been demonstrated that enactment (i.e., performing representative gestures during encoding) enhances memory for concrete words, in particular action words. Here, we investigate the impact of enactment on abstract word learning in a foreign language. We further ask if learning novel words with gestures facilitates sentence production. In a within-subjects paradigm, participants first learned 32 abstract sentences from an artificial corpus conforming with Italian phonotactics. Sixteen sentences were encoded audiovisually. Another set of 16 sentences was also encoded audiovisually, but, in addition, each single word was accompanied by a symbolic gesture. Participants were trained for 6 days. Memory performance was assessed daily using different tests. The overall results support the prediction that learners have better memory for words encoded with gestures. In a transfer test, participants produced new sentences with the words they had acquired. Items encoded through gestures were used more frequently, demonstrating their enhanced accessibility in memory. The results are interpreted in terms of embodied cognition. Implications for teaching and learning are suggested.
The impact of iconic gestures on foreign language word learning and its neural substrate
Vocabulary acquisition represents a major challenge in foreign language learning. Research has demonstrated that gestures accompanying speech have an impact on memory for verbal information in the speakers’ mother tongue and, as recently shown, also in foreign language learning. However, the neural basis of this effect remains unclear. In a within-subjects design, we compared learning of novel words coupled with iconic and meaningless gestures. Iconic gestures helped learners to significantly better retain the verbal material over time. After the training, participants’ brain activity was registered by means of fMRI while performing a word recognition task.
Neural correlates of high performance in foreign language vocabulary learning
Learning vocabulary in a foreign language is a laborious task which people perform with varying levels of success. Here, we investigated the neural underpinning of high performance on this task. In a within-subjects paradigm, participants learned ninety-two vocabulary items under two multimodal conditions: One condition paired novel words with iconic gestures, and the other with meaningless gestures. Memory performance was assessed through single word translation tests. High performers consistently learned more items than low performers, regardless of the training condition, the time and the difficulty of the task.
Games and foreign language teaching
Active spoken mastery of a foreign language all too often remains an illusive wish on the part of language learners. There is a tendency to seek the causes of non-fluency and accurate speech outside the classroom, for example, too little involvement, interest and time investment on the part of learners. In this article Manuela Macedonia asserts that the problem is attributed primarily to the type of exercises that are employed to process foreign language input. Traditional transmission of morphology and syntax by way of rules, and practising such rules via written exercises, does not lead to spoken language, for with this type of practice the retrieval of learned material is too slow and often incomplete to enable successful speech. While games in language and SEN instruction are not new, in this article their targeted usage based on cognitive/neurological evidence is proposed in order to proceduralise declarative knowledge and thereby to elevate accuracy and fluency to a level that enables real-time speech.