The Effect of Attention on Audiovisual Speech Integration

Abstract

Abstract: The present research used the square and the triangle as visual stimuli and spoken words as auditory stimuli to investigate the effect of attention on multisensory speech integration. The visual and auditory stimuli were semantically congruent or incongruent. The results from a discrimination task showed that: 1) In the condition of divided attention, participants required less time to react to audiovisual semantic-congruent targets than other types of targets; that is the redundant signal effect; 2) According to the race model, the redundant signal effect in the condition of divided attention originated from the multisensory integration. To conclude, the multisensory integration occurred when the attention was divided between visual and auditory modality only in the semantic-congruent audiovisual stimuli.

Key words: attention, multisensory integration, semantic congruence, race model

摘要： 采用2×3的被试内实验设计，将注意条件和目标刺激类型作为实验变量，考察了指向不同感觉通道的注意对视听语义整合加工的不同影响。结果发现，只有在同时注意视觉和听觉刺激时，被试对语义一致的视听刺激反应最快，即产生冗余信号效应。而在选择性注意一个感觉通道时，语义一致的视听刺激并不具有加工优势。进一步分析发现，在同时注意视觉和听觉时语义一致视听刺激的加工优势源自于其视觉和听觉成分产生了整合。也就是说，只有在同时注意视觉和听觉时，语义一致视听刺激才会产生整合，语义不一致视听刺激不会产生整合。而在选择性注意一个感觉通道时，不论语义是否一致，视听刺激均不会产生整合。

关键词: 注意, 多感觉整合, 语义联结, 竞争模型

CLC Number:

B842.1

LYU Yong, GU Jiyou. The Effect of Attention on Audiovisual Speech Integration[J]. Studies of Psychology and Behavior, 2017, 15(5): 587-592.

吕勇, 顾吉有. 注意对视听语义整合加工的影响[J]. 心理与行为研究, 2017, 15(5): 587-592.

References

刘强.(2010). 多感觉整合脑机制研究 (博士学位论文). 西南大学.
孙远路, 胡中华, 张瑞玲, 寻茫茫, 刘强, 张庆林. (2011). 多感觉整合测量范式中存在的影响因素探讨. 心理学报, 43(11), 1239-1246.
Frassinetti, F., Bolognini, N., & Làdavas, E. (2002). Enhancement of visual perception by crossmodal visuo-auditory interaction. Experimental Brain Research, 147, 332-343.
Friston, K. (2010). The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2), 127-138.
Haxby, J. V., Horwitz, B., Ungerleider, L. G., Maisog, J. M., Pietrini, P., & Grady, C. L. (1994). The functional organization of human extrastriate cortex: A PET-rCBF study of selective attention to faces and locations. Journal of Neuroscience, 14(11), 6336-6353.
Johnson, J. A., & Zatorre, R. J. (2005). Attention to simultaneous unrelated auditory and visual events: Behavioral and neural correlates. Cerebral Cortex, 15(10), 1609-1620.
Johnson, J. A., & Zatorre, R. J. (2006). Neural substrates for dividing and focusing attention between simultaneous auditory and visual events. NeuroImage, 31(4), 1673-1681.
Kawashima, R., O'Sullivan, B. T., & Roland, P. E. (1995). Positron-emission tomography studies of cross-modality inhibition in selective attentional tasks: Closing the “mind's eye”. Proceedings of the National Academy of Sciences of the United States of America, 92(13), 5969-5972.
Koelewijn, T., Bronkhorst, A., & Theeuwes, J. (2010). Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta Psychologica, 134(3), 372-384.
Laurienti, P. J., Burdette, J. H., Wallace, M. T., Yen, Y. F., Field, A. S., & Stein, B. E. (2002). Deactivation of sensory-specific cortex by cross-modal stimuli. Journal of Cognitive Neuroscience, 14(3), 420-429.
Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158(4), 405-414.
Lovelace, C. T., Stein, B. E., & Wallace, M. T. (2003). An irrelevant light enhances auditory detection in humans: A psychophysical analysis of multisensory integration in stimulus detection. Cognitive Brain Research, 17(2), 447-453.
Miller, J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14(2), 247-279.
Miller, J. (1986). Timecourse of coactivation in bimodal divided attention. Perception & Psychophysics, 40(5), 331-343.
Mittag, M., Alho, K., Takegata, R., Makkonen, T., & Kujala, T. (2013). Audiovisual attention boosts letter-speech sound integration. Psychophysiology, 50(10), 1034-1044.
Raab, D. H. (1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences, 24, 574-590.
Su, Y. H. (2014). Content congruency and its interplay with temporal synchrony modulate integration between rhythmic audiovisual streams. Frontiers in Integrative Neuroscience, 8, 92.
Talsma, D. (2015). Predictive coding and multisensory integration: An attentional account of the multisensory mind. Frontiers in Integrative Neuroscience, 9, 19.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14(9), 400-410.
Talsma, D., & Woldorff, M. G. (2005). Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17(7), 1098-1114.
Ten Oever, S., Sack, A. T., Wheat, K. L., Bien, N., & van Atteveldt, N. (2013). Audio-visual onset differences are used to determine syllable identity for ambiguous audio-visual stimulus pairs. Frontiers in Psychology, 4, 331.
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598-607.
Vatakis, A., & Spence, C. (2007). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics, 69(5), 744-756.

[1]	YAN Qin, ZHANG Qingfang. Temporal Processing of Rhythm in Speech: Evidence from a Meta-Analysis [J]. Studies of Psychology and Behavior, 2022, 20(5): 607-614.
[2]	LI Shiyi, XIE Yanfeng, ZHAO Guang, BAI Xuejun. The Influence of Media Multitasking Experience on Implicit Memory in Different Attention Patterns [J]. Studies of Psychology and Behavior, 2022, 20(4): 433-440.
[3]	LIU Yanyan, LI Jie, ZHAO Qichao, CHEN Beibei, ZHANG Yu. Diffusion of Inhibition of Return in 3D Dynamic and Static Scenes [J]. Studies of Psychology and Behavior, 2022, 20(1): 22-28.
[4]	SU Mingyu, WANG Guangming. STEAM Training Can Improve Pupils’ Attentional Inhibition and Attentional Concentration Ability [J]. Studies of Psychology and Behavior, 2021, 19(5): 701-705.
[5]	WU-REN Qimuge, ZHAO Aoxue, WANG Yuhan, ZHUANG Xiangling, MA Guojie. Visual Field Preference and Reference Strategy in Comparative Visual Search [J]. Studies of Psychology and Behavior, 2021, 19(2): 166-171.
[6]	GAO Chao, LI Shiyi, ZHANG Peng, ZHAO Guang. Modulation of Depth Perception on Contextual Cueing Effect Based on Binocular Disparity [J]. Studies of Psychology and Behavior, 2021, 19(1): 16-21,29.
[7]	JIAO Jiangli, LIU Yi, WEN Suxia. Bilingual Advantage of Bilingualism in Emotional Attention Bias [J]. Studies of Psychology and Behavior, 2020, 18(5): 590-595.
[8]	CHE Xiaowei, WANG Kaixuan, SHANGGUAN Mengqi, LI Shouxin. The Representation of Attention Template in Visual Working Memory: An EROS Study [J]. Studies of Psychology and Behavior, 2020, 18(3): 297-303.
[9]	WU Huifen, LI Xiaona, ZHANG Qiong, TANG Yichun, LIU Huashan. You Own Something Because of Paying Attention to Them: Exploring the Internal Mechanism of Mere Ownership Effect [J]. Studies of Psychology and Behavior, 2020, 18(2): 161-167,175.
[10]	LI Hengping, HE Shengxi, LIU Zhengkui. ERPs Study on the Effect of Fast Reading Training on Attention Based on Visual Beam Search [J]. Studies of Psychology and Behavior, 2020, 18(1): 78-84.
[11]	YU Yongju. Effects of Anxiety and Depression on Attentional Bias: The Mediating Role of Attentional Control [J]. Studies of Psychology and Behavior, 2020, 18(1): 121-127.
[12]	ZENG Qing, ZHENG XiFu. The Time-course Difference of the Attention Paid by High Disgust Sensitivity and Low Disgust Sensitivity Individuals when Processing Different Emotional Pictures: An Event-related Potentials Study [J]. Studies of Psychology and Behavior, 2019, 17(6): 743-749.
[13]	WANG Tingzhao, WANG Fengli, AO Yongqian, WANG Li. A Comparative Study on the Effects of Emotional Stimulation on Attention Blink in Deaf and Normal-Hearing People under the RSVP Paradigm [J]. Studies of Psychology and Behavior, 2019, 17(6): 803-809.
[14]	WU Lei, ZHAO Boyao, ZHAN Zhao, FAN Bijuan, ZHAO Yaping. The Effect of Mindful Parenting from Parents on Adolescent Depression and its Mechanism [J]. Studies of Psychology and Behavior, 2019, 17(4): 561-568.
[15]	LIU Li, LI Shiyi, YUE Junbing, BAI Xuejun. Location-value Association Induces Attentional Capture and Underlying Mechanism [J]. Studies of Psychology and Behavior, 2019, 17(3): 289-295.