古诗多媒体空间邻近效应的神经基础—来自fNIRS的证据

doi:10.12139/j.1672-0628.2022.06.007

摘要/Abstract

摘要：

为了考察存在冗余的多媒体图文之间是否存在空间邻近效应，并探究其产生机制，本研究采用了功能性近红外光谱成像技术（fNIRS），监测并分析了35名普通大学生在整合格式和分散格式下完成古诗多媒体阅读时的大脑皮层活动。学习结果通过保留成绩、迁移成绩和总成绩来衡量，认知负荷主观测量指标为自我报告心理努力。结果发现：与分散格式相比，整合格式下学习效果更好，且背外侧前额叶皮质、左侧额极有更大的神经激活。结果证明，即使图文存在冗余，古诗多媒体仍然存在空间邻近效应，且背外侧前额叶皮质和左侧额极的参与可能是古诗多媒体空间邻近效应的神经基础，为多媒体学习认知理论对空间邻近效应的假设提供了新的证据。

关键词: 空间邻近效应, 多媒体学习, 背外侧前额叶, 功能性近红外光谱成像技术

Abstract:

The present study explored the influence of redundancy on spatial contiguity effect during multimedia learning as well as its mechanism. We collected and analyzed the cerebral cortex activities, which were measured by functional near-infrared spectroscopy (fNIRS). Participants were 35 college students, who were randomly assigned to integrated conditions or separated conditions. Learning outcomes were measured by scores on retention, transfer, and total test. The subjective measure of the learning process was self-reported cognitive load ratings. Results showed that: 1) text-picture integration facilitated better comprehension than a separated presentation, and 2) dorsolateral prefrontal cortex (DLPFC) and left frontopolar cortex (FPC) tended to be more activated when learning under integrated conditions. The findings suggest that even though there is redundancy, the spatial contiguity effect still exists. The involvement of DLPFC and left FPC might be the neural basis of the spatial contiguity effect. The current findings provide new evidence for the theoretical hypothesis of the spatial contiguity effect of the Cognitive Theory of Multimedia Learning (CTML).

Key words: spatial contiguity effect, multimedia learning, dorsolateral prefrontal cortex, functional near-infrared spectroscopy (fNIRS)

中图分类号:

B842

陈夏妮, 章鹏, 严世振, 贾丽娜, 马晓博, 俞怡帆, 金花. 古诗多媒体空间邻近效应的神经基础—来自fNIRS的证据[J]. 心理与行为研究, 2022, 20(6): 760-767.

Xiani CHEN, Peng ZHANG, Shizhen YAN, Lina JIA, Xiaobo MA, Yifan YU, Hua JIN. The Neural Basis of Spatial Contiguity Effect in Chinese Ancient Poetry: Evidence from fNIRS[J]. Studies of Psychology and Behavior, 2022, 20(6): 760-767.

图/表 7

参考文献

	雷震, 毕蓉, 莫李澄, 于文汶, 张丹丹外显和内隐情绪韵律加工的脑机制: 近红外成像研究. 心理学报, 2021, 53 (1): 15- 25.
	田媛, 亓栀, 黄湘琳, 向虹钰, 汪颖社会线索促进在线学习的认知神经机制. 电化教育研究, 2021, 42 (2): 63- 69.
	王福兴, 杨晓梦, 范颖平, 胡祥恩空间邻近对7~9岁儿童图文阅读的影响: 来自眼动的证据. 心理与行为研究, 2020, 18 (4): 503- 509.
	杨海波, 刘和珺, 章鹏, 李量掩蔽刺激对目标识别加工的作用: 来自fNIRS的证据. 心理学报, 2019, 51 (11): 1187- 1197.
	Basso Moro, S., Cutini, S., Ursini, M. L., Ferrari, M., & Quaresima, V. Prefrontal cortex activation during story encoding/retrieval: A multi-channel functional near-infrared spectroscopy study. Frontiers in Human Neuroscience, 2013, 7, 925.
	Cierniak, G., Scheiter, K., & Gerjets, P. (2009). Expertise reversal in multimedia learning: Subjective load ratings and viewing behavior as cognitive process indicators. Paper presented at the Proceedings of the 31st Annual Conference of the Cognitive Science Society, Austin, TX.
	Csipo, T., Lipecz, A., Mukli, P., Bahadli, D., Abdulhussein, O., Owens, C. D., … Yabluchanskiy, A. Increased cognitive workload evokes greater neurovascular coupling responses in healthy young adults. PLoS ONE, 2021, 16 (5): e0250043. DOI
	de Koning, B. B., Rop, G., & Paas, F. Learning from split-attention materials: Effects of teaching physical and mental learning strategies. Contemporary Educational Psychology, 2020, 61, 101873. DOI
	Fishburn, F. A., Norr, M. E., Medvedev, A. V., & Vaidya, C. J. Sensitivity of fNIRS to cognitive state and load. Frontiers in Human Neuroscience, 2014, 8, 76.
	Hagoort, P., Baggio, G., & Willems, R. M. (2009). Semantic unification. In M. S. Gazzaniga (Ed.). The cognitive neurosciences (4ed, pp. 819–836). Cambridge, MA: MIT Press.
	Hagoort, P., & van Berkum, J. (2007). Beyond the sentence given. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 801–811.
	Jahani, S., Fantana, A. L., Harper, D., Ellison, J. M., Boas, D. A., Forester, B. P., & Yücel, M. A. fNIRS can robustly measure brain activity during memory encoding and retrieval in healthy subjects. Scientific Reports, 2017, 7 (1): 9533. DOI
	Johnson, C. I., & Mayer, R. E. An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 2012, 18 (2): 178- 191. DOI
	Kim, C., Kroger, J. K., Calhoun, V. D., & Clark, V. P. The role of the frontopolar cortex in manipulation of integrated information in working memory. Neuroscience Letters, 2015, 595, 25- 29. DOI
	Lau, E. F., Phillips, C., & Poeppel, D. A cortical network for semantics: (De) constructing the N400. Nature Reviews Neuroscience, 2008, 9 (12): 920- 933. DOI
	Makransky, G., Terkildsen, T. S., & Mayer, R. E. Role of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity effect. Learning and Instruction, 2019, 61, 23- 34. DOI
	Mar, R. A. The neuropsychology of narrative: Story comprehension, story production and their interrelation. Neuropsychologia, 2004, 42 (10): 1414- 1434. DOI
	Mayer, R. E. Systematic thinking fostered by illustrations in scientific text. Journal of Educational Psychology, 1989, 81 (2): 240- 246. DOI
	Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31–48). Cambridge: Cambridge University Press.
	Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge: Cambridge University Press.
	Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning (2nd ed.). New York: Cambridge University Press.
	Mayer, R. E. (2020). Multimedia learning (3rd ed.). New York: Cambridge University Press.
	Midha, S., Maior, H. A., Wilson, M. L., & Sharples, S. Measuring mental workload variations in office work tasks using fNIRS. International Journal of Human-Computer Studies, 2021, 147, 102580. DOI
	Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91(2), 358–368.
	Noble, W. S. How does multiple testing correction work. Nature Biotechnology, 2009, 27 (12): 1135- 1137. DOI
	Owens, P., & Sweller, J. Cognitive load theory and music instruction. Educational Psychology, 2008, 28 (1): 29- 45. DOI
	Paas, F. G. W. C., & van Merriënboer, J. J. G. Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 1994, 6 (4): 351- 371. DOI
	Pinti, P., Tachtsidis, I., Hamilton, A., Hirsch, J., Aichelburg, C., Gilbert, S., & Burgess, P. W. The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Annals of the New York Academy of Sciences, 2020, 1464 (1): 5- 29. DOI
	Pouw, W., Rop, G., de Koning, B., & Paas, F. The cognitive basis for the split-attention effect. Journal of Experimental Psychology: General, 2019, 148 (11): 2058- 2075. DOI
	Prabhakaran, V., Narayanan, K., Zhao, Z., & Gabrieli, J. D. E. Integration of diverse information in working memory within the frontal lobe. Nature Neuroscience, 2000, 3 (1): 85- 90. DOI
	Puma, S., Matton, N., Paubel, P. V., Raufaste, É., & El-Yagoubi, R. Using theta and alpha band power to assess cognitive workload in multitasking environments. International Journal of Psychophysiology, 2018, 123, 111- 120. DOI
	Purnell, K. N., Solman, R. T., & Sweller, J. (1991). The effects of technical illustrations on cognitive load. Instructional Science, 20(5–6), 443–462.
	Schroeder, N. L., & Cenkci, A. T. Spatial contiguity and spatial split-attention effects in multimedia learning environments: A meta-analysis. Educational Psychology Review, 2018, 30 (3): 679- 701. DOI
	Schroeder, N. L., & Cenkci, A. T. Do measures of cognitive load explain the spatial split-attention principle in multimedia learning environments? A systematic review. Journal of Educational Psychology, 2020, 112 (2): 254- 270. DOI
	Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. Cognitive architecture and instructional design. Educational Psychology Review, 1998, 10 (3): 251- 296. DOI
	Villringer, A., & Chance, B. Non-invasive optical spectroscopy and imaging of human brain function. Trends in Neurosciences, 1997, 20 (10): 435- 442. DOI
	Xu, Y. W., Lin, Q. X., Han, Z. Z., He, Y., & Bi, Y. C. Intrinsic functional network architecture of human semantic processing: Modules and hubs. NeuroImage, 2016, 132, 542- 555. DOI
	Yum, Y. N., Cohn, N., & Lau, W. K. W. Effects of picture-word integration on reading visual narratives in L1 and L2. Learning and Instruction, 2021, 71, 101397. DOI

格式	保留成绩	迁移成绩	总成绩	心理努力	学习时间（s）
整合	3.39±0.47	1.31±0.32	4.70±0.60	3.98±1.33	53.65±26.32
分散	3.20±0.49	1.27±0.38	4.47±0.60	4.29±1.75	52.86±27.95

格式	保留成绩	迁移成绩	总成绩	心理努力	学习时间（s）
整合	3.39±0.47	1.31±0.32	4.70±0.60	3.98±1.33	53.65±26.32
分散	3.20±0.49	1.27±0.38	4.47±0.60	4.29±1.75	52.86±27.95

	t检验			Bootstrap
	t(34)	p	d	SE	p	95%CI下限	95%CI上限
保留成绩	2.09	0.044	0.40	0.003	0.045	0.01	0.36
迁移成绩	0.44	0.664		0.000	0.677	−0.15	0.22
总成绩	2.16	0.038	0.38	0.003	0.047	0.02	0.46
心理努力	−1.86	0.072	0.20	−0.004	0.072	−0.64	−0.01
学习时间	0.44	0.664		0.001	0.661	−2.63	4.43

	t检验			Bootstrap
	t(34)	p	d	SE	p	95%CI下限	95%CI上限
保留成绩	2.09	0.044	0.40	0.003	0.045	0.01	0.36
迁移成绩	0.44	0.664		0.000	0.677	−0.15	0.22
总成绩	2.16	0.038	0.38	0.003	0.047	0.02	0.46
心理努力	−1.86	0.072	0.20	−0.004	0.072	−0.64	−0.01
学习时间	0.44	0.664		0.001	0.661	−2.63	4.43

通道	脑区	t检验			Bootstrap
通道	脑区	t(34)	p	d	SE	p	95%CI下限	95%CI上限
7	额极	3.80	0.001	0.34	0.04	0.008	1.77	5.28
8	背外侧前额叶皮质	2.27	<0.001	0.36	0.04	0.001	2.74	7.45
9		3.56	0.001	0.21	0.02	0.001	1.25	4.40
10		3.24	0.003	0.27	0.03	0.005	1.40	5.32