Assessing the Reliability of Statistical Learning Measures: Effects of Modality, Material, and Task

doi:10.12139/j.1672-0628.2024.06.013

Abstract

Abstract:

Statistical learning (SL) ability is often examined as an independent variable in predicting individual language development. However, traditional experimental tasks designed to assess group differences typically exhibit low reliability, failing to meet basic psychometric standards. This study aimed to address these limitations by synthesizing learning materials with mixed-length target structures and utilizing two testing tasks: a two-alternative forced-choice task and a familiarity rating task. Additionally, the study incorporated auditory speech and visual graphic modalities, calculating the reliability of the tasks using Cronbach’s alpha and split-half reliability. The results showed that experimental tasks utilizing mixed-length target structures demonstrated higher reliability compared to previous studies. Notably, tasks in the visual modality showed greater reliability than those in the auditory modality, and the forced-choice task yielded higher reliability than the familiarity rating task. These findings highlight the advantages of using mixed-length learning materials and visual stimuli when assessing SL ability.

Key words: statistical learning ability, reliability, two-alternative forced choice task, familiarity rating task

摘要：

统计学习能力常作为自变量预测个体语言能力的发展，但以组间差异为目标所编制的实验任务普遍信度不高，难以满足心理测量学的要求。为探究统计学习能力的测量方案，本研究使用混合长度的目标结构合成学习材料，在测验任务中对比了二选一的迫选任务和熟悉度评分任务；同时设计了听觉语音和视觉图形两种材料模态。信度指标计算了测验的内部一致性系数和分半信度。结果发现混合长度目标结构编制的实验任务的信度普遍高于以往使用等长度目标结构的任务；同时，视觉模态下的信度要高于听觉模态；前者迫选任务的信度也要高于熟悉度评分任务。综上，本研究推荐在视觉模态下，使用混合长度的目标结构合成学习材料、以迫选任务来考察被试的统计学习能力。

关键词: 统计学习能力, 信度, 迫选任务, 熟悉度评分任务

Wenbo YU, Hetong QI, Tianlin WANG, Dandan LIANG. Assessing the Reliability of Statistical Learning Measures: Effects of Modality, Material, and Task[J]. Studies of Psychology and Behavior, 2024, 22(6): 814-821.

于文勃, 亓鹤潼, 王天琳, 梁丹丹. 统计学习能力测量的信度研究：基于模态、材料特征和测试任务的证据[J]. 心理与行为研究, 2024, 22(6): 814-821.

Figures/Tables 7

References 0

	徐贵平, 范若琳, 金花. 统计学习的认知神经机制及其与语言的关系. 心理科学进展, 2020, 28 (9): 1525- 1538.
	于文勃, 王璐, 程幸悦, 王天琳, 张晶晶, 梁丹丹. 语言经验对概率词切分的影响. 心理科学进展, 2021, 29 (5): 787- 795.
	于文勃, 王璐, 瞿邢芳, 王天琳, 张晶晶, 梁丹丹. 转换概率和词长期待对语音统计学习的影响. 心理学报, 2021, 53 (6): 565- 574.
	Arnon, I. Do current statistical learning tasks capture stable individual differences in children? An investigation of task reliability across modality. Behavior Research Methods, 2020, 52 (1): 68- 81. DOI
	Batterink, L. J., Reber, P. J., Neville, H. J., & Paller, K. A. Implicit and explicit contributions to statistical learning. Journal of Memory and Language, 2015, 83, 62- 78. DOI
	Bogaerts, L., Frost, R., & Christiansen, M. H. Integrating statistical learning into cognitive science. Journal of Memory and Language, 2020, 115, 104167. DOI
	Bogaerts, L., Siegelman, N., & Frost, R. Splitting the variance of statistical learning performance: A parametric investigation of exposure duration and transitional probabilities. Psychonomic Bulletin & Review, 2016, 23 (4): 1250- 1256.
	Elazar, A., Alhama, R. G., Bogaerts, L., Siegelman, N., Baus, C., & Frost, R. When the “Tabula” is anything but “Rasa”: What determines performance in the auditory statistical learning task. Cognitive Science, 2022, 46 (2): e13102. DOI
	Erickson, L. C., Kaschak, M. P., Thiessen, E. D., & Berry, C. A. S. Individual differences in statistical learning: Conceptual and measurement issues. Collabra, 2016, 2 (1): 14. DOI
	Estes, K. G., Evans, J. L., Alibali, M. W., & Saffran, J. R. Can infants map meaning to newly segmented words? Statistical segmentation and word learning. Psychological Science, 2007, 18 (3): 254- 260. DOI
	Estes, K. G., Gluck, S. C. W., & Bastos, C. Flexibility in statistical word segmentation: Finding words in foreign speech. Language Learning and Development, 2015, 11 (3): 252- 269. DOI
	Frost, R., Armstrong, B. C., & Christiansen, M. H. Statistical learning research: A critical review and possible new directions. Psychological Bulletin, 2019, 145 (12): 1128- 1153. DOI
	Gabay, Y., Thiessen, E. D., & Holt, L. L. (2015). Impaired statistical learning in developmental dyslexia. Journal of Speech, Language, and Hearing Research, 58(3), 934–945.
	Hedge, C., Powell, G., & Sumner, P. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 2018, 50 (3): 1166- 1186. DOI
	Hoch, L., Tyler, M. D., & Tillmann, B. Regularity of unit length boosts statistical learning in verbal and nonverbal artificial languages. Psychonomic Bulletin & Review, 2013, 20 (1): 142- 147.
	Isbilen, E. S., & Christiansen, M. H. Statistical learning of language: A meta-analysis into 25 years of research. Cognitive Science, 2022, 46 (9): e13198. DOI
	Isbilen, E. S., McCauley, S. M., & Christiansen, M. H. Individual differences in artificial and natural language statistical learning. Cognition, 2022, 225, 105123. DOI
	Isbilen, E. S., McCauley, S. M., Kidd, E., & Christiansen, M. H. Statistically induced chunking recall: A memory-based approach to statistical learning. Cognitive Science, 2020, 44 (7): e12848. DOI
	Kidd, E., & Arciuli, J. Individual differences in statistical learning predict children’s comprehension of syntax. Child Development, 2016, 87 (1): 184- 193. DOI
	Kidd, E., Arciuli, J., Christiansen, M. H., Isbilen, E. S., Revius, K., & Smithson, M. Measuring children’s auditory statistical learning via serial recall. Journal of Experimental Child Psychology, 2020, 200, 104964. DOI
	Lammertink, I., Boersma, P., Rispens, J., & Wijnen, F. Visual statistical learning in children with and without DLD and its relation to literacy in children with DLD. Reading and Writing, 2020, 33 (6): 1557- 1589. DOI
	Lee, S. M. K., Cui, Y. M. N., & Tong, S. X. Toward a model of statistical learning and reading: Evidence from a meta-analysis. Review of Educational Research, 2022, 92 (4): 651- 691. DOI
	Newport, E. L. Statistical language learning: Computational, maturational, and linguistic constraints. Language and Cognition, 2016, 8 (3): 447- 461. DOI
	Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
	Ordin, M., & Polyanskaya, L. The role of metacognition in recognition of the content of statistical learning. Psychonomic Bulletin & Review, 2021, 28 (1): 333- 340.
	Perruchet, P. What mechanisms underlie implicit statistical learning? Transitional probabilities versus chunks in language learning. Topics in Cognitive Science, 2019, 11 (3): 520- 535. DOI
	Qi, Z. H., Sanchez Araujo, Y., Georgan, W. C., Gabrieli, J. D. E., & Arciuli, J. Hearing matters more than seeing: A cross-modality study of statistical learning and reading ability. Scientific Studies of Reading, 2019, 23 (1): 101- 115. DOI
	Saffran, J. R., Aslin, R. N., & Newport, E. L. Statistical learning by 8-month-old infants. Science, 1996, 274 (5294): 1926- 1928. DOI
	Saffran, J. R., Johnson, E. K., Aslin, R. N., & Newport, E. L. Statistical learning of tone sequences by human infants and adults. Cognition, 1999, 70 (1): 27- 52. DOI
	Saffran, J. R., & Kirkham, N. Z. Infant statistical learning. Annual Review of Psychology, 2018, 69 (1): 181- 203. DOI
	Siegelman, N. Statistical learning abilities and their relation to language. Language and Linguistics Compass, 2020, 14 (3): e12365. DOI
	Siegelman, N., Bogaerts, L., Elazar, A., Arciuli, J., & Frost, R. Linguistic entrenchment: Prior knowledge impacts statistical learning performance. Cognition, 2018, 177, 198- 213. DOI
	Siegelman, N., Bogaerts, L., & Frost, R. Measuring individual differences in statistical learning: Current pitfalls and possible solutions. Behavior Research Methods, 2017, 49 (2): 418- 432. DOI
	Siegelman, N., Bogaerts, L., Kronenfeld, O., & Frost, R. (2018). Redefining “Learning” in statistical learning: What does an online measure reveal about the assimilation of visual regularities? Cognitive Science, 42(Suppl.3), 692–727.
	Siegelman, N., & Frost, R. (2015). Statistical learning as an individual ability: Theoretical perspectives and empirical evidence. Journal of Memory and Language, 81, 105–120.
	Tong, X. H., Leung, W. W. S., & Tong, X. L. Visual statistical learning and orthographic awareness in Chinese children with and without developmental dyslexia. Research in Developmental Disabilities, 2019, 92, 103443. DOI
	van Witteloostuijn, M., Boersma, P., Wijnen, F., & Rispens, J. (2021). The contribution of individual differences in statistical learning to reading and spelling performance in children with and without dyslexia. Dyslexia, 27(2), 168–186.
	von Koss Torkildsen, J., Arciuli, J., & Wie, O. B. Individual differences in statistical learning predict children’s reading ability in a semi-transparent orthography. Learning and Individual Differences, 2019, 69, 60- 68. DOI

学习材料版本	目标词	跨界词	非词
语音学习材料A	nueruote	tediafo	nuemeilai
	diafolai	lainueruo	diasete
	remei	meirou	refo
	rouse	sere	rouruo
语音学习材料B	tediafo	diafolai	terouruo
	lainueruo	nueruote	lairefo
	meirou	rouse	meinue
	sere	remei	sedia

学习材料版本	目标词	跨界词	非词
语音学习材料A	nueruote	tediafo	nuemeilai
	diafolai	lainueruo	diasete
	remei	meirou	refo
	rouse	sere	rouruo
语音学习材料B	tediafo	diafolai	terouruo
	lainueruo	nueruote	lairefo
	meirou	rouse	meinue
	sere	remei	sedia

模态	听觉语音模态				视觉图形模态
测试任务	熟悉度评分任务		迫选任务		熟悉度评分任务		迫选任务
对照材料	A	B	A	B	A	B	A	B
α系数	0.66	0.75	0.65	0.68	0.72	0.74	0.86	0.74
分半信度	0.41～0.82	0.59～0.91	0.20～0.90	0.33～0.89	0.56～0.86	0.40～0.88	0.59～0.97	0.38～0.94

模态	听觉语音模态				视觉图形模态
测试任务	熟悉度评分任务		迫选任务		熟悉度评分任务		迫选任务
对照材料	A	B	A	B	A	B	A	B
α系数	0.66	0.75	0.65	0.68	0.72	0.74	0.86	0.74
分半信度	0.41～0.82	0.59～0.91	0.20～0.90	0.33～0.89	0.56～0.86	0.40～0.88	0.59～0.97	0.38～0.94

[1]	LIAO Wenli, ZHONG Shuang, FAN Qian, XU Huachun. Revision of Positive and Negative Risk-Taking Questionnaire Among Chinese College Students [J]. Studies of Psychology and Behavior, 2024, 22(6): 839-846.
[2]	Hongwei NIU, Jiajia HAO, Xuejun BAI. Development and Validation of Chinese College Students Failure Mindset Scale [J]. Studies of Psychology and Behavior, 2024, 22(3): 379-386.
[3]	Shouying ZHAO, Rongrong REN, Wei CHEN, Bangjun MU. Development and Validation of Numerical Reasoning Test for Grade 7~8 Students [J]. Studies of Psychology and Behavior, 2024, 22(3): 387-394.
[4]	Xin GUO, Wen LIU, Jiaqi ZHANG, Lingxiang XIA, Jie WU, Yining WANG. Social Adjustment Questionnaire for Students in Grades 4 to 9: Reliability and Validity Test in a Large Sample [J]. Studies of Psychology and Behavior, 2024, 22(3): 395-401.
[5]	Zhao ZHOU, Shunxin JI, Tuo LIU. Development of the Problematic Mobile Phone Use Scale Short and Very Short Version [J]. Studies of Psychology and Behavior, 2023, 21(6): 846-853.
[6]	Haibo YANG, Xinyu ZHANG. Revision and Evaluation of Diagnostic Efficiency for the Simplified Chinese Version of the Ten-Item Internet Gaming Disorder Test [J]. Studies of Psychology and Behavior, 2023, 21(5): 658-666.
[7]	Ping FANG, Zijian YI, Yuan JIANG, Hui PANG. The Development of People’s Social Adaptation Scale [J]. Studies of Psychology and Behavior, 2023, 21(4): 433-439.
[8]	Hui PANG, Xiaonan JIN, Yuan JIANG, Ping FANG. The Development of People’s Social Risk Perception Scale [J]. Studies of Psychology and Behavior, 2023, 21(4): 440-445.
[9]	Liu YANG, Wei DU, Ping FANG, Yuan JIANG. The Development and Validation of Social Emotion Scale [J]. Studies of Psychology and Behavior, 2023, 21(4): 446-453.
[10]	Hongxia LI, Yanli XU, Yiquan FANG, Kaiyue GUO, Jiwei SI. Reliability and Validity of the Abbreviated Math Anxiety Scale in Chinese Primary School Students [J]. Studies of Psychology and Behavior, 2022, 20(6): 850-857.
[11]	MIAO Xiuying, CHI Lizhong. The Impact of the Affect Heuristic on Risky Choice [J]. Studies of Psychology and Behavior, 2022, 20(3): 325-331.
[12]	WANG Yanli, CHEN Hong. The Development of Sense of Gain Questionnaire Among Urban Community Residents [J]. Studies of Psychology and Behavior, 2021, 19(5): 665-670.
[13]	LIU Wen, MENG Xiangrui, CHE Hanbo, ZHANG Jiaqi, SHEN Yi. Development, Psychometric Reliability and Validity of Pupils’ Creative Personality Questionnaire Based on Teacher’s Descriptions [J]. Studies of Psychology and Behavior, 2020, 18(5): 624-630.
[14]	WANG Zhanqi, ZHANG Xingli. The Validity and Reliability of the Chinese Version of the Creative Achievement Questionnaire [J]. Studies of Psychology and Behavior, 2020, 18(3): 390-397.
[15]	ZHANG Lina, LU Haidong. Development of Chinese High School Students Perceived Classroom Teaching Approaches Scale [J]. Studies of Psychology and Behavior, 2019, 17(6): 773-779.