The Application of the Comparative Judgment in Chinese Text Difficulty Assessment

doi:10.12139/j.1672-0628.2026.02.002

Abstract

Abstract: One effective way to obtain a trustworthy assessment of text difficulty is to employ comparative judgment, a holistic assessment method that has not yet been utilized in Chinese research. The current study gathered 80 evaluators and evaluated the difficulty of 80 texts using the comparative judgment approach. The results showed that the results obtained from comparative judgment had high reliability and were significantly correlated with the number of volumes and readability scores. The reliability and validity tended to be stable with the increase in the number of comparative judgments. No effect of the evaluator characteristics on the comparative judgment results was found, which implies that comparative judgment is also reliable in the assessment of the difficulty of Chinese texts.

Key words: Chinese leveled reading, text difficulty, comparative judgment, elementary-school-level Chinese language and literature textbooks

摘要： 两两比较作为一种相对整体并能够高效获得可靠结果的文本难度评估手段，在汉语文本评估中的效果还有待探索。本研究通过80名评估者对80篇文本的两两比较，探讨了比较次数对两两比较信效度的影响。结果表明，两两比较获得的结果具有较高的信度，且与文本册数和可读性分数显著相关，随着比较次数的增加，信效度逐渐增加且趋于稳定，同时未发现评估者特征对比较结果的影响，这意味着两两比较在汉语文本难度评估中具备一定可靠性。

关键词: 汉语分级阅读, 文本难度, 两两比较, 小学语文教材

CLC Number:

B842

YANG Guandoudou, TAN Jingwen, LIU Miaomiao, LI Hong. The Application of the Comparative Judgment in Chinese Text Difficulty Assessment[J]. Studies of Psychology and Behavior, 2026, 24(2): 151-160.

杨官豆豆, 谭静文, 刘苗苗, 李虹. 两两比较在汉语文本难度评估中的应用[J]. 心理与行为研究, 2026, 24(2): 151-160.

References

陈茹玲, 蔡鑫廷, 宋曜廷, 李宜宪. (2015). 文本适读性分级架构之建立研究. 教育科学研究期刊, 60(1), 1–32
刘苗苗, 李燕, 王欣萌, 甘琳琳, 李虹. (2021). 分级阅读初探: 基于小学教材的汉语可读性公式研究. 语言文字应用, (2), 116–126
杨慊, 贺文洁, 王海龙. (2021). 单参数单维度Rasch模型的优势与意义. 心理科学, 44(6), 1491–1498
中国新闻出版研究院. (2022). 第十九次全国国民阅读调查结果. 2022-11-30取自https://society.huanqiu.com/article/47ix20UIt5x
Bartholomew, S. R., Ruesch, E. Y., Hartell, E., & Strimel, G. J. (2020). Identifying design values across countries through adaptive comparative judgment. International Journal of Technology and Design Education, 30(2), 321–347
Bloxham, S. (2009). Marking and moderation in the UK: False assumptions and wasted resources. Assessment & Evaluation in Higher Education, 34(2), 209–220
Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3–4), 324–345
Bramley, T. (2007). Paired comparison methods. In P. Newton, J. A. Baird, H. Goldstein, H. Patrick, & P. Tymms (Eds.), Techniques for monitoring the comparability of examination standards (pp. 246–300). London: Qualifications and Curriculum Authority.
Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment. Cambridge: Cambridge University Press & Assessment.
Bramley, T., & Vitello, S. (2019). The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(1), 43–58.
Chall, J. S., & Conard, S. S. (1991). Should textbooks challenge students? :The case for easier or harder books. New York: Teachers College Press.
Chen, S. Y., & Fang, S. P. (2015). Developing a Chinese version of an author recognition test for college students in Taiwan. Journal of Research in Reading, 38(4), 344–360
Coertjens, L., Lesterhuis, M., Verhavert, S., van Gasse, R., & De Maeyer, S. (2017). Judging texts with rubrics and comparative judgement: Taking into account reliability and time investment. Pedagogische Studien, 94(4), 283–303
Crompvoets, E. A. V., Béguin, A. A., & Sijtsma, K. (2020). Adaptive pairwise comparison for educational measurement. Journal of Educational and Behavioral Statistics, 45(3), 316–338
Crossley, S., Heintz, A., Choi, J. S., Batchelor, J., Karimi, M., & Malatinszky, A. (2023). A large-scaled corpus for assessing text readability. Behavior Research Methods, 55(2), 491–507
Crossley, S. A., Skalicky, S., & Dascalu, M. (2019). Moving beyond classic readability formulas: New methods and new models. Journal of Research in Reading, 42(3–4), 541–561
Dale, E., & Chall, J. S. (1949). The concept of readability. Elementary English, 26(1), 19–26
Fountas, I. C., & Pinnell, G. S. (2012). Guided reading: The romance and the reality. The Reading Teacher, 66(4), 268–284
Fry, E. (2002). Readability versus leveling. The Reading Teacher, 56(3), 286–291
Jones, I., & Inglis, M. (2015). The problem of assessing problem solving: Can comparative judgement help? Educational Studies in Mathematics, 89(3), 337–355
Jones, I., Swan, M., & Pollitt, A. (2015). Assessing mathematical problem solving using comparative judgement. International Journal of Science and Mathematics Education, 13(1), 151–177
Kuhn, M. R., Schwanenflugel, P. J., & Meisinger, E. B. (2010). Aligning theory and assessment of reading fluency: Automaticity, prosody, and definitions of fluency. Reading Research Quarterly, 45(2), 230–251
Landrieu, Y., De Smedt, F., van Keer, H., & De Wever, B. (2022). Assessing the quality of argumentative texts: Examining the general agreement between different rating procedures and exploring inferences of (dis)agreement cases. Frontiers in Education, 7, 784261
Lesterhuis, M., Bouwer, R., van Daal, T., Donche, V., & De Maeyer, S. (2022). Validity of comparative judgment scores: How assessors evaluate aspects of text quality when comparing argumentative texts. Frontiers in Education, 7, 823895
Lesterhuis, M., van Daal, T., van Gasse, R., Coertjens, L., Donche, V., & De Maeyer, S. (2018). When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality. L1-Educational Studies in Language and Literature, 18(1), 1–22
Liu, M. M., Li, Y. X., Su, Y. Q., & Li, H. (2024). Text complexity of Chinese elementary school textbooks: Analysis of text linguistic features using machine learning algorithms. Scientific Studies of Reading, 28(3), 235–255
Luce, R. D. (1959). Individual choice behavior: A theoretical analysis. New York: John Wiley & Sons, Inc.
Meng, X. L., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin, 111(1), 172–175
Paquot, M., Rubin, R., & Vandeweerd, N. (2022). Crowdsourced adaptive comparative judgment: A community-based solution for proficiency rating. Language Learning, 72(3), 853–885
Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.
Pollitt, A., & Murray, N. L. (1996). What raters really pay attention to. In M. Milanovic & N. Saville (Eds.), Studies in language testing 3: Performance testing, cognition and assessment (pp. 74–91). Cambridge: Cambridge University Press.
Renaissance. (2022). What kids are reading report 2022. Retrieved November 30, 2022, from https://www.renaissance.com/2022/03/01/news-renaissance-shares-findings-of-worlds-largest-annual-k12-reading-survey/
Sheehan, K. M., Kostin, I., Napolitano, D., & Flor, M. (2014). The TextEvaluator tool: Helping teachers and test developers select texts for use in instruction and assessment. The Elementary School Journal, 115(2), 184–209
Smith, D. R., Stenner, A. J., Horabin, I., & Smith, M. (1989). The lexile scale in theory and practice: Final report for NIH Grant HD-19448. Bethesda, MD: National Institutes of Health.
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286
Thwaites, P., Kollias, C., & Paquot, M. (2024). Is CJ a valid, reliable form of L2 writing assessment when texts are long, homogeneous in proficiency, and feature heterogeneous prompts? Assessing Writing, 60, 100843
Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562.
Verhavert, S., De Maeyer, S., Donche, V., & Coertjens, L. (2018). Scale separation reliability: What does it mean in the context of comparative judgment? Applied Psychological Measurement, 42(6), 428–445
Wheadon, C., Barmby, P., Christodoulou, D., & Henderson, B. (2020). A comparative judgement approach to the large-scale assessment of primary writing in England. Assessment in Education: Principles, Policy & Practice, 27(1), 46–64.

[1]	WANG Yake, ZHANG Yuxuan, FENG Linlin, KA Mingfang, LIANG Feifei. Developmental Trajectories of Eye Movement in Reading Among Third- to Fifth-Grade Children and Their Relationship with Reading Comprehension [J]. Studies of Psychology and Behavior, 2026, 24(2): 161-169.
[2]	LU Linxin, LIU Zaihua, LIU Hongping, LIN Xiuyun, ZHOU Hanxiang, BAN Yongfei, SUN Ji, LI Xiaoqing, ZHANG Yiqing, HUANG Haizhen. Intercultural Sensitivity Climate in Special Education Schools and Prosocial Behavior of Visually Impaired Students: A Multilevel Bayesian Mediation Model of Bicultural Identity Integration [J]. Studies of Psychology and Behavior, 2026, 24(2): 170-177.
[3]	WANG Danyun, WANG Yulong, TANG Zhuo. Dynamic Relationships Between Adolescents’ Exercise Habits, Self-Esteem, and Resilience: Based on the Cross-Lagged Panel Model and the Random Intercept Cross-Lagged Panel Model Analysis [J]. Studies of Psychology and Behavior, 2026, 24(2): 178-186.
[4]	Yan MA, Zhenhong WANG. The Effect of Moral Self-Perception on Deceptive Behavior: The Moderating Role of Construal Level [J]. Studies of Psychology and Behavior, 2024, 22(1): 39-45.
[5]	Siyuan LIU, Lin ZHU, Ruibing WANG, Chuyan XU, Yunping WANG, Conghui LIU. Is there Dialect Effect in Moral Decision-Making? [J]. Studies of Psychology and Behavior, 2024, 22(1): 31-38.
[6]	Yuanqing YAO, Yi’an GUO, Chunmei LI, Yanan WU, Lei SHI, Guangping ZHAO. The Mapping Mechanism of Social Role Metaphors in Geometric Shapes: Behavioral and ERPs Study [J]. Studies of Psychology and Behavior, 2024, 22(1): 23-30.
[7]	Chunye FU, Yong LYU. The Effect of Expectation and Temporal Attention on Visual Perception [J]. Studies of Psychology and Behavior, 2024, 22(1): 15-22.
[8]	Cheng QIAN, Yue ZHAO, Xixi NIU, Jiacan GU, Aijun WANG. Effects of Emotional Faces on Inhibition of Return in Three-Dimensional Space [J]. Studies of Psychology and Behavior, 2024, 22(1): 8-14.
[9]	Meihua GUO, Zebo LAN, Jingen WU, Sainan LI, Junjie WU, Guoli YAN. The Effect of Word Segmentation and Font Size on Perceptual Span in Chinese Reading: Evidence from Eye Movements [J]. Studies of Psychology and Behavior, 2024, 22(1): 1-7.
[10]	Ruqi CHEN, Yaqian BAO, Linjieqiong HUANG, Xingshan LI. An Introduction to the Chinese Reading Model (CRM) [J]. Studies of Psychology and Behavior, 2023, 21(6): 725-735.
[11]	Feifei LIANG, Linlin FENG, Ying LIU, Changhao WANG, Jie WANG. The Role of Character Positional Probability in Chinese Two-Word Identification: Moderation of Lexical Contextual Diversity [J]. Studies of Psychology and Behavior, 2023, 21(6): 736-743.
[12]	Miao YU, Wendi WANG, Xiaoxiao CHEN. Prosodic Constraints in Chinese “N de V” Construction [J]. Studies of Psychology and Behavior, 2023, 21(6): 744-750.
[13]	Wanting CHEN, Yifei ZHANG, Qinghua HE. The Role of Accuracy Nudge in False Information Sharing [J]. Studies of Psychology and Behavior, 2023, 21(6): 751-759.
[14]	Dafu MA, Chunying QIN, Xiaofeng YU, Cui HE. Research on Test Assembly of Item Discrimination Index Under Polytomous Attributes and Mixed Scoring Items [J]. Studies of Psychology and Behavior, 2023, 21(6): 760-769.
[15]	Lei LIU, Yanan LI, Ruoyu NIU, Wenting YU, Yuxue CHEN, Ying LIU. Neural Basis of Motor Coordination in Stepping Tasks [J]. Studies of Psychology and Behavior, 2023, 21(5): 600-607.