Exploration of Adaptive Item Bank Development for Emotional Stability Based on ChatGPT

doi:10.12139/j.1672-0628.2025.05.014

Abstract

Abstract:

The significant human and material resources required for traditional item development have hindered the advancement of computer adaptive testing. However, with the progress of natural language processing technology, it has become possible to automatically generate items based on specific measurement objectives and to construct adaptive testing item banks on this basis. Nevertheless, the performance of the constructed item bank still requires validation. Therefore, this study utilized ChatGPT to generate a large number of Chinese personality items related to emotional stability. Through a series of item bank construction steps and simulations of adaptive testing, the quality of the generated items and the constructed item bank was evaluated. The results reveal that GPT-generated items exhibit performance similar to that of classical items, thereby validating a cost-effective and efficient supplementary solution for item bank development.

Key words: computerized adaptive testing, automatic item generation, natural language processing, emotional stability, item response theory

摘要：

传统项目开发所耗费的大量人力物力制约着计算机自适应测验的发展，而随着自然语言处理技术的进步，根据特定测量目标自动生成项目并以此为基础建立自适应测验题库成为可能，但构建出的题库性能仍有待验证。因此本研究使用ChatGPT生成了大量中文情绪稳定性的人格项目，通过系列题库构建步骤以及模拟自适应测验，检验了生成项目及其构建出的题库质量，并发现GPT生成项目与经典项目有类似的性能，以此验证了一种经济高效的题库开发补充方案。

关键词: 计算机化自适应测验, 自动项目生成, 自然语言处理, 情绪稳定性, 项目反应理论

Yaojie GAO, Yunxiao QI, Yuanqiu MA, Tuo LIU. Exploration of Adaptive Item Bank Development for Emotional Stability Based on ChatGPT[J]. Studies of Psychology and Behavior, 2025, 23(5): 685-694.

高垚杰, 齐运晓, 马苑秋, 刘拓. 基于ChatGPT的情绪稳定性自适应题库开发的探索[J]. 心理与行为研究, 2025, 23(5): 685-694.

Figures/Tables 13

References 0

	王孟成, 戴晓阳, 姚树桥. 中国大五人格问卷的初步编制Ⅲ: 简式版的制定及信效度检验. 中国临床心理学杂志, 2011, 19 (4): 454- 457.
	Akaike, H. Stochastic theory of minimal realization. IEEE Transactions on Automatic Control, 1974, 19 (6): 667- 674. DOI
	Bolukbasi, T., Chang, K. W., Zou, J., Saligrama, V., & Kalai, A. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (pp. 4349–4357). Barcelona, Spain: Curran Associates Inc.
	Choi, S. W., Gibbons, L. E., & Crane, P. K. Lordif: An R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software, 2011, 39 (8): 1- 30.
	DeYoung, C. G., Quilty, L. C., & Peterson, J. B. Between facets and domains: 10 aspects of the Big Five. Journal of Personality and Social Psychology, 2007, 93 (5): 880- 896. DOI
	Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. The Satisfaction with Life Scale. Journal of PERSonality Assessment, 1985, 49 (1): 71- 75. DOI
	Fliege, H., Becker, J., Walter, O. B., Bjorner, J. B., Klapp, B. F., & Rose, M. Development of a computer-adaptive test for depression (D-CAT). Quality of Life Research, 2005, 14 (10): 2277- 2291. DOI
	Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences of the United States of America, 2018, 115 (16): E3635- E3644.
	Gierl, M. J., & Haladyna, T. M. (Eds.). (2013). Automatic item generation: Theory and practice. New York: Routledge.
	Goldberg, Y. (2017). Neural network methods for natural language processing. Cham, Switzerland: Springer.
	Gosling, S. D., Rentfrow, P. J., & Swann, Jr. W. B. A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 2003, 37 (6): 504- 528. DOI
	Götz, F. M., Maertens, R., Loomba, S., & van Der Linden, S. Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development. Psychological Methods, 2024, 29 (3): 494- 518. DOI
	Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Thousand Oaks, CA: Sage Publications, Inc.
	Hernandez, I., & Nie, W. W. The AI‐IP: Minimizing the guesswork of personality scale item development through artificial intelligence. Personnel Psychology, 2023, 76 (4): 1011- 1035. DOI
	Hommel, B. E., Wollang, F. J. M., Kotova, V., Zacher, H., & Schmukle, S. C. Transformer-based deep neural language modeling for construct-specific automatic item generation. Psychometrika, 2022, 87 (2): 749- 772. DOI
	Kendler, K. S., Gardner, C. O., & Prescott, C. A. Toward a comprehensive developmental model for major depression in women. American Journal of Psychiatry, 2002, 159 (7): 1133- 1145. DOI
	Lee, P., Fyffe, S., Son, M., Jia, Z. H., & Yao, Z. Y. A paradigm shift from “human writing” to “machine generation” in personality test development: An application of state-of-the-art natural language processing. Journal of Business and Psychology, 2023, 38 (1): 163- 190. DOI
	Liu, X. R., Lu, H., Zhou, Z., Chao, M., & Liu, T. Development of a computerized adaptive test for problematic mobile phone use. Frontiers in Psychology, 2022, 13, 892387. DOI
	May, S., Littlewood, C., & Bishop, A. Reliability of procedures used in the physical examination of non-specific low back pain: A systematic review. Australian Journal of Physiotherapy, 2006, 52 (2): 91- 102. DOI
	Muraki, E. A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 1992, 16 (2): 159- 176. DOI
	Norton, P. J. (2007). Depression Anxiety and Stress Scales (DASS-21): Psychometric analysis across four racial groups. Anxiety, Stress, and Coping, 20(3), 253–265.
	Polit, D. F., Beck, C. T., & Owen, S. V. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 2007, 30 (4): 459- 467.
	Reckase, M. D. Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 1979, 4 (3): 207- 230. DOI
	Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., .. PROMIS Cooperative Group. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5 Suppl. 1), S22–S31.
	Samejima, F. Estimation of latent ability using a response pattern of graded scores. Psychometrika, 1969, 34 (S1): 1- 97. DOI
	Schwarz, G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6 (2): 461- 464.
	Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., .. Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 6000-6010). Long Beach, CA: Curran Associates Inc.
	von Davier, M. Automated item generation with recurrent neural networks. Psychometrika, 2018, 83 (4): 847- 857. DOI
	Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000). Computerized adaptive testing: A primer (2nd ed.). New York: Routledge.
	Widiger, T. A., & Oltmanns, J. R. Neuroticism is a fundamental domain of personality with enormous public health implications. World Psychiatry, 2017, 16 (2): 144- 145. DOI
	Womick, J., & King, L. A. (2020). Personality and well-being. In B. J. Carducci & C. Nave. (Eds.), The wiley-blackwell encyclopedia of personality and individual differences: Vol. IV: Clinical, applied, and cross-cultural research (pp. 371–376). Hoboken, NJ: John Wiley & Sons.
	Xu, L. L., Jin, R. Y., Huang, F. F., Zhou, Y. H., Li, Z. L., & Zhang, M. Q. Development of computerized adaptive testing for emotion regulation. Frontiers in Psychology, 2020, 11, 561358. DOI
	Zhang, B., Li, Y. M., Li, J., Luo, J., Ye, Y. H., Yin, L., ... John, O. P. The Big Five Inventory-2 in China: A comprehensive psychometric evaluation in four diverse samples. Assessment, 2022, 29 (6): 1262- 1284. DOI
	Zhang, X. T., Wang, M. C., He, L. N., Jie, L., & Deng, J. X. The development and psychometric evaluation of the Chinese Big Five Personality Inventory-15. PLoS One, 2019, 14 (8): e0221621. DOI

模型	AIC	BIC	Loglik
经典题库
GRM	47240.97	48167.08	−23398.48
GPCM	47820.24	48746.35	−23688.12
GPT题库
GRM	74640.70	76242.64	−36936.35
GPCM	75556.12	77158.05	−37394.06
总题库
GRM	122730.8	125258.8	−60759.38
GPCM	124201.5	126729.6	−61494.76

模型	AIC	BIC	Loglik
经典题库
GRM	47240.97	48167.08	−23398.48
GPCM	47820.24	48746.35	−23688.12
GPT题库
GRM	74640.70	76242.64	−36936.35
GPCM	75556.12	77158.05	−37394.06
总题库
GRM	122730.8	125258.8	−60759.38
GPCM	124201.5	126729.6	−61494.76

题库	终止规则	M_题数	SD_题数	M_SE(θ)	M_MR	r_真值
经典题库	全部	42		0.15	0.98	0.986***
	SE(θ)≤0.447	2.60	0.88	0.41	0.83	0.900***
	SE(θ)≤0.387	3.77	1.73	0.36	0.87	0.929***
	SE(θ)≤0.316	5.87	3.23	0.30	0.91	0.950***
	SE(θ)≤0.224	12.90	4.77	0.22	0.95	0.978***
GPT题库	全部	75		0.09	0.99	0.992***
	SE(θ)≤0.447	2.40	1.06	0.40	0.84	0.921***
	SE(θ)≤0.387	3.06	1.18	0.36	0.87	0.930***
	SE(θ)≤0.316	4.90	3.76	0.30	0.91	0.953***
	SE(θ)≤0.224	10.11	6.94	0.22	0.95	0.976***
GPT 题库(42)	全部	42		0.13	0.98	0.996***
	SE(θ)≤0.447	2.48	0.86	0.40	0.84	0.994***
	SE(θ)≤0.387	3.35	1.75	0.36	0.87	0.995***
	SE(θ)≤0.316	5.15	3.05	0.30	0.91	0.996***
	SE(θ)≤0.224	10.51	4.12	0.22	0.95	0.996***
总题库	全部	117		0.08	0.99
	SE(θ)≤0.447	2.42	0.96	0.39	0.84	0.916***
	SE(θ)≤0.387	3.15	1.39	0.36	0.87	0.932***
	SE(θ)≤0.316	4.79	3.30	0.30	0.91	0.948***
	SE(θ)≤0.224	10.14	8.38	0.22	0.95	0.971***

题库	终止规则	M_题数	SD_题数	M_SE(θ)	M_MR	r_真值
经典题库	全部	42		0.15	0.98	0.986***
	SE(θ)≤0.447	2.60	0.88	0.41	0.83	0.900***
	SE(θ)≤0.387	3.77	1.73	0.36	0.87	0.929***
	SE(θ)≤0.316	5.87	3.23	0.30	0.91	0.950***
	SE(θ)≤0.224	12.90	4.77	0.22	0.95	0.978***
GPT题库	全部	75		0.09	0.99	0.992***
	SE(θ)≤0.447	2.40	1.06	0.40	0.84	0.921***
	SE(θ)≤0.387	3.06	1.18	0.36	0.87	0.930***
	SE(θ)≤0.316	4.90	3.76	0.30	0.91	0.953***
	SE(θ)≤0.224	10.11	6.94	0.22	0.95	0.976***
GPT 题库(42)	全部	42		0.13	0.98	0.996***
	SE(θ)≤0.447	2.48	0.86	0.40	0.84	0.994***
	SE(θ)≤0.387	3.35	1.75	0.36	0.87	0.995***
	SE(θ)≤0.316	5.15	3.05	0.30	0.91	0.996***
	SE(θ)≤0.224	10.51	4.12	0.22	0.95	0.996***
总题库	全部	117		0.08	0.99
	SE(θ)≤0.447	2.42	0.96	0.39	0.84	0.916***
	SE(θ)≤0.387	3.15	1.39	0.36	0.87	0.932***
	SE(θ)≤0.316	4.79	3.30	0.30	0.91	0.948***
	SE(θ)≤0.224	10.14	8.38	0.22	0.95	0.971***

	SE_CBF-PI-B=0.34	SE_BFI-2=0.27	SE_TIPI=0.46	SE_BFAS=0.21
传统测验题数	8	12	2	20
经典题库题数	5.07	8.44	2.51	15.13
GPT题库题数	4.18	6.86	2.27	11.63
GPT题库(42)题数	4.42	7.12	2.37	11.97
总题库题数	4.20	7.08	2.27	11.52