Research on Test Assembly of Item Discrimination Index Under Polytomous Attributes and Mixed Scoring Items

doi:10.12139/j.1672-0628.2023.06.006

Abstract

Abstract: In this study, two item discrimination indices (CDI, ADI) and two general modified item discrimination indices (GMCDI, GMADI) were used for polytomous attribute diagnosis test assembly. Two simulation studies were conducted considering three real-world testing scenarios: Study 1, an independent test of dichotomous and polytomous scoring items; Study 2, a mixed test of dichotomous and polytomous scoring items. The results indicate that GMCDI and GMADI can obtain the highest diagnosis accuracy, and are not affected by attribute hierarchy and test type.

Key words: item discrimination index, CD-ATA, polytomous attributes, dichotomous scoring, polytomous scoring

摘要： 将两种项目区分度指标(CDI、ADI)及两种扩展项目区分度指标(GMCDI、GMADI)用于属性多水平诊断测验组卷。考虑三种实际测验情景，开展两项研究：研究1，二级与多级计分项目独立组卷测验；研究2，二级与多级计分项目混合组卷测验。结果表明，GMCDI与GMADI组卷方法可获得最高诊断精度，且不受属性层级结构与测验类型影响。

关键词: 项目区分度指标, 认知诊断自动测验组卷, 属性多水平, 二级计分, 多级计分

CLC Number:

B842

Dafu MA, Chunying QIN, Xiaofeng YU, Cui HE. Research on Test Assembly of Item Discrimination Index Under Polytomous Attributes and Mixed Scoring Items[J]. Studies of Psychology and Behavior, 2023, 21(6): 760-769.

马大付, 秦春影, 喻晓锋, 何催. 项目区分度指标在属性多水平和混合计分项目下的组卷研究[J]. 心理与行为研究, 2023, 21(6): 760-769.

Figures/Tables 8

References

	蔡艳, 涂冬波. 属性多级化的认知诊断模型拓展及其Q矩阵设计. 心理学报, 2015, 47 (10): 1300- 1308.
	丁树良, 汪文义, 罗芬, 熊建华. (2016). 可达阵功能的不可替代性. 江西师范大学学报(自然科学版), 40(3), 290–294, 298.
	丁树良, 杨淑群, 汪文义. 可达矩阵在认知诊断测验编制中的重要作用. 江西师范大学学报(自然科学版), 2010, 34 (5): 490- 494.
	唐小娟, 丁树良, 毛萌萌, 俞宗火. 基于属性层级结构的认知诊断测验的组卷. 心理学探新, 2013, 33 (3): 252- 259.
	涂冬波, 蔡艳, 戴海琦, 丁树良. 一种多级评分的认知诊断模型: P-DINA模型的开发. 心理学报, 2010, 42 (10): 1011- 1020.
	汪文义, 宋丽红, 丁树良. 分类视角下认知诊断测验项目区分度指标及应用. 心理科学, 2018, 41 (2): 475- 483.
	王立君, 赵少勇, 昌维, 唐芳, 詹沛达. (2022). 重参数化多分属性DINA模型的多级评分拓广——基于等级反应模型. 心理科学, 45(1), 195–203.
	詹沛达, 边玉芳, 王立君. 重参数化的多分属性诊断分类模型及其判准率影响因素. 心理学报, 2016, 48 (3): 318- 330.
	Becker, B., Debeer, D., Sachse, K. A., & Weirich, S.. Automated test assembly in R: The eatATA package. Psych, 2021, 3 (2): 96- 112. DOI
	Chang, H. H., & Ying, Z. L.. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 1996, 20 (3): 213- 229. DOI
	Chen, J. S., & de la Torre, J.. A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 2013, 37 (6): 419- 437. DOI
	Finkelman, M., Kim, W., & Roussos, L. A.. Automated test assembly for cognitive diagnosis models using a genetic algorithm. Journal of Educational Measurement, 2009, 46 (3): 273- 292. DOI
	Finkelman, M. D., de la Torre, J., & Karp, J. A.. Cognitive diagnosis models and automated test assembly: An approach incorporating response times. International Journal of Testing, 2020, 20 (4): 299- 320. DOI
	Finkelman, M. D., Kim, W., Roussos, L., & Verschoor, A.. A binary programming approach to automated test assembly for cognitive diagnosis models. Applied Psychological Measurement, 2010, 34 (5): 310- 326. DOI
	Henson, R., & Douglas, J.. Test construction for cognitive diagnosis. Applied Psychological Measurement, 2005, 29 (4): 262- 277. DOI
	Henson, R., Roussos, L., Douglas, J., & He, X. M.. Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 2008, 32 (4): 275- 288. DOI
	Huebner, A., & Wang, C.. A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 2011, 71 (2): 407- 419. DOI
	Kuo, B. C., Pai, H. S., & de la Torre, J.. Modified cognitive diagnostic index and modified attribute-level discrimination index for test construction. Applied Psychological Measurement, 2016, 40 (5): 315- 330. DOI
	Leighton, J. P., & Gierl, M. J.. Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 2007, 26 (2): 3- 16. DOI
	Lin, Y., Gong, Y. J., & Zhang, J.. An adaptive ant colony optimization algorithm for constructing cognitive diagnosis tests. Applied Soft Computing, 2017, 52, 1- 13. DOI
	Lin, Y., Jiang, Y. S., Gong, Y. J., Zhan, Z. H., & Zhang, J.. A discrete multiobjective particle swarm optimizer for automated assembly of parallel cognitive diagnosis tests. IEEE Transactions on Cybernetics, 2019, 49 (7): 2792- 2805. DOI
	Mullis, I. V. S., Martin, M. O., Ruddock, G. J., O’Sullivan, C. Y., Christine, Y., Arora, A., & Erberber, E. (2005). Timss 2007 assessment frameworks. Boston: International Association for the Evaluation of Educational Achievement.
	Rupp, A. A., Templin, J., & Henson, R. J. (2010). Diagnostic measurement: Theory, methods, and applications. New York: The Guilford Press.
	von Davier, M.. A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 2008, 61 (2): 287- 307. DOI
	von Davier, M., & Lee, Y. S. (2019). Handbook diagnostic classification models. New York: Springer.
	Wang, S. Y., Zheng, Y., Zheng, C. J., Su, Y. H., & Li, P. Z.. An automated test assembly design for a large-scale Chinese proficiency test. Applied Psychological Measurement, 2016, 40 (3): 233- 237. DOI
	Zeng, L. Y., Ding, S. L., & Gan, D. W. (2010). Test construction for cognitive diagnosis. In 2010 Asia-Pacific conference on wearable computing systems (pp. 12–15). Shenzhen: IEEE.
	Zhan, P. D., Wang, W. C., & Wang, L. (2017). A higher-order cognitive diagnosis model for polytomous attributes and polytomous responses. Poster presented at the annual meeting of the National Council on Measurement in Education, San Antonio.

因素	水平
题库类型	二级计分、多级计分
属性结构	独立型、线性型、分支型、收敛型
组卷长度	20、30、40
组卷方法	RD(随机组卷)、CDI、ADI、GMCDI、GMADI

因素	水平
题库类型	二级计分、多级计分
属性结构	独立型、线性型、分支型、收敛型
组卷长度	20、30、40
组卷方法	RD(随机组卷)、CDI、ADI、GMCDI、GMADI

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $
独立型	RD	0.265	0.281	0.016	0.380	0.391	0.011	0.469	0.505	0.036
	CDI	0.513	0.511	−0.002	0.689	0.671	−0.018	0.782	0.762	−0.020
	ADI	0.470	0.443	−0.027	0.652	0.626	−0.026	0.754	0.741	−0.013
	GMCDI	0.599	0.602	0.003	0.765	0.771	0.006	0.860	0.868	0.008
	GMADI	0.562	0.539	−0.023	0.732	0.735	0.003	0.832	0.846	0.014
线性型	RD	0.642	0.655	0.013	0.764	0.783	0.019	0.829	0.853	0.024
	CDI	0.606	0.597	−0.009	0.674	0.655	−0.019	0.715	0.695	−0.020
	ADI	0.680	0.671	−0.009	0.761	0.755	−0.006	0.797	0.792	−0.005
	GMCDI	0.906	0.915	0.009	0.950	0.958	0.008	0.977	0.983	0.006
	GMADI	0.845	0.852	0.007	0.895	0.900	0.005	0.943	0.948	0.005
分支型	RD	0.508	0.557	0.049	0.652	0.679	0.027	0.754	0.772	0.018
	CDI	0.598	0.569	−0.029	0.697	0.688	−0.009	0.779	0.769	−0.010
	ADI	0.520	0.522	0.002	0.671	0.663	−0.008	0.757	0.752	−0.005
	GMCDI	0.852	0.877	0.025	0.929	0.939	0.010	0.942	0.952	0.010
	GMADI	0.803	0.811	0.008	0.884	0.889	0.005	0.938	0.946	0.008
收敛型	RD	0.505	0.518	0.013	0.638	0.667	0.029	0.713	0.747	0.034
	CDI	0.626	0.612	−0.014	0.721	0.703	−0.018	0.776	0.763	−0.013
	ADI	0.519	0.489	−0.030	0.662	0.649	−0.013	0.750	0.754	0.004
	GMCDI	0.840	0.856	0.016	0.905	0.918	0.013	0.928	0.936	0.008
	GMADI	0.811	0.827	0.016	0.902	0.916	0.014	0.917	0.925	0.008

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $
独立型	RD	0.265	0.281	0.016	0.380	0.391	0.011	0.469	0.505	0.036
	CDI	0.513	0.511	−0.002	0.689	0.671	−0.018	0.782	0.762	−0.020
	ADI	0.470	0.443	−0.027	0.652	0.626	−0.026	0.754	0.741	−0.013
	GMCDI	0.599	0.602	0.003	0.765	0.771	0.006	0.860	0.868	0.008
	GMADI	0.562	0.539	−0.023	0.732	0.735	0.003	0.832	0.846	0.014
线性型	RD	0.642	0.655	0.013	0.764	0.783	0.019	0.829	0.853	0.024
	CDI	0.606	0.597	−0.009	0.674	0.655	−0.019	0.715	0.695	−0.020
	ADI	0.680	0.671	−0.009	0.761	0.755	−0.006	0.797	0.792	−0.005
	GMCDI	0.906	0.915	0.009	0.950	0.958	0.008	0.977	0.983	0.006
	GMADI	0.845	0.852	0.007	0.895	0.900	0.005	0.943	0.948	0.005
分支型	RD	0.508	0.557	0.049	0.652	0.679	0.027	0.754	0.772	0.018
	CDI	0.598	0.569	−0.029	0.697	0.688	−0.009	0.779	0.769	−0.010
	ADI	0.520	0.522	0.002	0.671	0.663	−0.008	0.757	0.752	−0.005
	GMCDI	0.852	0.877	0.025	0.929	0.939	0.010	0.942	0.952	0.010
	GMADI	0.803	0.811	0.008	0.884	0.889	0.005	0.938	0.946	0.008
收敛型	RD	0.505	0.518	0.013	0.638	0.667	0.029	0.713	0.747	0.034
	CDI	0.626	0.612	−0.014	0.721	0.703	−0.018	0.776	0.763	−0.013
	ADI	0.519	0.489	−0.030	0.662	0.649	−0.013	0.750	0.754	0.004
	GMCDI	0.840	0.856	0.016	0.905	0.918	0.013	0.928	0.936	0.008
	GMADI	0.811	0.827	0.016	0.902	0.916	0.014	0.917	0.925	0.008

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $	二级计分	多级计分	P$\uparrow $
独立型	RD	0.694	0.708	0.014	0.762	0.771	0.009	0.806	0.827	0.021
	CDI	0.879	0.880	0.001	0.928	0.926	−0.002	0.952	0.948	−0.004
	ADI	0.849	0.843	−0.006	0.908	0.904	−0.004	0.937	0.936	−0.001
	GMCDI	0.902	0.906	0.004	0.948	0.948		0.968	0.971	0.003
	GMADI	0.879	0.876	−0.003	0.930	0.932	0.002	0.957	0.962	0.005
线性型	RD	0.899	0.902	0.003	0.940	0.945	0.005	0.959	0.965	0.006
	CDI	0.853	0.850	−0.003	0.875	0.874	−0.001	0.890	0.888	−0.002
	ADI	0.848	0.838	−0.010	0.882	0.878	−0.004	0.897	0.895	−0.002
	GMCDI	0.974	0.977	0.003	0.989	0.991	0.002	0.995	0.996	0.001
	GMADI	0.937	0.939	0.002	0.964	0.965	0.001	0.985	0.987	0.002
分支型	RD	0.844	0.866	0.022	0.900	0.909	0.009	0.935	0.940	0.005
	CDI	0.890	0.882	−0.008	0.921	0.919	−0.002	0.944	0.941	−0.003
	ADI	0.822	0.821	−0.001	0.879	0.874	−0.005	0.910	0.907	−0.003
	GMCDI	0.960	0.966	0.006	0.983	0.986	0.003	0.986	0.988	0.002
	GMADI	0.925	0.929	0.004	0.963	0.963		0.985	0.987	0.002
收敛型	RD	0.835	0.841	0.006	0.892	0.904	0.012	0.920	0.933	0.013
	CDI	0.899	0.899		0.929	0.927	−0.002	0.945	0.943	−0.002
	ADI	0.832	0.824	−0.008	0.884	0.880	−0.004	0.913	0.914	0.001
	GMCDI	0.957	0.962	0.005	0.971	0.975	0.004	0.979	0.982	0.003
	GMADI	0.932	0.939	0.007	0.970	0.974	0.004	0.975	0.977	0.002