项目区分度指标在属性多水平和混合计分项目下的组卷研究

doi:10.12139/j.1672-0628.2023.06.006

摘要/Abstract

摘要： 将两种项目区分度指标(CDI、ADI)及两种扩展项目区分度指标(GMCDI、GMADI)用于属性多水平诊断测验组卷。考虑三种实际测验情景，开展两项研究：研究1，二级与多级计分项目独立组卷测验；研究2，二级与多级计分项目混合组卷测验。结果表明，GMCDI与GMADI组卷方法可获得最高诊断精度，且不受属性层级结构与测验类型影响。

关键词: 项目区分度指标, 认知诊断自动测验组卷, 属性多水平, 二级计分, 多级计分

Abstract: In this study, two item discrimination indices (CDI, ADI) and two general modified item discrimination indices (GMCDI, GMADI) were used for polytomous attribute diagnosis test assembly. Two simulation studies were conducted considering three real-world testing scenarios: Study 1, an independent test of dichotomous and polytomous scoring items; Study 2, a mixed test of dichotomous and polytomous scoring items. The results indicate that GMCDI and GMADI can obtain the highest diagnosis accuracy, and are not affected by attribute hierarchy and test type.

Key words: item discrimination index, CD-ATA, polytomous attributes, dichotomous scoring, polytomous scoring

中图分类号:

B842

马大付, 秦春影, 喻晓锋, 何催. 项目区分度指标在属性多水平和混合计分项目下的组卷研究[J]. 心理与行为研究, 2023, 21(6): 760-769.

Dafu MA, Chunying QIN, Xiaofeng YU, Cui HE. Research on Test Assembly of Item Discrimination Index Under Polytomous Attributes and Mixed Scoring Items[J]. Studies of Psychology and Behavior, 2023, 21(6): 760-769.

图/表 8

图1 属性多水平插补过程

表1 研究1四因素水平

因素	水平
题库类型	二级计分、多级计分
属性结构	独立型、线性型、分支型、收敛型
组卷长度	20、30、40
组卷方法	RD(随机组卷)、CDI、ADI、GMCDI、GMADI

图2 五种组卷方法多级计分组卷模式判准率

图3 五种组卷方法多级计分组卷属性判准率

表2 二级与多级计分项目组卷模式判准率比较

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$
独立型	RD	0.265	0.281	0.016	0.380	0.391	0.011	0.469	0.505	0.036
	CDI	0.513	0.511	−0.002	0.689	0.671	−0.018	0.782	0.762	−0.020
	ADI	0.470	0.443	−0.027	0.652	0.626	−0.026	0.754	0.741	−0.013
	GMCDI	0.599	0.602	0.003	0.765	0.771	0.006	0.860	0.868	0.008
	GMADI	0.562	0.539	−0.023	0.732	0.735	0.003	0.832	0.846	0.014
线性型	RD	0.642	0.655	0.013	0.764	0.783	0.019	0.829	0.853	0.024
	CDI	0.606	0.597	−0.009	0.674	0.655	−0.019	0.715	0.695	−0.020
	ADI	0.680	0.671	−0.009	0.761	0.755	−0.006	0.797	0.792	−0.005
	GMCDI	0.906	0.915	0.009	0.950	0.958	0.008	0.977	0.983	0.006
	GMADI	0.845	0.852	0.007	0.895	0.900	0.005	0.943	0.948	0.005
分支型	RD	0.508	0.557	0.049	0.652	0.679	0.027	0.754	0.772	0.018
	CDI	0.598	0.569	−0.029	0.697	0.688	−0.009	0.779	0.769	−0.010
	ADI	0.520	0.522	0.002	0.671	0.663	−0.008	0.757	0.752	−0.005
	GMCDI	0.852	0.877	0.025	0.929	0.939	0.010	0.942	0.952	0.010
	GMADI	0.803	0.811	0.008	0.884	0.889	0.005	0.938	0.946	0.008
收敛型	RD	0.505	0.518	0.013	0.638	0.667	0.029	0.713	0.747	0.034
	CDI	0.626	0.612	−0.014	0.721	0.703	−0.018	0.776	0.763	−0.013
	ADI	0.519	0.489	−0.030	0.662	0.649	−0.013	0.750	0.754	0.004
	GMCDI	0.840	0.856	0.016	0.905	0.918	0.013	0.928	0.936	0.008
	GMADI	0.811	0.827	0.016	0.902	0.916	0.014	0.917	0.925	0.008

表2 二级与多级计分项目组卷模式判准率比较

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$
独立型	RD	0.265	0.281	0.016	0.380	0.391	0.011	0.469	0.505	0.036
	CDI	0.513	0.511	−0.002	0.689	0.671	−0.018	0.782	0.762	−0.020
	ADI	0.470	0.443	−0.027	0.652	0.626	−0.026	0.754	0.741	−0.013
	GMCDI	0.599	0.602	0.003	0.765	0.771	0.006	0.860	0.868	0.008
	GMADI	0.562	0.539	−0.023	0.732	0.735	0.003	0.832	0.846	0.014
线性型	RD	0.642	0.655	0.013	0.764	0.783	0.019	0.829	0.853	0.024
	CDI	0.606	0.597	−0.009	0.674	0.655	−0.019	0.715	0.695	−0.020
	ADI	0.680	0.671	−0.009	0.761	0.755	−0.006	0.797	0.792	−0.005
	GMCDI	0.906	0.915	0.009	0.950	0.958	0.008	0.977	0.983	0.006
	GMADI	0.845	0.852	0.007	0.895	0.900	0.005	0.943	0.948	0.005
分支型	RD	0.508	0.557	0.049	0.652	0.679	0.027	0.754	0.772	0.018
	CDI	0.598	0.569	−0.029	0.697	0.688	−0.009	0.779	0.769	−0.010
	ADI	0.520	0.522	0.002	0.671	0.663	−0.008	0.757	0.752	−0.005
	GMCDI	0.852	0.877	0.025	0.929	0.939	0.010	0.942	0.952	0.010
	GMADI	0.803	0.811	0.008	0.884	0.889	0.005	0.938	0.946	0.008
收敛型	RD	0.505	0.518	0.013	0.638	0.667	0.029	0.713	0.747	0.034
	CDI	0.626	0.612	−0.014	0.721	0.703	−0.018	0.776	0.763	−0.013
	ADI	0.519	0.489	−0.030	0.662	0.649	−0.013	0.750	0.754	0.004
	GMCDI	0.840	0.856	0.016	0.905	0.918	0.013	0.928	0.936	0.008
	GMADI	0.811	0.827	0.016	0.902	0.916	0.014	0.917	0.925	0.008

表3 二级与多级计分项目组卷属性判准率比较

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$
独立型	RD	0.694	0.708	0.014	0.762	0.771	0.009	0.806	0.827	0.021
	CDI	0.879	0.880	0.001	0.928	0.926	−0.002	0.952	0.948	−0.004
	ADI	0.849	0.843	−0.006	0.908	0.904	−0.004	0.937	0.936	−0.001
	GMCDI	0.902	0.906	0.004	0.948	0.948		0.968	0.971	0.003
	GMADI	0.879	0.876	−0.003	0.930	0.932	0.002	0.957	0.962	0.005
线性型	RD	0.899	0.902	0.003	0.940	0.945	0.005	0.959	0.965	0.006
	CDI	0.853	0.850	−0.003	0.875	0.874	−0.001	0.890	0.888	−0.002
	ADI	0.848	0.838	−0.010	0.882	0.878	−0.004	0.897	0.895	−0.002
	GMCDI	0.974	0.977	0.003	0.989	0.991	0.002	0.995	0.996	0.001
	GMADI	0.937	0.939	0.002	0.964	0.965	0.001	0.985	0.987	0.002
分支型	RD	0.844	0.866	0.022	0.900	0.909	0.009	0.935	0.940	0.005
	CDI	0.890	0.882	−0.008	0.921	0.919	−0.002	0.944	0.941	−0.003
	ADI	0.822	0.821	−0.001	0.879	0.874	−0.005	0.910	0.907	−0.003
	GMCDI	0.960	0.966	0.006	0.983	0.986	0.003	0.986	0.988	0.002
	GMADI	0.925	0.929	0.004	0.963	0.963		0.985	0.987	0.002
收敛型	RD	0.835	0.841	0.006	0.892	0.904	0.012	0.920	0.933	0.013
	CDI	0.899	0.899		0.929	0.927	−0.002	0.945	0.943	−0.002
	ADI	0.832	0.824	−0.008	0.884	0.880	−0.004	0.913	0.914	0.001
	GMCDI	0.957	0.962	0.005	0.971	0.975	0.004	0.979	0.982	0.003
	GMADI	0.932	0.939	0.007	0.970	0.974	0.004	0.975	0.977	0.002

表3 二级与多级计分项目组卷属性判准率比较

属性结构	组卷方法	20			30			40
属性结构	组卷方法	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$	二级计分	多级计分	P $↑$
独立型	RD	0.694	0.708	0.014	0.762	0.771	0.009	0.806	0.827	0.021
	CDI	0.879	0.880	0.001	0.928	0.926	−0.002	0.952	0.948	−0.004
	ADI	0.849	0.843	−0.006	0.908	0.904	−0.004	0.937	0.936	−0.001
	GMCDI	0.902	0.906	0.004	0.948	0.948		0.968	0.971	0.003
	GMADI	0.879	0.876	−0.003	0.930	0.932	0.002	0.957	0.962	0.005
线性型	RD	0.899	0.902	0.003	0.940	0.945	0.005	0.959	0.965	0.006
	CDI	0.853	0.850	−0.003	0.875	0.874	−0.001	0.890	0.888	−0.002
	ADI	0.848	0.838	−0.010	0.882	0.878	−0.004	0.897	0.895	−0.002
	GMCDI	0.974	0.977	0.003	0.989	0.991	0.002	0.995	0.996	0.001
	GMADI	0.937	0.939	0.002	0.964	0.965	0.001	0.985	0.987	0.002
分支型	RD	0.844	0.866	0.022	0.900	0.909	0.009	0.935	0.940	0.005
	CDI	0.890	0.882	−0.008	0.921	0.919	−0.002	0.944	0.941	−0.003
	ADI	0.822	0.821	−0.001	0.879	0.874	−0.005	0.910	0.907	−0.003
	GMCDI	0.960	0.966	0.006	0.983	0.986	0.003	0.986	0.988	0.002
	GMADI	0.925	0.929	0.004	0.963	0.963		0.985	0.987	0.002
收敛型	RD	0.835	0.841	0.006	0.892	0.904	0.012	0.920	0.933	0.013
	CDI	0.899	0.899		0.929	0.927	−0.002	0.945	0.943	−0.002
	ADI	0.832	0.824	−0.008	0.884	0.880	−0.004	0.913	0.914	0.001
	GMCDI	0.957	0.962	0.005	0.971	0.975	0.004	0.979	0.982	0.003
	GMADI	0.932	0.939	0.007	0.970	0.974	0.004	0.975	0.977	0.002

表4 五种组卷方法30题时的混合计分模式判准率

属性结构	组卷方法	组卷比例									二级计分	多级计分
属性结构	组卷方法	1∶9	2∶8	3∶7	4∶6	5∶5	6∶4	7∶3	8∶2	9∶1	二级计分	多级计分
独立型	RD	0.401	0.385	0.384	0.382	0.370	0.377	0.379	0.370	0.367	0.380	0.391
	CDI	0.607	0.603	0.600	0.608	0.605	0.616	0.620	0.628	0.647	0.689	0.671
	ADI	0.604	0.606	0.595	0.595	0.595	0.597	0.591	0.606	0.616	0.652	0.626
	GMCDI	0.768	0.777	0.776	0.784	0.776	0.767	0.775	0.782	0.773	0.765	0.771
	GMADI	0.736	0.739	0.738	0.745	0.749	0.745	0.742	0.743	0.737	0.732	0.735
线性型	RD	0.774	0.775	0.761	0.764	0.759	0.759	0.752	0.766	0.755	0.764	0.783
	CDI	0.630	0.622	0.621	0.619	0.625	0.627	0.629	0.631	0.638	0.674	0.655
	ADI	0.725	0.722	0.715	0.705	0.700	0.702	0.704	0.706	0.719	0.761	0.755
	GMCDI	0.954	0.954	0.954	0.949	0.948	0.946	0.944	0.942	0.941	0.950	0.958
	GMADI	0.875	0.875	0.873	0.870	0.864	0.863	0.865	0.868	0.870	0.895	0.900
分支型	RD	0.663	0.680	0.672	0.653	0.651	0.654	0.659	0.647	0.642	0.652	0.679
	CDI	0.676	0.663	0.660	0.658	0.667	0.684	0.680	0.680	0.691	0.697	0.688
	ADI	0.695	0.693	0.680	0.682	0.676	0.681	0.685	0.685	0.694	0.671	0.663
	GMCDI	0.931	0.931	0.929	0.929	0.927	0.925	0.923	0.923	0.922	0.929	0.939
	GMADI	0.894	0.894	0.894	0.894	0.892	0.890	0.888	0.885	0.883	0.884	0.889
收敛型	RD	0.671	0.655	0.649	0.661	0.650	0.642	0.644	0.633	0.651	0.638	0.667
	CDI	0.708	0.701	0.703	0.691	0.690	0.696	0.692	0.694	0.696	0.721	0.703
	ADI	0.652	0.635	0.629	0.622	0.630	0.627	0.646	0.647	0.662	0.662	0.649
	GMCDI	0.922	0.922	0.922	0.921	0.918	0.916	0.914	0.911	0.908	0.905	0.918
	GMADI	0.918	0.918	0.917	0.917	0.915	0.913	0.910	0.907	0.904	0.902	0.916

表5 五种组卷方法30题时的混合计分属性判准率

属性结构	组卷方法	组卷比例									二级计分	多级计分
属性结构	组卷方法	1∶9	2∶8	3∶7	4∶6	5∶5	6∶4	7∶3	8∶2	9∶1	二级计分	多级计分
独立型	RD	0.778	0.768	0.765	0.764	0.756	0.762	0.762	0.756	0.752	0.762	0.771
	CDI	0.911	0.910	0.910	0.912	0.911	0.913	0.913	0.916	0.920	0.928	0.926
	ADI	0.898	0.899	0.896	0.895	0.895	0.895	0.894	0.898	0.900	0.908	0.904
	GMCDI	0.949	0.951	0.951	0.953	0.951	0.949	0.951	0.952	0.949	0.948	0.948
	GMADI	0.934	0.935	0.935	0.937	0.938	0.937	0.936	0.936	0.934	0.930	0.932
线性型	RD	0.944	0.944	0.940	0.941	0.940	0.939	0.938	0.942	0.939	0.940	0.945
	CDI	0.868	0.866	0.865	0.865	0.867	0.866	0.866	0.868	0.870	0.875	0.874
	ADI	0.867	0.867	0.864	0.859	0.857	0.857	0.859	0.860	0.865	0.882	0.878
	GMCDI	0.990	0.990	0.990	0.989	0.989	0.988	0.988	0.987	0.987	0.989	0.991
	GMADI	0.956	0.956	0.954	0.952	0.949	0.949	0.950	0.952	0.955	0.964	0.965
分支型	RD	0.906	0.908	0.906	0.899	0.897	0.900	0.903	0.899	0.894	0.900	0.909
	CDI	0.919	0.915	0.914	0.914	0.916	0.920	0.920	0.920	0.922	0.921	0.919
	ADI	0.890	0.888	0.885	0.886	0.884	0.888	0.889	0.888	0.891	0.879	0.874
	GMCDI	0.983	0.984	0.983	0.983	0.982	0.981	0.981	0.981	0.981	0.983	0.986
	GMADI	0.966	0.966	0.966	0.966	0.966	0.965	0.964	0.963	0.962	0.963	0.963
收敛型	RD	0.906	0.898	0.896	0.904	0.898	0.896	0.895	0.892	0.896	0.892	0.904
	CDI	0.928	0.926	0.928	0.925	0.923	0.925	0.924	0.923	0.924	0.929	0.927
	ADI	0.880	0.875	0.871	0.869	0.872	0.873	0.880	0.881	0.884	0.884	0.880
	GMCDI	0.977	0.977	0.977	0.977	0.976	0.976	0.975	0.974	0.974	0.971	0.975
	GMADI	0.976	0.976	0.976	0.975	0.975	0.974	0.974	0.973	0.972	0.970	0.974

参考文献

	蔡艳, 涂冬波. 属性多级化的认知诊断模型拓展及其Q矩阵设计. 心理学报, 2015, 47 (10): 1300- 1308.
	丁树良, 汪文义, 罗芬, 熊建华. (2016). 可达阵功能的不可替代性. 江西师范大学学报(自然科学版), 40(3), 290–294, 298.
	丁树良, 杨淑群, 汪文义. 可达矩阵在认知诊断测验编制中的重要作用. 江西师范大学学报(自然科学版), 2010, 34 (5): 490- 494.
	唐小娟, 丁树良, 毛萌萌, 俞宗火. 基于属性层级结构的认知诊断测验的组卷. 心理学探新, 2013, 33 (3): 252- 259.
	涂冬波, 蔡艳, 戴海琦, 丁树良. 一种多级评分的认知诊断模型: P-DINA模型的开发. 心理学报, 2010, 42 (10): 1011- 1020.
	汪文义, 宋丽红, 丁树良. 分类视角下认知诊断测验项目区分度指标及应用. 心理科学, 2018, 41 (2): 475- 483.
	王立君, 赵少勇, 昌维, 唐芳, 詹沛达. (2022). 重参数化多分属性DINA模型的多级评分拓广——基于等级反应模型. 心理科学, 45(1), 195–203.
	詹沛达, 边玉芳, 王立君. 重参数化的多分属性诊断分类模型及其判准率影响因素. 心理学报, 2016, 48 (3): 318- 330.
	Becker, B., Debeer, D., Sachse, K. A., & Weirich, S.. Automated test assembly in R: The eatATA package. Psych, 2021, 3 (2): 96- 112. DOI
	Chang, H. H., & Ying, Z. L.. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 1996, 20 (3): 213- 229. DOI
	Chen, J. S., & de la Torre, J.. A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 2013, 37 (6): 419- 437. DOI
	Finkelman, M., Kim, W., & Roussos, L. A.. Automated test assembly for cognitive diagnosis models using a genetic algorithm. Journal of Educational Measurement, 2009, 46 (3): 273- 292. DOI
	Finkelman, M. D., de la Torre, J., & Karp, J. A.. Cognitive diagnosis models and automated test assembly: An approach incorporating response times. International Journal of Testing, 2020, 20 (4): 299- 320. DOI
	Finkelman, M. D., Kim, W., Roussos, L., & Verschoor, A.. A binary programming approach to automated test assembly for cognitive diagnosis models. Applied Psychological Measurement, 2010, 34 (5): 310- 326. DOI
	Henson, R., & Douglas, J.. Test construction for cognitive diagnosis. Applied Psychological Measurement, 2005, 29 (4): 262- 277. DOI
	Henson, R., Roussos, L., Douglas, J., & He, X. M.. Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 2008, 32 (4): 275- 288. DOI
	Huebner, A., & Wang, C.. A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 2011, 71 (2): 407- 419. DOI
	Kuo, B. C., Pai, H. S., & de la Torre, J.. Modified cognitive diagnostic index and modified attribute-level discrimination index for test construction. Applied Psychological Measurement, 2016, 40 (5): 315- 330. DOI
	Leighton, J. P., & Gierl, M. J.. Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 2007, 26 (2): 3- 16. DOI
	Lin, Y., Gong, Y. J., & Zhang, J.. An adaptive ant colony optimization algorithm for constructing cognitive diagnosis tests. Applied Soft Computing, 2017, 52, 1- 13. DOI
	Lin, Y., Jiang, Y. S., Gong, Y. J., Zhan, Z. H., & Zhang, J.. A discrete multiobjective particle swarm optimizer for automated assembly of parallel cognitive diagnosis tests. IEEE Transactions on Cybernetics, 2019, 49 (7): 2792- 2805. DOI
	Mullis, I. V. S., Martin, M. O., Ruddock, G. J., O’Sullivan, C. Y., Christine, Y., Arora, A., & Erberber, E. (2005). Timss 2007 assessment frameworks. Boston: International Association for the Evaluation of Educational Achievement.
	Rupp, A. A., Templin, J., & Henson, R. J. (2010). Diagnostic measurement: Theory, methods, and applications. New York: The Guilford Press.
	von Davier, M.. A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 2008, 61 (2): 287- 307. DOI
	von Davier, M., & Lee, Y. S. (2019). Handbook diagnostic classification models. New York: Springer.
	Wang, S. Y., Zheng, Y., Zheng, C. J., Su, Y. H., & Li, P. Z.. An automated test assembly design for a large-scale Chinese proficiency test. Applied Psychological Measurement, 2016, 40 (3): 233- 237. DOI
	Zeng, L. Y., Ding, S. L., & Gan, D. W. (2010). Test construction for cognitive diagnosis. In 2010 Asia-Pacific conference on wearable computing systems (pp. 12–15). Shenzhen: IEEE.
	Zhan, P. D., Wang, W. C., & Wang, L. (2017). A higher-order cognitive diagnosis model for polytomous attributes and polytomous responses. Poster presented at the annual meeting of the National Council on Measurement in Education, San Antonio.

编辑推荐 0

Metrics

阅读次数

全文

253

HTML			PDF

最新录用	在线预览	正式出版	最新录用	在线预览	正式出版
0	0	10	0	0	243

来源	本网站	其他网站

次数	193	60
比例	76%	24%

摘要

220

最新录用	在线预览	正式出版

0	0	220

	来源	本网站

	次数	220
	比例	100%