[1]秦春影,吴龙月,王爱平*.计算机自适应测验中试题泄露的实时监控方法研究与应用[J].江西师范大学学报(自然科学版),2022,(02):118-125.[doi:10.16357/j.cnki.issn1000-5862.2022.02.02]
　QIN Chunying,WU Longyue,WANG Aiping*.The Study and Application of Real-Time Monitoring Methods for Item Leakage in Computerized Adaptive Tests[J].Journal of Jiangxi Normal University:Natural Science Edition,2022,(02):118-125.[doi:10.16357/j.cnki.issn1000-5862.2022.02.02]
点击复制

计算机自适应测验中试题泄露的实时监控方法研究与应用()
分享到：

《江西师范大学学报》（自然科学版）[ISSN:1006-6977/CN:61-1281/TN]

卷:
期数:
2022年02期
页码:
118-125
栏目:
心理与教育测量
出版日期:
2022-03-25

文章信息/Info

Title:
The Study and Application of Real-Time Monitoring Methods for Item Leakage in Computerized Adaptive Tests
文章编号:
1000-5862(2022)02-0118-08
作者:
秦春影1; 吴龙月1; 王爱平2*
1.南昌师范学院数学与信息科学学院,江西 南昌 330032; 2.亳州学院电子与信息工程系,安徽 亳州 246800
Author(s):
QIN Chunying1; WU Longyue1; WANG Aiping2*
1.School of Mathematics and Information Science,Nanchang Normal University,Nanchang Jiangxi 330032,China; 2.Department of Electronic and Information Engineering,Bozhou University,Bozhou Anhui 246800,China
关键词:
计算机自适应测验;  被试拟合指标;  序贯监测程序;  残差;  试题安全
Keywords:
computerized adaptive test;  test fit index;  sequential monitoring procedures;  residual;  item security
分类号:
B 814.7
DOI:
10.16357/j.cnki.issn1000-5862.2022.02.02
文献标志码:
A
摘要:
在连续施测下计算机自适应测验(CAT)中的试题被曝光的可能性急剧增加,因此需要对试题进行实时监控,当试题的参数发生显著性变化时必须将其进行强制“退休”.序贯监测程序(SMP)通过检测CAT中的试题统计特征的变化来判断试题是否泄露; 然而在用SMP监控试题时会出现较大的Ⅰ类错误率,并且在一些条件下其统计检验力较低.该文以残差的R指标作为考生拟合统计量(PFS),与SMP方法相结合,构建了一种新的监测方法(PFS_SMP); 该方法以被试作答信息为依据判断被SMP标记的试题是否泄露,从试题和被试这2个层面保证测验的安全性和公平性.最后,通过模拟实验和实证分析来对基于R的PFS_SMP的表现进行评价,实验结果表明:PFS_SMP方法能降低在SMP监测试题时的Ⅰ类错误,并能提高其统计检验力.
Abstract:
Computerized Adaptive Test(CAT)makes the possibility of each item being exposed increase,so the item needs to be monitored in real time.When the item parameters change significantly,it must be forced to "retire".The sequential monitoring program(SMP)is proposed in 2014 to determine whether an item is leaking by detecting changes in the statistical characteristics of the item in CAT.However,when using SMP to monitor the item,there will be a relatively high error rate of type I,and the statistical test will also have a greater impact. In this paper,based on the residual person fit statistic R,combined with the SMP method,a new monitoring method(PFS_SMP)is proposed.The PFS_SMP method can be applied to determine whether each respondent takes aberrant response behavior,and each item is known by the future respondents during the CAT,and to ensure the safety and fairness of the test.Finally,a simulation study and an empirical study are considered,and the results show that the PFS_SMP method can yield a well-controlled error rate of type I,and have a promising power as well.

参考文献/References:

[1] VAN DER LINDEN W J,GLAS C A W.Computerized adaptive testing:theory and practice[M].Berlin:Springer,2000.
[2] MAGIS D,YAN Duanli,VON DAVIER A A.Computerized adaptive and multistage testing with R:using packages catR and mstR[M].Switzerland:Springer International Publishing,2017.
[3] 唐倩,毛秀珍,何明霜,等.认知诊断计算机化自适应测验的选题策略[J].心理科学进展,2020,28(12):2160-2168.
[4] 于建芳,徐振国,刘剑.计算机自适应测试系统研究综述[J].中国教育技术装备,2015(6):28-29.
[5] 杨业兵.应用项目反应理论对《中国士兵人格测验》的项目分析及计算机自适应施测方案[D].西安:第四军医大学,2008.
[6] 涂冬波.项目自动生成的小学儿童数学问题解决认知诊断CAT编制[D].南昌:江西师范大学,2009.
[7] 邓远平.基于展开反应机制的计算机化自适应人格测验研究[D].南昌:江西师范大学,2014.
[8] YI Qing,ZHANG Jinming,CHANG Huahua.Severity of organized item theft in computerized adaptive testing:a simulation study[J].Applied Psychological Measurement,2008,32(7):543-558.
[9] ZHANG Jinming,CHANG Huahua,YI Qing.Comparing single-pool and multiple-pool designs regarding test security in computerized testing[J].Behavior Research Methods,2012,44(3):742-752.
[10] STOCKING M L,LEWIS C.Controlling item exposure conditional on ability in computerized adaptive testing[J].Journal of Educational and Behavioral Statistics,1998,23(1):57-75.
[11] MARTHA L S,CHARLES L.Controlling item exposure conditional on ability in computerized adaptive testing[EB/OL].[2021-09-12].https://onlinelibrary.wiley.com/doi/10.1002/j.2333-8504.1995.tb01659.x.
[12] WAINER H.Rescuing computerized testing by breaking Zipf 's law[J].Journal of Educational and Behavioral Statistics,2000,25(2):203-224.
[13] WAY W D.Protecting the integrity of computerized testing item pools[J].Educational Measurement:Issues and Practice,1998,17(4):17-27.
[14] ZHANG Jinming.A sequential procedure for detecting compromised items in the item pool of a CAT system[J].Applied Psychological Measurement,2014,38(2):87-104.
[15] MEIJER R R,SIJTSMA K.Methodology review:evaluating person fit[J].Applied Psychological Measurement,2001,25(2):107-135.
[16] BAKER F B,KIM S H.Item response theory:parameter estimation techniques[M].2nd ed.New York:Marcel Dekker,2004.
[17] MCLEOD L,LEWIS C,THISSEN D.A Bayesian method for the detection of item preknowledge in computerized adaptive testing[J].Applied Psychological Measurement,2003,27(2):121-137.
[18] 郭磊,刘伟.CAT中结合贝叶斯方法与序贯监测程序的题库质量监控技术[J].心理科学,2018,41(1):189-195.
[19] GEORGIADOU E,TRIANTAFILLOU E,ECONOMIDES A A.A review of item exposure control strategies for computerized adaptive testing developed from 1983 to 2005[J].The Journal of Technology,Learning,and Assessment,2007,5(8):4-38.
[20] 张金明,曹灿兮,揭勇菁.实时监控计算机自适应考题的两种方法及其稳健性比较[J].中国考试,2017(2):20-32.
[21] YU Xiaofeng,CHENG Ying.A change-point analysis procedure based on weighted residuals to detect back random responding[J].Psychological Methods,2019,24(5):658-674.
[22] PETRIDOU A,Williams J.Accounting for aberrant test response patterns using multilevel models[J].Journal of Educational Measurement,2007,44(3):227-247.
[23] 汪文义,张华华.统计测量视角下考试公平推动教育公平的对策[J].江西师范大学学报(自然科学版),2017,41(4):383-393.

相似文献/References:

[1]胡姗,丁树良,程艳,等.CAT分层终止规则探究[J].江西师范大学学报(自然科学版),2014,(05):445.
　HU Shan,DING Shu-liang,CHENG Yan,et al.Exploration of Hierarchical Termination Rules for CAT[J].Journal of Jiangxi Normal University:Natural Science Edition,2014,(02):445.

备注/Memo

备注/Memo:
收稿日期:2021-10-11
基金项目:江西省教育科学十四五规划2021课题(21YB257)资助项目.
作者简介:秦春影(1981—),女,安徽宿州人,副教授,主要从事教育大数据、教育统计与测量有关的研究.E-mail:qcy_qin@qq.com
通信作者:王爱平(1956—),女,甘肃庆阳人,教授,主要从事人工智能、信息安全等有关的研究.E-mail:710875983@qq.com
更新日期/Last Update: 2022-03-25