Microsoft Word - 19陈平辛涛jz12-477_new_.doc

Size: px

Start display at page:

Download "Microsoft Word - 19陈平辛涛jz12-477_new_.doc"

矿抄柯
5 years ago
Views:

1 2013, Vol. 21, No. 10, Advances in Psychological Science DOI: /SP.J * 在线标定技术在计算机化自适应测验中的应用 ( 1, ) ( 2, ) 计算机化自适应测验 (Computerized Adaptive Testing, CAT) 近年来得到迅猛发展, 题目增补对 CAT 的题库建设与维护至关重要新题标定作为题目增补过程中的技术难点, 它的精度直接影响被试能力估计的准确性, 目前在线标定技术经常用于标定新题从在线标定设计和在线标定方法两个方面, 对两类 CATs ( 以项目反应理论为基础的传统 CAT 以及以认知诊断理论为基础的认知诊断 CAT (Cognitive Diagnostic CAT, CD-CAT) ) 中的在线标定相关研究进行述评传统 CAT 领域有着较丰富的研究, 而 CD-CAT 的在线标定研究则刚刚起步未来研究应进一步探讨在线标定设计 / 方法之间的比较与结合, 以及 CD-CAT 和多维 CAT 的在线标定研究在线标定 ; 计算机化自适应测验 ; 题库建设 ; 认知诊断计算机化自适应测验 B841 1 引言,, (Computerized Adaptive Testing, CAT) (, NCLEX GMAT ASVAB) CAT (Item Response Theory, IRT),,, (,,, 2002) CAT,, CAT (Wainer et al., 1990; Weiss, 1982) CAT ( ) : * :, pchen@bnu.edu.cn 1883 (continuous testing) (Cheng, 2008) CAT 5, CAT, (,, 2011a), Flaugher (2000) CAT 5 (1) (test specification), ; (2)(specialist review) (sensitivity review), ( ) ; (3), ; (4) (Classical Test Theory, CTT) IRT, ; (5) ( SQL Server MySQL) (item stem) (item option) (item key), CAT, 转载

2 , (Wainer & Mislevy, 1990), (2010) ;,, (item replenishing) 1,, (Guo & Wang, 2003),,, (Cheng & Yuan, 2010),, (,, 2011a) Wainer Mislevy (1990) CAT (1) ( ), (calibration sample), Stocking-Lord (Stocking & Lord, 1983) ( A B),, ( ), (linking error) ; (2) (online calibration strategy),, (Wainer & Mislevy, 1990), ( ),, (anchor person),, 1 (calibration), (,, 2011b), CAT (e.g., Chang & Lu, 2010; Makransky, 2009;, 2010) Chang (2012) (2012) CAT IRT CAT, (Cognitive Diagnostic Theory, CDT) CAT (Cognitive Diagnostic CAT, CD-CAT) CD-CAT CAT, /,, CAT, CD-CAT, CD-CAT,, CD-CAT ( ) CAT, CD-CAT ( ), CD-CAT CAT,, Q Chen, Xin, Wang Chang (2010, 2012) CD-CAT, CAT 3 CD-CAT, CD-CAT, CAT CD-CAT, IRT (Multidimensional IRT, MIRT) CAT (Multidimensional CAT, MCAT) CAT (, 2011) MCAT (multitrait), (formative assessment) (Wang & Chang, 2011) MCAT CAT,, ;, MCAT (subscale), (Wang & Chang, 2011) CAT CD-CAT,

3 10 : 1885 Reckase (2009) MCAT MCAT (Reckase, 2009),, MCAT, MCAT, CAT ( CAT CD-CAT) (online calibration design) (online calibration method), 2 CAT, 3 CAT, 2 CAT 中的在线标定设计,,,, Wainer Mislevy (1990)(2010) (seeding new items), CAT CD-CAT 2.1 传统 CAT 中的在线标定设计 Wainer Mislevy (1990), 2 CAT (1),, CAT,,, CAT, CAT,, (Jones & Jin, 1994),,, (2010) CAT (), ; (2) Lord (1980),,,, Wainer Mislevy (1990) Jones Jin (1994) Chang Lu (2010),, Wainer Mislevy (1990), (subjective judgment) Makransky (2009) (occupational testing) ( ), CAT, 3, (Two-Phase Strategy, P2) (Multi-Phase Strategy, M) (Continuous Updating Strategy, C) P2,, 0,,, () ( 50 ), P2 M P2 2,, (M 1) C P2 M, (),, Makransky (2009) C, Makransky (2009) 3

4 , CAT,, 3, CAT CAT,, 0 表 1 多阶段策略实例示意表 Chang Lu (2010) CAT (sequential design), 2,, D- (Fedorov, 1972; Silvey, 1980) ( ), ( ),, CAT CAT,,, CAT,,,,, CAT, 2 (experimental design) (design points), 2.2 CD-CAT 中的在线标定设计 CAT, CD-CAT, (2011) CD-CAT (), CD-CAT, CAT, Chen (2010, 2012) Makransky (2009),, Makransky (2009), (data-based) ( 25%), ( 3 CAT ) (, PC); ( 75%), CD-CAT ; (, RC), Chen (2010, 2012) 3 (1:3, 1:1 3:1), 3 ( 1) 3, 3,, (),,,, (the Deterministic Inputs, Noisy and Gate, DINA)( g s) ( gs, (0.25,0.45) ),

10 : 1887,, 4 ; ( 5 10 ), CAT,, CD-CAT, DINA,, Chen (2010, 2012) 3 CAT 中的在线标定方法 1 3 2.

5 10 : 1887,, 4 ; ( 5 10 ), CAT,, CD-CAT, DINA,, Chen (2010, 2012) 3 CAT 中的在线标定方法 CAT 中在线标定设计的优劣对比 2 CAT ( CAT CD-CAT), 2,, CAT, CAT, Makransky (2009) 3 (C ) CAT, Wainer Mislevy (1990), CAT,, CAT CD-CAT 3.1 传统 CAT 中的在线标定方法 20,, 条件极大似然估计方法 Stocking (1988) A (Method A) B (Method B),, (Conditional Maximum Likelihood Estimation, CMLE) A CMLE ;, 表 2 CAT 中各种在线标定设计优劣对比 CAT (Wainer & Mislevy, 1990) CAT CAT (Wainer & Mislevy, 1990) (Makransky, 2009) CAT, CAT, 0 4 (Chang & Lu, 2010) CAT (, 2011) CAT Chen (2010, 2012) CD-CAT (Chen et al., 2010, 2012),,,

6 (), CMLE CMLE, -(Newton-Raphson, N-R) (bisection method)( N-R, ) A, A,, (scale drift) B A, B,, A, (, ), Stocking-Lord (Stocking & Lord, 1983) B, (2010), CMLE CMLE, b CMLE θ, ˆb, ( ), CMLE a CMLE, CMLE p z ( 2 t 1 + ( e 2 dt = p 2π z (0) rb z b ), r b (0) 2 b b = ), ( a = r 1 r ) (b (0) ) (a (0) ), ˆ θ CMLE b a (2010) ( b (0), ˆb ), a CMLE,,, ( ) CMLE CMLE, CMLE EM 方法 Wainer Mislevy (1990) EM (Marginal Maximum Likelihood Estimate with one EM cycle, OEM) OEM MMLE/EM EM E (), ; M (), E, OEM EM,, Ban, Hanson, Wang, Yi Harris (2001) OEM EM (Marginal Maximum Likelihood Estimate with Multiple EM cycles, MEM)MEM OEM, EM EM 1, MEM OEM, EM EM MEM EM, (E ),, (M ), E M M,

7 10 : 1889 MEM OEM IRT 软件方法 Ban (2001) IRT BILOG (Mislevy & Bock, 1990) BILOG/ Prior, BILOG A B, BILOG/Prior ( ln a ~ N(0,1) b~ N (0,1) ),,,,, (sparse matrix), (e.g., 50) BILOG Ban (2001) 5 ( A B OEM MEM BILOG/ Prior ) 3 ( ),, MEM,, MEM ; A, 3.2 CD-CAT 中的在线标定方法 CD-CAT,, Chen (2010, 2012) CAT 3 (Method A OEM MEM) CD-CAT, CD- Method A CD-OEM CD-MEM,, 3, DINA g s ( gs, (0.05,0.25) ), CD-Method A CD-OEM CD-MEM, A, CD-Method A,, DINA g s ( gs, (0.25,0.45) ) ( ), CD-Method A 3.3 CAT 中在线标定方法的特点比较 3 CAT, 3,, CAT, MEM ;, BILOG/Prior, BILOG MEM, 100 MEM, OEM, CD-CAT, DINA, ( 0.25), CD-Method A ; ( 0.25), CD-OEM 4 问题与展望 CAT,, ( ),, A (Stocking, 1988) CD-Method A (Chen et al., 2012),,,,,, Jones Jin (1994) (Full Functional Maximum Likelihood

8 Estimation, FFMLE), Stefanski Carroll (1985) FFMLE, ( 60) MLE, FFMLE A CD-Method A,, CD-CAT, Chen (2010, 2012) 3 (1:3, 1:1 3:1), ( 1:3, 1:2, 2:3, 4:5, 1:1, 5:4, 3:2, 2:1 3:1),, (DINA )( ), ( Fusion NIDA DINO ) ( ), CD-CAT, Chen (2010, 2012), CD-CAT, ; 表 3 CAT 中各种在线标定方法的特点 CAT CAT CD-CAT A (Stocking, 1988) B (Stocking, 1988) CMLE CMLE (, 2010) OEM (Wainer & Mislevy, 1990) MEM (Ban et al., 2001) BILOG/Prior (Ban et al., 2001) CD-Method A (Chen et al., 2010, 2012) CD-OEM (Chen et al., 2010, 2012) CD-MEM (Chen et al., 2010, 2012), A 3 BILOG, 100 DINA, DINA, CD-Method A CD-OEM CMLE 5, 3 Ban (2001) , BILOG/Prior MEM

9 10 : 1891 Q,, Chen (2010, 2012) DINA,,, IRT (unidimensional),, IRT (,, 2010), IRT IRT (Multidimensional IRT, MIRT) (,,,, 2011) MIRT, () (Reckase, 2009), MIRT MCAT, CAT CD-CAT,, MCAT, MCAT, CAT CD-CAT MCAT,, ;, CAT, Method A OEM MEM MCAT, MCAT,,, MCAT 陈平. (2011). 认知诊断计算机化自适应测验的项目增补以 DINA 模型为例. 博士学位论文, 北京师范大学. 陈平, 辛涛. (2011a). 认知诊断计算机化自适应测验中在线标定方法的开发. 心理学报, 43, 陈平, 辛涛. (2011b). 认知诊断计算机化自适应测验中的项目增补. 心理学报, 43, 康春花, 辛涛. (2010). 测验理论的新发展 : 多维项目反应理论. 心理科学进展, 18, 漆书青, 戴海崎, 丁树良. (2002). 现代教育与心理测量学原理. 北京 : 高等教育出版社. 唐小娟, 丁树良, 俞宗火. (2012). 计算机化自适应测验在认知诊断中的应用. 心理科学进展, 20, 涂冬波, 蔡艳, 戴海琦, 丁树良. (2011). 多维项目反应理论 : 参数估计及其在心理测验中的应用. 心理学报, 43, 汪文义, 丁树良, 游晓锋. (2011). 计算机化自适应诊断测验中原始题的属性标定. 心理学报, 43, 游晓锋, 丁树良, 刘红云. (2010). 计算机化自适应测验中原始题项目参数的估计. 心理学报, 42, Ban, J.-C., Hanson, B. A., Wang, T. Y., Yi, Q., & Harris, D. J. (2001). A comparative study of on-line pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, Chang, H. H. (2012). Making computerized adaptive testing diagnostic tools for schools. In R. W. Lissitz & H. Jiao (Eds.), Computers and their impact on state assessment: Recent history and predictions for the future (pp ). Charlotte, NC: Information Age. Chang, Y. -C. I., & Lu, H. Y. (2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75, Chen, P., Xin, T., Wang, C., & Chang, H. (2010). A comparative study on on-line calibration methods in cognitive diagnostic computerized adaptive testing. Paper presented at the 75 th meeting of the Psychometric Society. Athen, Georgia. Chen, P., Xin, T., Wang, C., & Chang, H. H. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77, Cheng, Y. (2008). Computerized adaptive testing: New developments and applications. Unpublished doctoral thesis, University of Illinois at Urbana-Champaign. Cheng, Y., & Yuan, K. H. (2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75, Flaugher, R. (2000). Item pools. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, & R. J.Mislevy (Eds.), Computerized adaptive testing: A primer (Chap. 3, 2nd ed., pp ). Mahwah, NJ: Erlbaum. Fedorov, V. V. (1972). Theory of optimal design. New York: Academic Press. Guo, F. M., & Wang, L. (2003). Online calibration and scale stability of a CAT program. Paper presented at the annual meeting of National Council on Measurement in Education. Chicago, IL. Jones, D. H., & Jin, Z. Y. (1994). Optimal sequential designs

10 for on-line item estimation. Psychometrika, 59, Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Lawrence Erlbaum Associates. Makransky, G. (2009). An automatic online calibration design in adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA. Mislevy, R. J., & Bock, R. D. (1990). BILOG3: Item analysis and test scoring with binary logistic models (2 nd ed.) [Computer program]. Mooresville, IN: Scientific Software. Reckase, M. D. (2009). Multidimensional item response theory (Chap. 10, pp ). New York: Springer. Silvey, S. D. (1980). Optimal design: An introduction to the theory for parameter estimation. London: Chapman and Hall. Stefanski, L. A., & Carroll, R. J. (1985). Covariate measurement error in logistic regression. Annals of Statistics, 13, Stocking, M. L. (1988). Scale drift in on-line calibration (Research Rep ). Princeton, NJ: ETS. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L., & Thissen, D. (1990). Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum. Wainer, H., & Mislevy, R. J. (1990). Item response theory, item calibration, and proficiency estimation. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (Chap. 4, pp ). Hillsdale, NJ: Erlbaum. Wang, C., & Chang, H. H. (2011). Item selection in multidimensional computerized adaptive testing gaining information from different angles. Psychometrika, 76, Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, Application of Online Calibration Technique in Computerized Adaptive Testing CHEN Ping 1 ; ZHANG Jiahui 2 ; XIN Tao 2 ( 1 State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing , China) ( 2 Institute of Developmental Psychology, Beijing Normal University, Beijing , China) Abstract: Computerized Adaptive Testing (CAT) has developed rapidly in recent years, and item replenishing is essential for item bank construction and maintenance in CAT. Calibration of new items is a technical challenge in item replenishing, and the precision of which directly impacts the accuracy of the estimation of examinees abilities. Now the online calibration technique has been commonly used to calibrate the new items. Studies on online calibration for two kinds of CATs (traditional CAT based on item response theory and cognitive diagnostic CAT (CD-CAT) based on cognitive diagnostic theory) are reviewed in the aspects of online calibration design and online calibration method. There have been relatively abundant research results for the traditional CAT, while online calibration studies of CD-CAT have just started. Future studies could further explore the comparison and combination of different online calibration designs/methods, as well as the online calibration for CD-CAT and multidimensional CAT. Key words: online calibration; computerized adaptive testing; item bank construction; cognitive diagnostic computerized adaptive testing

Microsoft Word - 3906.doc

Microsoft Word - 3906.doc 在线测试离我们有多远? 语言在线测试调研报告陈晓红, 蔡静 ( 北京外国语大学网络教育学院, 北京 100089) 摘要 : 本文是一篇关于英语在线测试的调研报告, 由三部分组成第一部分介绍计算机在语言测试中的重要作用及各种基于计算