* EduG 2014 6 181 FOREIGN LANGUAGE RESEARCH 2014 No. 6 Serial No. 181 200083 3 EduG H319. 5 A 1000-0100 2014 06-0113 - 9 A Brief Introduction to English Teaching Application of the New Generalizability Theory Software EduG Hu Jia-sheng Shanghai International Studies University Shanghai 200083 China Generalizability Theory is one of the contemporary mainstream testing theories. The theory uses the ANOVA procedure of modern statistics to conduct the measurement. It attaches importance to studying and isolating different types of variances of error and develops a complete procedure to break down various sources of those variances of error. The theory has also created a unique model called the Decision Stage. This thesis briefly introduces a new piece of software based upon the Generalizability Theory designed by Swiss Society for Educational Research and it also demonstrates its usefulness in language testing with a small exam sample. Key words Generalizability Theory software application test paper analysis foreign language teaching test 1 dependability random parallel strict parallel test test * EduG ydh53@ yahoo. com. cn. 113
2014 EduG 6 4 1 subject effect the effect of object of measurement 2 item effect 3 4 random error error confounding 3 universe of admissible observations 2 Universe Score Generalizability Coefficient G Facets of Measurement universe score level condition 20 20 Random Facet Fixed Facet 4 G G 114
2014 EduG 6 G norm-referenced test G G = / + relative decision G G criterion-referenced test G / + Decision Stage D D G = + absolute decision G D D D G G D G Classical Testing Theory Item Response Theory G G 6 object of measurement facet of measurement Generalizing Stage G 3 3 5 D 115
2014 EduG 6 7 EduG EduG EduG Workscreen S Instrumentation Facet S EduG Q Differenciation 8 Facet 7. 1 EduG EduG 2 2 Facet Label Level Universe 12 Students 1 16 12 Question 1 S Q 16 12 INF 12 16 EduG 1 16 12 16 16 12 1 2 3 4 5 6 7 8 9 10 11 12 12 1 1 1 1 1 1 0 1 0 0 1 0 0 INF INF 2 1 1 1 1 1 0 0 0 1 0 1 1 3 1 0 0 0 0 0 0 0 0 0 0 0 4 1 1 0 0 0 1 0 1 0 0 0 0 5 1 1 1 1 1 0 1 1 0 1 1 1 6 1 1 1 1 1 1 1 1 1 1 1 0 7 1 0 1 0 0 0 0 0 0 0 0 0 8 1 1 1 0 0 1 0 0 0 0 0 0 9 1 1 1 1 0 0 0 0 0 0 0 0 7. 2 EduG 10 1 1 0 1 1 0 0 1 0 0 0 0 EduG File 11 1 1 1 1 1 0 1 0 0 0 0 0 New 12 1 1 1 0 1 1 1 0 0 0 0 0 EduG File to be Created 13 1 1 1 1 1 1 1 0 0 0 0 0 N 14 1 1 1 1 0 1 1 1 1 1 0 0 Example One 15 1 1 1 1 1 1 1 1 1 1 1 0 16 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 12 12 1 EduG 116
2014 EduG 6 1-12 Title Ex- 16 12 ample One. Title Number of facets Data 2 Data 2 Number of facets Observation and estimation designs 12 6 2 Observation and estimation designs 16 16 Q 16 5 Keying scores 2 Observation and estimation designs Insert data 3 3 The basis does not include any data Do you want to add... Q /S Measurement design Scores Measurement design Report Sums of Squares Text Format RTF Format Word RTF Scores Keying scores Format 4 4 Standard Deviation St. D SDVariance V Var. Standard Error of Mean SEM 5 Estimate Observation and estimation designs S Optimization D Q G-facets analysis G S 12 1 12 2 12 3 12 5 Save 5 period Compute 5 ANOVA Analysis of Variance 4 Coef. G G Coef. G relative. G Coef. G abs. G of Phi Lambda Phi Lambda ANOVA Co- 117
2014 EduG 6 ef. G SE standard error G Mean SQ 51. 5% Edit report Save Save as... 3 0. 6 028 0. 026 0. 015 6 7. 3 Compute ANOVA Coef. - G ANOVA G-coefficient G Variance 3 Analysis of 3 Components SQ 63. 3% Source SS sum of square df degree of freedom Relative SE MS mean square Random 0. 117 0. 093 Mixed SQ 0. 009 Corrected % 3 SQ 0. 005 0. 009 0. 014 Total 100% 118 4 G Q Source of variance Sum of variances Standard deviation Differentiation variance Q Differentiation variance 0. 053 true score Standard deviation 0. 229 0. 229 0. 053 S SQ Source of variance Relative error variance 4 G % relative SQ 0. 009 100% 100% Absolute error variance S SQ 0. 005 0. 009 % Absolute Absolute SE 0. 093 0. 117 S
2014 EduG 6 Standard deviation 0. 093 0. 117 0. 229 5 16 12 Coef. G relative 0. 083 G Coef G absolute G 1. 083 0. 276 0. 500 G 0. 86 G 0. 79 6 0. 8 G 0. 8 G 0. 01 Means S Q Means 7 7 Means Compute Grand mean 0. 563 Variance 0. 257 Standard Dev. 0. 506 3 5 3 5 Students S Mean Variance Std. Dev. 1 0. 583 0. 243 0. 493 2 0. 667 0. 222 0. 471 3 0. 083 0. 076 0. 276 4 0. 333 0. 222 0. 471 5 0. 833 0. 139 0. 373 6 0. 917 0. 076 0. 276 7 0. 167 0. 139 0. 373 8 0. 333 0. 222 0. 471 9 0. 333 0. 222 0. 471 10 0. 417 0. 243 0. 493 11 0. 500 0. 250 0. 500 12 0. 500 0. 250 0. 500 13 0. 583 0. 243 0. 493 14 0. 750 0. 188 0. 433 15 0. 917 0. 076 0. 276 16 1. 083 0. 076 0. 276 1 0 0. 875 86% 19% D D Coef G relative Coef G absolute Compute Optimization D Optimization 8 8 Observation and estimation designs Opt 1... Opt 5 1 5 5 5 copy Observation and estimation designs 5 119
2014 EduG 6 16 Opt 1 Obs. 16 17Opt 2 Obs. 16 18Opt 3 Obs. 16 19 Opt 4 Obs. 16 20 Opt 5 Obs. 16 21 copy 12 5 Opt Optimization 9 9 17 16 Optimization 10 OK Optimization D 7 7 7 1 Option 1 16 17 Observ. 192 204 192 + 12 Coef-G rel. 0. 858 0. 865 rounded Coe-f abs. rounded 0. 80 Rel. Err. Var. Rel. Std. Err. of M. Abs. Err. Var. Abs. Std. Err. of M. Option 2 Option 5 216 252 216 + 12 + 12 + 12 Coef-G rel. rounded 0. 87 0. 89 Coef abs. rounded 0. 81 0. 83 Rel. Err. Var. Rel. Std. Err. of M. Abs. Err. Var. Abs. Std. Err. of M. 4 Option 3 16 Option 4 17 0. 87 0. 81 Optimization 1 Univ. INF. 12 13Opt 2 Univ. INF. 12 14Opt 3 Univ. INF. 12 15Opt 4 Univ. INF. 12 16Opt 5 Univ. INF. 12 7 rounded 0. 87 18 13 14 Option 2 15 Option 3 16 Option 4 120 10 OK Optimization D 8 8 8 Option 1 12 13 Coef-G rel. 0. 858 0. 868 rounded 0. 87Coe-G abs. 0. 792 0. 807 rounded 0. 81 13 14 Option 2 15 Option 5 Coef-G rel. 0. 868
2014 EduG 6 17 Option 5 Coe-G abs. rounded 0. 81 Option 1 Option 5 Rel. Err. Var. Rel. Std. Err. of M. Abs.. M. 1986. Err. Var. Abs. Std. Err. of M. 4. J. 12 2013 2. 4. Z. 2008. 8. M. 2002. 13 Bachman L. F. Fundamental Considerations in Language 0. 87 Testing M. Oxford Oxford University Press 1997. 0. 81 Baker F. Methodology Review of Item Parameter Estimation J. Applied Psychological Measurement 1987 11. 4 Beglar D. Estimating Grammatical Competence testing and Evaluation J. SIG Newsletter 2004 1. Davies A. & A. Brown. Dictionary of Language Testing M. Cambridge Cambridge University Press 2001 4 Henning G. A Guide to Language Testing Development Evaluation and Research M. Beijing Foreign Language Teaching and Research Press 2003 Linda C. & A. James. Introduction to Classical and Modern Test Theory M. New York Holt Rinchart and Winston 1986. Lyons L. Linguistic Semantics An Introduction M. Cambridge Cambridge University Press 1995. 8 McNamara T. Language Testing M. Oxford Oxford University Press 2001. Meara P. & G. Jones. Tests of Grammar in English as a Foreign Language J. Polyglot 1989 2. Nation I. S. P. Evaluating Reading Ability in Another language A. In T. D. Brown ed.. English Language Institute Occasional Publication C. Wellington Victoria University of Wellington Press 2005. 2013 02 EduG Thormas F. A Short Survey of Generalizability Theory M. Poutsmouth Poutsmouth NH Heinemann Press 1999. 2013-12 - 29 121