2016 3 Foreign Language Testing and Teaching Jul 2016 H319 A 2095-1167 2016 03-0020 - 12 * CFA 1 Ute Knoch 2 3 Abstract Understanding the factor structure of a language test is crucial to the establishment of its construct validity. Despite the growing number of factor structure studies in the field of language testing few have used multi-year data to investigate the factor structure and factorial invariance of language tests. Adopting longitudinal design and multisample Confirmatory Factor Analysis CFA as its analytic methodology this study investigated the factor structure and factorial invariance of a university-based English proficiency test. Results indicated that the higher-order factor model fit the test data well and was comparatively parsimonious. Multi-sample CFA analyses demonstrated that this model had basically the same configuration on the two samples. While lending important empirical support to the construct validity of this test this study also has implications for other testing agencies in their test validation endeavors. Key words factor structure factorial invariance Confirmatory Factor Analysis multi-sample analysis 1. test validity argument AERA et al. 2014 Chapelle et al. 2008 AERA et al. 2014 Gu 2014 Sawaki et al. 2009 In nami & Koizumi 2012 TEAP In nami et al. 2016 EPE Sims & Kunnan 2016 * 13CYY032 20
CFA Alderson & Banerjee 2002 Confirmatory Factor Analysis CFA cross-sectional design longitudinal design AERA et al. 2014 2014 2009 2001 2007 2007 CFA 2. 2. 1 70 Oller 1976 Principal Component Analysis PCA Oller Oller Oller PCA Bachman & Palmer 1982 CFA 116 Sawaki et al. 2009 CFA 2720 Sawaki Llosa 2007 Sawaki 2007 2014 Gu 2014 CFA 1000 In nami & Koizumi 2012 569 Bachman & Palmer 1981 Sang et al. 1986 21
2016 3 CFA Fudan English Test FET 2. 2 FET FET FET Bachman & Palmer 1996 2010 FET 1 FET 2 FET 3 FET FET 2013 FET FET FET FET FET 1 2014 CFA 2 Fan & Ji 2014 FET 3 Fan et al. 2014 FET FET 2014 2015 FET FET 3. 3. 1 1 FET 2 CFA FET 3. 2 2014 2015 FET 7157 3389 47. 4% 3768 52. 6% 2014 3988 22
CFA 1847 46. 3% 2141 53. 7% 2015 3169 1542 48. 7% 1627 51. 3% 3. 3 2014 2015 FET 4 L1 * L2 2 L3 L4 5 3 R1 - R3 2 R4 - R5 2 W1 W2 3 S1 S2 S3 CFA 3. 4 Bachman & Palmer 1996 2010 Gu 2014 In nmai & Koizumi 2012 Llosa 2007 Sawaki et al. 2009 a 1 FET FET 1 b 2 FET FET * CFA 23
2016 3 2 c A 3 FET + + FET 3 A d B 4 FET + + FET 24
CFA 3. 5 4 B 1 2 3 CFA EQS 6. 3 Bentler & Wu 2005 1 2 3 Byrne 2006 Sawaki et al. 2009 Comparative Fit Index CFI Non-Normed Fit Index NNFI Goodness of Fit Index GFI Standardized Root Mean Square Residual SRMR Root Mean Square Error of Approximation RMSEA Byrne 2006 In nami & Koizumi 2011 Byrne 2006 CFI NNFI GFI 0. 95 SRMR 0. 05 RMSEA 0. 08 RMSEA 90% In nami & Koizumi 2011 nested models Byrne 2006 4. 4. 1 2. 13 2014 25
2016 3 S2 ± 2 Mardia 21. 38 23. 59 5 Bentler & Wu 2005 EQS robust estimation method 4. 2 1 2 1 2014 CFI = 0. 959 GFI = 0. 972 SRMR = 0. 030 RMSEA = 0. 049 1 S - B x 2 = 483. 83 df = 5 p < 0. 05 1 S - B x 2 = 558. 4 df = 5 p < 0. 05 1 CFI = 0. 954 GFI = 0. 969 SRMR = 0. 035 RMSEA = 0. 051 S - B x 2 = 89. 43 df = 2 p < 0. 05 Cheung & Rensvold 2002 CFI CFI < 0. 01 Sawaki et al. 2009 CFI 0. 005 FET 1 2014 N = 3988 S - B x 2 df CFI NNFI GFI SRMR 739. 37 * 71 0. 959 0. 950 0. 972 0. 030 826. 67 * 73 0. 954 0. 945 0. 969 0. 035 RMSEA 90% C. I. 0. 049 0. 045-0. 052 0. 051 0. 048-0. 054 26
CFA 1440. 36 * 76 0. 916 0. 904 0. 946 0. 051 1328. 22 * 76 0. 923 0. 908 0. 951 0. 046 0. 067 0. 064-0. 070 0. 064 0. 061-0. 067 1 * p < 0. 05 2 A B 3 S - B x 2 EQS Satorra-Bentler C. I. Confidence Interval RMSEA 90% 2 2015 N = 3169 S - B x 2 df CFI NNFI GFI SRMR 583. 05 * 71 0. 963 0. 956 0. 972 0. 031 650. 96 * 73 0. 959 0. 952 0. 969 0. 035 1256. 34 * 76 0. 916 0. 906 0. 941 0. 059 999. 42 * 76 0. 934 0. 921 0. 953 0. 044 RMSEA 90% C. I. 0. 048 0. 044-0. 051 0. 051 0. 048-0. 055 0. 070 0. 067-0. 073 0. 062 0. 059-0. 065 1 2015 2 2015 2014 S - B x 2 = 66. 99 df = 2 p < 0. 05 CFI CFI = 0. 004 FET 4. 3 CFA FET CFA Byrne 2006 metric invariance factorial invariance CFA configural model A B C 3 27
2016 3 3 CFA S - B x 2 df CFI NNFI GFI SRMR Model A 1475. 52 * 146 0. 956 0. 948 0. 969 0. 035 Model B 1666. 66 * 156 0. 950 0. 945 0. 965 0. 045 Model C 1679. 09 * 159 0. 950 0. 946 0. 965 0. 046 RMSEA 90% C. I. 0. 050 0. 048-0. 053 0. 052 0. 050-0. 054 0. 052 0. 049-0. 054 1 p < 0. 05 2 Model A Model B Model C 3 CFI = 0. 956 GFI = 0. 969 SRMR = 0. 035 RMSEA = 0. 050 Byrne 2006 3 CFI = 0. 950 GFI = 0. 965 SRMR = 0. 045 RMSEA = 0. 052 S - B x 2 = 189. 34 df = 10 p < 0. 05 CFI RMSEA Sawaki et al. 2009 3 CFI = 0. 950 GFI = 0. 965 SRMR = 0. 046 RMSEA = 0. 052 S - B x 2 = 11. 69 df = 3 p < 0. 05 CFI RMSEA 5. CFA FET A 2013 FET Exploratory Factor Analysis EFA + + EFA CFA EFA EFA 28
CFA CFA Byrne 2006 EFA CFA CFA In nami & Koizumi 2011 B FET CFA 2014 2015 Llosa 2007 Sawaki 2007 Sawaki et al. 2009 FET Bachman & Palmer 1996 2010 FET Sawaki et al. 2009 * 0. 46-0. 91 2014 0. 49-0. 92 2015 0. 63-0. 93 2014 0. 66-0. 98 2015 Sims & Kunnan 2016 CFA AERA et al. 2014 FET Sawaki et al. 2009 In nami et al. 2016 Sim & Kunnan 2016 CFA Byrne 2006 Chapelle et al. 2008 * 29
2016 3 1 AERA APA & NCME. AERA 2014. Standards for Educational and Psychological Testing Z. Washington D.C. 2 Alderson J C & Banerjee J. Language testing and assessment Part 2 J. Language Teaching 2002 35 2 79-113. 3 Bachman L F & Palmer A S. The construct validation of the FSI Oral Interview 1 J. Language Learning 1981 31 1 67-86. 4 Bachman L F & Palmer A S. The construct validation of some components of communicative proficiency J. TESOL Quarterly 1982 16 4 449-465. 5 Bachman L F & Palmer A S. Language Assessment in Practice Designing and Developing Useful Language Tests M. Oxford Oxford University Press 1996. 6 Bachman L F & Palmer A S. Language Assessment in Practice Developing Language Assessments and Justifying their Use in the Real World M. Oxford Oxford University Press 2010. 7 Bentler P M & Wu E J. EQS 6. 1 for Windows M. Encino CA Multivariate Software 2005. 8 Byrne B M. Structural Equation Modeling with EQS Basic Concepts Applications and Programming 2nd Ed. M. Mahwah New Jersey Psychology Press 2006. 9 Chapelle C A Enright M K & Jamieson J M. Building a Validity Argument for the Test of English as a Foreign Language C. New York and London Routledge Taylor & Francis Group 2008. 10 Cheung G W & Rensvold R B. Evaluating goodness of fit indexes for testing measurement invariance J. Structural Equation Modeling 2002 9 233-255. 11 Fan J & Ji P. Test candidates attitudes and their test performance The case of the Fudan English Test J. University of Sydney Papers in TESOL 2014 9 1-35. 12 Fan J Ji P & Song X. Washback of university-based English language tests on students learning A case study J. The Asian Journal of Applied Linguistics 2014 1 2 178-192. 13 Gu L. At the interface between language testing and second language acquisition Language ability and context of learning J. Language Testing 2014 31 1 111-133. 14 In nami Y & Koizumi R. Structural equation modeling in language testing and learning research A review J. Language Assessment Quarterly 2011 8 3 250-276. 15 In nami Y & Koizumi R. Factor structure of the revise TOEFL test A multi-sample analysis J. Language Testing 2012 29 1 131-152. 16 In nami Y Koizumi R & Nakamura K. Factor structure of the Test of English for Academic Purposes TEAP test in relation to the TOEFL ibt P test J. Language Testing in Asia 2016 6 3 1-23. 17 Llosa L. Validating a standards-based classroom assessment of English proficiency A multitrait-multimethod approach J. Language Testing 2007 24 4 489-515. 18 Oller J W. Evidence for a general language proficiency factor An expectancy grammar J. Die Neueren Sprachen 1976 75 2 165-174. 19 Sang F Schmitz B Vollmer H J Baumert J & Roeder P. Models of second language competence A structural equation approach J. Language Testing 1986 3 1 54-79. 20 Sawaki Y. Construct validation of analytic rating scale in speaking assessment Reporting a score profile and a composite J. Language Testing 2007 24 3 355-390. 21 Sawaki Y Stricker L J & Oranje A H. Factor structure of the TOEFL Internet-based test J. Language Testing 2009 26 1 5-30. 37 30
4 5 1 2 3 4 lexical approach 1 Bachman L F & Palmer A S. Language Testing in Practice M. Oxford Oxford University Press 1996. 2 Taylor W L. Cloze procedure A new tool for measuring readability J. Journalism Quarterly 1953 30 415-453. 3. CET J. 2003 11 3 39-42. 4. M. 1997. 5. M. 2002. 510631 櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟櫟 30 22 Sims J M & Kunnan A J. Developing evidence for a validity argument for an English placement exam from multi-year test performance data J. Language Testing in Asia 2016 6 1 1-14. 23. J. 2013 2 45-53. 24. J. 2015 3 85-94. 25. J. 2014 34-40. 26. J. 2001 11 16-18. 27. J. 2007 2 89-96. 28. TEM J. 2009 8 287-295. 29. J. 2007 5 35-39. 1. 200433 2. 3. 200090 37