SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2026 / Volume: 8 Issue: 1

Comparison of confirmatory factor analysis estimation methods on mixed-format data

Abdullah Faruk Kılıç(Adıyaman Üniversitesi)0000-0003-3129-1763
Nuri Doğan(HACETTEPE ÜNİVERSİTESİ, EĞİTİM FAKÜLTESİ)0000-0001-6274-2016
Pages
—
Publication Date
2021-03-15
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Weighted least squares (WLS), weighted least squares mean-and-variance-adjusted (WLSMV), unweighted least squares mean-and-variance-adjusted (ULSMV), maximum likelihood (ML), robust maximum likelihood (MLR) and Bayesian estimation methods were compared in mixed item response type data via Monte Carlo simulation.
📊
Key Findings & Data:The percentage of polytomous items, distribution of polytomous items, categories of polytomous items, average factor loading, sample size and test length conditions were manipulated.
💡
Conclusion & Impact:No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.
🔬Design:Research Article
📍Scope:Comprehensive Sample
⚙️Variables:Bayesian estimation · Monte-carlo simulations · CFA

Abstract

Weighted least squares (WLS), weighted least squares mean-and-variance-adjusted (WLSMV), unweighted least squares mean-and-variance-adjusted (ULSMV), maximum likelihood (ML), robust maximum likelihood (MLR) and Bayesian estimation methods were compared in mixed item response type data via Monte Carlo simulation. The percentage of polytomous items, distribution of polytomous items, categories of polytomous items, average factor loading, sample size and test length conditions were manipulated. ULSMV and WLSMV were found to be the more accurate methods under all simulation conditions. All methods except WLS had acceptable relative bias and relative standard error bias. No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.

Özet

Weighted least squares (WLS), weighted least squares mean-and-variance-adjusted (WLSMV), unweighted least squares mean-and-variance-adjusted (ULSMV), maximum likelihood (ML), robust maximum likelihood (MLR) and Bayesian estimation methods were compared in mixed item response type data via Monte Carlo simulation. The percentage of polytomous items, distribution of polytomous items, categories of polytomous items, average factor loading, sample size and test length conditions were manipulated. ULSMV and WLSMV were found to be the more accurate methods under all simulation conditions. All methods except WLS had acceptable relative bias and relative standard error bias. No method gives accurate results with small sample sizes and low factor loading, however, the ULSMV method can be recommended to researchers because it gives more appropriate results in all conditions.

🔖

Cite This Article

Kılıç, A. F., Doğan, N. (2021). Comparison of confirmatory factor analysis estimation methods on mixed-format data. International Journal of Assessment Tools in Education, Volume: 8 Issue: 1. https://doi.org/10.21449/ijate.782351

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (49)

  1. AERA, APA, NCME, American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement In Education (NCME). (2014). Standards for educational and psychological testing. American Educational Research Association.
  2. Babakus, E., Ferguson, C. E., & Jöreskog, K. G. (1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24(2), 222 228. https://doi.org/10.2307/3151512
  3. Bandalos, D. L. (2014). Relative performance of categorical diagonally weighted least squares and robust maximum likelihood estimation. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 102 116. https://doi.org/10.1080/10705511.2014.859510
  4. Bandalos, D. L., & Leite, W. (2013). Use of Monte Carlo studies in structural equation modeling research. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 625-666). Information Age.
  5. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186 203. https://doi.org/10.1207/s15328007sem1302_2
  6. Bollen, K. A. (1989). Structural equations with latent variables. John Wiley & Sons, Inc. https://doi.org/10.1002/9781118619179
  7. Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). Routledge.
  8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
  9. Depaoli, S., & Scott, S. (2015). Frequentist and bayesian estimation of CFA measurement models with mixed item response types: A monte carlo investigation. Structural Equation Modeling: A Multidisciplinary Journal, (September), 1 16. https://doi.org/10.1080/10705511.2015.1044653 (Retraction published 2015, Structural Equation Modeling: A Multidisciplinary Journal, 318)
  10. DiStefano, C. (2002). The impact of categorization with confirmatory cactor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 9(3), 327 346. https://doi.org/10.1207/S15328007SEM0903_2
  11. DiStefano, C., & Hess, B. (2005). Using confirmatory factor analysis for construct validation: An empirical review. Journal of Psychoeducational Assessment, 23(3), 225–241. https://doi.org/10.1177/073428290502300303
  12. Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309–326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x
  13. Ferguson, E., & Rigdon, E. (1991). The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data. Journal of Marketing Research, 28(4), 491–497. https://doi.org/10.2307/3172790
  14. Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Information Age.
  15. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
  16. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625 641. https://doi.org/10.1080/10705510903203573
  17. Green, S. B., Akey, T. M., Fleming, K. K., Hershberger, S. L., & Marquis, J. G. (1997). Effect of the number of scale points on chi‐square fit indices in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 4(2), 108–120. https://doi.org/10.1080/10705519709540064
  18. Guilford, J. P. (1946). New standards for test evaluation. Educational and Psychological Measurement, 6(4), 427–438. https://doi.org/10.1177/001316444600600401
  19. Hallquist, M., & Wiley, J. (2017). MplusAutomation: Automating Mplus model estimation and interpretation. Retrieved from https://cran.r-project.org/package=MplusAutomation
  20. Holtmann, J., Koch, T., Lochner, K., & Eid, M. (2016). A comparison of ML, WLSMV, and bayesian methods for multilevel structural equation models in small samples: A simulation study. Multivariate Behavioral Research, 51(5), 661 680. https://doi.org/10.1080/00273171.2016.1208074
  21. Jin, S., Luo, H., & Yang-Wallentin, F. (2016). A simulation study of polychoric instrumental variable estimation in structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 23(5), 680 694. https://doi.org/10.1080/10705511.2016.1189334
  22. Lee, T. K., Wickrama, K., & O’Neal, C. W. (2018). Application of latent growth curve analysis with categorical responses in social behavioral research. Structural Equation Modeling: A Multidisciplinary Journal, 25(2), 294 306. https://doi.org/10.1080/10705511.2017.1375858
  23. Lee, S.-Y., & Song, X.-Y. (2004). Evaluation of the bayesian and maximum likelihood approaches in analyzing structural equation models with small sample sizes. Multivariate Behavioral Research, 39(4), 653–686. https://doi.org/10.1207/s15327906mbr3904_4
  24. Lei, P. (2009). Evaluating estimation methods for ordinal data in structural equation modeling. Quality and Quantity, 43(3), 495–507. https://doi.org/10.1007/s11135-007-9133-z
  25. Li, C.-H. (2014). The performance of MLR, USLMV, and WLSMV estimation in structural regression models with ordinal variables [Unpublished Doctoral dissertation]. Michigan State University.
  26. Li, C.-H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7
  27. Liang, X., & Yang, Y. (2014). An evaluation of WLSMV and Bayesian methods for confirmatory factor analysis with categorical indicators. International Journal of Quantitative Research in Education, 2(1), 17 38. https://doi.org/10.1504/IJQRE.2014.060972
  28. Liang, X., & Yang, Y. (2016). Confirmatory factor analysis under violations of distributional and structural assumptions: A comparison of robust maximum likelihood and bayesian estimation methods. Journal of Psychological Science, 39(5), 1256–1267. https://doi.org/10.1504/IJQRE.2013.055642
  29. Lorenzo-Seva, U., & Ferrando, P. J. (2020). Factor (Version 10.10.03) [Computer software]. Universitat Rovira i Virgili.
  30. MoNE. (2017). Monitoring and evaluation of academic skills report for eight graders. MONE. https://odsgm.meb.gov.tr/meb_iys_dosyalar/2017_11/30114819_iY-web-v6.pdf
  31. Morata-Ramirez, M. de los A., & Holgado-Tello, F. P. (2013). Construct validity of likert scales through confirmatory factor analysis: A simulation study comparing different methods of estimation based on Pearson and polychoric correlations. International Journal of Social Science Studies, 1(1), 54-61. https://doi.org/10.11114/ijsss.v1i1.27
  32. Moshagen, M., & Musch, J. (2014). Sample size requirements of the robust weighted least squares estimator. Methodology, 10(2), 60 70. https://doi.org/10.1027/1614-2241/a000068
  33. Muthén, B. O., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189. https://doi.org/10.1111/j.2044-8317.1985.tb00832.x
  34. Muthén, B. O., & Kaplan, D. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45(1), 19–30. https://doi.org/10.1111/j.2044-8317.1992.tb00975.x
  35. Muthén, L. K., & Muthén, B. O. (2012). Mplus statistical modeling software: Release 7.0 [Computer software]. Muthén & Muthén.
  36. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd. ed.). McGraw-Hill.
  37. Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling: A Multidisciplinary Journal, 7(4), 557–595. https://doi.org/10.1207/S15328007SEM0704_3
  38. Oranje, A. (2003, April 21-25). Comparison of estimation methods in factor analysis with categorized variables: Applications to NEAP data [Paper presentation]. Annual Meeting of the National Council on Measurement in Education. Chicago, IL, USA.
  39. Osborne, J. W., & Banjanovic, E. S. (2016). Exploratory factor analysis with SAS®. SAS Intitute Inc.
  40. Potthast, M. J. (1993). Confirmatory factor analysis of ordered categorical variables with large models. British Journal of Mathematical and Statistical Psychology, 46(2), 273–286. https://doi.org/10.1111/j.2044-8317.1993.tb01016.x
  41. R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.r-project.org/.
  42. Revelle, W. (2019). psych: Procedures for psychological, psychometric, and personality research. https://cran.r-project.org/package=psych
  43. Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
  44. Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66(2), 201–223. https://doi.org/10.1111/j.2044-8317.2012.02049.x
  45. Shi, D., DiStefano, C., McDaniel, H. L., & Jiang, Z. (2018). Examining chi-square test statistics under conditions of large model size and ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 924 945. https://doi.org/10.1080/10705511.2018.1449653
  46. Thompson, B., & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: A historical overview and some guidelines. Educational and Psychological Measurement, 56(2), 197–208. https://doi.org/10.1177/0013164496056002001
  47. Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. National Institutes of Health, 76(6), 913–934. https://doi.org/10.1177/0013164413495237
  48. Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling: A Multidisciplinary Journal, 17(3), 392–423. https://doi.org/10.1080/10705511.2010.489003
  49. Zhao, Y. (2015). The performance of model fit measures by robust weighted least squares estimators in confirmatory factor analysis [Doctoral dissertation, The Pennsylvania State University]. https://etda.libraries.psu.edu/catalog/24901