SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2026 / Volume: 9 Issue: 3

Which scale short form development method is better? A Comparison of ACO, TS, and SCOFA

Hakan Koğar(AKDENİZ ÜNİVERSİTESİ)0000-0001-5749-9824
Pages
—
Publication Date
2022-09-30
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The purpose of this study is to identify which scale short-form development method produces better findings in different factor structures.
📊
Key Findings & Data:According to the findings of this study, the increase in the level of misspecification and the decrease in the sample size can significantly affect the model-data fit.
💡
Conclusion & Impact:For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:short form · scale development · ant colony optimization

Abstract

The purpose of this study is to identify which scale short-form development method produces better findings in different factor structures. A simulation study was designed based on this purpose. Three different factor structures and three simulation conditions were selected. As the findings of this simulation study, the model-data fit and reliability coefficients were reported for each factor structure in each simulation condition. All analyses were conducted under the R environment. According to the findings of this study, the increase in the level of misspecification and the decrease in the sample size can significantly affect the model-data fit. In a situation where the factor structure of the scale is getting more and more complex, model-data fit and Omega coefficients decrease. For scales with a unidimensional factor structure, all of the scale short-form development methods are recommended. For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended. When viewed from the framework of metaheuristic algorithms, it has been identified that ACO produces better findings than Tabu Search.

Özet

The purpose of this study is to identify which scale short-form development method produces better findings in different factor structures. A simulation study was designed based on this purpose. Three different factor structures and three simulation conditions were selected. As the findings of this simulation study, the model-data fit and reliability coefficients were reported for each factor structure in each simulation condition. All analyses were conducted under the R environment. According to the findings of this study, the increase in the level of misspecification and the decrease in the sample size can significantly affect the model-data fit. In a situation where the factor structure of the scale is getting more and more complex, model-data fit and Omega coefficients decrease. For scales with a unidimensional factor structure, all of the scale short-form development methods are recommended. For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended. When viewed from the framework of metaheuristic algorithms, it has been identified that ACO produces better findings than Tabu Search.

🔖

Cite This Article

Koğar, H. (2022). Which scale short form development method is better? A Comparison of ACO, TS, and SCOFA. International Journal of Assessment Tools in Education, Volume: 9 Issue: 3. https://doi.org/10.21449/ijate.946231

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (28)

  1. Anastasi, A. (1982). Psychological Testing (5th ed.). Macmillan.
  2. Batley, R.M., & Boss, M.W. (1993). The effects on parameter estimation of correlated dimensions and a distribution-restricted trait in a multidimensional item response model. Applied Psychological Measurement, 17(2), 131 141. https://doi.org/10.1177/014662169301700203
  3. Cayanus, J.L., & Martin, M.M. (2004). An instructor self‐disclosure scale. Communication Research Reports, 21(3), 252-263. https://doi.org/10.1080/08824090409359987
  4. Colorni, A., Dorigo, M., & Maniezzo, V. (1991). Distributed optimization by ant colonies. In: Varela, F. and Bourgine, P., Eds., Proceedings of the European Conference on Artificial Life, ECAL’91, Paris, Elsevier Publishing, Amsterdam, 134-142.
  5. Ebesutani, C., McLeish, A.C., Luberto, C.M., Young, J., & Maack, D.J. (2014). A bifactor model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of Psychopathology and Behavioral Assessment, 36(3), 452 464. https://doi.org/10.1007/s10862-013-9400-3
  6. French, B.F., & Finch, W.H. (2011). Model misspecification and invariance testing using confirmatory factor analytic procedures. The Journal of Experimental Education, 79(4), 404-428. https://doi.org/10.1080/00220973.2010.517811
  7. Gatignon, H. (2010). Confirmatory Factor Analysis. In Statistical Analysis of Management Data (pp. 59-122). Springer. https://doi.org/10.1007/978-1-4419-1270-1_4
  8. Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
  9. Janssen, A.B., Schultze, M., & Grötsch, A. (2017). Following the ants: Development of short scales for proactive personality and supervisor support by Ant Colony Optimization. European Journal of Psychological Assessment, 33(6), 409. https://doi.org/10.1027/1015-5759/a000299
  10. Jiang, S., Wang, C., & Weiss, D.J. (2016). Sample size requirements for estimation of item parameters in the multidimensional graded response model. Frontiers in Psychology, 7(Article:109), 1-10. https://doi.org/10.3389/fpsyg.2016.00109
  11. Jorgensen, T.D., Pornprasertmanit, S., Schoemann, A. M., Rosseel, Y., Miller, P., Quick, C.,., & Enders, C. (2016). semTools: Useful Tools for Structural Equation Modeling. R package version 0.5 4. Retrieved from https://cran.r project.org/web/packages/semTools/index.html
  12. Kleka, P., & Soroko, E. (2018). How to avoid the sins of questionnaires abridgement?. Survey Research Methods, 12(2), 147-160. https://doi.org/10.31234/osf.io/8jg9u
  13. Kruyen, P.M., Emons, W.H., & Sijtsma, K. (2013). On the shortcomings of shortened tests: A literature review. International Journal of Testing, 13(3), 223 248. https://doi.org/10.1080/15305058.2012.703734
  14. LaNoue, M., Harvey, A., Mautner, D., Ku, B., & Scott, K. (2015). Confirmatory factor analysis and invariance testing between Blacks and Whites of the Multidimensional Health Locus of Control scale. Health Psychology Open, 2(2), 1 16. https://doi.org/10.1177/2055102915615045
  15. Leite, W.L., Huang, I.-C., & Marcoulides, G. A. (2008). Item selection for the development of short forms of scales using an Ant Colony Optimization Algorithm. Multivariate Behavioral Research, 43, 411–431. https://doi.org/10.1080/00273170802285743
  16. Marcoulides, K.M., & Falk, C. (2018). Model specification searches in structural equation modeling with R. Structural Equation Modeling, 25(3), 484 491. https://doi.org/10.1080/10705511.2017.1409074
  17. Nunnally, J.C. (1978). Psychometric Theory (2nd ed.). McGraw-Hill.
  18. Olaru, G., Witthöft, M., & Wilhelm, O. (2015). Methods matter: Testing competing models for designing short-scale big-five assessments. Journal of Research in Personality, 59, 56-68. https://doi.org/10.1016/j.jrp.2015.09.001
  19. Raborn, A.W., & Leite, W.L. (2018). ShortForm: An R package to select scale short forms with the ant colony optimization algorithm. Applied psychological measurement, 42(6), 516. https://doi.org/10.1177/0146621617752993
  20. Raborn, A.W., Leite, W.L., & Marcoulides, K.M. (2020). A comparison of metaheuristic optimization algorithms for scale short-form development. Educational and Psychological Measurement, 80(5), 910 931. https://doi.org/10.1177/0013164420906600
  21. Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behav. Res. 47, 667–696. https://doi.org/10.1080/00273171.2012.715555
  22. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5 12 (BETA). Journal of Statistical Software, 48(2), 1 36. https://doi.org/10.18637/jss.v048.i02
  23. Schroeders, U., Wilhelm, O., & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant colony optimization and genetic algorithm. PLoS One, 11(11), 1-19. https://doi.org/10.1371/journal.pone.0167110
  24. Singh, K., Junnarkar, M., & Kaur, J. (2016). Measures of Positive Psychology: Development and Validation. Springer.
  25. Van Abswoude, A.A., van der Ark, L.A., & Sijtsma, K. (2004b). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24. https://doi.org/10.1177/0146621603259277
  26. Van Abswoude, A.A., Vermunt, J.K., Hemker, B.T., & van der Ark, L.A. (2004a). Mokken scale analysis using hierarchical clustering procedures. Applied Psychological Measurement, 28(5), 332-354. https://doi.org/10.1177/0146621604265510
  27. Yang, Y., & Liang, X. (2013). Confirmatory factor analysis under violations of distributional and structural assumptions. International Journal of Quantitative Research in Education, 1(1), 61-84. https://doi.org/10.1504/ijqre.2013.055642
  28. Jorgensen, T.D., Pornprasertmanit, S., Schoemann, A. M., Rosseel, Y., Miller, P., Quick, C.,., & Enders, C. (2016). semTools: Useful Tools for Structural Equation Modeling. R package version 0.5 4. Retrieved from https://cran.r project.org/web/packages/semTools/index.html