SavedAdd your journal

Journals / International Journal of Assessment Tools in Education / 2026 / Volume: 9 Issue: 3

A Comparison of type I error and power rates in procedures used determining test dimensionality

Gül Güler(İSTANBUL AYDIN ÜNİVERSİTESİ)0000-0001-8626-4901
Nükhet Çıkrıkçı(İstanbul Aydın Üniversitesi)0000-0001-8853-4733
Pages
—
Publication Date
2022-09-30
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition.
📊
Key Findings & Data:The simulative data were generated for the study using the SAS program.
💡
Conclusion & Impact:On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.
🔬Design:Research Article
📍Scope:Comprehensive Sample
⚙️Variables:Dimensionality · Construct validity · DIMTEST T statistic

Abstract

The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.

Özet

The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.

🔖

Cite This Article

Güler, G., Çıkrıkçı, N. (2022). A Comparison of type I error and power rates in procedures used determining test dimensionality. International Journal of Assessment Tools in Education, Volume: 9 Issue: 3. https://doi.org/10.21449/ijate.1059628

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (30)

  1. Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
  2. Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
  3. Davey, T., Nering M.L., & Thompson, T. (1997). Realistic simulation of item response data. ACT Research Report Series, 97-4. https://files.eric.ed.gov/fulltext/ED414297.pdf
  4. Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
  5. Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARMbased statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307. https://doi.org/10.1177/0146621606294490
  6. Fleisman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
  7. Froelich, A.G., & Habing, B. (2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138 155. https://doi.org/10.1177/0146621607300421
  8. Gessaroli, M.E., & De Champlain, A.F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157 179. https://doi.org/10.1111/j.1745 3984.1996.tb00487.x Göçer Şahin, S. (2016). Examining parameter estimation when treating semi mixed multidimensional constructs as unidimensional [Unpublished doctoral dissertation]. Hacettepe University
  9. Hattie, J. (1985). Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(8), 139 – 145. http://dx.doi.org/10.1177/014662168500900204
  10. Hattie, J., Krakowski, K., Rogers, H.J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential dimensionality. Applied Psychological Measurement, 20, 1-14. https://doi.org/10.1177%2F014662169602000101
  11. Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 6(1), 53-60. https://doi.org/10.21427/D7CF7R
  12. Hu, L-T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
  13. Kaya, K.Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice. 16(1), 153-171. http://dx.doi.org/10.12738/estp.2016.1.0220
  14. Kim, H.R. (1994). New techniques for dimensionality assessment of standardized test data. [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, Department of Statistics.
  15. Ledasma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis. Practical Assessment, Research & Evaluation, 12 (2). https://doi.org/10.7275/wjnc-nm63
  16. Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19 (1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
  17. Nandakumar, R., & Stout, W. (1993). Refinement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41 68. https://psycnet.apa.org/doi/10.2307/1165182
  18. Özbek Baştuğ, Ö.Y. (2012). Assessment of Dimensionality in Social Science Subtest. Educational Sciences: Theory & Practice. 12(1), Winter: 382-385.
  19. Reichenberg, R.E. (2013). A comparison of DIMTEST and generalized dimensionality discrepancy approaches to assessing dimensionality in item response theory [M.S. dissertation, Arizona State University, Arizona]. https://doi.org/10.3102%2F10769986018001041
  20. Reckase, M.D. (2009). Multidimensional item response theory. Springer Dordrecht Heidelberg.
  21. Roussos, L.A., & Özbek, O.Y. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
  22. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617. https://doi.org/10.1007/BF02294821
  23. Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354. https://doi.org/10.1177%2F014662169602000403
  24. Sünbül, Ö. (2011). Çeşitli boyutluluk özelliklerine sahip yapılarda, madde parametrelerinin değişmezliğinin klasik test teorisi, tek boyutlu madde tepki kuramı ve çok boyutlu madde tepki kuramı çerçevesinde incelenmesi [Unpublished doctoral dissertation]. Mersin University.
  25. Sünbül, Ö., & Seo, M. (2012). Performance of test statistics for verifying unidimensionality, [Conference presentation abstract]. 2012 Annual Meeting, April 12-16, Vancouver, British Columbia, CANADA
  26. Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures [Unpublished doctoral dissertation]. Arizona State University
  27. Svetina, D., & Levy, R. (2014). A framework for dimensionality assessment for multidimensional item response models. Educational Assessment, 19(1), 35-57. https://doi.org/10.1080/10627197.2014.869450
  28. Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203. https://doi.org/10.1177/0146621603027003001
  29. Touron, J., Lizasoain, L., & Joaristi, L. (2012). Assessing the unidimensionality of the School and College Ability Test (SCAT, Spanish version) using non-parametric methods based on item response theory. High Ability Studies. 23(2), 183 202. https://doi.org/10.1080/13598139.2012.735401
  30. Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimension. The Journal of Experimental Education, 77 (2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166