Can TIMSS Mathematics Assessments be Implemented as Computerized Adaptive Test?

Abstract

In recent years, there has been a growing interest and extensive use of computerized adaptive testing (CAT)especially in large-scale assessments. Numerous simulation studies have been conducted on both real andsimulated data sets to determine the optimum conditions and develop CAT versions. Being one of the mostpopular large-scale assessment programs, Trends in International Mathematics and Science Study (TIMSS) hasbeen implemented as paper and pencil tests to monitor student achievement in mathematics and science at fourthand eighth grade levels since 1995. The purpose of this study is to investigate the optimum CAT algorithm forTIMSS eighth grade mathematics assessments. Since Turkey and USA participated in 2007, 2011 and 2015administrations, their data were combined and then 393 items were calibrated on the same scale by usingmarginal maximum likelihood estimation method. With this item pool, several scenarios were proposed andtested to determine not only the optimum starting rule, ability estimation method, test termination rule but alsothe efficiency of exposure control method. The results of the study indicated that estimating abilities withexpected a posteriori method after 6 random items, terminating the fixed-length test after 20 items seemed to bethe optimum algorithm for TIMSS eighth grade mathematics assessments. Also, it was found that using itemexposure control had a prior importance for the effective use of the item pool. This study has some implicationsfor both national and international large-scale test developers in determining the optimum CAT algorithm andits consequences compared with paper and pencil versions.