A Study on Detecting Differential Item Functioning of PISA 2006 Science Literacy Items in Turkish and American Samples

Özet

Problem Statement: Item bias occurs when individuals from differentgroups (different genders, cultural backgrounds, etc.) have differentprobabilities of responding correctly to a test item despite having the sameskill levels. It is important that tests or items do not have bias in order toensure the accuracy of decisions taken according to test scores. Thus, itemsshould be tested for bias during the process of test development andadaptation. Items used in testing programs, such as the Program forInternational Student Assessment (PISA) study, whose results informeducational policies throughout the participating countries, should bereviewed for bias. The study examines whether item bias of the 2006 PISAscience literacy test, in Turkish sample versus American sample,Purpose of the Study: The aim of this study is to analyze the measurementequivelance of the PISA science literacy test of 2006 in Turkish andAmerican groups in terms of structural invariance and also to determinewhether the science literacy items include inter-cultural bias.Methods: The study included data for 757 Turkish and 856 American 15-year-old students. Exploratory factor analysis (EFA) and confirmatoryfactor analysis (CFA) tests were performed to determine whether the PISAscience literacy test was equivalent in measurement construct in bothgroups; multi group confirmatory factor analysis (MCFA) was used toidentify differences in the factor structure according to cultures.Differential item functioning (DIF) was detected via the Mantel–Haenszel (MH), Simultaneous Item Bias Test (SIBTEST), and Item Response TheoryLikelihood- Ratio Analysis (IRT-LR) procedures.Findings and Results: : According to the MCFA results PISA 2006 scienceliteracy test, both Turkish and American groups showed equivalentmeasurement constructs. Moreover, the three analyses methods agreed atB and C levels for 15 items in the Turkish sample and 25 items in theAmerican sample in terms of DIF. According to expert opinions, commonsources for item bias were: familiarity with item content and differing skilllevels between cultures.Conclusions and Recommendations: The 38 items that showed DIF by each ofthe three methods were accepted as having DIF. The findings of thepresent study, that there is a possible source of bias in the items, will notchange the average level of student performance in participatingcountries. However, it will be beneficial to review item content before testadministration, in order to reduce the erroneous items with DIF acrossdifferent language and cultural groups in international comparativestudies.