SavedAdd your journal

Journals / ODÜ SOSYAL BİLİMLER ARAŞTIRMALARI DERGİSİ-ODÜSOBİAD / 2026 / Cilt: 16 Sayı: 1

Structural and behavioral implications of wording effects: An integrated investigation using exploratory graph analysis and explanatory item response modeling¹

Kelime etkisinin yapısal ve davranışsal yansımaları: Açıklayıcı grafik analizi ve açıklayıcı madde tepki modelleriyle bütüncül bir inceleme

Sinem Demirkol(Ordu Üniversitesi)0000-0002-9526-6156
Pages
—
Publication Date
2026-03-26
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The findings indicate that in these scales, which were theoretically designed as unidimensional, positive and negative items form separate dimensions, and positive items have a higher likelihood of being endorsed.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:The inclusion of both positive and negative items in self-report scales can generate extraneous variance unrelated to the target construct, potentially leading to the word effect (WE).
📊
Key Findings & Data:The study group consisted of 7,245 students who participated in the PISA 2022 Turkey application.
💡
Conclusion & Impact:The findings indicate that in these scales, which were theoretically designed as unidimensional, positive and negative items form separate dimensions, and positive items have a higher likelihood of being endorsed.
🔬Design:Modeling & Simulation
📍Scope:2022 Verileri
⚙️Variables:kelime etkisi · açıklayıcı grafik analizi · Açıklayıcı MTK

Abstract

The inclusion of both positive and negative items in self-report scales can generate extraneous variance unrelated to the target construct, potentially leading to the word effect (WE). The aim of this study is to examine the effect of WE on both scale structure and respondent behavior. The study group consisted of 7,245 students who participated in the PISA 2022 Turkey application. The Perseverance and Self-control scales used in the study consisted of 10 items (6 positive, 4 negative) and were administered in a five-point Likert format. Data analysis was conducted in two stages. First, the dimensionality of the scales was examined using exploratory graphical analysis, an extension of psychometric network analysis. Subsequently, the effect of KE on response behavior was analyzed using Exploratory Item Response Models. The findings indicate that in these scales, which were theoretically designed as unidimensional, positive and negative items form separate dimensions, and positive items have a higher likelihood of being endorsed. The results indicate that ignoring WE may weaken measurement validity and produce bias, especially in comparative studies. Therefore, it is necessary to systematically include word effects and method effects in the modeling and reporting stages. For future studies, it is recommended that the WE be systematically addressed in scale structures, process data such as response time/re-reading be collected, and item direction be examined.

Özet

Öz-bildirim temelli ölçeklerde olumlu ve olumsuz maddelerin birlikte yer alması, hedef yapıyla ilgisiz ek varyans üreterek kelime etkisi (KE) doğurabilir. Bu çalışmanın amacı, KE’nin hem ölçek yapısı hem de yanıtlayıcı davranışları üzerindeki etkisini incelemektir. Araştırmanın çalışma grubunu PISA 2022 Türkiye uygulamasına katılan 7.245 öğrenci oluşturmaktadır. Çalışmada ele alınan Azim ve Öz-kontrol ölçekleri 10 maddeden (6 olumlu, 4 olumsuz) oluşmakta ve beşli Likert formatında uygulanmaktadır. Verilerin analizi iki aşamada yürütülmüştür. Öncelikle psikometrik ağ analizinin bir uzantısı olan açıklayıcı grafik analiziyle ölçeklerin boyutlanması incelenmiş, ardından KE’nin yanıt davranışı üzerindeki etkisi Açıklayıcı Madde Tepki Modelleri ile analiz edilmiştir. Bulgular, kuramsal olarak tek boyutlu tasarlanan bu ölçeklerde olumlu ve olumsuz maddelerin ayrı boyutlar oluşturduğunu ve olumlu maddelerin onaylanma olasılıklarının daha yüksek olduğunu göstermektedir. Sonuçlar, KE’nin göz ardı edilmesinin ölçüm geçerliğini zayıflatıp özellikle karşılaştırmalı çalışmalarda yanlılık üretebileceğini, bu nedenle kelime etkisi ve yöntem etkilerinin modelleme ile raporlama aşamalarına sistematik biçimde dâhil edilmesi gerektiğini göstermektedir. Gelecek çalışmalar için, ölçek yapılarında KE’nin sistematik olarak ele alınması, yanıt süresi/yeniden okuma gibi süreç verilerinin toplanması ve madde yönünün incelenmesi önerilir.

🔖

Cite This Article

Demirkol, S. (2026). Structural and behavioral implications of wording effects: An integrated investigation using exploratory graph analysis and explanatory item response modeling¹. ODÜ SOSYAL BİLİMLER ARAŞTIRMALARI DERGİSİ-ODÜSOBİAD, Cilt: 16 Sayı: 1. https://doi.org/10.48146/odusobiad.1810609

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (64)

  1. Alessandri, G., Vecchione, M., Fagnani, C., Bentler, P. M., Barbaranelli, C., Medda, E., Nisticò, L., Stazi, M. A., & Caprara, G. V. (2010). Much more than model fitting? Evidence for the heritability of method effect associated with positively worded items of the Life Orientation Test–Revised. Structural Equation Modeling, 17(4), 642–653. https://doi.org/10.1080/10705511.2010.510064
  2. Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411–423. https://doi.org/10.1037/0033-2909.103.3.411
  3. Bassili, J. N., & Scott, S. B. (1996). Response latency as a signal to question problems in survey research. Public Opinion Quarterly, 60(3), 390–399. https://doi.org/10.1086/297760
  4. Benson, J., & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales on elementary school children. Journal of Educational Measurement, 22(3), 231–240.
  5. Bulut, H. C., & Bulut, O. (2022). Item wording effects in self-report measures and reading achievement: Does removing careless respondents help? Studies in Educational Evaluation, 72, 101126. https://doi.org/10.1016/j.stueduc.2022.101126
  6. Bulut, O. (2021). eirm: Explanatory item response modeling for dichotomous and polytomous item responses (R package version 0.3.0). https://CRAN.R-project.org/package=eirm https://doi.org/10.5281/zenodo.4556285
  7. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
  8. Chessa, A. G., & Holleman, B. C. (2007). Answering attitudinal questions: Modelling the response process underlying contrastive questions. Applied Cognitive Psychology, 21(2), 203–225. https://doi.org/10.1002/acp.1337
  9. Christensen, A. P., Golino, H., & Silvia, P. J. (2020). A psychometric network perspective on the validity and validation of personality trait questionnaires. European Journal of Personality, 34(6), 1095–1108.
  10. Clark, H. H. (1976). Semantics and comprehension. The Hague, Netherlands: Mouton.
  11. Cordery, J. L., & Sevastos, P. P. (1993). Responses to the original and revised Job Diagnostic Survey: Is education a factor in responses to negatively worded items? Journal of Applied Psychology, 78(1), 141–143. https://doi.org/10.1037/0021-9010.78.1.141
  12. Corwyn, R. F. (2000). The factor structure of global self-esteem among adolescents and adults. Journal of Research in Personality, 34(4), 357–379.
  13. Cote, J. A., & Buckley, R. (1987). Estimating trait, method, and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 24(3), 315–318.
  14. Dalal, D. K., & Carter, N. T. (2015). Negatively worded items negatively impact survey research. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 112–132). New York, NY: Routledge.
  15. De Boeck, P., & Wilson, M. (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer.
  16. DeVellis, R. F. (2005). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.
  17. DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13(3), 440–464.
  18. DiStefano, C., & Motl, R. W. (2009). Personality correlates of method effects due to negatively worded items on the Rosenberg Self-Esteem Scale. Personality and Individual Differences, 46(3), 309–313. https://doi.org/10.1016/j.paid.2008.10.020
  19. Ebesutani, C., Drescher, C. F., Reise, S. P., Heiden, L., Hight, T. L., & Young, J. (2012). The importance of modeling method effects: Resolving the (uni)dimensionality of the Loneliness Questionnaire. Journal of Personality Assessment. https://doi.org/10.1080/00223891.2011.627967
  20. Fiske, D. (1982). Convergent–discriminant validation in measurement and research strategies. In D. Brinberg & L. Kidder (Eds.), New directions for methodology of social and behavioral science: Forms of validity in research. San Francisco, CA: Jossey-Bass.
  21. Fuller, J. B., Patterson, C. E. P., Hester, K., & Stringer, S. Y. (1996). A quantitative review of research on charismatic leadership. Psychological Reports, 78(1), 271–287.
  22. Golino, H. F., & Christensen, A. P. (2022). EGAnet: Exploratory graph analysis—A framework for estimating the number of dimensions in multivariate data using network psychometrics (R package). https://CRAN.R-project.org/package=EGAnet
  23. Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035. https://doi.org/10.1371/journal.pone.0174035
  24. Greenleaf, E. A. (1992). Measuring extreme response style. Public Opinion Quarterly, 56(3), 328–351. https://doi.org/10.1086/269326
  25. Harvey, R. J., Billings, R. S., & Nilan, K. J. (1985). Confirmatory factor analysis of the Job Diagnostic Survey: Good news and bad news. Journal of Applied Psychology, 70(3), 461–468.
  26. Hevey, D., Pertl, M., Thomas, K., Maher, L., Craig, A., & Ní Chuinnéagain, S. (2010). Consideration of Future Consequences Scale: Confirmatory factor analysis. Personality and Individual Differences, 48(5), 654–657. https://doi.org/10.1016/j.paid.2010.01.006
  27. Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967–988.
  28. Hinz, A., D. M., Schwarz, R., & Herzberg, P. Y. (2007). The acquiescence effect in responding to a questionnaire. GMS Psycho-Social Medicine, 4, 1–9.
  29. Holleman, B. (1999). Wording effects in survey research: Using meta-analysis to explain the forbid/allow asymmetry. Journal of Quantitative Linguistics, 6(1), 29–40. https://doi.org/10.1076/jqul.6.1.29.4145 Horan, P. M., DiStefano, C., & Motl, R. W. (2003). Wording effects in self-esteem scales: Methodological artifact or response style? Structural Equation Modeling, 10(3), 435–455. https://doi.org/10.1207/S15328007SEM1003_6
  30. Idaszak, J., & Drasgow, F. (1987). A revision of the Job Diagnostic Survey: Elimination of a measurement artifact. Journal of Applied Psychology, 72(1), 69–74.
  31. John, T., Kulas, O., Rachael, K., & Lindsey, K. (2019). Confound it!: Social desirability and the “reverse-scoring” method effect. European Journal of Psychological Assessment, 35(6), 855–867. https://doi.org/10.1027/1015-5759/a000459
  32. Kam, C. C. S. (2018). Why do we still have an impoverished understanding of the item wording effect? An empirical examination. Sociological Methods & Research, 47(3), 574–597. https://doi.org/10.1177/0049124115626177
  33. Kam, C. C. S., & Fan, X. (2020). Investigating response heterogeneity in the context of positively and negatively worded items by using factor mixture modeling. Organizational Research Methods, 23(2), 322–341. https://doi.org/10.1177/1094428118790371
  34. Kamoen, N., Holleman, B., Mak, P., Sanders, T., & van den Bergh, H. (2011). Agree or disagree? Cognitive processes in answering contrastive survey questions. Discourse Processes, 48(5), 355–385. https://doi.org/10.1080/0163853X.2011.578910
  35. Lee, P. H., Chang, L. I., & Ravens-Sieberer, U. (2008). Psychometric evaluation of the Taiwanese version of the Kiddo-KINDL generic children’s health-related quality of life instrument. Quality of Life Research, 17(4), 603–611. https://doi.org/10.1007/s11136-008-9328-3
  36. Lindwall, M., Barkoukis, V., Grano, C., Lucidi, F., Raudsepp, L., Liukkonen, J., & Thøgersen-Ntoumani, C. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94(2), 196–204. https://doi.org/10.1080/00223891.2011.645936
  37. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–55.
  38. Lowe, K. B., Kroeck, K. G., & Sivasubramaniam, N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7(3), 385–425.
  39. Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810–819. https://doi.org/10.1037/0022-3514.70.4.810
  40. MEB. (2022). PISA 2022 OECD ülke raporu: Türkiye (ISBN: 978-975-11-7448-2).
  41. McLarnon, M. J., Goffin, R. D., Schneider, T. J., & Johnston, N. G. (2016). To be or not to be: Exploring the nature of positively and negatively keyed personality items in high-stakes testing. Journal of Personality Assessment, 98(5), 480–490. https://doi.org/10.1080/00223891.2016.1170691
  42. Motl, R. W., Conroy, D. E., & Horan, P. M. (2000). The Social Physique Anxiety Scale: An example of the potential consequences of negatively worded items in factorial validity studies. Journal of Applied Measurement, 1(4), 327–345.
  43. Nunnally, J. C. (1967). Psychometric theory. New York, NY: McGraw-Hill.
  44. OECD. (2024). PISA 2022 Technical Report. Paris: OECD Publishing. https://doi.org/10.1787/01820d6d-en
  45. Paiva-Salisbury, M. L., Gill, A. D., & Stickle, T. R. (2017). Isolating Trait and Method Variance in the Measurement of Callous and Unemotional Traits. Assessment, 24(6), 763–771. https://doi.org/10.1177/1073191115624546
  46. Peterson, R. A. (1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Research, 21(September), 381–391.
  47. Pilotte, W. J., & Gable, R. K. (1990). The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement, 50(3), 603–610.
  48. Podsakoff, P. M., MacKenzie, S. B., Paine, J. B., & Bachrach, D. G. (2000). Organizational citizenship behavior: A critical review of the theoretical and empirical literature and suggestions for future research. Journal of Management, 26(3), 513–563.
  49. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879
  50. R Core Team. (2025). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
  51. Roszkowski, M. J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 31(1), 113–130. https://doi.org/10.1080/02602930802618344
  52. Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114. https://doi.org/10.1177/001316448104100420
  53. Schwartz, S. J., Zamboanga, B. L., Wang, W., & Olthuis, J. V. (2009). Measuring identity from an Eriksonian perspective: Two sides of the same coin? Journal of Personality Assessment, 91(2), 143–154. https://doi.org/10.1080/00223890802634266
  54. Stanke, L., & Bulut, O. (2019). Explanatory item response models for polytomous item responses. International Journal of Assessment Tools in Education, 6(2), 259–278. https://doi.org/10.21449/ijate.515085
  55. Wagner, J. A., III, & Gooding, R. Z. (1987). Shared influence and organizational behavior: A meta-analysis of situational variables expected to moderate participation–outcome relationships. Academy of Management Journal, 30(3), 524–541.
  56. Watson, D. (1992). Correcting for acquiescent response bias in the absence of a balanced scale: An application to class consciousness. Sociological Methods & Research, 21(1), 52–88. https://doi.org/10.1177/0049124192021001003
  57. Weems, G. H., Onwuegbuzie, A. J., Schreiber, J. B., & Eggers, S. J. (2003). Characteristics of respondents who respond differently to positively and negatively worded items on rating scales. Assessment & Evaluation in Higher Education, 28(6), 587–607. https://doi.org/10.1080/0260293032000130234
  58. Weijters, B., & Baumgartner, H. (2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5), 737–747.
  59. Williams, L. J., Cote, J. A., & Buckley, M. R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? Journal of Applied Psychology, 74(3), 462–468. https://doi.org/10.1037/0021-9010.74.3.462
  60. Winkler, J. D., Kanouse, D. E., & Ware, J. E. (1982). Controlling for acquiescence response set in scale development. Journal of Applied Psychology, 67(5), 555–561. https://doi.org/10.1037/0021-9010.67.5.555
  61. Worcester, R. M., & Burns, T. R. (1975). Statistical examination of relative precision of verbal scales. Journal of the Market Research Society, 17, 181–197.
  62. Xin, Z., & Chi, L. (2010). Wording effect leads to a controversy over the construct of the Social Dominance Orientation Scale. The Journal of Psychology, 144(5), 473–488. https://doi.org/10.1080/00223980.2010.496672
  63. Ye, S. (2009). Factor structure of the General Health Questionnaire (GHQ-12): The role of wording effects. Personality and Individual Differences, 46(2), 197–201. https://doi.org/10.1016/j.paid.2008.09.027
  64. Zeng, B., Wen, H., & Zhang, J. (2020). How does the valence of wording affect features of a scale? The method effects in the Undergraduate Learning Burnout Scale. Frontiers in Psychology, 11, 585179. https://doi.org/10.3389/fpsyg.2020.585179