Assessing Measurement Invariance with Dichotomous Items: The Case of Early Grade Mathematic Assessment from the Zambian Sample

Brian Mumba*, Devrim Alici**, N. Bilge Uzun***
*-*** Department of Measurement and Evaluation in Education, Mersin University, Mersin, Turkey.
Periodicity:May - July'2022

Abstract

Assessment of measurement invariance is an essential component of construct validity in psychological measurement. However, the procedure for assessing measurement invariance with dichotomous items partially differs from that of invariance testing with continuous items. However, many studies have focused on invariance testing with continuous items or ordered polytomous items. Hence, this study assessed the measurement invariance with dichotomous items using Confirmatory Factor Analysis. The study was conducted with data from primary school students who participated in the Early Grade Mathematics Assessment Survey (EGMA) from different provinces of Zambia in 2018. The study determined measurement invariance for three subtests of EGMA according to gender. The total sample consisted of 4698 individuals (49% male, n = 2305 and 51% female, n = 2393). The method used was a Confirmatory Factor Analysis with three highly correlated factors, number identification, number discrimination, and recognition of number patterns. The findings revealed measurement invariance of the EGMA model as defined by three subtests (number identification, number discrimination, and recognition of number patterns) across gender subgroups. However, individual analysis of each subtest revealed that the test scores have the same unit of measurement for both male and female on two factors, excluding those of number identification that lack measurement invariance. Finally, these results contribute to evidence of validity for EGMA scores in terms of their internal structure from the perspective of equality of measurement.

Keywords

Measurement Invariance with Dichotomous Items, Early Grade Mathematics Assessment, Multiple Group Confirmatory Factor Analysis, Validity.

How to Cite this Article?

Mumba, B., Alici, D., and Uzun, N. B. (2022). Assessing Measurement Invariance with Dichotomous Items: The Case of Early Grade Mathematic Assessment from the Zambian Sample. i-manager’s Journal on Educational Psychology, 16(1), 1-11.

References

[1]. Bofah, E. A. T., & Hannula, M. S. (2015). TIMSS data in an African comparative perspective: Investigating the factors influencing achievement in mathematics and their psychometric properties. Large-Scale Assessments in Education, 3(1), 1-36. https://doi.org/10.1186/s40536-015-0014-y
[2]. Brown, T. A. (2006). Confirmatory Factor Analysis for Applied Research. New York, NY: Guilford.
[3]. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464-504. https://doi.org/10.1080/10705510701301834
[4]. Cimetta, A. D., Marx, R. W., Yaden, D. B., Alkhadim, G. S., & Cutshaw, C. A. (2018). Latent variable structure and measurement invariance of a modified early literacy assessment. International Journal of School & Educational Psychology, 6(3), 208-222. https://doi.org/10.1080/21683603.2017.1322016
[5]. Demir, E. (2017). Testing Measurement Invariance of the Students' Affective Characteristics Model across Gender Sub-Groups. Educational Sciences: Theory and Practice, 17(1), 47-62.
[6]. Hansson, Å., & Gustafsson, J. E. (2013). Measurement invariance of socioeconomic status across migrational background. Scandinavian Journal of Educational Research, 57(2), 148-166. https://doi.org/10.1080/00313831.2011.625570
[7]. Hu, E., Stavropoulos, V., Anderson, A., Clarke, M., Beard, C., Papapetrou, S., & Gomez, R. (2019). Assessing online flow across cultures: A two-fold measurement invariance study. Frontiers in Psychology, 10, 407.
[8]. Kim, E. S., & Yoon, M. (2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling, 18(2), 212-228. https://doi.org/10.1080/10705511.2011.557337
[9]. Kim, S. H., & Cohen, A. S. (1995). A comparison of Lord's chi-square, Raju's area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8(4), 291-312. https://doi.org/10.1207/s15324818ame0804_2
[10]. Koziol, N. A. (2010). Evaluating Measurement Invariance with Censored Ordinal Data: A Monte Carlo Comparison of Alternative Model Estimators and Scales of Measurement. Theses and Dissertations from the College of Education and Human Sciences. University of Nebraska – Lincoln.
[11]. Koziol, N., & Bovaird, J. A. (2009). Using Experimental Designs to Evoke Construct Validity Evidence. Poster presented at the annual meeting of the American Psychological Association, Toronto, Canada.
[12]. Millsap, R. E. (2011). Statistical Approaches to Measurement Invariance. New York: Taylor and Francis Group.
[13]. Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479-515.
[14]. Molenaar, D., & Borsboom, D. (2013). The formalization of fairness: Issues in testing for measurement invariance using subtest scores. Educational Research and Evaluation, 19(2-3), 223-244. https://doi.org/10.1080/13803611.2013.767628
[15]. Muthén, B., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Mplus Web Notes, 4(5), 1-22.
[16]. Muthén, L. K., & Muthén, B. O. (2017). Mplus (Version 8)[Computer software]. Los Angeles, CA: Author.
[17]. Rojas, L., Rojas, G., & Brizuela, A. (2018). The Use of Measurement Invariance with Dichotomous Variables as Evidence of Validity. Instituto de Investigaciones Psicológicas, Universidad de Costa Rica.
[18]. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1-36. https://doi.org/10.18637/jss.v048.i02
[19]. RTI International. (2014). Early Grade Mathematics Assessment (EGMA) Toolkit. Retrieved from https://shared.rti.org/content/early-grade-mathematics-assessmentegma-toolkit
[20]. Sideridis, G. D., Tsaousis, I., & Al-harbi, K. A. (2015). Multi-population invariance with dichotomous measures: combining multi-group and mimic methodologies in evaluating the general aptitude test in the arabic language. Journal of Psychoeducational Assessment, 33(6), 568-584. https://doi.org/10.1177/0734282914567871
[21]. Sinnes, A. (2014). Understanding student participation and choice in science and technology education. Springer, 17-19.
[22]. USAID. (2016). Early Grade Reading and Mathematics Assessment in the Republic of Macedonia: Study Report. Retrieved from https://www.stepbystep.org.mk/WEBprostor/EGRA_and_EGMA_Study_Report_- _May_2015.pdf
[23]. Uzun, B., & Ögretmen, T. (2010). Assessing the Measurement Invariance of Factors that are Related to Students' Science Achievement across Gender in TIMSS-R Turkey Sample. Egitim ve Bilim, 35(155), 26–35.
[24]. Wendt, H., Kasper, D., & Trendtel, M. (2017). Assuming measurement invariance of background indicators in international comparative educational achievement studies: a challenge for the interpretation of achievement differences. Large-scale Assessments in Education, 5(1), 1-34. https://doi.org/10.1186/s40536-017-0043-9
[25]. Widaman, K. F., & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. Educational and Psychological Measurement, 43, 729-740.
[26]. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58-79. https://doi.org/10.1037/1082-989X.12.1.58
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Pdf 35 35 200 20
Online 35 35 200 15
Pdf & Online 35 35 400 25

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.