Better Quality in Assessments: Consideration of Contextual Effects on Item Bias and Differential Item Functioning

Brandon K. Vaughn*
* Assistant Professor, The University of Texas at Austin.
Periodicity:September - November'2008
DOI : https://doi.org/10.26634/jsch.4.2.620

Abstract

This study considers the importance of contextual effects on the quality of assessments on item bias and differential item functioning (DIF) in measurement.  Often, in educational studies, students are clustered in teachers or schools, and the clusters could impact psychometric issues yet are largely ignored by traditional item analyses.  A statistical model for incorporating these cluster effects will be presented.  By considering DIF from this perspective, it is possible for DIF to be consistent or vary across clusters (e.g., teachers or schools).  By using this model, researchers can be shown much more detail into the nature and source of DIF and provide greater quality in the analysis and use of their assessments.  This can be of benefit in determining whether the nature of DIF is exclusively due to student attributes, or a particular combination of student and school attributes.  In addition to this, DIF may exist only among clusters, but not for students.  This is an extra detection not possible with traditional DIF analysis.  For example, in some educational situations, DIF may not exist among the subjects of interest, but in certain type of schools.  The use of this procedure will be demonstrated using real assessment data.

Keywords

Social Context, Cluster Effects, Multilevel Model, Differential Item Functioning, Assessments.

How to Cite this Article?

Brandon K. Vaughn (2008). Better Quality in Assessments: Consideration of Contextual Effects on Item Bias and Differential Item Functioning. i-manager’s Journal on School Educational Technology. 4(2), 29-39. https://doi.org/10.26634/jsch.4.2.620

References

[1]. Angoff, W. H. (1993). Perspectives on differential item functioning methodology. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 3-24). Hillsdale, NJ: Erlbaum.
[2]. Angoff, W. H. (September 1972). A technique for the investigation of cultural differences. Paper presented at the annual meeting of the American Psychological Association, Honolulu.
[3]. Borsboom, D., Mellenbergh, G. J., & van der Linden, W. J. (2002). Different kinds of DIF: A distinction between absolute and relative forms of measurement invariance and bias. Applied Psychological Measurement, 26, 433- 450.
[4]. Buu, Y. A. (2003). Statistical analysis of rater effects. Unpublished doctoral dissertation, University of Florida, Gainesville, FL.
[5]. Camilli, G., & Shepard, L. A. (1987). The inadequacy of ANOVA for detecting test bias. Journal of Educational Statistics, 12, 87-99.
[6]. Cardall, C., & Coffman, W. E. (1964). A method for comparing the performance of different groups on the items of a test. (No. Research Bulletin 64-61). Princeton, N.J: Educational Testing Service.
[7]. Chaimongkol, S. (2005). Modeling differential item functioning (DIF) using multilevel logistic regression models: A Bayesian perspective. Unpublished doctoral dissertation, Florida State University, Tallahassee, FL.
[8]. Cole, N. S. (1978). Approaches to examining bias in achievement test items. Paper presented at the national meeting of the American Personnel and Guidance Association, Washington, DC.
[9]. Cole, N. S., & Moss, P. A. (1989). Bias in test use. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 201- 219 ) . NewYork : American Council on Education/Macmillan.
[10]. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.
[11]. Holland, P. W., & Wainer, H. (Eds.). (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum Associates.
[12]. Hunter, J. E. (1975). A critical analysis of the use of item means and item-test correlations to determine the pressure or absence of content bias in achievement test items. Paper presented at the National Institute of Education conference on test bias, Annapolis, MD.
[13]. Jensen, A. R. (1980). Bias in mental testing. New York: Free Press.
[14]. Kamata, A., & Binici, S. (2003). Random-effect DIF analysis via hierarchical generalized linear models. Paper presented at the annual meeting of the Psychometric Society, Sardinia, Italy.
[15]. Kamata, A., & Vaughn, B. K. (2004). An introduction to differential item functioning analysis. Learning Disability: A Contemporary Journal, 2(2), 49-69.
[16]. Lord, F. M. (1977). A study of item bias, using item characteristic curve theory. In Y. H. Poortinga (Ed.), Basic problems in cross-cultural psychology (pp. 19-29). Amsterdam: Swets & Zeitlinger.
[17]. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., & Congdon, R. T. (2000). HLM 5: Hierarchical linear and nonlinear modeling [Computer Software]. Lincolnwood, IL: Scientific Software International, Inc.
[18]. Raudenbush, S. W., Bryk, A. S., & Congdon, R. T. (2005). HLM 6: Hierarchical linear and nonlinear modeling (Version 6.02) [Computer software]. Lincolnwood, IL: Scientific Software International, Inc.
[19]. Raudenbush, S. W., Yang, M. l., & Yosef, M. (2000). Maximum likelihood for hierarchical models via high order, multivariate LaPlace approximation. Journal of Computational and Graphical Statistics, 9(1), 141-157.
[20]. Raudenbush, S. W., Yang, M.-L., & Yosef, M. (2000). Maximum likelihood for generalized linear models with nested random effects via high-order, multivariate Laplace approximation. Journal of Computational and Graphical Statistics, 9, 141-157.
[21]. Rock, D. A., Pollack, J. M., & Quinn, P. (1995). Psychometric Report for the NELS:88 Base Year through Second Follow-Up. National Education Longitudinal Study of 1988. (No. NCES-95-382). Washington, DC: National Center for Education Statistics.
[22]. Shealy, R. T., & Stout, W. F. (1993b). An item response theory model for test bias and differential item functioning. In P. Holland & H. Wainer (Eds.), Differential item functioning. Hillsdale, NJ: Erlbaum.
[23]. Shealy, R. T., & Stout, W. F. (1993a). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159-194.
[24]. Shepard, L. A. (1981). Identifying bias in test items. In B. F. Green (Ed.), New direction in testing and measurement: Issues in testing, coaching, disclosure and test bias (Vol. 11, pp. 79-104). San Francisco: Jossey- Bass.
[25]. Swanson, D. B., Clauser, B. E., Case, S. M., Nungster, R. J., & Featherman, C. (2002). Analysis of differential item functioning (DIF) using hierarchical logistic regression models. Journal of Educational and Behavioral Statistics, 27, 53-75.
[26]. Vaughn, B. K. (2006). A Hierarchical Generalized Linear Model of Random Differential Item Functioning for Polytomous Items: A Bayesian Multilevel Approach. Unpublished doctoral dissertation, Florida State University, Tallahassee, FL.
[27]. Whitmore, M. L., & Schumacker, R. E. (1999). A comparison of logistic regression and analysis of variance differential item functioning detection methods. Educational and Psychological Measurement, 59, 910- 927.
[28]. Williams, V. S. L. (1997). The "unbiased" anchor: Bridging the gap between DIF and item bias. Applied Measurement in Education, 10, 253-267.
[29]. Yang, M. L. (1988). Increasing the efficiency in estimating multilevel Bernoulli models. Unpublished doctoral dissertation, Michigan State University, East Lansing, MI.
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Online 15 15

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.