Applied Mathematical Statistical Modeling to Determine the Validity and Reliability of Independent Instructional Measures

James Edward Osler II*, Mahmud A. Mansaray**
* North Carolina Central University.
** Research Analyst, Department of Research, Evaluation and Planning, North Carolina Central University.
Periodicity:July - September'2013
DOI : https://doi.org/10.26634/jmat.2.3.2458

Abstract

Most universities and colleges in United States and elsewhere use student ratings of instruction for summative (e.g. faculty promotion, etc.) and formative (e.g. instructional improvement, etc.) evaluation purposes. This research examines the reliability and validity of the rating instrument used to evaluate teaching effectiveness and student learning outcomes. The sample consisted of 7,919 students of Historic Black College and University [HBCU] who completed the 2012 spring semester. Student Ratings of Instruction [SRI] survey (a measurement instrument used by students to assess instruction in courses in which they are currently enrolled) was used. The application of the Cronbach's Alpha Reliability Model, in the study helps to find strong internal reliability [consistency] of the rating instrument used to comprehensively assess teaching quality. At the same time, the application of Principal Component Factor Analysis with Varimax Rotation also finds strong construct validity of the assessment tool used to evaluate faculty.

Keywords

Student Ratings of Instruction; Construct Validity; Reliability; Cronbach's Alpha, Principal Component Factor Analysis, Statistics

How to Cite this Article?

Osler, J. E., II. and Mansaray, M.A. (2013). Applied Mathematical Statistical Modeling to Determine The Validity And Reliability of Independent Instructional Measures. i-manager’s Journal on Mathematics, 2(3), 12-31. https://doi.org/10.26634/jmat.2.3.2458

References

[1]. Agbetsiafa, D. (2010). Evaluating effective teaching in college level Economics using student ratings of instruction: A factor analytic approach, Journal of College Teaching & Learning, 7(5), 57-66.
[2]. Anastasiadou, S. D. (2011). Reliability and validity testing of a new scale for measuring attitudes toward learning statistics with technology. Acta Didactica Napocensia, 4(1), 1-10.
[3]. Anderson, J., Brown, G., & Spaeth, S. (2006). Online student evaluations and response rates reconsidered. Innovate: Journal of Online Education, 2(6). Retrieved from https://rugby.ou.edu/content/dam/provost/documents/evaluations/ evaluate-Online- Student-Evaluations-and-Response-Rates.pdf
[4]. Barry, R. A. (2010). Teaching effectiveness and why it matters. Marylhurst University and the Chalkboard Project, Retrieved from http://chalkboardproject.org/wp-content/uploads/2010/12/teacher-effectiveness-and-why-it-matters.pdf
[5]. Beran, T., & Rokosh, J. (2009). Instructors' perspectives on the utility of student ratings of instruction. Instructional Science, 37(2), 171-184. doi:10.1007/s11251-007-9045-2.
[6]. Beran, T., Violato, C., Kline, D., & Frideres, J. (2009). What do students consider useful about student ratings?. Assessment & Evaluation in Higher Education, 34(5), 519-527.
[7]. Beran, T., Violato, C., & Kline, D. (2007). What's the "Use" of student ratings of instruction for administrators? One university's experience. Canadian Journal of Higher Education, 37(1), 27-43.
[8]. Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48-62.
[9]. Bland, J., & Altman, D. G. (1997). Cronbach's Alpha . BMJ: British Medical Journal (International Edition), 314(7080), 572.
[10]. Bo, Z., Johnston, L., & Kilic, G. (2008). Assessing the reliability of self- and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329-340. doi:10.1080/02602930701293181.
[11]. Chen, G., & Watkins, D. (2010). Stability and correlates of student evaluations of teaching at a Chinese university. Assessment & Evaluation in Higher Education, 35(6), 675-685. doi:10.1080/02602930902977715.
[12]. Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn?. Journal of Marketing Education, 31(1), 16-30.
[13]. Donnon, T., Delver, H., & Beran, T. (2010). Student and teaching characteristics related to ratings of instruction in medical sciences graduate programs, Medical Teacher, 32(4), 327-332. doi:10.3109/01421590903480097.
[14]. El Hassan, K. (2009). Investigating substantive and consequential validity of student ratings of instruction. Higher Education Research & Development, 28(3), 319-333. doi:10.1080/07294360902839917.
[15]. Field, A. (2009). Discovering statistics using spss. (3rd ed.). Thousand Oaks, CA: SAGE Publications Inc.
[16]. Fitzpatrick, J. L., & Miller-Stevens, K. (2009). A case study of measuring outcomes in an MPA program. Journal of Public Affairs Education, 15(1), 17-31.
[17]. Frick, T., Chadha, R., Watson, C., Wang, Y., & Green, P. (2009). College student perceptions of teaching and learning quality. Educational Technology Research & Development, 57(5), 705-720. doi:10.1007/s11423-007-9079-9.
[18]. Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The generalizability of student ratings of Instruction: Estimation of the teacher and course components. Journal of Educational Measurement, 15, 1-13.
[19]. Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness: A research synthesis. National Comprehensive Center for Teacher Quality, Retrieved from http://www.tqsource.org/publications/EvaluatingTeach Effectiveness.pdf
[20]. Gravestock, P., & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto, ON: Higher Education Quality Council of Ontario. Retrieved from http://www.heqco.ca/SiteCollectionDocuments/Student%20 Course%20Evaluations.pdf
[21]. Haladyna, T. M., & Amrein-Beardsley, A. (2009). Validation of a research-based student survey of instruction in a college of education. Educational Assessment, Evaluation and Accountability, 21(3), 255-276. doi:10.1007/s11092-008-9065-8.
[22]. Hassel, B. C. (2009). How should state define teacher effectiveness?. Center for American Progress, Retrieved from http://publicimpact.com/publications/PublicImpact-How_Should_States_Define_Teacher_Effectiveness.pdf
[23]. Héfer, B. (2009). Teaching effectiveness, course evaluation, and academic performance: The role of academic delay of gratification. Journal of Advanced Academics, 20(2), 326-355.
[24]. Hogan, T. P. (1973). Similarity of student ratings across instructors, courses, and time. Research in Higher Education, 1, 149-154.
[25]. Hunsaker, S. L., Nielsen, A., & Bartlett, B. (2010). Correlates of teacher practices influencing s t u d e n t o u t c ome s i n reading instruction for advanced readers. Gifted Child Quarterly, 54(4), 273-282. doi:10.1177/0016986210374506.
[26]. Ibrahim, A. (2011). Using generalizability theory to estimate the relative effect of class size and number of items on the dependability of student ratings of instruction. Psychological Reports, 109(1), 252-258. doi:10.2466/03.07.11.PR0.109.4.252-258.
[27]. Jones, B. (2010). An examination of motivation model components in face-to-face and online instruction. Electronic Journal of Research in Educational Psychology, 8(3), 915-944.
[28]. Keeley, J., Furr, R., & Buskist, W. (2010). Differentiating psychology students' perceptions of teachers using the teacher behavior checklist. Teaching of Psychology, 37(1), 16-20. doi:10.1080/00986280903426282.
[29]. Kneipp, L. B., Kelly, K. E., Biscoe, J. D., & Richard, B. (2010). The impact of instructor's personality characteristics on quality of instruction. College Student Journal, 44(4), 901-905.
[30]. Kozub, R. M. (2010). Relationship of course, instructor, and student characteristics to dimensions of student ratings of teaching effectiveness in business schools. American Journal of Business Education, 3(1), 33-40.
[31]. Leech, N. A., Barrett, K. C., Morgan, G. A. (2011). IBM: SPSS for intermediate statistics use and interpretation. (4th ed.). New York, NY: Taylor & Francis Group.
[32]. Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9(3), 215-230. doi:10.1007/s10984-006-9014-8.
[33]. Madden, T. J., Dillon, W. R., & Leak, R. L. (2010). Students' evaluation of teaching: Concerns of item diagnosticity. Journal of Marketing Education, 32(3), 264-274. doi:10.1177/0273475310377759.
[34]. Marsh, H. W. (1982). The use of path analysis to estimate teacher and course effects in student rating's of instructional effectiveness, Applied Psychological Measurement, 6, 47-59.
[35]. Marsh, H. W. (2007). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and usefulness. In R. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319-383). Netherlands: Springer.
[36]. Marsh, H. W., Ginns, P., Morin, A. S., Nagengast, B., & Martin, A. J. (2011). Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian Course Experience Questionnaire (CEQ). Journal of Educational Psychology, 103(3), 733-748. doi:10.1037/a0024221.
[37]. McKeachie, W. J. (2007). Good teaching makes a difference—And we know what it is. In R. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 457-474). Netherlands: Springer.
[38]. Perry, R. & Smart, J. C. (2007). The scholarship of teaching and learning in higher education: An evidence-based perspective. Netherlands: Springer.
[39]. Pounder, J. (2007). Is student evaluation of teaching worthwhile? An analytical framework for answering the question. Quality Assurance in Education, 18(1), 47-63.
[40]. Roche, L. A., & Marsh, H. W. (2000). Multiple dimensions of university teacher selfconcept. Instructional Science, 28(5), 439-468.
[41]. Safavi, S., Bakar, K., Tarmizi, R., & Alwi, N. (2012). The role of student ratings of instruction from perspectives of the higher education administrators. International Journal of Business and Social Science, 3(9), 233-239.
[42]. Schrodt, P., Witt, P. L., Myers, S. A., Turman, P. D., Barton, M. H., & Jernberg, K. A. (2008). Communication Education, 57(2), 180-200. doi:10.1080/03634520701840303.
[43]. Skowronek, J., Friesen, B., & Masonjones, H. (2011). Developing a statistically valid and practically useful student evaluation instrument, International Journal for the Scholarship of Teaching and Learning, 5(1), 1-19.
[44]. Spooren, P. P., Mortelmans, D. D., & Denekens, J. J. (2007). Student evaluation of teaching quality in higher education: Development of an instrument based on 10 Likert-scales, Assessment & Evaluation in Higher Education, 32(6), 667-679. doi:10.1080/02602930601117191.
[45]. Sprinkle, J. E. (2008). Student perceptions of effectiveness: An examination of the influence of student biases, College Student Journal, 42(2), 276-293.
[46]. Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction, Assessment & Evaluation in Higher Education, 37(4), 465-473. doi:10.1080/02602938.2010.545869.
[47]. Theall, M. (2010). Evaluating teaching: From reliability to accountability, New Directions for Teaching and Learning, 2010(123), 85-95. doi:10.1002/tl.412.
[48]. Titus, J. (2008). Student ratings in a consumerist academy: Leveraging pedagogical control and authority, Sociological Perspective, 51, 397-422.
[49]. Vogt, W. (1984). Developing a teacher evaluation system, Spectrum, 2(1), 41-46.
[50]. Young, S., & Shaw, D. G. (1999). Profiles of effective college and university teachers, Higher Education, 70(6), 670-686.
[51]. Zerbini, T., & Abbad, G. (2009). Reação aos procedimentos instrucionais de um curso via internet: Validação de uma escala. Estudos De Psicologia, 26(3), 363-371. doi:10.1590/S0103-166X2009000300009.
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Pdf 35 35 200 20
Online 35 35 200 15
Pdf & Online 35 35 400 25

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.