i-manager Publications

Technology Engineering Online Learner Metrics ToAnalyze Instructional Efficacy

James Edward Osler II*, Mahmud A. Mansaray**

* North Carolina Central University

** North Carolina Central University

Periodicity:July - September'2013
DOI : https://doi.org/10.26634/jet.10.2.2413

Abstract

The online deployment of Technology Engineered online Student Ratings of Instruction [SRIs] by colleges and universities in the United States has dynamically changed the deployment of course evaluation. This research investigation is the fourth part of a post hoc study that analytically and psychometrically examines the design, reliability, and validity of the items used in the delivery of an SRI instrument online during the 2012 Spring Semester at a United States Historical Black College and University [HBCU]. The sample under analysis consisted of 7,919 distributed Student Ratings Instruments with a Grand Total of 56451 analyzed instrument items. The application of several statistical measures was used to validate the Technologically Engineered SRI instrument used in the study. Results of the study included: a Cronbach's Alpha Test that yielded an extraordinarily high internal consistency of the student rejoinders to the items on the rating scale; and a Tri–Squared Test supported the research findings attributed to the Cronbach's Alpha Test results. This research is the continuation of in–depth investigations into Technology Engineering as discussed in iManager's Journal on School Educational Technology, further explored in iManager's Journal of Educational Technology, and data analyzed with the Tri–Squared statistical measure first introduced in iManager's Journal of Mathematics.

Keywords

Advanced Statistics, Construct Validity, Cronbach's Alpha, Efficacy, Goodman & Kruskal's Lambda, Principal Component Factor Analysis, Reliability, Technology Engineering, Statistics, Student Ratings of Instruction [SRI], Tri–Squared Test, and Validity

How to Cite this Article?

Osler , J. E., and Mansaray , M. A. (2013). Technology Engineering Online Learner Metrics To Analyze Instructional Efficacy. i-manager’s Journal of Educational Technology, 10(2), 43-59. https://doi.org/10.26634/jet.10.2.2413

References

[1]. Agbetsiafa, D. (2010). Evaluating effective teaching in college level Economics using student ratings of instruction: A factor analytic approach. Journal of College Teaching & Learning, 7(5), 57–66.

[2]. Anastasiadou, S. D. (2011). Reliability and validity testing of a new scale for measuring attitudes toward learning statistics with technology. Acta Didactica Napocensia, 4(1), 1–10.

[3]. Barry, R. A. (2010). Teaching effectiveness and why it matters. Marylhurst University and the Chalkboard Project.

[4]. Beran, T., Violato, C., Kline, D., & Frideres, J. (2009). What do students consider useful about student ratings?. Assessment & Evaluation in Higher Education, 34(5), 519–527.

[5]. Beran, T., Violato, C., & Kline, D. (2007). What's the "Use" of student ratings of instruction for administrators? One university's experience. Canadian Journal of Higher Education, 37(1), 27–43.

[6]. Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62.

[7]. Bland, J., & Altman, D. G. (1997). Cronbach's Alpha . BMJ: British Medical Journal (International Edition), 314(7080), 572.

[8]. Bo, Z., Johnston, L., & Kilic, G. (2008). Assessing the reliability of self– and peer rating in student group work. Assessment & Evaluation in Higher Education, 33(3), 329–340.

[9]. Chen, G., & Watkins, D. (2010). Stability and correlates of student evaluations of teaching at a Chinese university. Assessment & Evaluation in Higher Education, 35(6), 675–685.

[10]. Clayson, D. E. (2009). Student evaluations of teaching: Are they related to what students learn?. Journal of Marketing Education, 31(1), 16–30.

[11]. Donnon, T., Delver, H., & Beran, T. (2010). Student and teaching characteristics related to ratings of instruction in medical sciences graduate programs. Medical Teacher, 32(4), 327–332.

[12]. El Hassan, K. (2009). Investigating substantive and consequential validity of student ratings of instruction. Higher Education Research & Development, 28(3), 319– 333.

[13]. Field, A. (2009). Discovering statistics using spss. (3rd ed.). Thousand Oaks, CA: SAGE Publications Inc.

[14]. Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The generalizability of student ratings of Instruction: Estimation of the teacher and course components. Journal of Educational Measurement, 15, 1–13.

[15]. Hassel, B. C. (2009). How should state define teacher effectiveness?. Center for American Progress.

[16]. Hogan, T. P. (1973). Similarity of student ratings across instructors, courses, and time, Research in Higher Education, 1, 149–154.

[17]. Hunsaker, S. L., Nielsen, A., & Bartlett, B. (2010). Correlates of teacher practices influencing student outcomes in reading instruction for advanced readers. Gifted Child Quarterly, 54(4), 273–282.

[18]. Ibrahim, A. (2011). Using generalizability theory to estimate the relative effect of class size and number of items on the dependability of student ratings of instruction. Psychological Reports, 109(1), 252–258.

[19]. Jones, B. (2010). An examination of motivation model components in face–to–face and online instruction. Electronic Journal of Research in Educational Psychology, 8(3), 915–944.

[20]. Keeley, J., Furr, R., & Buskist, W. (2010). Differentiating psychology students' perceptions of teachers using the teacher behavior checklist. Teaching of Psychology, 37(1), 16–20.

[21]. Kneipp, L. B., Kelly, K. E., Biscoe, J. D., & Richard, B. (2010). The impact of instructor's personality characteristics on quality of instruction. College Student Journal, 44(4), 901–905.

[22]. Leech, N. A., Barrett, K. C., Morgan, G. A. (2011). IBM: SPSS for intermediate statistics use and interpretation. (4th ed.). New York, NY: Taylor & Francis Group.

[23]. Marsh, H. W. (1981). The use of path analysis to estimate teacher and course effects in student rating's of instructional effectiveness. Applied Psychological Measurement, 6, 47–59.

[24]. Marsh, H. W. (2007). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and usefulness. In R. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence–based perspective (pp. 319–383). Netherlands: Springer.

[25]. Marsh, H. W., Ginns, P., Morin, A. S., Nagengast, B., & Martin, A. J. (2011). Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian Course Experience Questionnaire (CEQ). Journal of Educational Psychology, 103(3), 733–748.

[26]. McKeachie, W. J. (2007). Good teaching makes a difference—And we know what it is. In R. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence–based perspective (pp. 457–474). Netherlands: Springer.

[27]. Osler, J. E., Hollowell, P. G. & Nichols, S. M. (2012). Technology Engineering in Science Education: Where Instructional Challenges Interface Nonconforming Productivity to Increase Retention, Enhance Transfer, and Maximize Student Learning. July–September i-manager’s Journal of Educational Technology, 9 (2), pp. 31–39.

[28]. Osler, J. E., Hollowell, G. P., & Palmer, C. E. (2008). Technology Engineering Science Instruction in the Information Age: Integrating Instructional Technology in K12 Education, June–August, i-manager’s Journal on School Educational Technology, 4 (1), pp. 12–17.

[29]. Roche, L. A., & Marsh, H. W. (2000). Multiple dimensions of university teacher self concept. Instructional Science, 28(5), 439–468.

[30]. Safavi, S., Bakar, K., Tarmizi, R., & Alwi, N. (2012). The role of student ratings of instruction from perspectives of the higher education administrators. International Journal of Business and Social Science, 3(9), 233–239.

[31]. Schrodt, P., Witt, P. L., Myers, S. A., Turman, P. D., Barton, M. H., & Jernberg, K. A. (2008). Communication Education, 57(2), 180–200.

[32]. Skowronek, J., Friesen, B., & Masonjones, H. (2011). Developing a statistically valid and practically useful student evaluation instrument. International Journal for the Scholarship of Teaching and Learning, 5(1), 1–19.

[33]. Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom–based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37(4), 465–473. D

[34]. Vogt, W. (1984). Developing a teacher evaluation system. Spectrum, 2(1), 41–46.

[35]. Young, S., & Shaw, D. G. (1999). Profiles of effective college and university teachers. Higher Education, 70(6), 670–686.

[36]. Zerbini, T., & Abbad, G. (2009). Reação aos procedimentos instrucionais de um curso via internet: Validação de uma escala. Estudos De Psicologia, 26(3), 363–371.

[37]. Zhao, J., & Gallant, D. J. (2012). Student evaluation of instruction in higher education: exploring issues of validity and reliability. Assessment & Evaluation in Higher Education, 37(2), 227–235.

[38]. Zuberi, R. W., Bordage, G., & Norman, G. R. (2007). Validation of the SETOC instrument – student evaluation of teaching in outpatient clinics. Advances in Health Sciences Education, 12(1), 55–69.

If you have access to this article please login to view the article or kindly login to purchase the article

Single Article

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Online	15	15

ADD TO CART

Options for accessing this content:

If you would like institutional access to this content, please recommend the title to your librarian.
If you already have i-manager's user account: Login above and proceed to purchase the article.
New Users: Please register, then proceed to purchase the article.