i-manager Publications

Chatbot Usability Scale: Adaptation to Turkish and Validation/Reliability Analysis

Mehmet Yavuz*, Bünyami Kayali**, Şener Balat***, Mücahit Çalişan****

* Bingöl University's Distance Education and Research Center, Turkey.

** Department of Computer Technologies, Bayburt University, Merkez, Turkey.

***,**** Bingöl University, Turkey.

Periodicity:April - June'2023
DOI : https://doi.org/10.26634/jet.20.1.19758

Abstract

The aim of this study is to adapt the 15-item Chatbot Usability Scale to the Turkish language and culture and evaluate the validity and reliability of the scale in the Turkish language and culture after the adaptation process. The necessary permissions were obtained, and the process was initiated. Proficient translators in both cultures were selected to ensure linguistic equivalence, and they carried out the translation process proficiently in both languages. The scale was translated back to its original language to enhance its suitability, and linguistic equivalence was ensured. The sample for the study included 406 students from eight different undergraduate and associate degree programs at a state university. Following the item analysis, the Cronbach's Alpha coefficient of the scale was found to be 0.935. Exploratory and confirmatory factor analyses were applied and it was found that the scale had a two-factor structure for Turkish university students. As a result of this study, the Chatbot Usability Scale has been adapted to the Turkish language and culture. As a suggestion, the generalizability of the scale can be increased by applying it to individuals from different age groups, genders, educational levels, and occupational groups and evaluating the test-retest reliability of the scale in future studies.

Keywords

Chatbot, Usability Scale, Artificial Intelligence, Turkish Adaptation, Validation Analysis, Reliability Analysis, Turkish Language.

How to Cite this Article?

Yavuz, M., Kayali, B., Balat, Ş., and Çalişan, M. (2023). Chatbot Usability Scale: Adaptation to Turkish and Validation/Reliability Analysis. i-manager’s Journal of Educational Technology, 20(1), 11-19. https://doi.org/10.26634/jet.20.1.19758

References

[1]. Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006.

[2]. Balaji, D. (2019). Assessing user Satisfaction with İn formation Chatbots: A Preliminary İnvestigation (Master's thesis, University of Twente).

[3]. Başkaya, F., & Karacan, H. (2022). The effect of artificial ıntelligence-based systems on personal data privacy: An ınvestigation on chatbots. Journal of Information Technologies, 15(4), 481-491.

[4]. Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F. , Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022). The Chatbot Usability Scale: The design and pilot of a usability scale for interaction with AI-based conversational agents. Personal and Ubiquitous Computing, 26, 95-119.

[5]. Borsci, S., Prati, E., Malizia, A., Schmettow, M., Chamberlain, A., & Federici, S. (2023b). Ciao AI: The Italian adaptation and validation of the Chatbot Usability Scale. Personal and Ubiquitous Computing (pp. 1-10).

[6]. Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., & Van Der Velde, F. (2023a). A confirmatory factorial analysis of the Chatbot Usability Scale: A multilanguage validation. Personal and Ubiquitous Computing, 27(2), 317-330.

[7]. Büyüköztürk, Ş. (2007). Data Analysis Handbook. Pegem Akademi Yayıncılık, Ankara, Turkey (pp. 167-182).

[8]. Büyüköztürk, Ş. (2011). Data Analysis Handbook for Social Sciences. Pegem Akademi, Ankara.

[9]. Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264-75278.

[10]. Field, A. (2019). Discovering Statistics using IBM SPSS Statistics. SAGE Publications, India.

[11]. Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022). How does ChatGPT perform on the medical licensing exams? The implications of large language models for medical education and knowledge assessment. MedRxiv.

[12]. Gliem, J. A., & Gliem, R. R. (2003). Calculating, interpreting, and reporting Cronbach's alpha reliability coefficient for Likert-type scales. Midwest Research-toPractice Conference in Adult, Continuing, and Community Education (pp. 82-88).

[13]. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis. Pearson.

[14]. Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471.

[15]. Hidayat, A., & Nugroho, A. (2022, December). Usability evaluation on educational chatbot using the System Usability Scale (SUS). In 2022 Seventh International Conference on Informatics and Computing (ICIC) (pp. 1-5).IEEE.

[16]. Hiremath, G., Hajare, A., Bhosale, P., Nanaware, R., & Wagh, K. S. (2018). Chatbot for education system. International Journal of Advance Research, Ideas and Innovations in Technology, 4(3), 37-43.

[17]. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.

[18]. Juhnke, C., & Muehlbacher, A. C. (2013). Patientcentredness in integrated healthcare delivery systemsneeds, expectations and priorities for organised healthcare systems. International Journal of İntegrated Care, 13.

[19]. Kane, D. A. (2016). The role of chatbots in teaching and learning. E-Learning and the Academic Library: Essays on İnnovative İnitiatives (pp. 131-156).

[20]. Kilic, S. (2016). Cronbach's alpha reliability coefficient. Psychiatry and Behavioral Sciences, 6(1), 47-48.

[21]. Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling. Guilford publications.

[22]. Lin, M. P., & Chang, D. (2020). Enhancing postsecondary writers' writing skills with a chatbot: A mixedmethod classroom study. Journal of Educational Technology & Society, 23(1), 78-92.

[23]. Nilsson, J. (2018). Artificial İntelligence: A History And Future. Boğaziçi University Press, Istanbul.

[24]. Özdamar, K. (2004). Construction of tables, reliability and analysis of questions. Statistical Data Analysis With Package Programs-1.

[25]. Özgüven, İ. E. (1994). Psychological Tests. Yeni Doğuş Matbaası, Ankara.

[26]. Ren, R., Castro, J. W., Acuña, S. T., & de Lara, J. (2019). Usability of chatbots: A systematic mapping study. In A. Perkusich (Ed.), Proceedings of the 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019 (pp. 479-617).

[27]. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.

[28]. Shum, H. Y., He, X. D., & Li, D. (2018). From Eliza to XiaoIce: Challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19, 10-26.

[29]. Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151, 103862.

[30]. Sumer, N. (2000). Structural equation models: Basic concepts and sample applications. Turkish Psychology Articles, 3(6), 49-74.

[31]. Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2013). Using Multivariate Statistics. Pearson, Boston (pp.497-516).

[32]. Trapero, H., Ilao, J., & Lacaza, R. (2020, November). An integrated theory for chatbot use in air travel: questionnaire development and validation. In 2020 IEEE Regıon 10 Conference (TENCON) (pp. 652-657). IEEE.

[33]. Valério, F. A., Guimarães, T. G., Prates, R. O., & Candello, H. (2017, October). Here's what I can do: Chatbots' strategies to convey their features to users. In Proceedings of the Xvi Brazilian Symposium on Human Factors in Computing Systems (pp. 1-10).

[34]. Wartman, S. A., & Combs, C. D. (2018). Medical education must move from the information age to the age of artificial intelligence. Academic Medicine, 93(8), 1107-1109.

[35]. Weichbroth, P. (2018, September). Usability attributes revisited: a time-framed knowledge map. In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS) (pp. 1005-1008). IEEE.

[36]. Wu, E. H. K., Lin, C. H., Ou, Y. Y., Liu, C. Z., Wang, W. K., & Chao, C. Y. (2020). Advantages and constraints of a hybrid model K-12 E-Learning assistant chatbot. IEEE Access, 8, 77788-77801.

If you have access to this article please login to view the article or kindly login to purchase the article

Single Article

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Online	15	15

ADD TO CART

Options for accessing this content:

If you would like institutional access to this content, please recommend the title to your librarian.
If you already have i-manager's user account: Login above and proceed to purchase the article.
New Users: Please register, then proceed to purchase the article.