i-manager Publications

A Comprehensive Survey on AI-Powered Desktop Automation (COSMO)

Amanpreet Kaur Bamrah*, Roshani Talmale**, Ishika Jaiswal***, Jyotleenkaur Saggu****, Tejaswini Patle*****

*-***** Department of Computer Science and Engineering, S. B. Jain Institute of Technology, Management and Research, Nagpur, India.

Periodicity:July - December'2025
DOI : https://doi.org/10.26634/javr.3.2.22788

Abstract

This paper presents an overview of AI-powered desktop automation and its evolution toward intelligent assistants capable of performing real-time tasks using natural language interaction. COSMO (Conversational Smart Machine Operator) is a lightweight desktop assistant designed to execute system level operations, manage files, interact with external APIs, and enhance user productivity. We review existing intelligent assistants, RPA tools, and NLP-driven automation systems, identify persistent challenges (e.g., UI variability, noise, dependence on cloud APIs), and motivate COSMO’s design choices that favor local execution, privacy, and modularity.

Keywords

Desktop Automation, Natural Language Processing, Speech Recognition, Virtual Assistant, GUI Automation, COSMO

How to Cite this Article?

Talmale, R., Jaiswal, I., Bamrah, A., Saggu, J., and Patle, T. (2025). A Comprehensive Survey on AI-Powered Desktop Automation (COSMO). i-manager’s Journal on Augmented & Virtual Reality, 3(2), 38-46. https://doi.org/10.26634/javr.3.2.22788

References

[1]. Chatterjee, M., & Namin, A. S. (2019). Detecting phishing websites through deep reinforcement learning. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC) (Vol. 2, pp. 227-232). IEEE.

[2]. De Barcelos Silva, A., Gomes, M. M., Da Costa, C. A., da Rosa Righi, R., Barbosa, J. L. V., Pessin, G., & Federizzi, G. (2020). Intelligent personal assistants: A systematic literature review. Expert Systems with Applications, 147, 113193.

[3]. Gunaratna, K., Srinivasan, V., & Jin, H. (2023). Explainable and Accurate Natural Language Understanding for Voice Assistants and Beyond. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (pp. 3913-3917).

[4]. Korkmaz, M., Sahingoz, O. K., & Diri, B. (2020). Detection of phishing websites by using machine learning-based URL analysis. In 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT) (pp. 1-7). IEEE.

[5]. Lalwani, T., Bhalotia, S., Pal, A., Rathod, V., & Bisen, S. (2018). Implementation of a Chatbot System using AI and NLP. International Journal of Innovative Research in Computer Science & Technology (IJIRCST), 6(3).

[6]. López-Espejo, I., Tan, Z. H., Hansen, J. H., & Jensen, J. (2021). Deep spoken keyword spotting: An overview. IEEE Access, 10, 4169-4199.

[7]. Mulik, S., Sawant, P., & Bhosale, V. (2021). Application of NLP: Design of chatbot for new research scholars. Turkish Online Journal of Qualitative Inquiry, 12(8), 2817–2823.

[8]. Nayak, S., Jian, X., Lin, K. Q., Rodriguez, J. A., Kalsi, M., Awal, R., & Rajeswar, S. (2025). Ui-vision: A desktop-centric gui benchmark for visual perception and interaction. arXiv preprint arXiv:2503.15661.

[9]. Nayeem, M., Tabrej, M. S., Deb, K. J., Goswami, S., & Hakim, M. A. (2025). Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluation. arXiv preprint arXiv:2510.12827.

[10]. Sahai, S. Y., Liu, J., Muniyappa, T., Sathyendra, K. M., Alexandridis, A., Strimel, G. P., & Kunzmann, S. (2023). Dual-attention neural transducers for efficient wake word spotting in speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

[11]. Sermet, Y., & Demir, I. (2021). A semantic web framework for automated smart assistants: A case study for public health. Big Data and Cognitive Computing, 5(4), 57.

[12]. Soltanian, M., Malik, J., Raitoharju, J., Iosifidis, A., Kiranyaz, S., & Gabbouj, M. (2021). Speech command recognition in computationally constrained environments with a quadratic self-organized operational layer. In 2021 International Joint Conference on Neural Networks (IJCNN) (pp. 1-6). IEEE.

[13]. Tang, F., Xu, H., Zhang, H., Chen, S., Wu, X., Shen, Y., & Zhuang, Y. (2025). A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865.

[14]. Vilas, M. M., Ghansham, K. P., Jaypralash, S. P., & Shila, P. (2019). Detection of phishing website using machine learning approach. In 2019 4th International Conference on Electrical, Electronics, Communication, Computer Technologies and Optimization Techniques (ICEECCOT) (pp. 384-389). IEEE.

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Pdf	40	40	300
Online	15	15	300
Pdf & Online	40	40	300

A Comprehensive Survey on AI-Powered Desktop Automation (COSMO)

Abstract

Keywords

How to Cite this Article?

References

If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Options for accessing this content: