Language Detection using Multinomial Naïve Bayes Algorithm

Yashvi Vaghasiya*, Diya Vora**, Nehayadav***, Manish Rana****
*-**** Department of Computer Engineering, Thakur College of Engineering and Technology, Mumbai, Maharashtra, India.
Periodicity:June - August'2022

Abstract

In this multilingual world, automatic detection of written or spoken language using Language Identification (LID) technology is a boon in the global communication with people using different languages in different countries. For simplicity and for the purpose of this research, the process of automatically identifying the language(s) from a document is thought of as LID. Lot of ongoing research projects are in the field of Natural Language Processing (NLP) that uses LID as a part of NLP. This field exploits several algorithms evolved in the field of computer science, individually or in combination to achieve accuracy in identifying a language. Among the different approaches adopted in LID,NaïveBayes Classification n-gram text processing seems to be promising.This paper proposes the concept for categorising multiple language texts using Naïve Bayesian algorithms using Machine Learning approaches. Using techniques from existing researches, this paper proposes a way to recognize multilingual documents and calculate the relative proportions of these languages.

Keywords

Language Identification, N-Gram Model, Multilingual Naïve Bayes Algorithm, Classification, Natural Language Processing (NLP).

How to Cite this Article?

Vaghasiya, Y., Vora, D., Nehayadav, and Rana, M. (2022). Language Detection using Multinomial Naïve Bayes Algorithm. i-manager’s Journal on Computer Science, 10(2), 34-39.

References

[1]. Babu, V. J.,&Baskaran, S. (2005, February). Automatic language identification using multivariate analysis. In International Conference on Intelligent Text Processing and Computational Linguistics, (pp. 789-792). Springer, Berlin, Heidelberg.https://doi.org/10.1007/978-3-540-30586-6_89
[2]. Cavnar, W. B., &Trenkle, J. M. (1994, April). N-grambased text categorization. In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, (Vol. 161175).
[3]. Dunning, T. (1994). Statistical Identification of Language (pp. 940-273). Las Cruces: Computing Research Laboratory, New Mexico State University. Retrieved form https://www.researchgate.net/profile/Ted-Dunning/publication/2263394_Statistical_Identification_ of_Language/links/0deec51cb2675ae546000000/Statistical-Identification-of-Language.pdf
[4]. Gadgil, A., Joshi, S., Katwe, P., & Kshatriya, P. (2018). Automatic language identification using hybrid approach and classification algorithms.International Research Journal of Engineering and Technology, 5(3), 3178-3181.
[5]. Gazeau, V., &Varol, C. (2018). Automatic spoken language recognition with neural networks. I.J. Information Technology and Computer Science, 10(8), 11-17. https://doi.org/10.5815/ijitcs.2018.08.02
[6]. Giguet, E. (1995, September). Categorization according to language: A step toward combining linguistic knowledge and statistic learning. In International Workshop of Parsing Technologies (IWPT'95).
[7]. Gold, E. M. (1967). Language identification in the limit. Information and Control, 10(5), 447-474. https://doi.org/10.1016/S0019-9958(67)91165-5
[8]. Grefenstette, G. (1995, December). Comparing two language identification schemes. In Proceedings of JADT, 95, 263–268.
[9]. Johnson, S. (1993). Solving the Problem of Language Recognition. Technical Report, School of Computer Studies.
[10]. Kanaris, I., Kanaris, K., Houvardas, I., &Stamatatos, E. (2006). Words versus characters n-grams for anti-spam filtering. International Journal on Artificial Intelligence Tools, 16(6), 1047-1067. https://doi.org/10.1142/S0218213007003692
[11]. Lins, R. D., & Gonçalves, P. (2004, March). Automatic language identification of written texts. In Proceedings of the 2004 ACM Symposium on Applied Computing, (pp. 1128-1133). https://doi.org/10.1145/967900.968129
[12]. Malato, G. (2020, October 8). An Efficient Language Detection Model using Naive Bayes; a Simple Language Detection Model in Python, Retrieved from https://towardsdatascience.com/an-efficient-language-detectionmodel-using-naive-bayes-85d02b51cfbd
[13]. Muthusamy, Y. K. (1993). A Segmental Approach to Automatic Language Identification. (Doctoral dissertation). Oregon Graduate Identification of Science and Technology, Portland, Oregon, USA.
[14]. Qafmolla, N. (2017). Automatic language identification. European Journal of Language and Literature, 3(1), 140-150. https://doi.org/10.26417/ejls.v7i1.p140-150
[15]. Rahmoun, A., & Elberrichi, Z. (2007). Experimenting n-grams in text categorization. International Arab Journal of Information Technology, 4(4), 377-385.
[16]. Shanmugam, D. C. (2017, October 30). How Redbus uses Scikit-Learn ML Models to Classify Customer Complaints? Retrieved form https://medium.com/redbus-in/how-to-deploy-scikit-learn-ml-models-d390b4b8ce7a
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Pdf 35 35 200 20
Online 35 35 200 15
Pdf & Online 35 35 400 25

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.