Transfer Learning Based Effective Emotional Face Recognition using DCNN via Cropping Techniques

Anjani Suputri Devi D.*, Suneetha Eluri**
*-** Department of Computer Science and Engineering, Jawaharlal Nehru Technological University, Kakinada, Andhra Pradesh, India.
Periodicity:June - August'2022

Abstract

Facial Expression Recognition (FER) has grown in popularity as a result of the recent advancement and use of humancomputer interface technologies. Because the images can vary in brightness, backdrop, position, etc. it is challenging for current machine learning and deep learning models to identify facial expression. If the database is small, it doesn't operate well. Feature extraction is crucial for FER, and if the derived characteristics can be separated, even a straightforward approach can help tremendously. Deep learning techniques and automated feature extraction, allow some irrelevant features to conflict with important features. In this paper, we deal with limited data and simply extract useful features from images. To make data more numerous and allow for the extraction of just important facial features, we suggest innovative face cropping, rotation, and simplification procedures and advocate using the Transfer Learning technique to construct DCNN for building a very accurate FER system. By replacing the dense top layer(s) with FER, a pretrained DCNN model is adopted, and the model is then modified with facial expression data. The training of the dense layer(s) is followed by adjusting each of the pre-trained DCNN blocks in turn. This new pipeline technique has gradually increased the accuracy of FER to a higher degree. On the CK+ and JAFFE datasets, experiments were run to assess the suggested methodology. For 7-class studies on the CK+ and JAFFE databases, high average accuracy in recognition of 99.49% and 98.58% were acquired.

Keywords

Convolutional Neural Network (CNN), Deep CNN (DCNN), Transfer Learning, CK+, JAFFE.

How to Cite this Article?

Devi, D. A. S., and Eluri, S. (2022). Transfer Learning Based Effective Emotional Face Recognition using DCNN via Cropping Techniques. i-manager’s Journal on Computer Science, 10(2), 8-19.

References

[1]. Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., ...& Asari, V. K. (2019). A state-ofthe- art survey on deep learning theory and architectures. Electronics, 8(3), 292. https://doi.org/10.3390/electronics8030292
[2]. Autism and Developmental Disabilities Monitoring Network Surveillance Year 2010 Principal Investigators. (2014). Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2010. Morbidity and Mortality Weekly Report: Surveillance Summaries, 63(2), 1-21.
[3]. Boutorh, A., & Guessoum, A. (2016). Complex diseases SNP selection and classification by hybrid association rule mining and artificial neural networkbased evolutionary algorithms. Engineering Applications of Artificial Intelligence, 51, 58-70. https://doi.org/10.1016/j.engappai.2016.01.004
[4]. Chang, C. Y., & Huang, Y. C. (2010, July). Personalized facial expression recognition in indoor environments. In the 2010, International Joint Conference on Neural Networks (IJCNN), (pp. 1-8). IEEE. https://doi.org/10.1109/IJCNN.2010.5596316
[5]. Feng, X., Pietikainen, M., & Hadid, A. (2007). Facial expression recognition based on local binary patterns. Pattern Recognition and Image Analysis. 17, 592–598. https://doi.org/10.1134/S1054661807040190
[6]. Fernández-Caballero, A., Martínez-Rodrigo, A., Pastor, J. M., Castillo, J. C., Lozano-Monasor, E., López, M. T., ...& Fernández-Sotos, A. (2016). Smart environment architecture for emotion detection and regulation. Journal of Biomedical Informatics, 64, 55-73. https://doi.org/10.1016/j.jbi.2016.09.015
[7]. Gogić, I., Manhart, M., Pandžić, I. S., &Ahlberg, J. (2020). Fast facial expression recognition using local binary features and shallow neural networks. The Visual Computer, 36(1), 97-112. https://doi.org/10.1007/s00371-018-1585-8
[8]. Goh, K. M., Ng, C. H., Lim, L. L., & Sheikh, U. U. (2020). Micro-expression recognition: an updated review of current trends, challenges and solutions. The Visual Computer, 36(3), 445-468. https://doi.org/10.1007/s00371-018-1607-6
[9]. Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020). A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53(8), 5455-5516. https://doi.org/10.1007/s10462-020-09825-6
[10]. Ko, B. C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18(2), 401. https://doi.org/10.3390/s18020401
[11]. Lee, C. C., Shih, C. Y., Lai, W. P., & Lin, P. C. (2012). An improved boosting algorithm and its application to facial emotion recognition. Journal of Ambient Intelligence and Humanized Computing, 3(1), 11-17. https://doi.org/10.1007/s12652-011-0085-8
[12]. Liew, C. F., & Yairi, T. (2015). Facial expression recognition and analysis: a comparison study of feature descriptors. IPSJ Transactions on Computer Vision and Applications, 7, 104-120. https://doi.org/10.2197/ipsjtcva.7.104
[13]. Lin, C. H. (2016). Novel application of continuously variable transmission system using composite recurrent Laguerre orthogonal polynomials modified PSO NN control system. ISA Transactions, 64, 405-417. https://doi.org/10.1016/j.isatra.2016.05.013
[14]. Liu, M., Li, S., Shan, S., & Chen, X. (2015). Au-inspired deep networks for facial expression feature learning. Neurocomputing, 159, 126-136. https://doi.org/10.1016/j.neucom.2015.02.011
[15]. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010, June). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, (pp. 94-101). IEEE. https://doi.org/10.1109/CVPRW.2010.5543262
[16]. Lyons, M. J., Budynek, J., & Akamatsu, S. (1999). Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12), 1357-1362. https://doi.org/10.1109/34.817413
[17]. Matthews, I., & Baker, S. (2004). Active appearance models revisited. International Journal of Computer Vision, 60(2), 135-164. https://doi.org/10.1023/B:VISI.0000029666.37597.d3
[18]. Mayya, V., Pai, R. M., & Pai, M. M. (2016). Automatic facial expression recognition using DCNN. Procedia Computer Science, 93, 453-461. https://doi.org/10.1016/j.procs.2016.07.233
[19]. Mehrabian, A. (2008). Communication without words. In Communication Theory, (pp. 193-200). Routledge.
[20]. Mohammadi, M. R., Fatemizadeh, E., &Mahoor, M. H. (2014). PCA-based dictionary building for accurate facial expression recognition via sparse representation. Journal of Visual Communication and Image Representation, 25(5), 1082-1092. https://doi.org/10.1016/j.jvcir.2014.03.006
[21]. Mollahosseini, A., Chan, D., &Mahoor, M. H. (2016, March). Going deeper in facial expression recognition using deep neural networks. In 2016, IEEE Winter Conference on Applications of Computer Vision (WACV), (pp. 1-10). IEEE. https://doi.org/10.1109/WACV.2016.7477450
[22]. O'Toole, A. J., Harms, J., Snow, S. L., Hurst, D. R., Pappas, M. R., Ayyad, J. H., & Abdi, H. (2005). A video database of moving faces and people. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 812-816. https://doi.org/10.1109/TPAMI.2005.90
[23]. Sahu, M., & Dash, R. (2021). A survey on deep learning: Convolution neural network (CNN). In Intelligent and Cloud Computing, (pp. 317-325). Springer, Singapore. https://doi.org/10.1007/978-981-15-6202-0_32
[24]. Sánchez, D., Melin, P., & Castillo, O. (2017). Optimization of modular granular neural networks using a firefly algorithm for human recognition. Engineering Applications of Artificial Intelligence, 64, 172-186. https://doi.org/10.1016/j.engappai.2017.06.007
[25]. Shan, C., Gong, S., & McOwan, P. W. (2009). Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 27(6), 803-816. https://doi.org/10.1016/j.imavis.2008.08.005
[26]. Shih, F. Y., Chuang, C. F., & Wang, P. S. (2008). Per formance comparisons of facial expression recognition in JAFFE database. International Journal of Pattern Recognition and Artificial Intelligence, 22(3), 445-459. https://doi.org/10.1142/S0218001408006284
[27]. Sreedharan, N. P. N., Ganesan, B., Raveendran, R., Sarala, P., Dennis, B., &Boothalingam R, R. (2018). Grey wolf optimisation based feature selection and classification for facial emotion recognition. IET Biometrics, 7(5), 490-499. https://doi.org/10.1049/ietbmt.2017.0160
[28]. Suja, P. S., & Tripathi, S. (2016, February). Real-time emotion recognition from facial images using Raspberry Pi II. In 2016, 3rd International Conference on Signal Processing and Integrated Networks (SPIN), (pp. 666-670). IEEE. https://doi.org/10.1109/SPIN.2016.7566780
[29]. Thonse, U., Behere, R. V., Praharaj, S. K., & Sharma, P. S. V. N. (2018). Facial emotion recognition, sociooccupational functioning and expressed emotions in schizophrenia versus bipolar disorder. Psychiatry Research, 264, 354-360. https://doi.org/10.1016/j.psychres.2018.03.027
[30]. Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., & Xun, E. (2017). Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognitive Computation, 9(5), 597-610. https://doi.org/10.1007/s12559-017-9472-6
[31]. Xiao-xu, Q. I., & Wei, J. (2007, April). Application of wavelet energy feature in facial expression recognition. In 2007 International Workshop on Anti-Counterfeiting, Security and Identification (ASID), (pp. 169-174). IEEE. https://doi.org/10.1109/IWASID.2007.373720
[32]. Xie, S., & Hu, H. (2018). Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Transactions on Multimedia, 21(1), 211-220. https://doi.org/10.1109/TMM.2018.2844085
[33]. Yaddaden, Y., Bouzouane, A., Adda, M., & Bouchard, B. (2016, June). A new approach of facial expression recognition for ambient assisted living. In Proceedings of the 9th ACM International Conference on Pervasive Technologies Related to Assistive Environments, (pp. 1-8). https://doi.org/10.1145/2910674.2910703
[34]. Yang, M., Liu, Y., & You, Z. (2017). The Euclidean embedding learning based on convolutional neural network for stereo matching. Neurocomputing, 267, 195-200. https://doi.org/10.1016/j.neucom.2017.06.007
[35]. Zhang, K., Huang, Y., Wu, H., & Wang, L. (2015, November). Facial smile detection based on deep learning features. In 2015, 3rd IAPR Asian Conference on Pattern Recognition (ACPR), (pp. 534-538). IEEE. https://doi.org/10.1109/ACPR.2015.7486560
[36]. Zhao, L., Zhuang, G., & Xu, X. (2008, June). Facial expression recognition based on PCA and NMF. In 2008 7th World Congress on Intelligent Control and Automation, (pp. 6826-6829). IEEE. https://doi.org/10.1109/WCICA.2008.4593968
[37]. Zhao, X., Shi, X., & Zhang, S. (2015). Facial expression recognition via deep learning. IETE Technical Review, 32(5), 347-355. https://doi.org/10.1080/02564602.2015.1017542
[38]. Zhi, R., & Ruan, Q. (2008). Facial expression recognition based on two-dimensional discriminant locality preserving projections. Neurocomputing, 71(7-9), 1730-1734. https://doi.org/10.1016/j.neucom.2007.12.002
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Pdf 35 35 200 20
Online 35 35 200 15
Pdf & Online 35 35 400 25

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.