i-manager Publications

Speech Driven 3D Face Animation

Kavitha Dhanushkodi *, Vasantha Kumaran A. P. **, Vignesh K. ***, Vimal Raj V. ****

*-**** Department of Computer Science and Engineering, SRM Valliammai Engineering College, Chennai, India.

Periodicity:March - May'2021
DOI : https://doi.org/10.26634/jit.10.2.18167

Abstract

When considering the problem of disabled persons, a new concept known as Speech driven 3D face animation employing deep learning and neural networks has come into use. Nonverbal behaviour cues, such as facial expressions, are still intact and provide information about what we are thinking, doing or reacting to. When it comes to computerbased movies and digital games, expressive face animation is a must-have feature. It is necessary to get audio input from the user, after which the matching characteristics of the audio are extracted. Once the expressions have been analysed, they are combined with the intermediate 3D model to complete the process. The relevant result is generated with the assistance of the neural renderer. An overview of the whole implementation is presented in this paper.

Keywords

Facial Animation, Deep Learning, Neural Networks, 3D Model.

How to Cite this Article?

Dhanushkodi, K., Kumaran, A. P. V., Vignesh, K., and Raj, V. V. (2021). Speech Driven 3D Face Animation. i-manager's Journal on Information Technology, 10(2), 13-21. https://doi.org/10.26634/jit.10.2.18167

References

[1]. Arslan, L. M., & Talkin, D. (1999). Codebook based face point trajectory synthesis algorithm using speech input. Speech Communication, 27(2), 81-93.

[2]. Beskow, J. (1995). Rule-based visual speech synthesis. In EUROSPEECH '95. 4th European Conference on Speech Communication and Technology (pp. 299-302).

[3]. Cao, C., Weng, Y., Lin, S., & Zhou, K. (2013). 3D shape regression for real-time facial animation. ACM Transactions on Graphics (TOG), 32(4), 1-10.

[4]. Cao, X., Wei, Y., Wen, F., & Sun, J. (2012). Face alignment by explicit shape regression. In Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York, 2887−2894.

[5]. Caridakis, G., Raouzaiou, A., Bevacqua, E., Mancini, M., Karpouzis, K., Malatesta, L., & Pelachaud, C. (2007). Virtual agent multimodal mimicry of humans. Language Resources and Evaluation, 41(3), 367-388.

[6]. Chai, J. X., Xiao, J., & Hodgins, J. (2003, July). Visionbased control of 3Dfacial animation. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Aire-la-Ville, Switzerland: Eurographics Association, 193−206.

[7]. Choe, B., Lee, H., & Ko, H. S. (2001). Performance driven muscle based facial animation. The Journal of Visualization and Computer Animation, 12(2), 67-79.

[8]. Chuensaichol, T., Kanongchaiyos, P., & Wutiwi watchai, C. (2011, October). Thai Speech-Driven Facial Animation. In 2011, Second International Conference on Culture and Computing (pp. 121-122). IEEE.

[9]. Gutierrez-Osuna, R., Kakumanu, P. K., Esposito, A., Garcia, O. N., Bojórquez, A., Castillo, J. L., & Rudomín, I. (2005). Speech-driven facial animation with realistic dynamics. IEEE Transactions on Multimedia, 7(1), 33-42.

[10]. Le Gallou, S., Breton, G., Séguier, R., & Garcia, C. (2007, September). Avatar puppetry using real-time audio and video analysis. In International Workshop on Intelligent Virtual Agents (pp. 391-392). Springer, Berlin, Heidelberg.

[11]. Learning 3D face-animation model, (n.d.). whatwhen-how, In Depth Tutorials and Information.

[12]. Li, X., & Zhang, Z. (2008). Morphable linear fitting method for facial expression synthesis. Acta Automatica Sinica, 34(5), 593-597.

[13]. Lixiang, L. (2009, July). A research on facial animation driven by emotional speech. In 2009, 4th International Conference on Computer Science & Education (pp. 118-121). IEEE.

[14]. Melek, Z., & Akarun, L. (2000, July). Automated lip synchronized speech driven facial animation. In 2000, IEEE International Conference on Multimedia and Expo. ICME2000, Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No. 00TH8532) (Vol. 2, pp. 623-626). IEEE.

[15]. Morishima, S., & Harashima, H. (1991). A media conversion from speech to facial image for intelligent man-machine interface. IEEE Journal on Selected Areas in Communications, 9(4), 594-600.

[16]. Pelachaud, C., Badler, N. I., & Steedman, M. (1996). Generating facial expressions for speech. Cognitive Science, 20(1), 1-46.

[17]. Pham, H. X., Cheung, S., & Pavlovic, V. (2017). Speech-driven 3D facial animation with implicit emotional awareness: a deep learning approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 80-88).

[18]. Sadiq, R., & Erzin, E. (2020). Emotion Dependent Domain Adaptation for Speech Driven Affective Facial Feature Synthesis. IEEE Transactions on Affective Computing.

[19]. Sifakis, E., Neverov, I., & Fedkiw, R. (2005). Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics, 24(3), 417−425

[20]. Smart Lab (n.d). RAVDESS, Science of Music, Auditory Research, and Technology, Ryerson University.

[21]. Vlasic, D., Brand, M., Pfister, H., & Popovic, J. (2006). Face transfer withmultilinear models. In Proceedings of the International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH Courses, New York, 24(3), 426−433.

[22]. Waters, K., & Levergood, T. M. (1993). DECface: An automatic lip-synchronization algorithm for synthetic faces. Digital Equipment Corporation, Cambridge Research Laborartory.

[23]. Yamamoto, E., Nakamura, S., & Shikano, K. (1998). Lip movement synthesis from speech based on Hidden Markov Models. Speech Communication, 26(1-2), 105-115.

If you have access to this article please login to view the article or kindly login to purchase the article

Single Article

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Online	15	15

ADD TO CART

Options for accessing this content:

If you would like institutional access to this content, please recommend the title to your librarian.
If you already have i-manager's user account: Login above and proceed to purchase the article.
New Users: Please register, then proceed to purchase the article.