Multiple Style Transfer to Video for a Specific Region of Interest using Deep Learning

P. T. Weerasinghe *, R. Siyambalapitiya **
*-** Department of Statistics & Computer Science, University of Peradeniya, Sri Lanka.
Periodicity:January - March'2021
DOI : https://doi.org/10.26634/jip.8.1.17981

Abstract

With the advancement of Artificial Intelligence and Deep Learning techniques, Neural Style transfer has become a popular methodology used to apply complex styles and effects to images which might have been an impossible task to be done manually a few years back. After using one image to style another successfully, research has been done in order to transfer styles from still images to real time video. The major problem faced in doing this has been, unlike styling still images there were inconsistent “popping”, that is inconsistent stylization from frame to frame and "temporal inconsistency" which is flickering between consecutive stylized frames. Many video style transfer models have been succeeded in improving temporal inconsistency but have failed to guarantee fast processing speed and nice perceptual style quality at the same time. In this study, we look to incorporate multiple styles to a video for a defined specific region of interest. As many models are able to transfer styles to a whole single frame but not to a specific region. For this, a convolutional neural network with the pre-trained VGG-16 model for transfer learning approach has been used. In our results we were able to transfer style to a specific region of interest of the video frame. Also, different tests were conducted to validate the model (pre-trained VGG-16) and it has been shown for which domains it is suitable.

Keywords

Deep Learning, Style Transfer, Transfer Learning, Artificial Intelligence, Popping.

How to Cite this Article?

Weerasinghe, P. T., and Siyambalapitiya, R. (2021). Multiple Style Transfer to Video for a Specific Region of Interest using Deep Learning. i-manager's Journal on Image Processing, 8(1), 1-6. https://doi.org/10.26634/jip.8.1.17981

References

[1]. Chen, D., Yuan, L., Liao, J., Yu, N., & Hua, G. (2017). Stylebank: An explicit representation for neural image style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1897- 1906).
[2]. Cui, B., Qi, C., & Wang, A. (2017). Multi-style transfer: Generalizing fast style transfer to several genres (Report). https://cs231n.stanford.edu/reports/2017/pdfs/401.pdf
[3]. Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). A neural algorithm of artistic style. Journal of Vision, 16(12), 326. https://doi.org/10.1167/16.12.326
[4]. Handa, A., Garg, P., & Khare, V. (2018, July). Masked neural style transfer using convolutional neural networks. In 2018, International Conference on Recent Innovations in Electrical, Electronics & Communication Engineering (ICRIEECE) (pp. 2099-2104). IEEE. https://doi.org/10.1109/ ICRIEECE44171.2018.9008937
[5]. Johnson, J., Alahi, A., & Fei-Fei, L. (2016, October). Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (pp. 694-711). Cham: Springer.
[6]. Koidan, K. (2018). Real-time video styletransfer: Fast, accurate and temporally consistent (Report). https://neuro hive.io/en/state-of-the-art/real-time-video-style-transferfast- accurate-and-temporally-consistent/
[7]. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., & Yang, M. H. (2017). Universal style transfer via feature transforms. Advances in Neural Information Processing Systems, 2017(December), 386-396.
[8]. Marcelino, P. (2018, October 23). Transfer learning from pre-trained models. Towards Data Science [Website]. Retrieved from https://towardsdatascience.com/transferlearning- from-pre-trained-models-f2393f124751
[9]. Mroueh, Y. (2020). Wasserstein Style Transfer. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR (Vol.108, pp. 842-852).
[10]. Ruder, M., Dosovitskiy, A., & Brox, T. (2016, September). Artistic style transfer for videos. In German Conference on Pattern Recognition (pp. 26-36). Cham: Springer. https://doi.org/10.1007/978-3-319-45886-13
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Online 15 15

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.