i-manager Publications

Performance Analysis of Convolutional Neural Networks for Image Classification with Appropriate Optimizers

Sana Danish*, Jamshaid Ul Rahman**, Gulfam Haider***

* Abdus Salam School of Mathematical Sciences, GC University, Lahore, Pakistan.

** School of Mathematical Sciences, Jiangsu University 301 Xuefu road, Zhenjiang, China.

*** FAST University of Computer and Emerging Sciences (NUCES), Chiniot Faisalabad (CFD), Campus, Pakistan.

Periodicity:January - June'2023
DOI : https://doi.org/10.26634/jmat.12.1.19398

Abstract

Optimizers in Convolutional Neural Networks play an important role in many advanced deep learning models. Studies on advanced optimizers and modifications of existing optimizers continue to hold significant importance in the study of machine tools and algorithms. There are a number of studies to defend and the selection of these optimizers illustrate some of the challenges on the effectiveness of these optimizers. Comprehensive analysis on the optimizers and alteration with famous activation function Rectified Linear Unit (ReLU) offered to protect effectiveness. Significance is determined based on the adjustment with the original Softmax and ReLU. Experiments were performed with Adam, Root Mean Squared Propagation (RMSprop), Adaptive Learning Rate Method (Adadelta), Adaptive Gradient Algorithm (Adagrad) and Stochastic Gradient Descent (SGD) to examine the performance of Convolutional Neural Networks for image classification using the Canadian Institute for Advanced Research dataset (CIFAR-10).

Keywords

Adam, Convolutional Neural Network, Optimizers, RMSprop, Stochastic Gradient Descent.

How to Cite this Article?

Danish, S., Rahman, J. U., and Haider, G. (2023). Performance Analysis of Convolutional Neural Networks for Image Classification with Appropriate Optimizers. i-manager’s Journal on Mathematics, 12(1), 1-8. https://doi.org/10.26634/jmat.12.1.19398

References

[1]. Agostinelli, F., Hoffman, M., Sadowski, P., & Baldi, P. (2014). Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830.

[2]. Ahlawat, S., Choudhary, A., Nayyar, A., Singh, S., & Yoon, B. (2020). Improved handwritten digit recognition using convolutional neural networks (CNN). Sensors, 20(12), 3344.

[3]. Babu, D. V., Karthikeyan, C., & Kumar, A. (2020, December). Performance analysis of cost and accuracy for whale swarm and RMSprop optimizer. In IOP Conference Series: Materials Science and Engineering, 993(1), 012080. IOP Publishing.

[4]. Bashar, A. (2019). Survey on evolving deep learning neural network architectures. Journal of Artificial Intelligence, 1(2), 73-82.

[5]. Bottou, L. (2012). Stochastic gradient descent tricks. In G. Montavon, G. B. Orr, & K. R. Müller (Eds), Neural Networks: Tricks of the Trade (pp. 421-436).

[6]. Cui, X., Zhang, W., Tüske, Z., & Picheny, M. (2018). Evolutionary stochastic gradient descent for optimization of deep neural networks. Advances in Neural Information Processing Systems, 31.

[7]. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., ... & Chen, T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354-377.

[8]. Hayou, S., Doucet, A., & Rousseau, J. (2019, May). On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning (pp. 2672-2680). PMLR.

[9]. Jais, I. K. M., Ismail, A. R., & Nisa, S. Q. (2019). Adam optimization algorithm for wide and deep neural network. Knowledge Engineering and Data Science, 2(1), 41-46.

[10]. Ketkar, N., & Santana, E. (2017). Deep Learning with Python (Vol. 1). Springer, Berkeley, CA.

[11]. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

[12]. Lei, X., Pan, H., & Huang, X. (2019). A dilated CNN model for image classification. IEEE Access, 7, 124087-124095.

[13]. Li, H., Liu, H., Ji, X., Li, G., & Shi, L. (2017). Cifar10-dvs: an event-stream dataset for object classification. Frontiers in Neuroscience, 11, 309.

[14]. Li, Y. (2022, January). Research and application of deep learning in image recognition. In 2022 IEEE 2nd International Conference on Power, Electronics and Computer Applications (ICPECA) (pp. 994-999). IEEE.

[15]. Li, Y., & Yuan, Y. (2017). Convergence analysis of two-layer neural networks with relu activation. Advances in Neural Information Processing Systems, 30.

[16]. Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2021). A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, 33(12), 6999-7019.

[17]. Lydia, A., & Francis, S. (2019). Adagrad-an optimizer for stochastic gradient descent. International Journal of Information and Computing Science, 6(5), 566-568.

[18]. Mahajan, A., & Chaudhary, S. (2019, June). Categorical image classification based on representational deep network (RESNET). In 2019 3rd International conference on Electronics, Communication and Aerospace Technology (ICECA) (pp. 327-330). IEEE.

[19]. Maharjan, S., Alsadoon, A., Prasad, P. W. C., Al-Dalain, T., & Alsadoon, O. H. (2020). A novel enhanced softmax loss function for brain tumour detection using deep learning. Journal of Neuroscience Methods, 330, 108520.

[20]. Martin-Donas, J. M., Gomez, A. M., Gonzalez, J. A., & Peinado, A. M. (2018). A deep learning loss function based on the perceptual evaluation of the speech quality. IEEE Signal Processing Letters, 25(11), 1680-1684.

[21]. Monga, V., Li, Y., & Eldar, Y. C. (2021). Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2), 18-44.

[22]. Postalcıoğlu, S. (2020). Performance analysis of different optimizers for deep learning-based image recognition. International Journal of Pattern Recognition and Artificial Intelligence, 34(2), 2051003.

[23]. Qu, Z., Yuan, S., Chi, R., Chang, L., & Zhao, L. (2019). Genetic optimization method of pantograph and catenary comprehensive monitor status prediction model based on adadelta deep neural network. IEEE Access, 7, 23210-23221.

[24]. Reddy, R. V. K., Rao, B. S., & Raju, K. P. (2018, June). Handwritten Hindi digits recognition using convolutional neural network with RMSprop optimization. In 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS) (pp. 45-51). IEEE.

[25]. Shinde, P. P., & Shah, S. (2018, August). A review of machine learning and deep learning applications. In 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA) (pp. 1-6). IEEE.

[26]. Sun, S., Cao, Z., Zhu, H., & Zhao, J. (2019). A survey of optimization methods from a machine learning perspective. IEEE Transactions on Cybernetics, 50(8), 3668-3681.

[27]. Traoré, C., & Pauwels, E. (2021). Sequential convergence of AdaGrad algorithm for smooth convex optimization. Operations Research Letters, 49(4), 452-458.

[28]. Ul Rahman, J., Ali, A., Ur Rehman, M., & Kazmi, R. (2020b). A unit softmax with Laplacian smoothing stochastic gradient descent for deep convolutional neural networks. In Intelligent Technologies and Applications: Second International Conference, INTAP 2019 (pp. 162-174). Springer, Singapore.

[29]. Ul Rahman, J., Chen, Q., & Yang, Z. (2020a). Additive parameter for deep face recognition. Communications in Mathematics and Statistics, 8, 203-217.

[30]. Ul Rahman, J., Makhdoom, F., Ali, A., & Danish, S. (2023). Mathematical modeling and simulation of biophysics systems using neural network. International Journal of Modern Physics B, 2450066.

[31]. Van Rossum, G., & Drake Jr, F. L. (1995). Python Tutorial (Vol. 620). Centrum voor Wiskunde en Informatica, Amsterdam, The Netherlands.

[32]. Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational Intelligence and Neuroscience, 2018.

[33]. Xiang, J., Dong, T., Pan, R., & Gao, W. (2020). Clothing attribute recognition based on RCNN framework using L-Softmax loss. IEEE Access, 8, 48299-48313.

[34]. Xu, J., Zhang, Z., Friedman, T., Liang, Y., & Broeck, G. (2018, July). A semantic loss function for deep learning with symbolic knowledge. In International Conference on Machine Learning (pp. 5502-5511). PMLR.

[35]. Zeiler, M. D. (2012). Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Pdf	35	35	200	20
Online	35	35	200	15
Pdf & Online	35	35	400	25

Performance Analysis of Convolutional Neural Networks for Image Classification with Appropriate Optimizers

Abstract

Keywords

How to Cite this Article?

References

If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Options for accessing this content: