i-manager Publications

A Comparative Exploration of Activation Functions for Image Classification in Convolutional Neural Networks

Faiza Makhdoom*, Jamshaid Ul Rahman**

* Abdus Salam School of Mathematical Sciences, Government College University, Lahore, Pakistan.

** School of Mathematical Sciences, Jiangsu University, Zhenjiang, China.

Periodicity:January - June'2024
DOI : https://doi.org/10.26634/jaim.2.1.20225

Abstract

Activation functions play a crucial role in enabling neural networks to carry out tasks with increased flexibility by introducing non-linearity. The selection of appropriate activation functions becomes even more crucial, especially in the context of deeper networks where the objective is to learn more intricate patterns. Among various deep learning tools, Convolutional Neural Networks (CNNs) stand out for their exceptional ability to learn complex visual patterns. In practice, ReLu is commonly employed in convolutional layers of CNNs, yet other activation functions like Swish can demonstrate superior training performance while maintaining good testing accuracy on different datasets. This paper presents an optimally refined strategy for deep learning-based image classification tasks by incorporating CNNs with advanced activation functions and an adjustable setting of layers. A thorough analysis has been conducted to support the effectiveness of various activation functions when coupled with the favorable softmax loss, rendering them suitable for ensuring a stable training process. The results obtained on the CIFAR-10 dataset demonstrate the favorability and stability of the adopted strategy throughout the training process.

Keywords

Activation Functions, Image Classification, Convolutional Neural Network,DeepLearning, Machine Intelligence.

How to Cite this Article?

Makhdoom, F., and Rahman, J. U. (2024). A Comparative Exploration of Activation Functions for Image Classification in Convolutional Neural Networks. i-manager’s Journal on Artificial Intelligence & Machine Learning, 2(1), 9-17. https://doi.org/10.26634/jaim.2.1.20225

References

[1]. Agarap, A. F. (2018). Deep learning using rectified linear units (relu). arXiv.

[2]. Alpaydin, E. (2020). Introduction to Machine Learning. MIT press.

[3]. Bonnell, J. A. (2011). Implementation of a New Sigmoid Function in Back propagation Neural Networks (Master thesis, East Tennessee State University).

[4]. Chaithanya, B. N., Swasthika Jain, T. J., Usha Ruby, A., & Parveen, A. (2021). An approach to categorize chest X- ray images using sparse categorical cross entropy. Indonesian Journal of Electrical Engineering and Computer Science, 24(3), 1700-1710.

[5]. Covington, P., Adams, J., & Sargin, E. (2016, September). Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, (pp. 191-198).

[6]. da Silva, I. N., Hernane Spatti, D., Andrade Flauzino, R., Liboni, L. H. B., dos Reis Alves, S. F., da Silva, I. N., & dos Reis Alves, S. F. (2017). Multilayer perceptron networks. Artificial Neural Networks: A Practical Course, (pp. 55-115).

[7]. Danish, S., Rahman, J. U., & Haider, G. (2023). Performance analysis of convolutional neural networks for image classification with appropriate optimizers. i-manager's Journal on Mathematics, 12(1), 1-8.

[8]. De Brebisson, A., & Vincent, P. (2015). An exploration of softmax alternatives belonging to the spherical loss family. arXiv.

[9]. De Ryck, T., Lanthaler, S., & Mishra, S. (2021). On the approximation of functions by tanh neural networks. Neural Networks, 143, 732-750.

[10]. Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019). Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (pp. 4690- 4699).

[11]. Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503, 92-108.

[12]. Eger, S., Youssef, P., & Gurevych, I. (2019). Is it time to swish? Comparing deep learning activation functions across NLP tasks. arXiv.

[13]. Feng, X., Jiang, Y., Yang, X., Du, M., & Li, X. (2019). Computer vision algorithms and hardware implementations: A survey. Integration, 69, 309-320.

[14]. Gulli, A., & Pal, S. (2017). Deep Learning with Keras. Packt Publishing Ltd.

[15]. He, C., Shah, A. D., Tang, Z., Sivashunmugam, D. F. N., Bhogaraju, K., Shimpi, M., & Avestimehr, S. (2021). FedCV: A federated learning framework for diverse computer vision tasks. arXiv.

[16]. Ide, H., & Kurita, T. (2017, May). Improvement of learning for CNN with ReLU activation by sparse regularization. In 2017 International Joint Conference on Neural Networks (IJCNN) (pp. 2684-2691). IEEE.

[17]. Jais, I. K. M., Ismail, A. R., & Nisa, S. Q. (2019). Adam optimization algorithm for wide and deep neural network. Knowledge Engineering and Data Science, 2(1), 41-46.

[18]. Joseph, F. J. J., Nonsiri, S., & Monsakul, A. (2021). Keras and TensorFlow: A hands-on experience. Advanced Deep Learning for Engineers and Scientists: A Practical Approach, (pp.85-111).

[19]. Kalman, B. L., & Kwasny, S. C. (1992, June). Why tanh: Choosing a sigmoidal function. In Proceedings 1992 IJCNN International Joint Conference on Neural Networks 4, 578-581. IEEE.

[20]. Khalifa, A., & Al-Hamadi, A. (2021, September). A survey on loss functions for deep face recognition network. In 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS) (pp. 1-7). IEEE.

[21]. Kouretas, I., & Paliouras, V. (2019, May). Simplified hardware implementation of the softmax activation function. In 2019 8th International Conference on Modern Circuits and Systems Technologies (MOCAST) (pp. 1-4). IEEE.

[22]. Krizhevsky, A., & Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images.

[23]. Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2021). A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, 33(12), 6999-7019.

[24]. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., & Song, L. (2017). Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 212-220).

[25]. Liu, Y. H. (2018, September). Feature extraction and image recognition with convolutional neural networks. In Journal of Physics: Conference Series, 1087, 062032. IOP Publishing.

[26]. Liu, Y., Gao, Y., & Yin, W. (2020). An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 33, 18261-18271.

[27]. Masoudian, S., Arabzadeh, A., Siavoshani, M. J., Jalal, M., & Amouzad, A. (2019). Adaptive online learning for gradient-based optimizers. arXiv.

[28]. Mercioni, M. A., & Holban, S. (2020, November). P- swish: Activation function with learnable parameters based on swish activation function in deep learning. In 2020 International Symposium on Electronics and Telecommunications (ISETC) (pp. 1-4). IEEE.

[29]. Pratiwi, H., Windarto, A. P., Susliansyah, S., Aria, R. R., Susilowati, S., Rahayu, L. K., & Rahadjeng, I. R. (2020, February). Sigmoid activation function in selecting the best model of artificial neural networks. In Journal of Physics: Conference Series, 1471(1), 012010. IOP Publishing.

[30]. Rahman, J. U., Makhdoom, F., & Lu, D. (2023a). Amplifying sine unit: An oscillatory activation function for deep neural networks to recover nonlinear oscillations efficiently. arXiv.

[31]. Rahman, J. U., Makhdoom, F., & Lu, D. (2023b). ASU- CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations. arXiv.

[32]. Rahman, U.l.J., Chen, Q., & Yang, Z. (2020). Additive parameter for deep face recognition. Communications in Mathematics and Statistics, 8, 203-217.

[33]. Ramachandran, P., Zoph, B., & Le, Q. V. (2017). Searching for activation functions. arXiv.

[34]. Ramasubramanian, K., & Singh, A. (2019). Deep learning using keras and tensorflow. Machine Learning Using R: With Time Series and Industry-Based Use Cases in R (pp.667-688).

[35]. Ray, S. (2019, February). A quick review of machine learning algorithms. In 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon) (pp. 35-39). IEEE.

[36]. Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Müller, K. R. (2021). Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3), 247-278.

[37]. Sharma, S., Sharma, S., & Athaiya, A. (2017). Activation functions in neural networks. International Journal of Engineering Applied Sciences and Technology, 4(12), 310-316.

[38]. Tan, M., & Le, Q. (2019, May). Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, (pp. 6105-6114). PMLR.

[39]. Wang, A., Zhang, W., & Wei, X. (2019). A review on weed detection using ground-based machine vision and image processing techniques. Computers and Electronics in Agriculture, 158, 226-240.

[40]. Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., & Liu, W. (2018). Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 5265-5274).

[41]. Wightman, R., Touvron, H., & Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv.

[42]. Wright, A., & Välimäki, V. (2020, May). Perceptual loss function for neural modeling of audio systems. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 251-255). IEEE.

[43]. Zhang, C., & Lu, Y. (2021). Study on artificial intelligence: The state of the art and future prospects. Journal of Industrial Information Integration, 23, 100224.

[44]. Zhang, C., Benz, P., Argaw, D. M., Lee, S., Kim, J., Rameau, F., & Kweon, I. S. (2021). Resnet or densenet? Introducing dense shortcuts to resnet. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, (pp. 3550-3559).

[45]. Zhang, W., Shankar, A., & Antonidoss, A. (2022). Modern art education and teaching based on artificial intelligence. Journal of Interconnection Networks, 22(1), 2141005.

If you have access to this article please login to view the article or kindly login to purchase the article

Single Article

	North Americas,UK, Middle East,Europe		India	Rest of world
	USD	EUR	INR	USD-ROW
Online	15	15

ADD TO CART

Options for accessing this content:

If you would like institutional access to this content, please recommend the title to your librarian.
If you already have i-manager's user account: Login above and proceed to purchase the article.
New Users: Please register, then proceed to purchase the article.