Architecture Optimization Model for the Deep Neural Network for Binary Classification Problems

Efosa Charles Igodan*, Kingsley Chiwuike Ukaoha**
*,** Department of Computer Science, University of Benin, Nigeria.
Periodicity:October - December'2019
DOI : https://doi.org/10.26634/jse.14.2.17162

Abstract

The daunting and challenging tasks of specifying the optimal network architecture and its parameters are a major area of research in the field of Machine Learning (ML) to date. Although these tasks determine the success of building and training an effective and accurate model are yet to be considered on a deep network having three hidden layers with varying optimized parameters to the best of our knowledge. This is due to expert's opinion that it is practically difficult to determine a good Multilayer Perceptron (MLP) topology with more than two or three hidden layers, without considering the number of samples and complexity of the classification to be learned. In this study, a novel approach is proposed that combining an evolutionary genetic algorithm and an optimization algorithm, and a supervised Deep Neural Network (Deep-NN) using alternative activation functions with the view of modeling the prediction for the admission of prospective university students. The genetic algorithm is used to select the optimal network parameters for the Deep-NN. Thus, this study presents a novel methodology that is effective, automatic and less human-dependent in finding optimal solution to diverse binary classification benchmarks. The model is trained, validated and tested using various performance metrics to measure the generalization ability and its performance.

Keywords

Machine Learning, Deep-NN, Multilayer Perceptron, Artificial Neural Network, Evolutionary Algorithm.

How to Cite this Article?

Igodan, E. C., & Ukaoha, K. C. (2019). Architecture Optimization Model for the Deep Neural Network for Binary Classification Problems. i-manager's Journal on Software Engineering, 14(2), 18-31. https://doi.org/10.26634/jse.14.2.17162

References

[1]. Abdalla, O. A., Elfaki, A. O., & AlMurtadha, Y. M. (2014). Optimizing the multilayer feed-forward artificial neural networks architecture and training parameters using genetic algorithm. International Journal of Computer Applications, 96(10), 42-48.
[2]. Aghazadeh, M., & Gharehchopogh, F. S. (2018). A New Hybrid model of Multi-layer Perceptron Artificial Neural Network and Genetic Algorithms in Web Design Management Based on CMS. Journal of AI and Data Mining, 6(2), 409-415. https://doi.org/10.22044/ja dm.2017.989
[3]. Arifovic, J., & Gencay, R. (2001). Using genetic algorithms to select architecture of a feed forward artificial neural network. Physica A: Statistical Mechanics and its Applications, 289 (3-4), 574-594. https://doi.org/ 10.1016/S0378-4371(00)00479-9
[4]. Ashwood, A. J. (2014). Portfolio selection using artificial intelligence (Doctoral dissertation). Queensland University of Technology, Australia. Retrieved from http://eprints.qut.edu.au/66229/
[5]. Bahnsen, A. C., & Gonzalez, A. M. (2011, December). Evolutionary algorithms for selecting the architecture of a MLP neural network: A credit scoring case. In 2011, IEEE 11th International Conference on Data Mining Workshops (pp. 725-732). IEEE. https://doi.org/10.1 109/ICDMW.2011 .80
[6]. Balochian, S., Seidabad, E. A., & Rad, S. Z. (2013). Neural network optimization by genetic algorithms for the audio classification to speech and music. International Journal of Signal Processing, Image Processing and Pattern Recognition, 6(3), 47-54.
[7]. Batchis, P. (2013). An Evolutionary Algorithm for Neural Network Learning Using Direct Encoding. Resource 53, Chinese Digital Library. Retrieved from https://www.cs.rutgers.edu/~mlittman/courses/ml03/iCM L03/papers/batchis.pdf
[8]. Brownlee, J. (2016). Deep Learning with Python: Develop Deep Learning Models on Theano and Tensorflow using Keras. Machine Learning Mastery.
[9]. Carvalho, A. R., Ramos, F. M., & Chaves, A. A. (2011). Metaheuristics for the feed forward artificial neural network (ANN) architecture optimization problem. Neural Computing and Applications, 20(8), 1273-1284. https://doi.org/10.1007/s00521-010-0504-3
[10]. Castillo, P. A., Arenas, M. G., Castellano, J. G., Cillero, M., Merelo, J. J., Prieto, A., Rivas, V., & Romero, G. (2001). Function approximation with evolved multilayer perceptrons. In Proceedings of International Workshop on Evolutionary Computation (IWEC'2000)(pp. 209-224).
[11]. Castillo, P. A., Carpio, J., Merelo, J. J., Prieto, A., Rivas, V., & Romero, G. (2000). Evolving multilayer perceptrons. Neural Processing Letters, 12(2), 115-128. https://doi.org/10.1023/A:1009684907680
[12].Chiroma, H., Abdulkareem, S., Abubakar, A., & Herawan, T. (2017). Neural networks optimization through genetic algorithm searches: A review. Applied Mathematics & Information Sciences, 11(6), 1543-1564. https://doi.org/10.18576/amis%2F110602
[13]. Chow, T. T., Lin, Z., Song, C. L., & Zhang, G. Q. (2001). Applying neural network and genetic algorithm in chiller system optimization. In Proceedings of the Seventh International Building Performance Simulation Association, Rio de Janeiro, Brazil, 1059-1066.
[14]. Dorofki, M., Elshafie, A.H., Jaafar, O., Karim, O.A., & Mastura, S. (2012).Comparison of artificial neural network transfer functions abilities to simulate extreme runoff data. International Conference on Environment, Energy and Biotechnology IPCBEE (vol.33, pp.39-44).
[15]. Eng, M. H., Li, Y., Wang, Q. G., & Lee, T. H. (2008, December). Forecast forex with ANN using fundamental data. In 2008 International Conference on Information Management, Innovation Management and Industrial Engineering (Vol. 1, pp. 279-282). IEEE. https://doi.org/ 10.1109/ICIII.2008.302
[16]. Ettaouil, M., Lazaar, M., & Ghanou, Y. (2013). Architecture optimization model for the multilayer perceptron and clustering. Journal of Theoretical & Applied Information Technology, 47(1), 64-72.
[17]. Fakharudin, A. S., Sulaiman, M. N., Salihon, J., & Zainol, N. (2013, August). Implementing artificial neural networks and genetic algorithms to solve modeling and optimization of biogas production. In Proceedings of the 4th International Conference on Computing and Informatics (ICOCI) (Vol. 30,pp.121-126).
[18]. Ferentinos, K. P. (2005). Biological engineering applications of feedforward neural networks designed and parameterized by genetic algorithms. Neural Networks, 18(7), 934-950. https://doi.org/10.1016/ j.neunet.2005.03.010
[19]. Fischer, M. M., & Leung, Y. (1998). A geneticalgorithms based evolutionary computational neural network for modelling spatial interaction data neural network for modelling spatial interaction data. The Annals of Regional Science, 32(3), 437-458. https://doi.org/10.1007/s001680050082
[20]. Ganatra, A., Kosta, Y. P., Panchal, G., & Gajjar, C. (2011). Initial classification through back propagation in a neural network following optimization through GA to evaluate the fitness of an algorithm. International Journal of Computer Science and Information Technology, 3(1), 98-116.
[21]. Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems (2nd Ed). O'Reilly Media.
[22]. Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. United States : Addison-Wesley Longman Publishing Co., Inc.
[23]. Gorunescu, F. (2011). Data Mining: Concepts, Models and Techniques (Vol. 12). Springer Science & Business Media.
[24]. Güler, İ., Polat, H., & Ergün, U. (2005). Combining neural network and genetic algorithm for prediction of lung sounds. Journal of Medical Systems, 29(3), 217-231. https://doi.org/10.1007/s10916-005-5182-9
[25]. Hassan, A. K. A., & Jasim, S. S. (2010). Integrating neural network with genetic algorithms for the classification plant disease. Engineering and Technology Journal, 28(4), 686-702.
[26]. Idrissi, M. A. J., Ramchoun, H., Ghanou, Y., & Ettaouil, M. (2016, May). Genetic algorithm for neural network architecture optimization. In 2016, 3rd International Conference on Logistics Operations Management (GOL) (pp. 1-4). IEEE. https://doi.org/10.1109/GOL.2016.773 1699
[27]. Igodan, C. E. (2019).Optimization of a feed-forward neural network topologies and parameters. (Unpublished predoctoral thesis).
[28]. Jayaraj, V., & Raman, N. S. (2016). A genetic algorithm optimized multilayer perceptron for software defect prediction. International Journal of Advanced Technology in Engineering and Science, 4(2), 132-141.
[29]. Karsoliya, S. (2012). Approximating number of hidden layer neurons in multiple hidden layer BPNN architecture. International Journal of Engineering Trends and Technology, 3(6), 714-717.
[30]. Kokko, T. (2013). Neural networks for computationally expensive problems (Postgraduate thesis), Department of Mathematical Information Technology, University of Jyväskylä, Finland
[31]. Kumari, V. S. R., & Rajesh, K. P. (2015). Optimization of multi-layer perceptron neural network using genetic algorithm for arrhythmia classification. Communications, 3(5), 150-157. https://doi.org/10.11648/j.com.201503 05.21
[32]. Ludermir, T. B., Yamazaki, A., & Zanchettin, C. (2006). An optimization methodology for neural network weights and architectures. IEEE Transactions on Neural Networks, 17(6), 1452-1459. https://doi.org/10.1109/TNN.2006.8 81047
[33]. Manavazhahan, M. (2017). A study of activation functions for neural networks (Computer Science and Computer Engineering Undergraduate Honours Thesis), University of Arkansas, Fayetteville.
[34]. Negnevitsky, M. (2005). Artificial Intelligence: A Guide to Intelligent Systems. Pearson Education.
[35]. Nienhold, D., Schwab, K., Dornberger, R., & Hanne, T. (2015, December). Effects of weight initialization in a feedforward neural network for classification using a modified genetic algorithm. In 2015, 3rd International Symposium on Computational and Business Intelligence (ISCBI) (pp. 6-12). IEEE. https://doi.org/10.1109/ISCBI.20 15.9
[36]. Panchal, G., Ganatra, A., Kosta, Y. P., & Panchal, D. (2011). Behaviour analysis of multilayer perceptrons with multiple hidden neurons and hidden layers. International Journal of Computer Theory and Engineering, 3(2), 332- 337.
[37]. Pater, L. (2016). Application of artificial neural networks and genetic algorithms for crude fractional distillation process modeling. Neural and Evolutionary Computing, 1-9.
[38]. Plagianakos, V. P., Magoulas, G. D., & Vrahatis, M. N. (2006). Evolutionary training of hardware realizable multilayer perceptrons. Neural Computing & Applications, 15(1), 33-40. https://doi.org/10.1007/s005 21-005-0005-y
[39]. Rahman, M. M., & Setu, T. A. (2015). An implementation for combining neural networks and genetic algorithms. International Journal of Computer Science and Telecommunications (IJCST), 6(3), 218-222.
[40]. Rao, R. S., & Gupta, M. (2014). 9 Design pattern detection by multilayer neural genetic algorithm, International Journal of Computer Science and Network, 3(1), 9-14. https://doi.org/10.1.1.466.1544
[41]. Sagar, G. V. R., & Chalam, D. S. V. (2011). Evolutionary algorithm for connection weights in artificial neural networks. International Journal of Electronics and Communication Engineering, 4(5), 517-525.
[42]. Senhaji, K., Ramchoun, H., & Ettaouil, M. (2017, October). Multilayer perceptron: NSGA II for a new multiobjective learning method for training and model complexity. In First International Conference on Real Time Intelligent Systems (pp. 154-167). Springer, Cham. https://doi.org/10.1007/978-3-319-91337-7_15
[43]. Sewsynker-Sukai, Y., Faloye, F., & Kana, E. B. G. (2017). Artificial neural networks: an efficient tool for modelling and optimization of biofuel production (a mini review). Biotechnology & Biotechnological Equipment, 31(2), 221- 235. https://doi.org/10.1080/13102818. 2016.1269616
[44]. Sivanandam, S. N., & Deepa, S. N. (2006). Introduction to Neural Networks using MATLAB 6.0. Tata McGraw-Hill Education.
[45]. Svozil, D., Kvasnicka, V., & Pospichal, J. (1997). Introduction to multi-layer feed-forward neural networks. Chemometrics and Intelligent Laboratory Systems, 39(1), 43-62. https://doi.org/10.1016/S0169-7439(97)00061-0
[46]. Tan, P-N., Steinbach, M., & Kumar, V. (2006). Introduction to Data Mining. Pearson Addison-Wesley. Retrieved from https://www-users.cs.umn.edu/~kumar 001/dmbook/sol.pdf
[47]. Taşkıran, M., Çam, Z. G., & Kahraman, N. (2015). An efficient method to optimize multi-layer perceptron for classification of human activities. International Journal of Computing, Communications & Instrumentation Engineering (IJCCIE), 2(2), 191-195. https://doi.org/ 10.15242/IJCCIE.ER1215104
[48]. Thomas, A. J., Petridis, M., Walters, S. D., Gheytassi, S. M., & Morgan, R. E. (2015, December). On predicting the optimal number of hidden nodes. In 2015 International Conference on Computational Science and Computational Intelligence (CSCI) (pp. 565-570). IEEE. https://doi.org/10.1109/CSCI.2015.33
[49]. Urban, S. (2017). Neural network architectures and activation functions: A gaussian process approach. Thesis.
[50]. Zhou, J., & Li, L. (2004, September). Using genetic algorithm trained perceptrons with adaptive structure for the detection of premature ventricular contraction. In Computers in Cardiology (pp. 353-356). IEEE. https://doi.org/10.1109/CIC.2004.1442945
If you have access to this article please login to view the article or kindly login to purchase the article

Purchase Instant Access

Single Article

North Americas,UK,
Middle East,Europe
India Rest of world
USD EUR INR USD-ROW
Online 15 15

Options for accessing this content:
  • If you would like institutional access to this content, please recommend the title to your librarian.
    Library Recommendation Form
  • If you already have i-manager's user account: Login above and proceed to purchase the article.
  • New Users: Please register, then proceed to purchase the article.