In year 2020, the World Health Organization (WHO) estimates that 2.3 million women worldwide were diagnosed of breast cancer, which resulted in 685,000 deaths. According to projections, the number of women who have been diagnosed with breast cancer over the last five years before and by the end of 2020 was expected to reach 7.8 million, making it the most common type of cancer worldwide. Early diagnosis could prevent the ailment however, lack of availability of health facilities, cost of accessing treatment especially in developing nations are among the challenges confronting the solution. With the advent of cutting edge technologies, such as artificial intelligence, and machine learning models, access to solution in terms of early diagnosis are becoming possible. Based on literature, Convolutional Neural Networks (CNNs) considering its multiple architectures is promising in bringing the solution to bear. Therefore, this research aims to proposed an architecture of CNNs that gives the best accuracy, F1 score, and Cohen Kappa score among Custom Optimized CNN, ResNet, EfficientNet architectures being considered in this work. From the results based on the datasets, ResNet’s performance across the five metrics outweigh the other two architectures. For example, while ResNet reported an accuracy, precision, and F1 score of 0.9987, 0.9934, and 0.9950 respectively, EfficientNet, which has the second performance reported 0.9977, 0.9914, and 0.9939 as accuracy, precision, and F1 score respectively.