In 2020, the World Health Organization (WHO) estimated that 2.3 million women worldwide were diagnosed with breast cancer, which resulted in 685,000 deaths. According to the projections, the number of women who have been diagnosed with breast cancer over the last five years before and by the end of 2020 was expected to reach 7.8 million, making it the most common type of cancer worldwide. Early diagnosis could prevent the ailment, however, lack of availability of health facilities and the cost of accessing treatment, especially in developing nations, are among the challenges confronting the solution. With the advent of artificial intelligence and machine learning models, specifically Convolutional Neural Networks (CNNs), considering their multiple architectures is highly promising to address the challenge of early diagnosis. Therefore, this study aims to propose an architecture of CNNs that gives the best accuracy, F1 score, and Cohen Kappa score among the Custom Optimized CNN, ResNet, and EfficientNet architectures. From the results, ResNet's performance across the five metrics outweighs the other two architectures. While ResNet reported an accuracy, precision, and F1 score of 0.9987, 0.9934, and 0.9950, respectively, EfficientNet, which has the second performance, reported 0.9977, 0.9914, and 0.9939 as accuracy, precision, and F1 score, respectively. Therefore, the best-performing architecture can be deployed for other available breast cancer datasets in order to ensure its total efficiency.