The increasing adoption of electronic emails as a means of communication, both at the commercial, government, and individual levels, serves as an impetus for attackers to compromise communication. Consequently, numerous machine learning techniques have been developed for identifying unwanted emails, commonly known as spam. Despite the significant progress reported in existing literature, most studies do not integrate the detection of both textual and image-based spam. In this paper, two deep learning techniques that detect both textual and image-based spam were evaluated. First, the Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) is studied, training them on various textimage features to explore their effectiveness on an improved dataset. Subsequently, in an effort to outsmart current spam detection techniques, a bi-modal architecture capable of detecting textual spam, image spam, and mixed spam is designed. The experimental results in conjunction with existing transfer learning for effective spam detection is provided.