For ages, human body temperature has been used as an indicator for judging the health status. Over the years, the science of Medical Thermography has evolved to measure body surface temperatures that help in making relevant judgments about diseases. Unusual temperature patterns or the values above permissible limits indicate abnormality. In most of the cases, such temperature differences indicate a high chance of inflammation, infection or malignancy. Localized increase in temperature is termed as Hot spot and is indicative of such abnormalities. This paper develops an algorithm to detect the highest temperature region in breast thermogram to predict the breast disease. Thermal image is captured using an infrared camera. The temperature data are processed to find the hotspot. The location and shape of the hotspot is detected. The thermograms are further segmented in left and right part manually. Statistical parameters are calculated from temperature data of segmented region. Significant differences in these parameters is observed for healthy and sick cases showing asymmetry. The presence of asymmetric hotspot suggests the further follow up. Results are validated by a radiologist confirming the performance of the algorithm.