The paper proposes an Improved Gesture Precision Virtual Personal Assistant (IGP-VPA) system for speech impaired people that would act as a personal assistant for them. The proposed system first recognizes the sign language performed by the impeded individuals and converts it to text and speech format. The converted speech format will be given as input to the available assistant system such as Amazon Alexa. Then the VPA system responds to the speech, which would act as a personal assistant for speech impaired people. The previous method, Static and Dynamic Hand Gesture Recognition in Depth Data using Dynamic Time Warping also helps the speech impaired persons with 71.9% of gesture identification precision. To increase the gesture identification precision, IGP-VPA has been proposed. IGP-VPA uses iterative optimization process using Convolutional Neural Network (CNN), which overcomes drawbacks, such as memory intensive storage problem, poor generalization, and performance drops when training with voluminous samples. IGP-VPA is trained with 2262 gestures, each with approximately 1500 image. Indoctrinating with the huge dataset makes a difference to realize the precision of the recognition. The experimental results concluded the average precision in gesture identification as 82.9%, which is 11.78% higher compared to the Static and Dynamic Hand Gesture Recognition in Depth Data using Dynamic Time Warping.