Disease diagnosis using tongue color image is a traditional non-invasive method widely employed to determine the status of the patient’s internal organ. The elimination of dependencies on subjective and expert knowledge assessment for tongue diagnosis might considerably raise the scope of wide utilization of tongue diagnosis over the globe, including Western medicine. Computer based tongue diagnosis connected to light estimation, color correction, tongue segmentation, image analysis, geometry analysis, etc is a proficient method to diagnose diseases. This paper introduces a new Deep Learning with Depthwise Separable Convolution (Xception) Model called DLXM for tongue color image analysis. The presented model involves data augmentation and bilateral filtering (BF) based noise removal at the preprocessing stage. Additionally, the DLXM is applied for feature extraction process. At last, the bagging classifier (BC)and multilayer perceptron classifier (MLPC) models are employed to categorize the feature vectors into distinct types of diseases. The performance of the presented model is evaluated against benchmark tongue image dataset and the results depicted the effectual classification performance on the applied images. The experimental values notified that the DLXM-MLPC model has outperformed the compared methods by achieving a higher precision, recall, accuracy, and F1-Score of 97.35%, 97.01%, 97.01%, and 96.77% respectively.