The Content Based Medical Image Retrieval (CBMIR) system helps to fulfill the need to diagnose the disease and provide proper treatment. The numerous complex deep neural networks are being applied to solve the same problem. This paper aims to design a simpler and accurate deep neural network-based CBMIR system. To achieve the same, the original Xception model is compressed by 58.33% by reducing the number of layers. The features of intermediate layers are fused to increase the accuracy of the proposed modified Xception model. This modified Multi-Feature Fused Xception (MFF-Xception) model is tested for two different X-ray image datasets: Image Retrieval in Medical Application (IRMA) dataset and the human’s chest dataset. It gives an Average Precision Rate (APR) of 90.14% for the IRMA dataset while providing 100% APR for the human chest dataset. It is also tested for another dataset containing skin images of seven different diseases and gives an APR of 100% for the skin dataset. The proposed modified MFF-Xception model also shows significant performance improvement over the original Xception model for all three datasets.