Imperfection classification is the most involved task in the cotton sector for finding Fabric Defects (FDs) and improving fiber productivity. Several approaches have been suggested in ancient times to automatically classify FDs. Presently, an Enhanced Pairwise-Potential Activation Layer in Convolutional Neural Network (EPPAL-CNN) approach depends on improved external memory and Dynamic Conditional Random Fields (DCRFs) to solve the complex pattern correlation of FDs and detect the defective fabrics from the given images. On the contrary, the gradient-based optimization schemes for learning the weights of CNN tend to unusual convergence nature, resulting in inefficient classification. Hence in this paper, an EPPAL-Optimized CNN (EPPAL-OCNN) approach is proposed which introduces an individual weight optimization scheme depending on NWM-Adam for solving the unwanted convergence of CNN. In this approach, a novel first-order gradient descent optimization method is introduced, which applies an adaptive exponential decay percentage for second-moment approximation rather than a preconfigured and constant one. Also, it can simply modify the grade to which how much the previous gradients weigh in the approximation. This novel exponential moving mean deviation is designed based on the fact that assigning additional memory to the previous gradients compared to the current gradients. Thus, it guarantees effective convergence and increases detection accuracy. At last, the testing results reveal that the EPPAL-OCNN achieves 94.64% of accuracy to different state-of-the-art approaches on the TILDA database.