Breast cancer continues to be the common cancer which causes more decease among women and about more than two million cases is identified every year and according to the record 523,000 deaths are caused per year due to breast cancer. Mammographic mass identification and segmentation are accomplished usually as sequential and distinct tasks, where in previous studies segmentation was often manually performed only on true positive cases. Machine learning (ML) approaches have been grown from manually provided inputs to systematic initializations. The developments in ML techniques have produced independent and more intelligent computer-aided diagnosis (CAD) systems. Moreover, due to the learning ability, ML techniques have been upgraded constantly. Recently, ML techniques are progressive with deeper and varied representation methods, generally termed as deep learning (DL) techniques, and have produced significant impacts on increasing the ability of diagnosing using CAD systems. So this paper proposes the novel architecture in detecting the breast cancer. Here the input database is mammogram image dataset. Initially this image has been resized by pre-processing process, then this pre-processed image has been segmented using Mask RCNN (Region-based convolution neural networks). Then this segmented image has been sent for extracting the feature using inception V3 and ResNet 152 networks. The ensemble classifier of decision tree and random forest has been used for classification and this classified output is detected based on the color variation and tumor size gives the enhanced accuracy in detecting the cancer.