A novel system for detection and classification of masses in breast mammography is introduced. The system integrates a breast segmentation module together with a modified region-based convolutional network to obtain detection and classification of masses according to BI-RADS score. While most of the previous work on mass identification in breast mammography has focused on classification, this study proposes to solve both the detection and the classification problems. The method is evaluated on a large multi-centre clinical data-set and compared to ground truth annotated by expert radiologists. Preliminary experimental results show the high accuracy and efficiency obtained by the suggested network structure. As the volume and complexity of data in health care continues to accelerate generalising such an approach may have a profound impact on patient care in many applications.