In this paper, we propose a Robust Breast Cancer Diagnostic System (RBCDS) based on multimode Magnetic Resonance (MR) images. Firstly, we design a four-mode convolutional neural network (FMS-PCNN) model to detect whether an image contains a tumor. The features of the images generated by different imaging modes are extracted and fused to form the basis of classification. This classification model utilizes both spatial pyramid pooling (SPP) and principal components analysis (PCA). SPP enables the network to process images of different sizes and avoids the loss due to image resizing. PCA can remove redundant information in the fused features of multi-sequence images. The best accuracy of this model achieves 94.6%. After that, we use our optimized U-Net (SU-Net) to segment the tumor from the entire image. The SU-Net achieves a mean dice coefficient (DC) value of 0.867. Finally, the performance of the system is analyzed to prove that this system is superior to the existing schemes.