Women with locally advanced breast cancer are generally given neoadjuvant chemotherapy (NAC), in which chemotherapy and optionally targeted treatment is administered prior to the surgery. In current clinical practice, prior to the start of NAC, it is not possible to accurately predict whether the patient is likely to encounter metastasis after treatment. Metastasis (or distant recurrence) is the development of secondary malignant growths at a distance from a primary site of cancer. We explore the use of tumor thickness features computed from MRI imaging to predict the risk of post treatment metastasis. We performed a retrospective study on a cohort of 1738 patients who were administered NAC. Of these patients, 551 patients had magnetic resonance imaging (MRI) before the treatment started. We analyzed the multimodal data using deep learning and classical machine learning algorithms to increase the set of discriminating features. Our results demonstrate the ability to predict metastasis prior to the initiation of NAC treatment, using each modality alone. We then show the significant improvement achieved by combining the clinical and MRI modalities, as measured by the AUC, sensitivity, and specificity. The overall combined model achieved 0.747 AUC and 0.379 specificity at a sensitivity operation point of 0.99. We also use interpretability methods to explain the models and identify important clinical features for the early prediction of metastasis.