Understanding medication events in clinical narratives is essential to achieving a complete picture of a patient's medication history. While prior research has explored identification of medication changes in clinical notes, due to the longitudinal and narrative nature of clinical documentation, extraction of medication change alone without the necessary clinical context is insufficient for use in real-world applications, such as medication timeline generation and medication reconciliation. Here, we present a framework to capture multi-dimensional context of medication changes documented in clinical notes. We define specific contextual aspects pertinent to medication change events (i.e. Action, Negation, Temporality, Certainty, and Actor), describe the annotation process and challenges encountered while creating the dataset, and explore models based on state-of-the-art transformers to automate the task. The resulting dataset, Contextualized Medication Event Dataset (CMED), consisting of 9,013 medications annotated over 500 clinical notes, will be released to the community as a shared task in 2021-2022.