About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
PLOS 2021
Conference paper
PyMM: Heterogeneous Memory Programming for Python Data Science
Abstract
While persistent memory (PMEM) is a promising technology, leveraging it with legacy applications is non-trivial. This is primarily because legacy applications assume all memory is volatile and there is no notion of crash-consistency or state recovery. As new types of persistent and intelligent memory emerge, propelled by the CXL standard, the problem of integration and adoption remains. In this paper we present PyMM, a framework for heterogeneous memory management in Python. It provides a means to abstract upon different memory types and their underlying traits (e.g., persistence, near/far). PyMM focuses on ease-of-use and employs an approach of sub-classing existing heavily-used types such as NumPy ndarray and PyTorch tensors. By doing so, PyMM allows new memory adoption with only minor modification to the application.