MoLMamba: A Large State-Space-based Foundation Model for Chemistry

Abstract

Chemical foundation models (FMs) have emerged as potent catalysts for scientific discovery, leveraging extensive pretraining on vast unlabeled datasets. Typically built upon sequence architectures like Transformers, these models excel in processing a wide array of inputs, ranging from SMILES notations to 3D images. In this research, we present MoLMamba, a novel foundation model pretrained on a dataset encompassing 91 million SMILES (equivalent to 4.3 billion tokens) extracted from PubChem and carefully curated. Diverging from conventional Transformer architectures, MoLMamba adopts a state-space approach, offering advantages such as accelerated inference speeds and linear scalability with variable sequence length. Even when confronted with sequences spanning billions of tokens, MoLMamba maintains robust performance. Evaluation on the MoleculeNet benchmark dataset underscores MoLMamba's capabilities across diverse tasks and domains. Its performance is compared with the current state-of-the-art methodologies, affirming its efficacy in chemical machine learning applications. MoLMamba's introduction marks a significant advancement in the field, offering promising avenues for further exploration and application in real-world chemical contexts.