In recent years, language models have disrupted multiple application domains, from natural language to chemistry and material science. Since their inception, they have enabled a revolutionary way to hypothesize the design of novel materials, shown remarkable capabilities in modeling reactivity and successfully adopted in automating chemical synthesis planning. This talk will cover our recent research on applying language models to accelerate scientific discovery in chemistry, from small molecules to polymers and proteins. Our methodologies cover textual representation of molecules, natural language, and hybrid representations, which allow leveraging different data modalities to build holistic foundation models. Besides introducing the methodologies, we will also cover various applications of language models for material design and synthesis. By harnessing the power of language models and the growing availability of datasets, we can transform the discovery process at different stages, paving the way for a revolutionary computer-aided approach to designing, optimizing, and validating novel materials.