Carbon Aware Fine-Tuning of Large-Language-Models for Estimating Scope-3 Emission

Abstract

The GHG Protocol Corporate Standard [1] offers a structured framework for quantifying an enterprise's greenhouse gas (GHG) emissions, categorizing them into three scopes. Scope 1 emissions are direct emissions from organizations, Scope 2 emissions result from generation of purchased energy, while Scope 3 encompasses indirect emission from both upstream and downstream of the value chain. Around 80% of a company's greenhouse gas impacts stem from Scope 3 emissions across their value chain [2]. To develop a comprehensive greenhouse gas emissions inventory, it is crucial for companies to account for Scope 3 emissions in conjunction with their Scope 1 & 2 emissions. This inclusive approach allows companies to grasp their complete environmental impact and strategically target the most significant opportunities for its reduction. While many of the world's largest companies routinely track and report emissions from their direct operations (Scopes 1 & 2), only a few extend their reporting to include Scope 3 emissions, and even then, it is often limited to specific categories. This disparity arises primarily due to the formidable complexity associated with tracking and accounting for Scope 3 emissions [3]. Presently, enterprises face several key challenges in this realm such as lack of cooperation and transparency between stakeholders, difficulty in data collection due to large number of stake holders and lack of personal resources to collect the data [4]. The latest developments in deep learning-based foundation models designed for natural language processing (NLP) have demonstrated superior performance compared to traditional machine learning models in various NLP classification tasks, especially in situations where labelled data is scarce [5, 6]. In the present context, financial transaction text data can be used as a proxy for acquired goods & services, thereby leveraging domain adapted Natural Language Processing (NLP) foundation models to estimate Scope 3 emissions. However, existing approaches in the literature uses the default setting of using equal weight for all the classes for fine-tuning their foundation models [5]. This ignores the fact that classes with higher emission factor introduce larger error in scope-3 emission calculation in an event of a misclassification.