Addressing Data Scarcity and Distribution Shifts in Communication Networks Using Pre-trained Transformers and Transfer Learning

Abstract

Rapid progression of Digital Twins has made them a crucial tool for informed decision-making for urban planning. To accurately replicate the complexities of physical environments, critical infrastructure (e.g. telecommunication networks and network data) is an essential component that must be seamlessly integrated into the digital twin models. Consequently, it is vital to ensure that machine learning models for network data can adapt to changing environmental conditions and work with challenges caused by limited annotated data. Transformers provide a flexible framework to overcome the limitations of limited labeled network data by leveraging pre-trained transformers and applying transfer learning techniques. In this paper, we present the initial results using a transformer model on a source domain with sufficient data and fine-tune it on a target domain with limited data, where the source and target domains have different network data distributions. Initial experimental results on a real-world network dataset show that our approach outperforms competitive baselines by up to 72.9% and 74% in prediction accuracy across two different target domains, showcasing its effectiveness and scalability for data-scarce network domains.