Predicting Table Joinability in Data Lakes using a Metadata Knowledge Graph

Discovering joinable tables in data lakes is a crucial task for systems that provide insights to users such as business analysts, data engineers, and data scientists. Due to the lack of unifying schema for such data lakes, it is typically necessary to require some level of access to the contents of tables to inspect the contents of table columns and discover joinable pairs. We introduce an alternate approach for tackling this problem, wherein we utilize only the textual metadata of tables to predict joinability. Our approach, MeGNN-Join (Metadata Graph Neural Network for Joinability Prediction), models table metadata as a knowledge graph and frames predicting table joinability as a link prediction task over the graph. MeGNN-Join’s lack of reliance on table contents allows it to perform predictions in situations where typical content-based methods cannot operate, and it utilizes semantic enrichment of missing metadata to produce more effective predictions. To validate our approach, we design an evaluation framework using real world data from open government data portals and demonstrate that MeGNN-Join shows superior performance and generalizability to new tables compared to baselines.