Experiences with managing data ingestion into a corporate datalake

Abstract

We explain our experiences in designing, building and running a large corporate Datalake. Our platform has been running for over two years and makes a wide variety of corporate data assets, such as sales, marketing, customer information, as well as data from less conventional sources such as weather, news and social media available for analytics purposes to many teams across the company. We focus on describing the management of data and in particular how it is transferred and ingested into the platform.