Enabling Programmable Metric Flows

Abstract

In the evolving computing landscape, extending from centralized clouds to multi-cloud and edge, the need for adaptable observability is becoming increasingly critical. Traditional static monitoring approaches grapple with inefficient data transfer, limited scalability, heterogeneous environments, and rigid metric processing pipelines. This paper introduces a novel metric processing system, Programmable Metric Flows (PMF), which is rooted in the principle of dynamism. PMF is a first-of-its-kind, light-weight, SQL-based metric processor. It empowers optimization-driven transformations of metrics, tailored to evolving resource availability and application requirements. This paper demonstrates how PMF enables various transformations for dynamic and fine-grained metric collection. We also showcase the capability of PMF for dynamically tuning the frequency of metrics to reduce the WAN cost in edge environments. Our experiments show that PMF performs at par with state-of-the-art techniques in terms of metric processing capability, with 10X lesser resource utilization. We envision PMF to usher in an era of lightweight programmability for observability platforms. PMF is open-source and available at: https://github.com/observ-vol-mgt/PMF.