The Trick of the Tail: Segmenting Heavy-Tailed Distributions

Abstract

Many systems generate heavy-tailed data sets within the Site Reliability Engineering (SRE) domain. Such datasets are composed of many small and few large observations. Fitting such datasets to known continuous distributions can be challenging due to the pronounced 'head' and long 'tail' of said datasets. This study considers a novel technique to split a dataset into two parts (head and tail) to allow for subsequent data modelling using existing fitting techniques. Using two test system datasets, we address whether a dataset can be modelled by its distribution 'head' and 'tail'. Our framework can aid SRE teams in modelling their datasets without resorting to non-parametric approaches such as Kernel Density Estimation (KDE).