Illegal tapping of fuel pipelines has recently become one of the most relevant safety problems faced by the industry. Hundreds of illegal interventions have been reported around the world, causing a significant number of deaths, relevant impacts on the environment, and capital loss. Therefore, it is important to develop systems that are able to detect such scenarios at an early stage, enabling a fast counteract. To this end, machine learning algorithms can train models on available data for detecting future issues. Most recently, ensemble learning and dynamic classifier selection (DCS) techniques have been achieving promising results in supervised learning tasks. Such models are usually trained based on a single criterion. However, it is desirable to take into account both the number of false positives (FP) and false negatives (FN) for the illegal tapping detection task, since they are conflicting and both lead to financial losses and/or accidents. Therefore, this work proposes a novel DCS technique based on multiple criteria, namely overall local class-specific accuracy (OLCA), which employs multi-criteria decision making for dynamically selecting the best classifier for a new sample given the local true positive and negative ratios. A numerical experiment is conducted for assessing the generalization performance of the proposed method in an oil pipeline, with the goal of detecting illegal taping using pressure transient signals. Results show that OLCA is able to reduce the number of both FP and FN when dynamically selecting the classifiers of a baseline Random Forest ensemble.