Enhancing In-context Learning via Linear Probe Calibration

In-context learning (ICL) is a new paradigm for natural language processing that utilizes Generative Pre-trained Transformer (GPT)- like models. This approach uses prompts that include in-context demonstrations to gener- ate the corresponding output for a new query input. However, applying ICL in real cases does not scale with the number of samples, and lacks robustness to different prompt tem- plates and demonstration permutations. In this paper, we first show that GPT-like mod- els using ICL result in unreliable predictions based on a new metric based on Shannon en- tropy. Then, to solve this problem, we propose a new technique called the Linear Probe Cali- bration (LinC), a method that calibrates the model’s output probabilities, resulting in reli- able predictions and improved performance, while requiring only minimal additional sam- ples (as few as five labeled data samples). LinC significantly enhances the ICL test per- formance of GPT models on various bench- mark datasets, with an average improvement of up to 21%, and up to a 50% improvement in some cases, and significantly boosts the per- formance of PEFT methods, especially in the low resource regime. Moreover, LinC achieves lower expected calibration error, and is highly robust to varying label proportions, prompt templates, and demonstration permutations.