ICLR 2022
Conference paper

Multitask Prompt Tuning Enables Zero-Shot Task Generalization

Download paper


Very large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It is theorized that this is a consequence of implicit multitask learning in language model training (Radford et al., 2019). Can better zero-shot generalization can be directly induced by multitask learning on explicit prompts? To test this question at scale, we develop a system for easily mapping general natural language tasks into a human-readable prompted form. We annotate a large set of supervised datasets, each with multiple prompts using varying language. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020) on this multitask mixture covering a wide variety of tasks, from question-answering to text classification to summarization. The model attains strong zero-shot performance across datasets, often outperforming models 16× its size. All collected datasets, prompting tools, and pretrained models are available at


24 Apr 2022


ICLR 2022