Boosted Prompt Ensembles for Large Language Models - Summary

Arxiv URL: https://arxiv.org/abs/2304.05970

Authors: Silviu Pitis, Michael R. Zhang, Andrew Wang, Jimmy Ba

Summary:

The paper proposes a prompt ensembling method for large language models called 'boosted prompting', which uses a small dataset to construct a set of few shot prompts that together comprise a boosted prompt ensemble. The few shot examples for each prompt are chosen in a stepwise fashion to be 'hard' examples on which the previous step's ensemble is uncertain. The algorithm outperforms single-prompt output-space ensembles and bagged prompt-space ensembles on challenging datasets like GSM8k and AQuA.

Key Insights & Learnings:

• Boosted prompting outperforms single-prompt output-space ensembles and bagged prompt-space ensembles on challenging datasets like GSM8k and AQuA.
• The algorithm uses a small dataset to construct a set of few shot prompts that together comprise a boosted prompt ensemble.
• The few shot examples for each prompt are chosen in a stepwise fashion to be 'hard' examples on which the previous step's ensemble is uncertain.
• The algorithm proposes both train-time and test-time versions of boosted prompting that use different levels of available annotation.
• The algorithm leverages a small dataset to construct a set of few shot prompts that progressively solve more of the problems, inspired by classical boosting algorithms.

Terms Mentioned: Large Language Models, few shot prompts, boosted prompting, output-space ensembles, bagged prompt-space ensembles, GSM8k, AQuA, train-time boosting, test-time boosting, annotation

Technologies / Libraries Mentioned: PyTorch, Hugging Face Transformers