AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts - Summary

Arxiv URL: https://arxiv.org/abs/2010.15980

Authors: Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh

Summary:

The paper introduces AUTOPROMPT, an automated method to create prompts for a diverse set of tasks based on a gradient-guided search. The prompts elicit more accurate factual knowledge from masked language models (MLMs) than manually created prompts on the LAMA benchmark. MLMs can perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art. AUTOPROMPT provides certain practical advantages over finetuning, achieving higher average- and worst-case accuracy in low-data regimes and not requiring large amounts of disk space to store model checkpoints.

Key Insights & Learnings:

• AUTOPROMPT is an automated method to create prompts for a diverse set of tasks based on a gradient-guided search.
• MLMs can perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art.
• AUTOPROMPT prompts elicit more accurate factual knowledge from MLMs than manually created prompts on the LAMA benchmark.
• MLMs can be used as relation extractors more effectively than supervised relation extraction.
• AUTOPROMPT provides certain practical advantages over finetuning, achieving higher average- and worst-case accuracy in low-data regimes and not requiring large amounts of disk space to store model checkpoints.

Terms Mentioned: pretrained language models, finetuning, probing classifiers, attention visualization, prompting, AUTOPROMPT, masked language models, factual knowledge, LAMA benchmark, relation extraction

Technologies / Libraries Mentioned: PyTorch, Hugging Face Transformers