Language Models are Few-Shot Learners - Summary

Arxiv URL: https://arxiv.org/abs/2005.14165

Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

Summary:

The paper discusses the limitations of pre-trained language representations in NLP systems and the need for task-specific datasets and fine-tuning. The authors show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches.

Key Insights & Learnings:

• Scaling up language models improves task-agnostic, few-shot performance.
• GPT-3, an autoregressive language model with 175 billion parameters, achieves strong performance on many NLP datasets.
• GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans.
• There are datasets where GPT-3's few-shot learning still struggles.
• The paper discusses broader societal impacts of GPT-3.

Terms Mentioned: NLP, pre-training, fine-tuning, language models, few-shot learning, GPT-3, autoregressive, parameters, datasets, news articles

Technologies / Libraries Mentioned: arXiv