Paper Summaries

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes - Summary

The paper introduces a new mechanism called Distilling step-by-step that trains smaller models to outperform larger language models (LLMs) while using less training data and smaller model sizes. The mechanism extracts LLM rationales as additional supervision for small models within a multi-task tra

Arxiv URL: https://arxiv.org/abs/2305.02301

Authors: Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister

Summary:

The paper introduces a new mechanism called Distilling step-by-step that trains smaller models to outperform larger language models (LLMs) while using less training data and smaller model sizes. The mechanism extracts LLM rationales as additional supervision for small models within a multi-task training setup. The paper presents three findings across 4 NLP benchmarks: (1) Distilling step-by-step achieves better performance with much fewer labeled/unlabeled training examples compared to both fine-tuning and distillation, (2) smaller models outperform LLMs with much smaller model sizes, and (3) the mechanism reduces both the model size and the amount of data required to outperform LLMs.

Key Insights & Learnings:

Distilling step-by-step trains smaller models that outperform LLMs while using less training data and smaller model sizes.
The mechanism extracts LLM rationales as additional supervision for small models within a multi-task training setup.
Distilling step-by-step achieves better performance with much fewer labeled/unlabeled training examples compared to both fine-tuning and distillation.
Smaller models outperform LLMs with much smaller model sizes.
The mechanism reduces both the model size and the amount of data required to outperform LLMs.

Terms Mentioned: large language models, distillation, fine-tuning, multi-task training, rationales, NLP benchmarks, model size, training data

Technologies / Libraries Mentioned: BERT, T5

Instruction Tuning with GPT-4 - Summary

The paper presents the first attempt to use GPT-4 to generate instruction-following data for Large Language Models (LLMs) finetuning. The 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks compared to the instruction-following

Are We Really Making Much Progress in Text Classification? A Comparative Review - Summary

This paper reviews and compares methods for single-label and multi-label text classification, categorizing them into bag-of-words, sequence-based, graph-based, and hierarchical methods. The findings reveal that pre-trained language models outperform all recently proposed graph-based and hierarchy-b

A Survey of Large Language Models - Summary

This paper surveys the recent advances in Large Language Models (LLMs), which are pre-trained Transformer models over large-scale corpora. The paper discusses the background, key findings, and mainstream techniques of LLMs, focusing on pre-training, adaptation tuning, utilization, and capacity eval

Read next