Paper Summaries

GPT Understands, Too - Summary

The paper proposes a novel method called P-tuning, which employs trainable continuous prompt embeddings to improve the performance of GPTs on natural language understanding (NLU) tasks. The method is shown to be better than or comparable to similar-sized BERTs on NLU tasks and substantially improve

Arxiv URL: https://arxiv.org/abs/2103.10385

Authors: Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang

Summary:

The paper proposes a novel method called P-tuning, which employs trainable continuous prompt embeddings to improve the performance of GPTs on natural language understanding (NLU) tasks. The method is shown to be better than or comparable to similar-sized BERTs on NLU tasks and substantially improves the previous best on the knowledge probing (LAMA) benchmark. P-tuning also improves BERTs’ performance in both few-shot and supervised settings while reducing the need for prompt engineering. The paper shows that language models contain much more world knowledge and prior task knowledge than previously assumed.

Key Insights & Learnings:

GPTs can be as competitive as BERTs in natural language understanding with P-tuning, which can boost pre-trained language models’ performance.
P-tuning is a general method to improve GPTs and BERTs in both few-shot and fully-supervised settings.
Language models have grasped more world knowledge and prior-task knowledge during pre-training than previously thought.
Giant models suffer from poor transferability, and fine-tuning on downstream tasks hardly works for those trillion-scale models.
Handcraft prompt searching heavily relies on large validation sets and can result in overfitting.

Terms Mentioned: natural language understanding, pre-training, language models, GPT, BERT, P-tuning, knowledge probing, LAMA, SuperGlue, few-shot, supervised learning, world knowledge, prior task knowledge, transferability, fine-tuning, downstream tasks, trillion-scale models, handcrafted prompts, overfitting

Technologies / Libraries Mentioned: PyTorch

Instruction Tuning with GPT-4 - Summary

The paper presents the first attempt to use GPT-4 to generate instruction-following data for Large Language Models (LLMs) finetuning. The 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks compared to the instruction-following

Are We Really Making Much Progress in Text Classification? A Comparative Review - Summary

This paper reviews and compares methods for single-label and multi-label text classification, categorizing them into bag-of-words, sequence-based, graph-based, and hierarchical methods. The findings reveal that pre-trained language models outperform all recently proposed graph-based and hierarchy-b

A Survey of Large Language Models - Summary

This paper surveys the recent advances in Large Language Models (LLMs), which are pre-trained Transformer models over large-scale corpora. The paper discusses the background, key findings, and mainstream techniques of LLMs, focusing on pre-training, adaptation tuning, utilization, and capacity eval

Read next