A Survey of Large Language Models - Summary

Arxiv URL: https://arxiv.org/abs/2303.18223v1

Authors: Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, Ji-Rong Wen

Summary:

This paper surveys the recent advances in Large Language Models (LLMs), which are pre-trained Transformer models over large-scale corpora. The paper discusses the background, key findings, and mainstream techniques of LLMs, focusing on pre-training, adaptation tuning, utilization, and capacity evaluation. The paper also summarizes the available resources for developing LLMs and discusses the remaining issues for future directions.

Key Insights & Learnings:

• Language modeling is a major approach to advancing language intelligence of machines.
• LLMs display surprising emergent abilities that may not be observed in previous smaller PLMs.
• LLMs have revolutionized the way that humans develop and use AI algorithms.
• The development of LLMs no longer draws a clear distinction between research and engineering.
• LLMs are posing a significant impact on the AI community and the advent of ChatGPT and GPT-4 leads to the rethinking of the possibilities of artificial general intelligence (AGI).

Terms Mentioned: Large Language Models, Emergent Abilities, Adaptation Tuning, Utilization, Alignment, Capacity Evaluation, Language Modeling, Statistical Language Models, Neural Language Models, Pre-trained Language Models, Transformer Architecture, Self-Attention Mechanisms, Artificial Intelligence, Natural Language Processing, Recurrent Neural Networks, Word2Vec, Bidirectional LSTM, BERT, GPT-2, BART, ChatGPT, Artificial General Intelligence

Technologies / Libraries Mentioned: ELMo, Transformer, PyTorch, TensorFlow