We're Afraid Language Models Aren't Modeling Ambiguity - Summary

Arxiv URL: https://arxiv.org/abs/2304.14399v1

Authors: Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, Yejin Choi

Summary:

The paper discusses the importance of managing ambiguity in natural language understanding and evaluates the ability of language models (LMs) to recognize and disentangle possible meanings. The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity, and design a suite of tests based on it to evaluate pretrained LMs. They find that recognizing ambiguity remains extremely challenging for LMs, including for the recent GPT-4. The paper also presents a case study of how a multilabel NLI model can be used to detect misleading political claims in the wild.

Key Insights & Learnings:

• Ambiguity is an intrinsic feature of language and managing it is critical to the success of language models.
• Recognizing ambiguity remains extremely challenging for pretrained language models.
• The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity.
• A suite of tests based on AMBIENT is designed to evaluate pretrained LMs.
• A multilabel NLI model can be used to detect misleading political claims in the wild.

Terms Mentioned: natural language understanding, ambiguity, language models, AMBIENT, entailment, NLI, GPT-4, multilabel, political claims

Technologies / Libraries Mentioned: OpenAI