We're Afraid Language Models Aren't Modeling Ambiguity - Summary
The paper discusses the importance of managing ambiguity in natural language understanding and evaluates the ability of language models (LMs) to recognize and disentangle possible meanings. The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity
Arxiv URL: https://arxiv.org/abs/2304.14399v1
Authors: Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, Yejin Choi
Summary:
The paper discusses the importance of managing ambiguity in natural language understanding and evaluates the ability of language models (LMs) to recognize and disentangle possible meanings. The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity, and design a suite of tests based on it to evaluate pretrained LMs. They find that recognizing ambiguity remains extremely challenging for LMs, including for the recent GPT-4. The paper also presents a case study of how a multilabel NLI model can be used to detect misleading political claims in the wild.
Key Insights & Learnings:
- Ambiguity is an intrinsic feature of language and managing it is critical to the success of language models.
- Recognizing ambiguity remains extremely challenging for pretrained language models.
- The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity.
- A suite of tests based on AMBIENT is designed to evaluate pretrained LMs.
- A multilabel NLI model can be used to detect misleading political claims in the wild.
Terms Mentioned: natural language understanding, ambiguity, language models, AMBIENT, entailment, NLI, GPT-4, multilabel, political claims
Technologies / Libraries Mentioned: OpenAI
