Production Guides
⭐️ Decoding OpenAI Evals
Learn how to use the eval framework to evaluate models & prompts to optimise LLM systems for the best outputs.
Latest
paper summaries
Discovering Language Model Behaviors with Model-Written Evaluations - Summary
The paper explores the use of language models (LMs) to automatically generate evaluations for testing LM behaviors. The generated evaluations are diverse and of high quality, and the approach is significantly cheaper, lower effort, and faster than manual data creation. The paper discovers new cases
paper summaries
We're Afraid Language Models Aren't Modeling Ambiguity - Summary
The paper discusses the importance of managing ambiguity in natural language understanding and evaluates the ability of language models (LMs) to recognize and disentangle possible meanings. The authors present AMBIENT, a linguist-annotated benchmark of 1,645 examples with diverse kinds of ambiguity
paper summaries
Scaling Transformer to 1M tokens and beyond with RMT - Summary
The paper presents a method to extend the context length of BERT, a Transformer-based model in natural language processing, by incorporating token-based memory storage and segment-level recurrence with recurrent memory (RMT). The method enables the model to store task-specific information across up
paper summaries
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes - Summary
The paper introduces a new mechanism called Distilling step-by-step that trains smaller models to outperform larger language models (LLMs) while using less training data and smaller model sizes. The mechanism extracts LLM rationales as additional supervision for small models within a multi-task tra
paper summaries
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head - Summary
The paper proposes a multi-modal AI system named AudioGPT that complements Large Language Models (LLMs) with foundation models to process complex audio information and solve numerous understanding and generation tasks. AudioGPT is connected with an input/output interface (ASR, TTS) to support spoke
paper summaries
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning - Summary
The paper discusses the limitations of large language models (LMs) and proposes a neuro-symbolic architecture called the Modular Reasoning, Knowledge and Language (MRKL) system that combines LMs with external knowledge sources and discrete reasoning modules to overcome these limitations.
paper summaries
ReAct: Synergizing Reasoning and Acting in Language Models - Summary
The paper introduces ReAct, a novel prompt-based paradigm that synergizes reasoning and acting in language models for general task solving. ReAct generates both verbal reasoning traces and actions in an interleaved manner, allowing the model to perform dynamic reasoning to create, maintain, and adj
paper summaries
Just Tell Me: Prompt Engineering in Business Process Management - Summary
The paper discusses the use of prompt engineering to leverage pre-trained language models for business process management (BPM) tasks. It identifies the potentials and challenges of prompt engineering for BPM research.
paper summaries
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models - Summary
The paper introduces Chameleon, a plug-and-play compositional reasoning framework that augments large language models (LLMs) to address their inherent limitations and tackle a broad range of reasoning tasks. Chameleon synthesizes programs to compose various tools, including LLM models, off-the-shel
paper summaries
MiLMo:Minority Multilingual Pre-trained Language Model - Summary
The paper presents a multilingual pre-trained language model named MiLMo that performs better on minority language tasks, including Mongolian, Tibetan, Uyghur, Kazakh and Korean. The authors also construct a minority multilingual text classification dataset named MiTC, and train a word2vec model fo
paper summaries
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace - Summary
The paper proposes HuggingGPT, a system that uses large language models (LLMs) like ChatGPT to connect various AI models in machine learning communities like HuggingFace to solve complicated AI tasks. The system leverages the strong language capability of ChatGPT and abundant AI models in HuggingFa