Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference - Summary

Arxiv URL: https://arxiv.org/abs/2412.13663

Authors: Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli

Summary

The research paper introduces ModernBERT, an updated version of the original BERT model, which is an encoder-only transformer designed to improve retrieval and classification tasks. Despite the original BERT's widespread use, until now, there have been limited improvements in terms of efficiency and performance. This paper presents ModernBERT, which incorporates modern architectural optimizations to address these shortcomings. It has been trained on a vast dataset of two trillion tokens and supports a native sequence length of 8192 tokens, which is significantly longer than the original BERT model’s sequence length of 512. ModernBERT achieves state-of-the-art results on various classification tasks and showcases robust performance on single and multi-vector retrieval tasks.

The authors implemented numerous architectural advancements to enhance ModernBERT's efficiency and performance. These include the use of rotary positional embeddings (RoPE), Gated Linear Units (GLU) instead of GeLU for activation, and alternating local and global attention mechanisms. Another significant innovation is the use of complete unpadding to make processing more efficient, especially concerning memory usage. This contributes to ModernBERT's impressive throughput, processing sequences with unparalleled speed compared to previous models.

ModernBERT demonstrates excellent performance across several downstream evaluations, such as text retrieval and natural language understanding tasks. It outperforms other leading models on benchmarks like GLUE. Remarkably, on long-context retrieval tasks like MLDR, ModernBERT shows significant improvements, indicating its capability to manage longer sequences effectively. In addition, due to its training on datasets mixed with code data, it excels in code-related tasks, setting a new standard in both text and code processing.

In summary, the paper argues that encoder-based models, like ModernBERT, still hold significant potential compared to larger, decoder-based models. By incorporating the latest architectural insights and massive pretraining datasets, ModernBERT overcomes previous limitations and sets a new benchmark for efficiency and performance in encoder-only models. Furthermore, its compatibility with common GPUs makes it an accessible choice for various applications, promising a brighter future for encoder-only models in NLP tasks.

Key Insights & Learnings

• ModernBERT achieves state-of-the-art results with longer sequence lengths (up to 8192 tokens) while being more memory and speed efficient.
• Utilizes recent architectural innovations like rotary positional embeddings and gated linear units, resulting in better performance.
• Introduces full unpadding, significantly enhancing processing efficiency and throughput.
• ModernBERT is highly effective in retrieval tasks, outperforming other models on long-context benchmarks.
• First encoder-only model to combine comprehensive internet-scale pretraining data, including code, bolstering its versatility across various NLP tasks.

Terms Mentioned: ModernBERT, Encoder-only Transformer, Rotary Positional Embeddings, Memory Efficiency, Long Context Finetuning

Technologies / Libraries Mentioned: Flash Attention, torch.compile, Weights&Biases