Legal Domain Adaptation of Modern BERT Models

Researchers have adapted a modern BERT model for the legal domain, reporting significant performance gains over its base version on US court opinion datasets, according to a new paper [1]. The study, posted to arXiv, details further pre-training of ModernBERT on all US court opinions using a masked language modeling objective [1]. ModernBERT was trained on roughly 500x more data than the original BERT model, yet the authors found it still benefited from domain-specific adaptation [1]. The resulting models can process sequences up to 8,192 tokens, enabling them to compute embeddings of legal passages or rapidly rerank hundreds of passages for a given search query [1]. The performance improvements were similar to those reported in early work on domain adaptation of BERT-like models, though training a model from scratch did not match the performance of further pre-training an existing ModernBERT checkpoint [1]. Large language models, including BERT variants, are neural networks trained on vast text corpora for natural language processing tasks [2]. They are typically based on the transformer architecture, with generative pre-trained transformers representing a common type that is pre-trained to predict the next word and later fine-tuned for specific behaviors [2]. The legal domain adaptation work applies engineering principles to create a scalable, efficient AI-based solution for a specialized field, merging aspects of data engineering and software engineering [3]. The researchers have released all model checkpoints publicly [1].