TW-LegalBench: Measuring Taiwanese Legal Understanding

A new benchmark called TW-LegalBench evaluates large language models on Taiwanese law, revealing that top models can pass the bar for qualified lawyers but fail to meet the threshold for judges and prosecutors, according to a paper posted to arXiv on June 17, 2026 [1]. The dataset fills a gap in jurisdiction-specific legal reasoning, as most common-law benchmarks rely on English sources and civil-law benchmarks focus on Simplified Chinese [2]. TW-LegalBench draws on Taiwan's publicly available official legal corpus and includes three task types: over 16,000 multiple-choice questions spanning five years of official examinations across 18 professional domains, 117 open-ended essay questions with official scoring rubrics, and more than 14,000 legal judgment prediction instances covering hundreds of crime categories [2]. Researchers evaluated 13 large language models using accuracy for multiple-choice questions, a decomposed LLM-as-Judge framework for essays, and metrics for sentencing accuracy and statute citation for judgment prediction [2]. Top-performing models exceeded the passing threshold for qualified lawyers, which has an 11% passing rate, but fell short of the 1~2% passing rate required for judges and prosecutors [2]. In the legal judgment prediction task, models demonstrated reasonable accuracy in predicting verdict types and sentences, yet they struggled to cite exact legal articles [2]. The findings underscore that reliable legal text generation remains a challenge for large language models, even as their performance on qualification exams approaches human levels [2]. Large language models are machine learning systems with many parameters, trained on vast amounts of text through self-supervised learning [10]. Companies such as DeepSeek, a Chinese AI firm founded in July 2023, have developed open-weight models that rival offerings from OpenAI and Meta while reporting significantly lower training costs [9]. DeepSeek's R1 model, for instance, was trained for a reported US$6 million, compared to the US$100 million cost for OpenAI's GPT-4 in 2023 [9]. The TW-LegalBench paper was submitted to arXiv, a preprint repository that has integrated with Hugging Face Spaces to allow researchers to share interactive demos alongside their papers [6][7]. Through this collaboration, users can navigate to a paper's Demo tab on arXiv and try open-source machine learning applications directly in a browser without writing code [8].