Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model

An open-weight large language model extracted structured data from Dutch brain MRI reports with high accuracy, according to a new study that tested LLaMA 3.1 on 947 reports from a tertiary memory clinic [1]. The research, posted to arXiv on June 5, 2026, evaluated the model's ability to pull thirty annotated variables from free-text reports authored by consultant neuroradiologists between 2016 and 2021 [1]. Trained medical students double-annotated 100 reports to establish inter-rater reliability [1]. The study assessed performance using balanced accuracy for categorical variables, accuracy and mean absolute error for counts, and text similarity for free-text descriptions [1]. In zero-shot testing, LLaMA 3.1 achieved a mean balanced accuracy of 90 percent for Medial Temporal Atrophy on the left side and 96 percent on the right [1]. Global Cortical Atrophy was rated at 87 percent and Fazekas scale scores at 94 percent [1]. The model detected mentions of microbleeds with 93 percent accuracy and infarct mentions with 82 percent accuracy [1]. Text similarity for lesion location reached 0.95 [1]. Performance dropped for numerical variables. Accuracy for the number of microbleeds was 80 percent, while accuracy for infarct counts fell to 66 percent [1]. Translating the Dutch reports into English before processing yielded comparable results [1]. Few-shot prompting improved outcomes for numerical variables. Using a structural similarity-based example selection strategy, accuracy for microbleed counts rose to 92 percent and for infarct counts to 81 percent [1]. The authors concluded that challenges remain for location-specific variables [1]. The study contributes to a growing body of work on artificial intelligence in healthcare, where AI programs are being applied to diagnostics, treatment protocol development, and patient monitoring [5]. Research has identified inadequate implementation infrastructure, limited workflow integration, and the absence of ongoing performance monitoring as primary barriers preventing clinically validated AI tools from reaching routine use beyond their originating institutions [5]. The current study's use of an open-weight model on a non-English dataset addresses a gap, as few prior assessments have focused on Dutch neuroradiology reports [1].