Fine-tuning a multimodal large language model for clinician-grade autism behavioral scoring from short home videos

A multimodal large language model fine-tuned on short home videos can extract behavioral features for autism assessment with reliability approaching that of clinicians, according to a study published on arXiv. The approach improved inter-rater reliability by 40% and boosted direct diagnosis performance by 53% on held-out children [1]. Autism spectrum disorder affects 1 in 31 children in the United States, yet the median age at diagnosis remains above four years [1]. Researchers from the study fine-tuned Gemini 2.5 Pro using low-rank adaptation on 400 clinician-rated home videos, training the model on 30 behavioral features previously validated for reliable prediction when passed to machine learning classifiers [1]. On a held-out set of 99 children — 49 with ASD and 50 neurotypical — per-feature weighted Cohen's kappa between the model and clinicians rose by 40% (p<0.001), with 27 of 28 evaluable features showing improvement [1]. As an emergent zero-shot capability, the model's direct ASD diagnosis F1 score improved by 53% (p<0.001), matching or exceeding clinician outcomes [1]. Classifier-assisted pipelines that used the fine-tuned LLM-derived behavioral features matched the performance of pipelines relying on clinician-scored inputs across all tested pathways [1]. Those pipelines achieved 77% accuracy (95% CI: 68-85%) and an area under the curve of 86% (95% CI: 78-92%) [1]. The work builds on a broader push to apply foundation models to observational data in public health. Separate efforts have explored geospatial embeddings for agricultural mapping, where a U-Net trained on AlphaEarth embeddings achieved 99.19% pixel accuracy for tomato-field identification in California without manual feature engineering [3]. Other studies have tackled spatio-temporal misalignment in environmental exposure modeling, using Bayesian weighted predictor regression to estimate fine particulate matter from satellite and ground-monitor data in Northern California [5]. These parallel lines of inquiry share a common thread: large pre-trained models can extract structured signals from unstructured or irregularly sampled inputs, reducing the preprocessing burden that has historically limited scalability. The autism study's authors conclude that fine-tuned multimodal LLMs can serve as scalable behavioral feature extractors for use in autism assessment and diagnosis [1]. The approach relies on home video, an easy-to-access form of observational data, which could lower barriers to earlier identification and timely delivery of interventions [1].