Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis

A hybrid system that pairs a large language model with a traditional machine-learning classifier outperformed end-to-end LLMs in diagnosing pediatric appendicitis, according to research posted to arXiv on June 17. The system, called ClaMPAPP, uses the LLM solely to extract clinical features from narrative notes, reserving the final risk prediction for a gradient-boosted tree model. The work addresses a persistent tension in clinical artificial intelligence. Large language models can interpret free-text physician notes, making them easy to slot into existing workflows, but their direct use as diagnostic engines is limited by sensitivity to prompt wording, the order of information, and the generation of plausible but incorrect outputs [2]. Structured machine-learning models offer more stable risk estimates, yet they demand tabular inputs that are cumbersome to produce in narrative-driven clinical settings [2]. ClaMPAPP — short for Clinical Language-assisted Machine-learning Pipeline for Appendicitis — attempts to capture the strengths of both approaches while sidestepping their weaknesses [1][2]. The pipeline works in three stages. First, an LLM reads note-like narratives and extracts a predefined set of clinical features. Those features then pass through deterministic plausibility checks designed to catch extraction errors. Finally, validated features are fed into an XGBoost classifier trained on clinical, laboratory, and ultrasound variables [1][2]. The architecture means the LLM acts as an interface, not as the final decision-maker — a design the authors describe as “LLM-as-interface, ML-as-predictor” [2]. Researchers evaluated ClaMPAPP on two independent pediatric appendicitis cohorts drawn from German hospitals [1][2]. To preserve ground-truth labels while testing free-text input, the team generated narratives from structured electronic health records using template rendering and constrained LLM rewriting. They also permuted sentence order to measure how sensitive each system was to the sequence of information in a note [2]. ClaMPAPP delivered the strongest overall diagnostic performance in both internal and external validation, with particular strength in minimizing missed appendicitis cases — the primary safety concern in acute triage [1][2]. By contrast, end-to-end LLMs, including both open-source and proprietary models, showed unstable sensitivity-specificity trade-offs and greater performance degradation when sentence order was shuffled [1][2]. The findings add to a growing body of evidence that separating natural-language understanding from predictive inference can yield more auditable clinical decision-support tools [2]. The paper was submitted to arXiv on June 17, 2026, and is available under the Computation and Language category [1].