The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance

A new comparative study finds that domain-specific pre-training, not raw model size, is the decisive factor for detecting causal adverse drug events, with BioBERT outperforming larger medical language models within the InferBERT framework for pharmacovigilance [1]. The research, submitted on 15 Jun 2026, evaluates how the choice of underlying classification model affects the InferBERT framework, which integrates transformer models with Do-calculus to distinguish causal adverse drug events from spurious correlations [1]. The study tested four models — XGBoost as a baseline, ALBERT as the original InferBERT model, the biomedical transformer BioBERT, and the medical large language model Med-LLaMA — across two benchmarks: Analgesics-induced Acute Liver Failure (AILF) and Tramadol-related Mortalities (TRAM) [1]. Using 5-fold cross-validation repeated over 20 runs, the authors measured accuracy, Expected Calibration Error before and after isotonic regression, and Jaccard concordance of causal terms with traditional pharmacovigilance signals such as PRR, ROR, and EBGM [1]. BioBERT achieved the highest accuracy on both datasets, while Med-LLaMA underperformed despite its larger size and the use of parameter-efficient fine-tuning [1]. The results demonstrate that domain-specific pre-training provides a clear advantage over simpler baselines and larger LLMs [1]. Post-hoc calibration improved Expected Calibration Error but had mixed effects on accuracy and causal discovery [1]. BioBERT's superiority also yielded the strongest concordance with traditional pharmacovigilance signals [1]. The study contributes to a broader conversation about model selection in specialized domains. Large language models are defined as machine learning models with many parameters, trained with self-supervised learning on vast amounts of text [7]. While scaling model size has been a dominant trend — exemplified by systems such as DeepSeek-R1, which provided responses comparable to OpenAI's GPT-4 and o1 at a reported training cost of US$6 million [6] — the InferBERT findings suggest that targeted, domain-aware architectures can outperform generic scaling in niche scientific tasks [1]. The paper is available on arXiv, a platform that has expanded access to machine learning research through integrations such as Hugging Face Spaces, which allows users to explore open-source demos linked directly from paper abstract pages [3][4]. Researchers can add demos to arXiv papers by including a link in a Space README file or by associating a model on the Hugging Face Hub [5]. The InferBERT study's conclusion — that investing in manageable, domain-aware models is more effective for computational pharmacovigilance than simply scaling model size — aligns with a growing body of evidence that domain-specific pre-training yields practical advantages in high-stakes biomedical applications [1].