Empirical Bayes Conformal Prediction for Vision and Language Models

A new conformal prediction framework uses empirical Bayes r-values to incorporate score variability into uncertainty estimates, aiming to improve ranking stability and shrink prediction set sizes for vision and language models, according to a paper published on arXiv [1]. Standard conformal prediction methods for modern AI models often rely on a single, potentially unstable nonconformity score to decide which candidates to include in a prediction set. The authors of the paper note that averaging multiple realizations into a point estimate similarly discards information about inconsistency that could signal whether a candidate is genuinely stable [1]. A weak answer can enter the conformal set simply because one posterior sample or prompt phrasing made it appear strong [2]. The new framework addresses this by converting score variability into an uncertainty-informed nonconformity score using r-values, which estimate how likely a candidate's latent score belongs to the top-ranked group after accounting for both its mean score and its uncertainty [2]. The method offers two estimation approaches: a closed-form Normal-Normal empirical Bayes estimator and a nonparametric posterior-sampling estimator [2]. Using the r-value as the nonconformity score preserves the target conformal coverage while provably reducing the inclusion of high-variance false candidates under mild regularity conditions [2]. The framework was tested across image classification, CLIP-based vision-language model benchmarks, and large language models (LLMs) [2]. These model types rely on transformer architectures, which use multi-head attention mechanisms to contextualize tokens and have become foundational for both language and vision tasks since the 2017 paper "Attention Is All You Need" [3]. Vision-language models and LLMs are prominent examples of multimodal learning systems, which integrate data types such as text and images to improve performance on tasks like visual question answering and image captioning [5]. The reported experiments showed that r-value conformal prediction preserved target coverage while improving ranking stability and reducing set size when variability was informative. When variability vanished, the method reverted to behavior similar to standard conformal prediction [2]. The work does not introduce new training procedures like reinforcement learning from human feedback, a technique used to align models with human preferences by training a reward model on ranking data from annotators [4], but instead focuses on the calibration and inference stage of already-trained models.