Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

A new benchmark called Argus evaluates how well uncertainty quantification methods work for computer-use agents that translate vision-language model predictions into GUI clicks, according to a paper posted to arXiv. The study finds that while some methods perform well within a single model, rankings rarely transfer across different model classes or to closed-source systems. The benchmark, detailed in a preprint submitted on 24 June 2026, assembles a 27-method open-weight matrix spanning four VLM agents and four datasets, alongside an 8-method closed-source matrix covering three frontier vendors where internal model states are inaccessible [1]. The evaluated techniques include logit-based scores, sampling and consistency measures, hidden-state and density estimators such as Mahalanobis and SAPLMA, attention-based scores, verbalised-confidence prompting, and split-conformal prediction [2]. The authors report that within-model ranking transfer is strong, with a Spearman rho reaching 0.969, but cross-tier transfer to closed-source vendors averages only +0.08 [2]. This gap means that uncertainty rankings derived from open-weight models cannot be reliably extrapolated to proprietary systems; instead, closed-source UQ should be reranked directly on the target agent [2]. Hidden-state and density methods emerged as the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment won in specific regimes [2]. The paper also examines conformal click regions, showing that locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch [2]. The authors conclude that score-level discrimination alone is insufficient for deployment in GUI agents [1]. The work addresses a fragmentation problem: prior evidence on post-hoc UQ for computer-use agents was scattered across isolated model and dataset pairs, making it unclear whether UQ rankings remain stable when the agent, benchmark, or observable interface changes [2]. The researchers release per-item records, calibration and test splits, UQ scores, and analysis scripts to support regime-aware UQ selection [2].