Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation

Large language models are increasingly used for machine translation, but researchers have found that the models' internal confidence signals can be misleading and that verbalized confidence methods perform similarly, with little correlation between the two approaches [1]. The study, published on arXiv, examines the reliability of confidence in translation outputs from large language models (LLMs), which are machine learning models trained on vast amounts of text for natural language processing tasks [5]. The authors note that unsupervised approaches relying on internal signals, such as predicted probabilities, can be problematic because they reflect certainty among alternatives rather than actual correctness, and they require access to internal model data [1]. To address these shortcomings, the researchers devised five verbalized methods to extract an LLM's per-token confidence [1]. When evaluated for reliability using fine-grained error detection and calibration, the internal and verbalized methods performed similarly, though results varied by model [1]. The researchers found little to no correlation between the internal and verbalized confidence methods [1]. This work comes as LLMs are being deployed for a growing range of generation tasks. For example, DeepSeek, a Chinese AI company founded in July 2023, launched its DeepSeek-R1 model in January 2025, providing responses comparable to other contemporary LLMs such as OpenAI's GPT-4 [4]. The company reported training its V3 model for US$6 million, significantly less than the US$100 million cost for GPT-4 in 2023 [4]. Other major LLM families include Qwen, developed by Alibaba Cloud, which is distributed under open-source licenses such as Apache 2.0 [6]. The translation confidence study adds to a broader body of research assessing the reliability of generative AI systems. A separate scoping review of quantum circuit and quantum code generation systems, for instance, found that while all reviewed systems addressed syntactic validity and most addressed semantic correctness, none reported end-to-end evaluation on quantum hardware, leaving a gap between generated circuits and practical deployment [3]. The new translation research provides a framework for understanding when and how LLMs can self-assess their own outputs, a capability that remains uncertain even as the models are adopted for high-stakes language tasks [1].