Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

Researchers have proposed a new measure to detect hallucinations in sign language translation (SLT) models, a major flaw in vision-language models. The measure combines feature-based sensitivity and counterfactual signals to quantify visual grounding.

Hallucination in SLT occurs when models generate text not supported by visual evidence, a problem exacerbated in gloss-free models that map continuous signer movements directly into natural language^[1]. These models are vulnerable because they lack intermediate gloss supervision that serves as alignment. Sign language translation is typically evaluated using surface-form metrics like BLEU and ROUGE, but these do not capture the semantic accuracy of translations^[2]. The proposed measure predicts hallucination rates and generalizes across datasets and architectures, according to researchers. A new benchmark, SLU-2K, has also been introduced, consisting of 2,350 closed-ended video question-answer pairs based on the PHOENIX-2014T and CSL-Daily datasets. This benchmark evaluates systems on their ability to recover key semantic aspects of the original sentence. The measure has been shown to distinguish between grounded and guessed tokens, allowing for risk estimation without references.