AI translation of literary texts is "fine", but readers still prefer human translations

A new study finds that while AI-generated literary translations are technically adequate, avid readers consistently prefer human translations for their ease, clarity, and immersive quality, according to research published on arXiv [1]. The study, which evaluated translations of 15 recent novels from French, Polish, and Japanese into English, asked 15 avid readers to compare human translations (HT) with machine translations (MT) produced by an agentic large language model pipeline [1]. Readers assessed approximately 8,000-word excerpts under two conditions: immersive reading of the full excerpt and close reading of 386 aligned chunk pairs, yielding 772 chunk-level comparisons [1]. At the excerpt level, readers preferred human translations in 19 out of 30 cases; at the more granular chunk level, the preference for human translations was clearer, with readers favoring them in 522 of 772 comparisons [1]. Crucially, readers could not reliably distinguish between the two versions, guessing correctly only 17 out of 30 times, and they tended to prefer whichever version they believed was human [1]. The researchers note that automatic evaluation metrics, including large language model-based judges, failed to recover these reader preferences and instead favored machine translations [1]. The work highlights a gap between conventional translation quality assessment and the subjective experience of literary reading. Translation has long been understood as the communication of meaning across languages, a process that can inadvertently introduce source-language structures into the target text [7]. The study also found that the quality of machine translations varied more within a single book than that of human translations [1]. To support further research, the team released LAIT (Literary AI Translation), a reader-centered evaluation dataset containing 1,000 reader comments, 2,000 judgments and preference ratings, and 7,200 span-level annotations, along with the evaluation protocol and supporting interface [1]. The dataset is available on Hugging Face [1]. The findings arrive as AI translation tools become increasingly common for literary works, raising questions about how readers experience translated fiction beyond basic fluency and adequacy [1].