CN-NewsTTS Bench: a target-level automatic benchmark for raw-input Chinese news TTS pronunciation

A new open benchmark, CN-NewsTTS Bench v0.1, measures how accurately Chinese news text-to-speech products pronounce dense written forms — scores, abbreviations, and mixed-language names — directly from raw text without any preprocessing aids [1]. The benchmark, detailed in a paper submitted to arXiv on June 23, 2026, targets a persistent problem in Chinese news TTS: systems can read a written string aloud but alter its intended spoken meaning [2]. The release includes a 200-record development set, an 800-record public test set, and 992 public auto-evaluable targets [2]. Fixed transcripts are generated by a three-ASR ensemble, and an automatic target scorer evaluates outputs without relying on user-side rules, LLM rewriting, SSML hints, or manual edits [2]. Initial results for seven product TTS systems show a wide performance gap. The best system achieves 0.879 strict accuracy, while several systems remain below 0.60 [2]. The authors also report ASR-route diagnostics, ASR-subset ablations, category-level results, confidence intervals, and provider configuration metadata [2]. The work appears on arXiv through arXivLabs, a framework that lets collaborators build and share new features on the platform [1]. arXivLabs has integrated with Hugging Face Spaces, a service that hosts over 12,000 open-source machine learning demos as of November 2022 [3]. Through this integration, a Demos tab on a paper’s arXiv abstract page links to interactive applications that let users test models without writing code [4]. Researchers can link a Space to a paper by including the paper’s URL in the Space’s README file or by associating a model on the Hugging Face Hub with the Space [5]. Chinese TTS development sits within a broader landscape of rapid AI advancement in China. Hangzhou-based DeepSeek, founded in July 2023, released its DeepSeek-R1 model in January 2025 with performance comparable to OpenAI’s GPT-4 and o1, while reporting training costs of US$6 million — far below the US$100 million cost for GPT-4 [6]. DeepSeek’s models are open-weight, with parameters shared openly but training data kept proprietary [6]. The company trained its systems using weaker AI chips designed for export, navigating ongoing trade restrictions on advanced chip sales to China [6]. Large language models, defined as machine learning models with many parameters trained via self-supervised learning on vast text corpora, underpin both the TTS evaluation pipeline and the broader generative AI tools now competing in the market [7].