CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder

A research team has introduced CANDLE, a lightweight system that applies Connectionist Temporal Classification to the task of character-level Arabic noise deduplication, according to a paper posted on arXiv [1]. The system addresses the challenge of repeated characters in Arabic text, which can indicate either correct spelling or informal elongation common in social media posts. CANDLE operates without handcrafted rules, dictionaries, or morphological analyzers, instead framing normalization as a sequence alignment problem over a character-based encoder [1][2]. The researchers evaluated the CTC model on three benchmarks: clean newspaper text, manually curated ambiguous cases, and real-world social media content. It achieved a Sentence Error Rate as low as 5.37% and consistently outperformed a classification-based baseline by a large margin [1][2]. To reduce inference overhead, the team distilled the original 6-layer CTC model into a 2-layer student model, achieving a 3× depth reduction with minimal performance degradation [1][2]. Beyond deduplication accuracy, the normalization process produced a practical downstream benefit: a relative reduction in tokenizer fertility of up to 12.8% across a diverse set of Arabic large language model tokenizers, which directly lowers inference costs and improves context window utilization [1][2]. The authors have released all code and models publicly to support reproducibility and further research [1][2].