Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

A new evaluation framework reveals systematic gaps in the long-range statistical organization of text generated by large language models compared to natural language, according to research submitted on 24 May 2026 [1]. The study, led by Kumiko Tanaka-Ishii, proposes a method based on repeated subsequences to probe how texts reuse previously established structure under finite-length conditions [1]. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text [2]. The framework relates the distribution of repeated subsequences across scales to higher-order Rényi entropies, offering a quantitative structural diagnostic [2]. Experiments compared human-written texts with length-matched outputs from GPT models [1]. While power-law models can describe restricted ranges of block length, the observed entropy growth was often equally or better characterized by logarithmic-power forms [2]. Across datasets, natural language exhibited stable entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts [2]. In contrast, GPT-generated texts showed systematic and statistically significant shifts in estimated exponents with model size [1]. The findings touch on a long-standing question in quantitative linguistics. Zipf's law, an empirical observation that word frequency is approximately inversely proportional to its rank, has been documented across many natural languages [4]. For instance, in the Brown Corpus of American English, the word "the" accounts for nearly 7% of all word occurrences, while the second-place word "of" accounts for slightly over 3.5% [4]. Such statistical regularities have also been used to analyze disputed texts. The Voynich manuscript, a 15th-century codex written in an unknown script, has been studied by professional and amateur cryptographers including William Friedman and John Tiltman, but has never been demonstrably deciphered [5]. Statistical analyses of its text have shown it follows Zipf-like patterns, fueling debate over whether it represents a natural language, a constructed language, or a hoax [5]. Modern language models are built on the transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by researchers at Google [3]. Transformers process text by converting words into numerical tokens and contextualizing each token within a context window via a parallel multi-head attention mechanism [3]. This design has no recurrent units, requiring less training time than earlier architectures such as long short-term memory networks, and has been widely adopted for training large language models on large datasets [3]. The new evaluation framework suggests that despite the fluency these architectures produce, the deep structural organization of their outputs diverges measurably from that of natural language [1].