Tracing Computation Density in LLMs

A new method called s-Trace reveals that large language models organize computation in two distinct phases, with a sparse early-layer core providing a rough prediction that is later refined by denser computations, according to a paper posted to arXiv on May 26, 2026 [1]. Transformer-based large language models contain billions of parameters arranged in deep and wide computational graphs, yet it remains unclear whether they use their full capacity for every input [2]. Researchers introduced the s-Trace method to estimate the subgraph of size s that best approximates a full model output, finding that a small subgraph composed mostly of early-layer nodes can reconstruct the head of the full model output distribution [2]. Adding further nodes, mostly located in later layers and increasingly consisting of attention heads, leads to incremental refinements in approximating the full output distribution [2]. The paper was submitted by Corentin Kervadec and is available as a 2,311 KB PDF [1]. The findings arrive as large language models continue to proliferate globally. Generative pre-trained transformers, the architecture underlying most modern LLMs, were first applied by OpenAI with GPT-1 in 2018 and have since spawned systems such as ChatGPT, Gemini, DeepSeek, and Claude [3]. More recent iterations, including GPT-5 released in 2025, incorporate routers that automatically select between faster models and slower reasoning models depending on the task [3]. The s-Trace results suggest that even these advanced systems may rely on a consistent modular organization, with a sparse early-layer core providing a rough prediction that is further refined through denser computations in later layers [2]. The study also found that the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics such as unigram frequency [2]. This aligns with broader algorithmic principles: algorithms define sets of rules or procedures for calculations, data processing, pattern recognition, and automated reasoning [4]. As AI adoption accelerates in markets such as India, where the AI sector is projected to reach $8 billion by 2025 and ranks 10th globally for private-sector AI investment according to a 2025 UN Trade and Development report, understanding the internal efficiency of these models carries economic implications [5]. India accounted for the largest share of ChatGPT mobile app users and the third-largest user base for DeepSeek in 2025 [5]. The s-Trace method may offer a lens for evaluating how much computation is actually required per query, potentially informing deployment costs and infrastructure decisions.