The Concept Allocation Zone: Tracking How Concepts Form Across Transformer Depth

A preprint posted to arXiv on 24 May 2026 proposes that concepts in transformer language models do not form at a single layer but across a depth interval the authors call the Concept Allocation Zone (CAZ) [1][2]. The paper formalizes the CAZ through three layer-wise metrics — Separation, Concept Coherence, and Concept Velocity — and derives a boundary detection method that does not require manual layer sweeps [2]. A CAZ is defined as the depth region within which the model organizes its geometry to make a concept separable, not the concept itself [2]. The authors note that a single concept typically participates in multiple CAZes, and multiple concepts may share a single zone [2]. Empirical validation covered 34 models drawn from 8 architectural families and examined 7 concepts [2]. The separation curve S(l) was frequently multimodal, indicating that concept formation is more complex than a single peak [2]. A scored detector identified what the researchers term “gentle CAZes” — subtle allocation regions that standard peak detection misses but that were causally active in 93–100% of cases under ablation [2]. These gentle CAZes appeared in 16 of the 34 models, with a companion validation paper reporting them in 26 models [2]. The framework generated seven testable predictions [2]. Four yielded clear verdicts: two were not supported, one was partially supported, and one was supported [2]. One prediction had its precondition invalidated by the data, and two were underpowered [2]. Cross-architecture alignment was confirmed as depth-matched rather than monolithic under leave-one-concept-out cross-validation [2]. The reference implementation, rosetta_tools v1.3.1, has been deposited with the DOI 10.5281/zenodo.20361433 [2]. The work was submitted through arXivLabs, a framework that allows community collaborators to develop and share new features on the arXiv website [1].