The Context-Ready Transformer

A new recurrent neural network architecture called the context-ready transformer pre-loads each token with contextual information before it enters a transformer block, delivering faster generation and competitive performance against deeper standard models, according to research posted to arXiv on June 25, 2026 [1]. The architecture is built from a D-layer transformer block and uses a correction network that combines the previous position’s block output — a cached summary of past context — with the current token embedding. This means the token enters the block already contextualized rather than as a raw embedding [1]. At sequential inference, the correction chain makes the architecture a recurrent neural network [1]. For training, the researchers unroll the correction process K times over the full sequence, processing all positions in parallel at each step [1]. A pretrained transformer can be converted to a context-ready model by adding a zero-initialized correction feed-forward network and fine-tuning [1]. The transformer architecture underpins widely used generative pre-trained transformer models such as OpenAI’s GPT series, which are pre-trained on large unlabeled datasets and generate novel content [3]. OpenAI introduced the first GPT model in 2018 and has since released progressively larger versions, culminating in GPT-5 in August 2025 [3][5]. These models have driven rapid adoption of AI chatbots, with ChatGPT reaching 100 million monthly active users within two months of its November 2022 launch [4]. In evaluations, a context-ready transformer with D=5 layers outperformed a 12-layer standard transformer while generating 1.7x faster on an Nvidia A100 GPU [1]. With the correction process unrolled K=10 times, a single-layer model (D=1) beat a 6-layer transformer and achieved a 2.6x inference speedup [1]. Sequential inference matched parallel K=10 to within 0.01 perplexity [1]. The architecture benefited most from wide representations and long contexts [1]. On a pointer-chasing task, a D=1 model trained with backpropagation through time solved all 10 composition levels, while standard transformers showed a staircase-like dependence on depth [1].