Express Language Modeling

Researchers have introduced Express, a new tool that converts non-causal attention approximations into causal ones while preserving approximation guarantees, according to a paper posted to arXiv on June 9 [1]. The method improves upon the best known causal attention bounds when paired with the Thinformer approximation [2]. The paper details that Express, combined with Thinformer, achieves a $\log^{3/2}(n)/s$ approximation error for a sequence of length $n$, requiring only $O(s)$ memory and $O(s^2 \log^2(n))$ compression overhead [1][2]. The authors also built an I/O-aware implementation using the Triton language and reported substantial speedups over FlashAttention 2 [1][2]. The team deployed Express to address four specific resource bottlenecks in language modeling: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding [1][2]. The work sits within a broader landscape of modeling languages and structured data representation. The name Express is shared by an older ISO standard, ISO 10303-11, which defines a textual data modeling language for product data exchange [6]. That standard is part of the STEP framework and represents a formal notation for expressing data, information, or knowledge under a consistent set of rules [7]. While the new machine-learning tool operates in a different domain, the reuse of the name highlights the continued importance of structured representation across computer science disciplines. Modeling languages more generally can be graphical or textual, and executable variants are designed to amplify programmer productivity for complex problems such as parallel computing and distributed systems [7]. The systems modeling language SysML, for instance, extends a subset of the Unified Modeling Language to support specification, analysis, design, and verification of systems engineering applications [8]. The Express attention tool similarly aims to amplify efficiency, specifically by reducing the computational and memory footprint of causal attention mechanisms in transformer architectures [1][2]. The paper does not include direct quotations from the authors, and no independent replication studies were available in the research bundle. The claims rest on the preprint's abstract and technical presentation, which have not yet undergone peer review [1][2]. The authors frame Express as a general-purpose conversion tool, meaning it could theoretically be applied to other non-causal attention approximations beyond Thinformer, though the paper's primary results center on that pairing [1][2].