CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation

A new framework called CONCORD aims to speed up retrieval-augmented generation on edge devices by treating the cloud as an asynchronous evidence source rather than a continuously synchronized partner, according to a preprint posted to arXiv on June 13, 2026 [1][2]. Retrieval-augmented generation, or RAG, has become a key method for improving language models by pulling in external knowledge during inference [1][2]. As device-cloud collaborative inference makes it possible to run small language models on phones and other edge hardware, a new challenge has emerged: private documents stay on the device while public knowledge lives in the cloud, and privacy or policy rules often block raw document exchange between the two [1][2]. This creates what the authors call a document-isolated dual-end RAG setting [1][2]. Existing approaches in this space depend on frequent remote synchronization and dense evidence transfer, which drags down throughput under real-world latency and bandwidth limits [1][2]. CONCORD, short for an asynchronous sparse aggregation framework, takes a different path. Instead of requiring the cloud to act as a continuously synchronized co-generator, it treats the cloud as an evidence source that arrives asynchronously [1][2]. The framework introduces a mechanism called waiting debt control, which decides at each decoding step whether to keep waiting for remote input based on the observed return on that waiting [1][2]. It also uses a certificate-guided minimal supplementation mechanism that requests only the remote evidence needed to settle the current greedy decision [1][2]. Steps that do consult the cloud preserve the same greedy token as dense dual-end aggregation, while the remaining steps commit locally without any remote evidence [1][2]. In experiments on the Natural Questions and WikiText-2 datasets, CONCORD improved end-to-end throughput over baselines by 1.66× and 2.15×, respectively [1][2]. Per-token communication dropped by more than two orders of magnitude, while answer quality and perplexity remained comparable [1][2]. The paper was submitted to arXiv’s artificial intelligence section. arXiv, which began on August 14, 1991, is an open-access repository of electronic preprints that are moderated but not peer-reviewed [6]. As of November 2024, the repository was receiving about 24,000 articles per month [6]. The platform also hosts arXivLabs, a framework that allows community collaborators to build experimental tools on top of the article record page, such as citation explorers and recommender systems [4][5].