Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

A new adaptive method for semantic filtering with large language models addresses four persistent limitations of existing cascade systems, achieving a 90% accuracy target while running 1.6 to 2.0 times faster than the best prior approach across three document corpora, according to research posted to arXiv [1]. The work targets the semantic filter problem: evaluating a natural-language yes/no predicate over a document corpus under a strict accuracy target. Calling a large language model on every document — the oracle — is prohibitive, so production systems pair the oracle with a fast proxy in a cascade. The authors identify four shortcomings in current deployments. Each cascade family commits to a single representation and pipeline, winning only on a narrow query regime. The strongest online proxy relies on a custom-trained bi-encoder over dense embeddings, missing token-level evidence that richer predicates require. Proxies are trained against binary yes/no labels, discarding the oracle’s per-document confidence on boundary cases. And existing calibrations apply a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost [1][2]. The proposed framework tackles each limitation. It composes cascade families adaptively: a lightweight model-free clustering phase runs first, and an online-trained proxy is invoked only when clustering cannot classify all documents confidently. Oracle calls from the clustering phase are reused as training data for the proxy, minimizing the transition cost [3]. The clustering phase itself draws on the Clustering-Sampling-Voting paradigm, which embeds documents, groups them into semantically similar clusters, samples representatives for LLM evaluation, and infers labels for the remainder via voting, with theoretical guarantees that bound the discrepancy between voting results and expected LLM output [4]. For the online proxy, the researchers replace the cosine bi-encoder with a hybrid of off-the-shelf token-aware models that capture fine-grained semantics. The proxy is trained with the oracle’s per-document confidence as a soft label, so it learns to be unsure on documents where the oracle was unsure, rather than being forced toward a binary yes/no decision [2][3]. The calibration component separates proxy uncertainty from small-sample uncertainty and adds a safety margin only where the labeled sample is sparse, reducing the number of documents that must be escalated to the oracle [1][3]. The paper also introduces three novel uses of the oracle’s per-document confidence: as a query-level difficulty compass that explains which cascade family wins on which query, as a lower bound on the minimum oracle calls any proxy-based cascade can make at the target accuracy, and as the soft training label for the proxy [2][3]. Evaluated at a 90% accuracy target on three 10,000-document corpora, the adaptive two-phase method is 1.6 to 2.0 times faster than the best prior method per corpus and meets the accuracy target on 95% of queries. The lower bound derived from oracle confidence indicates a further 4 to 20 times of headroom for future work [1][3]. Prior systems such as ScaleDoc have demonstrated that adaptive cascade mechanisms can reduce expensive LLM invocations by up to 85% while achieving over a 2x end-to-end speedup, underscoring the practical value of the direction [5].