CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward

A new training system called CacheRL achieves 92 percent process accuracy on multi-step tool-calling tasks while requiring 100 times less compute than GPT-5, according to research posted to arXiv on June 12, 2026 [1]. The system, described in a preprint, addresses three challenges in training small agent foundation models: transferring tool-calling knowledge from large models at scale, enabling reinforcement learning without costly live tool execution, and learning robustly from noisy cached environments [1]. CacheRL introduces a hybrid thinking trajectory pipeline that augments agent trajectories with LLM-generated reasoning traces, a three-tier fuzzy cache called CacheAgentLoop that eliminates live execution costs while preserving trajectory fidelity through token-level masking, and a cache-tier-aware reward that dynamically adjusts answer-quality weights [1]. Through iterative supervised fine-tuning and Group Relative Policy Optimization, CacheRL improved the Qwen3-4B-Thinking model's validation reward from 0.43 to 0.78 [1]. On public agentic tool-calling benchmarks, the resulting model achieved competitive performance against frontier models such as GPT-5, which posted 94 percent process accuracy [1]. Ablation studies showed that removing knowledge transfer reduced performance by 41 percent, while cache-aware rewards contributed a 17 percent improvement [1]. The authors noted that reinforcement learning improved training stability but yielded limited gains beyond strong supervised fine-tuning, suggesting that data quality and reward design play a more important role than complex optimization methods in building practical small agent models [1]. The caching approach builds on a growing body of work aimed at accelerating reinforcement learning for tool-calling agents. A separate preprint introduced TVCache, a stateful tool-value cache that organizes cached tool calls into a tool call graph and uses longest-prefix matching for lookups, achieving cache hit rates up to 70 percent and reducing median tool-call execution time by up to 6.9 times across terminal-based, SQL, and video understanding workloads [5]. Another system, Speculative Rollout with Tree-Structured Cache, exploits redundancy across rollouts by organizing past continuations into per-prompt tree-structured caches and using speculative decoding with run-ahead generation, achieving up to 2.08 times rollout speedup across multiple RL algorithms and multi-turn settings [7]. Reward design remains a central difficulty in multi-turn tool-calling. Researchers applying Multi-Turn Group Relative Policy Optimization to customer-service tasks found that naively designed dense per-turn rewards degraded performance by up to 14 percentage points due to misalignment between reward discriminativeness and advantage direction [3]. They introduced Iterative Reward Calibration, a methodology that measures the empirical correlation between each reward tier and task success rather than assigning rewards by intuition, and showed that read-only tool calls should receive zero reward while non-golden state-changing calls should be penalized [3]. A related method, Reward-Conditioned Group Relative Policy Optimization, treats exploration as a controllable steering problem by injecting discrete reward tokens into prompts, explicitly injecting variance to restore non-degenerate group-relative advantages when within-group reward variation is low [4]. The CacheRL preprint appeared on arXiv, the open-access repository of electronic preprints that has hosted more than two million articles since its founding in 1991 and now receives roughly 24,000 submissions per month [11].