Not All Skills Help: Measuring and Repairing Agent Knowledge

A new framework called ASSAY separates the generation of agent skills from their curation, showing that many accumulated skills can harm performance on specific tasks even when they appear beneficial overall, according to research submitted in 2026 [1]. Large language model agents can improve over time by storing natural-language skills drawn from experience, without requiring weight updates. But current systems rely entirely on the LLM’s own judgment to decide which skills to retain and when to apply them [1]. The researchers argue this approach conflates two distinct functions. Generating a skill from experience is a creative act that judgment handles well, while determining whether that skill actually helps requires empirical evidence across many tasks [2]. Creativity, broadly understood, involves generating novel and valuable ideas or solutions, a capacity that has been studied across psychology, business, and cognitive science [4]. To test this separation, the team measured per-skill causal contributions using randomized masking. They found pervasive causal heterogeneity: individual skills routinely helped on some task types while hurting on others, yet their opposing effects canceled out in aggregate, making the problem invisible to global curation methods [2]. ASSAY addresses this by computing a per-skill causal attribution on a small development set, restructuring the skill library offline, and suppressing skills with a negative predicted effect for each test task [1]. The framework was evaluated across seven base models from four providers on two benchmarks, AppWorld and tau-bench [2]. On AppWorld’s hardest split, DeepSeek-V3 reached 69.3% task-goal completion, a 47.4% relative improvement and a new state of the art among all published methods, including weight-tuned approaches [1]. On tau-bench retail, GPT-4.1 improved by 8.7% relative, moving past o4-mini, o1, and GPT-4.5 on the public leaderboard without any weight modification [2]. Ablation analysis traced the dominant gain to per-task masking, confirming that the bottleneck lies in matching skills to tasks at inference time rather than removing bad skills globally [1]. The code for ASSAY has been released on GitHub [2].