Agentic Episodic Control

A new architecture called Agentic Episodic Control (AEC) integrates large language models into episodic reinforcement learning to address long-standing problems of data efficiency and generalization, according to a paper posted on arXiv [1][2]. The paper, submitted in June 2025 and revised in June 2026, proposes AEC as a way to overcome two key limitations of prior episodic RL methods: a representation bottleneck from shallow encoders, and a retrieval dilemma where memory is accessed indiscriminately [1][2]. AEC uses an LLM-based semantic augmenter to generate semantic representations from raw observations, and a critical state recognizer to selectively retrieve valuable experiences [2]. This transforms memory usage from passive similarity matching into strategic, context-aware recall [2]. In tests across five BabyAI-Text environments, AEC achieved 2-6x higher data efficiency than baselines and was the only method to solve complex tasks like UnlockLocal with over 90% success [1][2]. The architecture also demonstrated strong cross-task and cross-environment generalization, maintaining performance under distribution shifts [2]. The concept of episodic memory in cognitive science was coined by Endel Tulving in 1972, referring to the recollection of personal experiences tied to specific times and places [3]. Tulving identified three key properties: a subjective sense of time, connection to the self, and autonoetic consciousness [3]. Psychologists later expanded the concept to include mental time travel, the capacity to mentally reconstruct past events and imagine future scenarios [4]. AEC draws on these principles by giving an agent a memory system that recalls strategically rather than by simple similarity [2]. Reinforcement learning, one of the three basic machine learning paradigms alongside supervised and unsupervised learning, trains an agent through interactions with an environment to maximize a reward signal [6]. The field has long grappled with the exploration-exploitation dilemma, where an agent must balance trying new actions against using known strategies [6]. Episodic RL methods attempt to improve sample efficiency by storing past experiences in external memory modules, but AEC's authors argue those methods have been held back by representation and retrieval shortcomings [2]. By combining LLM-derived semantic priors with reinforcement learning, AEC produces agents that are more sample-efficient and adaptable [2]. The work was led by Xidong Yang [1].