Tree-of-Experience: A Structured Experience-Management Solution for Self-Evolving Agents under Low-Repetition and Implicit-Reward Environments

A team of researchers has proposed Tree-of-Experience, a structured method for managing how large language model agents learn from past actions in environments where tasks rarely repeat and feedback is indirect, according to a paper submitted on 5 June 2026 [1]. The work addresses a gap in current benchmarks for experience-based self-evolution, which typically assume explicit goals, stable task patterns, and clear feedback [1]. The authors instead examine low-repetition tasks with implicit rewards, a setting where past experience is difficult to reuse because feedback is delayed, noisy, and provided only at the outcome level [2]. To ground their investigation, they introduce FinEvolveBench, a temporally controlled benchmark that links daily news-driven financial sentiment predictions to future excess returns [3]. The proposed Tree-of-Experience method organizes, retrieves, validates, and updates agent experience in a structured manner [1]. Experiments across multiple foundation models and self-evolution strategies showed that general-purpose experience mechanisms did not consistently outperform a no-experience baseline, suggesting that reusing experience in low-repetition financial environments is non-trivial [4]. In contrast, the structured approach delivered stronger overall performance under the evaluation protocol [4]. The challenge of learning from limited, indirect feedback echoes broader themes in machine learning, where algorithms must generalize from data without explicit programming [6]. The paper’s focus on self-evolution also draws a conceptual parallel to evolutionary psychology, which examines how cognitive mechanisms adapt to solve recurrent ancestral problems [7]. While the Tree-of-Experience framework operates in a computational domain, its emphasis on selectively retaining and validating experience mirrors the adaptive pressures described in evolutionary theory. A related line of work, DeltaMem, tackles similar memory-management challenges through a dual-tree residual framework that decouples task-level strategy from environment-level knowledge and stores new episodes as compact deltas relative to prior experience [5]. That approach, inspired by neuroscientific memory consolidation, autonomously distills high-frequency convergent paths into new root nodes, allowing the memory structure to self-organize over time [5]. The Tree-of-Experience paper contributes to this growing body of research by demonstrating that effective self-evolution in implicit-reward settings requires agents to selectively retrieve, validate, and update experience rather than directly reuse historical patterns [3].