LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling

A new benchmark called LoHoSearch has been introduced to test the limits of long-horizon search agents, presenting a challenge that exceeds the difficulty of human-authored tests. The benchmark comprises 544 human-verified questions across 11 domains and is built using an automated pipeline over a knowledge graph of more than 7 million Wikipedia entities [1][2]. The benchmark was created in response to the rapid saturation of existing search agent tests, such as BrowseComp, where the strongest models have recently surpassed 90% accuracy [1][2]. Researchers note that because these prior benchmarks are predominantly human-authored, annotators cannot systematically maximize the size of the search space or the structural complexity of the questions, creating a difficulty ceiling that is hard to break [1][2]. To overcome this limitation, LoHoSearch uses an automated construction pipeline. This system selects relations with large search spaces from a knowledge graph and assembles them into structurally complex questions, each with a unique answer verified by the knowledge graph [1][2]. The result is a set of 544 questions that have been human-verified for quality [1][2]. When evaluated against the benchmark, even the most capable model achieved an accuracy of only 34.74% [1][2]. The study also found that existing context management strategies, which are designed to help agents handle large amounts of information during a search, provided limited benefit. The best of these strategies yielded a gain of just 6.8%, a far smaller improvement than what has been observed on earlier benchmarks [1][2]. The development of more demanding benchmarks like LoHoSearch comes as the field of artificial intelligence increasingly focuses on agents that can perform complex, multi-step reasoning over vast information spaces. The ability to manage context over a long horizon is critical for tasks that require synthesizing disparate pieces of information, a challenge that remains largely unsolved according to the new results [1][2].