- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces LoHoSearch, a really tough benchmark designed to test how well agents handle long-horizon search tasks. Building search benchmarks today usually comes with three major headaches: high manual costs, trouble making sure there's only one right answer, and difficulty controlling how hard the questions are. To fix this, the authors came up with an automated pipeline built on knowledge graphs.
They used a Wikipedia knowledge graph. Then, to keep the quality high, they ran the data through automated verification, difficulty filtering, and a final human review. In the end, they put together a benchmark with 544 questions across 7 different domains. They tested 16 mainstream models on it, and even the best one only hit 52.02% accuracy—which shows just how challenging this benchmark really is.
Critical strengths
-
It’s super practical and significantly cuts down the cost of building benchmarks. Usually, building tough search benchmarks relies heavily on experts manually designing and verifying questions, which gets really expensive. Instead, this paper uses a knowledge graph to automatically generate questions and controls the difficulty by tweaking the number of candidate entities and constraints. This approach doesn't just save a ton of manual effort—it also makes the whole data pipeline much more scalable. Plus, by combining answer-uniqueness checks from the knowledge graph with automated verification and human review, the authors made sure the generated questions are actually reliable.
-
The experiments are really thorough and validate the benchmark from multiple angles. The authors didn't just test many closed and open-source models; they also dug into factors like context management, tool-call frequency, evidence gathering, and parallel sampling. For example, their experiments show that in successful runs with DeepSeek-V4-Flash, LoHoSearch takes an average of 61 tool calls, compared to just 35 on BrowseComp. That gives us pretty direct proof of how tough its long-horizon search tasks really are.
Critical weaknesses
- Missing evaluations on mainstream agent harnesses, which limits how well the findings translate to real-world agent systems.
the paper mostly evaluates different models using a unified Search and Browse toolset, sticking to a standard ReAct framework for the context management experiments. However, it doesn't really explore how different agent harnesses impact final performance. In practice, an agent's capability depends just as much on its harness design as it does on the underlying model. Mainstream systems like Codex and Claude Code have their own built-in workflows for tool use, task planning, context management, and error recovery—all of which can make a huge difference in long-horizon search tasks.
Because of this, simply comparing models in a standardized tool environment doesn't give us the full picture of how real-world agent systems actually perform. Especially for long-horizon search and context management—which this paper heavily emphasizes—the harness itself is likely a major factor. I would suggest adding experiments with a few representative agent harnesses, ideally keeping the base model and toolset as consistent as possible to isolate the harness's impact on search performance. This would also help show that the benchmark doesn't just measure raw model capability, but can also effectively evaluate deployed agent systems in the wild.
Questions for authors
-
On the implementation details of the Agent Harness. In Section 3.1, the paper outlines a unified setup for the Search and Browse tools, and Section 3.3 mentions using the ReAct framework. However, I’d love to get more clarity on a few things: did the authors build the agent harness for the main experiments from scratch, or is it based on an existing open source framework? Also, do all models run on the exact same agent execution logic? It would be great if they could share more implementation details—like tool call scheduling, search step limits, retry mechanisms on failure, and context management—just to make the experiments easier to reproduce.
-
On comparing against mainstream Agent Harnesses. Have the authors considered testing LoHoSearch with popular agent harnesses like Codex or Claude Code? These systems run on execution mechanisms that are quite different from standard ReAct, so they might behave very differently when it comes to long-horizon planning, tool use, and context management. If we could compare different harnesses using the same underlying model and comparable tools, it would really help untangle whether the benchmark's difficulty comes mainly from the base model itself or from the agent's execution framework. Figuring this out is important for understanding how valuable LoHoSearch actually is in real world scenarios.
3 comments
Thank you for the clarification and additional experiments. I encourage the authors to include the Claude Code and Codex evaluation results in the paper, as they would strengthen its practical relevance. I will maintain my original score.
Dear Reviewer NF4F,
Thank you for your follow-up and for taking the time to review our work. We will include the Claude Code and Codex evaluation results in the revised paper.
Best regards,
The Authors
Q1
We implemented the harness for the main experiments ourselves, using a ReAct loop. Within each ReAct turn, the harness executes tool calls sequentially in the order emitted by the model. As described in Appendix D, a single
searchcall can process up to five queries in parallel, and a singlebrowsecall can process up to three URLs in parallel. The harness appends all tool results to the conversation before requesting the next model response. All models share the same loop, tool definitions, and system prompt. The default total context budget is 200K tokens, with 184K for input and 16K for output. The unified evaluation harness sets no limit on the number of tool calls or model requests. It disables tool calls and prompts the model to answer only when the token threshold is reached. Failed model and tool requests are retried separately. Search queries have up to 10 attempts, page retrieval up to 3, and model requests up to 15, including the initial attempt in each case. These retries are intended to prevent service instability from affecting evaluation results. In the context-management experiments, when context usage reaches 80%, Summary compresses the trajectory or Discard-all clears the model and tool history, after which the model continues working on the question. In configurations with Verify, the module evaluates the answer once the model proposes it. If it passes verification, the answer is submitted. If it is rejected, the model receives feedback and continues checking. In the context-management experiments, we cap the number of model calls at 1,000 per question.W1 & Q2
We evaluated DeepSeek-V4.1-Flash-1017 and GLM-5.3 on LoHoSearch using Claude Code and Codex. Both models achieved higher accuracy with either harness than with their respective ReAct baselines, and Claude Code scored higher than Codex for both models. Under both harnesses, search and answering were limited to eight hours per question. The context window was 200K tokens, with automatic context compaction triggered at 90% of the window.
\text{model} & \substack{\text{ReAct baseline}\\\\\text{accuracy}} & \text{Claude Code accuracy} & \substack{\text{Gain over ReAct}\\\\\text{(percentage points)}} & \text{Codex accuracy} & \substack{\text{Gain over ReAct}\\\\\text{(percentage points)}} \\\\ \hline \text{DeepSeek‑V4.1‑Flash-1017} & \text{31.34\\%} & \text{54.50\\%} & \text{+23.16} & \text{47.52\\%} & \text{+16.18} \\\\ \text{GLM‑5.3} & \text{20.96\\%} & \text{47.15\\%} & \text{+26.19} & \text{43.66\\%} & \text{+22.70} \end{array}$$ Mean assistant turns per correct trajectory $$\begin{array}{lccccc} & \substack{\text{ReAct baseline mean}\\\\\text{turns}} & \text{Claude Code mean turns} & \substack{\text{Difference from ReAct}\\\\\text{(turns)}} & \text{Codex mean turns} & \substack{\text{Difference from ReAct}\\\\\text{(turns)}} \\\\ \hline \text{DeepSeek‑V4.1‑Flash-1017} & \text{38.89} & \text{99.20} & \text{+60.31} & \text{111.46} & \text{+72.57} \\\\ \text{GLM‑5.3} & \text{17.74} & \text{71.71} & \text{+53.97} & \text{175.56} & \text{+157.82} \end{array}$$