acceptodds
Under review as a conference paper at ICLR 2027

LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling

Abstract

Rapid advances in the agentic capabilities of large language models have spurred the development of increasingly challenging benchmarks for deep search agents. Constructing high-quality questions for these benchmarks often relies on labor-intensive expert authoring to ensure factual validity and sufficient difficulty. To automate this process, we introduce LoHoSearch (Long-Horizon Search Agents), a benchmark comprising 544 automatically verified and human-reviewed questions across seven domains. Its automated pipeline samples tree- and graph-structured subgraphs from a Wikipedia-derived knowledge graph (KG) containing 7.62 million entities. The KG enables systematic control over candidate-set sizes and constraint structures to shape question difficulty, while supporting answer uniqueness checks across the full graph to help ensure question validity. The sampled subgraphs are then converted into natural-language questions, which undergo automated validation, difficulty filtering, and human review. Under a standardized evaluation setup, even the best-performing model achieves only 52.02% accuracy. With DeepSeek-V4-Flash, successful trajectories use 1.7× as many tool calls on average as on BrowseComp, highlighting the long-horizon search and reasoning demands of our questions. Existing context-management strategies yield a maximum accuracy gain of 6.8 percentage points, substantially smaller than the 14.03-point gain on BrowseComp. These findings highlight the need for more effective context-management methods. LoHoSearch provides a demanding testbed for evaluating evidence discovery, multi-constraint reasoning, and context management over extended search trajectories.

Primary area:
datasets and benchmarks
Keywords:
Deep Search Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NF4FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LoHoSearch, a really tough benchmark designed to test how well agents handle long-horizon search tasks. Building search benchmarks today usually comes with three major headaches: high manual costs, trouble making sure there's only one right answer, and difficulty controlling how hard the questions are. To fix this, the authors came up with an automated pipeline built on knowledge graphs.

They used a Wikipedia knowledge graph. Then, to keep the quality high, they ran the data through automated verification, difficulty filtering, and a final human review. In the end, they put together a benchmark with 544 questions across 7 different domains. They tested 16 mainstream models on it, and even the best one only hit 52.02% accuracy—which shows just how challenging this benchmark really is.

Critical strengths
  1. It’s super practical and significantly cuts down the cost of building benchmarks. Usually, building tough search benchmarks relies heavily on experts manually designing and verifying questions, which gets really expensive. Instead, this paper uses a knowledge graph to automatically generate questions and controls the difficulty by tweaking the number of candidate entities and constraints. This approach doesn't just save a ton of manual effort—it also makes the whole data pipeline much more scalable. Plus, by combining answer-uniqueness checks from the knowledge graph with automated verification and human review, the authors made sure the generated questions are actually reliable.

  2. The experiments are really thorough and validate the benchmark from multiple angles. The authors didn't just test many closed and open-source models; they also dug into factors like context management, tool-call frequency, evidence gathering, and parallel sampling. For example, their experiments show that in successful runs with DeepSeek-V4-Flash, LoHoSearch takes an average of 61 tool calls, compared to just 35 on BrowseComp. That gives us pretty direct proof of how tough its long-horizon search tasks really are.

Critical weaknesses
  1. Missing evaluations on mainstream agent harnesses, which limits how well the findings translate to real-world agent systems.

the paper mostly evaluates different models using a unified Search and Browse toolset, sticking to a standard ReAct framework for the context management experiments. However, it doesn't really explore how different agent harnesses impact final performance. In practice, an agent's capability depends just as much on its harness design as it does on the underlying model. Mainstream systems like Codex and Claude Code have their own built-in workflows for tool use, task planning, context management, and error recovery—all of which can make a huge difference in long-horizon search tasks.

Because of this, simply comparing models in a standardized tool environment doesn't give us the full picture of how real-world agent systems actually perform. Especially for long-horizon search and context management—which this paper heavily emphasizes—the harness itself is likely a major factor. I would suggest adding experiments with a few representative agent harnesses, ideally keeping the base model and toolset as consistent as possible to isolate the harness's impact on search performance. This would also help show that the benchmark doesn't just measure raw model capability, but can also effectively evaluate deployed agent systems in the wild.

Questions for authors
  1. On the implementation details of the Agent Harness. In Section 3.1, the paper outlines a unified setup for the Search and Browse tools, and Section 3.3 mentions using the ReAct framework. However, I’d love to get more clarity on a few things: did the authors build the agent harness for the main experiments from scratch, or is it based on an existing open source framework? Also, do all models run on the exact same agent execution logic? It would be great if they could share more implementation details—like tool call scheduling, search step limits, retry mechanisms on failure, and context management—just to make the experiments easier to reproduce.

  2. On comparing against mainstream Agent Harnesses. Have the authors considered testing LoHoSearch with popular agent harnesses like Codex or Claude Code? These systems run on execution mechanisms that are quite different from standard ReAct, so they might behave very differently when it comes to long-horizon planning, tool use, and context management. If we could compare different harnesses using the same underlying model and comparable tools, it would really help untangle whether the benchmark's difficulty comes mainly from the base model itself or from the agent's execution framework. Figuring this out is important for understanding how valuable LoHoSearch actually is in real world scenarios.

3 comments

Q1

We implemented the harness for the main experiments ourselves, using a ReAct loop. Within each ReAct turn, the harness executes tool calls sequentially in the order emitted by the model. As described in Appendix D, a single search call can process up to five queries in parallel, and a single browse call can process up to three URLs in parallel. The harness appends all tool results to the conversation before requesting the next model response. All models share the same loop, tool definitions, and system prompt. The default total context budget is 200K tokens, with 184K for input and 16K for output. The unified evaluation harness sets no limit on the number of tool calls or model requests. It disables tool calls and prompts the model to answer only when the token threshold is reached. Failed model and tool requests are retried separately. Search queries have up to 10 attempts, page retrieval up to 3, and model requests up to 15, including the initial attempt in each case. These retries are intended to prevent service instability from affecting evaluation results. In the context-management experiments, when context usage reaches 80%, Summary compresses the trajectory or Discard-all clears the model and tool history, after which the model continues working on the question. In configurations with Verify, the module evaluates the answer once the model proposes it. If it passes verification, the answer is submitted. If it is rejected, the model receives feedback and continues checking. In the context-management experiments, we cap the number of model calls at 1,000 per question.

W1 & Q2

We evaluated DeepSeek-V4.1-Flash-1017 and GLM-5.3 on LoHoSearch using Claude Code and Codex. Both models achieved higher accuracy with either harness than with their respective ReAct baselines, and Claude Code scored higher than Codex for both models. Under both harnesses, search and answering were limited to eight hours per question. The context window was 200K tokens, with automatic context compaction triggered at 90% of the window.

\text{model} & \substack{\text{ReAct baseline}\\\\\text{accuracy}} & \text{Claude Code accuracy} & \substack{\text{Gain over ReAct}\\\\\text{(percentage points)}} & \text{Codex accuracy} & \substack{\text{Gain over ReAct}\\\\\text{(percentage points)}} \\\\ \hline \text{DeepSeek‑V4.1‑Flash-1017} & \text{31.34\\%} & \text{54.50\\%} & \text{+23.16} & \text{47.52\\%} & \text{+16.18} \\\\ \text{GLM‑5.3} & \text{20.96\\%} & \text{47.15\\%} & \text{+26.19} & \text{43.66\\%} & \text{+22.70} \end{array}$$ Mean assistant turns per correct trajectory $$\begin{array}{lccccc} & \substack{\text{ReAct baseline mean}\\\\\text{turns}} & \text{Claude Code mean turns} & \substack{\text{Difference from ReAct}\\\\\text{(turns)}} & \text{Codex mean turns} & \substack{\text{Difference from ReAct}\\\\\text{(turns)}} \\\\ \hline \text{DeepSeek‑V4.1‑Flash-1017} & \text{38.89} & \text{99.20} & \text{+60.31} & \text{111.46} & \text{+72.57} \\\\ \text{GLM‑5.3} & \text{17.74} & \text{71.71} & \text{+53.97} & \text{175.56} & \text{+157.82} \end{array}$$

Thank you for the clarification and additional experiments. I encourage the authors to include the Claude Code and Codex evaluation results in the paper, as they would strengthen its practical relevance. I will maintain my original score.

Dear Reviewer NF4F,

Thank you for your follow-up and for taking the time to review our work. We will include the Claude Code and Codex evaluation results in the revised paper.

Best regards,

The Authors

Reviewer_mv6uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper constructs 544 search questions across seven domains from the Wikipedia hyperlink graph, using tree/graph sampling to control candidate ambiguity and constraint structure, followed by textual verification, agent-based filtering, and human review. The highest accuracy under a shared tool configuration is 52.02%, and the paper analyzes evidence acquisition, context management, and answer selection.

Input text to the LLM " explain this paper to me".

Critical strengths
  1. The use of graph-wide candidate sets and structural uniqueness to generate questions is clearly designed; compared with relying solely on manually written questions, it provides a parameterizable construction approach
  2. Quality verification does not rely solely on graph structure: it includes constraint-by-constraint checks, searches for alternative answers, human review, and a ranking check on a subset of 385 questions with high-confidence uniqueness.
Critical weaknesses
  1. The empirical evidence for the core claim of “controllable difficulty” remains largely descriptive. The trees and graphs already differ in their numbers of nodes and edges; Figure 5 is also affected by entity popularity and differences in entity extraction methods between the two benchmarks.

  2. Reproducibility details and the costs of scaling remain unclear. The main text refers only to “multiple” rounds of difficulty filtering, without specifying the number of rounds or exclusion thresholds; the graph snapshot date, an explicit retrieval backend, and the total tool budget are missing.

Questions for authors

Could the authors conduct ablations of structure/candidate thresholds with popularity, node counts, and budgets matched, and add statistics on complete successful and failed trajectories from strong models?

5 comments

W1 & Q1

1. Ablation of candidate-set size

We divided the 282 tree-structured questions into two groups based on the mean candidate-set size of each question. We matched questions with the same node count whose mean in-degree fell in the same interval of width 3. Each pair contained one question from each group, with a difference in mean candidate-set size of at least 5. Each question was used at most once. This yielded two sets of 88 questions each. Both sets averaged 7.77 nodes, and their average entity popularity was 31.61 versus 31.60, while their mean candidate-set sizes were 16.79 versus 8.67. We recomputed the evaluation results for all 16 models on the matched subsets. The larger-candidate-set group had a mean accuracy of 24.25% across the 16 models, 8.1 percentage points below the smaller-candidate-set group. For GPT-6 Astra, the strongest model in our experiments, accuracy was 13.07 percentage points lower and correct trajectories were 5.37 turns longer on average in the larger-candidate-set group. This experiment shows that when the other conditions are broadly comparable, questions with larger candidate sets are more difficult.

\text{Model} & \substack{\text{Accuracy, larger}\\\\\text{search space}} & \substack{\text{Accuracy, smaller}\\\\\text{search space}} & \substack{\text{Smaller-group}\\\\\text{advantage (percentage}\\\\\text{points)}} \\\\ \hline \text{GPT‑6 Astra} & \text{55.68\\%} & \text{68.75\\%} & \text{+13.07} \\\\ \text{GPT‑5.6 Sol} & \text{53.98\\%} & \text{68.75\\%} & \text{+14.77} \\\\ \text{GPT‑5.5} & \text{34.09\\%} & \text{48.30\\%} & \text{+14.20} \\\\ \text{Gemini 3.8 Flash} & \text{53.98\\%} & \text{57.39\\%} & \text{+3.41} \\\\ \text{Gemini 3.1 Pro} & \text{10.80\\%} & \text{12.50\\%} & \text{+1.70} \\\\ \text{Kimi‑K3} & \text{42.61\\%} & \text{57.95\\%} & \text{+15.34} \\\\ \text{Kimi‑K2.6} & \text{12.50\\%} & \text{14.77\\%} & \text{+2.27} \\\\ \text{Claude Opus 5} & \text{24.43\\%} & \text{50.57\\%} & \text{+26.14} \\\\ \text{Claude Opus 4.7} & \text{10.23\\%} & \text{17.05\\%} & \text{+6.82} \\\\ \text{Claude Opus 4.6} & \text{20.45\\%} & \text{21.59\\%} & \text{+1.14} \\\\ \text{DeepSeek‑V4.1‑Flash} & \text{23.30\\%} & \text{36.93\\%} & \text{+13.63} \\\\ \text{DeepSeek‑V4‑Flash} & \text{10.80\\%} & \text{13.64\\%} & \text{+2.84} \\\\ \text{GLM‑5.3} & \text{14.77\\%} & \text{33.52\\%} & \text{+18.75} \\\\ \text{GLM‑5.1} & \text{13.64\\%} & \text{7.95\\%} & \text{−5.69} \\\\ \text{MiniMax‑M3} & \text{3.41\\%} & \text{5.11\\%} & \text{+1.70} \\\\ \text{MiniMax‑M2.7} & \text{3.41\\%} & \text{2.84\\%} & \text{−0.57} \\\\ \text{Average across 16 models (equal weight)} & \text{24.25\\%} & \text{32.35\\%} & \text{+8.10} \end{array}$$

2. Ablation of structure

From the tree- and graph-structured questions in LoHoSearch, we selected pairs with the closest average node counts and average entity candidate-space sizes. This yielded 48 pairs, comprising 96 questions. For tree- versus graph-structured questions, average entity popularity was 25.05 versus 20.05, candidate-space size was 11.21 versus 5.42, and node count was 8.9 versus 9.9. We reevaluated all 16 models on these two sets. Their mean accuracy on graph-structured questions was 22.01%, 15.04 percentage points below that on tree-structured questions. For GPT-6 Astra, the strongest model in our experiments, accuracy was 34.38 percentage points lower and correct trajectories were 3.29 turns longer on average in the graph-structured group. Overall, this experiment shows that graph-structured questions are more difficult than tree-structured questions in this matched sample.

\text{Model} & \text{Tree group accuracy} & \substack{\text{General graph group}\\\\\text{accuracy}} & \substack{\text{Tree-group advantage}\\\\\text{(percentage points)}} \\\\ \hline \text{GPT-6 Astra} & \text{75.00\\%} & \text{40.62\\%} & \text{+34.38} \\\\ \text{GPT-5.6 Sol} & \text{71.88\\%} & \text{41.67\\%} & \text{+30.21} \\\\ \text{GPT-5.5} & \text{51.04\\%} & \text{26.04\\%} & \text{+25.00} \\\\ \text{Gemini-3.8-Flash} & \text{61.46\\%} & \text{42.71\\%} & \text{+18.75} \\\\ \text{Gemini-3.1-Pro} & \text{17.71\\%} & \text{19.79\\%} & \text{−2.08} \\\\ \text{Kimi-K3} & \text{63.54\\%} & \text{34.38\\%} & \text{+29.16} \\\\ \text{Kimi-K2.6} & \text{25.00\\%} & \text{10.42\\%} & \text{+14.58} \\\\ \text{Claude-Opus-5} & \text{51.04\\%} & \text{42.71\\%} & \text{+8.33} \\\\ \text{Claude-Opus-4.7} & \text{27.08\\%} & \text{3.12\\%} & \text{+23.96} \\\\ \text{Claude-Opus-4.6} & \text{25.00\\%} & \text{13.54\\%} & \text{+11.46} \\\\ \text{DeepSeek-V4.1-Flash} & \text{34.38\\%} & \text{26.04\\%} & \text{+8.33} \\\\ \text{DeepSeek-V4-Flash} & \text{19.79\\%} & \text{8.33\\%} & \text{+11.46} \\\\ \text{GLM-5.3} & \text{32.29\\%} & \text{22.92\\%} & \text{+9.37} \\\\ \text{GLM-5.1} & \text{21.88\\%} & \text{14.58\\%} & \text{+7.30} \\\\ \text{MiniMax-M3} & \text{10.42\\%} & \text{3.12\\%} & \text{+7.30} \\\\ \text{MiniMax-M2.7} & \text{5.21\\%} & \text{2.08\\%} & \text{+3.13} \\\\ \text{Average across 16 models} & \text{37.04\\%} & \text{22.01\\%} & \text{+15.04} \end{array}$$ **Statistics for complete successful and unsuccessful GPT-6 Astra trajectories** $$\begin{array}{lccccc} \text{Group} & \substack{\text{Assistant turns (mean}\\\\\text{/ median)}} & \substack{\text{Search tool calls}\\\\\text{(mean / median)}} & \substack{\text{Search queries (mean /}\\\\\text{median)}} & \substack{\text{Browse tool calls}\\\\\text{(mean / median)}} & \substack{\text{Browse URLs (mean /}\\\\\text{median)}} \\\\ \hline \text{All questions} & \text{29.40 / 25.5} & \text{20.60 / 16} & \text{75.93 / 56} & \text{7.82 / 7} & \text{20.27 / 17} \\\\ \text{Correctly answered questions} & \text{22.75 / 19} & \text{14.43 / 11} & \text{52.03 / 40} & \text{7.34 / 6} & \text{19.11 / 16} \\\\ \text{Incorrectly answered questions} & \text{36.46 / 38} & \text{27.14 / 26} & \text{101.29 / 97} & \text{8.32 / 7} & \text{21.50 / 18} \end{array}$$ **W2** - During difficulty filtering, the DeepSeek-V3.2 search agent independently attempted each question five times. We removed any question it answered correctly at least once. - We constructed the graph from the March 2025 full English Wikipedia dump and obtained entity types through the Wikidata SPARQL endpoint. - We used Serper for search in our experiments. Standard ReAct is governed solely by the context length, with no additional budget limits on tool or model calls. When the input reaches 184K tokens, the harness stops tool calls and prompts the model to answer. - To generate question–answer pairs, we sample subgraphs from a large, prebuilt knowledge graph. This makes scaling far less costly than traditional manual construction. We have used the same pipeline to synthesize more than 10,000 examples in batches for model training, and we plan to publicly release a subset of these question–answer pairs.

Dear Reviewer mv6u,

Thank you again for your feedback. We want to follow up and ask whether our response addressed your concerns. If you have any remaining questions, we would be happy to discuss them further.

Best regards,

The Authors

Reviewer_KN1yOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LoHoSearch, a benchmark of 544 questions across seven domains for evaluating long-horizon search agents. The authors construct questions from tree- and graph-structured subgraphs of a Wikipedia-derived knowledge graph, followed by automated verification, difficulty filtering, and human review. Experiments show that the best-performing model achieves only 52.02% accuracy on the benchmark.

Critical strengths
  1. The paper is clearly presented, with solid experiments and comprehensive evaluation and analysis.
  2. The proposed pipeline addresses key challenges in constructing benchmarks for long-horizon search tasks, enabling scalable generation of high-quality datasets with reduced human involvement.
  3. The proposed framework is well designed and scalable, and its underlying ideas could be applied to benchmark construction in other domains.
  4. This paper also introduces a new benchmark, which poses a significant challenge to state-of-the-art models.
Critical weaknesses
  1. The claimed scalability of the proposed pipeline is not sufficiently demonstrated. Despite using an automated framework, LoHoSearch contains only 544 questions, substantially fewer than BrowseComp, which is constructed manually and contains 1,266 instances.
  2. The paper reports only 2,000 initial question–answer pairs from a knowledge graph containing 7.62 million entities. This raises concerns about the framework’s practical scalability and whether it can be used to generate substantially larger datasets.
  3. The models are evaluated using default reasoning effort, leaving it unclear how much their performance would improve with higher reasoning effort. Given the paper’s emphasis on benchmark difficulty, the authors should also report results under the highest reasoning effort, for example for DeepSeek-V4.1-Flash, and report the gains over the default setting.
Questions for authors
  1. Why Gemini-3.1-Pro performs substantially worse than Gemini-3.8-Flash, Claude-Opus-4.7 underperforms Claude-Opus-4.6, and both MiniMax models score below 3%? Is it because of the calibration error?
  2. How do you ensure that the generated questions remain natural when many constraints are combined? Could you clarify how naturalness is assessed during human review?
  3. Have you considered how time-sensitive facts might affect the benchmark? If a question’s answer changes or new valid answers emerge over time, how would you ensure that the benchmark remains valid? (Just a random thought, not critical to the review)
  4. Given that models may have encountered the underlying Wikipedia content during pretraining, why do they still struggle with these questions? How would performance change if models first generated supporting facts for each condition from their internal knowledge and then combined them to answer the question?
  5. Why are the calibration errors of some models so high, and why do they vary substantially even within the same model family? Given their similar accuracy, could GPT-5.6 Sol with Verify outperform GPT-6 Astra’s baseline?
  6. What is the effect of using Verify alone, without Summary or Discard-all?
8 comments

W1 & W2

  • The 544 questions released in LoHoSearch underwent automated verification, difficulty filtering, and human review. The dataset is comparable in size to other search-agent benchmarks. BrowseComp contains 1,266 questions, while other benchmarks for multi-step web search and browsing typically contain several hundred questions. For example, WebWalkerQA contains 680 questions, DeepSearchQA 900, and WideSearch 200.
  • We randomly sampled 2,000 subgraphs from the full graph while maintaining coverage across domains and difficulty levels. We have used the same pipeline to synthesize more than 10,000 question–answer pairs and used the resulting data for model training. Producing these training data demonstrates the scalability of the pipeline. We will make a subset of the training data publicly available to support research in this area.

W3

For some models in our evaluation, including Kimi-K3 and GLM-5.3, the default reasoning effort is already set to the maximum. For models whose default is below the maximum, the effect of increasing reasoning effort varies by model. We compared the default and maximum settings for DeepSeek-V4.1-Flash-1017 and Claude Opus 4.6. For DeepSeek-V4.1-Flash-1017, accuracy rose from 31.34% to 35.02%, and calibration error fell from 50.87% to 50.06%. For Claude Opus 4.6, accuracy fell from 15.62% to 12.78%, and calibration error fell from 31% to 28.69%.

Q1

Calibration error and accuracy measure different aspects of model performance. Calibration error does not directly cause poorer performance.

  • Regarding the substantially lower performance of Gemini-3.1-Pro relative to Gemini-3.8-Flash, we observed that Gemini-3.1-Pro tended to end its searches earlier on LoHoSearch, with less exploration and verification in its trajectories. It averaged 9.59 turns, the lowest among all evaluated models.
  • Claude Opus 4.7's lower performance relative to Claude Opus 4.6 is consistent with the trend reported in their official BrowseComp results. Claude Opus 4.7 scored 79.3%, considerably below Claude Opus 4.6's 84%. We believe this reflects a weakness of Claude Opus 4.7 in agentic search.
  • We observed that both MiniMax models had difficulty identifying key entities and often selected entities that matched only some of the clues. For example, MiniMax-M2.7 averaged 84.68 turns, the highest among all evaluated models.

Q2

When generating question–answer pairs, we prompted the LLM to phrase each question as naturally and fluently as possible. Human reviewers check each question for readability, whether the constraints fit together naturally, whether any wording is redundant, and whether revisions preserve the original factual constraints. We also used Qwen3-4B-Base to compare question perplexity across the two benchmarks. For each question, we randomly selected a 64-token span, used the first 32 tokens as context, and scored the remaining 32. Perplexity was 9.91 for LoHoSearch and 12.62 for BrowseComp. Under this scoring setup, LoHoSearch questions had lower perplexity than BrowseComp questions. This result demonstrates the linguistic fluency of the LoHoSearch questions.

Q3

Temporal change can affect answer correctness in a benchmark. Some benchmarks, such as LiveSearchBench, focus on model performance on highly time-sensitive questions. LoHoSearch has a different focus, and its construction primarily uses relations and entities that remain stable over time.

During question construction, reviewers verified each constraint and actively searched for alternative answers. If the reference answer failed to satisfy the question or another answer satisfied all constraints, reviewers revised or discarded the question to safeguard answer correctness and uniqueness as far as possible.

Q4

LoHoSearch questions combine constraints with large candidate spaces and complex subgraph structures. The clues have complex dependencies. Even if a model recalls some relevant facts, it may still lack the key information needed to connect the clues and rule out other candidates. As reported in Section 3.4.1, providing the pages needed to solve the questions directly raises accuracy to 92.92% for DeepSeek-V4-Flash and 98.07% for GPT-5.5. Both results point to the same difficulty: locating and verifying the key facts that distinguish among candidates.

We compared four methods on 100 questions using DeepSeek-V4.1-Flash. Direct answering, single-round candidate aggregation, and multi-round candidate aggregation achieved mean scores of 1.50%, 1.50%, and 2.00%, respectively, while open search achieved 39.00%. In the multi-round method, the model first generated a list of candidates for each constraint. It then checked the candidates against the full question over multiple rounds, eliminated those that failed to meet the constraints, updated the ranking, and answered based on the remaining candidates. These results show that the model struggles to perform well on LoHoSearch using its internal knowledge alone.

Q5

GPT-5.6 Sol was more overconfident than GPT-6 Astra. On the full 544-question LoHoSearch benchmark, the two models achieved similar ReAct accuracy (51.7% vs. 52.0%), but GPT-5.6 Sol had higher expected calibration error (41.8% vs. 27.7%). Among answers with reported confidence of at least 80%, GPT-5.6 Sol was 10.6 percentage points less accurate than GPT-6 Astra (55.5% vs. 66.1%), even though their mean reported confidence was similar (95.4% vs. 96.4%). GPT-5.6 Sol reported confidence of at least 80% more often than GPT-6 Astra (92.5% vs. 77.6% of questions). About 95% of the difference in calibration error came from this high-confidence group.

We tested whether GPT-5.6 Sol could outperform the GPT-6 Astra ReAct baseline with Verify enabled. Verify checked draft answers against the model’s search history and prompted further search when it rejected a draft. We are applying Verify to the original ReAct trajectories on the full LoHoSearch benchmark.

\text{Model} & \text{Verify trigger rate} & \substack{\text{ReAct accuracy on}\\\\\text{rejected items}} & \text{Accuracy after REJECT} & \text{Overall ReAct accuracy} & \substack{\text{Overall accuracy with}\\\\\text{Verify}} \\\\ \hline \text{DeepSeek-V4.1-Flash} & \text{544/544 (100\\%)} & \text{124/542 (22.9\\%)} & \text{135/542 (24.9\\%)} & \text{124/544 (22.8\\%)} & \text{135/544 (24.8\\%)} \\\\ \text{GPT-6 Astra} & \text{542/544 (99.6\\%)} & \text{25/41 (61.0\\%)} & \text{25/41 (61.0\\%)} & \text{283/544 (52.0\\%)} & \text{283/544 (52.0\\%)} \\\\ \text{GPT-5.6 Sol} & \text{543/544 (99.8\\%)} & \text{279/533 (52.3\\%)} & \text{260/533 (48.8\\%)} & \text{281/544 (51.7\\%)} & \text{262/544 (48.2\\%)} \end{array}$$ On the full 544-question LoHoSearch benchmark, GPT-5.6 Sol scored 48.2% with Verify, 3.9 percentage points below the GPT-6 Astra ReAct baseline. Verify rejected a draft on 533 of the 543 questions it checked. On those 533 questions, further search led to 19 fewer correct answers, lowering GPT-5.6 Sol’s overall accuracy from 51.7% to 48.2%. **Q6** Using Verify alone produced small changes in accuracy, with gains for some models and a slight decline for another. We evaluated all four models on the same questions with and without Verify. For each model, both settings shared the same search trajectory up to the first draft. When Verify rejected a draft, the model continued searching. This helped DeepSeek-V4.1-Flash correct four answers. For GPT-6.1 Sol, further search led the model to replace one correct answer with an incorrect one. These results show that the benefit of further search depends on how often it corrects errors and how often it introduces them. $$\begin{array}{lccccc} \text{Model} & \text{Verify trigger rate} & \substack{\text{ReAct accuracy on}\\\\\text{rejected items}} & \text{Accuracy after REJECT} & \text{Overall ReAct accuracy} & \substack{\text{Overall accuracy with}\\\\\text{Verify}} \\\\ \hline \text{GPT-6 Astra} & \text{91/100 (91\\%)} & \text{17/26 (65.4\\%)} & \text{17/26 (65.4\\%)} & \text{59\\%} & \text{59\\%} \\\\ \text{GPT-5.6 Sol} & \text{98/100 (98\\%)} & \text{44/98 (44.9\\%)} & \text{47/98 (48.0\\%)} & \text{44\\%} & \text{47\\%} \\\\ \text{GPT-6.1 Sol} & \text{92/100 (92\\%)} & \text{23/33 (69.7\\%)} & \text{22/33 (66.7\\%)} & \text{57\\%} & \text{56\\%} \\\\ \text{DeepSeek-V4.1-Flash} & \text{66/100 (66\\%)} & \text{37/65 (56.9\\%)} & \text{41/65 (63.1\\%)} & \text{41\\%} & \text{45\\%} \end{array}$$

Thank you for the clarification and additional experiments. Given the popularity and page-length criteria for entity selection, could you estimate the maximum number of valid QA pairs the full 7.62M entities could yield?

Comments on the paper

2 comments
Area_Chair_cydfOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

The authors have provided the rebuttal. Please have a look and let the authors know if you need further clarification or still have follow-up questions.

Thanks, Your AC

AuthorsOpenReview ↗

For AC, SAC and PC: Summary of Rebuttal Process

Dear Area Chair, Senior Area Chairs, and Program Chairs,

Thank you for considering LoHoSearch. To support your final assessment, we summarize the reviewers’ main comments, the additional evidence provided during the rebuttal, and the discussion outcomes.

The reviews highlighted three strengths: (1) a graph-based, parameterizable construction pipeline that reduces reliance on manual question design (KN1y, mv6u, NF4F), (2) multi-stage quality assurance combining automated verification and human review (mv6u, NF4F), and (3) a challenging benchmark supported by comprehensive analyses of evidence gathering and context management (KN1y, NF4F).

During the rebuttal, we strengthened the evidence in three areas:

1. Support for difficulty control (mv6u).

To address the question of whether our framework supports difficulty control, we added experiments examining how candidate-set size and question structure relate to model performance. Across models, larger candidate sets and graph-structured questions were generally associated with lower accuracy, providing additional empirical support for using construction parameters to adjust question difficulty.

2. Broader agent evaluation and stronger inference settings (NF4F, KN1y).

To assess practical relevance and difficulty under stronger inference settings, we added evaluations with Claude Code and Codex (NF4F) and at maximum reasoning effort (KN1y). Both harnesses achieved higher accuracy than the corresponding ReAct baselines, while the benchmark remained challenging under the tested settings. We also compared closed-book methods with open search, analyzed calibration and the effects of Verify, and reported statistics on successful and failed trajectories to address questions about model behavior (KN1y, mv6u).

3. Scalability, quality assurance, and reproducibility (KN1y, mv6u, NF4F).

To address questions about scalability, we reported our use of the pipeline to generate additional QA pairs for training and estimated how many pairs it could produce based on pilot experiments (KN1y, mv6u). Following KN1y’s questions about relaxing entity-selection thresholds, we analyzed the relationship between entity popularity and question difficulty. We also clarified quality checks, human review, filtering criteria, and the shared evaluation framework to address concerns about question quality and reproducibility (KN1y, mv6u, NF4F).

Discussion outcomes. KN1y followed up on the number of QA pairs the pipeline could generate and how relaxing the entity in-degree thresholds would affect question quality and difficulty. After we provided these estimates and compared questions involving entities of different popularity levels, the reviewer stated that the additional experiments partially addressed their scalability concerns and raised the rating from 3 to 4. NF4F recommended including the harness results to strengthen the paper’s practical relevance and maintained a positive rating of 3. mv6u acknowledged the clarification and maintained a rating of 2, with no further questions in the follow-up.

We have incorporated the additional results and clarifications into the revised manuscript. We hope the revised manuscript and discussion provide a fuller basis for your final assessment of LoHoSearch’s contributions.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.