acceptodds
Under review as a conference paper at ICLR 2027

Beyond Finding Facts: DISCOBench for Evaluating Clarification in Deep Search

Abstract

Search agents powered by large language models (LLMs) increasingly tackle complex information-seeking tasks through multi-step retrieval and reasoning. However, existing benchmarks often assume complete and explicit queries, leaving agents' ability to clarify underspecified requests insufficiently evaluated. In multi-step search, retrieved facts may support several plausible intermediate targets without revealing which one the user intends. Final-answer correctness alone offers limited insight into whether agents recognize and resolve these information gaps. We introduce DISCOBench, a benchmark for evaluating whether search agents recognize clarification needs, ask effective questions, and use feedback to advance through interdependent retrieval steps. Starting from verified multi-hop questions, we introduce ambiguous checkpoints with annotated discriminative clues and design a user simulator that provides these clues in response to clarification questions. The annotated checkpoints provide verifiable intermediate targets for assessing whether agents ask effective clarification questions and successfully advance after receiving feedback. DISCOBench contains 211 samples and 463 ambiguity instances across 11 knowledge domains and four ambiguity types. We evaluate task utility, ambiguity detection, clarification effectiveness, and interaction cost. Experiments on 12 LLMs show that even the best-performing model achieves only 63.7% task accuracy, highlighting the challenge of coordinating retrieval with proactive clarification in multi-step search.

Primary area:
datasets and benchmarks
Keywords:
Deep Search, Search Agents, Benchmark, Ambiguity Resolution

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_d5MWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DISCOBench, a Chinese benchmark containing 211 multi-step retrieval tasks and 463 annotated ambiguity instances. Through sequential checkpoints and a simulated user, it evaluates clarification coverage, question quality, task completion, and interaction cost. Experiments on 12 models reveal persistent challenges in recognizing clarification needs and completing multi-step retrieval tasks.

Critical strengths
  1. The evaluation usefully separates clarification coverage from conditional success. Table 2 illustrates this distinction through DeepSeek-V4.1-Flash’s 94.4% CE-B and 41.9% clarification recall. These complementary metrics reveal failures that either final-answer accuracy or conditional interaction quality alone would obscure.
  2. The experiments provide informative diagnostic comparisons. Table 4 examines different interaction configurations, while Appendix B shows that increased clarification recall does not consistently improve task accuracy. These findings support evaluating clarification behavior and task utility jointly.
  3. The protocol is documented and meaningfully validated. Appendices K–M provide human audits, responder-prompt checks, and statistical analyses. The paper also appropriately qualifies behavioral comparisons as conditional associations and explicitly acknowledges the scope of its supplied decomposition.
Critical weaknesses
  1. The distinction from a closely related benchmark is insufficiently established. Clarify-Then-Search [1] already evaluates clarification for underspecified deep-search queries through simulated feedback, downstream utility, and interaction diagnostics. DISCOBench’s interleaved checkpoint protocol appears meaningfully different, but this work is not discussed. A direct comparison of clarification timing, agent decisions, and evaluation targets would better establish the benchmark’s incremental contribution.
  2. The feedback setting limits the generalizability of clarification success. Each injected ambiguity is designed to be resolved with a single verified clue, supplied after a correctly targeted question. This controlled design supports reproducibility, but performance with incomplete or uncertain user feedback remains unclear.

[1] Huang D, Zhou J, Lu X, et al. Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 9113-9124.

Questions for authors
  1. Which capabilities and diagnostic findings distinguish DISCOBench from Clarify-Then-Search, particularly regarding clarification during retrieval rather than before search?
  2. clarify how the single-clue feedback assumption limits the interpretation of CE-B when users cannot provide sufficient discriminative information.
  3. All five positions in the “Ambiguity Point Identification” chain are labeled “Hop 1.”. Might small typo.
  4. In Table 14, why is the non-leakage mean a perfect 3.000/3 while raw agreement is only 0.658? Are these calculated from the same ratings, or do they use different aggregation rules?
3 comments

W1 / Q1. Relation to Clarify-Then-Search

Thank you for pointing us to Clarify-Then-Search [1] and suggesting a direct comparison. Clarify-Then-Search sets a question budget before retrieval, uses clarification to gather information and rewrite the query, and passes it to a fixed WebDancer backend. It measures the downstream benefit of clarification with weighted nugget recall. DISCOBench places clarification decisions within multi-step retrieval. Based on the evidence gathered so far, the agent decides whether to keep searching, ask the user, or submit an answer. Each checkpoint specifies what the agent has seen and provides a verifiable intermediate target. This lets us assess whether the agent asks when needed, asks about the right condition, and uses the clue to advance.

For example, DeepSeek-V4.1-Flash asks about the right condition and passes at 94.4% of checkpoints where it asks (CE-B). Its correctly targeted questions cover 41.9% of the ambiguous checkpoints it reaches (Recall). The trace breakdown shows that 98.8% of its clarification false negatives occur at checkpoints where it never asks (see our response to Reviewer qf8K, W1 / Q1, Table S1). The model often asks effectively and advances when it does ask, yet leaves many clarification needs uncovered. We will compare the protocols in the revision by clarification timing, agent decisions, and evaluation targets, and use this example to illustrate what checkpoint diagnostics can reveal.

References

[1] Huang D, Zhou J, Lu X, et al. Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 9113-9124.

W2 / Q2. A single clue from a cooperative user

Thank you for asking how the single-clue assumption affects the interpretation of CE-B. After a correctly targeted question, the simulator provides one verified clue sufficient to resolve the current ambiguity. This controls the information the model receives, and the model still needs to use the clue to continue searching and pass the checkpoint. CE-B is the fraction of checkpoints with a clarification attempt at which the model asks about the right condition and subsequently passes. Checkpoints where every question misses the ambiguity remain in the denominator. Together, CE-A and CE-B distinguish failures to ask about the right condition from failures to pass after receiving a sufficient clue.

A high CE-B therefore indicates that, under this feedback condition, models often ask about the right condition and advance when they attempt clarification. With partial feedback, they also need to determine what remains ambiguous and whether to ask a follow-up. CE-B under the current protocol does not measure those decisions. We will clarify this scope in Section 5.1 and Appendix A.4 and use the current results as a reference for comparisons across feedback conditions.

Q3. Hop labels in Figure 3

Thank you for catching the labeling error. We will label the five positions Hop 1 through Hop 5.

Q4. Non-leakage scores in Table 14

Thank you for pointing out that Table 14 needs clearer definitions. Both columns use the same annotations. Every numerical rating was 3, so the mean after excluding N/A is 3.000. Raw agreement (0.658) includes N/A and therefore also captures disagreement about whether the dimension applies.

We will specify the population, N/A handling, and agreement definition for each column. We will also report agreement on applicability and agreement among valid numerical ratings separately.

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Thank you for the clarificaiton and additional experiements, which has resolved some of my previous concerns. I maintain by positive rating.

Reviewer_qf8KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper introduces DISCOBENCH, a Chinese benchmark for clarification during multi-step web search. Its 211 tasks contain 463 annotated ambiguities. A checkpoint-based harness supplies intermediate questions, simulates a user who can provide discriminative clues, and evaluates both task completion and clarification behavior. The experiments compare 12 models and examine prompting, available actions, and reasoning effort. The contribution is a controlled diagnostic benchmark for interactive retrieval.

Strengths

S1. The paper addresses a useful evaluation gap. Final-answer accuracy does not reveal whether an agent recognized missing information, asked a relevant question, or successfully used the reply. Annotated checkpoints make these distinctions more inspectable.

S2. The dataset construction and harness receive substantial validation. Appendix J describes multiple quality-control stages; Appendix K checks judgments and simulator replies against human annotations. The repeated runs, bootstrap analysis, and responder-prompt checks in Appendices L–M are useful additions to an otherwise small benchmark.

S3. The results go beyond a model leaderboard. For example, DeepSeek-V4.1-Flash has high clarification-to-advance performance but low clarification recall, while guided prompting increases GPT-6-ASTRA's recall and reduces its task accuracy. These observations show why interaction frequency and conditional success should be examined separately.

Weaknesses

W1. A more explicit breakdown would improve the interpretation of the detection scores. Appendix A.5 explains that recall combines asking behavior and question targeting, and CE-A reports targeting quality over asked checkpoints. However, the remaining false negatives are not separated into checkpoints where the agent never asks and those where it asks but fails to target the ambiguity. Reporting these two counts within reached ambiguous checkpoints would make the diagnostics more useful, particularly when interpreting low recall alongside high CE-A. This can be derived from the existing traces.

W2. Cross-model differences in conditional metrics would be easier to assess with the reached-state counts. The paper correctly notes that early termination changes which checkpoints each model encounters, and Appendix M distinguishes this issue from the stability of overall task rankings. Please report reached-checkpoint counts by position and ambiguity type alongside recall and CE-B. This would show how much the diagnostic comparisons involve different state populations. A small common-checkpoint analysis could further help if the existing logs permit it. The current results are informative, but should be interpreted as performance on each model's encountered states.

Critical strengths

S1. The paper addresses a useful evaluation gap. Final-answer accuracy does not reveal whether an agent recognized missing information, asked a relevant question, or successfully used the reply. Annotated checkpoints make these distinctions more inspectable.

Critical weaknesses

W1. A more explicit breakdown would improve the interpretation of the detection scores. Appendix A.5 explains that recall combines asking behavior and question targeting, and CE-A reports targeting quality over asked checkpoints. However, the remaining false negatives are not separated into checkpoints where the agent never asks and those where it asks but fails to target the ambiguity. Reporting these two counts within reached ambiguous checkpoints would make the diagnostics more useful, particularly when interpreting low recall alongside high CE-A. This can be derived from the existing traces.

Questions for authors
  1. Can the existing logs separate no-ask failures, wrong-target asks, and correctly targeted asks, and show how this breakdown changes the interpretation of Table 2?
  2. Do the main clarification comparisons remain similar on checkpoint states shared across models? What proportion of later ambiguous checkpoints is excluded for each model?
  3. Appendix A distinguishes checkpoint ambiguity from ambiguity in the complete rewritten question, and Appendix J already includes full-query no-clarification probes. How often were retained items solvable in those probes, and why were they retained? A summary from the existing quality-control records would make the benchmark's scope clearer without introducing another evaluation setting.
4 comments

W1 / Q1. False negatives in clarification

Thank you for suggesting this breakdown. We use the existing evaluation traces to assign each reached ambiguous checkpoint to one of three categories: at least one correctly targeted question, questions that all miss the ambiguity, or no question. The analysis covers the full 211 tasks and 463 annotated ambiguity instances, with three runs per model. Counts are averaged across runs. Rates are computed on valid records within each run and then averaged.

Across the 12 models, 93.2%–100.0% of false negatives came from checkpoints where the agent never asked, and 0.0%–6.8% came from checkpoints where it asked but never targeted the ambiguity. For example, DeepSeek-V4.1-Flash reaches an average of 322.3 ambiguous checkpoints per run. It asks a correctly targeted question at 135.0, asks without ever targeting the ambiguity at 2.3, and never asks at 185.0. Its low Recall primarily reflects clarification needs left uncovered.

Models can thus perform well on the checkpoints where they ask while leaving many clarification needs unmet. Across models, an average of 25.0–45.0 checkpoints per run include an initial miss followed by a correctly targeted question. We count those checkpoints as correctly targeted because the breakdown measures eventual coverage, even when the first question misses. We will add this definition and the results to the revision.

Table S1. Clarification coverage and false negatives at reached ambiguous checkpoints

ModelReachedCorrectly targetedAsked, never targetedNever askedMistargeted share of FNNo-ask share of FN
GPT-6-ASTRA393.7309.05.779.06.8%93.2%
Claude-Fable-5.1373.7246.74.3122.73.4%96.6%
Claude-Opus-5386.0218.33.3164.32.0%98.0%
Muse-Spark-1.3328.0223.06.798.36.4%93.6%
Hunyuan-4-Preview354.7203.02.3149.31.5%98.5%
Gemini-3.8-Flash378.3219.76.7152.04.2%95.8%
Grok-4.6357.0234.36.7116.05.5%94.5%
DeepSeek-V4.1-Flash322.3135.02.3185.01.2%98.8%
GLM-5.3-Flash349.3173.02.3174.01.3%98.7%
Kimi-K3338.0133.01.0204.00.5%99.5%
Qwen3.8-Max290.0120.31.0168.70.6%99.4%
Doubao-Seed-2.1-Pro329.0128.30.0200.70.0%100.0%

Q3. Full-query quality-control probes and retention rules

In the open-book quality check (Appendix J), the model saw the full question with the injected ambiguity. It could search and read web pages with up to 25 tool calls and had to answer without asking the user. On the 211 evaluation tasks, the model answered 121 (57.4%) correctly with web access and 3 (1.4%) in the closed-book check. Among the 236 candidates that passed quality control, the corresponding counts were 134 (56.8%) and 7 (3.0%).

Later constraints can help select a unique candidate when the full query is available. At a progressively disclosed checkpoint, the same constraints are still hidden. The open-book probe reported ambiguity on 150 of the 211 tasks, including 77 of the 121 it answered correctly. Since it had to submit an answer, agreement with the reference answer alone cannot establish that the task is unambiguous.

Open-book success alone was not an exclusion criterion. We removed 21 items where open-book success coincided with knowledge leakage and 3 where it coincided with superficial ambiguity (Table 12). The 121 items in the final evaluation set that the open-book probe answered correctly were retained after human review. We will report the probe results for both populations and explain the retention rule in the revision.

W2 / Q2. Checkpoints reached by different models

Thank you for pointing out that models may reach different checkpoints. We now report how many checkpoints each model reaches by position, ambiguity type, and order of appearance, alongside Recall, CE-B, and their denominators. We distinguish checkpoints excluded because of earlier failure, invalid task records, and the absence of an evaluable event. The analysis covers 211 tasks and 463 ambiguity instances. Rates are computed per run and averaged over three runs.

Across models, 25.5%–54.0% of the second and third ambiguity points combined are excluded from the statistics. Table S3 reports the breakdown for each model.

A model's own set comprises the ambiguous checkpoints it reached with an evaluable decision in that run. The shared set is the intersection of the 12 models' own sets within each run, aligned by task ID and checkpoint position. Each of the three runs has 165 shared checkpoints, or 35.6% of the 463 instances. Using the shared set changes each model’s Recall averaged over the three runs by −4.5 to +6.2 points, and the Spearman correlation between the model rankings is 0.970. GPT-6-ASTRA remains highest in Recall, and the four models with the lowest Recall on their own sets still form the lowest group (Table S2).

The ordering by CE-B on ambiguous checkpoints is more sensitive (ρ ≈ 0.74 using the values displayed in Table S2), so small differences warrant caution. Coverage and conditional success can still diverge. For example, DeepSeek-V4.1-Flash has 41.8% Recall and 96.1% CE-B on the shared set. Both CE-B columns in Table S2 use ambiguous checkpoints only. The original metric in Table 2 also includes questions at unambiguous checkpoints.

The shared set holds the checkpoints fixed. Search histories and the subsets where models actually ask can still differ. In each run, 127 of the 165 shared checkpoints (77.0%) are the first ambiguity point in their task. The shared-set analysis supports the main Recall comparison. Interpreting CE-B also requires knowing which checkpoints each model reached and how many of those it asked about. We will clarify this scope and report the position-wise counts in the revision.

Table S2. Clarification metrics on models' own and shared reached sets

ModelRecall (own)Recall (shared)Δ Recall (pp)CE-B on ambiguous points (own)CE-B on ambiguous points (shared)Δ CE-B (pp)
GPT-6-ASTRA78.5%77.8%-0.794.3%94.0%-0.3
Claude-Fable-5.166.0%68.3%+2.396.4%96.1%-0.3
Claude-Opus-556.7%55.4%-1.394.5%94.0%-0.5
Muse-Spark-1.368.0%68.9%+0.992.7%93.3%+0.6
Hunyuan-4-Preview57.2%52.7%-4.592.0%91.9%-0.1
Gemini-3.8-Flash58.1%64.2%+6.294.1%92.7%-1.4
Grok-4.665.6%64.2%-1.490.3%91.7%+1.4
DeepSeek-V4.1-Flash41.9%41.8%-0.195.9%96.1%+0.2
GLM-5.3-Flash49.5%45.3%-4.393.0%95.5%+2.5
Kimi-K339.3%40.0%+0.796.7%93.5%-3.2
Qwen3.8-Max41.5%38.6%-2.997.2%94.7%-2.5
Doubao-Seed-2.1-Pro39.0%39.2%+0.298.3%98.2%-0.1

Table S3. Later ambiguity points excluded from the analysis

ModelSecond, earlier failureThird, earlier failureSecond and third, earlier failureSecond and third, all excluded
GPT-6-ASTRA19.8%36.9%25.5%25.5%
Claude-Fable-5.118.0%40.0%25.4%30.6%
Claude-Opus-524.0%41.2%29.8%29.8%
Muse-Spark-1.315.6%30.6%20.6%39.7%
Hunyuan-4-Preview32.9%45.9%37.3%38.1%
Gemini-3.8-Flash24.0%48.2%32.1%32.1%
Grok-4.632.9%47.1%37.7%38.5%
DeepSeek-V4.1-Flash32.7%55.3%40.3%48.3%
GLM-5.3-Flash35.9%51.4%41.1%41.1%
Kimi-K334.1%57.6%42.1%46.0%
Qwen3.8-Max28.7%48.2%35.3%54.0%
Doubao-Seed-2.1-Pro40.7%69.4%50.4%50.4%

Dear Reviewer qf8K,

We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your thoughtful feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.

I appreciate the authors' efforts in addressing my concerns through the additional results. After considering the rebuttal, my final rating is 2.

Reviewer_jTDmOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DISCOBENCH, a Chinese-language benchmark for evaluating clarification behavior in LLM-based search agents. The benchmark is constructed by transforming deterministic multi-hop factual QA chains and injecting four types of ambiguity—Entity, Version, Criteria, and Factual Inaccuracy—at intermediate checkpoints. Each ambiguity is paired with a discriminative clue that a simulated user provides when the agent asks an appropriately targeted question.

The benchmark contains 211 questions and 463 ambiguity instances across 11 domains. Agents interact through SEARCH, ASK, and ANSWER actions, and are evaluated on task completion, checkpoint progress, ambiguity detection, clarification effectiveness, over-asking, and interaction cost. Experiments on 12 models show a best end-to-end accuracy of 63.7%. The authors also provide ablations, behavioral analyses, human validation of the evaluation harness, repeated runs, and prompt-sensitivity analyses.

Critical strengths

(1) The checkpoint-level evaluation provides useful diagnostic information beyond final-answer accuracy. The benchmark distinguishes failure to detect ambiguity, asking an irrelevant clarification question, and failing to use the returned clue. This is a meaningful contribution because these failure modes are conflated in conventional answer-only search benchmarks.

(2) The paper demonstrates substantial experimental and benchmark-engineering effort. The evaluation covers 12 models and several interaction settings, including no-ASK, unambiguous, guided-prompting, and reasoning-effort variants. The multi-stage quality control, human audits, repeated runs, and robustness analyses increase confidence that the reported results are internally consistent under the proposed protocol.

Critical weaknesses

(1) The benchmark construction supports only a narrow and insufficiently validated subset of clarification needs. All instances are derived from deterministic multi-hop factual QA chains. Consequently, every ambiguity must preserve a single hidden factual target, remain compatible with a predefined downstream chain, and be resolvable using a discriminative fact. The four operationalized types cover ambiguity about entity identity, version or time, ranking criteria, and factual errors, but they do not capture many important clarification needs in realistic deep search, including:

  • ambiguity about the user’s underlying goal or intent;
  • subjective preferences and trade-offs;
  • task scope, geographic or temporal boundaries;
  • desired answer granularity or output format;
  • acceptable evidence or source-quality standards;
  • conflicting constraints or evolving user intent;
  • ambiguities requiring multiple rounds of clarification.

For example, “Which cloud database is best for our startup?” requires clarification of budget, scale, latency, consistency, and operational preferences. Similarly, “Analyze the recent impact of AI regulation” requires clarification of jurisdiction, time range, meaning of “impact,” evidence standard, and desired depth. These cannot generally be resolved by identifying one missing factual clue.

The paper does not provide an empirical coverage analysis over naturally occurring search queries or user–agent conversations. Therefore, the results support conclusions about a specific class of synthetically transformed fact-retrieval ambiguities, rather than clarification in deep search more generally. This scope mismatch affects the central contribution and is the main reason for my recommendation.

(2) The oracle checkpoint protocol further limits the validity of the claimed end-to-end setting. The agent receives a predefined sequence of checkpoint subquestions while later constraints are withheld. It does not autonomously decompose the complete query, choose its own retrieval trajectory, or recover from an earlier error. The paper also acknowledges that an Ambi checkpoint may not remain ambiguous when the complete rewritten question is visible. Thus, some measured clarification needs may be created by the progressive-disclosure protocol rather than by the original query. The benchmark primarily evaluates clarification decisions under a gold decomposition, not autonomous clarification during realistic deep search.

Questions for authors

(1) How were the four ambiguity types selected? Were they derived from an analysis of naturally occurring search conversations, or primarily selected because they could be injected into multi-hop factual QA chains? Can the authors provide an empirical coverage study showing what proportion of real clarification needs can be represented by these four types, including examples that cannot be represented?

(2) How would DISCOBENCH evaluate non-factual or multi-dimensional clarification needs? For example, how would it handle preference elicitation, scope negotiation, requested output granularity, source-quality requirements, conflicting constraints, or cases requiring several rounds of clarification? If these are intentionally outside scope, what is the precise capability the authors claim DISCOBENCH measures?

(3) Is the complete ambiguity-injected question shown to the agent during evaluation? If only checkpoint subquestions are shown, what fraction of the 463 ambiguity instances remain genuinely underdetermined when the complete rewritten question is visible? Results under full-query visibility and autonomous decomposition would materially affect my assessment.

4 comments

W1 / Q2. Scope and research contribution

DISCOBench measures whether, during multi-step factual retrieval, an agent recognizes an ambiguity that can change the subsequent search, asks for information that distinguishes the candidates, and uses the reply to advance along the intended path. Consider your AI regulation example. A user may have a particular version of a regulation in mind but refer to it by a shorthand name. Search can find the competing versions and their provisions, yet the agent may still need the user to confirm which version they mean. Choosing the wrong version changes the provisions and cases that the agent retrieves, even if every retrieved fact is accurate. This is why we measure the ability to clarify the intended target separately.

DISCOBench focuses on users who already have a goal but have left out information needed to distinguish the candidates. Preference negotiation, changing goals, and follow-up questions after partial feedback fall outside this evaluation and require different interaction and scoring designs. A correctly targeted question receives a sufficient clue for that ambiguity point, and a task can still require clarification at multiple checkpoints.

Verifiable intermediate targets and sufficient clues let us distinguish whether a model asks, whether it asks about the right condition, and whether it advances after receiving feedback. DeepSeek-V4.1-Flash asks about the right condition and passes at 94.4% of checkpoints where it asks (CE-B), while correctly targeted questions cover 41.9% of the ambiguous checkpoints it reaches (Recall, Table 2). With search still available, disabling ASK lowers task accuracy by 10.9–31.5 points under progressive disclosure (Table 4). These results show that clarification can affect task completion when intermediate targets are verifiable. Checkpoint-level diagnostics reveal where clarification breaks down. We will make this contribution explicit in the abstract, introduction, and limitations.

Q1. Choice of ambiguity types and coverage in natural search

We started from multi-hop factual question-answering chains and selected ambiguity types that can change an intermediate retrieval target and can be resolved with a discriminative clue. AmbigQA [1] motivates the entity category, SituatedQA [2] motivates the time or version category, and CondAmbigQA [3] motivates the category of unstated conditions. We focus the last category on decision criteria, such as which metric to rank by.

An incorrect premise can also prevent the agent from identifying the intended target. CREPE [4] found false presuppositions in roughly a quarter of the Reddit information-seeking questions it collected. This motivates including factual inaccuracies, where the agent may need to verify a condition the user has already stated.

We have yet to estimate what share of clarification needs they cover in natural search conversations. When an agent does not ask, the user’s unstated constraints often remain unverifiable from the transcript, making it difficult to count all cases that warranted clarification. A reliable estimate would require verifying user intent on a representative sample of conversations. We will clarify this scope in the revision.

References

[1] Min S, Michael J, Hajishirzi H, et al. AmbigQA: Answering Ambiguous Open-domain Questions[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 5783-5797.

[2] Zhang M, Choi E. SituatedQA: Incorporating Extra-Linguistic Contexts into QA[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 7371-7387.

[3] Li Z, Li Y, Xie H, et al. CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025: 2269-2288.

[4] Yu X, Min S, Zettlemoyer L, et al. CREPE: Open-Domain Question Answering with False Presuppositions[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023: 10457-10480.

W2 / Q3. Progressive disclosure and autonomous full-query retrieval

Thank you for requesting an evaluation with full-query visibility and autonomous decomposition. In the main evaluation, the agent sees the current subquestion and the interaction history, with the rest of the question disclosed step by step. We added an autonomous evaluation on all 211 tasks using the same four models as Table 4. From the outset, the agent sees the complete query with injected ambiguity and chooses how to decompose it, which retrieval path to follow, and when to ask.

We manually reviewed all 211 complete queries independently of model performance. In 112 tasks (53.1%), later constraints exclude other candidates or lead different paths to the same reference answer. In 99 tasks (46.9%), candidate ambiguity, factual conflicts, or conflicting sources remain unresolved by the later constraints. We audited whole tasks because the autonomous evaluation scores the final answer, and different intermediate paths can lead to that same answer. A separate audit of all 463 ambiguity instances would require checking each candidate branch against the later constraints and remains to be done.

All four models score 16.2–39.5 percentage points lower on tasks with unresolved issues than on tasks resolved by later constraints (Table S4). They also ask more often on the unresolved group. GPT-6-ASTRA asks on 68.4% of those tasks versus 31.2% of the resolvable group. The other models ask on 2.0%–22.7% of the unresolved tasks.

With the full query visible, an agent can use later constraints to filter earlier candidates or keep several paths open until it answers. At each checkpoint, we know what the agent has seen and can verify its intermediate answer. This lets us distinguish a missed opportunity to ask, a mistargeted question, and a failure to advance after a correctly targeted question. Under progressive disclosure, the ASK ablation in Table 4 lowers accuracy by 10.9–31.5 points across four models. The trace breakdown attributes 93.2%–100.0% of false negatives to checkpoints where the model never asked (see our response to Reviewer qf8K, W1 / Q1, Table S1). Together, these results show how access to clarification affects task completion under the checkpoint protocol and identify missed opportunities to ask as a specific failure mode.

Table S4. Autonomous retrieval on full queries

ModelOverall accuracyResolvable accuracyUnresolved accuracyResolvable ask rateUnresolved ask rateMean searches per task
GPT-6-ASTRA85.78%93.8%77.6%31.2%68.4%43.5
DeepSeek-V4.1-Flash66.35%83.5%51.6%4.6%14.7%56.2
Doubao-Seed-2.1-Pro65.40%83.9%44.4%0.0%2.0%41.5
Hunyuan-4-Preview60.66%78.4%42.3%4.5%22.7%19.9

Dear Reviewer jTDm,

We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your thoughtful feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Reviewer_xgqNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DiscoBench, a benchmark for evaluating whether LLM-based search agents can proactively recognize ambiguity and ask effective clarification questions during multi-step web retrieval. The authors argue that mainstream retrieval benchmarks (GAIA, BrowseComp, AgentBench) assume explicit queries, while ambiguity-focused datasets (AmbigQA, DeepAmbigQA) are static and non-interactive; interactive benchmarks (IN3, UserBench, InteractComp) offer limited coverage of clarification within open-domain multi-step retrieval. DiscoBench is positioned as a checkpoint-level diagnostic benchmark bridging this gap.

Critical strengths
  1. Well-motivated problem and clear gap analysis. The related-work section carefully distinguishes DiscoBench from static ambiguity QA, multi-hop retrieval benchmarks, and existing interactive benchmarks. The identification of "checkpoint-level diagnosis" as the missing evaluation axis is crisp and convincing.
  2. Checkpoint-level annotation is a genuine methodological contribution. Pre-annotating ambiguity positions, discriminative clues, and sub-goals — rather than comparing free-form trajectories — yields repeatable intermediate targets and localizes failures to specific steps. This is more informative than final-answer-only evaluation and more tractable than fully autonomous decomposition.
  3. Controlled, auditable ambiguity injection. Replacing strong constraints with shared attributes (rather than random perturbation) ensures each ambiguous state is retrieval-grounded, has multiple plausible candidates, and is resolvable by a single user clue. The four-type taxonomy (Entity/Version/Criteria/Factual Inaccuracy) is sensible and covers qualitatively different failure modes, as illustrated by the two case studies (factual inaccuracy requiring an ASK rather than a guess; version ambiguity requiring a precise date).
  4. Behavioral analysis adds actionable insight. The SearchThenAsk vs. SearchThenGuess contrast (~94.2% vs. ~57.0% mean checkpoint pass) is an interesting, practically relevant finding: asking after retrieval is strongly associated with checkpoint success. The profile analysis, while correlational, suggests a concrete design principle for agent policies.
Critical weaknesses
  1. The Chinese-only construction limits external validity. DiscoBench is built in Chinese "to reflect ambiguity patterns and retrieval behaviors in Chinese web environments," with Baidu Baike prioritized in prompts. Ambiguity patterns, entity popularity distributions, and search-engine behavior differ substantially across languages and retrieval backends. The paper does not discuss whether its conclusions (e.g., SearchThenAsk superiority) are language- or index-dependent. At minimum, a small English pilot or a discussion of expected transfer would strengthen the generality claim.
  2. The "Neutral" protocol is not neutral, and the checkpoint harness constrains autonomy. The harness supplies a fixed decomposition and standard sub-questions, stops at the first error, and withholds future constraints (Appendix A). This measures clarification given a decomposition, not autonomous decomposition — which the authors acknowledge but then interpret broadly ("search agents can proactively clarify"). The Neutral prompt also explicitly instructs search-first, which mechanically depresses DirectAsk (0–5.5%) and may inflate the SearchThenAsk profile share. The behavioral finding that "retrieval followed by clarification" beats "retrieval followed by answering" is therefore partly a consequence of protocol design rather than a free-standing empirical regularity. A stronger design would compare policies under matched information states, or randomize the order of available actions.
Questions for authors
  1. Is that possible to extend the whole methodology to other language?
  2. How would a simple heuristic baseline perform — e.g., "ask whenever top-k retrieval returns ≥2 distinct candidate entities satisfying the constraints"? This would calibrate whether the reported model gaps reflect genuine clarification capability or merely the absence of a trivial policy.
  3. Given the $7,877 full-run cost and 105h48m total runtime, do you envision a lighter-weight subset (e.g., 50–80 items stratified by ambiguity type) for routine model development, with the full set reserved for leaderboard evaluation? If so, how would you select the subset to preserve ranking fidelity?
4 comments

Thank you for the thoughtful review and for raising questions about language transfer and the search-first prompt.

W1 / Q1. Chinese construction and transfer to English

We built DISCOBench in Chinese and use Tavily for SEARCH. We removed the instruction to prioritize Baidu Baike. To test English transfer, we randomly sampled 61 tasks and evaluated both language versions with the four models in Table 4.

The four models rank in the same order by task accuracy in both languages, with English accuracy 3.3–11.5 percentage points lower (Table S5). For example, GPT-6-ASTRA scores 75.4% in Chinese and 67.2% in English. This pilot demonstrates that the evaluation can be run in English on the selected tasks. Absolute performance may depend on the language, translation, and retrieval environment.

Three models have English Recall 5.7–20.1 points lower than Chinese Recall. Hunyuan's Recall rises slightly. English CE-A and CE-B do not consistently decline alongside Recall. For example, DeepSeek and Doubao achieve English Recall of 28.9% and 31.2%, with CE-B of 92.6% and 96.8%, respectively. High conditional success can therefore coexist with limited clarification coverage in the English pilot. Our SearchThenAsk analysis applies to the original Chinese setting.

Table S5. Chinese and English results on 61 paired tasks

ModelAccuracy (ZH / EN)Δ Accuracy (EN − ZH, pp)Recall (ZH / EN)CE-A (ZH / EN)CE-B (ZH / EN)
GPT-6-ASTRA75.4% / 67.2%−8.280.7% / 75.0%88.5% / 87.3%87.5% / 84.3%
DeepSeek-V4.1-Flash49.2% / 37.7%−11.549.0% / 28.9%94.1% / 96.3%92.2% / 92.6%
Doubao-Seed-2.1-Pro37.7% / 34.4%−3.346.9% / 31.2%95.7% / 96.8%91.5% / 96.8%
Hunyuan-4-Preview55.7% / 45.9%−9.857.9% / 60.7%93.9% / 100.0%92.4% / 98.2%

W2. Search-first prompting and behavior analysis

The original Neutral prompt explicitly requires search first and gives no reminder to check for ambiguity. We will rename it Base (search-first). The instruction tests whether agents can use retrieved evidence to decide when to ask and may influence how often they ask before searching.

Ablation of the search-first instruction

To examine sensitivity to this instruction, we ran an ablation on the same 211 tasks with the same four models. We removed the explicit search-first sentence and kept the decomposition, tool budget, user simulator, and scoring rules fixed. We ran each model three times and averaged the per-run rates. Table S6 reports reached ambiguity points, path shares, path pass rates, and task accuracy.

After removing the instruction, the DirectAsk share rises by 0.4–0.7 points. The SearchThenAsk share falls by 0.4–0.7 points and remains higher than DirectAsk under both prompts. Mean task accuracy changes by −0.5 to +0.3 points, across the four models.

Under No-SF, the SearchThenAsk pass rate still exceeds the SearchThenGuess pass rate by 35.6–49.1 percentage points. The association between asking after search and higher checkpoint pass rates thus also appears without an explicit search-first instruction. Models select their own paths and may reach different checkpoints or have different search histories. A causal comparison would require matching the information states.

Table S6. Behavior and task performance with and without the search-first instruction

ModelReached ambiguous points (Base / No-SF)DirectAsk (Base / No-SF)SearchThenAsk (Base / No-SF)Task accuracy (Base / No-SF)Δ Accuracy (pp)SearchThenAsk pass rate (Base / No-SF)SearchThenGuess pass rate (Base / No-SF)
GPT-6-ASTRA393.7 / 395.35.5% / 6.2%74.4% / 73.7%63.7% / 63.2%−0.597.0% / 96.8%61.4% / 61.2%
DeepSeek-V4.1-Flash322.3 / 320.70.7% / 1.4%41.9% / 41.2%38.7% / 39.0%+0.394.4% / 94.7%58.1% / 58.0%
Hunyuan-4-Preview354.7 / 354.30.0% / 0.6%57.9% / 57.3%46.4% / 46.1%−0.392.2% / 92.0%55.6% / 55.9%
Doubao-Seed-2.1-Pro329.0 / 329.30.0% / 0.4%39.0% / 38.6%28.4% / 28.2%−0.298.7% / 98.5%49.1% / 49.4%

We will specify the prompts and protocol when reporting these associations. See our response to Reviewer jTDm, W2 / Q3 (Table S4), for autonomous retrieval. Under the main protocol, disabling ASK while retaining search lowers accuracy by 10.9–31.5 percentage points across the four models (Table 4). That ablation measures the value of access to clarification under this protocol and leaves the best order of actions open.

Q2. Candidate-count heuristic

Thank you for proposing this baseline. It can flag cases where the retrieved evidence supports multiple candidates that satisfy the disclosed constraints. Its scope depends on how the system identifies candidates and determines which constraints they satisfy. The rule provides a reference for assessing how many checkpoints that need clarification can be detected from the number of qualifying candidates in the top-k search results. If the search retrieves only one qualifying candidate, the rule stays silent even when other qualifying candidates exist.

For a question such as “Which company is larger?”, filtering by revenue alone may yield a single candidate even though the user's intended criterion still needs clarification. Considering both revenue and headcount requires recognizing that the criterion was left unstated. A factual error can leave no candidate that satisfies all the stated conditions. In that case, the conflict between the query and the evidence may still call for verification.

Candidate count is one signal for clarification, with coverage that varies by ambiguity type. Once the rule triggers, the agent must still identify which difference between the candidates to ask the user about. Our checkpoint diagnostics distinguish whether the model detects an unstated criterion or a factual conflict, asks about the relevant condition, and advances after receiving feedback.

Q3. Lightweight development subset

Thank you for the suggestion. We plan to release a fixed DISCOBench-Lite subset for routine development and use all 211 tasks for the formal leaderboard. We will use a fixed seed to sample roughly 50–80 complete tasks by ambiguity type and difficulty, retaining all checkpoints and clues. Coverage and agreement with the full-set rankings will determine the final size.

Using per-task results from 12 models and three runs, we will compare subset and full-set rankings with Spearman ρ, Kendall τ, and top-three agreement. We will hold out each model in turn, use the remaining models for selection and size adjustments, and then test the held-out model's relative position. We will assess stability across runs and with a task-level bootstrap. If smaller subsets produce unstable rankings, we will use a larger subset.

We will release the fixed task IDs, selection rules, and validation results, and report cost savings based on per-task logs. The costs and runtime in the paper are cumulative across all evaluated models for a full round. Lite will support rapid configuration screening, with final comparisons confirmed on the full set.

Dear Reviewer xgqN,

We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your thoughtful feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.