- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DISCOBench, a Chinese benchmark containing 211 multi-step retrieval tasks and 463 annotated ambiguity instances. Through sequential checkpoints and a simulated user, it evaluates clarification coverage, question quality, task completion, and interaction cost. Experiments on 12 models reveal persistent challenges in recognizing clarification needs and completing multi-step retrieval tasks.
Critical strengths
- The evaluation usefully separates clarification coverage from conditional success. Table 2 illustrates this distinction through DeepSeek-V4.1-Flash’s 94.4% CE-B and 41.9% clarification recall. These complementary metrics reveal failures that either final-answer accuracy or conditional interaction quality alone would obscure.
- The experiments provide informative diagnostic comparisons. Table 4 examines different interaction configurations, while Appendix B shows that increased clarification recall does not consistently improve task accuracy. These findings support evaluating clarification behavior and task utility jointly.
- The protocol is documented and meaningfully validated. Appendices K–M provide human audits, responder-prompt checks, and statistical analyses. The paper also appropriately qualifies behavioral comparisons as conditional associations and explicitly acknowledges the scope of its supplied decomposition.
Critical weaknesses
- The distinction from a closely related benchmark is insufficiently established. Clarify-Then-Search [1] already evaluates clarification for underspecified deep-search queries through simulated feedback, downstream utility, and interaction diagnostics. DISCOBench’s interleaved checkpoint protocol appears meaningfully different, but this work is not discussed. A direct comparison of clarification timing, agent decisions, and evaluation targets would better establish the benchmark’s incremental contribution.
- The feedback setting limits the generalizability of clarification success. Each injected ambiguity is designed to be resolved with a single verified clue, supplied after a correctly targeted question. This controlled design supports reproducibility, but performance with incomplete or uncertain user feedback remains unclear.
[1] Huang D, Zhou J, Lu X, et al. Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 9113-9124.
Questions for authors
- Which capabilities and diagnostic findings distinguish DISCOBench from Clarify-Then-Search, particularly regarding clarification during retrieval rather than before search?
- clarify how the single-clue feedback assumption limits the interpretation of CE-B when users cannot provide sufficient discriminative information.
- All five positions in the “Ambiguity Point Identification” chain are labeled “Hop 1.”. Might small typo.
- In Table 14, why is the non-leakage mean a perfect 3.000/3 while raw agreement is only 0.658? Are these calculated from the same ratings, or do they use different aggregation rules?
3 comments
Dear reviewers,
Will you please check the authors' response and reply it?
Best, AC
Thank you for the clarificaiton and additional experiements, which has resolved some of my previous concerns. I maintain by positive rating.
W1 / Q1. Relation to Clarify-Then-Search
Thank you for pointing us to Clarify-Then-Search [1] and suggesting a direct comparison. Clarify-Then-Search sets a question budget before retrieval, uses clarification to gather information and rewrite the query, and passes it to a fixed WebDancer backend. It measures the downstream benefit of clarification with weighted nugget recall. DISCOBench places clarification decisions within multi-step retrieval. Based on the evidence gathered so far, the agent decides whether to keep searching, ask the user, or submit an answer. Each checkpoint specifies what the agent has seen and provides a verifiable intermediate target. This lets us assess whether the agent asks when needed, asks about the right condition, and uses the clue to advance.
For example, DeepSeek-V4.1-Flash asks about the right condition and passes at 94.4% of checkpoints where it asks (CE-B). Its correctly targeted questions cover 41.9% of the ambiguous checkpoints it reaches (Recall). The trace breakdown shows that 98.8% of its clarification false negatives occur at checkpoints where it never asks (see our response to Reviewer qf8K, W1 / Q1, Table S1). The model often asks effectively and advances when it does ask, yet leaves many clarification needs uncovered. We will compare the protocols in the revision by clarification timing, agent decisions, and evaluation targets, and use this example to illustrate what checkpoint diagnostics can reveal.
References
[1] Huang D, Zhou J, Lu X, et al. Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration[C]//Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026: 9113-9124.
W2 / Q2. A single clue from a cooperative user
Thank you for asking how the single-clue assumption affects the interpretation of CE-B. After a correctly targeted question, the simulator provides one verified clue sufficient to resolve the current ambiguity. This controls the information the model receives, and the model still needs to use the clue to continue searching and pass the checkpoint. CE-B is the fraction of checkpoints with a clarification attempt at which the model asks about the right condition and subsequently passes. Checkpoints where every question misses the ambiguity remain in the denominator. Together, CE-A and CE-B distinguish failures to ask about the right condition from failures to pass after receiving a sufficient clue.
A high CE-B therefore indicates that, under this feedback condition, models often ask about the right condition and advance when they attempt clarification. With partial feedback, they also need to determine what remains ambiguous and whether to ask a follow-up. CE-B under the current protocol does not measure those decisions. We will clarify this scope in Section 5.1 and Appendix A.4 and use the current results as a reference for comparisons across feedback conditions.
Q3. Hop labels in Figure 3
Thank you for catching the labeling error. We will label the five positions Hop 1 through Hop 5.
Q4. Non-leakage scores in Table 14
Thank you for pointing out that Table 14 needs clearer definitions. Both columns use the same annotations. Every numerical rating was 3, so the mean after excluding N/A is 3.000. Raw agreement (0.658) includes N/A and therefore also captures disagreement about whether the dimension applies.
We will specify the population, N/A handling, and agreement definition for each column. We will also report agreement on applicability and agreement among valid numerical ratings separately.