- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how search-augmented LLMs perform on abstract search queries, where queries are underspecified and may admit multiple valid answers. The authors propose a taxonomy based on two axes: specification and directness, and construct a benchmark of 515 queries spanning two settings, "entity recall" and "open discovery". To avoid requiring an exhaustive ground-truth answer set, the authors evaluate returned candidates against query-specific criteria using an automated Web-enabled verifier. Seven LLMs are evaluated using a shared search harness, measuring target recovery, candidate precision, coverage, and confidence calibration.
The results suggest that models struggle to recover intended entities from indirect descriptions, face a trade-off between retrieval breadth and candidate validity, and remain overconfident even with Web search. The paper contributes a benchmark and an empirical analysis of search behaviour under underspecification.
Critical strengths
I find this paper investigates an interesting and timely problem. Existing search benchmarks largely focus on predetermined answers, whereas real-world queries may be incomplete, indirectly expressed, or admit several valid answers. The distinction between specification and directness is useful, particularly in framing the entity recall (ER) and open discovery (OD) tasks. I think this is a meaningful contribution as it highlights search behaviours that existing benchmarks may overlook.
The criteria-based evaluation framework is a practical contribution. Rather than requiring exhaustive ground-truth answer sets, the authors assess whether the candidates returned by LLMs satisfy the stated constraints. This is especially relevant for OD, where enumerating every valid answer may be infeasible. The empirical evaluation is also reasonably thorough, covering seven LLMs under a shared search harness. The 90.4% agreement between human annotations and automated verification is reassuring, and the sensitivity and calibration analyses in the appendices further strengthen the evaluation. This framework makes the benchmark usable in settings where gold answer sets are impractical, which is a key reason for my positive assessment.
Critical weaknesses
However, it is not clear if the benchmark isolates difficulties specific to abstract search, as compared to general retrieval difficulty. In Section 3, the authors explain that ER queries are modified by removing or rewriting clues until a solver fails. While this produces challenging queries, it is difficult to ascertain if lower performance is attributable specifically to underspecification or indirect expression, rather than to general retrieval difficulty (e.g., obscure entities or queries tuned to defeat a particular solver). Figure 2 partially addresses this by examining different types of query revision, but the authors also acknowledge that edit types can co-occur. A more controlled comparison would strengthen the paper's central claim that abstract search exposes distinct challenges.
The interpretation of hit rate in ER could be clearer. The authors explicitly allow multiple valid candidates to satisfy a query, yet hit rate measures whether the model recovers the annotator's intended seed entity. This means a model may return candidates that satisfy every stated criterion, and still miss the seed entity. Since Hit Rate is the primary metric for ER, it would be informative to report how often seed-recovery failures can still produce at least one valid alternative. This would help distinguish the failure to recover the user's intended entity from the failure to satisfy the query as written.
Questions for authors
Clarification requests:
-
Figure 2 shows that indirect descriptions are associated with reduced hit rate, but query revisions may involve several simultaneous changes. Can the authors isolate the effect of indirect expression from clue removal or other changes, for example through matched revisions that preserve the underlying information? This would help clarify whether the observed degradation is specific to indirectness rather than general retrieval difficulty.
-
For ER, what fraction of queries where the intended seed entity is not recovered nevertheless yield at least one candidate satisfying every stated criterion? This would help contextualize Hit Rate as a measure of intended-entity recovery rather than overall validity of returned answers.
Minor Comments:
- A few typographical errors should be corrected, including "descrubing" (line 114), the malformed wording around "sampFinally" (lines 201–202), and "Tanl C" (line 230). These are minor presentation issues rather than soundness concerns.
3 comments
Author Rebuttal (2/2)
W2. Contextualise Hit Rate as a measure of intended-entity recovery and not validity
Q. For ER, what fraction of queries where the intended seed entity is not recovered nevertheless yield at least one candidate satisfying every stated criterion?
A. Across all models, 35.47% of model–query pairs where the user-intended seed entity is not recovered still return at least one candidate satisfying every stated criterion.
Evidence. Our existing Precision and Coverage@4 metrics already show the validity and breadth of returned candidates. However, they do not explicitly show how often a model returns at least one valid candidate when the user-intended entity is not recovered. We now report Validity Rate (%), which measures the percentage of all 255 ER queries per model where at least one candidate satisfies every stated criterion, alongside Hit Rate.
| Model | Hit Rate (%) ↑ | Validity Rate (%) ↑ | Valid alternative if seed missed (%) ↑ | Neither seed nor valid (%) ↓ |
|---|---|---|---|---|
| GPT-5.6 Sol | 67.45 | 79.61 | 43.37 | 18.43 |
| Claude Opus 5 | 64.71 | 81.57 | 47.78 | 18.43 |
| Gemini 3.7 Flash | 61.18 | 72.55 | 31.31 | 26.67 |
| DeepSeek V4 Pro | 53.33 | 69.80 | 36.13 | 29.80 |
| GPT-5.6 Luna | 47.84 | 66.67 | 37.59 | 32.55 |
| Claude Sonnet 5 | 38.04 | 54.90 | 28.48 | 44.31 |
| Qwen 3.8 Max | 36.86 | 55.69 | 31.68 | 43.14 |
| All models | 52.77 | 68.68 | 35.47 | 30.48 |
Hit Rate and Validity Rate use all ER queries. Valid alternative if seed missed is conditional on missing the seed; Neither seed nor valid uses all model–query pairs.
Takeaway. The results reveal two distinct failure points: failing to recover the entity the user had in mind and failing to return any valid candidate. Across models, 47.23% of model–query pairs miss the intended entity. Of these, 35.47% still return a valid alternative, while 64.53% do not (equivalent to 30.48% of all model–query pairs).
Action. We have updated the main Entity Recall results table to include Validity Rate (%) and added the per-model breakdown to the appendix.
We have corrected the typographical errors identified by the reviewer. We hope these additional analyses and revisions address the reviewer's concerns and clarify the contributions of our work.
As the discussion period comes to an end, we wanted to check whether our responses and revisions have addressed your concerns. We have updated the manuscript, with changes highlighted in purple:
- W1. Isolating the effect of indirectness — L343–364; Appendix F.3.
- W2. Contextualise Hit Rate as a measure of intended-entity recovery and not validity — Table 1, L332–341; Appendix F.1.
Thank you again for your feedback. Please let us know if any concerns remain or if we can clarify anything further.
Author Rebuttal (1/2)
We thank the reviewer for the positive assessment of our work, particularly for recognising the gaps in existing search benchmarks and the value of our framework in addressing them. The reviewer's questions and feedback have been helpful in strengthening our contribution. We address the specific concerns below and have updated the manuscript accordingly.
W1. Isolating the effect of indirect expression
A. Yes. We compared matched revisions that preserve the underlying information and found an average 8.8 percentage-point drop in Hit Rate and an additional 11.0 percentage-point loss of previously recovered entities beyond variations in unchanged query reruns (95% CI: [−19.1, −3.8]).
Evidence. In Figure 2, we show that queries with indirect descriptions as an edit type are associated with some of the largest drops in Hit Rate. However, we agree that this alone does not isolate the effect of indirect descriptions because users often rewrite clues indirectly or remove them at the same time (r = 0.66) together with other edits, as shown in Appendix B.
We screened the 702 query revisions in our dataset and manually identified 38 matched query pairs that preserve the underlying information without removing clues. We then compared retrieval performance across three models, with three independent runs each. To account for variation across runs, we measured how often a model failed to recover a previously recovered entity when the original query was rerun unchanged. The table below reports the changes in Hit Rate, the loss after indirect rephrasing on entities that were previously recoverable, the loss from unchanged-query reruns, and the additional drop after accounting for rerun variability.
Negative values indicate a drop in performance. The pooled loss rates are calculated across paired runs. Confidence intervals use query-level bootstrap resampling.
Takeaway. The matched-revision experiment provides additional evidence that indirect expression is associated with reduced retrieval performance even when the underlying information is preserved. This helps distinguish its effect from clue removal and underlying query difficulty. However, the magnitude varies across models, and our analysis is limited to the 38 matched pairs we found in the dataset. Instances where clues are rephrased indirectly without removing information are relatively rare in our dataset (38 of 702 revisions; 5.4%), which is an interesting finding in itself, so this matched-pair analysis provides our best available test of the effect of indirect expression.
Action. We have revised Finding 1 to clarify that Figure 2 reports associations rather than isolated effects and added the matched-revision experiment to the appendix.