acceptodds
Under review as a conference paper at ICLR 2027

Searching in the Dark: How Do Search-Augmented LLMs Handle Abstract Queries?

Abstract

Current search benchmarks largely focus on well-specified, multi-constraint queries that determine a target answer. However, in practice, users often omit constraints and may describe what they are looking for indirectly. We study abstract search: queries that leave multiple plausible answers, and whose available constraints may be expressed explicitly or described indirectly. We construct a dataset of 515 queries spanning two search settings: 255 human-authored Entity Recall queries, where a user has a target entity in mind but provides partial or indirect cues that could fit other candidates and 260 Open Discovery queries, where users state their criteria directly but those criteria include multiple valid candidates. Across seven search-augmented LLMs, we find that models struggle to balance retrieval breadth and candidate validity: most models either retrieve too conservatively and miss valid candidates or retrieve too broadly and admit invalid candidates, with no model achieving both high precision and coverage. We observe that models struggle to resolve indirect descriptions in underspecified queries, with the strongest model recovering only 67.5% of user-intended entities in Entity Recall and indirect descriptions are associated with the largest drop in retrieval performance. Models also remain overconfident: Web search improves calibration relative to parametric memory, but this improvement comes from accuracy rising towards confidence, not from models becoming more cautious. Overall, our findings highlight the need to characterise search behaviour beyond well-specified settings and reveal important blind spots that emerge when users search abstractly.

Primary area:
datasets and benchmarks
Keywords:
search-augmented language models, conversational search, information retrieval, search agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u9nnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how search-augmented LLMs perform on abstract search queries, where queries are underspecified and may admit multiple valid answers. The authors propose a taxonomy based on two axes: specification and directness, and construct a benchmark of 515 queries spanning two settings, "entity recall" and "open discovery". To avoid requiring an exhaustive ground-truth answer set, the authors evaluate returned candidates against query-specific criteria using an automated Web-enabled verifier. Seven LLMs are evaluated using a shared search harness, measuring target recovery, candidate precision, coverage, and confidence calibration.

The results suggest that models struggle to recover intended entities from indirect descriptions, face a trade-off between retrieval breadth and candidate validity, and remain overconfident even with Web search. The paper contributes a benchmark and an empirical analysis of search behaviour under underspecification.

Critical strengths

I find this paper investigates an interesting and timely problem. Existing search benchmarks largely focus on predetermined answers, whereas real-world queries may be incomplete, indirectly expressed, or admit several valid answers. The distinction between specification and directness is useful, particularly in framing the entity recall (ER) and open discovery (OD) tasks. I think this is a meaningful contribution as it highlights search behaviours that existing benchmarks may overlook.

The criteria-based evaluation framework is a practical contribution. Rather than requiring exhaustive ground-truth answer sets, the authors assess whether the candidates returned by LLMs satisfy the stated constraints. This is especially relevant for OD, where enumerating every valid answer may be infeasible. The empirical evaluation is also reasonably thorough, covering seven LLMs under a shared search harness. The 90.4% agreement between human annotations and automated verification is reassuring, and the sensitivity and calibration analyses in the appendices further strengthen the evaluation. This framework makes the benchmark usable in settings where gold answer sets are impractical, which is a key reason for my positive assessment.

Critical weaknesses

However, it is not clear if the benchmark isolates difficulties specific to abstract search, as compared to general retrieval difficulty. In Section 3, the authors explain that ER queries are modified by removing or rewriting clues until a solver fails. While this produces challenging queries, it is difficult to ascertain if lower performance is attributable specifically to underspecification or indirect expression, rather than to general retrieval difficulty (e.g., obscure entities or queries tuned to defeat a particular solver). Figure 2 partially addresses this by examining different types of query revision, but the authors also acknowledge that edit types can co-occur. A more controlled comparison would strengthen the paper's central claim that abstract search exposes distinct challenges.

The interpretation of hit rate in ER could be clearer. The authors explicitly allow multiple valid candidates to satisfy a query, yet hit rate measures whether the model recovers the annotator's intended seed entity. This means a model may return candidates that satisfy every stated criterion, and still miss the seed entity. Since Hit Rate is the primary metric for ER, it would be informative to report how often seed-recovery failures can still produce at least one valid alternative. This would help distinguish the failure to recover the user's intended entity from the failure to satisfy the query as written.

Questions for authors

Clarification requests:

  1. Figure 2 shows that indirect descriptions are associated with reduced hit rate, but query revisions may involve several simultaneous changes. Can the authors isolate the effect of indirect expression from clue removal or other changes, for example through matched revisions that preserve the underlying information? This would help clarify whether the observed degradation is specific to indirectness rather than general retrieval difficulty.

  2. For ER, what fraction of queries where the intended seed entity is not recovered nevertheless yield at least one candidate satisfying every stated criterion? This would help contextualize Hit Rate as a measure of intended-entity recovery rather than overall validity of returned answers.

Minor Comments:

  • A few typographical errors should be corrected, including "descrubing" (line 114), the malformed wording around "sampFinally" (lines 201–202), and "Tanl C" (line 230). These are minor presentation issues rather than soundness concerns.
3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We thank the reviewer for the positive assessment of our work, particularly for recognising the gaps in existing search benchmarks and the value of our framework in addressing them. The reviewer's questions and feedback have been helpful in strengthening our contribution. We address the specific concerns below and have updated the manuscript accordingly.

W1. Isolating the effect of indirect expression

Q. Can the authors isolate the effect of indirect expression from clue removal or other changes, for example through matched revisions that preserve the underlying information?

A. Yes. We compared matched revisions that preserve the underlying information and found an average 8.8 percentage-point drop in Hit Rate and an additional 11.0 percentage-point loss of previously recovered entities beyond variations in unchanged query reruns (95% CI: [−19.1, −3.8]).

Evidence. In Figure 2, we show that queries with indirect descriptions as an edit type are associated with some of the largest drops in Hit Rate. However, we agree that this alone does not isolate the effect of indirect descriptions because users often rewrite clues indirectly or remove them at the same time (r = 0.66) together with other edits, as shown in Appendix B.

We screened the 702 query revisions in our dataset and manually identified 38 matched query pairs that preserve the underlying information without removing clues. We then compared retrieval performance across three models, with three independent runs each. To account for variation across runs, we measured how often a model failed to recover a previously recovered entity when the original query was rerun unchanged. The table below reports the changes in Hit Rate, the loss after indirect rephrasing on entities that were previously recoverable, the loss from unchanged-query reruns, and the additional drop after accounting for rerun variability.

ModelΔ Hit RateΔ Indirect rephrasingΔ Unchanged rerunAdditional drop vs. rerun (95% CI)
Gemini 3.7 Flash−9.6 pp−11.6 pp−2.8 pp−8.9 pp [−19.3, −0.6]
Claude Sonnet 5−14.9 pp−21.4 pp−3.2 pp−18.2 pp [−29.8, −7.9]
GPT-5.6 Luna−1.8 pp−11.3 pp−5.2 pp−6.2 pp [−14.6, +1.6]
Pooled−8.8 pp−14.6 pp−3.7 pp−11.0 pp [−19.1, −3.8]

Negative values indicate a drop in performance. The pooled loss rates are calculated across paired runs. Confidence intervals use query-level bootstrap resampling.

Takeaway. The matched-revision experiment provides additional evidence that indirect expression is associated with reduced retrieval performance even when the underlying information is preserved. This helps distinguish its effect from clue removal and underlying query difficulty. However, the magnitude varies across models, and our analysis is limited to the 38 matched pairs we found in the dataset. Instances where clues are rephrased indirectly without removing information are relatively rare in our dataset (38 of 702 revisions; 5.4%), which is an interesting finding in itself, so this matched-pair analysis provides our best available test of the effect of indirect expression.

Action. We have revised Finding 1 to clarify that Figure 2 reports associations rather than isolated effects and added the matched-revision experiment to the appendix.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

W2. Contextualise Hit Rate as a measure of intended-entity recovery and not validity

Q. For ER, what fraction of queries where the intended seed entity is not recovered nevertheless yield at least one candidate satisfying every stated criterion?

A. Across all models, 35.47% of model–query pairs where the user-intended seed entity is not recovered still return at least one candidate satisfying every stated criterion.

Evidence. Our existing Precision and Coverage@4 metrics already show the validity and breadth of returned candidates. However, they do not explicitly show how often a model returns at least one valid candidate when the user-intended entity is not recovered. We now report Validity Rate (%), which measures the percentage of all 255 ER queries per model where at least one candidate satisfies every stated criterion, alongside Hit Rate.

ModelHit Rate (%) ↑Validity Rate (%) ↑Valid alternative if seed missed (%) ↑Neither seed nor valid (%) ↓
GPT-5.6 Sol67.4579.6143.3718.43
Claude Opus 564.7181.5747.7818.43
Gemini 3.7 Flash61.1872.5531.3126.67
DeepSeek V4 Pro53.3369.8036.1329.80
GPT-5.6 Luna47.8466.6737.5932.55
Claude Sonnet 538.0454.9028.4844.31
Qwen 3.8 Max36.8655.6931.6843.14
All models52.7768.6835.4730.48

Hit Rate and Validity Rate use all ER queries. Valid alternative if seed missed is conditional on missing the seed; Neither seed nor valid uses all model–query pairs.

Takeaway. The results reveal two distinct failure points: failing to recover the entity the user had in mind and failing to return any valid candidate. Across models, 47.23% of model–query pairs miss the intended entity. Of these, 35.47% still return a valid alternative, while 64.53% do not (equivalent to 30.48% of all model–query pairs).

Action. We have updated the main Entity Recall results table to include Validity Rate (%) and added the per-model breakdown to the appendix.

We have corrected the typographical errors identified by the reviewer. We hope these additional analyses and revisions address the reviewer's concerns and clarify the contributions of our work.

As the discussion period comes to an end, we wanted to check whether our responses and revisions have addressed your concerns. We have updated the manuscript, with changes highlighted in purple:

  • W1. Isolating the effect of indirectness — L343–364; Appendix F.3.
  • W2. Contextualise Hit Rate as a measure of intended-entity recovery and not validity — Table 1, L332–341; Appendix F.1.

Thank you again for your feedback. Please let us know if any concerns remain or if we can clarify anything further.

Reviewer_c2sKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how search-augmented LLMs respond to underspecified queries that may admit multiple valid answers. It introduces 515 queries across two settings: Entity Recall, where users describe a particular entity through partial or indirect clues, and Open Discovery, where users seek candidates satisfying stated criteria. Seven models are evaluated using target recovery, candidate validity, Coverage@K, and confidence calibration. The paper reports differences in candidate selection, performance across query categories, calibration, and source usage.

Critical strengths
  • The paper addresses a practically relevant setting in which users cannot fully specify their information needs and multiple answers may be valid.
  • The dataset construction involves substantial human effort, iterative testing, filtering, and manual review. The shared search environment also improves comparability across models.
  • Evaluating candidate validity separately from target recovery is valuable. The supplementary examples explicitly demonstrate that models can return valid alternatives without recovering the intended target.
  • The cross-domain results and source-usage analysis provide useful descriptive observations. The supplementary calibration and sensitivity analyses help contextualize the main results.
Critical weaknesses
  • Target recovery does not adequately measure the ability to process indirect descriptions. When several candidates satisfy the stated constraints, the original target is distinguished by the annotator’s hidden intent rather than by information available to the model. Failing to return it cannot distinguish misunderstanding from correctly interpreting the query and returning valid alternatives. The paper acknowledges this distinction but does not systematically decompose target-recovery failures into cases with and without valid alternatives. Moreover, adversarial revisions combine information removal and indirect phrasing, preventing the analysis from isolating the effect of indirectness. The claimed difficulty in processing indirect descriptions therefore goes beyond what target recovery alone establishes.

  • The retrieval-breadth conclusion is not sufficiently supported by the output-level metrics. Coverage@K assumes that returning KK valid candidates is an appropriate objective, but the paper does not establish that each query has at least KK valid answers. Low coverage may therefore reflect a small answer set rather than insufficient retrieval. Furthermore, the evaluation does not distinguish candidates that were never retrieved from those encountered but omitted during verification or final selection. Differences in final candidate counts and precision establish different output behaviours, but do not by themselves demonstrate overly conservative search or an inability to balance retrieval breadth and validity.

  • The cross-domain finding is supported descriptively, but its cause remains unresolved. All evaluated models perform worst on the cross-domain subset, making this a meaningful observation. However, cross-domain queries may also involve more demanding multi-hop retrieval, entity linking, constraint combinations, or scarcer valid answers. The separate Nested Sequence category does not control for these factors. Consequently, the results do not isolate cross-domain integration as the bottleneck, and higher search-call counts do not rule out insufficient or inefficient search.

  • The calibration analysis requires a clearer definition of the predicted event. Confidence that a candidate satisfies the description differs from confidence that it is the user’s intended entity. The displayed prompt does not clearly separate these events, while the same confidence values are evaluated against both. Even if confidence concerns target identity, calibration depends on the target-selection distribution, which is shaped here by adversarial construction. The results support overconfidence under the reported evaluation setup, but do not establish that search specifically reduces knowledge uncertainty while residual error reflects failure to handle query uncertainty.

  • The source-usage findings should be interpreted more narrowly. The reported Wikipedia read share measures the proportion of model–query pairs that opened the domain, rather than its share of all page reads. Actual visits also depend on search-engine exposure, ranking, and accessibility, so they do not isolate model preferences. Additionally, the search backend did not enforce the requested site: restrictions: these requests indicate intended source selection, not successful domain-restricted retrieval. The analysis describes source-use patterns but does not establish their causes or effectiveness.

Questions for authors

See weaknesses above.

6 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We thank the reviewer for their detailed feedback on our work, which has been helpful in sharpening our claims. The main concerns raised relate to the interpretation of our findings. We address each concern below and, where needed, have clarified our claims or provided additional analyses to substantiate our findings.

W1. Target recovery and valid alternatives

Q. The paper does not systematically decompose target-recovery failures into cases with and without valid alternatives.

A. We now report Validity Rate alongside Hit Rate to decompose target-recovery failures into cases with and without valid alternatives. Across models, 47.23% of model–query pairs miss the intended entity. Of these, 35.47% still return a valid alternative, while 64.53% do not.

Clarification to Finding 1 (§5). We have updated the framing of our original finding to better distinguish failure to recover the intended entity from failure to return a valid answer:

  • Original: “Models struggle to resolve indirect descriptions in underspecified queries.”
  • → Clarified: “Models often fail to recover user-intended entities on Entity Recall queries, with only about a third of these failures returning a valid alternative.”

Evidence. The table below reports Validity Rate, the proportion of queries with at least one valid candidate, alongside the decomposition of target-recovery failures.

ModelHit Rate (%) ↑Validity Rate (%) ↑Valid alternative when seed missed (%) ↑Neither seed nor valid (%) ↓
GPT-5.6 Sol67.4579.6143.3718.43
Claude Opus 564.7181.5747.7818.43
Gemini 3.7 Flash61.1872.5531.3126.67
DeepSeek V4 Pro53.3369.8036.1329.80
GPT-5.6 Luna47.8466.6737.5932.55
Claude Sonnet 538.0454.9028.4844.31
Qwen 3.8 Max36.8655.6931.6843.14
All models52.7768.6835.4730.48

Valid alternative when seed missed is conditional on missing the seed; Neither seed nor valid uses all model–query pairs.


W2. Isolating the effect of indirectness

Q. Adversarial revisions combine information removal and indirect phrasing, preventing the analysis from isolating the effect of indirectness. The claimed difficulty in processing indirect descriptions therefore goes beyond what target recovery alone establishes.

A. We isolated revisions that preserve the underlying information while rephrasing clues indirectly. Across three models, we find that indirect rephrasing reduced Hit Rate by 8.8 percentage points and caused an additional 11.0 percentage-point loss of previously recovered entities relative to unchanged-query reruns (95% CI [−19.1, −3.8]).

Additional evidence supporting Finding 1 (§5). We have updated the paper to include the new matched-pair analysis, which provides further evidence that indirect rephrasing reduces intended-entity recovery even when the underlying information is preserved, although the magnitude of this effect varies across models.

Evidence. We screened 702 query revisions and manually identified 38 matched pairs that preserve the underlying clues. We evaluated three models across three independent runs, comparing indirect rephrasing against the original query and unchanged-query reruns to account for run-to-run variation.

ModelΔ Hit RateLoss after indirect rephrasingLoss on unchanged rerunAdditional loss (95% CI)
Gemini 3.7 Flash−9.6 pp−11.6 pp−2.8 pp−8.9 pp [−19.3, −0.6]
Claude Sonnet 5−14.9 pp−21.4 pp−3.2 pp−18.2 pp [−29.8, −7.9]
GPT-5.6 Luna−1.8 pp−11.3 pp−5.2 pp−6.2 pp [−14.6, +1.6]
Pooled−8.8 pp−14.6 pp−3.7 pp−11.0 pp [−19.1, −3.8]

Negative values indicate reduced recovery. Loss rates are measured among previously recovered entities. Confidence intervals use query-level bootstrap resampling.

Takeaway. The matched-revision experiment provides additional evidence that indirect expression is associated with reduced retrieval performance even when the underlying information is preserved. This helps distinguish its effect from clue removal and underlying query difficulty.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

W3. Retrieval-breadth conclusion is not sufficiently supported by the output-level metrics

Q. Coverage@K assumes that returning K valid candidates is an appropriate objective, but the paper does not establish that each query has at least K valid answers. Differences in final candidate counts and precision do not establish conservative search behaviour.

A. We pooled distinct valid candidates from all seven models to approximate answer availability and re-evaluated 158 Open Discovery queries with at least five observed valid answers. We observe that Precision–Coverage trade-offs persist on this subset.

Our output-level metrics cannot distinguish candidates never retrieved from those considered but omitted during verification or selection. We therefore limit Finding 2 to returned candidate sets.

Clarification to Finding 2 (§5).

Original paperRevised paper
Finding 2“Models struggle to balance retrieval breadth and candidate validity.”→ “Models vary in how well they balance candidate-set breadth and validity.”
Interpretation“Models either retrieve too conservatively, sacrificing Coverage, or retrieve broadly and include invalid candidates, sacrificing Precision.”→ “Models exhibit different Precision–Coverage trade-offs in their returned candidate sets, with some achieving higher Coverage at lower Precision, while others perform well on both.”

Evidence. The table below reports Precision, Coverage@5, and F1@5 on the 158/260 Open Discovery queries (60.8%) with at least five distinct valid candidates pooled across the seven models.

ModelPrecision (%) ↑Coverage@5 (%) ↑F1@5 (%) ↑
GPT-5.6 Sol79.981.077.6
Gemini 3.7 Flash74.772.871.5
Claude Opus 564.180.169.8
DeepSeek V4 Pro74.864.966.7
GPT-5.6 Luna70.867.266.1
Qwen 3.8 Max54.577.362.0
Claude Sonnet 549.353.750.3

Claude Opus 5 and Qwen 3.8 Max achieve relatively high Coverage at lower Precision, while GPT-5.6 Sol performs well on both. However, we note that the pooled candidates approximate answer availability based on what all models found rather than the exhaustive set of valid answers that actually exist.


W4. The cross-domain finding is supported descriptively, but its cause remains unresolved

Q. Cross-domain queries may involve more demanding multi-hop retrieval, entity linking, constraint combinations, or scarcer valid answers. The results do not isolate cross-domain integration as the bottleneck, and higher search-call counts do not rule out insufficient or inefficient search.

A. We tested whether cross-domain difficulty persists when sufficient valid answers are available. On queries with at least five observed valid answers, cross-domain remains the lowest-performing category (61.5% F1@5), compared with Nested Sequence (63.7%) and other categories (70.1%). This strengthens our finding, although it does not establish its underlying cause.

Clarification to Finding 3 (§5).

  • Original: “Models therefore search more but surface fewer candidates, suggesting that the bottleneck is not search effort itself, but finding candidates that satisfy constraints across domains.”

  • → Revised: “Models therefore search more but surface fewer candidates on cross-domain queries, which remain the lowest-performing category even when at least five valid answers are observed. However, these results alone do not establish the underlying cause of this difficulty.”

Evidence. We compared F1@5 across categories before and after restricting to queries with at least five distinct valid candidates pooled across the seven models.

Query typeAll queries F1@5 (%) ↑Queries with ≥5 valid answersSubset F1@5 (%) ↑
Cross-domain41.237/71 (52.1%)61.5
Nested Sequence48.144/68 (64.7%)63.7
Other categories51.777/121 (63.6%)70.1
Overall47.9158/260 (60.8%)66.3

Isolating the effects of multi-hop retrieval, entity linking, and constraint combinations would require further controlled experiments, which we leave for future work.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

W5. The calibration analysis requires a clearer definition of the predicted event

Q. The prompt does not clearly distinguish confidence in recovering the intended entity from confidence in satisfying the description. The results support overconfidence but do not isolate knowledge uncertainty from query uncertainty.

A. We clarify that our primary calibration event is whether the highest-confidence candidate matches the user-intended seed entity. Our Entity Recall prompt (Appendix D.3, Figure 16) frames the task as recovering a specific entity:

The user is trying to recall something from memory using an abstract or
underspecified description. They know what they’re looking for and will
recognise it among a set of candidates once they see it, even if they
cannot specify all the constraints and concrete details required to
retrieve it.
...
For each candidate, confidence is your calibrated estimate (0 to 100)
that it is correct, not a ranking.

We evaluate the reported confidence scores against two explicitly defined events:

  1. Intended-entity recovery: Whether the highest-confidence candidate matches the user-intended seed entity.

  2. Criterion satisfaction: Whether the highest-confidence candidate satisfies all criteria expressed in the query, regardless of whether it matches the seed entity.

We recognise that models may interpret “correct” as either event, and we cannot determine which interpretation each model follows. Because of this, we reported calibration against both outcomes (Appendix G, Tables 12–13 and Figure 20). We find that overconfidence persists for most models under both definitions.

Our parametric baseline shows that Web search improves target recovery and, for six of seven models, calibration, but does not separately isolate knowledge uncertainty from query uncertainty. We have narrowed Finding 4 accordingly. Our calibration results concern our constructed Entity Recall queries, not a general user-query distribution.

Clarification to Finding 4 (§5).

  • Original: “Search reduces knowledge uncertainty, but models remain overconfident under query uncertainty.”

  • → Revised: “Search improves target recovery, but models remain overconfident on abstract search queries.”


W6. Source-usage findings should be interpreted more narrowly

Q. Wikipedia read share measures model–query pairs rather than all page reads. Source visits depend on search-engine exposure, ranking, and accessibility. site: requests do not establish successful domain-restricted retrieval, and source-use patterns do not establish their causes or effectiveness.

A. We now report the share of all page reads in Appendix J and confirm that Wikipedia remains the most-read domain across all seven models. We also examine search-engine exposure and clarify that our source-prior findings concern domain-targeting requests, not successful domain-restricted retrieval.

Clarification to Finding 5 (§5).

  • Original: “Wikipedia dominates model reads, while search models exhibit source priors.”

  • → Clarified: “Wikipedia is the most-read domain across models, while model-generated search queries exhibit topic-specific source priors.”

We now report the share of all page reads, ranked by domain:

ModelWikipedia (%)Second domain (%)Third domain (%)
GPT-5.6 Sol8.68worldbank.org (3.42)sec.gov (2.49)
GPT-5.6 Luna11.52sec.gov (2.67)nih.gov (1.87)
Claude Opus 533.47sec.gov (4.23)wmcloud.org (1.39)
Claude Sonnet 523.24sec.gov (3.89)arxiv.org (1.04)
Gemini 3.7 Flash30.79garaph.info (4.19)sec.gov (3.35)
DeepSeek V4 Pro21.48junoawards.ca (2.61)sec.gov (2.56)
Qwen 3.8 Max26.27sec.gov (4.07)congress.gov (1.41)

On search-engine exposure. Wikipedia accounts for 7.10–11.35% of surfaced URLs but 8.68–33.47% of page reads. We observe that reading is not directly proportional to exposure, though we acknowledge that ranking and accessibility may also contribute to this. We believe that the shared harness gives a comparable environment for all the models though it may not fully eliminate the issues raised.

On source priors and effectiveness. As explained in Appendix J, Exa does not enforce site: requests, which indicate intended source targeting rather than pages actually read through Jina. Of 270 Entity Recall requests naming Reddit communities, 251 target four communities for identifying half-remembered things. This supports topic-specific source priors, not their effectiveness. Our aim is to characterise source selection; testing its effect on performance would require controlled experiments beyond this work and is a direction for future research.

Reviewer_v24EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper builds a search benchmark around queries that can be satisfied by several answers valid, either because the stated constraints don't pin down one answer or because they describe the target indirectly, through descriptions or vague recollections instead of direct names. Existing benchmarks mostly use queries that determine a single answer. The authors build 515 queries: 255 Entity Recall (partial clues) queries, human-written and refined, and 260 synthetic Open Discovery (multiple valid candidates) queries. With no exhaustive gold set possible, they check each returned candidate against web-verifiable criteria using an LLM verifier (human agreement κ = 0.78). Across several search-augmented LLMs, models struggle with indirect descriptions, trade precision against coverage, and stay overconfident: search improves calibration only because accuracy rises, while stated confidence remains well above accuracy.

Critical strengths

Strengths

  • The problem is well-motivated. Appendix A estimates about 65% of real search queries are underspecified. The paper identifies a clear gap in existing benchmarks.

  • The evaluation approach (web-verifiable criteria checked by a verifier) avoids needing an exhaustive gold set, and it was validated against human annotators (κ = 0.78).

  • Experiments separate target recovery (Hit Rate) from candidate validity (Precision, Coverage), which goes beyond single-answer benchmarks hide.

  • Experiments involve 515 queries, seven models, about 124k criterion judgments, a shared search harness, and documented filtering funnels.

  • The paper provides diagnostics beyond aggregate performance: invoking search raises accuracy toward already-high confidence without making models more cautious, and cross-domain queries are hardest despite the most search effort.

Critical weaknesses

Limitations

  • Repeated run samples may be preferable. With n = 255, the top three Hit Rates may be within noise bands.

  • Hit Rate is noisy because the seed need not be the only valid answer, and it is not normalised for list length. There is no human baseline.

  • Coverage@K is not faithful to (elusive) true recall, and K is set from the systems’ own median output.

  • There is a potential self-preference danger with Gemini 3.7 Flash as the criteria-based verifier. The paper might re-score with a different verifier or an ensemble.

  • Validation by humans is at the granularity of a single criterion. However, a candidate counts as valid only if all critera stand validated, and queries average about 6 criteria. So the candidate-level agreement is unclear and likely lower.

  • Order matters: a false NO wrongly invalidates a valid candidate, while a false YES only matters if it is the last failing criterion.

  • Lack of evidence (which can be common in a data set spanning 41 countries and 39 languages) counts as invalidation. I don't see a way around this, but the authors should note this as a limitation.

  • A few related work gaps: DETOUR and other close papers are not cited, and the difference from BLUR in the effect of search is not discussed.

Assessment

AxisRatingRationale
NoveltyModerateThe specificity-by-directness taxonomy and the fixed-query, open-set, criteria-verified setup are not combined in any single prior work. Close neighbours (BLUR, InteractComp, QUEST, DeepSearchQA, DETOUR) cover pieces, and the claimed gap is partly narrow.
CreativityModestUsing verification asymmetry to avoid an exhaustive gold set is a sensible idea, as is separating target recovery from candidate validity. The query construction pipelines (adversarial refinement, Generator–Solver loop) are creative but built from known ingredients, and have limitations.
Technical depthLowThere is no new method or theory. The Coverage@K and F1@K metrics are simple, and Coverage@K is a sufficiency measure, not recall. The depth sits in dataset engineering and verifier design.
Experimental rigorModerateThe shared harness, filtering funnels and human validation are good. Some rankings are within noise, the verifier is also an evaluated model, validation is only at criterion (not candidate) level, and there are confounds across models (reasoning effort, search budget, output format).
Data insightGoodThe calibration finding (accuracy rises toward high confidence, with no added caution), the cross-domain difficulty result and the case studies are informative.
ImpactModerateIt points evaluation toward realistic underspecified search, a common real-world case. Whether it changes practice depends on adoption of the benchmark and on fixes to the metric concerns.
UtilityGoodThe data, code and traces are released, the setup is reusable, and the criteria-based framework could apply to other open-ended retrieval tasks.

Additional references

Questions for authors

Suggestions about experiments

  • Report candidate-level human agreement.

  • Test a second verifier from a different family (or ensemble) to rule out self-preference.

  • Report how often verdicts are NO because evidence was missing as opposed to contradicted.

Suggestions about writing

"Red team" is used inappropriately.

906 versus 1,039 generated queries, ECE values that differ between text and tables, a truncated sentence in Section 3, and the weak lexical-overlap analysis in Appendix K.

Figure 1: "Select, verify, and stop among several valid candidates" is unclear, what does it mean to "stop among candidates"?

The authors make fair effort to define and explain, but I appreciate the specificity dimension better than the directness dimension. I am not sure I can conjure up more examples of my own on the directness axis. I also don't understand if/how these axes are centrally important to the submission.

L201 Fix typo: "returned in any sampFi-nally, the authors"

4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We thank the reviewer for their thorough review and for recognising the motivation of our work, its contribution beyond single-answer benchmarks, and the diagnostic insights provided by our evaluation framework. We address the concerns raised below and provide the suggested experiments.

W1. Report candidate-level human agreement

Q. Validation by humans is at the granularity of a single criterion. However, a candidate counts as valid only if all critera stand validated, and queries average about 6 criteria. So the candidate-level agreement is unclear and likely lower.

A. Candidate-level agreement is 88.9% (κ = 0.723), slightly lower than criterion-level agreement of 90.4% (κ = 0.780) but remains substantial.

Evidence. We now report candidate-level agreement alongside the existing criterion-level validation of the Gemini 3.7 Flash verifier.

ComparisonCriterion agreementCriterion κCandidate agreementCandidate κ
Annotators with each other85.9%0.67586.7%0.650
Verifier vs. annotators90.4%0.78088.9%0.723
Verifier vs. human majority94.6%0.87693.3%0.831

W2. Testing Gemini 3.7 Flash verifier self-preference

Q. There is a potential self-preference danger with Gemini 3.7 Flash as the criteria-based verifier. The paper might re-score with a different verifier or an ensemble.

A. We tested GPT-5.6 Luna as a second verifier from a different family and found no statistically significant evidence of self-preference for Gemini 3.7 Flash relative to GPT 5.6 Luna (−3.13 pp, 95% CI: [−7.94, +2.20]).

Evidence. We re-scored 1,133 candidates (7,117 criteria) from 50 random queries (25 per task) using GPT-5.6 Luna, and reran Gemini 3.7 Flash under identical conditions. Claude Opus 5 and DeepSeek V4 Pro serve as unrelated-family controls. The acceptance rate is set at candidate-level: a candidate is accepted if it meets all criteria. The table below provides the acceptance rate for both verifiers.

ModelnGemini 3.7 Flash accepts (%)GPT 5.6 Luna accepts (%)Difference (95% CI)
Gemini 3.7 Flash25153.050.2+2.8 pp [−2.1, +7.8]
Claude Opus 536345.538.3+7.2 pp [+1.2, +13.2]
DeepSeek V4 Pro27858.654.0+4.7 pp [−4.2, +11.9]
GPT-5.6 Luna24156.859.8−2.9 pp [−18.9, +10.2]

Positive differences indicate higher acceptance by Gemini; bold indicates higher acceptance. CIs use query-level bootstrap resampling.

Takeaway. We see that Gemini accepts 2.8 pp more of its own candidates than Luna, compared with 5.9 pp more for unrelated-family models. The adjusted difference is −3.13 pp (95% CI [−7.94, +2.20]), providing no statistically significant evidence of self-preference, although small effects cannot be fully excluded.

Action. We have added this robustness analysis to the appendix and referenced it in the main evaluation section.


W3. Missing evidence vs contradictory evidence

Q. Report how often verdicts are NO because evidence was missing as opposed to contradicted.

A. At the criterion level, 11.9% of NO verdicts were due to missing evidence, compared with 79.9% due to contradictory evidence. At the candidate level, where invalidation occurs, only 5.9% were rejected due to missing evidence, compared with 90.4% due to contradictory evidence found on the web.

Evidence. Using a separate model (Gemini 3.8 Flash), we classified 27,861 criterion-level NO verdicts across 9,792 invalid candidates, providing each criterion and its supporting evidence returned by the verifier without the rejection-reason label. At candidate level, a rejection is contradictory if any failed criterion contradicts available evidence, and insufficient-evidence only if all failed criteria lack evidence.

Reason for rejectionPer criterionPer candidate
Contradicted by available evidence79.9%90.4%
Insufficient evidence11.9%5.9%
Candidate not identifiable as a valid entity (e.g., a summary)8.2%3.6%
Rejections27,8619,792

However, our analysis on its own cannot determine whether candidates rejected for insufficient evidence are genuinely invalid or simply poorly documented on the web. We have acknowledged this in the limitation.

Action. We have added this analysis to the appendix and acknowledged that missing evidence may lead to valid candidates being invalidated in the limitations section.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

W4. Noise Bands in the Top Three Entity Recall Hit Rates

Q. Repeated run samples may be preferable. With n = 255, the top three Hit Rates may be within noise bands.

A. We report 95% bootstrap confidence intervals and paired significance tests. On the 255 Entity Recall queries, we do not observe statistically significant differences among the top three models.

Evidence. We agree that repeated runs would be preferable, but a single evaluation across all seven models costs approximately $1,795, making repeated evaluations expensive. We instead report confidence intervals across the 255 queries using 10,000 bootstrap resamples. The table below reports Hit Rates with their 95% bootstrap confidence intervals.

ModelHit Rate (%) ± 95% CI
GPT-5.6 Sol67.45 ± 5.69
Claude Opus 5`64.71 ± 5.88
Gemini 3.7 Flash61.18 ± 6.08
DeepSeek V4 Pro53.33 ± 6.08
GPT-5.6 Luna47.84 ± 6.08
Claude Sonnet 538.04 ± 6.27
Qwen 3.8 Max36.86 ± 5.88

Because all models answer the same queries, we also conduct paired McNemar tests. We find no statistically significant differences among the top three models, with the comparison between GPT-5.6 Sol and Gemini 3.7 Flash closest to the significance threshold (p = 0.056).

Action. We have added bootstrap confidence intervals to the main results table and paired significance tests to the appendix.


W5. Hit Rate is Noisy

Q. Hit Rate is noisy because the seed need not be the only valid answer, and it is not normalised for list length. There is no human baseline.

A. Hit Rate is our primary metric for Entity Recall because it measures whether a model returns what the user was looking for, but it is not the only metric we report. Our evaluation already includes Precision and Coverage@K to account for list length and candidate validity, and a model oracle to contextualise performance in the absence of a human baseline. We now additionally report Validity Rate to make alternative valid answers more explicit, but does not substitute finding what the user was looking for.

We address the three points briefly below.

1. Noisy because the seed need not be the only valid answer. Our existing Precision and Coverage@K metrics already evaluate the validity of other returned candidates. To make this distinction more explicit, we now report Validity Rate, the proportion of queries for which a model returns at least one valid candidate, regardless of whether it recovers the intended entity.

ModelHit Rate (%) ↑Validity Rate (%) ↑Valid alternative if seed missed (%) ↑Neither seed nor valid answer (%) ↓
GPT-5.6 Sol67.4579.6143.3718.43
Claude Opus 564.7181.5747.7818.43
Gemini 3.7 Flash61.1872.5531.3126.67
DeepSeek V4 Pro53.3369.8036.1329.80
GPT-5.6 Luna47.8466.6737.5932.55
Claude Sonnet 538.0454.9028.4844.31
Qwen 3.8 Max36.8655.6931.6843.14

↑ Higher is better; ↓ lower is better. Bold indicates the best value. The third percentage is conditional on missing the seed; the final percentage is calculated across all Entity Recall queries.

2. Noisy because it is not normalised for list length. Hit Rate checks whether the intended entity is in the returned list and does not, on its own, normalise for list length. This is why our original evaluation also reports Precision, which normalises by the number of returned candidates, and Coverage@4, which measures the number of valid candidates relative to a common target of K = 4.

3. Noisy because there is no human baseline. As discussed in L351, establishing a human baseline is challenging because the queries require prior linguistic and cultural familiarity, making it difficult to find suitable participants across all queries. This is why we instead provided a model oracle, which counts a query as solved if any of the seven models recovers the intended entity. The oracle achieves 84.3% Hit Rate, compared to 67.5% for the best individual model, leaving 15.7% of queries unsolved by any model. This provides a reference point for individual model performance, although it does not replace a human baseline.

Action. We have added Validity Rate to the main Entity Recall results. We have clarified that Hit Rate alone does not provide a diagnostic understanding of model failures and should be interpreted alongside the other evaluation metrics.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

W6. Coverage@K is not faithful to (elusive) true recall

Q. Coverage@K is not faithful to (elusive) true recall, and K is set from the systems’ own median output.

A. We agree that Coverage@K is a sufficiency measure, not true recall, as also noted in the reviewer's Technical Depth assessment. This is consistent with our definition in the manuscript: “Coverage@K measures whether enough valid candidates are returned.”

Evidence. Because the complete set of valid candidates is unknown and exhaustive retrieval is infeasible for abstract queries, true recall cannot be computed. That is why we measure whether models return a sufficient number of valid candidates. Following Wei et al. (2024) [1], we set a common K based on the median number of candidates returned across systems. This provides a reasonable threshold for comparing models, rather than choosing K separately for each system.

We already report sensitivity across K = 1 to 8 in Appendix H and find that Entity Recall rankings remain stable, while some Open Discovery models change rank as K varies.

Action. We have clarified that Coverage@K should be interpreted as a sufficiency measure rather than true recall and highlighted the existing sensitivity analysis in Appendix H. We also acknowledge that results can depend on the choice of K.

[1] Wei et al. (2024). Long-form factuality in large language models. NeurIPS 2024.


W7. Order matters: a false NO wrongly invalidates a valid candidate

Q. Order matters: a false NO wrongly invalidates a valid candidate, while a false YES only matters if it is the last failing criterion.

A. We examined this using our human-annotated validation set of 166 criterion judgments across 30 candidates. None of the six false NO judgments caused a valid candidate to be incorrectly rejected because those candidates already failed other criteria, while two false YES judgments caused invalid candidates to be marked as valid because they were the last failing criterion.

Evidence. Comparing the verifier against the majority labels of three human annotators, we find 94.7% sensitivity (95% CI [88.9, 98.0]) and 94.2% specificity (95% CI [84.1, 98.8]) at the criterion level. At the candidate level, 0 of 7 valid candidates were incorrectly rejected, while 2 of 23 invalid candidates were incorrectly accepted. Given the small number of valid candidates, these results do not rule out false rejections in larger samples.

Action. We have added the sensitivity and specificity analysis to the appendix.

We thank the reviewer for the writing suggestions and additional references shared, which we have added to better position our work and improve the discussion. We hope the additional experiments and clarifications address the concerns raised, and we welcome any further feedback.

As the discussion period comes to an end, we would like to check whether our responses and additional experiments have addressed your concerns. We have incorporated the changes into the revised manuscript, highlighted in purple. The changes are located at:

  • W1. Report candidate-level human agreement — L297–300; Appendix E.
  • W2. Testing Gemini 3.7 Flash verifier self-preference — L300–303; Appendix E.
  • W3. Missing evidence vs contradictory evidence — L304–306; Appendix E.
  • W4. Noise Bands in the Top Three Entity Recall Hit Rates — Table 1; Appendix F.2.
  • W5. Hit Rate is Noisy — L310–325, L332–341; Appendix F.1.
  • W6. Coverage@K is not faithful to (elusive) true recall — L313–317; Appendix H.
  • W7. Order matters: a false NO wrongly invalidates a valid candidate — Appendix E.

We have also incorporated the suggested references and corrected the typographical and presentation issues.

Thank you again for your feedback. Please let us know if anything remains unclear or if we can address any remaining concerns before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.