acceptodds
Withdrawn from ICLR 2027

SGR-BENCH: Benchmarking Search Agents on State-Gated Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in large language models and tool-using agents have expanded the range of benchmarked web tasks. Yet an important class of specialized retrieval tasks remains undercharacterized. On many specialized data-retrieval websites, answer-bearing evidence becomes accessible only after establishing the correct site-specific retrieval state through filters, views, hierarchies, or scopes. We term this capability state-gated retrieval (SGR). We introduce SGR-BENCH, a benchmark for this setting containing 134 task records across 14 public data ecosystems. Each task requires discovering the appropriate website and configuring its site-specific retrieval state to produce a structured answer. SGR-BENCH pairs constraint-guided and goal-oriented formulations of the same underlying problems, enabling controlled comparisons between explicit and implicit guidance for state-gated retrieval. We evaluate nine CLI-based agentic LLM systems and three commercial search-agent products on the 134-task benchmark. GPT-5.6-sol achieves the highest Overall Row-F1 at 62.49%. A manual audit of 156 analyzable failed CLI trajectories shows why: agents often reach a relevant web source, but establish the wrong site-specific retrieval state. Retrieval-scope drift (37.2%) and criterion mismatch (27.6%) dominate, whereas final answer composition accounts for only 10.3%. The dataset and single-case evaluation instructions are available at https://huggingface.co/datasets/sgr-bench/SGR-Bench.

Primary area:
datasets and benchmarks
Keywords:
Search Agents, Web Retrieval, Information Retrieval, Tool-Using Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nwwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SGR-BENCH, a benchmark for evaluating state-gated retrieval (SGR) in search agents. The key motivation is that finding the correct website is often insufficient for specialized retrieval tasks since agents must set appropriate filters, hierarchies, and retrieval scopes to access relevant evidence. The authors construct 134 task records across 14 public data ecosystems, with paired constraint-guided and goal-oriented formulations. They evaluate nine CLI-based agentic systems and three commercial search agents, and analyze failure trajectories to identify common retrieval errors.

Overall, the paper addresses a relevant problem and provides a useful benchmark with extensive empirical evaluation.

Critical strengths
  1. The paper identifies an important challenge that retrieving information from specialized websites often requires establishing the correct retrieval state rather than simply locating relevant sources. Explicitly benchmarking this capability is meaningful and practically relevant.

  2. The benchmark construction follows a systematic pipeline with six design requirements and three rounds of expert validation. The paired constraint-guided and goal-oriented tasks also provide a useful setting for studying how retrieval guidance affects agent performance.

  3. The experiments cover 12 search-agent systems and multiple specialized domains. The trajectory-level failure analysis provides useful insights beyond final-answer accuracy, particularly regarding retrieval-scope drift and criterion mismatch.

Critical weaknesses
  1. The benchmark does not clearly isolate state-gated retrieval from general multi-step search and information aggregation. For example, the Europe PMC task (Appendix B, europemc_003) primarily involves locating two articles and cross-referencing information across their tables, without clearly demonstrating which site-specific retrieval state must be set to access otherwise unavailable evidence.

  2. The evidence supporting the paper's central claim that retrieval-state configuration is the primary bottleneck remains insufficient. The gap between Item-F1 and Row-F1 does not necessarily indicate retrieval-state failures, since Row-F1 requires all fields in a row to be correct. Such a gap could also result from incorrect field extraction or answer formatting, even when the retrieval state is correct. More direct evidence is needed to support the claim that state configuration is the primary bottleneck.

  3. The failure analysis covers only 156 failed trajectories from eight CLI-based systems, excluding the best-performing GPT-5.6-sol. It remains unclear whether this subset is representative of the full benchmark and how the consistency of manual error categorization is ensured.

Questions for authors
  1. How does SGR-BENCH distinguish SGR from general multi-step search and information aggregation?

  2. Can the authors provide separate success rates for source discovery and retrieval-state establishment to support the claim that state configuration is the primary bottleneck?

  3. How representative is the manual failure analysis, and why is the best-performing GPT-5.6-sol excluded?

Reviewer_T3D5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Defines state-gated retrieval, answers only show up on a site after the right filters, views, or scopes are set. it releases 134 tasks (67 paired formulations) from 14 data sites, evaluates 12 systems, and says most failures come from retrieval-state errors.

Critical strengths

The task setting is relevant and underexplored. Paired formulations and rubrics are carefully built. I checked the release against the paper. 134 records, family sizes 36/22/22/22/22/10, and answers of 2-44 rows (mean 7.31) all match. Fig 2(a), the colors in Fig 3, and Fig 4 also match the underlying counts exactly, so I dont think numbers were made up.

Critical weaknesses

I checked the paper against the released dataset (accessed 10/15) and env repo. Several reported results cant come from the released 134-task benchmark.

  • Fig 3 does not seem to be computed on 134 tasks. Using the released family sizes, the overall Item-F1 in Table 5 forces impossible Vulnerability-family scores for 8 of 10 systems, e.g. GLM-5.1 136.2, Kimi K2.5 197.8, Seed-2.0 -65.6. The heatmap colors match its printed numbers so it was drawn from some real matrix, likely a subset. This matches the five families in the 22-task audit.

  • P.O.A. doesnt match its definition. App. A says each value is averaged over 67 tasks, but none of the 24 P.O.A. values in Table 2 are on the 1/67 grid. Ten are exact multiples of 1/11 (45.45 appears 4 times). This looks like 11 tasks per formulation.

  • arxiv_003 scores in App. F are from a different task version? A 3-row answer gets Row-F1 50.0 and Item-F1 68.8, but against the released 27-row gold answer both are capped at 20%. The reported numbers only work with a 5-row (or 1-row) reference.

  • Audit labels exist in two versions. The Scholarly bar in Fig 5 adds up to 33/7/8/6/5/3 (n=62), but Table 8 says 35/7/8/6/5/3 (n=64). In Fig 6 a trajectory labeled “Correct” scores 0, which explains the 95.0 mean in Table 9 and contradicts the label.

  • consumerfinance_012 is a task defect, not a model failure. The paper says the agent queried “medical bills OR doctors” instead of the required “medical OR doctor”, but the released prompt and rubric for both formulations say “medical bills or doctors”. How many of the criterion-mismatch failures (27.6%) are actually task defects?

  • Reproducibility claims dont match the release. Sec 3.3 says offline envs cover 60 records but the env repo ships 34. Neither repo has canonicalization or scoring code. Env snapshots are dated 2026-09-25 while all access dates in the paper are 2026-05-06. The run script routes Seed-2.0 Pro thru a non-OpenRouter API, contrary to App. C. genome_003 and genome_004 are cross-paired between the two splits, contrary to the README.

Questions for authors

Which task subset and benchmark version were used for Fig 3, the P.O.A. column, and the App. F scores?Can you release per-task, per-system outputs and the scoring code so Table 2 can be reproduced from the released data? Which version of the audit labels is used in Tables 7-9 and Figs 4-6? How was the “required” query in consumerfinance_012 derived from a prompt that does not state it?

If you could answer and correct the above integrity problems, i have a few more on novelty and design.

  1. Most tasks you have are on live websites, so gold answers can change over time. Offline envs only cover 34 of 134 tasks and were built after the experiments. Please explain how the live tasks can be reproduced. Or it is not intended to be reproduced.

  2. Your gold answers have different formats. Some have headers and one row per line, some (Census, NOAA) put all rows in one line. climategov_012 has two answers in the oracle field, with and without °Df. This should be handled in scoring, not in the answer.

  3. How were the tasks checked? consumerfinance_012 has a rubric that doesnt match its prompt. Please report how tasks were validated and if there was a second annotator.

  4. GPT-5.6-sol is the best model in table 2 but is not in fig 2, fig 3 or the trajectory audit. So the failure analysis does not include the strongest model.

If the authors release per-task outputs and scoring code, and Table 2, fig 3, P.O.A. and App. F all hold on the 134 tasks, I would raise my score. Otherwise I cant tell which results describe this benchmark.

Reviewer_raUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SGR-Bench for evaluating search agents on specialized data-retrieval websites. Agents need to find the appropriate website, set the correct filters and retrieval scope, and return structured answers. The benchmark contains 134 records arranged into 67 pairs, with explicit retrieval guidance in one variant and more implicit guidance in the other. The authors evaluate 12 systems and analyze their failures. The results suggest that maintaining the correct retrieval scope and criteria is difficult for current agents.

Critical strengths
  1. I find the task definition interesting. Finding a relevant website is often not enough, and correctly using its filters and collecting complete results is useful in practical search.
  2. The tasks seem challenging. The paired questions and trajectory analysis also help explain where agents fail, beyond reporting the final scores.
Critical weaknesses
  1. To me, the tasks currently look like a combination of source discovery and within-site retrieval. I am not yet convinced that they measure a distinct new capability. Without experiments that separate these parts, the difference from existing search and browser benchmarks remains unclear.
  2. The expert checks for shortcut resistance are useful, but quantitative evidence is limited. The audit shows that correct answers have official evidence, but this does not necessarily mean that the required state-setting operations were needed to obtain it.
Questions for authors
  1. Could you give the model the correct website and let it search only within that site? I want to see whether searching outside the site is actually necessary. If the scores are similar, how different is the remaining task from the list-filtering tasks in WorkArena or the navigation-and-filtering tasks in WorkArena++?
  2. Could you also give the model the correctly filtered results page? Comparing these settings would show whether the hard part is finding the website, setting the filters, or reading and organizing the results.
  3. Do you have numbers showing how often models can bypass the intended state-setting steps? Retrieving official evidence does not necessarily mean those steps were needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.