- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SGR-BENCH, a benchmark for evaluating state-gated retrieval (SGR) in search agents. The key motivation is that finding the correct website is often insufficient for specialized retrieval tasks since agents must set appropriate filters, hierarchies, and retrieval scopes to access relevant evidence. The authors construct 134 task records across 14 public data ecosystems, with paired constraint-guided and goal-oriented formulations. They evaluate nine CLI-based agentic systems and three commercial search agents, and analyze failure trajectories to identify common retrieval errors.
Overall, the paper addresses a relevant problem and provides a useful benchmark with extensive empirical evaluation.
Critical strengths
-
The paper identifies an important challenge that retrieving information from specialized websites often requires establishing the correct retrieval state rather than simply locating relevant sources. Explicitly benchmarking this capability is meaningful and practically relevant.
-
The benchmark construction follows a systematic pipeline with six design requirements and three rounds of expert validation. The paired constraint-guided and goal-oriented tasks also provide a useful setting for studying how retrieval guidance affects agent performance.
-
The experiments cover 12 search-agent systems and multiple specialized domains. The trajectory-level failure analysis provides useful insights beyond final-answer accuracy, particularly regarding retrieval-scope drift and criterion mismatch.
Critical weaknesses
-
The benchmark does not clearly isolate state-gated retrieval from general multi-step search and information aggregation. For example, the Europe PMC task (Appendix B, europemc_003) primarily involves locating two articles and cross-referencing information across their tables, without clearly demonstrating which site-specific retrieval state must be set to access otherwise unavailable evidence.
-
The evidence supporting the paper's central claim that retrieval-state configuration is the primary bottleneck remains insufficient. The gap between Item-F1 and Row-F1 does not necessarily indicate retrieval-state failures, since Row-F1 requires all fields in a row to be correct. Such a gap could also result from incorrect field extraction or answer formatting, even when the retrieval state is correct. More direct evidence is needed to support the claim that state configuration is the primary bottleneck.
-
The failure analysis covers only 156 failed trajectories from eight CLI-based systems, excluding the best-performing GPT-5.6-sol. It remains unclear whether this subset is representative of the full benchmark and how the consistency of manual error categorization is ensured.
Questions for authors
-
How does SGR-BENCH distinguish SGR from general multi-step search and information aggregation?
-
Can the authors provide separate success rates for source discovery and retrieval-state establishment to support the claim that state configuration is the primary bottleneck?
-
How representative is the manual failure analysis, and why is the best-performing GPT-5.6-sol excluded?