acceptodds
Under review as a conference paper at ICLR 2027

Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts

Abstract

Final-answer accuracy does not reveal whether a scientific-search agent failed to encounter a target paper, attempt to inspect it, or return an accepted answer af- ter inspection. We introduce decision checkpoints that record observations and tool actions without benchmark labels during inference, then join target identities and evaluator labels to assign outcome categories from recorded events. Across five conditions on 540 answerable AutoResearchBench Deep questions in a fixed, target-enriched environment, keyword search achieves 24.6% accuracy, compared with 17.8% for raw search. The keyword condition has fewer incorrect answers with neither target exposure nor inspection, but more incorrect answers after the target is exposed and left uninspected. Compared with keyword search, read-first has 27.4% more recorded evidence-search calls. Target inspection attempts occur on 199 questions under read-first and 191 under keyword search; both conditions achieve 24.6% accuracy. The checkpoint protocol makes these question-level dif- ferences explicit, distinguishing target exposure and inspection from aggregate accuracy and total tool use.

Primary area:
datasets and benchmarks
Keywords:
scientific search agents, agent evaluation, decision checkpoints, retrieval, failure attribution, evidence inspection

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_UzisOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a decision-checkpoint protocol for auditing scientific-search agents. It records search observations and executed inspection attempts during inference, then joins benchmark target identities and correctness labels afterward. The resulting categories distinguish incorrect answers without target exposure, incorrect answers after exposure without inspection, and incorrect answers after an inspection attempt.

Experiments cover five search conditions on 540 answerable AutoResearchBench Deep questions in a shared, target-enriched retrieval environment. Keyword search achieves 24.6% accuracy versus 17.8% for raw search. A read-first intervention increases evidence-search calls by 27.4% but leaves aggregate accuracy unchanged, with 60 correctness gains and 60 losses relative to keyword search.

Critical strengths

1.The distinction between target exposure, inspection attempts, and final correctness is useful. The paper explicitly avoids treating a logged inspection call as proof of successful evidence acquisition or a reasoning failure. This makes the event definitions interpretable.

2.Question-paired accounting reveals changes hidden by aggregate accuracy. The separation between overall tool calls and target-specific inspection attempts is particularly useful for understanding why additional tool activity need not improve performance.

Critical weaknesses

1.The principal empirical comparisons lack an adequately reproducible inference configuration. Section 4 states that the underlying model snapshots and effective reasoning settings are incompletely recorded; the gateway alias does not establish the model identity. Section 7 also reports no same-condition replicate runs. Therefore, the observed differences cannot be established as stable differences between policies under a verified common model configuration. Question and target bootstraps quantify variation in saved outcomes, but cannot resolve execution stochasticity or model drift. This is a central evidential gap that would require substantial additional experimentation.

2.The added diagnostic value of the categories is insufficiently demonstrated. Historical exposure does not establish that the target remains available in the current context, and 167 of the 277 keyword exposed-but-uninspected failures terminate at a context or turn limit. Inspection attempts likewise do not establish that the returned excerpts satisfy the question. The exploratory semantic review covers only 26 retained excerpts from ten selected records and does not provide independent human validation. Together with the target-enriched retrieval environment, these limitations leave the contribution primarily as descriptive trajectory accounting, without convincing evidence that it reliably identifies actionable failure sources.

Questions for authors

1.Can the exact upstream model identities and effective reasoning/generation settings for the primary runs be recovered? What evidence establishes that these remained constant across conditions?

2.Are repeated executions under a fully specified configuration available? Their results would materially affect whether the reported policy differences can be considered robust.

3.Can the 60 gains and 60 losses between keyword and read-first be cross-tabulated with changes in target inspection status and termination reason? This analysis would help establish whether the checkpoint measurements explain the changed outcomes beyond reporting separate marginal counts.

Reviewer_6bVnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a decision-checkpoint auditing framework for scientific literature search agents that distinguishes target-paper exposure in search results, inspection attempts directed at the target, and the correctness of the final answer. The authors compare five search conditions on 540 answerable questions from AutoResearchBench Deep within a shared, target-enriched retrieval environment. Through trajectory analysis and per-question paired comparisons, the paper demonstrates that overall accuracy and total tool-call counts can obscure differences in search behavior and answer outcomes.

Critical strengths
  1. Distinguishing target exposure, inspection attempts, and answer correctness provides a clear and practical perspective for analyzing search failures beyond final accuracy.
  2. Per-question paired comparisons provide additional insights, and the results are interpreted with appropriate caution.
Critical weaknesses
  1. Limited novelty. The framework primarily categorizes observable events, and its key findings are relatively intuitive. The prioritized-reading strategy is also a straightforward inspection-first heuristic. The paper does not yet demonstrate a substantial conceptual or methodological advance, nor does it develop these observations into deeper, broadly generalizable insights.
  2. The experiments lack repeated runs. As a result, the stability of the observed differences between strategies, particularly the consistency of per-question answer improvements and regressions, remains unclear.
  3. The paper is poorly written and difficult to follow.
Questions for authors

Null

Reviewer_ekqFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes using checkpoints to audit agents in AutoResearchBench across the scientific search workflow. They run the gpt-5.6-luna on the 540 answerable AutoResearchBench Deep questions comparing keyword search and raw search. The authors compare between keyword search and raw search and explore their failure cases.

Critical strengths
  • the idea of auditing agents at different steps is important.
Critical weaknesses
  • the paper is very difficult to understand. The "DECISION-CHECKPOINT AUDITING" section is unreadable and incoherent. For example, the sub section "Checkpoints (execution time, no ground truth)." contains no clear explanation, motivation and illustration of how it relates to what the authors are trying to propose. Each sub-section seems like an item in a glossary explaining different terms without any coherence.

  • Figure 1 is also unreadable, it looks like a set of block text with no clear explanation of what they represent and how they relate to ecah other. For example what is "Record without labels; attribute only after inference" - that is not explained. And what is "Different failure paths, identical terminal score" with 0 accuracies - this is poorly written.

  • "to benchmark target IDs after inference" what is target IDs thats not explained anywhere. "how do target exposure, target inspection attempts and accepted answers differ between search conditions?" what is exposure and inspection? that hasn't been defined anywhere.

  • The whole paper is poorly written and in fact this idea of auditing the agent is at different parts of the trajectories is not uncommon and has been explored in many existing works like RAGChecker [1]. The error analysis in AutoResearchBench is also quite exhaustive and it is not clear what insights the audits here add.

  • The related work section is weak in the sense that the authors haven't positioned their works against existing methods in terms what is the limitation of existing methods and what this works adds in comparison.

[1] https://arxiv.org/abs/2408.08067

Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.