- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a decision-checkpoint protocol for auditing scientific-search agents. It records search observations and executed inspection attempts during inference, then joins benchmark target identities and correctness labels afterward. The resulting categories distinguish incorrect answers without target exposure, incorrect answers after exposure without inspection, and incorrect answers after an inspection attempt.
Experiments cover five search conditions on 540 answerable AutoResearchBench Deep questions in a shared, target-enriched retrieval environment. Keyword search achieves 24.6% accuracy versus 17.8% for raw search. A read-first intervention increases evidence-search calls by 27.4% but leaves aggregate accuracy unchanged, with 60 correctness gains and 60 losses relative to keyword search.
Critical strengths
1.The distinction between target exposure, inspection attempts, and final correctness is useful. The paper explicitly avoids treating a logged inspection call as proof of successful evidence acquisition or a reasoning failure. This makes the event definitions interpretable.
2.Question-paired accounting reveals changes hidden by aggregate accuracy. The separation between overall tool calls and target-specific inspection attempts is particularly useful for understanding why additional tool activity need not improve performance.
Critical weaknesses
1.The principal empirical comparisons lack an adequately reproducible inference configuration. Section 4 states that the underlying model snapshots and effective reasoning settings are incompletely recorded; the gateway alias does not establish the model identity. Section 7 also reports no same-condition replicate runs. Therefore, the observed differences cannot be established as stable differences between policies under a verified common model configuration. Question and target bootstraps quantify variation in saved outcomes, but cannot resolve execution stochasticity or model drift. This is a central evidential gap that would require substantial additional experimentation.
2.The added diagnostic value of the categories is insufficiently demonstrated. Historical exposure does not establish that the target remains available in the current context, and 167 of the 277 keyword exposed-but-uninspected failures terminate at a context or turn limit. Inspection attempts likewise do not establish that the returned excerpts satisfy the question. The exploratory semantic review covers only 26 retained excerpts from ten selected records and does not provide independent human validation. Together with the target-enriched retrieval environment, these limitations leave the contribution primarily as descriptive trajectory accounting, without convincing evidence that it reliably identifies actionable failure sources.
Questions for authors
1.Can the exact upstream model identities and effective reasoning/generation settings for the primary runs be recovered? What evidence establishes that these remained constant across conditions?
2.Are repeated executions under a fully specified configuration available? Their results would materially affect whether the reported policy differences can be considered robust.
3.Can the 60 gains and 60 losses between keyword and read-first be cross-tabulated with changes in target inspection status and termination reason? This analysis would help establish whether the checkpoint measurements explain the changed outcomes beyond reporting separate marginal counts.