- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Evidence–Answer Tracing EAT, an offline framework that characterizes search-agent trajectories using gold-document exposure, gold-answer surfacing, and remaining interaction budget. Across 5927 trajectories from 25 systems on BrowseComp-Plus, the authors analyze performance gaps, action–outcome associations, and existing process metrics.
Critical strengths
- Tracking both evidence exposure and answer surfacing provides a useful perspective beyond simply counting interactions or evaluating agent behaviors.
- The analysis uses cross-validation and controls for differences between questions and agents, making the results more reliable.
- The authors also conduct several useful robustness checks, including excluding runs without final answers, testing the effect of the remaining budget, and comparing methods under the same supervision settings.
Critical weaknesses
- The TOP5–WEAK within-state conversion component in Table 7 is only −0.04 pp, but this near-zero aggregate does not establish similar within-state performance, positive and negative differences can cancel.
- The paper uses the presence of the gold-answer string to assess agents’ information states, but different systems may express the same answer differently. If the matching rule misses more valid answer variants for some systems, differences in wording may be mistaken for differences in information state, potentially affecting the paper’s conclusions.
- It remains unclear how much EAT’s ability to predict final correctness comes from knowing which actions the agent takes vs. what information it has already acquired.
- The observational results show that longer searching is associated with lower accuracy but do not establish that continued actions directly cause performance degradation.
Questions for authors
- Are the main decomposition results robust to variant-aware answer matching and different support thresholds?
- How much additional predictive value does action information provide beyond the evidence–answer state alone?
- Table 6 implies 97 correct E2A0 commitments, but App E.4 reports 131 such cases. Could the authors clarify this discrepancy
2 comments
Response to Reviewer UXPo (2/2)
W3 and Q2. Additional predictive value of action information
To isolate the contribution of actions, we compared with , which pools over SEARCH, VISIT, and COMMIT. Both estimators use the same evidence–answer state, system, remaining-turn interval, question folds, and outcome supervision, and are evaluated on the same held-out decisions.
At the decision level, action conditioning lowers overall Brier error from .10192 to .10025, with the largest reduction for COMMIT. The action-specific rows use the same out-of-fold predictions as the overall row.
Question-balanced decision-level Brier error (submitted state labels).
| Held-out decisions | Count | Actions pooled | Action-conditioned | Paired Brier reduction [95% CI] |
|---|---|---|---|---|
| Overall | 50,878 | .10192 | .10025 | .00167 [.00097, .00239] |
| SEARCH | 43,363 | .12313 | .12378 | −.00065 [−.00123, −.00004] |
| VISIT | 2,636 | .12021 | .11493 | .00528 [−.00057, .01079] |
| COMMIT | 4,879 | .04979 | .04235 | .00744 [.00580, .00906] |
At the trajectory level, adding action information lowers Brier error from .07441 to .06875. This comparison uses the E.4 labels and the same 5,624 trajectories and nested question folds as the predictive comparison.
Trajectory-level Brier error (E.4 labels, nested question folds).
| Common held-out trajectories | Actions pooled | Action-conditioned | Paired Brier reduction [95% CI] |
|---|---|---|---|
| 5,624 | .07441 | .06875 | .00566 [.00462, .00673] |
The overall decision-level and trajectory-level results support additional predictive value from action information beyond the evidence–answer state and the other shared covariates.
W4. Evaluating when agents continue or stop
We agree that the result concerns an association with final correctness. Eq. (4) compares conditional success rates for logged SEARCH and COMMIT decisions within the same system, evidence–answer state, and remaining-turn interval. We cross-fit by question, require training-fold support for both actions, and average within trajectories before averaging across them.
| SEARCH − COMMIT (pp) | ||
|---|---|---|
| Main comparison (Figure 3a, Table 9) | +8.7 | −7.4 |
| TOP5, equal system weights (Appendix D.4) | +29.1 | −7.1 |
The change in direction also appears within individual systems and across the support thresholds in Appendix D.4. When both fitting and evaluation are restricted to autonomous commitments, the contrast remains negative in every supported remaining-turn interval (Table 10).
For process evaluation, the useful finding is that the same action has different associations with final correctness depending on the evidence–answer state. This supports evaluating actions in the context of the information available to the agent.
Q3. Count of correct commitments
Thank you for checking this count. The correct count is 97, consistent with Table 6. We have corrected the count in Appendix E.4.
In the revised manuscript, changes and additions are marked in blue, and the existing definition of decision value (§2.3) is highlighted in yellow.
Thank you again for your helpful comments. We would be happy to discuss any further questions.
Response to Reviewer UXPo (1/2)
Thank you for your careful review and helpful suggestions. We address your concerns below.
W1. State attainment and within-state conversion
Our main finding concerns the decomposition of the overall accuracy gap. To examine the cancellation you raised, we quantified the conversion contribution from each state:
The total uses unrounded contributions. Their absolute values sum to 6.39 pp, or 9.7% of the overall gap. Table 6 reports each state’s accuracy and sample count; the conditional-accuracy comparison at E2A1 in §4.1 refers specifically to TOP5 and MID.
We also checked how the attainment share changes under alternative reference weights. On the same autonomous commitments, we used WEAK’s state shares and then TOP5’s for the conversion term, pairing each with the other group’s within-state accuracies for the attainment term. Both decompositions sum to the same observed accuracy gap (Appendix C.1, Table 7).
In a direct sensitivity check, setting WEAK’s E2A1 correctness to 5/6 instead of 6/6 leaves a state-attainment share of 93.3%. These checks support the same main finding: state attainment remains the dominant component of the observed gap.
W2 and Q1. Answer variants and minimum support
We recomputed Eq. (2) using the archived variant-aware labels from Appendix E.4, keeping the autonomous-commitment cohort, evidence states, and final grades fixed. State attainment remains the dominant component in all three tier comparisons.
The main finding holds under both label sets. We have revised the abstract to state this conclusion more clearly and precisely.
For the support-threshold question, Eq. (2) uses all autonomous commitments. The minimum training-cell support threshold applies to the decision estimates in Eq. (4). We tested thresholds of 5, 8, 10, 12, 15, and 20, fitting and evaluating on the same autonomous-commitment cohort. Under both label sets, SEARCH − COMMIT is positive at E0A0 and negative at E2A1 at every threshold tested.
Together, these results support the dominance of state attainment under both label sets and the stability of the endpoint action-contrast directions across the tested support thresholds.