acceptodds
Under review as a conference paper at ICLR 2027

To Continue or to Commit: Evidence-Answer Tracing of Agent Trajectories

Abstract

Deep-search agents address complex search tasks in an iterative search-and-reasoning process until the evidence is judged sufficient and an answer is committed. Evaluating an agent's search-and-reasoning trajectory is important for understanding how it arrives at a final answer, but it is also challenging because the correctness of the outcome depends not only on what the agent does, but also on what information it has acquired and when it acquires it. Existing process signals mostly focus on evaluating the action of an agent, while the acquired information of the agent is largely ignored. This confounds what an action contributes with what the agent already knew. To bridge this gap, we propose Evidence–Answer Tracing (EAT), which, to our knowledge, is the first attempt to integrate an agent's action with its information state along the trajectory by borrowing the resource-rational theory. Specifically, two events of a target trajectory are determined by comparing against the ground-truth trajectory: evidence exposure, whether the gold documents have surfaced, and answer surfacing, whether the gold answer has appeared. This principled framework re-evaluates existing deep-search agents on a fair and unified ground. On 25 state-of-the-art systems on the BrowseComp-Plus, we find that the outcome is largely decided before final commitment. Over 99% of the autonomous-commitment accuracy gap between the strongest and weakest systems lies in which evidence–answer state they reach before committing, and under 1% in within-state conversion, that is, in how well a system answers once that state is reached. The systems that make this progress behave differently along the way: stronger systems redirect when evidence is absent and favor reading over re-searching once partial evidence is available. Once agent and question are held fixed, effort counts reverse sign and LLM-judged behaviors lose much of their association with success, whereas evidence-grounded signals such as the EAT score keep theirs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, deep search agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_UXPoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Evidence–Answer Tracing EAT, an offline framework that characterizes search-agent trajectories using gold-document exposure, gold-answer surfacing, and remaining interaction budget. Across 5927 trajectories from 25 systems on BrowseComp-Plus, the authors analyze performance gaps, action–outcome associations, and existing process metrics.

Critical strengths
  1. Tracking both evidence exposure and answer surfacing provides a useful perspective beyond simply counting interactions or evaluating agent behaviors.
  2. The analysis uses cross-validation and controls for differences between questions and agents, making the results more reliable.
  3. The authors also conduct several useful robustness checks, including excluding runs without final answers, testing the effect of the remaining budget, and comparing methods under the same supervision settings.
Critical weaknesses
  1. The TOP5–WEAK within-state conversion component in Table 7 is only −0.04 pp, but this near-zero aggregate does not establish similar within-state performance, positive and negative differences can cancel.
  2. The paper uses the presence of the gold-answer string to assess agents’ information states, but different systems may express the same answer differently. If the matching rule misses more valid answer variants for some systems, differences in wording may be mistaken for differences in information state, potentially affecting the paper’s conclusions.
  3. It remains unclear how much EAT’s ability to predict final correctness comes from knowing which actions the agent takes vs. what information it has already acquired.
  4. The observational results show that longer searching is associated with lower accuracy but do not establish that continued actions directly cause performance degradation.
Questions for authors
  1. Are the main decomposition results robust to variant-aware answer matching and different support thresholds?
  2. How much additional predictive value does action information provide beyond the evidence–answer state alone?
  3. Table 6 implies 97 correct E2A0 commitments, but App E.4 reports 131 such cases. Could the authors clarify this discrepancy
2 comments
AuthorsOpenReview ↗

Response to Reviewer UXPo (1/2)

Thank you for your careful review and helpful suggestions. We address your concerns below.

W1. State attainment and within-state conversion

Our main finding concerns the decomposition of the overall accuracy gap. To examine the cancellation you raised, we quantified the conversion contribution from each state:

StateTOP5–WEAK conversion contribution (pp)
E0A0E_0A_0−0.68
E0A1E_0A_1−0.65
E1A0E_1A_0+0.12
E1A1E_1A_1+2.50
E2A0E_2A_0+0.55
E2A1E_2A_1−1.89
Total−0.04

The total uses unrounded contributions. Their absolute values sum to 6.39 pp, or 9.7% of the overall gap. Table 6 reports each state’s accuracy and sample count; the conditional-accuracy comparison at E2A1E_2A_1 in §4.1 refers specifically to TOP5 and MID.

We also checked how the attainment share changes under alternative reference weights. On the same autonomous commitments, we used WEAK’s state shares and then TOP5’s for the conversion term, pairing each with the other group’s within-state accuracies for the attainment term. Both decompositions sum to the same observed accuracy gap (Appendix C.1, Table 7).

Reference shares for conversionWeight on E2A1E_2A_1Attainment share of the TOP5–WEAK gap
Average of TOP5 and WEAK (Eq. 2)26.66%≈100%\approx 100\%
WEAK0.26%102.2%
TOP553.06%97.9%

In a direct sensitivity check, setting WEAK’s E2A1E_2A_1 correctness to 5/65/6 instead of 6/66/6 leaves a state-attainment share of 93.3%. These checks support the same main finding: state attainment remains the dominant component of the observed gap.

W2 and Q1. Answer variants and minimum support

We recomputed Eq. (2) using the archived variant-aware labels from Appendix E.4, keeping the autonomous-commitment cohort, evidence states, and final grades fixed. State attainment remains the dominant component in all three tier comparisons.

Tier comparisonSubmitted labels (Table 7)Archived E.4 variant labels
TOP5–WEAK≈100%\approx 100\%95.9%
TOP5–MID96.6%93.6%
MID–WEAK93.8%97.5%

The main finding holds under both label sets. We have revised the abstract to state this conclusion more clearly and precisely.

For the support-threshold question, Eq. (2) uses all autonomous commitments. The minimum training-cell support threshold applies to the decision estimates in Eq. (4). We tested thresholds of 5, 8, 10, 12, 15, and 20, fitting and evaluating on the same autonomous-commitment cohort. Under both label sets, SEARCH − COMMIT is positive at E0A0E_0A_0 and negative at E2A1E_2A_1 at every threshold tested.

Together, these results support the dominance of state attainment under both label sets and the stability of the endpoint action-contrast directions across the tested support thresholds.

AuthorsOpenReview ↗

Response to Reviewer UXPo (2/2)

W3 and Q2. Additional predictive value of action information

To isolate the contribution of actions, we compared V(s,a,r,m)V(s,a,r,m) with V(s,r,m)V(s,r,m), which pools over SEARCH, VISIT, and COMMIT. Both estimators use the same evidence–answer state, system, remaining-turn interval, question folds, and outcome supervision, and are evaluated on the same held-out decisions.

At the decision level, action conditioning lowers overall Brier error from .10192 to .10025, with the largest reduction for COMMIT. The action-specific rows use the same out-of-fold predictions as the overall row.

Question-balanced decision-level Brier error (submitted state labels).

Held-out decisionsCountActions pooledAction-conditionedPaired Brier reduction [95% CI]
Overall50,878.10192.10025.00167 [.00097, .00239]
SEARCH43,363.12313.12378−.00065 [−.00123, −.00004]
VISIT2,636.12021.11493.00528 [−.00057, .01079]
COMMIT4,879.04979.04235.00744 [.00580, .00906]

At the trajectory level, adding action information lowers Brier error from .07441 to .06875. This comparison uses the E.4 labels and the same 5,624 trajectories and nested question folds as the predictive comparison.

Trajectory-level Brier error (E.4 labels, nested question folds).

Common held-out trajectoriesActions pooledAction-conditionedPaired Brier reduction [95% CI]
5,624.07441.06875.00566 [.00462, .00673]

The overall decision-level and trajectory-level results support additional predictive value from action information beyond the evidence–answer state and the other shared covariates.

W4. Evaluating when agents continue or stop

We agree that the result concerns an association with final correctness. Eq. (4) compares conditional success rates for logged SEARCH and COMMIT decisions within the same system, evidence–answer state, and remaining-turn interval. We cross-fit by question, require training-fold support for both actions, and average within trajectories before averaging across them.

SEARCH − COMMIT (pp)E0A0E_0A_0E2A1E_2A_1
Main comparison (Figure 3a, Table 9)+8.7−7.4
TOP5, equal system weights (Appendix D.4)+29.1−7.1

The change in direction also appears within individual systems and across the support thresholds in Appendix D.4. When both fitting and evaluation are restricted to autonomous commitments, the E2A1E_2A_1 contrast remains negative in every supported remaining-turn interval (Table 10).

For process evaluation, the useful finding is that the same action has different associations with final correctness depending on the evidence–answer state. This supports evaluating actions in the context of the information available to the agent.

Q3. Count of correct E2A0E_2A_0 commitments

Thank you for checking this count. The correct count is 97, consistent with Table 6. We have corrected the count in Appendix E.4.

In the revised manuscript, changes and additions are marked in blue, and the existing definition of decision value (§2.3) is highlighted in yellow.

Thank you again for your helpful comments. We would be happy to discuss any further questions.

Reviewer_5DGLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence–Answer Tracing (EAT), an offline framework for analyzing deep-search agent trajectories by combining gold-document exposure and gold-answer surfacing into a joint state representation alongside agent actions, identity, and remaining interaction budget. The authors evaluate EAT on 5,927 trajectories from 25 systems across 242 BrowseComp-Plus questions. Concretely, the analysis examines where cross-system accuracy gaps arise, how actions taken from different information states correlate with final success, and whether existing process evaluation signals remain informative after controlling for question and agent fixed effects.

Critical strengths

This paper studies a practically relevant problem in agent process evaluation that incorporating acquired information states helps disentangle evidence acquisition from final-answer generation, capturing differences often overlooked by pure search-effort or behavioral metrics. The empirical analysis is well structured, leveraging a unified retrieval environment, question-level cross-fitting, and question/agent fixed effects to ensure fair comparisons. In addition, the supplementary analyses on mechanical turn-cap failures and matched-supervision comparisons with evidence recall usefully address important alternative explanations.

Critical weaknesses
  1. Unverified robustness of answer surfacing (A_t) to formatting variations: Answer surfacing is a core component distinguishing EAT from pure evidence recall, yet Appendix E.4 reveals severe sensitivity to surface formatting: among correct autonomous commitments, the default matcher identifies A_1 in 91.4%, 73.9%, and 37.5% of cases for Top5, Mid, and Weak tiers, respectively. Although accounting for name, date, number, and multi-part variations raises overall detection from 78.8% to 97.2%, all main analyses still rely on the unadjusted normalization procedure. Because A_t directly determines terminal-state distributions, capability-gap decompositions, and state-conditioned action contrasts, it remains unclear to what extent the reported cross-tier state differences reflect genuine information acquisition rather than matching artifacts.

  2. Cancellation effects and sparse-state sensitivity in the accuracy-gap decomposition: Because the within-state conversion term in Eq. (2) is a signed weighted sum across states, the near-zero aggregate contribution in Table 7 masks substantial state-level divergences that offset one another. Specifically, Table 6 shows a large gap in E_1 A_1 (Top5 at 91.7% vs. Weak at 65.2%), whereas Weak has only six commitments in E_2 A_1 (all correct) that pull the sum in the opposite direction while receiving non-negligible weight from Top5's large terminal share. While the authors explore alternative reference weights and commitment-time bins, these checks neither quantify cross-state cancellation nor account for small-sample uncertainty in E_2 A_1, weakening the quantitative claim that over 99% of the accuracy gap is explained by pre-commitment state attainment.

  3. Unverified incremental value of action-conditioned scoring: While Appendix E.3 (equal-supervision comparison with gold-document recall) and Appendix D.3 (remaining budget analysis) show that EAT captures useful predictive signals, they do not isolate the contribution of the action variable (Search, Visit, Commit) itself. In the current comparisons, EAT differs from evidence recall not only by conditioning on actions, but also by incorporating answer-surfacing states and additional covariates. Without comparing the full EAT estimator against a state-only baseline P(C=1∣E,A,R,m)P(C=1 \mid E, A, R, m) under identical cross-fitting folds and outcome supervision, it is unclear whether conditioning on actions provides meaningful gains beyond a richer state representation.

Questions for authors

see above weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5DGL (1/2)

Thank you for your careful review and helpful suggestions. We respond to each point below.

W1. Robustness to answer variants

We followed your suggestion and recomputed the decomposition, action contrasts, and recall comparison using the archived variant-aware labels that reproduce the 97.2% detection rate in Appendix E.4. We kept the autonomous-commitment cohort, evidence states, and final grades fixed. State attainment remains the dominant component in all three tier comparisons.

State attainment as a share of the accuracy gap (Eq. 2).

Tier comparisonSubmitted labels (Table 7)Archived E.4 variant labels
TOP5–WEAK≈100%\approx 100\%95.9%
TOP5–MID96.6%93.6%
MID–WEAK93.8%97.5%

In the three-factor decomposition in Table 2, evidence exposure and answer surfacing together account for 99.2% of the TOP5–WEAK gap with the submitted labels and 96.3% with the E.4 variant labels. The main finding holds under both label sets. We have revised the abstract to state this conclusion more clearly and precisely.

With the E.4 labels, the share of commitments from E2A1E_2A_1 rises from 53.1% to 57.5% for TOP5, from 11.3% to 13.1% for MID, and from 0.3% to 0.8% for WEAK. We also refit and evaluated the Table 9 estimator on the same 5,672 autonomous commitments under both label sets. All four contrasts below retain their directions.

StateAction contrastSubmitted labels (pp)E.4 variant labels (pp)
E0A0E_0A_0SEARCH − COMMIT+10.0+12.5
E1A1E_1A_1SEARCH − COMMIT−5.1−5.7
E1A1E_1A_1VISIT − COMMIT+3.3+4.8
E2A1E_2A_1SEARCH − COMMIT−4.1−7.9

For the comparison with recall, we retained the 5,624 autonomous commitments and prediction model used in Appendix E.3 and used nested question-level cross-fitting. The two runs differ only in their answer labels. Adding EAT to recall lowers held-out Brier error under both label sets.

Brier error on the 5,624 Appendix E.3 trajectories (nested question folds).

Answer labels for EATRecall onlyEAT onlyRecall + EAT
Submitted state labels.07530.07394.06269
Archived E.4 variant labels.07530.06875.05821

Taken together, these results support our main findings under both answer-label sets: state attainment remains the dominant component of the cross-tier accuracy gaps, the four action contrasts retain their directions, and EAT continues to provide predictive information beyond document recall.

AuthorsOpenReview ↗

Response to Reviewer 5DGL (2/2)

W2. Robustness of the main decomposition finding

To assess the influence of the six WEAK commitments in E2A1E_2A_1, we considered a sensitivity case with 5/65/6 correctness instead of 6/66/6. State attainment still accounts for 93.3% of the TOP5–WEAK accuracy gap in this case.

We also examined how much the result depends on the weight assigned to this cell. Using the submitted labels and the same commitments, we recomputed the decomposition with WEAK’s state shares and then TOP5’s as the reference weights for conversion (Appendix C.1). In each case, attainment and conversion sum to the same observed accuracy gap.

For reference group gg and the other group hh, the two terms are:

Conversion=∑sπg(s)[aTOP5(s)−aWEAK(s)],\text{Conversion}=\sum_s \pi_g(s)\left[a_{\mathrm{TOP5}}(s)-a_{\mathrm{WEAK}}(s)\right],

Attainment=∑s[πTOP5(s)−πWEAK(s)]ah(s).\text{Attainment}=\sum_s\left[\pi_{\mathrm{TOP5}}(s)-\pi_{\mathrm{WEAK}}(s)\right]a_h(s).

Here, πg(s)\pi_g(s) is group gg’s commitment share at state ss, and ag(s)a_g(s) is its accuracy at that state.

Reference shares for conversionWeight on E2A1E_2A_1Attainment share of the TOP5–WEAK gap
Average of TOP5 and WEAK (Eq. 2)26.66%≈100%\approx 100\%
WEAK0.26%102.2%
TOP553.06%97.9%

We also quantified the cancellation you raised: the signed conversion contributions sum to −0.04 pp, while their absolute values sum to 6.39 pp, or 9.7% of the overall gap. Taken together, the reference-weight comparisons and the 5/65/6 sensitivity check support the same conclusion: state attainment remains the main component of the observed gap.

W3. Additional predictive value of action information

To isolate the contribution of actions, we compared V(s,a,r,m)V(s,a,r,m) with V(s,r,m)V(s,r,m), which pools over SEARCH, VISIT, and COMMIT. Both estimators use the same evidence–answer state, system, remaining-turn interval, question folds, and outcome supervision, and are evaluated on the same held-out decisions.

At the decision level, action conditioning lowers overall Brier error from .10192 to .10025, with the largest reduction for COMMIT. The action-specific rows below use the same out-of-fold predictions as the overall row.

Question-balanced decision-level Brier error (submitted state labels).

Held-out decisionsCountActions pooledAction-conditionedPaired Brier reduction [95% CI]
Overall50,878.10192.10025.00167 [.00097, .00239]
SEARCH43,363.12313.12378−.00065 [−.00123, −.00004]
VISIT2,636.12021.11493.00528 [−.00057, .01079]
COMMIT4,879.04979.04235.00744 [.00580, .00906]

At the trajectory level, adding action information lowers Brier error from .07441 to .06875. This comparison uses the E.4 labels and the same 5,624 trajectories and nested question folds as the predictive comparison above.

Trajectory-level Brier error (E.4 labels, nested question folds).

Common held-out trajectoriesActions pooledAction-conditionedPaired Brier reduction [95% CI]
5,624.07441.06875.00566 [.00462, .00673]

These matched comparisons support the additional predictive value of action information beyond the evidence–answer state and the other shared covariates.

In the revised manuscript, changes and additions are marked in blue. The existing state definitions (§2.1) and accuracy formula (§2.2) are highlighted in yellow.

Thank you again for these helpful suggestions. We would be happy to discuss any further questions.

Thank you for the detailed rebuttal and additional experiments. The new robustness analyses and ablation results have substantially addressed my main concerns. Although some uncertainty remains regarding the precise decomposition percentages, I find the overall response convincing. I have therefore raised my rating from 2 to 3.

Thank you for taking another look at the results! We’re glad the additional experiments helped address your main concerns. You can find the robustness checks and ablations from our discussion in Sections F.1–F.4 (pp. 21–22) of the revised manuscript. We’ve marked them in blue, along with the related wording.

Reviewer_xSb1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Evidence–Answer Tracing (EAT), a framework for analyzing the trajectories of deep-search agents. It asks whether success or failure is decided at the final step or earlier, depending on whether the agent finds the right evidence. At each step, EAT records whether none, some, or all of the gold documents have appeared in the agent's search results or opened pages, and whether the gold answer has appeared in its context. Each decision (search, open a document, or answer) is then evaluated against what the agent had seen at that point. Based on the findings, the authors recommend that process signals be reported under question and agent controls, alongside an evidence-based baseline, rather than relying on effort counts or LLM-judged behaviors alone.

Critical strengths
  • The audit of existing process signals is a useful contribution. It shows that many existing metrics look predictive because stronger agents tend to score higher on them. Once the agent is fixed, their predictive value drops: search count even reverses sign. These metrics are thus unreliable for comparing runs from the same agent.

  • Using a subset of BrowseComp-Plus, the authors report several interesting findings: (a) strong and weak agents differ mainly in whether they reach the right evidence; once the gold documents and answer have surfaced, both answer correctly at similar rates; (b) when stuck, weak agents tend to repeat their queries and rarely recover, whereas strong agents change direction and recover more successfully; and (c) in an audit of existing process signals, effort counts and LLM-judged behaviors lose much of their predictive value once the agent is held fixed, while gold-document recall and the EAT score keep theirs.

  • The experiments are broad and fairly controlled, evaluated under same retriever, turn limit and cross-fitted, making the experiments sound.

Critical weaknesses

My main concern is that the practical value of EAT is neither well demonstrated nor clearly explained

  • The first major finding (lines 081-085) essentially says that once the gold evidence has been found and the gold answer appears in the context, both agents answer correctly, so the difference between strong and weak agents is in whether they find the right evidence. However, this seems like a trivial finding i.e., agent answers correctly if it finds the right answer. It is unclear what the significance of this result is and why this is presented as the lead finding.

  • Simple gold-document recall is comparable to Q_EAT. It is unclear the practical cases where Q_EAT provides information that recall alone does not.

  • Q_EAT currently requires gold documents/answers and it is still compared against other process signals. It isn't clear if this is meant to be presented as an alternative process metric or only as a reference baseline.

A second concern is that the paper is difficult to follow overall:

  • Sometimes the terms are used before they are defined. For example, introduction refers to "65.6-point autonomous-commitment accuracy gap" but "autonomous-commitment accuracy" is not defined there.

  • More generally, the writing is dense and the sections do not clearly explain why each analysis is done and explain the significance of findings.

Questions for authors

Could you please respond to my concerns in the weaknesses section?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xSb1 (1/2)

Thank you for your careful review. We provide the following responses to help you and other readers better understand our work.

W1. Why the first finding matters

We agree that agents are more likely to answer correctly when the answer has surfaced; this observation alone is not our contribution. Our key finding is a quantitative decomposition of where the accuracy gap between strong and weak agents arises, distinguishing differences in reaching evidence–answer states (state attainment) from differences in correctness conditional on those states (within-state conversion).

Across 5,927 trajectories from 25 systems evaluated with a common corpus and retriever, state attainment accounts for approximately 100% of the 65.6-percentage-point autonomous-commitment accuracy gap between TOP5 and WEAK under the submitted labels (Table 7), and 95.9% under the variant-aware labels. Specifically, 53.1% of TOP5’s autonomous commitments occur in E2A1E_2A_1, compared with only 0.3% for WEAK (Table 1). Among TOP5 and MID systems, correctness conditional on E2A1E_2A_1 is similar: 92.9% and 92.0%.

The significance is not that finding an answer improves accuracy, but that the decomposition identifies where the observed performance differences arise. It provides a diagnostic basis for prioritizing improvements: limited evidence exposure points toward query formulation, retrieval, and ranking; exposed evidence without a surfaced answer points toward document reading and answer synthesis; and incorrect commitments after answer surfacing point toward verification and final-answer decisions. In our data, 216 of the 1,350 commitments after answer surfacing were incorrect (Appendix E.4).

Prior results illustrate why these distinctions matter. On BrowseComp-Plus, Chen et al. (2026b, Table 1) report that replacing BM25 with Qwen3-Embedding-8B raises GPT-5 accuracy from 55.90% to 70.12%. Meng et al. (2026, Table 8) report 57.2% to 68.9% for gpt-oss-20b after adding monoT5 reranking to the same passage-level BM25 retriever. Sharifymoghaddam and Lin (2026) also find moderate reranking can be a cost-effective use of inference compute. Our comparison uses a shared retriever, so the decomposition helps identify which stage to examine even when the retriever is held fixed.

This is our lead finding because it establishes the diagnostic framework for subsequent analyses. Section 4.2 examines how the success rates associated with SEARCH, VISIT, and COMMIT vary across these states, while Section 4.3 evaluates whether existing process signals capture these distinctions. Together, the analyses move from locating performance gaps to examining decisions within states and assessing whether existing metrics capture these differences.

W2. What EAT adds beyond gold-document recall

A concrete example may help clarify this point. Full recall tells us that all gold documents reached the agent. EAT also records whether the gold answer surfaced before commitment. The following two trajectories on BrowseComp-Plus question 679 have the same document recall but different answer-surfacing states:

SystemGold-document recallState at commitmentFinal answer
GLM-5.11.0E2A1E_2A_1: answer surfacedCorrect
Kimi-K2.7-Code1.0E2A0E_2A_0: answer had not surfacedIncorrect

The terminal state of each trajectory is unchanged across the two answer-label sets. With recall at 100% in both runs, EAT shows that the answer surfaced before commitment in only one of them.

We also tested complementarity quantitatively. In the nested-fold comparison on the same 5,624 held-out trajectories as Appendix E.3, adding EAT with the E.4 answer labels to gold-document recall lowers Brier error from .07530 to .05821. This supports a complementary role for EAT when document coverage is already known.

AuthorsOpenReview ↗

Response to Reviewer xSb1 (2/2)

W3. Why we audit process signals and how we use QEATQ_{\mathrm{EAT}}

We use QEATQ_{\mathrm{EAT}} as an offline diagnostic and an evidence-grounded reference. Gold documents and answers anchor the state annotations used in the audit. We appreciate your question about this role, which should have been clearer in the paper.

Deep-search evaluation includes interaction counts, judged behaviors, reasoning competencies, step-quality labels, and evidence coverage. These signals describe different aspects of a trajectory. Comparing results from different systems or questions makes it difficult to tell which signals distinguish successful runs after accounting for question difficulty and persistent system differences.

Sections 2.4 and 4.3 bring these signals onto a common trajectory-level basis across 25 systems. We examine what each signal retains after question and system controls, whether it ranks correct runs above incorrect runs on the same question, and what it adds when entered jointly with QEATQ_{\mathrm{EAT}}. The held-out comparison then tests predictive complementarity under matched outcome supervision. The search-count association reverses after adding system controls, while recall and EAT both retain predictive information.

For metric designers, this provides a common protocol for testing whether a new signal distinguishes successful runs and adds information beyond evidence coverage. For agent developers, it helps connect a trajectory-level score to the stages and decisions they can inspect.

W4. Definition and organization

Thank you for pointing out that “autonomous-commitment accuracy” appears before it is explained. It is the fraction of correct runs among those ending with an agent-initiated COMMIT. Section 2 defines COMMIT, and §2.2 gives the accuracy formula. These existing passages are highlighted in yellow in the revised manuscript; changes and additions are marked in blue.

We also agree that the connection between the analyses should be easier to follow. They address three questions in sequence:

  1. Where does the accuracy gap arise? (§4.1) We compare states at commitment and correctness within each state, then decompose the gap into attainment and conversion.
  2. How does the action–outcome relationship vary by state? (§4.2) We compare SEARCH and VISIT with COMMIT within system, state, and remaining-budget interval.
  3. Which process signals distinguish success? (§4.3) We examine signals after question and system controls, compare runs on the same question, and test their complementarity with EAT.

Thank you again for helping us clarify the practical purpose of these analyses. We would be happy to discuss any further questions.

The responses answer my questions and adds clarity. I have raised the score accordingly. Thank you for the clarifications.

Thanks for getting back to us! We’re glad our replies answered your questions. Your question about why we lead with the accuracy gap helped us clarify how the three findings fit together, and we’ve also made the paper easier to follow overall. In the revised manuscript, we’ve marked the updated wording and the follow-up analyses in Sections F.1 and F.3 (pp. 21–22) in blue for easy reference.

Reviewer_SdCBOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Evidence–Answer Tracing (EAT) maps logged search-agent decisions to six states defined by gold-document exposure and literal gold-answer surfacing. It estimates cross-fitted conditional success rates by system, state, action, and remaining-position interval, decomposes accuracy gaps into state attainment and within-state conversion, and audits existing process signals with question and agent controls. The study covers 5,927 trajectories from 25 systems on 242 BrowseComp-Plus questions. It finds that evidence-grounded signals remain informative after additive controls, while effort and several judged behaviors lose predictive association.

Critical strengths

Across the common cohort, question/agent controls expose how effort and several judged behaviors inherit associations from agent composition, while evidence-grounded signals remain informative. Question-level cross-fitting and the equal-supervision comparison in E.3 go beyond a simple correlation analysis. In the latter, the joint recall/EAT model has Brier error .0585 versus .0753 for recall alone and .0694 for EAT alone, and the highest reported same-question concordance. This supports complementary predictive information on the studied cohort, not a causal account of which action to take.

Critical weaknesses

Literal answer matching detects 91.4% of graded-correct TOP5 commitments, 73.9% for MID, and 37.5% for WEAK (E.4). The available variant-aware normalization raises overall agreement to 97.2%, but the main decomposition and audit are not recomputed with it. Unequal false negatives can move trajectories between states and affect both attainment and conversion. Also, including the current reasoning step in A_t makes it a valid measurement at action selection, but potentially includes a solution formed immediately before commitment; an earlier boundary would measure a different, useful sensitivity case. These issues do not establish that the main trend will reverse, and they are not evidence of leakage. They limit how strongly “before commitment” can be interpreted as excluding the contribution of final reasoning.

Questions for authors

With the variant-aware normalization already described in E.4, do the main attainment/conversion decomposition and the complementary predictive benefit of EAT with recall persist? Please separate changes to the exact “over 99%” estimate from changes to the qualitative “mainly attainment” conclusion. A loss of the latter together with the complementary diagnostic value would materially weaken my recommendation; a lower but still predominant attainment share would chiefly require narrower wording.

Comments on the paper

1 comment
Area_Chair_LuzzOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

The authors have provided the rebuttal. Please have a look and let the authors know if you need further clarification or still have follow-up questions.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.