- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TRACE addresses credit assignment in long-horizon search agents by using a frozen reference model to measure changes in the predictability of the gold answer after each tool interaction. It converts these changes into turn-level progress rewards through a log-ratio of remaining negative-log-likelihood gaps, propagates credit over a local multi-step window, and combines it with terminal outcome supervision. The method requires no separately trained critic or process reward model. Experiments across multiple models and benchmarks evaluate training performance and transfer across retrieval environments, with ablations on the reward formulation, propagation window, and reference checkpoint.
Critical strengths
-
The method is intuitive and well motivated. Measuring whether a tool interaction makes the correct answer easier to predict gives a useful candidate signal for search progress. Local multi-step propagation accounts for delayed benefits, while terminal supervision keeps training tied to final correctness. The recipe fits into an existing policy optimization pipeline without requiring a separately trained critic or process reward model.
-
The experiments show improvements across multiple models and benchmarks under controlled training conditions. Gains also transfer from offline training to open-web evaluation. The ablations provide support for some design choices, including the credit transformation and propagation window, adding to the evidence for the recipe's practical value.
Critical weaknesses
- The contribution relative to IGPO [1] is not sufficiently established. IGPO already uses changes in the gold answer's mean log-probability to provide turn-level rewards and combines them with outcome supervision. TRACE changes the scoring model, reward transformation, and credit propagation, but does not directly discuss or compare the complete methods under the same setting. The raw-delta ablation retains other parts of TRACE's recipe and is not a complete IGPO comparison. These omissions leave the size of the methodological advance unclear.
[1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents, ICLR 2026
- The contribution of likelihood-based progress is not fully separated from terminal-outcome reweighting. Equations 9 and 10 add both progress credit and extra terminal-outcome fill to the tool-turn advantages. Setting the turn-reward weight to zero removes both. The credit-form ablations show that the transformation matters, but the paper does not clearly report a control that retains the same terminal fill while removing only the progress term. In particular, Figure 5(c)'s baseline described as being without reference-model progress scoring does not specify its terminal-fill setting. Clarifying this comparison would show what likelihood-based progress adds beyond reweighting outcome supervision.
Questions for authors
-
Could you directly compare TRACE with IGPO and clarify which changes constitute the main technical contribution? How does TRACE compare with the complete IGPO recipe under the same search-training conditions?
-
Does the progress-disabled baseline in Figure 5(c) retain the same terminal-outcome fill and other loss settings as TRACE? Please clarify the result of removing only the likelihood-progress term while keeping those settings fixed.