acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Turn-Level Reward Assignment via Credit Estimation for Long-Horizon Agents

26 views

Abstract

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent, post training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c6JgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE addresses credit assignment in long-horizon search agents by using a frozen reference model to measure changes in the predictability of the gold answer after each tool interaction. It converts these changes into turn-level progress rewards through a log-ratio of remaining negative-log-likelihood gaps, propagates credit over a local multi-step window, and combines it with terminal outcome supervision. The method requires no separately trained critic or process reward model. Experiments across multiple models and benchmarks evaluate training performance and transfer across retrieval environments, with ablations on the reward formulation, propagation window, and reference checkpoint.

Critical strengths
  1. The method is intuitive and well motivated. Measuring whether a tool interaction makes the correct answer easier to predict gives a useful candidate signal for search progress. Local multi-step propagation accounts for delayed benefits, while terminal supervision keeps training tied to final correctness. The recipe fits into an existing policy optimization pipeline without requiring a separately trained critic or process reward model.

  2. The experiments show improvements across multiple models and benchmarks under controlled training conditions. Gains also transfer from offline training to open-web evaluation. The ablations provide support for some design choices, including the credit transformation and propagation window, adding to the evidence for the recipe's practical value.

Critical weaknesses
  1. The contribution relative to IGPO [1] is not sufficiently established. IGPO already uses changes in the gold answer's mean log-probability to provide turn-level rewards and combines them with outcome supervision. TRACE changes the scoring model, reward transformation, and credit propagation, but does not directly discuss or compare the complete methods under the same setting. The raw-delta ablation retains other parts of TRACE's recipe and is not a complete IGPO comparison. These omissions leave the size of the methodological advance unclear.

[1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents, ICLR 2026

  1. The contribution of likelihood-based progress is not fully separated from terminal-outcome reweighting. Equations 9 and 10 add both progress credit and extra terminal-outcome fill to the tool-turn advantages. Setting the turn-reward weight to zero removes both. The credit-form ablations show that the transformation matters, but the paper does not clearly report a control that retains the same terminal fill while removing only the progress term. In particular, Figure 5(c)'s baseline described as being without reference-model progress scoring does not specify its terminal-fill setting. Clarifying this comparison would show what likelihood-based progress adds beyond reweighting outcome supervision.
Questions for authors
  1. Could you directly compare TRACE with IGPO and clarify which changes constitute the main technical contribution? How does TRACE compare with the complete IGPO recipe under the same search-training conditions?

  2. Does the progress-disabled baseline in Figure 5(c) retain the same terminal-outcome fill and other loss settings as TRACE? Please clarify the result of removing only the likelihood-progress term while keeping those settings fixed.

Reviewer_wRvAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TRACE, a critic-free turn-level credit assignment method for RLVR training of multi-turn search agents. At each tool-call boundary, a frozen copy of the initial policy computes the average log-probability of the ground-truth answer given the trajectory prefix. TRACE converts these scores into log-ratio state values and uses their temporal-difference (TD) changes as dense turn-level rewards, which are added to the GRPO outcome advantage. Trained with pure RL, TRACE improves Qwen3-4B on BrowseComp-Plus from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. Averaged over four benchmarks, TRACE outperforms GRPO, GSPO, GiGPO.

Critical strengths
  1. The paper is easy to follow, and the method is simple to understand.

  2. It's a critic-free credit assignment method that could be easily adopted by different algorithms.

Critical weaknesses
  1. The novelty is limited. TRACE's core signal, the per-turn change in the teacher-forced log-probability of the gold answer, was already proposed by IGPO[1], which the paper cites only within a list of related work. TRACE's own changes (a frozen reference model, a log-ratio transform, and a short-window backup) are incremental, and the paper never compares against IGPO to show that they help.

  2. Insufficient baselines. The controlled comparison covers only GRPO, GSPO and GiGRPO. It omits both stronger general-purpose baselines such as DAPO[2] and the RL methods for multi-turn agents that the paper itself cites (e.g., IGPO[1], StepSearch[3], SPA-RL[4], ARPO[5]). GRPO and GSPO have no multi-turn design, and GiGRPO falls below GRPO in this setting (26.5 vs. 29.5 at 4B). Outperforming them is therefore expected and does not show that TRACE improves on existing agentic credit-assignment methods.

[1] Wang G, Dai S, Ye G, et al. Information gain-based policy optimization: A simple and effective approach for multi-turn search agents[C]//International Conference on Learning Representations. 2026, 2026: 104502-104529.

[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. Advances in Neural Information Processing Systems, 2026, 38: 113222-113244.

[3] Wang Z, Zheng X, An K, et al. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization[J]. arXiv preprint arXiv:2505.15107, 2025.

[4] Wang H, Leong C T, Wang J, et al. Spa-rl: Reinforcing llm agents via stepwise progress attribution[J]. arXiv preprint arXiv:2505.20732, 2025.

[5] Dong G, Mao H, Ma K, et al. Agentic reinforced policy optimization[C]//International Conference on Learning Representations. 2026, 2026: 16981-17017.

Questions for authors
  1. Can you compare TRACE with IGPO [1] at the method level and clarify which of its components (the frozen reference, the log-ratio transform, the short-window backup) are new and how much each contributes?

  2. Can you add DAPO [2] and the agentic RL methods IGPO [1], StepSearch [3], SPA-RL [4] and ARPO [5] as baselines under the same controlled setup?

  3. Can you report wall-clock time and GPU-hours per training step for TRACE and GRPO, including reference-model scoring, and compare the two at matched compute?

  4. Please report the number of questions per benchmark and avg@k (e.g., k = 4, 8, 16) with standard deviations, since one evaluation sample per question (Table 3) is noisy on small benchmarks, especially for such long-horizon tasks.

[1] Wang G, Dai S, Ye G, et al. Information gain-based policy optimization: A simple and effective approach for multi-turn search agents[C]//International Conference on Learning Representations. 2026, 2026: 104502-104529.

[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. Advances in Neural Information Processing Systems, 2026, 38: 113222-113244.

[3] Wang Z, Zheng X, An K, et al. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization[J]. arXiv preprint arXiv:2505.15107, 2025.

[4] Wang H, Leong C T, Wang J, et al. Spa-rl: Reinforcing llm agents via stepwise progress attribution[J]. arXiv preprint arXiv:2505.20732, 2025.

[5] Dong G, Mao H, Ma K, et al. Agentic reinforced policy optimization[C]//International Conference on Learning Representations. 2026, 2026: 16981-17017.

Reviewer_63DbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE, which provides turn-level rewards for training long-horizon search agents. The key idea is to use a frozen model to scores how likely the correct answer is after each tool call, and score changes are transformed to rewards. Such rewards are considered together with the final answer correctness rewards. The authors conduct experiments on 2 model backbones and 4 benchmark datasets, showing consistent improvements over baselines including GRPO, GSPO, and GiGRPO.

Critical strengths
  • The key idea avoids training a separate progress scorer or credit assignment model.
  • Consistent performance improvements are observed. Besides, the authors conduct ablation studies to demonstrate the contribution of reward transformation, look-ahead horizon, reference checkpoint, etc.
Critical weaknesses
  • Novelty and performance improvement over more relevant prior work are unclear. Note that IGPO already uses changes in the probability of generating the final answer to construct turn-level rewards (and combines them with outcome rewards). I understand that TRACE makes multiple changes to IGPO. However, IGPO is not compared in the experiments. Technical significance of the proposed changes is not discussed in detail either.
  • Ablation studies have not fully isolated the contribution of dense credit from extra outcome weighting. Eq. (9) adds additional outcome advantage to the last few tool calls. The ablation version removes both this extra weighting and dense credit. Thus, it is unclear how much improvement comes from the proposed progress rewards (rather than just emphasizing the last few tool calls).
Questions for authors
  • Could you compare TRACE with IGPO?
  • Could you explain technical differences between TRACE and IGPO (and how significant they are)?
  • Could you add one more ablation version that fully isolates the contribution of dense credit from extra outcome weighting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.