- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACER for multi-turn search agent RL. It computes advantages separately for different reward types and combines them after format-based gating. The method is evaluated on QA benchmarks with multiple backbones.
Critical strengths
The paper addresses an important question: how different reward types should be handled when they serve different purposes in agent training.
Critical weaknesses
-
Attribution of performance gains is not fully clear: ProxMO-C is a useful control for auxiliary reward accumulation, but its comparison with TRACER changes several aspects of credit estimation together. The paper should better establish how much of the improvement comes from the proposed method beyond reward calibration.
-
The main design choices need stronger justification: TRACER combines several credit estimation and routing components, but their contributions and interactions are not sufficiently explained. The existing ablations are useful, but they do not fully establish why the proposed combination is necessary or how it improves the learning signal.
-
The empirical evidence does not fully support the scope of the claims: The failure-mode diagnostics are limited to a single run with the smallest backbone. The gains over ProxMO-C are positive, but their stability across runs is unclear. Overall, I am not fully convinced that the results establish a substantial practical benefit from the additional design complexity.
Overall, the idea is reasonable, but the supporting evidence and justification remain insufficient, and the gains over the calibrated baseline are modest. I am therefore not convinced that the contribution is substantial enough for acceptance at ICLR.
Questions for authors
-
What evidence distinguishes the benefit of the proposed credit estimation method from reward calibration and supports the main design choices beyond the complete system's performance?
-
How consistently do the diagnosed failure modes and performance gains occur across backbones and training runs?
4 comments
Follow-up on Gains Beyond Calibration, Cross-Backbone Consistency, and Practical Cost
Dear Reviewer karY,
Thank you again for your careful review and for highlighting the need to distinguish gains beyond reward calibration and better justify the additional design complexity. As the discussion period draws to a close, we wanted to respectfully follow up on our response.
Our rebuttal adds a timing-preserving calibration baseline, a cumulative comparison using common comparison pools, and a shared-support control for local process horizons. We also report consistent three-seed gains on Qwen2.5-1.5B and Gemma-3-4B-IT, failure-mode evidence on additional backbones, and measured practical costs, including a 0.9% increase in end-to-end training time in the tested 1.5B setting. These results support a more narrowly scoped claim, and we explicitly acknowledge that the cumulative increments depend on the order of component addition. We will incorporate the additional evidence and clarified limitations into the revised manuscript.
We hope this helps address your concerns about attribution, consistency, and practical benefit. If any concern remains unresolved, we would greatly appreciate the opportunity to clarify it promptly. If you find the new evidence sufficient to reconsider your assessment, we would sincerely appreciate your consideration of an upward rating revision. Thank you again for your time and thoughtful judgment.
Sincerely, The Authors
Brief Follow-up Before the Discussion Deadline
Dear Reviewer karY,
With the discussion deadline approaching, we would appreciate any brief feedback on whether our rebuttal addresses your main concerns. We understand that your time is limited and thank you for considering this follow-up.
The additional evidence directly targets the questions raised in your review:
- Gains beyond calibration: We added a timing-preserving calibration baseline and controlled cumulative comparisons, including a shared-support test for local process horizons.
- Consistency and practical benefit: Three-seed comparisons show improvements over ProxMO-C on both Qwen2.5-1.5B and Gemma-3-4B-IT, with a measured 0.9% increase in end-to-end training time in the tested 1.5B setting.
We will incorporate these results and their limitations into the revised manuscript, explicitly presenting the component analysis as cumulative and order-dependent.
If a concern still prevents a more favorable assessment, even a brief indication would help us respond before the deadline. If the additional evidence has sufficiently addressed your concerns, we would be grateful if you would consider revising your rating upward.
Thank you again for your time and thoughtful review.
Sincerely,
The Authors
I would like to clarify that I read and evaluated the paper myself. I used ChatGPT to look up related information, discuss the paper, and help organize and phrase my comments in English. The review contains substantial human input, and the concerns below reflect my own assessment of the paper.
My original concerns are as follows:
-
Attribution of performance gains. The comparison between ProxMO-C and TRACER changed several aspects of credit estimation together. It was therefore unclear how much of the improvement came from the proposed method beyond reward calibration.
-
Justification of the main design choices. TRACER combines several credit estimation and routing components. The original ablations did not sufficiently explain their individual contributions and interactions, or why the proposed combination was necessary.
-
Empirical support and practical benefit. The original failure-mode diagnostics were limited to a single run with the smallest backbone, and the stability of the gains across training runs was unclear. I was also not convinced that the improvements established a substantial practical benefit from the additional design complexity.
Additional Comments After Rebuttal:
I appreciate the additional experiments and the authors' clarification of the scope of their claims.
-
Attribution of performance gains. The new ProxMO-LC baseline and the controlled local-horizon experiment provide useful evidence of gains beyond simple reward calibration. These results substantially address this part of my concern.
-
Justification of the main design choices. The additional RSSMix and routing-gate experiments strengthen the justification for these components. However, the first step in the cumulative analysis changes reward separation, applicability, calibration, and normalization together. The contribution of reward separation itself therefore remains unclear. The trajectory anchor is also held fixed, and the rebuttal does not compare the history-kernel baseline with a uniform average within the same comparison pools. The cumulative analysis supports the combined design, but my concern about the contributions of the main design choices is only partially addressed.
-
Empirical support and practical benefit. The three-seed results on Qwen2.5-1.5B and Gemma-3-4B strengthen the evidence that the gains are consistent. I also appreciate the small increase in total training time measured in the 1.5B setting. However, the reported zero-search collapse occurs when type-specific comparison pools are removed from TRACER, while the authors state that ProxMO-C does not collapse. This supports the importance of these pools within TRACER, but does not establish how frequently the diagnosed failure modes occur in existing methods or how much they affect performance.
Overall, the rebuttal strengthens the empirical evidence and partially addresses my concerns. However, the contribution of the central design choices and the practical importance of the diagnosed failure modes remain insufficiently established. I am therefore still not convinced that the contribution is substantial enough for acceptance, and I will maintain my original score.
Response to Reviewer karY on Gains Beyond Reward Calibration, Multi-Seed Results, and Training Cost
Thank you. We agree that the original ProxMO-C comparison supported the complete system but did not isolate typed credit from calibration, state representation, gating, or scale. We added a fixed-protocol evidence chain and narrowed the claim.
Beyond trajectory-level calibration. All new comparisons use a common 50-step validation grid: one checkpoint is selected by mean NQ/HotpotQA validation EM, then its complete seven-test vector is reported. On Qwen2.5-1.5B, ProxMO-C gives 37.13 +/- 0.23 macro EM over seeds 0/1/2. A timing-preserving calibration control (ProxMO-LC), which calibrates length-growing auxiliary rewards while leaving them at their observed turns, gives 37.48 +/- 0.25; thus terminal relocation is not our only calibration counterfactual.
We then use common
(question, projected-action-type)history-kernel LOTO pools for a reproducible cumulative analysis. The scalar cell usesr_o + 0.0625 r_d/N_d + 0.3125 r_ewith union applicability; the first split cell restores rawr_d, pathway-specific applicability, separate RMS, and gamma=0.95 for every content pathway. The scalar-content formula excludes format: every row retains the same separate format pathway, final routing gate, Eq. 9 trajectory anchor, and fusion rule; only the content-estimator setting named in the row changes.The paired conditional increments are +0.39 +/- 0.02, +0.43 +/- 0.01, +0.46 +/- 0.03, and +0.23 +/- 0.01 pp. The first is explicitly a joint reward-separation/applicability/calibration/normalization change, and the table is cumulative, not an order-independent factorial. The horizon comparison keeps rewards, weights, masks, candidate rules, RMS pools, gate, anchor, and fusion fixed. Because nonlocal recursion can be interrupted by missing successors, local horizons also increase valid-credit coverage. For this common-support training control, each variant computes both horizon supports on its own rollout at every update and retains only turn-pathway records with valid targets under both rules before RMS and policy-loss fusion; test evaluation remains unfiltered. Retraining both variants under this shared-support mask leaves a +0.34 +/- 0.01 pp macro-EM gain for local horizons.
Reliability and generalization. Per-seed ProxMO-C to TRACER macro EM is 37.17 to 38.81, 36.88 to 38.42, and 37.34 to 38.95 (paired +1.60 +/- 0.05). On the 2025 Gemma-3-4B-IT family, the same comparison is 41.36 +/- 0.29 to 44.01 +/- 0.21 (paired +2.65 +/- 0.20, positive on 7/7 benchmarks). The common-pool scalar-to-full comparison also replicates on Gemma: 42.64/42.37/42.61 versus 44.20/43.78/44.05 by seed, or 42.54 +/- 0.15 to 44.01 +/- 0.21 (paired +1.47 +/- 0.08).
Routing attribution. Reward-side all-non-format conditioning gives 36.99 +/- 0.16. Adding the same final eligibility mask gives 38.35 +/- 0.24; retaining unmodified upstream rewards/statistics gives full TRACER at 38.73 +/- 0.27. Thus, in this tested chain, final eligibility supplies the larger improvement and preserving upstream statistics adds a further 0.38 pp. We do not claim universal superiority over all reward-side conditioning. Likewise, type pools are important specifically for separately normalized local pathways: removing them produces 92.4 +/- 2.1% and 89.6 +/- 2.8% zero-search trajectories on Qwen-7B/Llama-8B, whereas ProxMO-C does not collapse.
Practical cost. Three-seed means across 654 updates on 1.5B give ProxMO-C to TRACER estimator time of 2.17 to 2.57 s/update (+18.4%), end-to-end wall time of 51.3 to 51.8 h (+0.9%), GPU-hours of 410.4 to 414.4 (+1.0%), and peak memory per GPU of 71.6 to 72.0 GiB (+0.6%). These totals include training, scheduled validation, and retrieval waiting, and exclude the final seven-benchmark test for both methods. Training-plus-validation tokens/question decrease 3.6% and searches/trajectory 13.4%; inference has no TRACER component.
We will revise the claim to a repeatable, cost-compatible cumulative estimator design for this multi-reward search setting, not a universal credit-correctness theorem or a claim that every ingredient is novel.