acceptodds
Under review as a conference paper at ICLR 2027

Typed-Reward-Aware Credit Estimation and Routing for Multi-Reward Agentic Reinforcement Learning

Abstract

Practical language-model agents face diverse, task-dependent reward requirements. Common examples concern task outcomes, intermediate processes, and conditions on valid interaction. Reinforcement learning for these agents therefore faces a multi-reward credit-assignment problem, extending beyond distributing a final outcome score across turns. We identify three coupled difficulties: shared temporal propagation can let dense process feedback overwhelm outcome credit; mixed action-type statistics can encourage easily rewarded behaviors; and additive reward fusion can allow task gains to offset violated prerequisites. We propose **TRACER**, Typed-Reward-Aware Credit Estimation and Routing, to preserve these distinctions throughout credit assignment. TRACER estimates reward-specific advantages with separate horizons, baselines, and normalization scales, refines comparisons by action type where appropriate, and uses conditional rewards to gate which pathways enter each update. The normalized pathways are then fused into a single policy advantage. We instantiate three common reward types—outcome, process, and conditional rewards—in search-augmented question answering. Experiments across seven benchmarks and three backbones show improvements in answer accuracy over group-relative baselines. Component analyses connect these gains to auxiliary-reward horizons, action-type comparisons, and sustained format compliance. Together, these results support treating temporal responsibility, action comparability, and conditional eligibility as joint requirements for credit assignment in practical multi-turn agent RL.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning, Multi-Reward Agentic Training, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_karYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACER for multi-turn search agent RL. It computes advantages separately for different reward types and combines them after format-based gating. The method is evaluated on QA benchmarks with multiple backbones.

Critical strengths

The paper addresses an important question: how different reward types should be handled when they serve different purposes in agent training.

Critical weaknesses
  • Attribution of performance gains is not fully clear: ProxMO-C is a useful control for auxiliary reward accumulation, but its comparison with TRACER changes several aspects of credit estimation together. The paper should better establish how much of the improvement comes from the proposed method beyond reward calibration.

  • The main design choices need stronger justification: TRACER combines several credit estimation and routing components, but their contributions and interactions are not sufficiently explained. The existing ablations are useful, but they do not fully establish why the proposed combination is necessary or how it improves the learning signal.

  • The empirical evidence does not fully support the scope of the claims: The failure-mode diagnostics are limited to a single run with the smallest backbone. The gains over ProxMO-C are positive, but their stability across runs is unclear. Overall, I am not fully convinced that the results establish a substantial practical benefit from the additional design complexity.

Overall, the idea is reasonable, but the supporting evidence and justification remain insufficient, and the gains over the calibrated baseline are modest. I am therefore not convinced that the contribution is substantial enough for acceptance at ICLR.

Questions for authors
  • What evidence distinguishes the benefit of the proposed credit estimation method from reward calibration and supports the main design choices beyond the complete system's performance?

  • How consistently do the diagnosed failure modes and performance gains occur across backbones and training runs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer karY on Gains Beyond Reward Calibration, Multi-Seed Results, and Training Cost

Thank you. We agree that the original ProxMO-C comparison supported the complete system but did not isolate typed credit from calibration, state representation, gating, or scale. We added a fixed-protocol evidence chain and narrowed the claim.

Beyond trajectory-level calibration. All new comparisons use a common 50-step validation grid: one checkpoint is selected by mean NQ/HotpotQA validation EM, then its complete seven-test vector is reported. On Qwen2.5-1.5B, ProxMO-C gives 37.13 +/- 0.23 macro EM over seeds 0/1/2. A timing-preserving calibration control (ProxMO-LC), which calibrates length-growing auxiliary rewards while leaving them at their observed turns, gives 37.48 +/- 0.25; thus terminal relocation is not our only calibration counterfactual.

We then use common (question, projected-action-type) history-kernel LOTO pools for a reproducible cumulative analysis. The scalar cell uses r_o + 0.0625 r_d/N_d + 0.3125 r_e with union applicability; the first split cell restores raw r_d, pathway-specific applicability, separate RMS, and gamma=0.95 for every content pathway. The scalar-content formula excludes format: every row retains the same separate format pathway, final routing gate, Eq. 9 trajectory anchor, and fusion rule; only the content-estimator setting named in the row changes.

Content estimatorMacro EM, mean +/- SD
Calibrated scalar content, shared gamma, one RMS37.21 +/- 0.22
Split pathways, native applicability, separate RMS37.60 +/- 0.24
+ local process horizons38.03 +/- 0.24
+ RSSMix38.50 +/- 0.27
+ original turn-index rules (full TRACER)38.73 +/- 0.27

The paired conditional increments are +0.39 +/- 0.02, +0.43 +/- 0.01, +0.46 +/- 0.03, and +0.23 +/- 0.01 pp. The first is explicitly a joint reward-separation/applicability/calibration/normalization change, and the table is cumulative, not an order-independent factorial. The horizon comparison keeps rewards, weights, masks, candidate rules, RMS pools, gate, anchor, and fusion fixed. Because nonlocal recursion can be interrupted by missing successors, local horizons also increase valid-credit coverage. For this common-support training control, each variant computes both horizon supports on its own rollout at every update and retains only turn-pathway records with valid targets under both rules before RMS and policy-loss fusion; test evaluation remains unfiltered. Retraining both variants under this shared-support mask leaves a +0.34 +/- 0.01 pp macro-EM gain for local horizons.

Reliability and generalization. Per-seed ProxMO-C to TRACER macro EM is 37.17 to 38.81, 36.88 to 38.42, and 37.34 to 38.95 (paired +1.60 +/- 0.05). On the 2025 Gemma-3-4B-IT family, the same comparison is 41.36 +/- 0.29 to 44.01 +/- 0.21 (paired +2.65 +/- 0.20, positive on 7/7 benchmarks). The common-pool scalar-to-full comparison also replicates on Gemma: 42.64/42.37/42.61 versus 44.20/43.78/44.05 by seed, or 42.54 +/- 0.15 to 44.01 +/- 0.21 (paired +1.47 +/- 0.08).

Routing attribution. Reward-side all-non-format conditioning gives 36.99 +/- 0.16. Adding the same final eligibility mask gives 38.35 +/- 0.24; retaining unmodified upstream rewards/statistics gives full TRACER at 38.73 +/- 0.27. Thus, in this tested chain, final eligibility supplies the larger improvement and preserving upstream statistics adds a further 0.38 pp. We do not claim universal superiority over all reward-side conditioning. Likewise, type pools are important specifically for separately normalized local pathways: removing them produces 92.4 +/- 2.1% and 89.6 +/- 2.8% zero-search trajectories on Qwen-7B/Llama-8B, whereas ProxMO-C does not collapse.

Practical cost. Three-seed means across 654 updates on 1.5B give ProxMO-C to TRACER estimator time of 2.17 to 2.57 s/update (+18.4%), end-to-end wall time of 51.3 to 51.8 h (+0.9%), GPU-hours of 410.4 to 414.4 (+1.0%), and peak memory per GPU of 71.6 to 72.0 GiB (+0.6%). These totals include training, scheduled validation, and retrieval waiting, and exclude the final seven-benchmark test for both methods. Training-plus-validation tokens/question decrease 3.6% and searches/trajectory 13.4%; inference has no TRACER component.

We will revise the claim to a repeatable, cost-compatible cumulative estimator design for this multi-reward search setting, not a universal credit-correctness theorem or a claim that every ingredient is novel.

AuthorsOpenReview ↗

Follow-up on Gains Beyond Calibration, Cross-Backbone Consistency, and Practical Cost

Dear Reviewer karY,

Thank you again for your careful review and for highlighting the need to distinguish gains beyond reward calibration and better justify the additional design complexity. As the discussion period draws to a close, we wanted to respectfully follow up on our response.

Our rebuttal adds a timing-preserving calibration baseline, a cumulative comparison using common comparison pools, and a shared-support control for local process horizons. We also report consistent three-seed gains on Qwen2.5-1.5B and Gemma-3-4B-IT, failure-mode evidence on additional backbones, and measured practical costs, including a 0.9% increase in end-to-end training time in the tested 1.5B setting. These results support a more narrowly scoped claim, and we explicitly acknowledge that the cumulative increments depend on the order of component addition. We will incorporate the additional evidence and clarified limitations into the revised manuscript.

We hope this helps address your concerns about attribution, consistency, and practical benefit. If any concern remains unresolved, we would greatly appreciate the opportunity to clarify it promptly. If you find the new evidence sufficient to reconsider your assessment, we would sincerely appreciate your consideration of an upward rating revision. Thank you again for your time and thoughtful judgment.

Sincerely, The Authors

AuthorsOpenReview ↗

Brief Follow-up Before the Discussion Deadline

Dear Reviewer karY,

With the discussion deadline approaching, we would appreciate any brief feedback on whether our rebuttal addresses your main concerns. We understand that your time is limited and thank you for considering this follow-up.

The additional evidence directly targets the questions raised in your review:

  • Gains beyond calibration: We added a timing-preserving calibration baseline and controlled cumulative comparisons, including a shared-support test for local process horizons.
  • Consistency and practical benefit: Three-seed comparisons show improvements over ProxMO-C on both Qwen2.5-1.5B and Gemma-3-4B-IT, with a measured 0.9% increase in end-to-end training time in the tested 1.5B setting.

We will incorporate these results and their limitations into the revised manuscript, explicitly presenting the component analysis as cumulative and order-dependent.

If a concern still prevents a more favorable assessment, even a brief indication would help us respond before the deadline. If the additional evidence has sufficiently addressed your concerns, we would be grateful if you would consider revising your rating upward.

Thank you again for your time and thoughtful review.

Sincerely,
The Authors

I would like to clarify that I read and evaluated the paper myself. I used ChatGPT to look up related information, discuss the paper, and help organize and phrase my comments in English. The review contains substantial human input, and the concerns below reflect my own assessment of the paper.

My original concerns are as follows:

  1. Attribution of performance gains. The comparison between ProxMO-C and TRACER changed several aspects of credit estimation together. It was therefore unclear how much of the improvement came from the proposed method beyond reward calibration.

  2. Justification of the main design choices. TRACER combines several credit estimation and routing components. The original ablations did not sufficiently explain their individual contributions and interactions, or why the proposed combination was necessary.

  3. Empirical support and practical benefit. The original failure-mode diagnostics were limited to a single run with the smallest backbone, and the stability of the gains across training runs was unclear. I was also not convinced that the improvements established a substantial practical benefit from the additional design complexity.

Additional Comments After Rebuttal:

I appreciate the additional experiments and the authors' clarification of the scope of their claims.

  1. Attribution of performance gains. The new ProxMO-LC baseline and the controlled local-horizon experiment provide useful evidence of gains beyond simple reward calibration. These results substantially address this part of my concern.

  2. Justification of the main design choices. The additional RSSMix and routing-gate experiments strengthen the justification for these components. However, the first step in the cumulative analysis changes reward separation, applicability, calibration, and normalization together. The contribution of reward separation itself therefore remains unclear. The trajectory anchor is also held fixed, and the rebuttal does not compare the history-kernel baseline with a uniform average within the same comparison pools. The cumulative analysis supports the combined design, but my concern about the contributions of the main design choices is only partially addressed.

  3. Empirical support and practical benefit. The three-seed results on Qwen2.5-1.5B and Gemma-3-4B strengthen the evidence that the gains are consistent. I also appreciate the small increase in total training time measured in the 1.5B setting. However, the reported zero-search collapse occurs when type-specific comparison pools are removed from TRACER, while the authors state that ProxMO-C does not collapse. This supports the importance of these pools within TRACER, but does not establish how frequently the diagnosed failure modes occur in existing methods or how much they affect performance.

Overall, the rebuttal strengthens the empirical evidence and partially addresses my concerns. However, the contribution of the central design choices and the practical importance of the diagnosed failure modes remain insufficiently established. I am therefore still not convinced that the contribution is substantial enough for acceptance, and I will maintain my original score.

Reviewer_r6U5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACER, a framework for credit assignment in multi-turn RL with heterogeneous rewards. It estimates reward-specific advantages using separate temporal horizons and baselines, compares actions within type-specific pools, and uses conditional requirements to gate which advantages enter policy updates. A trajectory-level anchor supplements the turn-level credit. Experiments combine outcome, process, and format rewards in search QA benchmarks. Across different benchmarks, TRACER improves accuracy over baselines by 1.6-4.0 percentage points.

Critical strengths
  1. Problem formulation is clear. The distinction between reward horizons, action comparability, and prerequisite conditions provides a useful framework for understanding failures in multi-reward agent training.
  2. Clear experimental improvement compared with strong baselines across several search QA benchmarks.
  3. Ablation studies are informative. Removing action-type isolation reduces acc a lot and leads to near-complete abandonment of retrieval. The gate ablation retains format rewards but exhibits deteriorating compliance, supporting the practical importance of routing within the proposed estimator.
Critical weaknesses
  1. TRACER changes both where gating is applied and which training signals it blocks. The comparison therefore does not clearly isolate the benefit of its particular gating design.
  2. Format failures include both malformed outputs and failure to stop searching in time. Reporting these separately would clarify which behavior it improves.
  3. All experiments use models released in 2024. It remains unclear whether the same performance gains would hold on newer model families.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer r6U5 on Gate Placement, Stopping and Structural Failures, and Transfer to Gemma 3

Thank you for recognizing the clear problem formulation, the gains over strong search QA baselines, and the evidence from the action-type and gate ablations. Your review identified three questions the original results left open. The new controls isolate gate placement, distinguish failure to stop from structural errors, and show gains on Gemma-3-4B-IT across all three seeds.

1. Gate placement with matched credit eligibility. On Qwen2.5-1.5B (seeds 0/1/2), all rows share rewards, LOTO/GAE/RMS estimator definitions, threshold, weights, budget, and checkpoint rule.

Conditioning and failed-turn scopeMacro EM, mean ± SD
Reward-side, process only, anchor retained36.43 ± 0.17
Advantage-side, process only, anchor retained37.35 ± 0.20
Reward-side, all non-format pathways, Eq. 9 anchor masked36.99 ± 0.16
Reward-side plus final pathway mask, Eq. 9 anchor masked38.35 ± 0.24
Advantage-side, all non-format pathways, Eq. 9 anchor masked (TRACER)38.73 ± 0.27

Every row constructs BgB_g from the same unmodified outcome and evidence-gain rewards. Reward-side zeroing affects only copies entering the turn-channel estimators. Rows 1–2 retain BgB_g, and rows 3–5 apply the same gateg,tBg\mathrm{gate}_{g,t}B_g anchor mask.

Post-estimation routing gains +0.92 pp in the process-only contrast and +1.74 pp in the all-non-format contrast. Rows 4 and 5 hold direct failed-turn eligibility fixed through identical final pathway and anchor masks. With the same anchor inputs and RSSMix denominator rule, preserving the original reward inputs through return estimation, baselines, and normalization adds +0.38 pp (38.35 to 38.73). Starting from row 5 (full TRACER), we change only the format-failing-turn denominator from the post-gate included-weight norm to the pre-gate L2 weight norm over all valid, applicable pathways; the numerator, anchor mask, and all other settings remain fixed. This gives 38.44 ± 0.26.

2. Failure to stop and structural errors. At each seed's validation-selected checkpoint, we evaluated one greedy trajectory per example across all seven test sets and report trajectory-weighted means over seeds. A/L/M partition terminal states into submitted answer, valid search at action 6 (failure to stop), and missing or malformed terminal action. X independently flags a pure structural violation at any turn (think-block, completeness, multiplicity, simultaneous types, empty action, or trailing text), excluding expected action type. The original failure metric is F=X∪LF=X\cup L. We checked for overlap and found none in this evaluation.

MethodALMXF
ProxMO-C82.5%13.8%3.7%5.2%19.0%
TRACER without gate78.4%16.7%4.9%6.8%23.5%
TRACER94.6%4.6%0.8%1.5%6.1%

At the exact step-650 checkpoint, we used Figure 4's fixed validation panel of 1,000 unique NQ and 1,000 unique HotpotQA questions (split seed 42), with one greedy trajectory per question and no checkpoint selection or format filtering. We pooled 2,000 trajectories per seed across three seeds. Equal denominators make this equivalent to averaging per-seed rates. L/X/F are 2,988/1,344/4,332 of 6,000 (49.8/22.4/72.2%) without the gate and 306/108/414 of 6,000 (5.1/1.8/6.9%) with TRACER. Failure to stop dominates the late collapse, and structural errors also rise.

EM uses all trajectories. In this evaluation, the exact equal-benchmark macro decomposition of TRACER's +1.60 pp over ProxMO-C gives +0.88 pp when both submit, +0.79 when only TRACER submits, −0.07 when only ProxMO-C submits, and 0 when neither submits. The both-submit subset thus contributes positively to the gain. This decomposition is descriptive.

3. Transfer to Gemma-3-4B-IT. ProxMO-C macro EM by seed is 41.40/41.05/41.62 (41.36 ± 0.29), and TRACER achieves 44.20/43.78/44.05 (44.01 ± 0.21). Paired gains are +2.80/+2.73/+2.43 pp (+2.65 ± 0.20 pp).

Each seed evaluates the same 125 Bamboogle questions once. Correct counts are 51/125, 50/125, 51/125 for ProxMO-C (152/375 = 40.53%) and 56/125, 55/125, 55/125 for TRACER (166/375 = 44.27%). In Table 1 order (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle), three-seed mean EM is [44.10, 62.20, 44.80, 40.40, 39.60, 17.90, 40.53] for ProxMO-C and [47.10, 63.30, 47.30, 43.20, 42.50, 20.40, 44.27] for TRACER. All seven differences are positive, with unweighted means of 41.36 and 44.01.

We will incorporate all three sets of results into the revised manuscript, add the gate controls and diagnostics alongside Figure 4, and limit our reliability claim to improved answer submission and the defined structural checks in this search setting. We will explicitly discuss the limits regarding factuality, constraint satisfaction, and possible indirect cross-turn effects. If these results address your concerns, we would be grateful if you would consider raising your overall rating.

AuthorsOpenReview ↗

Follow-up on Gate Placement, Failure Diagnostics, and Gemma-3 Results

Dear Reviewer r6U5,

Thank you again for your thoughtful review and for recognizing the clear problem formulation, empirical improvements, and informative ablations. With the discussion deadline approaching, we wanted to briefly follow up on the three concerns raised in your review.

Our rebuttal provides gate-placement controls with identical final pathway and anchor masks, helping distinguish the effects of final credit eligibility from those of preserving the original rewards during estimation. We also report failure to stop and structural errors separately, and extend the evaluation to Gemma-3-4B-IT, where TRACER improves over ProxMO-C across all three seeds, with a mean gain of 2.65 pp. We will incorporate these controls and diagnostics into the revised manuscript and explicitly delimit the reliability claims supported by this evaluation.

We hope these additions have addressed the questions you identified. If any issue remains unclear or requires further discussion, please let us know; we would be happy to respond promptly. If the new evidence resolves your concerns, we would be very grateful if you would consider revising your rating upward accordingly. We fully understand that the final assessment remains at your discretion and sincerely appreciate your time and careful consideration.

Sincerely, The Authors

Thanks to authors for their efforts in rebuttal and additional experiments, which address my concern. I will raise my score.

AuthorsOpenReview ↗

Thank You for Your Constructive Feedback and Updated Assessment

Dear Reviewer r6U5,

Thank you for reconsidering your assessment and raising your score. We sincerely appreciate your time and constructive feedback, which helped us strengthen the experimental evidence and clarify our claims. We will incorporate the additional results and clarifications into the revised manuscript as promised.

Best regards, The Authors

Reviewer_BTznOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TRACER, a framework for credit assignment in multi-turn agentic RL with multiple reward signals. The authors identify that different reward signals require different temporal horizons. To address this, the method separately estimates advantages for outcome, process, and conditional rewards using reward-specific temporal horizons and action-type comparisons. The authors evaluate TRACER on search-augmented question answering using three language models and seven benchmarks. The results show improvements over GRPO and other baselines.

Critical strengths
  1. The paper is well-motivated for handling different reward signals separately during credit assignment. The distinction between outcome, process, and conditional rewards is reasonable and intuitive for training multi-turn agents.

  2. The experimental evaluation is good and spans seven benchmarks and three model backbones. The authors provide comparisons with relevant baselines and useful ablations of the proposed components. The results show consistent improvements across most benchmarks.

Critical weaknesses
  1. My main concern is the technical novelty of the proposed approach. The method combines reward-specific GAE, normalization, action-type comparisons, and condition gating, which are individually standard techniques. Although the proposed combination is effective, it is unclear whether it constitutes a significant methodological contribution. I think the authors should better justify the novelty of the proposed credit-estimation framework.

  2. Another concern is the reliability of the experimental improvements. Appendix C.4 mentions that the results are obtained using only one training seed, without reporting variance across independent runs. Although TRACER improves over the baselines, some of the gains are marginal. Without multiple training seeds, it is difficult to determine whether these improvements are consistent.

Questions for authors
  1. Can the authors provide ablations isolating the contributions of RSSMix, trajectory anchor, etc? This would help understand whether the additional components of the proposed method are necessary.

  2. Can the authors evaluate TRACER across multiple training seeds and report the mean and variance of the results, particularly on Qwen2.5-1.5B, where the improvement over the strongest baseline is relatively small?

5 comments
AuthorsOpenReview ↗

Response to Reviewer BTzn on Gate Placement, Multi-Seed Results, and Component Ablations

Thank you for recognizing the motivation for treating outcome, process, and conditional rewards separately, and for highlighting the evaluation across seven benchmarks and three model backbones. Your questions about novelty, reliability, and the role of individual components helped us identify the controls needed to substantiate our claims.

1. Methodological contribution and gate placement

We ran a controlled gate-placement study on Qwen2.5-1.5B. Every variant uses the same gate, g_t = 1[r_format,t >= 0.95], conditions the same three pathways (outcome, redundancy, and evidence gain), and retains failed-format credit. We hold fixed the GAE parameters, LOTO/type pools, RMS definitions, RSSMix weights, training budget, and checkpoint rule. The anchor is computed from the same unconditioned trajectory scores and multiplied by the same g_t in every variant.

Conditioning of non-format pathwaysSeed 0 / 1 / 2 macro EMMean ± SD
Reward side, before LOTO/GAE/RMS37.12 / 36.81 / 37.0436.99 ± 0.16
Reward side + final advantage mask38.53 / 38.08 / 38.4438.35 ± 0.24
Advantage side, after estimation (TRACER)38.81 / 38.42 / 38.9538.73 ± 0.27

Reward-side conditioning replaces each non-format immediate reward with g_t * r_t^(k) before value estimation, GAE, and RMS normalization. In this variant, the estimated advantages flow directly into fusion. TRACER applies the same gate to normalized pathway advantages immediately before fusion, with estimation performed on the original rewards. The hybrid applies both operations.

  • Adding the final mask to reward-side conditioning recovers 1.36 pp (36.99 to 38.35). Propagated TD residuals and relative baselines can still assign non-format advantage to a failed action after its immediate rewards are masked.
  • Removing reward-side conditioning while keeping the same final pathway and anchor masks adds 0.38 pp, with paired gains of +0.28/+0.34/+0.51 pp. This comparison isolates the effect of preserving the temporal estimates and comparison statistics used at other actions. TRACER improves over reward-side conditioning by +1.69/+1.61/+1.91 pp across the three paired runs (mean +1.74 pp).

We agree that binary gating, GAE, and reward-wise normalization are established techniques. TRACER's contribution is to order these operations within a typed credit estimator as typed estimation → typed normalization → conditional routing → fusion. On a format-failing action, the fusion and anchor masks exclude direct outcome and process credit while retaining failed-condition credit. This property concerns direct credit at the failing action. Constraint satisfaction and indirect cross-turn effects require separate analysis. Positive reward rescaling is canceled for a fixed normalized pathway up to epsilon. The final anchor-mixed advantage can still depend on reward scale.

2. Reliability across training seeds

All reported multi-seed results use seeds 0/1/2, identical budgets, and a common 50-step validation grid. For each run, we select one checkpoint by mean NQ/HotpotQA validation EM and compute macro EM from its complete seven-benchmark vector. ProxMO-C scores 37.17/36.88/37.34 (37.13 ± 0.23) at steps 550/600/500. TRACER scores 38.81/38.42/38.95 (38.73 ± 0.27) at steps 550/600/600. The paired gains are +1.64/+1.54/+1.61 pp, with a mean ± SD of +1.60 ± 0.05 pp.

3. RSSMix and trajectory anchor

  • RSSMix. Replacing RSSMix with the identical weighted sum without its active-set L2 denominator gives 37.87 ± 0.23 after RMS matching. Full-minus-control gaps are +0.87/+0.81/+0.89 pp.
  • Trajectory anchor. Removing only the anchor while retaining the full-system 0.9 turn coefficient (rather than resetting it to 1) gives 38.19 ± 0.25. Full-minus-control gaps are +0.54/+0.51/+0.57 pp.

These controls are retrained: at every update, both estimators run on the control rollout; the control estimator's pre-clipped valid-turn advantages are matched to the fixed-setting full TRACER estimator's RMS, clipped to [-5, 5], and used for that update.

The cumulative matched analysis gives 37.21 (calibrated scalar content), 37.60 (typed pathways, applicability, and separate RMS), 38.03 (local process horizons), 38.50 (+RSSMix), and 38.73 (+turn-index rules). The increments depend on the order of addition. The first transition jointly changes separation, applicability, and calibration.

Manuscript revision

We will add the placement table, per-seed results, checkpoint-selection protocol, and RMS-matched controls to the revised manuscript. We will update Appendix C.4 to document the new multi-seed results, label the cumulative ablation as order-conditional, and revise the abstract and method claims to reflect the scope above. If these results address your concerns about the methodological contribution and reliability of the gains, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Follow-up on Methodological Contribution, Multi-Seed Reliability, and Component Ablations

Dear Reviewer BTzn,

Thank you again for your careful and constructive review, and for recognizing the motivation and breadth of our evaluation. As the discussion period is nearing its end, we wanted to respectfully follow up on our rebuttal.

In response to your concerns, we clarified the methodological contribution of typed estimation before fusion, provided controlled gate-placement comparisons, and reported three-seed results on Qwen2.5-1.5B showing consistent improvements over ProxMO-C (+1.60 pp on average). We also added RMS-matched, retrained ablations of RSSMix and the trajectory anchor to better establish their individual contributions. We will incorporate these results and the checkpoint-selection protocol into the revised manuscript, while making the scope and limitations of our claims explicit.

We hope these additions help address your concerns about novelty, reliability, and component necessity. If any concern remains unresolved, we would be very grateful if you could let us know, and we will respond promptly. If you find that the additional evidence adequately addresses your concerns, we would sincerely appreciate your consideration of an upward revision to your rating. We fully respect your final judgment and thank you again for your time and thoughtful feedback.

Sincerely, The Authors

I appreciate the multiple-seed evaluation and the controlled ablations of RSSMix and the anchor. However, I am still not convinced about the methodological novelty. In the gate-placement study, adding a final advantage mask to reward-side conditioning already recovers most of the performance difference. The proposed advantage-side conditioning further improves performance to 38.73, which is a marginal gain and could just be the noise. Hence, I do not believe that the proposed approach offers a significant methodological contribution. Therefore, I will maintain my Weak Reject rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.