- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to combine two step-level credit signals in agentic RL: GiGPO's outcome-based step advantage and a hindsight signal from on-policy self-distillation, where the policy rescores its own response given a successful peer's actions. Observing that the two signals are positively correlated on average but often disagree locally, the authors propose UniOPSD. A running correlation between the signals sets a global mixing weight, per-row precision proxies adjust it, and a bounded token modulation is applied on top. Experiments cover ALFWorld, WebShop and Search-QA with Qwen2.5-3B and 7B.
Critical strengths
The paper is clearly written and unusually careful about the limits of each design choice. The diagnostics on agreement, disagreement and coverage between the two signals are informative.
Critical weaknesses
-
The central mechanism is not shown to matter. With u = 0.5 as used in all experiments, Eq. (7) simplifies exactly to c̄ = 1 − 2ρ̂², since the denominator is always 0.25. The frozen-correlation variant (ρ0 = 0.25) gives c̄ = 0.875. The observed mean correlations on ALFWorld and WebShop (0.27 to 0.32) give c̄ between about 0.80 and 0.85 before shrinkage, and shrinkage moves them closer to 0.875. So the adaptive rule and the frozen control differ by a few percent in global weight, and only on the 10% to 33% of rows where both signals exist. On WebShop the all-row mean outcome weight is already 0.976. It is hard to see how this difference produces the 1.6 to 3.1 point gaps in Table 2. Also, the "w/o teacher" ablation removes the hindsight branch and token modulation together, so the gain over GiGPO cannot be attributed to the arbitration rule rather than simply adding hindsight credit.
-
The method does not outperform existing approaches. On ALFWorld it is below the strongest baseline at both scales (82.8 vs SDAR's 84.4 for 3B, 83.6 vs StepOPSD's 88.4 for 7B). On 7B WebShop it is below SDAR (82.0 vs 82.8), and on Search-QA it leads by under one point. The only clear gain is 3B WebShop. Existing fixed-weight combinations such as SDAR and StepOPSD therefore already handle this setting at least as well, which weakens the case that adaptive arbitration is needed.
-
The reported numbers are peak values from online validation with 128 episodes. For example, 3B ALFWorld is reported at 82.8 (update 125) while the final checkpoint is 75.0. With this protocol, differences of a few points are not reliable, which matters because nearly all the comparisons above are within that range.
Questions for authors
-
Can you compare UniOPSD with GiGPO plus hindsight credit and token modulation using a fixed mixing weight, for several values of c (e.g. 0.7, 0.8, 0.9)? This is the direct test of whether adaptive arbitration helps.
-
Please report results from a fixed final checkpoint on the full evaluation split for all methods, and state whether the baselines in Table 1 were rerun or taken from prior papers.
-
Given that c̄ = 1 − 2ρ̂² at u = 0.5, what is the actual range of c̄ during training on each benchmark, and how does it compare with the frozen variant's 0.875?
2 comments
We sincerely thank you for your careful review and constructive feedback, and we apologize for not responding during the rebuttal period. We greatly appreciate the time and effort you have devoted to evaluating our work. We will carefully consider your suggestions and revise the manuscript accordingly before our next submission. Thank you again for your valuable input. We also wish you every success in your research and future submissions!
The authors did not provide any response during the rebuttal phase, so I am maintaining my score.