- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies a "misguided reinforcement" problem in multi-turn agentic RL with outcome-only rewards: tokens belonging to bad local behaviors (invalid tool calls, repeated queries) frequently receive positive advantages on successful trajectories. C3 is a lightweight post-hoc correction applied between advantage estimation and the actor update — positive advantages on detector-flagged spans are flipped, scaled by the behavior policy's confidence on that span — leaving the reward, return, and critic untouched. Experiments cover both PPO and GDPO on Search QA (7 benchmarks) and WebShop.
Critical strengths
The phenomenon is documented convincingly. The token-level audit showing that ~29% of detector-flagged tokens still receive positive credit, together with the training-dynamics analysis, is a useful observation for anyone doing multi-turn RL. I also found the process-reward comparison genuinely informative: injecting the same signal as a turn-level reward makes things worse (query loops roughly triple, WebShop collapses late in training), which supports the paper's central point that where a process signal enters matters.
The method itself is simple, cheap, and genuinely plug-and-play, and it shows consistent — if modest — gains across two RL paradigms, two environments, and a third-party recipe (T2PO), which is more coverage than most papers of this type provide.
Critical weaknesses
The baseline coverage is thin for a paper titled "rethinking process supervision." None of the existing turn/step-level credit assignment methods discussed in the related work (Turn-PPO, StepSearch, GiGPO, etc.) are compared against — the only process-supervision baseline is the authors' own turn-level penalty. So we learn that C3 beats naive reward injection, but not whether it beats existing process supervision.
The evidence also has statistical weaknesses: the error bars come from sampling at test time, not from multiple RL training runs, which is where most of the variance lives; the flagship gain (2Wiki +11.5 under PPO) flips sign under GDPO (-2.2) and is never really explained; WebShop numbers use best-checkpoint selection. On the method side, the design is purely heuristic — the token-level confidence variant is worse than doing nothing, there is no ablation with constant correction strength, and the detector's precision/recall is never measured, so the damage from false positives is unknown.
Questions for authors
- How does C3 perform with constant correction strength (e.g. κ=1, severity only)? If that already matches, is the confidence weighting buying anything? And why should high-confidence bad behavior be corrected more strongly rather than less?
- What are the detector's precision and recall, especially for near-duplicate query loops? How sensitive are the results to the similarity threshold?
- Why does 2Wiki improve by +11.5 under PPO but degrade under GDPO? And how many independent RL training runs underlie each number in the main table?
4 comments
Response to Reviewer L5TC(2)
W2.4 & Q1 — The token-level confidence variant is worse than doing nothing. How does C³ perform with constant correction strength, and why correct high-confidence bad behavior more strongly?
A: We thank the reviewer for raising these important questions about the confidence design. To better examine its contribution, we have added constant-strength and masking controls. Figure 7 (left) uses the C³ endpoint from Table 1 and compares it with the following alternatives.
| Correction rule | Macro strict EM (%) |
|---|---|
| Token probability | 44.73 ± 0.18 |
| Geometric span mean | 49.06 ± 0.22 |
| Arithmetic span mean, C³ | 50.20 ± 0.10 |
| Whole-bad-turn masking | 48.74 ± 0.25 |
| Constant κ = 0.5 | 46.56 ± 0.44 |
| Constant κ = 1.0 | 50.16 ± 0.25 |
The token-level variant underperforms PPO (44.73% vs. 47.79%). We suspect that token likelihood may misallocate correction according to lexical predictability: within a flagged turn, reusable formatting may receive more weight than tokens defining the offending action, potentially interfering with valid behavior. Span averaging avoids this issue by applying a uniform gain across the behavioral event.
C³ and constant κ=1.0 achieve comparable strict EM (50.20% vs. 50.16%). However, in the separate four-generation behavior evaluation (Figure 7, right), C³ yields fewer query loops (69/206,852 = 0.0334% vs. 147/206,852 = 0.0711%) and behavior-union violations (90/206,852 = 0.0435% vs. 218/206,852 = 0.1054%). This supports the benefit of confidence guidance: confidence reflects the policy's preference for an action, not its correctness. Thus, stronger correction is applied when an undesirable behavior is more strongly reinforced by the current policy.
W2.5 & Q2 — What are the detector's precision and recall, especially for near-duplicate query loops? How sensitive are the results to the similarity threshold?
A: We thank the reviewer for raising this question. We conducted an LLM-assisted audit of 427 query-level items sampled across four policies and similarity strata. A single LLM reviewer first labeled the items blind to detector predictions; the reported estimates use the final adjudicated labels. The detector achieves 95% precision for identifying repeated information needs and 90% precision (95% CrI: 80–94%) for unnecessary repeated searches, with all 33 sampled exact-match triggers correctly identified. Recall is substantially lower because many semantic repetitions use different wording. The audit supports this qualitative gap, but does not provide a reliable population recall estimate.
We further examined the effect of varying the near-loop similarity threshold:
| Threshold | Flagged actions (est.) | Same-information precision | Unnecessary-repeat precision |
|---|---|---|---|
| ≥ 0.95 | ~212 | 96% | 96% |
| ≥ 0.90 (used) | ~532 | 95% | 90% |
| ≥ 0.85 | ~1,210 | 95% | 87% |
| ≥ 0.80 | ~2,684 | 95% | 81% |
Lower thresholds detect more potential repetitions but also introduce more false positives. The selected threshold prioritizes precision to avoid incorrectly suppressing useful behaviors. This analysis is based on fixed trajectories and does not measure training performance under different thresholds.
Thanks for the clarifications and I will maintain my scoring.
Thank you for taking the time to review our response and consider the additional experiments and clarifications. We sincerely appreciate your thoughtful feedback and constructive suggestions, which have helped us improve the manuscript.
Response to Reviewer L5TC(1)
We thank the reviewer for recognizing the value of our token-level audit, process-reward comparison, and lightweight integration. In response to the reviewer’s concerns, we have expanded the baseline evaluation with T²PO, added controlled ablations of confidence-guided correction, and conducted further analyses of detector reliability and the 2Wiki performance reversal.
W1 — The baseline coverage is thin: existing turn/step-level credit-assignment methods are not compared.
A: We thank the reviewer for pointing out the limited coverage of agent-specific process-supervision approaches. We have expanded the T²PO evaluation to Search QA and added Avg@4 evaluations for WebShop. T²PO is an agent-RL recipe featuring GiGPO-style credit assignment, and we evaluate it both with and without C³. This additional experiment tests whether C³ provides complementary benefits on top of an existing agent-specific credit assignment mechanism, rather than relying solely on the PPO/GDPO settings in our original evaluation.
The new comparisons are reported in Tables 1–2 of the revised manuscript, with the added setup and results highlighted in blue, particularly in Sections 5.1–5.2:
We appreciate that comparisons with additional agent-RL recipes such as Turn-PPO and StepSearch would provide further insights. In this revision, we have extended our evaluation with T²PO to broaden the coverage of agent-specific credit assignment methods. We also clarify that the process-reward PPO control examines a complementary intervention point: it applies event-level penalties before return and advantage estimation, whereas C³ operates on the resulting token-level advantages (Appendix B.2).
W2.1 & Q3 — The error bars come from sampling at test time. How many independent RL training runs underlie each number in the main table?
A: We thank the reviewer for raising this important point. Each experimental arm currently uses one independent training run. The reported error bars quantify variation across stochastic evaluations of a trained checkpoint, rather than variance across independent training runs. Due to limited computational resources and time during the rebuttal period, we were unfortunately unable to conduct additional independent training runs. We agree that multi-seed experiments would provide stronger evidence of training robustness.
Nevertheless, we have evaluated C³ across multiple agent-RL methods (PPO, GDPO, and T²PO) and different tasks, observing consistent improvements across these settings. These results provide complementary evidence across training methods and tasks. For completeness, Table 1 reports the PPO+C³ result of 50.20 ± 0.10, corresponding to a 2.41-point macro improvement over PPO, and Appendix B.3 details the evaluation samples and standard-deviation calculation.
W2.2 & Q3 — Why does 2Wiki improve by +11.5 under PPO but degrade under GDPO?
A: We thank the reviewer for highlighting this difference between PPO and GDPO. The opposite changes are concentrated in comparison questions, particularly those requiring yes/no answers. We break down 2Wiki performance by question type to identify which subsets drive the PPO gain and the GDPO decline.
Among the 1,295 yes/no questions, PPO+C³ improves EM from 26.72% to 82.01%, whereas GDPO+C³ reduces it from 48.73% to 20.25%. These changes contribute +5.69 and −2.93 percentage points, respectively, to overall 2Wiki EM. On the remaining questions, GDPO+C³ contributes a positive aggregate gain of +0.81 points.
Trajectory analysis further identifies answer-type selection as a key behavioral difference. Among all evaluated responses to the 1,295 yes/no questions, including responses without an answer, the fraction that contains an answer but not a standalone "yes" or "no" decreases from 63.06% to 6.72% with PPO+C³, but increases from 41.80% to 71.95% with GDPO+C³. For example, on the same nationality-comparison question, PPO changes its answer from "Germany" to "yes" after adding C³, whereas GDPO changes from "Yes" to "German." These observations suggest that the reversal is associated with how C³ affects answer-type selection under different optimization algorithms. A plausible explanation is that PPO and GDPO's different advantage estimation and optimization dynamics lead to different uses of the modified token-level learning signals.