- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoRLA, an uncertainty-aware framework that calibrates teacher-generated process labels. It constrains return-based potential learning using reward intervals, and combines uncertainty-gated temporal credit with selective hindsight distillation.
Critical strengths
-
The paper addresses a meaningful challenge in long-horizon agent RL: local next-state feedback can provide dense supervision, but its reliability and relationship to eventual task success are uncertain.
-
CoRLA combines teacher-calibrated reward intervals, return-based potential learning, uncertainty-gated credit assignment, and selective hindsight-guided on-policy distillation. This integration provides a structured framework for utilizing noisy intermediate feedback.
-
This paper provide promising empirical improvements and relatively comprehensive evaluation. Beyond downstream performance, the paper includes component ablations, reward-noise experiments, trajectory-length analysis, and counterfactual action-replacement evaluations.
Critical weaknesses
W1. Insufficient justification for using teacher-calibrated process rewards to constrain long-horizon value estimates.
CoRLA assumes that local process reward intervals provide meaningful constraints on discounted potential differences. However, these quantities are not necessarily semantically aligned. A locally unsuccessful action may provide valuable diagnostic information.
The paper does not sufficiently establish when or why the proposed interval constraints improve long-horizon credit estimation (although mention in section 2.2). Section 3.5 evaluates the correlation between the final credit estimates and counterfactual return changes, it does not directly validate whether teacher rewards are aligned with long-term contributions or whether narrower intervals indicate more reliable credit estimates.
W2. Limited validation of teacher quality and the reliability of uncertainty-based supervision.
The framework relies on a frozen Qwen3-4B teacher for both process labels and hindsight hints, including when training a Qwen3-8B terminal agent. Although judging observed transitions may be easier than solving the complete task. However, the reliability of this teacher on complex agent trajectories is not sufficiently established. More importantly, conformal calibration measures predictive agreement with teacher labels. A systematically incorrect but consistent teacher may produce narrow intervals, which could influence both potential regularization and credit gating.
Questions for authors
Q1. Alignment between local process rewards and long-horizon contributions (W1).
Section 3.5 already estimates counterfactual return changes through action replacement. Could the authors use these measurements to report:
- The correlation between raw teacher process labels and counterfactual return changes?
- The correlation between predicted process rewards and counterfactual return changes?
- Whether narrower conformal intervals correspond to more accurate credit estimates, for example through credit-direction error rates stratified by interval width?
I think these would directly test whether the calibrated local supervision is informative about long-horizon action contributions.
Q2. Necessity of conformal constraints and the uncertainty gate (W1).
Could the authors report a potential-learning ablation with , keeping the terminal-return supervision and other training components unchanged? And, since the proposed gate uses teacher-label uncertainty rather than potential uncertainty, what evidence supports using this uncertainty to weight potential differences? Have the authors considered estimating uncertainty directly from the potential model?
Q3. Reliability of the frozen teacher and hindsight hints (W2).
What empirical evidence supports using Qwen3-4B to generate reliable process labels and corrective hints for the evaluated agent tasks? Could the authors provide an analysis of teacher-label agreement and examine whether performance changes with a stronger teacher? For selective OPD, does lower process-label uncertainty correlate with higher hint correctness or usefulness?
Q4. Training cost. It would also be helpful to report the additional teacher-query and training costs relative to OpenClaw-RL.
Thank you for the detailed response. The revision adds a matched-compute PPO + critic + GAE baseline (Table H.3) and an ablation without the interval penalty (Table H.1), and it corrects Figure 3. Following the paper, the code now uses separate judge and potential adapters and samples calibration probes uniformly first. However, the central question I raised in W3, whether the results were obtained as the paper describes, remains open, so I am keeping my score at 2 for the following reasons.
I would be glad to reconsider if the authors provide per-seed results and runnable task code, and state which code version produced the main results.