acceptodds
Desk rejected at ICLR 2027

CoRLA: From Next-State Feedback to Long-Horizon Credit for LLM Agent Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Next-state feedback provides dense supervision for large language model agents, but noisy local evaluations may not reflect an action's contribution to eventual task success. We propose CoRLA (Conformal Reward and Long-horizon Attribution), an uncertainty-aware reinforcement learning framework that connects reliable reward inference with long-horizon credit assignment. CoRLA uses conformal prediction to construct reward intervals from labels supplied by a frozen LLM teacher, updating calibration online as the policy evolves. The framework learns a potential from terminal returns under these interval constraints and propagates uncertainty-gated potential differences to assign long-horizon credit. Selective on-policy distillation adds complementary token-level supervision through hindsight corrections from low-uncertainty transitions. Policy optimization integrates local progress, long-horizon credit, and token-level supervision. Experiments show that CoRLA exceeds OpenClaw-RL by 11.2 percentage points in Terminal Bench 2.0 success rate, 8.2 points in SWE-bench Verified resolved rate, and 10.0 points in AIME 24/25 accuracy. Compared with local process rewards, CoRLA's credits also align more closely with return changes under sampled action replacements.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Long-horizon Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fZkiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoRLA, an uncertainty-aware framework that calibrates teacher-generated process labels. It constrains return-based potential learning using reward intervals, and combines uncertainty-gated temporal credit with selective hindsight distillation.

Critical strengths
  1. The paper addresses a meaningful challenge in long-horizon agent RL: local next-state feedback can provide dense supervision, but its reliability and relationship to eventual task success are uncertain.

  2. CoRLA combines teacher-calibrated reward intervals, return-based potential learning, uncertainty-gated credit assignment, and selective hindsight-guided on-policy distillation. This integration provides a structured framework for utilizing noisy intermediate feedback.

  3. This paper provide promising empirical improvements and relatively comprehensive evaluation. Beyond downstream performance, the paper includes component ablations, reward-noise experiments, trajectory-length analysis, and counterfactual action-replacement evaluations.

Critical weaknesses

W1. Insufficient justification for using teacher-calibrated process rewards to constrain long-horizon value estimates.

CoRLA assumes that local process reward intervals provide meaningful constraints on discounted potential differences. However, these quantities are not necessarily semantically aligned. A locally unsuccessful action may provide valuable diagnostic information.

The paper does not sufficiently establish when or why the proposed interval constraints improve long-horizon credit estimation (although mention in section 2.2). Section 3.5 evaluates the correlation between the final credit estimates and counterfactual return changes, it does not directly validate whether teacher rewards are aligned with long-term contributions or whether narrower intervals indicate more reliable credit estimates.

W2. Limited validation of teacher quality and the reliability of uncertainty-based supervision.

The framework relies on a frozen Qwen3-4B teacher for both process labels and hindsight hints, including when training a Qwen3-8B terminal agent. Although judging observed transitions may be easier than solving the complete task. However, the reliability of this teacher on complex agent trajectories is not sufficiently established. More importantly, conformal calibration measures predictive agreement with teacher labels. A systematically incorrect but consistent teacher may produce narrow intervals, which could influence both potential regularization and credit gating.

Questions for authors

Q1. Alignment between local process rewards and long-horizon contributions (W1).

Section 3.5 already estimates counterfactual return changes through action replacement. Could the authors use these measurements to report:

  1. The correlation between raw teacher process labels and counterfactual return changes?
  2. The correlation between predicted process rewards and counterfactual return changes?
  3. Whether narrower conformal intervals correspond to more accurate credit estimates, for example through credit-direction error rates stratified by interval width?

I think these would directly test whether the calibrated local supervision is informative about long-horizon action contributions.

Q2. Necessity of conformal constraints and the uncertainty gate (W1).

Could the authors report a potential-learning ablation with λCP=0\lambda_{CP}=0, keeping the terminal-return supervision and other training components unchanged? And, since the proposed gate uses teacher-label uncertainty rather than potential uncertainty, what evidence supports using this uncertainty to weight potential differences? Have the authors considered estimating uncertainty directly from the potential model?

Q3. Reliability of the frozen teacher and hindsight hints (W2).

What empirical evidence supports using Qwen3-4B to generate reliable process labels and corrective hints for the evaluated agent tasks? Could the authors provide an analysis of teacher-label agreement and examine whether performance changes with a stronger teacher? For selective OPD, does lower process-label uncertainty correlate with higher hint correctness or usefulness?

Q4. Training cost. It would also be helpful to report the additional teacher-query and training costs relative to OpenClaw-RL.

Reviewer_t1DBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The central problem that this paper focus on is the noisy next-state evaluations in LLM RL methods like OpenClaw-RL. It introduces a framework based on conformal prediction, more specifically using conformal interval as gates for (1) long horizon credit assignment and (2) selective on-policy distillation. Empirical results on 4 tasks show improvement over the representative next-state evaluation method OpenClaw-RL. Ablations and analysis also justify the design choice, showcase the method's robustness and the method's efficacy on long-horizon problems and credit attribution.

Critical strengths
  1. The uncertainty-aware training framework the paper introduced is sound and has strong empirical performance
  2. Analysis on length, credit attribution and robustness justifies the introduction of the framework
Critical weaknesses
  1. The paper has already done some ablations. However, the design is way more complex since we have hybrid RL (2 components), multiple components in the introduced uncertainty framework, and hyper parameters like βΦ\beta_{\Phi} and λOPD\lambda_{OPD} etc.. Is it possible to conduct finer-grained ablations for a better understanding of the framework?
  2. Some more preliminaries on conformal prediction may be helpful for readers who are not familiar.
Questions for authors
  1. What's the conformal reward mentioned in the ablation study? Is it \mu_t or something else?
Reviewer_8o9pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoRLA uses next-state feedback to train LLM agents. Its judge predicts a frozen teacher's step labels along with an uncertainty scale. Online conformal calibration converts this scale to reward intervals. A potential is fitted to terminal returns, with a penalty pulling it towards these intervals. Differences in the potential between steps receive less weight when intervals are wide and are propagated through a GAE-style trace. Hindsight OPD is restricted to steps with narrow intervals. The reported gains over OpenClaw-RL are 11.2, 8.2 and 10.0/10.0 points on Terminal-Bench 2.0, SWE-bench Verified and AIME 2024/2025. Further experiments examine label noise, trajectory length, counterfactual credit and gate variants.

Critical strengths
  1. Weighting step-level feedback by its reliability makes sense for agent RL.
  2. The counterfactual test is useful. By replacing an action and rerunning the task, it checks whether the steps assigned high credit actually affect the outcome.
Critical weaknesses

I cannot tell how much the new components add over plain actor-critic. With all gates set to one, the credit is standard GAE with a learned critic (Sec. 2.2). It is unclear whether "ORM Only" includes a critic. If not, a value baseline alone could account for much of the "+Potential" gain. The remaining components are difficult to isolate too. The authors acknowledge a conflict between the interval penalty and the return-fitted potential. For CP Soft Gate, the only difference from Raw-σ Gate is a scalar shared within each round. A PPO + critic + GAE baseline with the same compute budget, together with a run without the interval penalty, would clarify what these components contribute.

The results in Section 3.4 seem inconsistent with the calculation described. Only the interval width changes across the three LS variants, while Appendix C assigns classes using the predicted mean alone. With a shared predictor, all three should have the same StepAcc. Multiplying uncertainty by a constant should also leave Noise AUROC unchanged. Under symmetric noise, however, the reported StepAcc increases from 67.4% to 70.1%, and AUROC from 0.68 to 0.79.

Several implementation details differ from the paper, leaving open whether the analyses were run as described.

In ualca_critic.py, the judge and potential share one adapter and one optimizer. Updating the potential therefore changes the judge as well. The paper specifies separate adapters.

The calibration procedure in the paper samples probes uniformly before selecting uncertain transitions. In ualca_training.py, these operations happen in reverse order. This biases the probes towards low-uncertainty steps and can produce intervals that are too narrow.

A training script is provided only for the Qwen3-4B OpenClaw setting. CoRLA code for terminal, SWE and tool-call tasks is absent.

Questions for authors

For "ORM + Point Reward" and "ORM + Conformal Reward", where is the judge's output used in the policy update? The predicted progress is absent from both the step credit (Eq. 9) and the policy loss (Eq. 14).

According to Table 2, every gate variant retains OPD. Yet No Gate scores 22.5 on Terminal-Bench and 33.3 on AIME 2024, matching the potential variant without OPD implied by Sec. 3.3. Does this mean OPD provides no gain when the gate is removed?

How much additional compute does CoRLA need over OpenClaw-RL once the extra critic GPU, teacher queries and hint-conditioned scoring passes are counted?

Did you reproduce the OpenClaw-RL results using your models and budgets? For AIME, is the reported score based on one sample or an average over several?

2 comments

Thank you for the detailed response. The revision adds a matched-compute PPO + critic + GAE baseline (Table H.3) and an ablation without the interval penalty (Table H.1), and it corrects Figure 3. Following the paper, the code now uses separate judge and potential adapters and samples calibration probes uniformly first. However, the central question I raised in W3, whether the results were obtained as the paper describes, remains open, so I am keeping my score at 2 for the following reasons.

  1. The key number for W1 does not match the evaluation protocol. Under the protocol in Appendix H (500 tasks, one attempt per task, mean over three seeds), every SWE-bench mean must be a multiple of 1/1500, but the 18.50% reported for PPO + critic + GAE in Table H.3 is not. This experiment is the main evidence that CoRLA improves on a plain actor-critic, yet the table reports no standard deviations and does not state how many rounds the baseline completed under the matched budget.
  2. Other numbers are also inconsistent with the described computation. In the original Figure 3, the three LS variants had different StepAcc and Noise AUROC values, although under the paper's computation both metrics should be identical across them. The authors call this a reporting error but do not explain where those values came from. In addition, the Q4 response states that the AIME scores are three-seed means. Such a mean is a multiple of 1/30 only when the three seeds' correct-answer counts sum to a multiple of 3 (roughly one case in three), yet every AIME score of the trained models in Table 1, Table 2, and Figure 2 is a multiple of 1/30, which is not quite consistent with that statement.

I would be glad to reconsider if the authors provide per-seed results and runnable task code, and state which code version produced the main results.

Thank you for the clarifications, which have resolved many of my concerns. Please include them in the final version, in particular the per-seed results and the AIME evaluation protocol. I hope the released code will reproduce the results as described in the paper. I have raised my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.