acceptodds
Under review as a conference paper at ICLR 2027

PRICE the Action Chunks: Physical Relational Credit Assignment for Embodied Reinforcement Learning

Abstract

Outcome-based reinforcement learning (RL) post-trains vision–language–action policies using terminal success signals, but assigns the same trajectory-level advantage to every action chunk. A failed episode can thus penalize useful early actions as if they caused the failure. Existing approaches seek finer-grained feedback through learned evaluators, adding task-specific supervision or additional model training. We explore, for the first time to our knowledge, whether physical relations across trajectories can provide action-chunk credit in embodied RL from terminal outcomes alone, without an auxiliary evaluator. The key insight is that rollouts reaching corresponding physical situations can serve as references for one another: their terminal outcomes provide evidence for assessing local progress. We introduce Physical Relations for Inferring Credit from Episodes (PRICE), with two components: (i) a physical relational graph that pools current and historical outcomes at corresponding chunk boundaries to estimate success potentials; and (ii) confidence-gated credit assignment that uses changes in these potentials to refine trajectory-level supervision. Our analysis connects oracle potential changes to the terminal-success objective and provides a finite-sample directional bound for outcome-independent evidence pools. Independent continuation tests show that PRICE's retained credits align with local progress, while experiments on LIBERO, RoboTwin 2.0, and real robots demonstrate improved task success over outcome-based baselines and faster learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_467mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRICE, a way to assign action-chunk-level credit in GRPO-style RL post-training of VLAs using only terminal success. Chunk boundaries are matched across rollouts by clustering frozen visual and proprioceptive features. Each node’s success rate is estimated from current and past rollouts, and the change in success rate across a chunk, kept only when a Hoeffding-based gate deems it reliable, is added to the GRPO advantage.

Critical strengths

The problem is relevant: outcome-based GRPO for VLAs gives every chunk the same advantage, and PRICE addresses this without training an extra reward or value model. The method is simple and plugs into existing GRPO pipelines, and the experiments cover two simulated benchmarks, a world-action-model backbone, and two real-robot tasks. The paper also goes beyond task success by checking whether the assigned credit agrees with independent Monte Carlo estimates of progress, and it is upfront about the limits of its theoretical guarantee.

Critical weaknesses
  1. The technical novelty is limited. At its core, the method is GiGPO/SALT-style cross-trajectory step credit added to the GRPO advantage, with node values given by a non-parametric MC table over a frozen embedding. This is close to episodic control (NEC) and graph-based value memories (VMG, ERLAM), none of which are discussed. Theorem 1 is the standard unbiasedness of the one-step TD residual (as in GAE) and should not be presented as a contribution. Matching continuous states and gating graph-based credit by confidence also already appear in works the paper cites (BiPACE, Gated-BEPO). What remains is the choice of descriptor, the reuse of past rounds, and a Hoeffding gate. These are engineering refinements rather than a new idea, and the “first” claim in the abstract is overstated.

  2. It is unclear where the improvement comes from. Only a small fraction of chunks receive nonzero credit, and its magnitude is small relative to the GRPO advantage, so it is not obvious how such a sparse signal produces the reported gains. The ablations do not resolve this. By the paper’s own analysis, the “w/o History” variant reduces to GRPO, and the comparisons are from single runs. Learned-critic and GiGPO-style baselines under the same setup would make the picture much clearer.

  3. The credit evaluation does not test the failure mode that matters most. The MC check restores states from the same evidence pools used to compute the credit, so it only measures sampling noise. It cannot reveal whether a query trajectory was actually in the same physical situation as the pooled states. That is the central concern in the embodied setting, and one the paper itself emphasizes. Running continuations from each trajectory’s own states would be a more direct test.

Questions for authors
  1. How much of the improvement comes from groups where all rollouts succeed or all fail, where GRPO gives no gradient? How does PRICE compare to simply filtering or resampling those groups?
  2. How does PRICE compare to a learned critic (PPO with GAE) and to a GiGPO-style baseline under the same setup?
  3. Does the credit still agree with MC progress if continuations are run from each trajectory’s own states rather than from the pooled evidence states?
Reviewer_nBSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRICE adds action-chunk credit to outcome-based GRPO for post-training vision-language-action policies, without training an extra model. Two main parts. It builds a graph by matching visual and proprioceptive features across current and past rollouts to estimate node success rates, and then uses a confidence gate to add the potential difference to the GRPO advantage. The Authors experiment on LIBERO, RoboTwin 2.0, and two real-robot tasks using OpenVLA-OFT and OpenPI backbones.

Critical strengths
  1. Problem definition is clear, and the motivation for fixing trajectory-level credit in GRPO is easy to understand.
  2. The approach avoids training an extra reward or critic network, and the authors show it works across multiple policy types, including autoregressive VLAs, flow-matching policies, and world-action models.
  3. The authors provide strong diagnostic insights into local credit quality by using approximately 12,000 independent Monte Carlo continuation rollouts to verify sign agreement and rank correlation against true continuation progress.
Critical weaknesses
  1. In Tables 9 and 12, only around 3.8% of the action chunks actually receive non-zero credit, and it is even worse during the online RL training in Table 7 (only 2–4% of chunks). As 96–98% of chunks get zero credit, this is really sparse credit, so the paper's framing on "Dense credit & Simplicity" in Figure 1 does not hold.
  2. In Section 3.2 and Table 3, the authors argue that the confidence gate is essential and removing it hurts performance. But in the real-robot evaluation, the confidence gate was absent and raw contrasts were used instead. Also how many seeds were run for the real-robot experiments in Figure 4?
  3. The comparisons in Table 1 and Table 2 do not fully hold because the methods seem to be compared under different settings and inputs. For example, in Table 1, PRICE is listed as using only T+I inputs, but Equation 5 shows it actually uses robot proprioception during training to build the graph.
  4. There is no vanilla GRPO baseline run on the exact same setup across all suites in Table 1 and Table 2.
  5. Cosine similarity is very simplistic, so taking that over global visual embeddings and proprioception can easily merge states where the robot arm is in the same pose, but the object has slightly slipped in the gripper.
Questions for authors
  1. How can the authors justify the "dense credit" claim in Figure 1 when Tables 7, 9, and 12 show that 96–98% of action chunks receive zero credit?
  2. Why was the confidence gate dropped in the real-robot experiments (Appendix F.3), what is the performance if the gate is kept on, and how many seeds were run for Figure 4?
  3. Can the authors provide a controlled vanilla GRPO baseline under the exact same checkpoints and inputs across all tasks in Table 1 and Table 2, especially since PRICE uses proprioception during training?
  4. How does the cosine similarity matching in Equation 5 avoid merging states where the robot arm pose is identical but the object has slightly slipped in the gripper?
Reviewer_Y3DzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PRICE, a method that gives short sequences of robot actions additional training credit according to whether they increase the estimated chance of eventual task success, using the outcomes of other attempts that passed through similar observations rather than a separately trained reward or value model. The robot policies are trained with feedback indicating whether the overall attempt succeeded. The authors argue that assigning the same training credit throughout an attempt misses useful distinctions: a robot might grasp and transport an object correctly but fail during placement. Intermediate rewards or learned reward, value, and world models can provide more detailed feedback, but require task-specific reward design or an additional trained evaluator. PRICE instead uses the final outcomes of other attempts that passed through similar visual and physical situations to estimate how the chance of eventual success changes across each action sequence. It adds this estimated progress to the existing trajectory-level training signal, while retaining the terminal reward. The motivation is therefore to extract more useful local feedback from collected experience without training an additional evaluator, rather than to show that terminal rewards or learned evaluators are fundamentally inadequate. The core question is whether an action sequence leaves the robot in a more promising situation. Consider a grasp: PRICE separately finds other attempts resembling the situation before the grasp and the situation afterward, then checks how often each group eventually succeeded. If success was more common in the latter group, it gives the grasp positive local credit, even if this particular attempt later failed during placement. A lower estimated success rate afterward gives negative credit. The basic operation is to group similar observations, count eventual successes, and subtract the before estimate from the after estimate; it does not train an additional model to predict future outcomes. The camera features come from a frozen copy of the image encoder in the robot policy's initial supervised-trained checkpoint; measurements of the robot's physical configuration are normalized with fixed checkpoint statistics. This keeps the representation used for matching stable as the acting policy changes. PRICE therefore reuses learned visual features, but does not train an additional evaluator. The attempt being judged is excluded from its own evidence, repeated visits within a reference attempt are not counted repeatedly, and historical data are screened for policy changes. If the statistical test finds the direction too uncertain, no local credit is added. The difficult assumption is that observations judged similar really describe situations with comparable prospects of success. A confident estimate for a mismatched group can still be misleading for the particular action receiving credit.

PRICE estimates local credit by combining current experience with historical outcomes collected before policy updates. Historical data are retained only when a measure of policy change stays below a threshold. A confidence gate emits local credit only when the estimated difference has an unambiguous sign under its uncertainty interval; otherwise the local contribution is suppressed. This integrates existing value-difference credit ideas without a separately trained evaluator or hand-labeled task stages. The mathematics specifies the success estimates, uncertainty filter, and training update, and explains why changes in true success probabilities would provide useful training credit. A separate result bounds the chance of adding credit with the wrong sign relative to the reference groups' true averages, under assumptions that include selecting those groups independently of terminal outcomes. These are useful idealized justifications, but the method's basic operation can be understood without the proofs. The theory does not establish that groups formed from camera features and robot measurements accurately represent the particular situation being judged, nor does it guarantee the full adaptive training procedure. Whether matching provides useful local credit remains an empirical question. This is a limit on what the mathematics establishes, not evidence that the mathematical results are incorrect.

The strongest simulation evidence is the controlled comparison with GRPO, a baseline that gives each action the training credit assigned to its whole attempt. In the controlled comparison on LIBERO-Spatial and LIBERO-Object, the robot policies start from the same model and receive the same environment-interaction and optimization budgets; adding PRICE improves task success on both suites. Experiments also cover selected RoboTwin tasks involving grasping, lifting, transport, placement, and handover. These comparisons support practical usefulness more directly than a ranking against separately published results. Gains extend beyond the main OpenVLA-OFT robot policy architecture: a controlled Fast-WAM comparison also favors PRICE. Training-efficiency comparisons with SimpleVLA-RL, an existing framework for outcome-based GRPO training of robot policies, reach success milestones with less training and computation. This breadth makes the result less dependent on a particular setup, without establishing general superiority across robotics. Real-robot insertion and sorting use advantage-weighted regression (AWR): train on recorded actions, giving more weight to those judged favorable. PRICE-AWR adds local credit to those weights, whereas GRPO-AWR uses whole-attempt credit. With the same recorded data and optimization budget, PRICE-AWR performs better. No new robot experience is collected during this adaptation, and it omits the confidence gate, so the hardware result supports the local-credit idea rather than every part of the online method. Fresh continuations test whether retained credits agree with eventual-success estimates for their matched reference groups. The confidence gate compares favorably with selecting the same quantity of credits using only evidence counts. These checks support the estimator on its reference groups, but do not establish that it correctly describes progress at the particular action being judged. Main PRICE results include repeated training runs, but uncertainty is not reported clearly for every baseline and component comparison. Small incremental gains therefore deserve more caution. Overall, the experiments support a useful training modification more strongly than they establish accurate credit for individual actions.

Critical strengths

The controlled simulation comparisons support a practical benefit: PRICE improves success over whole-attempt GRPO training while holding the starting model, interaction budget, and optimization settings fixed. Improvement with another robot policy architecture and the fixed-data hardware adaptation provides evidence beyond a single setup. The hardware version omits the gate, so I credit it as support for local feedback, not independent validation of the full online method.

The authors examine the proposed feedback rather than relying only on final task scores. They collect fresh continuations from matched reference situations and compare credit-selection rules while holding the quantity of selected credits fixed. These tests support the confidence gate on the reference groups. Their explicit separation of sampling uncertainty, state-matching error, and historical-policy mismatch makes the remaining limitations identifiable.

Critical weaknesses

The authors check PRICE's estimates by restarting the robot from situations used as reference examples and measuring how often it subsequently succeeds. This tests success rates in those reference situations. It does not directly test whether the particular action sequence receiving credit improved the robot's chance of success: its starting and ending situations may differ in consequential ways from the examples PRICE matched them to. For example, similar camera images and robot measurements need not imply the same prospects of completing the task. When testing historical reference examples, the authors also use the policies that originally collected them, which can differ from the policy that produced the action sequence being credited. Agreement with the reference examples therefore does not resolve errors caused by mismatched situations or changes in policy.

This limits the explanation of why the local credit works; it does not invalidate the reported task-success gains. A separate qualification is repeatability: the main PRICE results include independent training runs, but uncertainty for each baseline and component comparison is not clearly reported. I therefore place more weight on the controlled overall improvement and its support across setups than on small component gains. I lean toward acceptance for a useful training modification, without treating the experiments as proof of accurate local credit at each action.

Questions for authors
  1. Is there an existing check that continues from the credited action sequence's actual before/after histories using the same frozen policy that collected that attempt? Each starting point should retain its own remaining episode allowance and policy inputs. This would test whether the assigned credit agrees in sign with the actual change in success probability, rather than with reference-pool averages.

  2. Section 4.1 reports independent training repeats for the main PRICE results. Were GRPO and every Table 3 variant also retrained, and what uncertainty across these repeats is available for the PRICE–GRPO improvement?

Comments on the paper

7 comments

Thank you for the detailed rebuttal and the substantial new experiments. The PPO+GAE and GiGPO comparisons, the group-type ablations, and the query-level MC validation address my Q1–Q3 well and strengthen the empirical side of the paper.

That said, two main concerns remain.

  1. For conceptual novelty: the core pipeline of cross-trajectory state aggregation, graph-based progress estimation, and local credit assignment has been explored in prior work, particularly GraphGPO (ICML 2026), GiGPO, and SALT. GraphGPO, for example, already builds a unified state-transition graph across rollouts and combines graph-derived local advantages with episode-level supervision, and SALT already merges continuous states across trajectories by embedding similarity. PRICE uses a different progress estimator, but its main extensions appear to be approximate physical-state matching, historical outcome pooling, and confidence gating. While the new experiments show that these designs are effective, the distinct algorithmic contribution is still not clearly established. A more explicit method-level comparison, especially with GraphGPO, and controlled baselines that adapt related graph-based credit assignment methods to the same VLA setting would help clarify both the novelty and the empirical benefit of PRICE. The authors may also wish to discuss recent concurrent work such as G2PO.

  2. Validation of physical state correspondence is somewhat insufficient. The paper identifies physical correspondence across continuous robotic states as a central challenge in extending cross-trajectory credit assignment to embodied RL. However, the proposed approach relies on frozen visual-proprioceptive features and cosine-similarity-based matching, which does not necessarily preserve equivalence in continuation-success probabilities. States with similar observations and robot configurations may differ in contact conditions, object stability, or interaction history, resulting in substantially different future outcomes. Importantly, confidence gating addresses sampling uncertainty in the pooled estimates but cannot directly correct systematic errors caused by inaccurate state matching.

The paper’s own results apper to point to this. In the micro-slip diagnostic, some pairs with clearly different continuation values are merged (Table 13). During online training, only few of audited chunks receive nonzero credit (Table 10). On the real robot, the default gate produces no credit on either task, so the reported gains may come from ungated credit (Appendix F.4). The query-level MC results are encouraging, but it remains unclear how well PRICE works when matching is imperfect or reference data is scarce. Experiments that vary state overlap and the amount of reference data, ideally compared against matching on privileged states, would help clarify this. It would also help to show qualitative examples of success/failure cases, such as the merged pairs with different continuation values, credited chunks whose sign disagrees with MC progress, and samples of states grouped into the same node, to illustrate when the matching breaks down.

AuthorsOpenReview ↗

Response to Reviewer 467m: GraphGPO Comparison and Physical State Correspondence Validation (1/2)

Thank you for taking the time to review our revisions and for the opportunity to further clarify and strengthen our work. All manuscript revisions remain highlighted in red.

Q1. Empirical comparison with GraphGPO

We added GraphGPO to the same VLA framework. Success rates (%) at matched cumulative GPU-hour budgets are:

Cumulative budget (GPU hours)PPO+GAEGiGPO-styleGraphGPOPRICE
20063.481.077.482.4
40082.887.884.689.8
60092.090.888.096.0
81896.695.894.699.6

PRICE achieves the highest observed success at every budget, outperforming GraphGPO and GiGPO-style by 5.0 and 3.8 pp, respectively, at 818 GPU hours. To connect these training gains with credit quality, we progressively introduce PRICE's components into GraphGPO:

Credit variantDirAcc (%)RankCorrAlignedGap (pp)
GraphGPO57.500.101+4.41
+ PRICE current success-rate calibration71.740.248+7.56
+ Historical evidence73.910.378+13.50
+ Adapted confidence gating80.950.447+17.86
Full PRICE (separate reference)97.500.709+28.47

PRICE calibration using current success rates raises GraphGPO's DirAcc from 57.50% to 71.74%. Historical evidence and adapted gating further improve it to 73.91% and 80.95%, respectively, with RankCorr reaching 0.447 and AlignedGap reaching +17.86 pp. The earlier GiGPO-style comparison shows DirAcc increasing from 60.00% to 79.55% with successor-value estimation, then to 82.98% with historical evidence and 85.00% with adapted gating. These gains within both existing advantage frameworks support PRICE's contribution: using pooled outcome evidence and its uncertainty to construct local credit that better tracks continuation progress, consistent with the training improvements above (see Appendix G.6, Tables 19–21).

Novelty and contribution

Our contribution is a learning mechanism that turns physical experience across attempts into reusable local supervision for continuous action chunks. In manipulation, different action sequences can reach comparable physical situations, while identical terminal outcomes can conceal different local progress. PRICE uses these correspondences to assess how an action changes the prospects of eventual success, without training an auxiliary evaluator.

The distinction from existing cross-trajectory methods lies in how this physical experience is converted into local supervision. GiGPO compares realized returns at shared anchors, leaving an action's credit tied to its trajectory's eventual outcome. SALT reduces conflicting supervision by averaging trajectory-level advantages over matched steps. GraphGPO derives local advantages from shortest-path graph returns, which capture routes to success but do not directly estimate continuation-success probabilities. PRICE evaluates a chunk through the estimated change in success prospects between its physical endpoints. Pooling outcomes from other trajectories lets it assess local progress separately from the query episode's final outcome, without requiring identical action chunks. This is valuable in manipulation, where useful intermediate behavior can be obscured by a later failure.

PRICE also changes the evidence available for this assessment. Concurrent G2PO shares the use of aggregated values and endpoint differences, but estimates them within the sampled group. PRICE extends reference evidence across same-task groups and eligible training rounds. A current group can therefore receive informative local supervision even when its own outcomes lack variation, provided past experience distinguishes the chunk endpoints. This broadens what the policy can learn from already collected interaction, an important capability when robot experience is costly.

Making this accumulated evidence useful also requires deciding when it should influence a policy update. PRICE manages historical reuse as the policy evolves and selects endpoint differences according to their sampling uncertainty. Whereas Gated-BEPO activates credit based on visit and branching support, PRICE evaluates the evidence for the particular progress contrast being credited. Evidence accumulation and selection thus work together to determine both the availability and use of local supervision.

Our experiments substantiate this mechanism. The group-type ablations show that accumulated evidence helps recover missing supervision and refine existing feedback. Component-transfer and equal-coverage comparisons support the specific estimator and selection rules, while continuations from the query's own histories examine whether the resulting credit tracks progress where it is applied. The controlled training comparisons demonstrate the practical learning value of this supervision in the evaluated embodied tasks.

AuthorsOpenReview ↗

Response to Reviewer 467m: GraphGPO Comparison and Physical State Correspondence Validation (2/2)

Q2. Physical state correspondence

We agree that observation similarity does not guarantee equivalent continuation values, and that confidence gating addresses sampling uncertainty but cannot directly correct systematic correspondence errors. We therefore evaluate credit against independent continuations from each query's own before/after history and explicitly analyze residual matching failures.

We first evaluate credit under reduced reference support and state overlap, including privileged matching, then examine learning benefits and qualitative success and failure cases.

Reference amount and state overlap

We evaluate 160 fixed queries across all LIBERO-Spatial tasks: 80 credited by full-reference PRICE and 80 additional manipulation-related queries. We run 64 independent continuations from each query's actual before/after history under its frozen collection policy, preserving the remaining horizon (20,480 continuations; Appendix E.3).

We first reduce the historical references:

Historical references retainedPRICE coverage (%)PRICE DirAcc (%)Privileged coverage (%)Privileged DirAcc (%)PRICE AlignedGap (pp)
100%2.8490.63.0593.8+20.0
50%1.7689.31.9192.0+17.5
25%0.9287.51.0290.0+14.0

With one quarter of the references, DirAcc decreases by 3.1 pp, and AlignedGap retains 70% of its full-reference value. Fewer references reduce credit coverage, but the retained feedback remains aligned with progress at the queries themselves.

We then reduce the near-reference fraction while keeping the reference count fixed and the outcome composition comparable:

Near-reference fractionPRICE coverage (%)PRICE DirAcc (%)Privileged coverage (%)Privileged DirAcc (%)PRICE AlignedGap (pp)
80%1.9092.32.0694.4+20.0
50%1.4090.01.5492.3+17.0
20%0.7586.70.8690.0+12.0

At the lowest overlap, retained credit still achieves 86.7% DirAcc and +12.0 pp AlignedGap. Both tests show that reduced reference availability mainly limits how often PRICE supplies local feedback, while retained credit continues to track query-level progress.

Privileged matching and correspondence errors

We compare the original matcher with a privileged descriptor containing object and end-effector poses, relative motion, gripper state, and finger contacts, keeping the PRICE estimator and gate fixed. Across both reference tests, privileged matching improves query-level DirAcc by 2.1–3.3 pp, while PRICE retains 87.2%–93.1% of its coverage. This limited observed gap, together with PRICE's positive query-level AlignedGap across conditions, supports the usefulness of standard observations for local supervision. With fewer references or lower overlap, PRICE credits fewer action chunks, but the retained credits still agree well with query-level MC progress.

Under the standard training setup, ungated PRICE improves over GRPO, showing that feedback derived from approximate physical correspondence has practical learning value. Confidence gating further improves online training (Appendix G.1). These results complement the query-level evaluations by connecting the usefulness of the estimated credit to policy improvement.

Real-robot adaptation without gating

In the real-robot setting, we use ungated PRICE-AWR with a fixed buffer of 60 trajectories per task. Applying the default online gate to these buffers produces no credit (Appendix F.3–F.4).

With matched data, initialization, and adaptation budgets, three seeds give (Appendix G.1, Table 7):

MethodRemote Insertion (%)Produce Sorting (%)
GRPO-AWR42.5 ± 5.052.5 ± 2.5
Ungated PRICE-AWR58.3 ± 3.869.2 ± 2.9

The 15.8 and 16.7 pp gains support the usefulness of relational credit on real hardware, including with limited fixed data.

Qualitative examples of state sharing and local feedback

Following your suggestion, we added visualizations of shared-node states and action-chunk feedback, together with MC case analyses (Appendix I.3, Figures 9–11). These examples show how comparable physical configurations allow different attempts to share outcome evidence, and how this evidence distinguishes local progress from setbacks. In particular, useful actions within failed episodes can still receive positive local credit, illustrating the additional supervision that PRICE extracts from terminally labeled experience.

We also examine states merged despite different continuation values and credits whose directions disagree with MC progress. These cases identify contact distinctions that matching can miss and explain their effects on local feedback. The qualitative analysis makes PRICE's experience-sharing mechanism concrete while clarifying the physical conditions under which correspondence errors can arise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.