- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PRICE, a way to assign action-chunk-level credit in GRPO-style RL post-training of VLAs using only terminal success. Chunk boundaries are matched across rollouts by clustering frozen visual and proprioceptive features. Each node’s success rate is estimated from current and past rollouts, and the change in success rate across a chunk, kept only when a Hoeffding-based gate deems it reliable, is added to the GRPO advantage.
Critical strengths
The problem is relevant: outcome-based GRPO for VLAs gives every chunk the same advantage, and PRICE addresses this without training an extra reward or value model. The method is simple and plugs into existing GRPO pipelines, and the experiments cover two simulated benchmarks, a world-action-model backbone, and two real-robot tasks. The paper also goes beyond task success by checking whether the assigned credit agrees with independent Monte Carlo estimates of progress, and it is upfront about the limits of its theoretical guarantee.
Critical weaknesses
-
The technical novelty is limited. At its core, the method is GiGPO/SALT-style cross-trajectory step credit added to the GRPO advantage, with node values given by a non-parametric MC table over a frozen embedding. This is close to episodic control (NEC) and graph-based value memories (VMG, ERLAM), none of which are discussed. Theorem 1 is the standard unbiasedness of the one-step TD residual (as in GAE) and should not be presented as a contribution. Matching continuous states and gating graph-based credit by confidence also already appear in works the paper cites (BiPACE, Gated-BEPO). What remains is the choice of descriptor, the reuse of past rounds, and a Hoeffding gate. These are engineering refinements rather than a new idea, and the “first” claim in the abstract is overstated.
-
It is unclear where the improvement comes from. Only a small fraction of chunks receive nonzero credit, and its magnitude is small relative to the GRPO advantage, so it is not obvious how such a sparse signal produces the reported gains. The ablations do not resolve this. By the paper’s own analysis, the “w/o History” variant reduces to GRPO, and the comparisons are from single runs. Learned-critic and GiGPO-style baselines under the same setup would make the picture much clearer.
-
The credit evaluation does not test the failure mode that matters most. The MC check restores states from the same evidence pools used to compute the credit, so it only measures sampling noise. It cannot reveal whether a query trajectory was actually in the same physical situation as the pooled states. That is the central concern in the embodied setting, and one the paper itself emphasizes. Running continuations from each trajectory’s own states would be a more direct test.
Questions for authors
- How much of the improvement comes from groups where all rollouts succeed or all fail, where GRPO gives no gradient? How does PRICE compare to simply filtering or resampling those groups?
- How does PRICE compare to a learned critic (PPO with GAE) and to a GiGPO-style baseline under the same setup?
- Does the credit still agree with MC progress if continuations are run from each trajectory’s own states rather than from the pooled evidence states?
Thank you for the detailed rebuttal and the substantial new experiments. The PPO+GAE and GiGPO comparisons, the group-type ablations, and the query-level MC validation address my Q1–Q3 well and strengthen the empirical side of the paper.
That said, two main concerns remain.
For conceptual novelty: the core pipeline of cross-trajectory state aggregation, graph-based progress estimation, and local credit assignment has been explored in prior work, particularly GraphGPO (ICML 2026), GiGPO, and SALT. GraphGPO, for example, already builds a unified state-transition graph across rollouts and combines graph-derived local advantages with episode-level supervision, and SALT already merges continuous states across trajectories by embedding similarity. PRICE uses a different progress estimator, but its main extensions appear to be approximate physical-state matching, historical outcome pooling, and confidence gating. While the new experiments show that these designs are effective, the distinct algorithmic contribution is still not clearly established. A more explicit method-level comparison, especially with GraphGPO, and controlled baselines that adapt related graph-based credit assignment methods to the same VLA setting would help clarify both the novelty and the empirical benefit of PRICE. The authors may also wish to discuss recent concurrent work such as G2PO.
Validation of physical state correspondence is somewhat insufficient. The paper identifies physical correspondence across continuous robotic states as a central challenge in extending cross-trajectory credit assignment to embodied RL. However, the proposed approach relies on frozen visual-proprioceptive features and cosine-similarity-based matching, which does not necessarily preserve equivalence in continuation-success probabilities. States with similar observations and robot configurations may differ in contact conditions, object stability, or interaction history, resulting in substantially different future outcomes. Importantly, confidence gating addresses sampling uncertainty in the pooled estimates but cannot directly correct systematic errors caused by inaccurate state matching.
The paper’s own results apper to point to this. In the micro-slip diagnostic, some pairs with clearly different continuation values are merged (Table 13). During online training, only few of audited chunks receive nonzero credit (Table 10). On the real robot, the default gate produces no credit on either task, so the reported gains may come from ungated credit (Appendix F.4). The query-level MC results are encouraging, but it remains unclear how well PRICE works when matching is imperfect or reference data is scarce. Experiments that vary state overlap and the amount of reference data, ideally compared against matching on privileged states, would help clarify this. It would also help to show qualitative examples of success/failure cases, such as the merged pairs with different continuation values, credited chunks whose sign disagrees with MC progress, and samples of states grouped into the same node, to illustrate when the matching breaks down.