acceptodds
Under review as a conference paper at ICLR 2027

ESCAPO: Entropy-guided Segment-level Credit Assignment for Policy Optimization in VLA Post-Training

Abstract

Long-horizon vision-language-action (VLA) policies are increasingly post-trained with on-policy reinforcement learning using sparse terminal rewards. Critic-free methods such as Group Relative Policy Optimization (GRPO) derive a trajectory-level advantage from these outcomes, providing only coarse temporal credit: a failed rollout may contain useful intermediate behavior, yet every action receives the same advantage. We find that the policy itself provides a finer signal. Human-annotated rollout analysis shows that, after supervised fine-tuning (SFT), locally successful actions tend to have lower policy entropy than failed actions within the same trajectory, and that this separation becomes stronger during on-policy RL post-training. Based on this observation, we propose Entropy-guided Segment-level Credit Assignment for Policy Optimization (), which uses policy entropy to refine GRPO advantages without a critic, dense reward, teacher policy, or action-quality labels for training. In failed rollouts, reliable low-entropy segments receive increased credit while high-entropy segments are penalized; in successful rollouts, uncertain actions supported by successful outcomes receive additional positive credit. On LIBERO-10 and LIBERO-Plus-10, improves long-horizon learning over GRPO from both SFT and shared RL warm-up initializations. From SFT, it raises late-stage success from to and from to , respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reinforcement learning, vision-language-action models, policy optimization, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vbuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ESCAPO, an entropy-guided refinement of GRPO for VLA post-training. It adjusts credit across segments in failed trajectories and reinforces high-entropy actions in successful trajectories, without requiring a critic, a learned reward model, or human action-quality labels for policy optimization. The reported improvements on long-horizon tasks make this a potentially useful approach.

Critical strengths

The paper addresses a meaningful problem and presents a practical design. The proposed method adjusts credit across segments in failed trajectories and reinforces high-entropy actions in successful trajectories, without requiring a critic, a learned reward model, or human action-quality labels for policy optimization. The reported improvements on long-horizon tasks make this a potentially useful approach.

Critical weaknesses

First, the experiments do not sufficiently establish the specific contribution of entropy-guided segmentation, and secondly, the reliability of the reported gains is difficult to assess from the main paper.

The entropy analysis in Section 3 provides useful evidence of an association between entropy and local action quality. However, the analysis presented does not establish whether entropy provides information beyond trajectory position or execution stage. This leaves open whether the observed association reflects local action quality or primarily task progression, weakening the motivation for using entropy to assign local credit. The ablation in Section 5.5 shows that adding the failure branch improves performance when the success branch is active. This supports the usefulness of the branch, but does not isolate the benefit of segmentation: removing the failure branch also removes all failure-side entropy corrections. Comparisons with chunk-wise entropy correction without segmentation and a simple position-based correction, under the same training budget, would more directly test the central contribution. The performance gains are encouraging but need stronger evidence of consistency. Table 1 reports improvements over GRPO of 7.40 and 3.72 percentage points on the two long-horizon suites. However, the main paper does not report the number of independent training runs or uncertainty across runs. Averaging over the final 15% of training does not address this uncertainty. The smaller warm-up gains are particularly difficult to assess, while the real-robot result of 4/10 versus 2/10 successes provides only preliminary support.

Questions for authors
  1. Does entropy retain its ability to distinguish successful from failed actions after controlling for trajectory position or execution stage, for example through position-matched or stage-matched comparisons?
  2. Under the same training budget, how does ESCAPO compare with chunk-wise entropy correction without segmentation and a simple position-based correction?
  3. How many independent training runs underlie the main results? Can the authors report variability across seeds, such as mean and standard deviation or confidence intervals, to establish the consistency of the gains over GRPO?
1 comment

Thank you for the careful review and for suggesting concrete controls to test the contribution of segmentation and the reliability of the gains.

W1. Entropy beyond task progress and the contribution of segmentation. The new position- and stage-matched analyses show that entropy retains information about local action quality after controlling for either factor (Q1). We also added separate comparisons with chunk-wise entropy correction and a fixed-position credit rule. The chunk-wise comparison directly evaluates the contribution of grouping while retaining failure-side entropy correction (Q2).

W2. Reliability across training runs. The revision reports the mean and sample standard deviation across three independent LIBERO-10 training runs per method for ESCAPO and GRPO (Q3). Given the available compute budget, the warm-up comparisons use one training seed per configuration (Appendix D.6).

Q1. Entropy after matching stage or position. On human-labeled chunks from failed rollouts, negative chunk entropy achieves ROC-AUC of 0.67–0.94 within the same rollout and execution stage, and 0.56–0.94 at the same absolute chunk index. AUC exceeds 0.5 at every reported checkpoint. See Sec. 3.3 (lines 211–216) and Appendix B.2 (lines 899–908), Fig. B.2.

Q2. Chunk-wise and position-based comparisons. With matched rollout budgets, optimization settings, and evaluation intervals on LIBERO-10, ESCAPO achieves 60.18% vs. 56.99% Late-15% success and 41.92% vs. 42.11% normalized AUC against chunk-wise correction. Against the simple position-based rule, it achieves 32.65% vs. 29.74% Late-15% success and 24.37% vs. 21.46% normalized AUC. See Sec. 5.5, Table 5, Fig. 8, and Appendix D.1 (lines 1227–1290).

Q3. Independent training runs. Given the available time and compute, we used one training seed per configuration for the main experiments and ablations (Appendix D.1). We additionally trained ESCAPO and GRPO on LIBERO-10 with three independent runs per method (seeds 1234, 2026, and 2027). Using each run’s success rate at the final training step (step 75), ESCAPO achieves 44.30% mean success across seeds versus 38.74% for GRPO, with sample standard deviations of 0.90 and 6.53 percentage points, respectively. Section 5.2 summarizes the three-seed comparison, and Appendix D.3 describes the final training step (step 75) analysis (lines 1385–1393). Table D.3 (lines 1405–1413) reports the paired results: ESCAPO exceeded GRPO by 1.56, 12.01, and 3.13 percentage points for seeds 1234, 2026, and 2027, respectively. Fig. D.2 (caption at lines 1370–1371) shows the mean learning curves with shading for ±1 sample standard deviation.

Reviewer_fMuiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates temporal credit assignment in sparse-reward vision-language-action reinforcement learning and proposes ESCAPO, an entropy-guided advantage-shaping method. Specifically, it adds upward advantage corrections to eligible low-entropy segments of failed trajectories and downward corrections to high-entropy segments, while reinforcing uncertain actions in successful trajectories. Its analysis examines the relationship between entropy and local action quality during training. Experiments on LIBERO benchmarks report improved long-horizon task success, with preliminary real-robot evaluation.

Critical strengths
  • Under shared initialization, LIBERO-10 late success rises from 46.90% to 54.30%; LIBERO-Plus-10 rises from 27.66% to 31.38%. These gains address a consequential credit-assignment problem without adding an external supervision channel.
  • Shared-checkpoint continuation and failure-branch ablations provide evidence beyond one favorable comparison. The manuscript also discloses failed initializations, weak short-horizon gains, and the limited scope of its physical evaluation. These counterexamples improve the credibility of the bounded empirical claim.
Critical weaknesses
  • The paper does not establish what the proposed credit mechanism adds beyond close prior art and temporal weighting. Previous works [1, 2] have already used entropy for segment-level advantage shaping and VLA quality assessment, respectively. The omitted LESS [1] constructs contiguous low-entropy spans and shapes GRPO advantages using correctness. Its cross-response occurrence rule differs from ESCAPO's additive failed-robot-rollout correction, so exact duplication is not established. The cited GTPO [3] is also a direct shaping antecedent, yet no direct comparison is supplied. The omitted work [2] relates action-token entropy to human-rated VLA quality. The residual claim therefore depends on local failed-rollout credit. The supplemental chunk-position AUC of about 0.92–0.99 and inconsistent within-window entropy discrimination expose an unresolved progress confound. Different initialization and AI annotations limit that counter-evidence; they do not remove the need for position-matched human-label analysis and a matched position-based update.
  • The reported gains lack a sufficiently specified replication protocol. Independent training-seed uncertainty is deferred, and Appendix D does not clearly provide simulation evaluation denominators or the relationship between training rollouts and reported success curves. The coefficient mismatch above also leaves the implemented update ambiguous. Small warm-up and ablation differences therefore remain descriptive. The 4/10 versus 2/10 physical result is appropriately preliminary, not independent confirmation of a stable transfer advantage.

References

[1] Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs

[2] Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots

[3] GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy

Questions for authors
  • What exact LOW/HIGH update produced the tables, and how does it relate to βfail\beta_{fail}? Please specify the evaluation episodes and human-annotation protocol.
  • Does entropy add predictive value beyond chunk position on the main human-labeled trajectories?
  • Which gains require the new failure correction rather than a suitably adapted GTPO/LESS-style rule or duration-matched position weighting, and do they persist across independent training seeds?
  • How does the entropy-only credit signal compare with previous work [1], which already assigns stage-specific advantages to VLA action intervals? Its privileged simulator predicates distinguish it from ESCAPO, but the missing citation is directly relevant to the local-credit motivation.

References

[1] Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

1 comment

Thank you for the detailed review. Your comments on prior work, the role of task progress, and reproducibility helped us sharpen both the analysis and the experimental protocol.

W1. Prior work and the role of task progress. We agree that temporal progress is an important control. We added human-labeled analyses that control for stage and position (Q2), along with direct comparisons against GTPO and a position-based update (Q3). Sec. 7 now discusses LESS’s cross-response matching of low-entropy spans and prior work relating VLA uncertainty to human-rated execution quality. For each failed rollout that passes the reliability gate, ESCAPO increases the GRPO advantages of participating LOW segments and decreases those of HIGH segments.

W2. Replication and uncertainty. We appreciate the request for a clearer protocol. The revision specifies the rollout denominator and curve aggregation (Appendix D.1), the annotation procedure (Appendix B.1), and the correction weights (Table D.1). It also adds a three-seed LIBERO-10 comparison (Appendix D.3). Given the available compute budget, we used one training seed per configuration for the warm-up and ablation experiments (Appendix D.6).

Q1. Exact LOW/HIGH update and evaluation protocol. For each failed rollout that passes the gate, Eq. (6) standardizes the binary segment scores (LOW = 1, HIGH = 0), weighting each participating segment equally. Equation (7) selects the correction weight by segment type. We use 0.5 for LOW segments and 0.25 for HIGH segments (Table D.1). Equation (9) adds the weighted correction to the original GRPO advantage on participating segments, with zero failure-side correction elsewhere (Sec. 4.2, lines 258–291). Each LIBERO learning-curve point reports the fraction of 1,024 on-policy training rollouts that succeed at least once within 512 environment actions (Appendix D.1, lines 1315–1319). Appendix B.1 describes the human annotation protocol (lines 829–842).

Q2. Predictive value beyond position. When we matched chunks by absolute position, negative chunk entropy distinguished locally successful from failed actions with ROC-AUC of 0.56–0.94 (Fig. B.2(b)). Matching by execution stage within the same rollout yielded ROC-AUC of 0.67–0.94 (Fig. B.2(a)). On held-out human-labeled trajectories from the main analysis, adding entropy to a position-only model improves ROC-AUC from 88.55% to 94.98% for LIBERO-10 GRPO, 88.12% to 90.64% for LIBERO-Plus-10 GRPO, and 91.48% to 93.74% for LIBERO-10 PPO. Log loss also decreases in all three settings. These results show that policy entropy adds predictive information about local action quality beyond absolute chunk position. Table B.2 reports the regression results (see Appendix B.2, lines 980–982). In a separate analysis, negative chunk entropy also distinguished action quality among chunks from the same rollout and execution stage (ROC-AUC 0.67–0.94, Fig. B.2). Section 3.3 (lines 211–215) summarizes the stage- and position-matched analyses.

Q3. Failure correction, baselines, and training seeds. LESS matches low-entropy spans across correct and incorrect responses. ESCAPO assigns credit within each failed rollout, allowing it to preserve useful behavior even when the same operation requires different action sequences (Sec. 7). GTPO uses inverse entropy to penalize low-entropy tokens more strongly in failed rollouts. ESCAPO gives sustained low-entropy segments more credit and high-entropy segments less credit when the reliability gate is active (Sec. 4.2). In the direct comparison with VLA-adapted GTPO, ESCAPO achieves 54.30% vs. 49.42% Late-15% success (Sec. 5.2, Table 1). With the success branch fixed, enabling the failure correction raises Late-15% success from 50.26% to 55.36% (Sec. 5.5, Table 4). We also trained a position-based credit assignment baseline that raises credit for the first 32 valid chunks of each eligible failed rollout and lowers it thereafter. ESCAPO achieves 32.65% vs. 29.74% Late-15% success, supporting the use of rollout-specific entropy over a fixed temporal split. Section 5.5 describes the baseline, and Table 5 reports the results. A rollout-specific duration-matched comparison remains open. Given the available compute budget, we used one training seed per configuration for the failure-branch ablation and the comparisons with VLA-adapted GTPO and the fixed-position baseline. Appendix D.3 reports the three-seed comparison of the full ESCAPO method with GRPO.

Q4. Comparison with Temporal GRPO. Thank you for pointing us to this relevant work. Temporal GRPO assigns stage-level credit using task-specific completion predicates and privileged simulator states, while ESCAPO derives local corrections from policy entropy and terminal task outcomes. We added this comparison in Sec. 7.

Reviewer_jjWrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ESCAPO, an entropy-guided segment-level credit assignment method for VLA RL post-training without critics or process rewards. It uses policy entropy to refine GRPO advantages, preserving useful behaviors in failed trajectories and encouraging exploration in successful ones. Experiments demonstrate improvements over GRPO on long-horizon LIBERO benchmarks, with preliminary real-robot validation.

Critical strengths
  1. Novel and practically relevant approach to VLA RL. The critic-free and process-reward-free design is particularly appealing for real-robot VLA RL, where learning reliable critics and obtaining dense rewards can be challenging.

  2. Meaningful improvements over prior entropy-guided methods. Compared with GTPO, ESCAPO introduces temporal segment grouping, protection of useful behaviors in failed trajectories, and reliability gating. These designs are well motivated by the characteristics of long-horizon robotic manipulation.

Critical weaknesses
  1. Relatively modest gains and insufficient comparison with GTPO. The performance improvements are relatively modest, and the lack of a direct comparison with GTPO adapted to VLA makes it difficult to isolate the benefits of the proposed innovations. More targeted ablations of the key components would strengthen the claims.

  2. Insufficient analysis of segment grouping and limited generality. The paper lacks a direct comparison between segment-level and chunk-level credit assignment to demonstrate the effectiveness of temporal grouping. Additionally, the absence of experiments on flow-based VLA policies limits the demonstrated generality of the approach.

Questions for authors
  1. Comparison with GTPO. Could the authors provide a direct comparison with GTPO adapted to VLA under the same training setup? This would help clarify the improvements attributable to ESCAPO's proposed mechanisms beyond existing entropy-guided credit assignment.

  2. Effectiveness of segment grouping. How much does temporal segment grouping contribute compared with directly applying entropy-based corrections at the chunk level? An ablation isolating segment grouping and reliability gating would help substantiate these key design choices.

  3. Generalization to flow-based policies. Can the proposed approach be extended to flow-based VLA policies, where action entropy is less straightforward to compute? A discussion of potential adaptations or preliminary results would strengthen the practical relevance of the method.

2 comments

Thank you for your thoughtful review and for recognizing the practical motivation of critic-free credit assignment.

W1. Magnitude of the gains and comparison with GTPO. We agree that a direct comparison is important for interpreting the gains over GRPO. We added a VLA-adapted GTPO baseline (Q1) and ablations of the reliability gate and temporal grouping (Sec. 5.5).

W2. Segment grouping and generality. We appreciate the suggestion to isolate temporal grouping. The new chunk-level comparison shows gains in late-stage success from grouping, with similar normalized AUC (Q2). We also expanded the discussion of flow-based policies in Sec. 6 (Q3).

Q1. Direct comparison with GTPO. On LIBERO-10, ESCAPO and VLA-adapted GTPO use the same SFT checkpoint, rollout budget, optimization settings, and training interval (steps 0–118); only their advantage construction differs. ESCAPO achieves 38.42% normalized AUC and 54.30% Late-15% success, compared with 36.27% and 49.42% for GTPO. This corresponds to gains of 2.15 and 4.88 percentage points, respectively. Section 5.2 reports the comparison in Table 1, and Appendix D.1 (lines 1293–1313) defines the token-level GTPO rewards and advantages in Eqs. (D.1)–(D.3).

Q2. Segment grouping and reliability gating. Compared with chunk-wise correction, segment grouping increases Late-15% success from 56.99% to 60.18%, while normalized AUC remains similar (42.11% vs. 41.92%). In a separate matched comparison, enabling the gate increases Late-15% success from 47.82% to 54.30% and normalized AUC from 34.73% to 37.99%. These results support the contributions of both segment grouping and reliability gating in the evaluated settings. See Sec. 5.5, Table 5, Fig. 8, and Appendix D.1 (lines 1263–1290).

Q3. Extension to flow-based policies. Thank you for raising this direction. Sec. 6 now discusses two candidate signals, Action-Chunk Entropy estimated from repeated samples under the same observation and instruction, and VLM-backbone token entropy where available. Whether either signal can guide local credit assignment in flow-matching VLAs remains an open empirical question.

Thanks for the author's response. My concerns have been largely addressed. I have therefore decided to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.