- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ESCAPO, an entropy-guided refinement of GRPO for VLA post-training. It adjusts credit across segments in failed trajectories and reinforces high-entropy actions in successful trajectories, without requiring a critic, a learned reward model, or human action-quality labels for policy optimization. The reported improvements on long-horizon tasks make this a potentially useful approach.
Critical strengths
The paper addresses a meaningful problem and presents a practical design. The proposed method adjusts credit across segments in failed trajectories and reinforces high-entropy actions in successful trajectories, without requiring a critic, a learned reward model, or human action-quality labels for policy optimization. The reported improvements on long-horizon tasks make this a potentially useful approach.
Critical weaknesses
First, the experiments do not sufficiently establish the specific contribution of entropy-guided segmentation, and secondly, the reliability of the reported gains is difficult to assess from the main paper.
The entropy analysis in Section 3 provides useful evidence of an association between entropy and local action quality. However, the analysis presented does not establish whether entropy provides information beyond trajectory position or execution stage. This leaves open whether the observed association reflects local action quality or primarily task progression, weakening the motivation for using entropy to assign local credit. The ablation in Section 5.5 shows that adding the failure branch improves performance when the success branch is active. This supports the usefulness of the branch, but does not isolate the benefit of segmentation: removing the failure branch also removes all failure-side entropy corrections. Comparisons with chunk-wise entropy correction without segmentation and a simple position-based correction, under the same training budget, would more directly test the central contribution. The performance gains are encouraging but need stronger evidence of consistency. Table 1 reports improvements over GRPO of 7.40 and 3.72 percentage points on the two long-horizon suites. However, the main paper does not report the number of independent training runs or uncertainty across runs. Averaging over the final 15% of training does not address this uncertainty. The smaller warm-up gains are particularly difficult to assess, while the real-robot result of 4/10 versus 2/10 successes provides only preliminary support.
Questions for authors
- Does entropy retain its ability to distinguish successful from failed actions after controlling for trajectory position or execution stage, for example through position-matched or stage-matched comparisons?
- Under the same training budget, how does ESCAPO compare with chunk-wise entropy correction without segmentation and a simple position-based correction?
- How many independent training runs underlie the main results? Can the authors report variability across seeds, such as mean and standard deviation or confidence intervals, to establish the consistency of the gains over GRPO?
Thank you for the careful review and for suggesting concrete controls to test the contribution of segmentation and the reliability of the gains.
W1. Entropy beyond task progress and the contribution of segmentation. The new position- and stage-matched analyses show that entropy retains information about local action quality after controlling for either factor (Q1). We also added separate comparisons with chunk-wise entropy correction and a fixed-position credit rule. The chunk-wise comparison directly evaluates the contribution of grouping while retaining failure-side entropy correction (Q2).
W2. Reliability across training runs. The revision reports the mean and sample standard deviation across three independent LIBERO-10 training runs per method for ESCAPO and GRPO (Q3). Given the available compute budget, the warm-up comparisons use one training seed per configuration (Appendix D.6).
Q1. Entropy after matching stage or position. On human-labeled chunks from failed rollouts, negative chunk entropy achieves ROC-AUC of 0.67–0.94 within the same rollout and execution stage, and 0.56–0.94 at the same absolute chunk index. AUC exceeds 0.5 at every reported checkpoint. See Sec. 3.3 (lines 211–216) and Appendix B.2 (lines 899–908), Fig. B.2.
Q2. Chunk-wise and position-based comparisons. With matched rollout budgets, optimization settings, and evaluation intervals on LIBERO-10, ESCAPO achieves 60.18% vs. 56.99% Late-15% success and 41.92% vs. 42.11% normalized AUC against chunk-wise correction. Against the simple position-based rule, it achieves 32.65% vs. 29.74% Late-15% success and 24.37% vs. 21.46% normalized AUC. See Sec. 5.5, Table 5, Fig. 8, and Appendix D.1 (lines 1227–1290).
Q3. Independent training runs. Given the available time and compute, we used one training seed per configuration for the main experiments and ablations (Appendix D.1). We additionally trained ESCAPO and GRPO on LIBERO-10 with three independent runs per method (seeds 1234, 2026, and 2027). Using each run’s success rate at the final training step (step 75), ESCAPO achieves 44.30% mean success across seeds versus 38.74% for GRPO, with sample standard deviations of 0.90 and 6.53 percentage points, respectively. Section 5.2 summarizes the three-seed comparison, and Appendix D.3 describes the final training step (step 75) analysis (lines 1385–1393). Table D.3 (lines 1405–1413) reports the paired results: ESCAPO exceeded GRPO by 1.56, 12.01, and 3.13 percentage points for seeds 1234, 2026, and 2027, respectively. Fig. D.2 (caption at lines 1370–1371) shows the mean learning curves with shading for ±1 sample standard deviation.