acceptodds
Under review as a conference paper at ICLR 2027

PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Models

Abstract

Reinforcement Fine-Tuning (RFT) has emerged as a promising paradigm for improving Vision-Language-Action (VLA) policies, yet sparse task-level outcomes provide limited credit for intermediate transitions, especially in long-horizon manipulation. A natural approach is to model intermediate task progress and use it as dense feedback for policy improvement. Despite their architectural differences, existing progress-aware methods commonly formulate task progress as an explicit scalar prediction, providing limited structure for modeling how intermediate observations relate to the task goal, which may hinder effective transition-level credit assignment. We introduce Progress Field Reinforcement Learning (PF-RL), which learns a structured goal-conditioned progress representation over pretrained VLA features and converts it into dense credit for policy optimization. A lightweight shared Progress Field head maps current and goal representations into a compact progress space, where geometric distance induces goal-conditioned value, while complementary temporal and goal-structure objectives shape the learned geometry. Transition-level value changes naturally yield dense progress advantages, enabling fine-grained credit assignment for both offline policy improvement and online reinforcement fine-tuning. Extensive experiments on LIBERO, RoboTwin2.0, and real-world bimanual manipulation tasks show that PF-RL consistently improves policy performance over strong supervised fine-tuning, reinforcement fine-tuning, and progress-aware baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Vision-Language-Action Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_156MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PF-RL learns a goal-conditioned value function from VLA features and uses value changes to estimate action progress. This signal supports offline policy training and online refinement. Experiments cover LIBERO, four RoboTwin tasks, and four real-world bimanual manipulation tasks.

Critical strengths
  • PF-RL improves over the reproduced GRPO baseline under matched initialization, data, and interaction budgets. The improvement on RoboTwin, from approximately 79% to 92%, provides useful evidence beyond the nearly saturated LIBERO benchmark.

  • Comparisons with alternative value functions and ablations of the training objectives support the proposed design. Experiments on four real-world tasks also show improvements over EvoRL, extending the evaluation beyond simulation.

Critical weaknesses
  • Comparisons with strong existing RL methods are incomplete. SimpleVLA-RL and SRPO report similar LIBERO success rates, but direct comparisons are missing. SRPO is particularly relevant because it also uses successful trajectories to provide progress-related feedback for failed attempts. Comparisons under comparable training budgets would better establish the practical benefit of PF-RL.

  • Real-world training includes human corrections, but their frequency and duration are not reported. Following the same collection protocol does not necessarily provide comparable amounts of corrective supervision. This makes it difficult to assess how much of the improvement comes from the proposed method and how much depends on human assistance.

Questions for authors
  • How does PF-RL compare with SimpleVLA-RL and, particularly, SRPO under comparable data and interaction budgets, accounting for additional computation?

  • How much human intervention was required by each method in the real-world experiments, and how was the corrective supervision controlled across methods?

3 comments

[W1 & Q1: Incomplete comparisons with strong existing RL methods.]

Thank you for the constructive comments. To provide a more comprehensive comparison with existing RL methods, we additionally evaluate SimpleVLA-RL and SRPO on LIBERO. Both methods are initialized from the same 30K-step SFT checkpoint used by RLinf-GRPO and trained for 100 online RL updates, with 64 groups per update, 8 trajectories per group, and 512 trajectories in total per update. For PF-RL, we retain our original offline-to-online training pipeline. All methods are evaluated under the same protocol with the same training budget. As shown in Table R1, PF-RL achieves an average success rate of 99.1%, outperforming RLinf-GRPO (97.1%), SimpleVLA-RL (97.5%), and SRPO (98.0%). In particular, PF-RL improves over SRPO by 1.4 and 2.0 percentage points on LIBERO-Goal and LIBERO-Long, respectively, both of which demand stronger long-horizon decision-making capabilities. These results provide further evidence of the effectiveness of PF-RL compared with strong existing RL methods. We have included the additional experimental details and results in Appendix D.

Table R1. Comparison with recent RL baselines on LIBERO under matched data and training budgets.

MethodSpatialObjectGoalLongAvg.
RLinf-GRPO97.698.795.296.897.1
SimpleVLA-RL98.498.895.497.497.5
SRPO98.099.497.497.098.0
PF-RL (Ours)99.299.598.899.099.1

We additionally report the online training time of each method using the same cluster of 8 NVIDIA A800 GPUs, as summarized in Table R2. Despite differences in their implementation pipelines, PF-RL incurs only a modest additional computational cost compared with RLinf-GRPO, while requiring less training time than SimpleVLA-RL and SRPO.

Table R2. Online training time comaprison of different online RL methods under matched data and training budgets.

RLinf-GRPOSimpleVLA-RLSRPOPF-RL (Ours)
27.6 h30.2h33.6h28.7h

[W2 & Q2: Real-world training includes human corrections, but their frequency and duration are not reported. ]

Thanks for raising this important concern. We additionally report the human intervention details for Evo-RL and PF-RL across four real-world tasks and two rounds of RL training (40 rollouts per task), as summarized in Table R3. All methods follow the same intervention protocol: human operators take over when the robot exhibits execution errors or becomes unable to make meaningful task progress, and return control to the policy once execution recovers to a stable state. As shown in Table R3, the corrective-supervision budgets are broadly comparable between PF-RL and Evo-RL, with overall human-control durations of 4.6 and 5.3 minutes and human-control ratios of 6.9% and 7.2%, respectively, where the human-control ratio represents the percentage of training timesteps involving human intervention. These results support the fairness of the real-world comparison by showing that PF-RL's improvements are not attributable to receiving more human corrective supervision. We have added the human intervention details to the revised Appendix B.2.

Table R3. Human intervention details during real-world RL training (40 rollouts per task).

TaskMethodIntervention SegmentsHuman-Controlled TimestepsHuman-Control Duration (min)Human-Control Ratio (%)
Fold ClothesEvo-RL1128711.67.3
PF-RL820581.15.6
Stack all PlatesEvo-RL810320.65.2
PF-RL78610.54.7
Put Blocks into BucketEvo-RL1018301.06.6
PF-RL1019461.17.8
Fill Vase and Arrange FlowersEvo-RL1338222.18.3
PF-RL1233841.98.6
OverallEvo-RL4295555.37.2
PF-RL3782494.66.9

Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.

Thank you for your positive feedback and for updating your score. We sincerely appreciate your time and constructive suggestions, which have helped us improve our paper.

Reviewer_mUB3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PF-RL, which learns a goal-conditioned progress representation from VLA features and defines value as the negative Euclidean distance between the current and goal embeddings. The representation is trained with expectile TD learning, step-to-goal calibration, goal contrastive learning, and auxiliary subgoals. The resulting TD residuals are used to reweight offline imitation and supplement GRPO advantages during online simulation training. On real robots, newly collected trajectories instead support progress-weighted flow-matching updates. For failed online trajectories, the progress advantage uses a task-level goal prototype derived from successful endpoints rather than the failed terminal observation.

The main LIBERO experiment reports 99.1% average success against 97.1% for a reproduced RLinf-GRPO baseline, with three training seeds. The paper also reports results on four RoboTwin tasks and four real-robot tasks, together with comparisons against scalar progress prediction and ablations of the training objectives.

Critical strengths
  1. A coherent way to use goal-conditioned value estimates for VLA fine-tuning. The shared projection makes task progress depend on the distance between current and goal observations, rather than on a standalone scalar head. The paper connects this representation to offline advantage-weighted imitation and online policy updates, and handles failed online trajectories by using success-based goal prototypes. This is a practical formulation of progress-aware VLA adaptation, even though several individual ingredients build on prior work.

  2. Useful controlled comparisons and results across settings. The main LIBERO experiment matches the initial checkpoint, data, and optimization budgets and reports three training seeds, with PF-RL improving the average success rate from 97.1% to 99.1%. Table 4 includes both a distance-based variant without contrastive learning and an MLP readout over the learned projections, making the geometry comparison more informative than a single scalar baseline. The RoboTwin and real-world results provide additional evidence of potential practical benefit, although their statistical strength is more limited.

Critical weaknesses
  1. RL uses progress-weighted imitation during offline optimization, whereas the reproduced RLinf-GRPO baseline uses standard SFT before online training. Thus, the 99.1% versus 97.1% result changes both the offline initialization and the online advantage. Table 6 varies the progress weight, but it is unclear whether its zero-weight setting starts from the same offline PF-RL checkpoint. This matters because the central claim concerns transition-level credit assignment, not just the performance of the combined pipeline. Moreover, with γ = 1, Eqs. 6 and 9 imply a zero TD residual along a trajectory if the predicted value exactly equals the negative number of remaining action chunks. This is not an error in itself, but it makes independent validation of what the actual nonzero residuals reward important. The qualitative examples and the small manually annotated sample in Figure 11 do not yet establish that the signal reliably distinguishes useful actions from timing or value-estimation effects.

  2. Several supporting results have limited statistical or supervision-budget controls. Appendix B states that the main LIBERO results use three seeds but other experiments generally use one. This applies to the key scalar-readout and loss ablations, where the performance differences are used to support claims about the value geometry and its components. The real-robot results use only 20 evaluation rollouts per setting; in the original conditions, the improvements over Evo-RL correspond to one or two additional successes per task. Human corrective interventions are also incorporated into training, but the amount of intervention per method is not reported. Without those counts, it is difficult to assess whether the compared methods use equivalent amounts of human supervision.

Questions for authors
  1. Offline versus online attribution. Does the GRPO (λ_prog = 0) row in Table 6 start from the standard SFT checkpoint or from the offline PF-RL checkpoint? Could the authors compare GRPO and PF-RL online updates from the same offline PF-RL policy and report the performance after each offline stage? Table 1 reports an offline PF-RL mean of 81.7%, which makes the intermediate checkpoints particularly important for interpreting the final result.

  2. Meaning of the progress advantage. Beyond the qualitative examples, can the authors evaluate whether A_prog ranks beneficial and harmful transitions on independently held-out trajectories, preferably alongside the direct scalar-progress baseline? Including pauses, detours, recovery actions, and failed trajectories would help determine whether the residual captures credit for improving task outcomes rather than mainly deviations from the calibrated time-to-goal estimate.

  3. Reliability of the representation ablations. Can the main scalar/MLP comparisons in Table 4 and the principal loss ablations in Table 5 be repeated across training seeds, with uncertainty on the differences relative to full PF-RL? These are the most direct experiments supporting the claimed advantage of the geometry.

  4. Real-robot protocol and human assistance. For each method, how many interventions were made during training, how long did they last, and what fraction of executed actions came from the human operator? Were evaluations fully autonomous, and were success criteria identical across methods? The answers would help interpret the 20-trial success rates and compare the effective supervision budgets.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

[W1.1 & Q1: Insufficient isolation of offline and online contributions.]

Thanks for the constructive comment. We clarify that the original GRPO baseline in Table 6 is initialized from the standard-SFT checkpoint. To isolate the contribution of online progress guidance, we additionally evaluate vanilla GRPO (λprog=0\lambda_{\mathrm{prog}}=0) initialized from the same offline PF-RL checkpoint as the full method. As shown in Table R1, offline PF-RL initialization improves the final average performance of vanilla GRPO from 97.1% to 97.8%, demonstrating the contribution of progress-weighted imitation during offline optimization. More importantly, under the same offline initialization, PF-RL further improves the average performance to 99.1%, consistently outperforming vanilla GRPO across all four LIBERO suites. This comparison isolates the additional benefit of incorporating the progress advantage during online RL, beyond the gains from offline progress-weighted imitation. We have incorporated this additional comparison into the revised Appendix F.3.

Table R1. Offline-to-online performance comparison under different offline initializations and online progress guidance settings.

Methodλprog\lambda_{\mathrm{prog}}SpatialObjectGoalLongAvg.
GRPO (SFT initialization)088.6→97.676.6→98.783.2→95.275.2→96.880.9→97.1
GRPO (Offline PF-RL initialization)087.4→98.077.8→98.884.6→96.476.8→97.881.7→97.8
PF-RL (Ours)1.087.4→99.277.8→99.584.6→98.876.8→99.081.7→99.1

[W1.2 & Q2: Insufficient validation of the meaning and reliability of progress advantages.]

Thanks for the constructive suggestions. The progress advantage measures goal-conditioned value changes relative to the learned temporal scale instead of the elapsed time itself, allowing it to distinguish productive progress from stagnation or regression. With γ=1\gamma=1, a zero TD residual under exact temporal calibration simply indicates that the value change matches the expected temporal progression, rather than implying an absence of task progress. To validate this, we extend Appendix H with a comparison between Progress Field and the matched scalar readout on manually annotated progress, stagnation, and regression segments from successful and failed trajectories. All evaluation trajectories are independently collected and held out from the training data. For failed trajectories, successful-goal prototypes are used as goal references. As shown in revised Fig. 11 and Table R2, Progress Field achieves consistently higher pairwise ranking probabilities than Scalar Prediction across all three comparisons on both successful and failed trajectories. These results demonstrate that Progress Field provides more reliable transition-level behavioral discrimination than scalar prediction, including on held-out failed trajectories with successful-goal prototypes. We have incorporated these analyses into the revised Appendix H.

Table R2. Pairwise ranking probabilities (%) averaged across four real-world tasks. P, S, and R denote progress, stagnation, and regression, respectively. Each metric measures the probability that a randomly sampled segment from the first category receives a higher average advantage than one from the second category, with both segments sampled from the same trajectory within the same task.

MethodP > SP > RS > R
Successful trajectories
Scalar Prediction61.585.379.5
Progress Field (Ours)85.299.788.7
Failed trajectories
Scalar Prediction61.088.980.3
Progress Field (Ours)81.499.992.6

[W2.1 & Q3: Limited multi-seed validation of key representation and loss ablations.]

We appreciate your valuable comment. We conduct additional multi-seed experiments to strengthen the evidence for our geometric parameterization and training objectives. First, we evaluate Progress Field against the matched scalar MLP readout using three matched seeds and identical training budgets as shown in Table R3. Both methods employ the same contrastive objective to isolate the effect of value parameterization. Progress Field consistently outperforms the scalar baseline, with improvements of 4.7 and 3.4 percentage points, respectively. Second, we repeat the principal loss ablations on LIBERO-Long across three seeds as shown in Table R4. The results validate the contributions of the individual objectives.

Table R3. Multi-seed comparison between Progress Field and scalar MLP readout on selected LIBERO suites. Results are reported over 3 seeds.

MethodGoalLongAvg.
Matched Scalar MLP Readout (w Lcon\mathcal L_\text{con})94.1±1.295.6±1.094.9
PF-RL (w Lcon\mathcal L_\text{con})98.8±1.099.0±0.598.9

Table R4. Multi-seed ablation of the training objectives on LIBERO-Long.

Methodw/o TDw/o Stepw/o ContrastiveTD-onlyFull PF-RL
LIBERO-Long94.5±0.395.3±0.698.0±0.495.5±0.599.0±0.5
AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

[W2.2: Limited statistical precision of real-world evaluation.]

Thanks for this constructive concern. We expand the real-world evaluation from 20 to 30 rollouts per condition across four manipulation tasks, including original and perturbed settings. As shown in Table R5, π0.5\pi_{0.5}+PF-RL outperforms Evo-RL in various real-world task settings, demonstrating improved real-world performance and robustness. We have incorporated the expanded real-world evaluation results into the revised manuscript.

Table R5. Real-world success rates over 30 rollouts per condition. Cloth, Bowl, Blocks, and Flowers denote the four manipulation tasks: Fold Clothes, Stack all Plates, Put Blocks into Bucket, and Fill Vase and Arrange Flowers, respectively.

MethodCloth Orig.Cloth−ObjCloth-BGCloth−LPlates Orig.Plates−ObjPlates−BGPlates−L
π0.5\pi_{0.5}80.066.7 (−13.3)70.0 (−10.0)50.0 (−30.0)70.070.0 (0.0)56.7 (−13.3)63.3 (−6.7)
Evo-RL86.776.7 (−10.0)83.3 (−3.3)60.0 (−26.7)93.386.7 (−6.7)80.0 (−13.3)83.3 (−10.0)
π0.5\pi_{0.5} + PF-RL80→96.786.7 (−10.0)90.0 (−6.7)83.3 (−13.3)85→96.796.7 (0.0)93.3 (−3.3)90.0 (−6.7)
MethodBlocks Orig.Blocks−ObjBlocks−BGBlocks−LFlowers Orig.Flowers−ObjFlowers−BGFlowers-L
π0.5\pi_{0.5}76.766.7 (−10.0)66.7 (−10.0)43.3 (−33.3)50.033.3 (−16.7)40.0 (−10.0)26.7 (−23.3)
Evo-RL83.383.3 (0.0)76.7 (−6.7)70.0 (−13.3)76.760.0 (−16.7)50.0 (−26.7)66.7 (−10.0)
π0.5\pi_{0.5} + PF-RL80→93.383.3 (−10.0)86.7 (−6.7)76.7 (−16.7)65→86.776.7 (−10.0)73.3 (−13.3)76.7 (−10.0)

[W2.3 & Q4: Unquantified human corrective supervision and insufficiently specified real-robot evaluation protocol.]

Thanks for raising this important suggestion. We additionally report the human intervention details for Evo-RL and PF-RL across four real-world tasks and two rounds of RL training (40 rollouts per task), as summarized in Table R6. All methods follow the same intervention protocol: human operators take over when the robot exhibits execution errors or becomes unable to make meaningful task progress, and return control to the policy once execution recovers to a stable state. As shown in Table R6, the corrective-supervision budgets are broadly comparable between PF-RL and Evo-RL, with overall human-control durations of 4.6 and 5.3 minutes and human-control ratios of 6.9% and 7.2%, respectively. These results support the fairness of the real-world comparison and the effectiveness of PF-RL in real-world manipulation. As for evaluation, all methods execute autonomously without human intervention, while task completion is assessed by human evaluators using identical criteria across all methods. We have added the human intervention details to the revised Appendix B.2.

Table R6. Human intervention details during real-world RL training (40 rollouts per task).

TaskMethodIntervention SegmentsHuman-Controlled TimestepsHuman-Control Duration (min)Human-Control Ratio (%)
Fold ClothesEvo-RL1128711.67.3
PF-RL820581.15.6
Stack all PlatesEvo-RL810320.65.2
PF-RL78610.54.7
Put Blocks into BucketEvo-RL1018301.06.6
PF-RL1019461.17.8
Fill Vase and Arrange FlowersEvo-RL1338222.18.3
PF-RL1233841.98.6
OverallEvo-RL4295555.37.2
PF-RL3782494.66.9

Thank you very much for the additional experiments. They have addressed most of my concerns, and I will consider increasing my confidence score.

Thank you for your thoughtful feedback and consideration. We sincerely appreciate your time and constructive suggestions, which have helped us further improve the paper.

Reviewer_acWgOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progress Field Reinforcement Learning (PF-RL), a method for improving credit assignment during reinforcement fine-tuning of vision-language-action policies. A shared projection head maps current and goal-conditioned VLA representations into an embedding space, where negative Euclidean distance defines a goal-conditioned value. The representation is trained using expectile temporal-difference learning, remaining-step calibration, goal contrastive learning, and auxiliary future subgoals. Temporal changes in the learned value provide progress advantages, which reweight offline imitation learning and augment GRPO during online simulation training. Real-robot adaptation uses advantage-weighted flow matching. Successful-goal prototypes provide task-completion references when scoring failed online trajectories.

The method is evaluated on LIBERO, four RoboTwin2.0 tasks, and four real-world manipulation tasks. The strongest controlled result is an improvement from 97.1% to 99.1% average LIBERO success under matched offline-to-online training budgets. Additional comparisons investigate the geometric parameterization, training objectives, and transfer of progress models across tasks.

Critical strengths
  1. The main controlled evaluation provides credible evidence of practical benefit.

The LIBERO comparison against the reproduced RLinf-GRPO baseline uses the same initial checkpoint, offline data, offline optimization budget, and online training budget, with three independent training seeds (Table 1 and Appendix B.1). PF-RL improves all four suites, including an increase from 96.8% to 99.0% on LIBERO-Long. The larger improvements on the evaluated RoboTwin2.0 tasks and the encouraging real-robot results provide complementary evidence across manipulation settings. These results make a reasonably convincing case that the complete training procedure improves policy performance.

  1. The paper includes informative controls for its central design choices.

The comparisons in Appendix D are particularly useful: PF-RL without goal contrastive learning outperforms direct scalar prediction, and the full method outperforms a scalar MLP readout that retains the projection architecture and training objectives (Table 4). The HILP-style TD-only variant in Table 5 further tests the value of the additional supervision. These controls support the usefulness of the proposed geometric inductive bias and training design. The contribution builds on established goal-conditioned metric value learning, which the paper acknowledges, and provides a practical integration with VLA policy optimization. The reported online computational overhead is also modest under the authors’ feature-caching implementation (Appendix I).

Critical weaknesses
  1. The interpretation of the learned signal as reliable task-progress credit is less firmly established than the policy-level improvement.

The step-calibration targets measure the realized temporal separation within a trajectory. Consequently, they can reflect execution speed, pauses, and recovery behavior in addition to task completion. With the chosen discount factor, the progress advantage is (Atprog=Dt−Dt+−1)(A_t^{\mathrm{prog}}=D_t-D_{t+}-1), which measures progress relative to the learned temporal scale. Moreover, failed trajectories train the field against their observed endpoints, whereas their policy advantages are evaluated against successful-goal prototypes. This requires useful generalization between these two goal references.

These choices are plausible, and the failure/recovery visualizations and Appendix G provide helpful supporting evidence. However, the paper offers limited aggregate comparison against scalar alternatives on the reliability of the resulting transition-level credit, particularly for failed or inefficient executions. This matters because improved credit assignment is the principal explanation for the method’s gains. The current results establish the utility of the overall procedure more clearly than they establish this mechanism.

  1. The decisive component comparisons and real-robot results have limited statistical precision.

Appendix B states that most experiments beyond the main LIBERO comparison use a single training seed, while the full PF-RL results are reused from the three-seed experiment. Thus, the comparisons most directly supporting the geometric parameterization and individual objectives lack comparable estimates of training variability. This limits confidence in how consistently the proposed inductive bias improves over the matched scalar alternatives.

The real-robot evaluation is also based on 20 trials per condition; the improvements over Evo-RL in the original settings correspond to one or two additional successes per task. Furthermore, corrective human interventions are retained in the training data, but their frequency or duration is not quantified. The reported common protocol is reassuring, but the amount of corrective data is relevant to interpreting these relatively small differences. These limitations do not invalidate the main LIBERO result, but they constrain the strength of the broader empirical conclusions.

Questions for authors

Interpretation and validation of progress credit. How should the learned value be interpreted when trajectories contain substantial pauses or inefficient recovery behavior? Could the authors provide an aggregate comparison between PF and the matched scalar readout on the annotated progress/stagnation/regression segments, including failed trajectories scored against successful-goal prototypes? Such an analysis would help assess whether the proposed geometry produces more reliable local credit.

Stability of the central parameterization comparison. How stable are the Table 4 results across training seeds? Repeating the most diagnostic distance-versus-MLP-readout comparison with matched seeds and reporting means and standard deviations would substantially strengthen the evidence for the central geometric contribution. A repetition of every ablation would not be necessary to address this concern.

Isolation of the online contribution. In Table 6, does the (\lambda_{\mathrm{prog}}=0) setting start from the same offline PF-RL checkpoint as the other settings, or from the standard-SFT checkpoint used by the GRPO baseline? If the initializations differ, please clarify how much of the reported comparison can be attributed specifically to the online progress advantage.

Real-world corrective-data budget. How many intervention segments, intervention timesteps, or minutes of human control were collected for each method? Were the amounts comparable, and were intervention-triggering criteria applied consistently? This information would help interpret the real-world gains independently of differences in corrective supervision.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

[W1 & Q1: The interpretation of the learned signal as reliable task-progress credit is less firmly established than the policy-level improvement.]

Thanks for the constructive comments. We agree with the interpretation of the learned progress signal and the need for generalization across goal references. The progress advantage measures goal-conditioned value changes relative to the learned temporal scale, rather than elapsed time itself, allowing it to distinguish productive progress from stagnation or regression.

To directly address the concern regarding transition-level credit reliability, we have extended Appendix H with an aggregate comparison between Progress Field and the matched scalar readout. Specifically, we evaluate both methods on the same annotated progress, stagnation, and regression segments from successful and failed trajectories. For failed trajectories, Progress Field advantages are computed using successful-goal prototypes as goal references. As shown in the updated Appendix H, Progress Field exhibits clearer separation between execution behaviors than Scalar Prediction. Meanwhile, we further quantify this distinction through pairwise ranking probabilities as shown in Table R1. The results shows that Progress Field consistently achieves higher ranking probabilities than Scalar Prediction across all three comparisons. Importantly, this advantage is maintained on failed trajectories when using successful-goal prototypes as goal references, providing more direct evidence for the reliability of local progress credit and its generalization across goal references.

Table R1. Pairwise ranking probabilities (%) averaged across four real-world tasks. P, S, and R denote progress, stagnation, and regression, respectively. Each metric measures the probability that a randomly sampled segment from the first category receives a higher average advantage than one from the second category, with both segments sampled from the same trajectory within the same task.

MethodP > SP > RS > R
Successful trajectories
Scalar Prediction61.585.379.5
Progress Field (Ours)85.299.788.7
Failed trajectories
Scalar Prediction61.088.980.3
Progress Field (Ours)81.499.992.6

[W2.1 & Q2: Limited multi-seed validation]

We appreciate your valuable comment. We conduct additional multi-seed experiments to strengthen the evidence for our geometric parameterization and training objectives. First, we evaluate Progress Field against the matched scalar MLP readout on LIBERO-Long and LIBERO-Goal using three matched seeds and identical training budgets as shown in Table R2. Both methods employ the same contrastive objective to isolate the effect of value parameterization. Progress Field consistently outperforms the scalar baseline, with improvements of 4.7 and 3.4 percentage points, respectively. Second, we repeat the principal loss ablations on LIBERO-Long across three seeds as shown in Table R3. The results validate the contributions of the individual objectives.

Table R2. Multi-seed comparison between Progress Field and scalar MLP readout on selected LIBERO suites. Results are reported over 3 seeds.

MethodGoalLongAvg.
Matched Scalar MLP Readout (w Lcon\mathcal L_\text{con})94.1±1.295.6±1.094.9
PF-RL (w Lcon\mathcal L_\text{con})98.8±1.099.0±0.598.9

Table R3. Multi-seed ablation of the training objectives on LIBERO-Long.

Methodw/o TDw/o Stepw/o ContrastiveTD-onlyFull PF-RL
LIBERO-Long94.5±0.395.3±0.698.0±0.495.5±0.599.0±0.5
AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

[W2.2: Statistical precision of real-world evaluation]

Thanks for this insightful concern. We expand the real-world evaluation from 20 to 30 rollouts per condition across four manipulation tasks, including original and perturbed settings. As shown in Table R4, π0.5\pi_{0.5}+PF-RL outperforms Evo-RL in various real-world task settings, demonstrating improved real-world performance and robustness. We have incorporated the expanded real-world evaluation results into the revised manuscript.

Table R4. Real-world success rates over 30 rollouts per condition. Cloth, Bowl, Blocks, and Flowers denote the four manipulation tasks: Fold Clothes, Stack all Plates, Put Blocks into Bucket, and Fill Vase and Arrange Flowers, respectively.

MethodCloth Orig.Cloth−ObjCloth-BGCloth−LPlates Orig.Plates−ObjPlates−BGPlates−L
π0.5\pi_{0.5}80.066.7 (−13.3)70.0 (−10.0)50.0 (−30.0)70.070.0 (0.0)56.7 (−13.3)63.3 (−6.7)
Evo-RL86.776.7 (−10.0)83.3 (−3.3)60.0 (−26.7)93.386.7 (−6.7)80.0 (−13.3)83.3 (−10.0)
π0.5\pi_{0.5} + PF-RL80→96.786.7 (−10.0)90.0 (−6.7)83.3 (−13.3)85→96.796.7 (0.0)93.3 (−3.3)90.0 (−6.7)
MethodBlocks Orig.Blocks−ObjBlocks−BGBlocks−LFlowers Orig.Flowers−ObjFlowers−BGFlowers-L
π0.5\pi_{0.5}76.766.7 (−10.0)66.7 (−10.0)43.3 (−33.3)50.033.3 (−16.7)40.0 (−10.0)26.7 (−23.3)
Evo-RL83.383.3 (0.0)76.7 (−6.7)70.0 (−13.3)76.760.0 (−16.7)50.0 (−26.7)66.7 (−10.0)
π0.5\pi_{0.5} + PF-RL80→93.383.3 (−10.0)86.7 (−6.7)76.7 (−16.7)65→86.776.7 (−10.0)73.3 (−13.3)76.7 (−10.0)

[W2.3 & Q4: Unquantified real-world corrective-data budget]

Thanks for raising this important concern. We additionally report the human intervention details for Evo-RL and PF-RL across four real-world tasks and two rounds of RL training (40 rollouts per task), as summarized in Table R5. All methods follow the same intervention protocol: human operators take over when the robot exhibits execution errors or becomes unable to make meaningful task progress, and return control to the policy once execution recovers to a stable state. As shown in Table R5, the corrective-supervision budgets are broadly comparable between PF-RL and Evo-RL, with overall human-control durations of 4.6 and 5.3 minutes and human-control ratios of 6.9% and 7.2%, respectively. These results support the fairness of the real-world comparison and the effectiveness of PF-RL in real-world manipulation. We have added the human intervention details to the revised Appendix B.2.

Table R5. Human intervention details during real-world RL training (40 rollouts per task).

TaskMethodIntervention SegmentsHuman-Controlled TimestepsHuman-Control Duration (min)Human-Control Ratio (%)
Fold ClothesEvo-RL1128711.67.3
PF-RL820581.15.6
Stack all PlatesEvo-RL810320.65.2
PF-RL78610.54.7
Put Blocks into BucketEvo-RL1018301.06.6
PF-RL1019461.17.8
Fill Vase and Arrange FlowersEvo-RL1338222.18.3
PF-RL1233841.98.6
OverallEvo-RL4295555.37.2
PF-RL3782494.66.9

[Q3: Isolation of the Online Contribution]

Thanks for your insightful comment. We clarify that the original GRPO baseline in Table 6 is initialized from the standard-SFT checkpoint. To isolate the contribution of online progress guidance, we additionally evaluate vanilla GRPO (λprog=0\lambda_{\mathrm{prog}}=0) initialized from the same offline PF-RL checkpoint as the full method. As shown in Table R6, initializing vanilla GRPO from the offline PF-RL checkpoint improves performance over the standard-SFT initialization, but still falls short of the full PF-RL across all four LIBERO suites. This comparison demonstrates the additional benefit of incorporating the online progress advantage beyond offline progress-weighted imitation. We have incorporated this additional comparison into the revised Appendix F.3.

Table R6. Ablation of the online progress advantage under different offline initializations across four LIBERO suites.

Methodλprog\lambda_{\mathrm{prog}}SpatialObjectGoalLongAvg.
GRPO (SFT initialization)097.698.795.296.897.1
GRPO (Offline PF-RL initialization)098.098.896.497.897.8
PF-RL (Ours)1.099.299.598.899.099.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.