- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LVF improves online reinforcement learning on top of a frozen VLA by adding predictions of intermediate events, such as grasping and collision, to the critic’s TD target. The event predictors learn from binary labels, including events observed in failed episodes. Experiments on three real-robot tasks and eleven LIBERO tasks show that the method mainly benefits policies with low initial success.
Critical strengths
-
The paper provides real-robot evidence under a limited interaction budget. After 100 online training episodes, LVF improves over the corresponding actor-critic baseline from 82.5% to 95% on book-long and from 52.5% to 70% on hammer placement. Simulation experiments support the benefit for weaker initial policies.
-
The evaluation includes direct event rewards, potential-based shaping, parameter sweeps, and frozen-predictor experiments. These comparisons help assess how event supervision contributes to policy improvement. Event predictors and labels are not required during deployment.
Critical weaknesses
-
The consequences of the modified objective need further analysis. Adding event predictions directly to the TD target changes the optimization objective, as the authors acknowledge. The policy may therefore favor intermediate events without proportionally improving task completion. The current shaping baseline uses binary event history rather than learned forecasts, leaving open whether shaping with comparable learned predictions could achieve similar gains. The effects on event timing and collision behavior also remain unclear.
-
The supervision cost and scope of applicability are not fully established. Real-robot training requires an operator to mark events throughout collection, but annotation time and robustness to missed or delayed labels are not quantified. The reduced-label experiment is conducted only in simulation with exact event labels. Real-world evaluation covers three tasks, two of which are closely related, with fixed initial positions. The results support improvement within individual tasks, while transfer of the learned event predictors remains untested.
Questions for authors
-
Can you compare the proposed objective with shaping based on learned event forecasts using the same supervision? Does the modified objective affect event timing or collision frequency?
-
How much annotation time is required, and how sensitive is performance to imperfect labels? Can the frozen-predictor protocol retain its benefit on the real robot or under changed object placements?
-
Please discuss the relationship to SRPO, which also addresses sparse rewards by extracting feedback from successful and failed trajectories. A comparison would be informative if the initial policy and interaction budgets can be made comparable.
4 comments
Thank you for the review. As the discussion period ends soon, we wanted to come back to your concern that optimizing intermediate-event signals may not improve task completion.
- Over the seven low-base LIBERO tasks, LVF raises success from 48.7% (RLT) to 65.1%, and it matches or beats RLT on all four real-robot tasks. The policy reaches the grasp much more often than RLT (89% vs. 59% on the Long tasks), and success after the grasp stays at RLT's level (61% vs. 58%). So the extra grasps turn into completed tasks rather than replacing them. Collision rates are 0–3.5% for every method.
- At a large weight (), policies reach the grasp in 87–91% of episodes but succeed in only 1–2%. This is the failure you describe, and it is why we use on every task and robot without retuning (Appendix N).
- The same GVFs as a PBRS potential. PBRS keeps the optimal policy of the sparse task unchanged, so it cannot trade task success for events. With the same learned GVFs, it reaches 56.0%, against 65.1% for LVF. The event signal helps task completion most when it enters the critic's target as discounted credit.
Do these results address your concern? We would be glad to clarify anything further before the discussion closes.
Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.
Thank you for the follow-up and for raising your score. Your question about task completion made us look at how often the policy reaches the grasp and what happens after it, which helped make the paper clearer.
We thank the reviewer for the careful reading.
The reviewer was concerned that "the policy may therefore favor intermediate events without proportionally improving task completion", asked whether "shaping with comparable learned predictions could achieve similar gains", and noted that "annotation time and robustness to missed or delayed labels are not quantified". In the revision, we added
The main new finding is that LVF reaches the grasp much more often than RLT and succeeds after it at about the same rate, so the gain does not come at the cost of finishing the task.
Q1: Consequences of the modified objective
Q2: Supervision cost and imperfect labels
Q3: Relationship to SRPO
We would be glad to answer any further questions.