acceptodds
Under review as a conference paper at ICLR 2027

Layered Value Forecasts for Online RL Fine-Tuning of Vision-Language-Action Models

18 views

Abstract

Vision-language-action (VLA) models provide strong priors for robot manipulation, but adapting them to a new task with online reinforcement learning (RL) is costly, since every episode has to be run on a real robot. Part of this cost comes from the reward, which is sparse and arrives only when the task succeeds. A decision such as closing the gripper on an object is credited only through the value of the states that follow it, and that value stays near zero until the policy completes the task from them, so the critic separates a good decision from a bad one only once success is already frequent. Intermediate events, in contrast, occur in failed episodes as well as successful ones. To exploit this, we propose Layered Value Forecasts (LVF), which augments the critic of an off-policy actor-critic with event-based general value functions (GVFs). Each GVF predicts the discounted probability that one event occurs, such as a grasp or a collision, and is learned from binary event labels with minimal supervision. Their prediction at the chosen action enters the critic's TD target as a cumulant alongside the terminal reward, so the critic receives credit at the steps leading to each event. Across three real-robot manipulation tasks, LVF improves most where success is rare, reaching 95.0% and 70.0% on the two tasks where the initial policy succeeds in 32.5% of trials.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reinforcement learning, VLA, embodied-AI, general value functions

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_neKnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LVF improves online reinforcement learning on top of a frozen VLA by adding predictions of intermediate events, such as grasping and collision, to the critic’s TD target. The event predictors learn from binary labels, including events observed in failed episodes. Experiments on three real-robot tasks and eleven LIBERO tasks show that the method mainly benefits policies with low initial success.

Critical strengths
  • The paper provides real-robot evidence under a limited interaction budget. After 100 online training episodes, LVF improves over the corresponding actor-critic baseline from 82.5% to 95% on book-long and from 52.5% to 70% on hammer placement. Simulation experiments support the benefit for weaker initial policies.

  • The evaluation includes direct event rewards, potential-based shaping, parameter sweeps, and frozen-predictor experiments. These comparisons help assess how event supervision contributes to policy improvement. Event predictors and labels are not required during deployment.

Critical weaknesses
  • The consequences of the modified objective need further analysis. Adding event predictions directly to the TD target changes the optimization objective, as the authors acknowledge. The policy may therefore favor intermediate events without proportionally improving task completion. The current shaping baseline uses binary event history rather than learned forecasts, leaving open whether shaping with comparable learned predictions could achieve similar gains. The effects on event timing and collision behavior also remain unclear.

  • The supervision cost and scope of applicability are not fully established. Real-robot training requires an operator to mark events throughout collection, but annotation time and robustness to missed or delayed labels are not quantified. The reduced-label experiment is conducted only in simulation with exact event labels. Real-world evaluation covers three tasks, two of which are closely related, with fixed initial positions. The results support improvement within individual tasks, while transfer of the learned event predictors remains untested.

Questions for authors
  • Can you compare the proposed objective with shaping based on learned event forecasts using the same supervision? Does the modified objective affect event timing or collision frequency?

  • How much annotation time is required, and how sensitive is performance to imperfect labels? Can the frozen-predictor protocol retain its benefit on the real robot or under changed object placements?

  • Please discuss the relationship to SRPO, which also addresses sparse rewards by extracting feedback from successful and failed trajectories. A comparison would be informative if the initial policy and interaction budgets can be made comparable.

4 comments

We thank the reviewer for the careful reading.

The reviewer was concerned that "the policy may therefore favor intermediate events without proportionally improving task completion", asked whether "shaping with comparable learned predictions could achieve similar gains", and noted that "annotation time and robustness to missed or delayed labels are not quantified". In the revision, we added

  • shaping with the learned GVFs, the grasp rate and success after the grasp, and event timing and collisions (Q1)
  • label-noise and event-removal experiments, the annotation cost, and a fourth real-robot task (Q2)
  • a discussion of SRPO (Q3)

The main new finding is that LVF reaches the grasp much more often than RLT and succeeds after it at about the same rate, so the gain does not come at the cost of finishing the task.

Q1: Consequences of the modified objective

  • The reviewer is right that the event term changes the objective (Section 3.3). In practice, though, neither LVF nor RLT maximizes task return freely. Both actors are pulled toward the VLA's reference chunk by a BC penalty, so the policy stays close to the SFT policy, and RLT's TD3+BC solution is not the return-optimal policy either.
  • On the Long tasks, LVF reaches the grasp in 89% of episodes vs. 59% for RLT, and succeeds after it at about the same rate (61% vs. 58%, Table 1). On Goal/Spatial, both grasp in about 90% of episodes, and success after the grasp is 77% for LVF vs. 65% for RLT.
  • The failure the reviewer describes does happen when the event term dominates. With decayed labels at λ=1.0\lambda = 1.0, where it makes up 0.98–0.99 of the critic's target, the policy grasps in 87–91% of episodes on Long 8 and 9 but succeeds in only 1–2% (Appendix N).
  • Shaping with the same learned forecasts gets part of the gain on the seven low-base tasks (Figure 5, Appendix N), and its policy reaches the grasp no more often than RLT (59%).
MethodSuccess (%)
RLT48.7
Event reward30.0
PBRS, learned GVFs as potential56.0
LVF, γψ=0\gamma_\psi = 045.8
LVF65.1
  • On timing and collisions (Appendix L.4), LVF reaches the grasp more often, in 472 of 528 Long episodes vs. 313 for RLT. Among successful episodes, the median first-grasp step is similar, 61 for LVF and 62 for RLT. Collisions stay infrequent under both, in 3.0% of Goal/Spatial episodes for LVF and 2.1% for RLT, and never in a successful episode.

Q2: Supervision cost and imperfect labels

  • Marking events takes at most K=2K = 2 or 33 key presses per episode, about 2 s of pressing keys (Section 4.1). This is not the operator's full time, since the operator still watches each episode, as is also needed for the success label in RLT.
  • We tested delayed, missed, false and combined marks on Long 0, Goal 3 and Long 8 (Appendix M). On Long 0 and Goal 3, LVF stays above RLT under any single error type at the lower rate. Long 8 is the most sensitive. With all errors combined, the three-task mean drops from 66.9% to 48.1%, close to RLT at 45.4%, which we list as a limitation.
  • The frozen-GVF experiment is in simulation only. There, LVF with GVFs frozen after 20 successful episodes reaches 64.1%, vs. 65.1% with labels throughout. A bonus computed from the labels with the same horizon reaches 64.5% while labels continue, but drops to 50.8% once they stop (Figure 6). We have not tested freezing on the real robot or with changed object placements.
  • We added a fourth real-robot task, mug in drawer, with a drawer to open and three events. SFT succeeds in 27.5% of trials, RLT in 75.0% and LVF in 92.5% (Figure 3). We do not test whether the event predictors transfer across tasks.

Q3: Relationship to SRPO

  • SRPO fine-tunes the VLA's weights and rewards each failed trajectory by its latent distance to the successful ones in the same batch of parallel rollouts. Its online RL runs only in simulation. On the real robot, its authors train offline instead, because online exploration raised safety concerns and resets were slow.
  • We target online RL on a single robot, one episode at a time, where early episodes rarely succeed, so there are usually no successes in a batch to compare against. LVF's events also occur in failed episodes and credit individual chunks rather than whole trajectories.
  • The two could be combined, with SRPO's progress reward replacing our terminal reward. SRPO is now in the related work (Appendix A.4).

We would be glad to answer any further questions.

Thank you for the review. As the discussion period ends soon, we wanted to come back to your concern that optimizing intermediate-event signals may not improve task completion.

  • Over the seven low-base LIBERO tasks, LVF raises success from 48.7% (RLT) to 65.1%, and it matches or beats RLT on all four real-robot tasks. The policy reaches the grasp much more often than RLT (89% vs. 59% on the Long tasks), and success after the grasp stays at RLT's level (61% vs. 58%). So the extra grasps turn into completed tasks rather than replacing them. Collision rates are 0–3.5% for every method.
  • At a large weight (λ=1.0\lambda = 1.0), policies reach the grasp in 87–91% of episodes but succeed in only 1–2%. This is the failure you describe, and it is why we use λ=0.3\lambda = 0.3 on every task and robot without retuning (Appendix N).
  • The same GVFs as a PBRS potential. PBRS keeps the optimal policy of the sparse task unchanged, so it cannot trade task success for events. With the same learned GVFs, it reaches 56.0%, against 65.1% for LVF. The event signal helps task completion most when it enters the critic's target as discounted credit.

Do these results address your concern? We would be glad to clarify anything further before the discussion closes.

Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.

Thank you for the follow-up and for raising your score. Your question about task completion made us look at how often the policy reaches the grasp and what happens after it, which helped make the paper clearer.

Reviewer_xKWhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies online RL on top of a frozen pi0.5, following RLT that a compressed "RL token" serves as the state, and a small actor-critic is trained on it. Under a sparse terminal reward setting, the critic at a grasp step receives almost no signal until the policy already succeeds often. LVF adds one event GVF per hand-chosen event (grasp, contact, collision, placement, open). Each GVF is trained from binary event marks and predicts the discounted probability that the event occurs, conditioned on the state and the action chunk. The weighted signed GVF is then added to the critic's TD target at every transition. During inference, the GVF heads are discarded. For experiment part, the work chooses three real-robot tasks and 11 LIBERO tasks with pi0.5 as base policy, LVF outperforms RLT over all the real-world setting and the seven low-success tasks, which then matches RLT on the four high-success tasks. The same labels used as an event reward or as potential-based shaping do better than the base policy on some tasks but average about 30% on those seven tasks. Freezing the GVFs after 20 successes retains most of the gain.

Critical strengths

My evaluation on strengths goes as follows:

  1. The idea is simple and fits the stated problem. For important online RL of robot, events occur in failed episodes, so a policy-conditioned forecast of them can be learned before success is frequent, and it costs nothing at deployment, the ideas and method flow make sense. The definitions are precise enough to reimplement, and the algorithm and appendix give the full recipe.
  2. The simulation study is designed around the right variable. Tasks are selected by the base policy's success rate, which is appropriate, and the baseline η is tuned separately from LVF's η. The gain and how it comes are clear and helps understanding.
  3. The paper is frank about its own weak points. It states that the cumulant is not potential-based and that estimated Q is not a return estimate, that the real-robot intervals overlap, and that operator labels are still needed for a large part of training.
  4. The frozen-GVF experiment is a useful probe of what the forecasts need. In simulation and in the range of freeze times reported, it shows that the learned GVL keeps supplying a usable cumulant after labels stop.
Critical weaknesses

My evaluation on weaknesses goes as follows:

  1. The main explanation for why event labels fail as rewards is not supported. The paper shows that the event term supplies about two thirds of the critic's value and that LVF therefore needs a larger behavior-cloning coefficient (0.2 instead of 0.1). The event-reward and shaping baselines were run with the smaller RLT coefficient, and the event reward with a single weight, so they did not get the per-run tuning that RLT and LVF got. An event reward paid once per episode is also far smaller in total than a bonus paid at every chunk. The collapse of shaping on Long 8 at the largest weight (0 of 144 episodes) may therefore reflect the actor drifting from the VLA under a mis-scaled BC term, not the way the labels enter the critic. The baselines need per-arm tuning of the BC coefficient and weight, or a matched total bonus.
  2. The shaping argument rests on a premise that does not hold. PBRS preserves the optimal policy only if the potential depends on the state the agent observes. The potential in Appendix J depends on whether an event has already happened in the episode, while the critic sees only the RL token and proprioception (a grasped-then-dropped object still counts as grasped). The statement that shaping is designed to preserve the optimum yet still fails therefore cannot rule out a change of objective. The support is also narrow: the weight sweep covers three Long tasks, while the headline number comes from seven tasks at one weight. On Long 0, shaping at the largest weight reaches 83.3%, greatly outperforms LVF (71.5%) and RLT (44.4%).
  3. The evidence does not separate a learned forecast from simple smoothing of the same labels. With the reported discount (0.95 per step, chunks of 10 steps), a forecast loses about 40% of its value for each chunk further from the event. In effect, the added term is a bonus that fades over two to three chunks before the event, which is shorter than the critic's own horizon. The obvious method ablation is a bonus with the same decay computed directly from the stored trajectories, without a learned forecaster, and a critic with n-step returns. Without it, the gain cannot be attributed to forecasting itself.
  4. The mechanism claimed in the introduction is not measured. The paper says that at a grasp step the critic's target is near zero regardless of grasp quality. All evidence is final task success at a single budget. In the Appendix, the hyperparameter table (Table 4) says evaluations were run at iterations 50, 95, and 140, but only the last one is shown, so there are no learning curves for a sample-efficiency claim. There is no check of how well the critic separates good and bad grasps, no accuracy measure for the forecasts, and no statistics on how often the final policies trigger events without succeeding. The paper concedes that maximizing the augmented critic need not maximize success, yet behavior at large cumulant weights is not examined.
  5. The real-robot evidence is not strong. There are three tasks, one training run per method, and 40 trials from fixed initial positions, with one task at 100% for both methods. The two headline gaps (38/40 against 33/40, and 28/40 against 21/40) are good but not significantly gapped under its setting. On simulation, the labels are noise-free simulator predicates, and the grasp predicate fires on any two-finger contact, which the paper says is weaker than what the operator marks. The operator key-press pipeline is therefore tested only in the three real runs, with no analysis of label delay, missed presses or false presses, such extra human-in-the-loop design should be more carefully reported. The comparison with human-in-the-loop alternatives is weaker than the framing suggests. The introduction presents LVF as an alternative to RLT's human-designated critical segment, but the RLT baseline removes exactly that segment. LVF itself uses operator marks in every episode until the freeze point and needs a hand-chosen event set for each task. The cost of this supervision is not quantified, and the full RLT is not included.
Questions for authors

I have the following questions to ask the authors:

  1. Were the event-reward and shaping baselines run with the BC coefficient and weight tuned per arm? Please report details on all seven low-success tasks, and if the total bonus scale matched.
  2. As a simple variant I thought in Weakness 3, how does LVF compare with a bonus of the same decay computed directly from the stored trajectories, and with an RLT critic using n-step returns?
  3. Please report results at iterations 50 and 95 for all methods, together with a critic diagnostic at event steps and the accuracy of the forecasts.
  4. Please report the operator effort made per episode, and how robust is LVF to delayed, missed or spurious marks? How does it compare with the full RLT when the annotation effort of both is stated?
5 comments

We thank the reviewer for the detailed review.

The reviewer asked for "per-arm tuning of the BC coefficient and weight, or a matched total bonus" (Q1), for "a bonus with the same decay computed directly from the stored trajectories" and "a critic with n-step returns" (Q2), for "results at iterations 50 and 95", "a critic diagnostic at event steps and the accuracy of the forecasts" (Q3), and for "the operator effort made per episode" and robustness "to delayed, missed or spurious marks" (Q4).

Q1: Tuning and scale of the label-based baselines

  • The reviewer is right that the original event reward used RLT's η=0.1\eta = 0.1 and a single weight.
  • Each new variant either uses LVF's own η=0.2\eta = 0.2 and λ=0.3\lambda = 0.3 (matched), or the η\eta and λ\lambda chosen in a single-seed sweep on two pilot tasks, Long 0 and Goal 3 (tuned).
  • Event reward, PBRS and look-ahead use RLT's η\eta with LVF's λ\lambda. They add a shaping term to RLT's reward. A potential-based term shifts QQ by at most one potential value instead of accumulating like LVF's cumulant, and with a state potential it leaves the action gradient of QQ unchanged, so we kept RLT's η\eta.
  • Matching η\eta to LVF keeps the BC strength the same, but it is not a joint search over η\eta and λ\lambda.
  • Results are on the seven low-base tasks over three seeds (Figure 5, Appendix N, Table 18).
Methodη\etaλ\lambdaHow chosenSuccess (%)
RLT0.1–η\eta swept48.7
Event reward0.10.3RLT's η\eta, LVF's λ\lambda30.0
PBRS, learned GVFs0.10.3RLT's η\eta, LVF's λ\lambda56.0
Look-ahead, learned GVFs0.10.3RLT's η\eta, LVF's λ\lambda51.7
LVF, γψ=0\gamma_\psi = 00.20.3matched to LVF45.8
LVF, γψ=0\gamma_\psi = 00.21.0tuned on pilot tasks41.5
Decayed labels0.20.3matched to LVF64.5
Decayed labels0.21.0tuned on pilot tasks53.6
LVF, γψ=0.99\gamma_\psi = 0.990.10.1tuned on pilot tasks67.0
LVF0.20.3η\eta and λ\lambda swept (App. D, I)65.1
  • The tuned scores are lower than the matched ones because the pilot tasks are easier than the hardest tasks. On Long 0 and Goal 3, λ=1.0\lambda = 1.0 scored best. On Long 8 and 9, the same λ\lambda makes the added term 0.98–0.99 of the critic's target, and success drops to 0–4% (Appendix N).
  • At the matched λ=0.3\lambda = 0.3, the decayed-labels bonus makes up a share of the critic's target similar to LVF's (0.73–0.92 vs. 0.71–0.86, Table 20).
  • The reviewer is also right about the PBRS premise. The label-history potential depends on what happened earlier in the episode, so it does not guarantee policy invariance. Appendix J now says this.

Q2: Bonus with the same decay, and n-step returns

  • We ran both. The hindsight bonus with the same decay reaches 64.5%, against 65.1% for LVF. An RLT critic with 3-chunk returns, with η\eta and nn tuned on the pilot tasks, reaches 48.3%.
  • Under continuous annotation, the matched-decay result identifies temporally extended event credit as the main driver.
  • The forecaster provides a distinct benefit when annotation stops. In simulation, frozen LVF retains 64.1%, versus 50.8% for the hindsight-label baseline (Figure 6).

Q3: Learning curves and diagnostics

  • We report 50, 95 and 140 episodes for RLT and LVF (Appendix K). On the seven low-base tasks, LVF is ahead at 50 (35.8% vs. 26.8%), tied at 95 (45.0% vs. 45.4%), and ahead at 140 (65.1% vs. 48.7%).
  • The contact and grasp forecasts correlate with realized outcomes at 0.89 and 0.82 (Appendix L.2).
  • LVF's critic does not rank grasps by eventual success better than RLT's (AUROC 0.58 vs. 0.70 at 100 episodes, Appendix L.3).
  • What changes is the policy. On the Long tasks, LVF reaches the grasp in 89% of episodes vs. 59% for RLT, with similar success after it (61% vs. 58%, Table 1).

Q4: Operator effort, label noise and RLT's supervision

  • Marking adds at most K=2K = 2 or 33 key presses per episode, about 2 s of pressing keys, while the operator still watches each episode (Section 4.1).
  • With delayed, missed and false marks on Long 0, Goal 3 and Long 8 (Appendix M), LVF stays above RLT on Long 0 and Goal 3 under each single error type at the lower rate. With all three combined, the mean drops from 66.9% to 48.1%, vs. 45.4% for RLT, which we list as a limitation.
  • We did not run RLT with its original supervision, where in each episode the operator hands control to RL at the critical phase and may take over with teleoperated corrections. Our goal was minimal supervision, where the operator only marks events and never controls the robot, so the agent finds better actions by itself.
  • We added a fourth real-robot task, mug in drawer (LVF 92.5%, RLT 75.0%).

We would be glad to answer any further questions.

Thank you for the detailed response and the new experiments. My concerns about the PBRS premise, the n-step critic and the operator cost are resolved, and the drawer task makes the real-robot results more consistent. I have raised my soundness score.

Remaining concerns:

  1. With continuous labels, a hindsight bonus with the same decay matches LVF (64.5% vs. 65.1%), so the gain comes from temporally extended event credit rather than from learned forecasts. The forecaster’s specific value rests on the label-stopping experiment (64.1% vs. 50.8%). If new episodes receive no bonus there while old ones keep it, the critic is told that events stopped mattering. A fairer baseline would train an event detector on the labeled episodes and keep computing the hindsight bonus from its predictions.
  2. It is unclear whether the same procedure keeps LVF out of the event-dominated regime on new tasks or on the robot. LVF’s critic ranks grasps by eventual success worse than RLT’s, and pilot-task selection did not transfer for the control variants.
  3. The improvement is not robust to noise.
  4. LVF and RLT are similiar at 95 episodes and the gap appears only at 140, with non-monotone per-task curves. More evaluation points or a longer budget would show whether it persists.
  5. Minor: event reward remains a single untuned setting and falls below RLT, which slow propagation alone does not explain.

The paper is now more careful, but the supported contribution is a decayed event bonus, and the forecaster’s specific advantage is not yet established. I maintain my overall rating.

Thank you for the follow-up. We are glad the PBRS, the n-step critic, the operator cost and the real-robot results are now clear.

  • We agree that the GVF and the hindsight bonus have the same target. Each GVF learns the expected discounted event bonus by TD (Eq. 9), while the hindsight bonus is a Monte Carlo estimate of the same return along the stored episode. So with labels in every episode they should perform about the same, and they do (65.1% vs. 64.5%).
  • We also agree that our stopped-labels baseline was unfair, because new episodes got no bonus. So we ran the experiment you suggested. We train an event detector on the labels and compute every bonus, in old and new episodes, from its detections, re-scoring stored episodes as the detector improves. The labels are only used to train the detector, the same way LVF only uses them to train its GVFs. The detector is frozen with the same rule as LVF's GVFs (after 20 successful episodes), with the same events, η\eta and λ\lambda.
MethodGoal 3Sp 5Sp 9L0L7L8L9Mean
RLT68.893.133.344.459.722.918.848.7
MC, labels stopped (hindsight bonus)62.566.043.867.462.533.320.150.8
MC, frozen detector78.585.438.971.569.425.015.354.9
LVF, frozen GVFs83.388.246.564.688.941.036.164.1
  • For contact and grasp, the detector's precision and recall are mostly above 90%, usually within one chunk, so the gap does not come from missed detections of these events.

  • The detector recovers some of what was lost when labels stopped (50.8% to 54.9%), but frozen GVFs keep more of the gain (64.1%). In this comparison, learning the event return as a forecast works better after labeling stops than computing it in hindsight from a detector.

  • This also supports our main finding. What helps under sparse reward is a discounted event return in the critic's target. Event rewards, shaping and a classifier without a horizon use the same labels and do not get this gain. LVF learns that return directly as a GVF, without a separate detector or relabeling at the end of each episode. We added this to Figure 6 and Appendix N.

  • On the event-dominated setting, we used the same λ=0.3\lambda = 0.3 and η=0.2\eta = 0.2 on all 11 LIBERO tasks and all 4 real-robot tasks, without retuning. The event term's share of the critic target stayed between 0.71 and 0.86 on every low-base task, and it can be monitored during training. The critic's weaker ranking of grasps does not lead to failure here, because the gain comes from the policy reaching the grasp more often, with similar success after it (Table 1).

  • For robustness to noise, most of the drop in the mean comes from Long 8. Even with delayed, missed and false marks combined, LVF stays above RLT on Long 0 (50.0% vs. 44.4%) and Goal 3 (82.6% vs. 68.8%). We agree that heavy noise on the hardest task removes the gain, and we list this as a limitation.

  • We agree that more evaluation points would help. Our goal, however, is improvement within a limited interaction budget, about 100 to 140 episodes, so that the method is practical on a real robot. A much longer budget falls outside this setting. Within it, on the real robot, LVF matches or improves on RLT on all four tasks after 100 episodes.

  • On the event reward, you are right that it uses a single setting. The tuned label-based baseline is the hindsight bonus at LVF's η\eta, which reaches 64.5%.

We would be happy to answer any further questions.

Reviewer_BTdaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Layered Value Forecasts (LVF), an online RL fine-tuning method for vision-language-action (VLA) policies that augments an off-policy actor-critic with event-based general value functions (GVFs). Each GVF predicts the discounted probability that a specific intermediate event (e.g., grasp, collision) will occur under the current policy, and the ensemble of GVF predictions is injected as a cumulant into the critic’s TD target, providing dense, action-conditioned credit assignment even when terminal successes are rare. The authors demonstrate that LVF yields substantial gains over a strong RL-token (RLT) baseline when the base policy’s success is low, matching RLT when the base policy is already strong, across three real-robot tasks and eleven LIBERO tasks (with ablations contrasting LVF against directly using event labels as rewards or potential-based shaping).

Critical strengths

Addresses a core bottleneck in VLA online RL (sparse and delayed rewards) with a solution that fits naturally with off-policy actor-critic and requires only minimal binary event labels.

Critical weaknesses

The event-reward and PBRS baselines use the same labels but do not include a variant that uses learned event-prediction signals directly as dense rewards; this would isolate whether the “forecasting-and-layering” versus “direct reward” distinction is the only driver of gains.

Questions for authors
  1. Have you tried using a learned event classifier’s probability (from visual/proprio inputs) as a direct dense reward, or as a PBRS potential, to test whether “forecasting” per se (with termination) or just probability signals explain the gains?

  2. What is the runtime/compute overhead of training K GVFs in parallel? Do you observe scaling issues with increasing K, and is there evidence that a shared multi-head GVF network would retain benefits while reducing compute?

  3. How does LVF perform if the event set is incomplete or slightly mis-specified (e.g., missing a key collision type)? Any evidence of negative transfer when λ is moderate?

3 comments

We thank the reviewer for the careful reading.

The reviewer noted that our baselines "do not include a variant that uses learned event-prediction signals directly as dense rewards", which would show whether "the 'forecasting-and-layering' versus 'direct reward' distinction is the only driver of gains" (Q1), and asked about the "runtime/compute overhead of training K GVFs in parallel" (Q2) and how LVF performs "if the event set is incomplete or slightly mis-specified" (Q3). In the revision, we added

  • the same learned GVFs used as a dense reward, a PBRS potential and look-ahead advice, a hindsight bonus with the same decay, and how often each final policy reaches the grasp (Q1)
  • a compute benchmark with up to eight event channels (Q2)
  • event-removal experiments and delayed, missed and false labels (Q3)

Q1: Learned event probabilities as a dense reward or potential

  • LVF with γψ=0\gamma_\psi = 0 is exactly the learned classifier the reviewer describes. With γψ=0\gamma_\psi = 0, the GVF target in Eq. (9) reduces to the event mark of the next chunk, so each GVF becomes a classifier of whether its event occurs within the next chunk, trained on the same inputs and labels. Its probability then enters the critic's target at every transition as a dense reward.
  • For the potential, we added PBRS with the learned GVFs as a state potential, Φ(x)=∑iwiψˉi(x,μθ(x))\Phi(x) = \sum_i w_i \bar\psi_i(x, \mu_\theta(x)), and look-ahead advice with Φ(x,a)\Phi(x, a).
  • All variants use the same events, backbone and budget, on the seven low-base tasks over three seeds (Figure 5, Appendix N).
MethodSuccess (%)
RLT48.7
Event reward (labels at the event)30.0
Learned classifier as dense reward (LVF, γψ=0\gamma_\psi = 0)45.8
PBRS, learned GVFs as potential56.0
Look-ahead, learned GVFs51.7
Hindsight decayed labels64.5
LVF (γψ=0.95\gamma_\psi = 0.95)65.1
LVF (γψ=0.99\gamma_\psi = 0.99)67.0

These results point to three findings.

  • With labels in every episode, the gain comes from temporally extended event credit. A bonus computed in hindsight from the labels, with the same decay as the GVF target, reaches 64.5%, close to LVF at 65.1%, and a longer horizon (γψ=0.99\gamma_\psi = 0.99) reaches 67.0%. Without a horizon, the learned classifier reaches 45.8%, below RLT at 48.7%, and the learned potentials reach 56.0% and 51.7%.
  • We learn forecasts so that labeling can stop. The hindsight bonus needs a label in every new episode, and when labels stop after 20 successful episodes, it falls to 50.8%, close to RLT. In simulation, LVF with its GVFs frozen at the same point still reaches 64.1% (Figure 6).
  • The event credit makes the policy reach bottleneck states such as the grasp far more often. On the Long tasks, LVF's final policies reach the grasp in 89% of episodes against 59% for RLT, and succeed after it at a similar rate, 61% against 58% (Table 1). The learned classifier reaches the grasp in 68% of episodes but succeeds after it in only 44%, and the forecast-based potential reaches it as often as RLT.

Q2: Compute overhead and scaling with K

  • The GVF heads add 5.96M trainable parameters, 0.16% of the VLA (Appendix E, Table 8).
  • With four channels, 1,000 gradient updates take 26.3 s against 9.9 s without GVFs, so a simulated iteration takes 52.1 s against 35.8 s.
  • Our RLT baseline trains the same heads with λ=0\lambda = 0, so all comparisons are at equal compute.
  • The time per update step grows linearly with K, from 7.7 ms with no channels to 31.5 ms with eight, about 3 ms per channel, so we did not observe scaling issues up to K = 8.
  • We did not test a shared multi-head network, which would reduce this cost further.

Q3: Incomplete event sets and negative transfer

  • We removed one event at a time on Long 0, Goal 3 and Long 8 (Appendix M). Removing the grasp event lowers the three-task mean from 66.9% to 53.0%, still above RLT at 45.4%, although Long 8 falls below RLT. Removing the collision event on Goal 3 lowers success from 86.1% to 75.0%, still above RLT at 68.8%.
  • With 30% false marks per episode and event, the mean falls to 44.9%, close to RLT, and with delayed, missed and false marks combined, it falls to 48.1%.
  • The clearest negative transfer appears at large λ\lambda. In our λ\lambda sweep on Spatial (Appendix I), where RLT already reaches 0.979, success stays at 0.979 up to λ=0.3\lambda = 0.3 and drops to 0.729 at λ=0.7\lambda = 0.7.
  • At the λ=0.3\lambda = 0.3 we use, LVF is close to RLT on the four LIBERO tasks where the base policy is already strong (86.1% vs. 88.9%). It is below RLT on four tasks (Spatial 5, Long 1, Long 4 and Long 6). The per-seed results (Appendix F) overlap on each, but with three seeds we cannot rule out a small regression.

We would be glad to answer any further questions.

Thank you for the review. As the discussion period ends soon, we wanted to point to the two comparisons that answer your main question, whether learned event predictions used directly would give the same gain.

  • LVF with γψ=0\gamma_\psi = 0 turns each GVF into a classifier of whether its event happens in the next chunk, trained on the same inputs and labels, and adds its probability to the critic's target at every step. It reaches 45.8%, below RLT at 48.7%. So predicting the event is not enough. The signal needs a horizon that credits the chunks leading up to the event.
  • With the same learned forecasts as a state potential, success is 56.0%, against 65.1% for LVF. PBRS keeps the optimal policy of the sparse task unchanged, so it can only speed up learning. LVF instead puts the forecast into the critic's target as event credit, which is what moves the policy toward the event.
  • The forecast supplies discounted event credit over a few chunks. Learning it as a GVF also keeps more of the gain once labeling stops than computing the same credit from a frozen event detector (64.1% vs. 54.9%).

Do these comparisons address your concern? If anything is still unclear, we would be glad to clarify before the discussion closes.

Thank you again for your review. Since the discussion closes today, here is a short recap of how we answered your three questions.

  • You asked whether using learned event predictions directly, as a dense reward or a PBRS potential, would give the same gain. LVF with γψ=0\gamma_\psi = 0 is exactly that classifier, and it reaches 45.8%, a bit below RLT at 48.7%. PBRS with the same learned GVFs, after tuning the BC coefficient and shaping weight, reaches 56.0%, while LVF reaches 65.1% (Figure 5, Appendix N). LVF's policy also gets to the grasp much more often than RLT's, 89% vs. 59% on the Long tasks.
  • On compute, the GVF heads add only 0.16% of the VLA's parameters, and each extra channel costs about 3 ms per update step, growing linearly up to K = 8. Our RLT baseline trains the same heads, so the comparisons are at equal compute (Appendix E).
  • On incomplete or mis-specified events, LVF stays above RLT on average when we remove the grasp event (53.0% vs. 45.4%), and on Goal 3 when we remove the collision event (75.0% vs. 68.8%). At the λ\lambda we use, we don't see significant negative transfer on tasks where the base policy is already strong. Heavy false or combined label noise does bring LVF back to RLT's level, and we list that as a limitation (Appendix M).

We hope this answers your questions, and we'd be grateful if you could take it into account in your final assessment.

Reviewer_9n5EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Layered Value Forecasts (LVF), an approach for improving online reinforcement learning fine-tuning of vision-language-action (VLA) models under sparse terminal rewards. The main motivation is that standard actor-critic methods struggle with credit assignment when successful trajectories are infrequent. Nevertheless, intermediate manipulation events, such as grasps and collisions, can be observed in both successful and unsuccessful episodes. The authors propose leveraging this information through event-specific general value functions (GVFs), which estimate the discounted likelihood of future events under the current policy.

The method builds directly upon RL Token (RLT), retaining the frozen π0.5 backbone, compressed RL-token representation, lightweight actor-critic, and behavior-cloning regularization. Its main modification is introducing the weighted predictions of event-specific GVFs as additional cumulants in the task critic's TD target. The objective is to provide a denser training signal without modifying the pretrained VLA or increasing the deployment-time model size.

The paper evaluates LVF on three real-world manipulation tasks and several selected LIBERO tasks. On the physical robot, LVF reaches high success rates outperforming RLT. In simulation, LVF improves the success rate over RLT in most cases, but not always. The authors also provide comparisons with PLD, direct event-reward shaping, and ablations on event supervision suggesting interesting future works.

Critical strengths
  1. Fair and well-controlled comparison with RLT. The authors carefully isolate the contribution of event forecasts by maintaining the same underlying VLA, actor-critic architecture, and training pipeline as RLT. I particularly appreciate the matched network capacity and independently tuned hyperparameters, which make the reported improvements more convincing.

  2. Promising results supported by real-world experiments. LVF achieves meaningful improvements over RLT, particularly on tasks with low initial success rates. The inclusion of physical robot experiments is especially valuable, as it demonstrates the practical potential of the approach under limited online interaction budgets, that is a very relevant problem in robotics.

  3. Effective use of intermediate events to address sparse rewards. The idea of leveraging event predictions learned from both successful and unsuccessful trajectories is practically appealing, especially when successful episodes are rare. The ablations comparing LVF against direct event rewards and potential-based shaping provide useful evidence that predictive event supervision can improve learning efficiency.

Critical weaknesses
  1. Limited methodological novelty beyond RLT. The proposed method largely retains the RLT architecture and training procedure, with the main modification being the introduction of event-specific GVF predictions into the critic's TD target. While this is a reasonable extension, GVFs and predictive reward shaping are well-established concepts. I am not fully convinced that this constitutes a sufficiently substantial methodological contribution for ICLR.

  2. Insufficient comparison with existing VLA-RL methods. The experimental evaluation focuses primarily on RLT and PLD, while several relevant approaches, including πRL, VLA-RL, SimpleVLA-RL, DSRL, and EXPO-FT, are not experimentally compared. Although differences in architectures and training budgets make direct comparisons challenging, stronger positioning against these methods is important to establish the significance of the contribution. In particular, VLA-RL's use of learned process rewards is closely related to the paper's motivation. Additionally, all experiments are conducted using a single VLA architecture, π0.5. Although the results are promising, it remains unclear whether the proposed event-forecast mechanism generalizes to other VLA models with different representations and action-generation mechanisms. Evaluating LVF on additional VLA backbones would strengthen the generality and broader applicability of the proposed approach.

  3. Inconsistent improvements over RLT. LVF underperforms RLT on several LIBERO tasks, including Spatial 5, Long 1, Long 4, and Long 6. While accurately detected intermediate events do not necessarily guarantee better performance, these regressions raise questions about when the additional forecasts are beneficial. The paper lacks a convincing analysis of whether these failures result from inaccurate predictions, uninformative events, or conflicting optimization signals.

  4. Insufficient isolation of the proposed mechanism. The event-reward and potential-based shaping baselines use event labels rather than the same learned GVF predictions employed by LVF. Consequently, these comparisons do not fully establish whether the improvements arise from the specific layered-value formulation or simply from incorporating learned dense predictive signals. A comparison using the same GVFs through alternative shaping or auxiliary critic formulations would considerably strengthen the paper's central claim.

Questions for authors
  1. Novelty and comparison with existing VLA-RL methods. Could the authors clarify the main methodological contribution beyond extending RLT with event-specific GVFs? In particular, how does LVF compare conceptually and empirically with πRL, VLA-RL, SimpleVLA-RL, DSRL, and EXPO-FT? I recognize that differences in architectures and training budgets may make direct comparisons challenging, but additional comparison results where feasible, especially with methods using learned intermediate rewards, would help establish the significance of the contribution.

  2. Performance regressions relative to RLT. Why does LVF underperform RLT on Spatial 5, Long 1, Long 4, and Long 6? Are these regressions caused by inaccurate event predictions, uninformative event definitions, or conflicting optimization signals? An analysis of these cases, potentially including event-specific ablations, would help clarify when the proposed method is beneficial and when it may degrade performance.

  3. Isolating the contribution of the layered critic. Could the authors compare LVF against approaches that use the same learned GVF predictions through alternative mechanisms, such as potential-based reward shaping or auxiliary critic features? The current shaping baselines use event labels rather than learned forecasts, making it difficult to determine whether the improvements come from the specific layered-value formulation or simply from introducing dense predictive supervision.

  4. Generalization across VLA architectures. Since all experiments use π0.5, is there evidence that LVF generalizes to other VLA architectures? Even a limited experiment with one or two different backbones would help establish whether the benefits arise from the proposed event-forecast mechanism rather than properties specific to the RLT/π0.5 setup.

7 comments

We thank the reviewer for the careful review.

The reviewer was "not fully convinced that this constitutes a sufficiently substantial methodological contribution", noted that LVF "underperforms RLT on several LIBERO tasks", and asked whether the gains come "from the specific layered-value formulation or simply from incorporating learned dense predictive signals". Our setting is low-budget online RL on a single robot, one episode at a time for about 100 episodes. In the revision, we added

  • controlled comparisons that use the same learned GVFs or the same labels in other ways (Q1, Q3)
  • per-seed results for the tasks where LVF is below RLT (Q2)
  • a fourth real-robot task and an analysis of the final policies

These show that the main driver is event credit spread over a horizon in the critic's target, that the learned GVFs keep this benefit when labeling stops in simulation, and that LVF's policies reach bottleneck states such as the grasp far more often.

Q1: Contribution and comparison with other VLA-RL methods

  • Our goal was to help online RL on a real robot without a person controlling the robot. HIL-SERL relies on an operator who corrects the robot mid-episode, and in RLT the operator hands control to RL at the critical phase of each episode and may take over with corrections. We wanted the operator only to mark when an event such as a grasp happens, so the agent finds better actions by itself.
  • LVF learns predictions of these events as GVFs and adds them to the critic's target. It shares RLT's backbone, RL token, networks and optimizer, with the BC coefficient tuned separately for each (0.1 for RLT, 0.2 for LVF).
  • With the same labels, an event reward reaches 30.0%, a learned event probability without a horizon 45.8%, and shaping with the learned forecasts 56.0%, while LVF reaches 65.1% on the seven low-base tasks (Figure 5).
  • LVF makes the policy reach bottleneck states far more often. On the Long tasks, its final policies reach the grasp in 89% of episodes against 59% for RLT, and succeed after it at a similar rate, 61% against 58% (Table 1).
  • πRL, SimpleVLA-RL and VLA-RL update the VLA's weights on many parallel simulated rollouts, which does not fit one episode per iteration, so we did not run them (Section 4.2). VLA-RL's process reward is trained on successful trajectories split at gripper changes and then frozen, while LVF's forecasts are learned online from all episodes.
  • DSRL and EXPO change how actions are produced. DSRL chooses the noise fed to the frozen VLA, and EXPO edits the base policy's actions. Both still learn from the sparse terminal reward, and DSRL rarely collects a success when initial success is low (Lu et al., 2026). LVF instead changes what the critic learns from. We did not run them, so we make no claim about which performs better (Appendix A).

Q2: Regressions relative to RLT

  • LVF's benefit is concentrated on tasks where the base policy is weak (Figure 7), and the reviewer is right that it is below RLT on four tasks. The table gives per-seed successes out of 48 (Appendix F).
TaskRLT (seeds)LVF (seeds)RLT / LVF pooled
Spatial 547, 42, 4546, 34, 4693.1% / 87.5%
Long 148, 47, 4648, 45, 4597.9% / 95.8%
Long 437, 40, 3330, 32, 4076.4% / 70.8%
Long 643, 42, 4244, 36, 3988.2% / 82.6%
  • The seed ranges overlap on every task, and on Spatial 5 and Long 6 one LVF seed accounts for most of the gap. With three training seeds, we cannot rule out a small regression.
  • Long 1, 4 and 6 already have a frequent terminal reward, so the forecasts add little.
  • On Spatial 5, the grasp forecast is least accurate (correlation 0.44 vs. 0.75–0.91 elsewhere, Appendix L.2), and in the single-seed λ\lambda sweep LVF reaches 0.979 with η=0.1\eta = 0.1 instead of 0.2 (Appendix I).
  • Our event ablations were run on Long 0, Goal 3 and Long 8, not on these tasks.

Q3: Using the same learned GVFs through other mechanisms

  • The table compares these variants with the same learned GVFs, events and budget, on the seven low-base tasks (Figure 5, Appendix N).
MethodSuccess (%)
RLT48.7
PBRS, learned GVFs as potential56.0
Look-ahead advice, learned GVFs51.7
LVF, γψ=0\gamma_\psi = 0 (no horizon)45.8
LVF65.1
  • Shaping with the same forecasts gets part of the gain, and its policy reaches the grasp as often as RLT (Table 1).

Q4: Other VLA backbones

  • All experiments use π0.5, which we state as a limitation.
  • LVF relies only on the representation and action chunks a VLA provides. The event GVFs and the critic take these as inputs, and the VLA's weights and the way it generates actions are never changed.
  • Any VLA that exposes a compact state and action chunks, autoregressive or flow-based, could in principle be used. Building this interface for other backbones is out of the scope of this paper.

We would be glad to answer any further questions.

Thank you for the detailed rebuttal and additional experiments. I really appreciate and recognize the new real-world task and the further ablations, which address several of my concerns.

My main remaining concern is methodological novelty. In particular, the Monte Carlo hindsight baseline achieves 64.5% versus 65.1% for LVF, suggesting that much of the gain comes from propagating intermediate-event credit rather than learning GVFs specifically. While the frozen-GVF results are promising, the ability to maintain performance after stopping event supervision is demonstrated only in simulation, whereas the real-world experiments still require event annotations throughout training.

Could the authors clarify why GVF-based PBRS was evaluated with a single fixed hyperparameter setting rather than independently tuning the BC coefficient and shaping weight, as was done for LVF?

Overall, the revision strengthens the paper, but I remain somewhat concerned about the novelty of the contribution beyond RLT.

Thank you for the follow-up.

Hindsight bonus

  • The hindsight bonus is not an existing method. We built it as a diagnostic for this revision, after Reviewer xKWh asked for a bonus with the same decay computed from the stored trajectories. It takes the true event labels and computes, along each stored episode, the same discounted event return that each GVF learns by TD. It is a Monte Carlo version of LVF's target, so with labels in every episode we expected the two to perform about the same, and they do (64.5% vs. 65.1%).
  • So this result tells us where the gain comes from, not that a similar method already exists. Event credit has to enter the critic's target as a discounted return over a few chunks, and event rewards, shaping and a classifier without a horizon do not get there (Figure 5).
  • The two separate once labels stop. We added a second experiment in which every bonus comes from a learned event detector, trained only on the labels and frozen with the same rule as LVF's GVFs. Over the seven low-base tasks, frozen GVFs reach 64.1%, the frozen detector 54.9%, and the bonus with no labels after the stop 50.8% (Figure 6, Appendix N).

Contribution beyond RLT

  • In RLT, the operator controls the robot during training, handing control to RL at the critical phase and taking over with teleoperated corrections. In LVF, the operator never controls the robot and only presses a key when an event like a grasp happens, so the agent has to find better actions by itself. Without those corrections, RLT's critic only sees the sparse terminal reward, and LVF is our way around this.
  • Our contribution is a way to get the benefit of intervention without anyone controlling the robot. The operator just marks events, and LVF learns from them how much each action brings the robot closer to an event, with one extra module next to the critic. In practice, LVF reaches the grasp much more often than RLT (89% vs. 59% on the Long tasks) and raises the seven-task mean from 48.7% to 65.1%.
  • It would help us to know which concern remains. Is it that a similar method already exists, or that the improvement over RLT is not large enough? If the reviewer has particular related work in mind, we would be glad to see it and explain how LVF differs.

Other concerns

  • We agree. Freezing the GVFs was only tested in simulation, and our real-robot runs use labels throughout. We list this as a limitation. On the robot, labeling costs at most 2 or 3 key presses per episode, and testing frozen GVFs there is our next step.

  • PBRS went through the same tuning as the other variants. On the two pilot tasks, Long 0 and Goal 3 (seed 0), we swept η∈{0.1,0.2}\eta \in \{0.1, 0.2\} and the shaping weight λ∈{0.1,0.3,1.0}\lambda \in \{0.1, 0.3, 1.0\}.

η\etaλ\lambdaLong 0Goal 3Mean
0.10.10.4380.5830.510
0.20.10.5420.5000.521
0.10.30.6250.7080.667
0.20.30.3540.5420.448
0.11.00.3330.3130.323
0.21.00.2080.3960.302
  • The setting we reported, η=0.1\eta = 0.1 and λ=0.3\lambda = 0.3, was the best one. On the same tasks and seed, RLT gets 0.615 and LVF 0.854. Larger weights only made things worse. We swept look-ahead with the learned GVFs the same way, and its reported setting was also the best. We have added this in Appendix N (Table 18).

We would be glad to answer any further questions.

Comments on the paper

1 comment

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.