- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tackles adaptation of hierarchical robot policies beyond the data used for pretraining, to adapt to never-seen challenges. The authors introduce two frameworks: HARP, which uses hindsight corrections of interactions by a VLM to fine-tune the high-level policy, and FLUTE, a foresight critic, which uses another VLM to choose the best action among a given set based on pre-defined criteria. They evaluate them individually and combined in a subset of the Bench2Drive and Fail2Drive benchmarks, where they yield a substantial improvement in mean performance.
Critical strengths
- Clear presentation with good visuals and reproducability: The paper reads nicely, the components are explained well and the visuals aid understanding. All details are given in the appendix, so I am certain that even without the code (which is also attached), results are very reproducible.
Critical weaknesses
- Novelty: All individual components are well-studied, there are no novel ideas to make the combination work, but rather just combining them.
- Evaluation: evaluation falls short in several ways:
- Baselines: While the behavioral cloning policies are varied and recent, the autonomous improvement ones are lacking and only used in one of the two experiments. The authors only consider residual RL (an older method from 2018) and YAY robot (which in its original formulation is human-in-the-loop, not autonomous). They cite several in their related work, but do not compare their method against them in the evaluation.
- FLUTE seems to depend on hand-crafted criteria, which are then evaluated by the LLM. No sensitivity to these are reported, and the ones chosen seem to be directly related to the evaluation metric, the driving score. (The driving score said to combine "collision avoidance, traffic-rule following, and progress in the route", FLUTE evaluates "expected progress along the route, alignment with [the language instructions], correctness of the proposed action, and whether the proposed action will lead to a collision, a traffic violation, or an off-road violation", which seems like a direct optimization towards the metric.)
- Low confidence results: Driving scores are calculated based on 3 seeds, and SDs of individual results are not reported (only SDs of what seems to be the means of the route driving scores, averaged over the seeds are given), so no statistical significance can be derived from the results.
- No inference times are reported: FLUTE uses a VLM-based metric for every action decision, but no inference times are reported. This makes the described use-case of real-world robotics questionable, as capable VLMs usually have higher inference times than uses such as autonomous driving require.
- The subset of the benchmarks is selected by the worst performances of the BC baselines. This may inflate the reported improvements.
- The weight parameters are never ablated, their values are chosen without an apparent reason.
Questions for authors
- Is there a good reason for using the specific improvement methods as baselines and omitting others from the related work?
- Is the 20Hz the simulator runs at real-time or simulation time when including the inference time needed for running FLUTE? Is real-time inference on an autonomous car possible?
- What are the confidence intervals of the improvements in driving score?
- What is the sensitivity of the methods to the chosen VLM? Why were different VLMs chosen for the two methods?
6 comments
[Weakness 2.2] "...FLUTE seems to depend on hand-crafted criteria…"
The criteria used in FLUTE (progress along the route, collisions, traffic violations, off-road violations) are commonplace reward terms for autonomous driving and for navigation in general. Prior works, such as Raw2Drive (Yang et al., 2025) and LSTC (Hu et al., 2024) use similar criteria, wherein progress is rewarded, and violations are penalized.
We acknowledge that these objectives overlap with components of the Bench2Drive driving score. However, FLUTE does not directly observe or optimize the driving score; instead, the VLM estimates the likely consequences of candidate actions from the available observations. The same criteria are applied across routes without route-specific tuning. We agree that sensitivity to the relative weights is an important question and will clarify the rationale behind our coefficient choices in the revised manuscript.
[Weakness 2.3 & Question 3] …no statistical significance can be derived from the results… What are the confidence intervals of the improvements in driving score?
Due to the compute-intensive nature of the CARLA driving simulator and running inference with large VLAs, prior works (such as SimLingo (Renz et al., 2025)) typically use a maximum of three seeds.
As the routes differ significantly and therefore have different variations in performance, we used the typical method for computing performance on Bench2Drive and Fail2Drive, which is computing the per-seed averages and then computing the standard deviation across these seeds. This is reflective of prior work such as SimLingo, BLUE, ORION, and DriveMoE.
We report here 95% confidence intervals on improvement for the Bench2Drive and Fail2Drive for HARP + FLUTE relative to the strongest base policy we evaluated with (SteerVLA high-level policy and the low-level policy)
- Bench2Drive: +5.26 [-5.90, +16.42]
- Fail2Drive: +12.32 [+0.67, +23.97]
The result on Bench2Drive indicates the estimated mean gain is positive, but the confidence interval includes 0. However, we invite the reviewer to review the examples in Figures 1,2, and 4 that demonstrate qualitatively the improvements HARP + FLUTE can achieve. On Fail2Drive, the estimated mean gain and its 95% confidence interval lie entirely above zero.
We also hope the reviewer understands that we are resource-constrained, and the experiments in the paper took over 2000 GPU hours across policies, routes, and seeds with access to only 8 H200 GPUs. While we would have strongly preferred to increase the number of seeds used in these experiments, this is a significant undertaking.
We will include this version of the results and more discussion in our updated manuscript.
[Weakness 2.4 & Question 2] No inference times are reported… Is the 20Hz the simulator runs at real-time or simulation time?
The reported 20 Hz refers to simulation time. CARLA advances synchronously, allowing inference to finish before simulation proceeds. We select action chunks every three simulator steps, corresponding to 0.15 seconds of simulated driving.
The policies have the following inference times. We also report the amortized inference time, as we only query the high-level policy every 6 steps and the low-level policy every 3 steps:
| Model | ms/query | ms/simulator step |
|---|---|---|
| SteerVLA high-level | 1186 | 198 |
| SteerVLA low-level | 144 | 48 |
| high-level | 329 | 55 |
| low-level | 76 | 25 |
As described in Appendix B, for FLUTE, we first prompt the VLM to generate a short description of the scene, then score each candidate. We overlap the description generation with the generation of each candidate, as these two steps are independent.
| Component | Time (s) |
|---|---|
| Description generation | 5.50 |
| Candidate scoring (4 candidates) | 2.16 |
| Total | 7.66 |
We acknowledge that this implementation does not meet real-time autonomous driving requirements. Our primary objective is to investigate whether VLM-based foresight evaluation can improve policy behavior in challenging and out-of-distribution scenarios, rather than to demonstrate real-time deployment. Reducing the computational cost of candidate generation and evaluation remains an important direction for future work. We will clarify this limitation and report the measured inference latency in the revised manuscript.
[Weakness 2.5] "The subset of the benchmarks is selected by the worst performances of the BC baselines. This may inflate the reported improvements."
As we seek to investigate autonomous improvement methods, we selected routes where improvement is required. We specifically chose routes where all BC baselines struggled because we believe that these routes highlight skills that current autonomous driving policies struggle with. Focusing on these failure modes makes them a strong testbed for studying autonomous improvement. We acknowledge that these reported gains are relevant only to the selected subsets and do not make broader claims about improving on the benchmarks overall.
[Weakness 2.6] "[weight parameter] values are chosen without an apparent reason…"
For each criterion other than collision probability and expected progress, we use the same weight of 0.5. We use a smaller coefficient for expected progress because its values range from 0 to 10. We upweight collision risk for both Fail2Drive and Bench2Drive, as collisions pose high safety risks, and because collision-avoidant behaviors often oppose progress. We will clarify this in our updated manuscript.
We unfortunately do not have the resources to perform a closed-loop ablation of these hyperparameters, as our experiments are very compute-intensive. We instead perform an open-loop analysis of the changed decisions, if we were instead to weight each term equally (0.5):
| Benchmark | Changed decisions | Changed ADE (0.15s) | Changed ADE (0.5s) | Changed ADE (1s) |
|---|---|---|---|---|
| B2D | 177/1,995 (8.87%) | 0.0416 | 0.1287 | 0.2942 |
| F2D | 223/3,394 (6.57%) | 0.0505 | 0.1497 | 0.3274 |
Here, changed ADE refers to the average ADE in meters between the action chunks corresponding to changed decisions and the originally executed action chunk over one seed. As shown, the average ADE is quite low across both benchmarks; we note that, at each decision point, the policy can produce multiple reasonable actions, so changes in decisions may not necessarily induce drastic differences in behavior.
[Question 4] "What is the sensitivity of the methods to the chosen VLM? Why were different VLMs chosen for the two methods?":
HARP uses a VLM with strong video understanding to analyze long trajectories. FLUTE uses a local VLM because candidate scoring requires access to model logits. Moreover, as HARP is a train-time method that reasons over trajectory videos, we find that a more powerful VLM is better suited to its use case, and as FLUTE is a test-time method, utilizing a local VLM allows for local deployment without reliance on external API calls. We will clarify this in our updated manuscript.
To evaluate the sensitivity of FLUTE to the VLM backbone, we perform experiments with FLUTE and HARP with smaller, older models over one seed, using the mixed policy. We also include the single-seed results in brackets for Qwen3.8-27B:
| VLM Backbone | Bench2Drive | Fail2Drive |
|---|---|---|
| Qwen3.5-9B | 73.23 | 71.97 |
| Gemma3-4B | 73.68 | 70.17 |
| Qwen3.8-27B (Ours) | 73.8 (77.00) | 75.8 (77.67) |
We find that Bench2Drive performance is largely unchanged, and that Fail2Drive performance has a small degradation in performance, although we still see an improvement over all BC baselines. Empirically, both smaller models had the largest drop in performance on generalization-animals-1076, wherein the vehicle must yield to an elephant crossing the road. We observe that, while the VLMs are able to identify the elephant within the frame, the difference in performance is mainly attributed to the candidate scoring step; Qwen3.5-9B does not identify accelerating actions as having sufficient collision risk and collides, while Gemma3-4B is overly cautious and assigns high collision risk to all actions, regardless of whether obstacles are present in the frame.
We unfortunately do not believe that ablations of the backbone or closed-loop experiments are feasible within the review period, as they would either require high API costs or intensive compute costs.
Thank you for the thoughtful review. Please let us know if our responses address your concerns, and we would be happy to clarify anything further.
We sincerely thank the reviewer for the constructive and insightful feedback. In this response, we clarify our method and contributions and propose additional experiments to substantiate our claims further.
[Weakness 1] "...there are no novel ideas to make the combination work…"
We agree that our work builds on a number of prior components, but we are not aware of any previous work that combines high-level policy corrections with an autonomous improvement framework for autonomous driving. This framework specifically allows HARP to introduce new behavior modes and improve the strategy of the high-level policy, all without human intervention.
A key element of HARP that ties together high-level supervision and autonomous improvement is “hindsight refinement”, which is specifically novel in the context of robot policy improvement. As outlined in the related work, some LLM refinement work corrects the exact step that led to failure. In robot policy work, this is even more limited to selecting subtasks or intervening with human guidance before the policy fails or just after it fails. None of these works attempt to reflect on the larger history of what the policy did and relabel behaviors to improve the policy’s overall strategy. This is evident in Figure 4, where the policy learns to stop for the vehicle blocking the road rather than navigate around it.
Are there any related works beyond those that we discuss in the related work that you believe are sufficiently close to what our paper is doing, and that we should cite?
[Weakness 2.1 & Question 1] "...the autonomous improvement [baseline]s are lacking and only used in one of the two experiments…Is there a good reason for using the specific improvement methods as baselines and omitting others from the related work?"
For our autonomous improvement experiments, we selected two baselines that are representative of broader classes of autonomous improvement methods. During this rebuttal period, we attempted an implementation of GRPO and discussed our findings in this response.
Our Residual RL baseline measures the effectiveness of low-level scalar reward optimization. Although residual RL was first introduced in 2018, it remains relevant to modern VLA post-training methods, such as EXPO and EXPO-FT (Dong et al., 2025 and 2026), KL-regularized methods (such as Maximum a Posteriori Policy Optimization (Abdolmaleki et al., 2018)), and DSRL (Wagenmaker et al., 2025). Residual RL is uniquely suited to fine-tuning VLA policies since it avoids backpropagation through large VLM/VLA backbones.
Our YAY Robot baseline measures the effectiveness of high-level subtask intervention and refinement. We intentionally modified the design and training scheme from its original implementation to create a more controlled comparison with HARP. We believe YAY Robot is a particularly useful baseline because it isolates the importance of hindsight reflection.
The other works introduced in the related work, such as LLM-specific methods like GRPO, Self-Refine, R^3V, Critic-V, and GUI-Reflection, would require adaptation to be utilized in a robot learning setting.
Some prior robotics work uses GRPO to improve low-level actions (Autodrive-R^2, Yuan et al., 2025) or alignment in the high-level policy (Poutine, Rowe et al., 2025 and AutoVLA, Zhou et al., 2025). Our residual RL baseline is already representative of low-level improvements, so we focused on implementing a preliminary version of GRPO for high-level policy improvement. Since trajectory-level group rollouts are infeasible (and low-signal in long-horizon settings), our implementation of GRPO requires setting up a reward to evaluate the quality of potential subtasks before they are executed. We tested two approaches to reward design:
It’s worth noting that the disadvantages of GRPO are illustrative of the advantages of HARP. HARP can perform its updates asynchronously without interrupting the current trajectory, sidestep concerns with reward design by leveraging prior knowledge from VLMs, and can ground its evaluation of subtasks in actual outcomes rather than predictions.