acceptodds
Under review as a conference paper at ICLR 2027

HARP: Hindsight Reflection Autonomously Improves Robot Policies

Abstract

Real-world robot policies must operate in unseen settings and perform tasks beyond the support of their pretraining data. While scaling up pretraining has enabled broadly capable robot policies, learning autonomously from experience allows policies to continually improve as they encounter new scenarios. A central challenge to achieving this is incorporating prior knowledge in the learning process, allowing robots to interpret their mistakes and determine how to improve their behavior, as humans do. To address this, we introduce HARP, or Hindsight-guided Adaptation for Robot Policies, a general framework for autonomous improvement of language-conditioned robot policies through hindsight reflection. HARP uses the prior knowledge of VLMs to reflect on robot experience, determining which decisions led to successful outcomes, caused failures, or introduced risks even when no failure occurred. This feedback is converted into subtask-level corrections, aligned with an overall task strategy. These corrections supervise ongoing updates to a high-level planner, improving its ability to guide the language-conditioned policy through subsequent interactions. We instantiate HARP for autonomous driving and demonstrate improvements over base policies on challenging routes from Bench2Drive and Fail2Drive.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot learning, autonomous improvment, behavior cloning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_URcMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles adaptation of hierarchical robot policies beyond the data used for pretraining, to adapt to never-seen challenges. The authors introduce two frameworks: HARP, which uses hindsight corrections of interactions by a VLM to fine-tune the high-level policy, and FLUTE, a foresight critic, which uses another VLM to choose the best action among a given set based on pre-defined criteria. They evaluate them individually and combined in a subset of the Bench2Drive and Fail2Drive benchmarks, where they yield a substantial improvement in mean performance.

Critical strengths
  • Clear presentation with good visuals and reproducability: The paper reads nicely, the components are explained well and the visuals aid understanding. All details are given in the appendix, so I am certain that even without the code (which is also attached), results are very reproducible.
Critical weaknesses
  • Novelty: All individual components are well-studied, there are no novel ideas to make the combination work, but rather just combining them.
  • Evaluation: evaluation falls short in several ways:
    • Baselines: While the behavioral cloning policies are varied and recent, the autonomous improvement ones are lacking and only used in one of the two experiments. The authors only consider residual RL (an older method from 2018) and YAY robot (which in its original formulation is human-in-the-loop, not autonomous). They cite several in their related work, but do not compare their method against them in the evaluation.
    • FLUTE seems to depend on hand-crafted criteria, which are then evaluated by the LLM. No sensitivity to these are reported, and the ones chosen seem to be directly related to the evaluation metric, the driving score. (The driving score said to combine "collision avoidance, traffic-rule following, and progress in the route", FLUTE evaluates "expected progress along the route, alignment with [the language instructions], correctness of the proposed action, and whether the proposed action will lead to a collision, a traffic violation, or an off-road violation", which seems like a direct optimization towards the metric.)
    • Low confidence results: Driving scores are calculated based on 3 seeds, and SDs of individual results are not reported (only SDs of what seems to be the means of the route driving scores, averaged over the seeds are given), so no statistical significance can be derived from the results.
    • No inference times are reported: FLUTE uses a VLM-based metric for every action decision, but no inference times are reported. This makes the described use-case of real-world robotics questionable, as capable VLMs usually have higher inference times than uses such as autonomous driving require.
    • The subset of the benchmarks is selected by the worst performances of the BC baselines. This may inflate the reported improvements.
    • The weight parameters are never ablated, their values are chosen without an apparent reason.
Questions for authors
  • Is there a good reason for using the specific improvement methods as baselines and omitting others from the related work?
  • Is the 20Hz the simulator runs at real-time or simulation time when including the inference time needed for running FLUTE? Is real-time inference on an autonomous car possible?
  • What are the confidence intervals of the improvements in driving score?
  • What is the sensitivity of the methods to the chosen VLM? Why were different VLMs chosen for the two methods?
6 comments

We sincerely thank the reviewer for the constructive and insightful feedback. In this response, we clarify our method and contributions and propose additional experiments to substantiate our claims further.

[Weakness 1] "...there are no novel ideas to make the combination work…"

We agree that our work builds on a number of prior components, but we are not aware of any previous work that combines high-level policy corrections with an autonomous improvement framework for autonomous driving. This framework specifically allows HARP to introduce new behavior modes and improve the strategy of the high-level policy, all without human intervention.

A key element of HARP that ties together high-level supervision and autonomous improvement is “hindsight refinement”, which is specifically novel in the context of robot policy improvement. As outlined in the related work, some LLM refinement work corrects the exact step that led to failure. In robot policy work, this is even more limited to selecting subtasks or intervening with human guidance before the policy fails or just after it fails. None of these works attempt to reflect on the larger history of what the policy did and relabel behaviors to improve the policy’s overall strategy. This is evident in Figure 4, where the policy learns to stop for the vehicle blocking the road rather than navigate around it.

Are there any related works beyond those that we discuss in the related work that you believe are sufficiently close to what our paper is doing, and that we should cite?

[Weakness 2.1 & Question 1] "...the autonomous improvement [baseline]s are lacking and only used in one of the two experiments…Is there a good reason for using the specific improvement methods as baselines and omitting others from the related work?"

For our autonomous improvement experiments, we selected two baselines that are representative of broader classes of autonomous improvement methods. During this rebuttal period, we attempted an implementation of GRPO and discussed our findings in this response.

Our Residual RL baseline measures the effectiveness of low-level scalar reward optimization. Although residual RL was first introduced in 2018, it remains relevant to modern VLA post-training methods, such as EXPO and EXPO-FT (Dong et al., 2025 and 2026), KL-regularized methods (such as Maximum a Posteriori Policy Optimization (Abdolmaleki et al., 2018)), and DSRL (Wagenmaker et al., 2025). Residual RL is uniquely suited to fine-tuning VLA policies since it avoids backpropagation through large VLM/VLA backbones.

Our YAY Robot baseline measures the effectiveness of high-level subtask intervention and refinement. We intentionally modified the design and training scheme from its original implementation to create a more controlled comparison with HARP. We believe YAY Robot is a particularly useful baseline because it isolates the importance of hindsight reflection.

The other works introduced in the related work, such as LLM-specific methods like GRPO, Self-Refine, R^3V, Critic-V, and GUI-Reflection, would require adaptation to be utilized in a robot learning setting.

Some prior robotics work uses GRPO to improve low-level actions (Autodrive-R^2, Yuan et al., 2025) or alignment in the high-level policy (Poutine, Rowe et al., 2025 and AutoVLA, Zhou et al., 2025). Our residual RL baseline is already representative of low-level improvements, so we focused on implementing a preliminary version of GRPO for high-level policy improvement. Since trajectory-level group rollouts are infeasible (and low-signal in long-horizon settings), our implementation of GRPO requires setting up a reward to evaluate the quality of potential subtasks before they are executed. We tested two approaches to reward design:

  1. VLM grader to score candidate subtasks (closest to HARP): Generation of 4 candidates took ~8 seconds and grading took ~10 seconds, making this approach prohibitively slow. We attempted to mitigate this issue by querying the VLM less frequently, but this resulted in noticeable performance degradation.
  2. Simulator-based reward (includes route progress, lane deviations, potential collision risks, red lights, etc.): While this version is much faster, it requires predicting which subtasks will lead to good behavior, similar to YAY Robot, rather than confirming in hindsight, like HARP or residual RL. With initial experiments, we can anecdotally say that we observed poor performance on key routes where HARP as well as the other baselines performed well.

It’s worth noting that the disadvantages of GRPO are illustrative of the advantages of HARP. HARP can perform its updates asynchronously without interrupting the current trajectory, sidestep concerns with reward design by leveraging prior knowledge from VLMs, and can ground its evaluation of subtasks in actual outcomes rather than predictions.

[Weakness 2.2] "...FLUTE seems to depend on hand-crafted criteria…"

The criteria used in FLUTE (progress along the route, collisions, traffic violations, off-road violations) are commonplace reward terms for autonomous driving and for navigation in general. Prior works, such as Raw2Drive (Yang et al., 2025) and LSTC (Hu et al., 2024) use similar criteria, wherein progress is rewarded, and violations are penalized.

We acknowledge that these objectives overlap with components of the Bench2Drive driving score. However, FLUTE does not directly observe or optimize the driving score; instead, the VLM estimates the likely consequences of candidate actions from the available observations. The same criteria are applied across routes without route-specific tuning. We agree that sensitivity to the relative weights is an important question and will clarify the rationale behind our coefficient choices in the revised manuscript.

[Weakness 2.3 & Question 3] …no statistical significance can be derived from the results… What are the confidence intervals of the improvements in driving score?

Due to the compute-intensive nature of the CARLA driving simulator and running inference with large VLAs, prior works (such as SimLingo (Renz et al., 2025)) typically use a maximum of three seeds.

As the routes differ significantly and therefore have different variations in performance, we used the typical method for computing performance on Bench2Drive and Fail2Drive, which is computing the per-seed averages and then computing the standard deviation across these seeds. This is reflective of prior work such as SimLingo, BLUE, ORION, and DriveMoE.

We report here 95% confidence intervals on improvement for the Bench2Drive and Fail2Drive for HARP + FLUTE relative to the strongest base policy we evaluated with (SteerVLA high-level policy and the π0.5AD\pi^{AD}_{0.5} low-level policy)

  • Bench2Drive: +5.26 [-5.90, +16.42]
  • Fail2Drive: +12.32 [+0.67, +23.97]

The result on Bench2Drive indicates the estimated mean gain is positive, but the confidence interval includes 0. However, we invite the reviewer to review the examples in Figures 1,2, and 4 that demonstrate qualitatively the improvements HARP + FLUTE can achieve. On Fail2Drive, the estimated mean gain and its 95% confidence interval lie entirely above zero.

We also hope the reviewer understands that we are resource-constrained, and the experiments in the paper took over 2000 GPU hours across policies, routes, and seeds with access to only 8 H200 GPUs. While we would have strongly preferred to increase the number of seeds used in these experiments, this is a significant undertaking.

We will include this version of the results and more discussion in our updated manuscript.

[Weakness 2.4 & Question 2] No inference times are reported… Is the 20Hz the simulator runs at real-time or simulation time?

The reported 20 Hz refers to simulation time. CARLA advances synchronously, allowing inference to finish before simulation proceeds. We select action chunks every three simulator steps, corresponding to 0.15 seconds of simulated driving.

The policies have the following inference times. We also report the amortized inference time, as we only query the high-level policy every 6 steps and the low-level policy every 3 steps:

Modelms/queryms/simulator step
SteerVLA high-level1186198
SteerVLA low-level14448
π0.5AD\pi^{AD}_{0.5} high-level32955
π0.5AD\pi^{AD}_{0.5} low-level7625

As described in Appendix B, for FLUTE, we first prompt the VLM to generate a short description of the scene, then score each candidate. We overlap the description generation with the generation of each candidate, as these two steps are independent.

ComponentTime (s)
Description generation5.50
Candidate scoring (4 candidates)2.16
Total7.66

We acknowledge that this implementation does not meet real-time autonomous driving requirements. Our primary objective is to investigate whether VLM-based foresight evaluation can improve policy behavior in challenging and out-of-distribution scenarios, rather than to demonstrate real-time deployment. Reducing the computational cost of candidate generation and evaluation remains an important direction for future work. We will clarify this limitation and report the measured inference latency in the revised manuscript.

[Weakness 2.5] "The subset of the benchmarks is selected by the worst performances of the BC baselines. This may inflate the reported improvements."

As we seek to investigate autonomous improvement methods, we selected routes where improvement is required. We specifically chose routes where all BC baselines struggled because we believe that these routes highlight skills that current autonomous driving policies struggle with. Focusing on these failure modes makes them a strong testbed for studying autonomous improvement. We acknowledge that these reported gains are relevant only to the selected subsets and do not make broader claims about improving on the benchmarks overall.

[Weakness 2.6] "[weight parameter] values are chosen without an apparent reason…"

For each criterion other than collision probability and expected progress, we use the same weight of 0.5. We use a smaller coefficient for expected progress because its values range from 0 to 10. We upweight collision risk for both Fail2Drive and Bench2Drive, as collisions pose high safety risks, and because collision-avoidant behaviors often oppose progress. We will clarify this in our updated manuscript.

We unfortunately do not have the resources to perform a closed-loop ablation of these hyperparameters, as our experiments are very compute-intensive. We instead perform an open-loop analysis of the changed decisions, if we were instead to weight each term equally (0.5):

BenchmarkChanged decisionsChanged ADE (0.15s)Changed ADE (0.5s)Changed ADE (1s)
B2D177/1,995 (8.87%)0.04160.12870.2942
F2D223/3,394 (6.57%)0.05050.14970.3274

Here, changed ADE refers to the average ADE in meters between the action chunks corresponding to changed decisions and the originally executed action chunk over one seed. As shown, the average ADE is quite low across both benchmarks; we note that, at each decision point, the policy can produce multiple reasonable actions, so changes in decisions may not necessarily induce drastic differences in behavior.

[Question 4] "What is the sensitivity of the methods to the chosen VLM? Why were different VLMs chosen for the two methods?":

HARP uses a VLM with strong video understanding to analyze long trajectories. FLUTE uses a local VLM because candidate scoring requires access to model logits. Moreover, as HARP is a train-time method that reasons over trajectory videos, we find that a more powerful VLM is better suited to its use case, and as FLUTE is a test-time method, utilizing a local VLM allows for local deployment without reliance on external API calls. We will clarify this in our updated manuscript.

To evaluate the sensitivity of FLUTE to the VLM backbone, we perform experiments with FLUTE and HARP with smaller, older models over one seed, using the mixed policy. We also include the single-seed results in brackets for Qwen3.8-27B:

VLM BackboneBench2DriveFail2Drive
Qwen3.5-9B73.2371.97
Gemma3-4B73.6870.17
Qwen3.8-27B (Ours)73.8 (77.00)75.8 (77.67)

We find that Bench2Drive performance is largely unchanged, and that Fail2Drive performance has a small degradation in performance, although we still see an improvement over all BC baselines. Empirically, both smaller models had the largest drop in performance on generalization-animals-1076, wherein the vehicle must yield to an elephant crossing the road. We observe that, while the VLMs are able to identify the elephant within the frame, the difference in performance is mainly attributed to the candidate scoring step; Qwen3.5-9B does not identify accelerating actions as having sufficient collision risk and collides, while Gemma3-4B is overly cautious and assigns high collision risk to all actions, regardless of whether obstacles are present in the frame.

We unfortunately do not believe that ablations of the backbone or closed-loop experiments are feasible within the review period, as they would either require high API costs or intensive compute costs.

Thank you for the thoughtful review. Please let us know if our responses address your concerns, and we would be happy to clarify anything further.

Reviewer_8shTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HARP, a framework for improving hierarchical robot policies through VLM-based hindsight reflection. A VLM reviews past rollouts, identifies bad or risky decisions, and generates corrected subtask instructions, including corrections for earlier decisions that contributed to later failures. These corrections are used to fine-tune the high-level policy while keeping the low-level controller frozen. The authors also introduce FLUTE, a VLM-based critic that selects among candidate language-action pairs at test time using best-of-N sampling.

Critical strengths
  1. Hindsight correction of earlier decisions. Unlike feedback that only identifies immediate failures, HARP can attribute bad outcomes to earlier decisions and generate corresponding corrections. The language-subtask interface makes these corrections directly usable for policy training without modifying the low-level controller.

  2. Complementarity between hindsight learning and test-time selection. HARP can introduce useful behaviors that the base policy rarely generates, while FLUTE helps select them at test time. The experiments provide encouraging evidence for this combination, particularly on out-of-distribution scenarios.

Critical weaknesses
  1. The benefit of hindsight reflection is not well isolated. There is no ablation of precursor credit assignment, strategy memory, or simple VLM-based relabeling without hindsight. The comparison with the modified YAY Robot baseline is useful, but the correction and training pipelines differ. It remains unclear how much of the improvement comes specifically from hindsight reasoning rather than additional VLM-generated supervision.

  2. Limited evidence of generalization beyond the adaptation scenarios. The policies are adapted and evaluated on the same routes. This demonstrates scenario-specific improvement, but does not establish whether the learned corrections transfer to unseen routes. Moreover, FLUTE requires multiple VLM evaluations at inference time, and its latency is not reported.

Questions for authors
  1. How much of HARP's improvement comes from precursor credit assignment and strategy memory? Can the authors compare against simpler VLM relabeling without these components under the same training budget?

  2. Does HARP improve performance on held-out routes after adaptation? What is the inference latency of FLUTE?

5 comments

We sincerely thank the reviewer for the constructive and insightful feedback. In this response, we clarify our method and contributions and propose additional experiments to further substantiate our claims.

[Weakness 1 & Question 1] "...There is no ablation of precursor credit assignment, strategy memory, or simple VLM-based relabeling without hindsight…How much of HARP's improvement comes from precursor credit assignment and strategy memory? Can the authors compare against simpler VLM relabeling without these components under the same training budget?"

We believe that our current YAY Robot baseline sufficiently isolates the importance of credit assignment and strategy memory. The YAY Robot training pipeline in our experiments was purposely designed to be as similar to HARP as possible: the same number of gradient steps were taken at the same intervals, the same VLM was used to produce corrected subtasks, etc. However, it cannot perform precursor credit assignment and does not have strategy memory as it does not perform hindsight reflection to produce corrected subtasks. HARP adds both of these components and achieves an 8.1 point performance improvement because it is able to ground its corrections in actual outcomes and can use information about prior attempts to inform future strategies.

Strategy memory allows historical information about prior attempts to impact strategies of future episodes. Qualitatively, we found this to be valuable during the development of our method as otherwise, HARP has no way of knowing to modify strategies that did not work out. While performing an ablation of strategy memory could be valuable, online HARP experiments are expensive and time-consuming to run. Therefore, we will run the experiment if we have time, but we hope that the reviewer understands that this would be a significant undertaking on top of the other experiments we are running.

[Weakness 2 & Question 2.1] "Limited evidence of generalization beyond the adaptation scenarios… Does HARP improve performance on held-out routes after adaptation?"

We agree that held-out-route evaluation would strengthen our findings. We performed generalization experiments wherein we take a set of 5 routes, train on 4 routes with HARP labels, and evaluate on 1 held-out route. Specifically, we take the pools of data collected during HARP training per route, and do offline gradient updates with the pooled data across routes. We do this for similar groups of routes (variations of turning at a junction or stopping for an obstacle) and diverse groups of routes. We use a batch size of 128 and perform 200 gradient updates, following the same protocol as HARP training otherwise. We hold out each of the routes in the group once and, as these evaluations are relatively inexpensive to run compared to the full HARP runs, we evaluate each of these policies over 9 seeds. We perform these evaluations with the SteerVLA high-level and the $\pi^{AD}_{0.5} low-level policy.

Here are the route groups used to evaluate generalization:

Bench2Drive:

  • Junction routes: various scenarios of making turns with oncoming traffic, with and without signalized intersections (non-signalized-junction-left-turn-001, non-signalized-junction-left-turn-enter-flow-002, non-signalized-junction-right-turn-003, signalized-junction-left-turn-001, signalized-junction-left-turn-enter-flow-001)
  • Highway/merging routes: various scenarios of merging into other lanes while avoiding other driving agents (enter-actor-flow-004, hazard-at-side-lane-002, highway-exit-002, interurban-actor-flow-004, merger-into-slow-traffic-v2-005)
  • B2D Diverse Group 1: diverse scenarios in Bench2Drive (enter-actor-flow-004, hazard-at-side-lane-002, highway-exit-002, pedestrian-crossing-004, signalized-junction-left-turn-001)
  • B2D Diverse Group 2: diverse scenarios in Bench2Drive (accident-two-ways-002, crossing-bicycle-flow-004, interurban-actor-flow-004, merger-into-slow-traffic-v2-005, signalized-junction-right-turn-004)

Fail2Drive:

  • Blocked routes: various scenarios where the agent encounters an obstacle or sign and must come to a complete stop (fully-blocked-1032, hard-brake-1036, obscured-stop-1046, pedestrians-on-road-1085, wall-1097)
  • Obstacles: various scenarios where the agent encounters an obstacle that partially blocks the road and must avoid it (bad-parking-1004, construction-permutations-1019, custom-obstacles-1020, custom-obstacles-1024, right-construction-1094)
  • F2D Diverse: diverse scenarios in Fail2Drive (animals-1076, construction-pedestrian-1011, image-on-object-1041, pedestrian-other-blocker-1072, right-of-way-1056)
GroupBase policySingle-route HARPPooled HARP (9 seeds)HARP − basePooled − basePooled − HARP
Junction routes57.6 ± 14.959.9 ± 3.268.3 ± 10.4+2.3+10.7+8.4
Highway / merging routes73.6 ± 11.780.5 ± 5.673.7 ± 7.1+6.9+0.1−6.8
B2D Diverse Group 162.7 ± 12.474.7 ± 9.772.1 ± 12.0+12.1+9.5−2.6
B2D Diverse Group 275.2 ± 14.678.6 ± 4.873.9 ± 5.5+3.3−1.3−4.6
Blocked routes72.3 ± 4.188.0 ± 8.755.8 ± 4.5+15.7−16.6−32.2
Obstacle routes58.8 ± 2.653.6 ± 5.756.5 ± 4.9−5.3−2.3+3.0
F2D Diverse67.4 ± 16.047.1 ± 11.469.7 ± 10.0−20.3+2.2+22.6

We observe that on Bench2Drive route groups, pooled HARP achieves a 10.7 point gain over the base policy on the “Junction routes group” and a 9.5 point gain on “B2D Diverse Group 1”. On Fail2Drive, single-route HARP achieves a 15.7 point gain over the base policy on the “Blocked routes” group while pooled HARP achieves a 2.2 point gain over the base policy and a 22.6 point gain over single-route HARP on “F2D Diverse”.

Grouping routes by route similarity did not consistently improve held-out performance. Among similar groups, the “Junction routes” group showed the largest improvement. Pooled HARP scored 10.7 points above the base policy and 8.4 points above single-route HARP. One possible explanation is that these routes share a similar intersection layout and focus on the same decision regarding turning in the presence of oncoming traffic. In the blocked-route group, by comparison, the required action is the same across routes, but the cues that tell the policy when to stop can differ. The gain on Bench2Drive Diverse Group 1 (+9.5 points over the base policy) also suggests that transfer is possible across different behaviors.

We also look at specific routes included in the similar and diverse route pools. Six Bench2Drive routes appear in both a similar route pool and a diverse route pool.

Held-out route (similar / diverse group)Base policySingle-route HARPPooled HARP (similar group)Pooled HARP (diverse group)Similar − baseDiverse − baseSimilar − HARPDiverse − HARPDiverse − similar
signalized-junction-left-turn-001 (Junction / B2D Diverse Group 1)39.3 ± 17.962.4 ± 32.650.3 ± 38.360.8 ± 26.0+10.9+21.4−12.1−1.6+10.5
enter-actor-flow-004 (Highway/merging / B2D Diverse Group 1)73.4 ± 46.060.0 ± 0.058.9 ± 25.055.3 ± 32.2−14.5−18.2−1.1−4.7−3.6
highway-exit-002 (Highway/merging / B2D Diverse Group 1)77.9 ± 0.077.9 ± 0.074.0 ± 10.377.9 ± 0.0−3.90.0−3.90.0+3.9
hazard-at-side-lane-002 (Highway/merging / B2D Diverse Group 1)55.9 ± 46.286.7 ± 23.173.3 ± 20.082.4 ± 28.6+17.4+26.5−13.3−4.3+9.1
interurban-actor-flow-004 (Highway/merging / B2D Diverse Group 2)97.2 ± 0.096.3 ± 0.092.6 ± 12.296.8 ± 0.5−4.6−0.4−3.7+0.5+4.2
merger-into-slow-traffic-v2-005 (Highway/merging / B2D Diverse Group 2)63.7 ± 31.581.8 ± 31.569.9 ± 28.687.9 ± 24.0+6.2+24.2−12.0+6.1+18.0
Average (6 routes)67.9 ± 12.777.5 ± 10.169.8 ± 9.676.8 ± 9.3+1.9+8.9−7.7−0.7+7.0

Across the six shared held-out routes, diverse-route pooling achieved a mean driving score of 76.8 compared with 69.8 for similar-route pooling and 77.5 for single-route HARP. Diverse-route pooling therefore gained 7.0 points over similar-route pooling and scored 0.7 points below single-route HARP.

We will include these results in our updated manuscript.

[Question 2.2] "What is the inference latency of FLUTE?"

As described in Appendix B, for FLUTE, we first prompt the VLM to generate a short description of the scene, then score each candidate. We overlap the description generation with the generation of each candidate, as these two steps are independent.

ComponentTime (s)
Description generation5.50
Candidate scoring (4 candidates)2.16
Total7.66

We acknowledge that this implementation does not meet real-time autonomous driving requirements. Our primary objective is to investigate whether VLM-based foresight evaluation can improve policy behavior in challenging and out-of-distribution scenarios, rather than to demonstrate real-time deployment. Reducing the computational cost of candidate generation and evaluation remains an important direction for future work. We will clarify this limitation and report the measured inference latency in the revised manuscript.

Thank you for the thoughtful review. Please let us know if our responses address your concerns, and we would be happy to clarify anything further!

Reviewer_KdwkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HARP, which uses a VLM to retrospectively analyze robot experience, convert mistakes and their precursors into corrected language subtasks, and fine-tune the high-level policy. FLUTE further evaluates and selects language-action candidates at test time. Together, they achieve relative driving-score improvements of 7.7% and 19.4% on selected Bench2Drive and Fail2Drive subsets.

Critical strengths
  1. HARP and FLUTE have a clear complementary design. HARP enables the policy to propose better behaviors, while FLUTE checks the resulting actions and selects a candidate for execution.
  2. The paper provides extensive visualizations and experiment results. For example, Figure 4 illustrates meaningful behavioral changes with HARP and FLUTE in two safety-critical cases. The detailed prompts and per-route results also help understand the implementation and its failure cases.
Critical weaknesses
  1. The policies are trained separately on each target route and subsequently evaluated on that same route. This demonstrates adaptation to encountered scenarios, but does not evaluate the generalizability to unseen routes or maintenance of previously learned capabilities. Held-out-route evaluation and performance checks on existing tasks are needed.
  2. In Line 148-150, the paper says "In robotics, addressing early mistakes or risky behaviors is especially important since failure can be irreversible. HARP is designed to address this kind of credit assignment problem through hindsight reflection." However, the experiments haven't compared correcting decisions at failure states with addressing early mistakes. This ablation is needed to determine whether correcting those earlier decisions provides additional performance gains.
Questions for authors
  1. As discussed in Line 421-422, "although HARP alone leads to a small decrease in performance on Fail2Drive, combined with FLUTE, we can improve performance by 12.3 points, far outperforming FLUTE alone." Is HARP's performance decrease caused by incorrect hindsight corrections, the high-level policy failing to select the correct behavior, or the low-level policy failing to execute that behavior correctly?
5 comments

We sincerely thank the reviewer for the constructive and insightful feedback. In this response, we clarify our method and contributions and introduce additional experimental results to further substantiate our claims.

[Weakness 1] "...Held-out-route evaluation and performance checks on existing tasks are needed."

We agree that held-out-route evaluation would strengthen our findings. We performed generalization experiments wherein we take a set of 5 routes, train on 4 routes with HARP labels, and evaluate on 1 held-out route. Specifically, we take the pools of data collected during HARP training per route and do offline gradient updates with the pooled data across routes. We do this for similar groups of routes (variations of turning at a junction or stopping for an obstacle) and diverse groups of routes. We use a batch size of 128 and perform 200 gradient updates, following the same protocol as HARP training otherwise. We hold out each of the routes in the group once and, as these evaluations are relatively inexpensive to run compared to the full HARP runs, we evaluate each of these policies over 9 seeds. We perform these evaluations with the SteerVLA high-level and the $\pi^{AD}_{0.5} low-level policy.

Here are the route groups used to evaluate generalization:

Bench2Drive:

  • Junction routes: various scenarios of making turns with oncoming traffic, with and without signalized intersections (non-signalized-junction-left-turn-001, non-signalized-junction-left-turn-enter-flow-002, non-signalized-junction-right-turn-003, signalized-junction-left-turn-001, signalized-junction-left-turn-enter-flow-001)
  • Highway/merging routes: various scenarios of merging into other lanes while avoiding other driving agents (enter-actor-flow-004, hazard-at-side-lane-002, highway-exit-002, interurban-actor-flow-004, merger-into-slow-traffic-v2-005)
  • B2D Diverse Group 1: diverse scenarios in Bench2Drive (enter-actor-flow-004, hazard-at-side-lane-002, highway-exit-002, pedestrian-crossing-004, signalized-junction-left-turn-001)
  • B2D Diverse Group 2: diverse scenarios in Bench2Drive (accident-two-ways-002, crossing-bicycle-flow-004, interurban-actor-flow-004, merger-into-slow-traffic-v2-005, signalized-junction-right-turn-004)

Fail2Drive:

  • Blocked routes: various scenarios where the agent encounters an obstacle or sign and must come to a complete stop (fully-blocked-1032, hard-brake-1036, obscured-stop-1046, pedestrians-on-road-1085, wall-1097)
  • Obstacles: various scenarios where the agent encounters an obstacle that partially blocks the road and must avoid it (bad-parking-1004, construction-permutations-1019, custom-obstacles-1020, custom-obstacles-1024, right-construction-1094)
  • F2D Diverse: diverse scenarios in Fail2Drive (animals-1076, construction-pedestrian-1011, image-on-object-1041, pedestrian-other-blocker-1072, right-of-way-1056)
GroupBase policySingle-route HARPPooled HARP (9 seeds)HARP − basePooled − basePooled − HARP
Junction routes57.6 ± 14.959.9 ± 3.268.3 ± 10.4+2.3+10.7+8.4
Highway / merging routes73.6 ± 11.780.5 ± 5.673.7 ± 7.1+6.9+0.1−6.8
B2D Diverse Group 162.7 ± 12.474.7 ± 9.772.1 ± 12.0+12.1+9.5−2.6
B2D Diverse Group 275.2 ± 14.678.6 ± 4.873.9 ± 5.5+3.3−1.3−4.6
Blocked routes72.3 ± 4.188.0 ± 8.755.8 ± 4.5+15.7−16.6−32.2
Obstacle routes58.8 ± 2.653.6 ± 5.756.5 ± 4.9−5.3−2.3+3.0
F2D Diverse67.4 ± 16.047.1 ± 11.469.7 ± 10.0−20.3+2.2+22.6

We observe that on Bench2Drive route groups, pooled HARP achieves a 10.7 point gain over the base policy on the “Junction routes group” and a 9.5 point gain on “B2D Diverse Group 1”. On Fail2Drive, single-route HARP achieves a 15.7 point gain over the base policy on the “Blocked routes” group while pooled HARP achieves a 2.2 point gain over the base policy and a 22.6 point gain over single-route HARP on “F2D Diverse”.

Grouping routes by route similarity did not consistently improve held-out performance. Among similar groups, the “Junction routes” group showed the largest improvement. Pooled HARP scored 10.7 points above the base policy and 8.4 points above single-route HARP. One possible explanation is that these routes share a similar intersection layout and focus on the same decision regarding turning in the presence of oncoming traffic. In the blocked-route group, by comparison, the required action is the same across routes, but the cues that tell the policy when to stop can differ. The gain on Bench2Drive Diverse Group 1 (+9.5 points over the base policy) also suggests that transfer is possible across different behaviors.

We also look at specific routes included in the similar and diverse route pools. Six Bench2Drive routes appear in both a similar route pool and a diverse route pool.

Held-out route (similar / diverse group)Base policySingle-route HARPPooled HARP (similar group)Pooled HARP (diverse group)Similar − baseDiverse − baseSimilar − HARPDiverse − HARPDiverse − similar
signalized-junction-left-turn-001 (Junction / B2D Diverse Group 1)39.3 ± 17.962.4 ± 32.650.3 ± 38.360.8 ± 26.0+10.9+21.4−12.1−1.6+10.5
enter-actor-flow-004 (Highway/merging / B2D Diverse Group 1)73.4 ± 46.060.0 ± 0.058.9 ± 25.055.3 ± 32.2−14.5−18.2−1.1−4.7−3.6
highway-exit-002 (Highway/merging / B2D Diverse Group 1)77.9 ± 0.077.9 ± 0.074.0 ± 10.377.9 ± 0.0−3.90.0−3.90.0+3.9
hazard-at-side-lane-002 (Highway/merging / B2D Diverse Group 1)55.9 ± 46.286.7 ± 23.173.3 ± 20.082.4 ± 28.6+17.4+26.5−13.3−4.3+9.1
interurban-actor-flow-004 (Highway/merging / B2D Diverse Group 2)97.2 ± 0.096.3 ± 0.092.6 ± 12.296.8 ± 0.5−4.6−0.4−3.7+0.5+4.2
merger-into-slow-traffic-v2-005 (Highway/merging / B2D Diverse Group 2)63.7 ± 31.581.8 ± 31.569.9 ± 28.687.9 ± 24.0+6.2+24.2−12.0+6.1+18.0
Average (6 routes)67.9 ± 12.777.5 ± 10.169.8 ± 9.676.8 ± 9.3+1.9+8.9−7.7−0.7+7.0

Across the six shared held-out routes, diverse-route pooling achieved a mean driving score of 76.8 compared with 69.8 for similar-route pooling and 77.5 for single-route HARP. Diverse-route pooling therefore gained 7.0 points over similar-route pooling and scored 0.7 points below single-route HARP.

We are also running maintenance experiments. We will update the response if these experiments come back in time.

We will include these results in our updated manuscript.

[Weakness 2] "...experiments haven't compared correcting decisions at failure states with addressing early mistakes…"

We agree that this comparison is important. Our comparison against YAY Robot directly contrasts correcting near-failure states with addressing early causal mistakes. The YAY Robot training pipeline in our experiments was purposely designed to be as similar to HARP as possible: the same number of gradient steps were taken at the same intervals, the same VLM was used to produce corrected subtasks, a similar prompt was used, etc. However, YAY Robot does not address early mistakes that contributed to failure later on in the episode. Because the VLM only sees the current state, it must identify mistakes before downstream consequences unfold. Early mistakes often appear locally benign in isolation, so the VLM only intervenes when it believes failure is imminent. HARP outperforms this baseline by 8.1 points on the Bench2Drive subset, demonstrating the value of addressing early causal mistakes rather than intervening only near failure.

[Question 1] "Is HARP's performance decrease caused by incorrect hindsight corrections, the high-level policy failing to select the correct behavior, or the low-level policy failing to execute that behavior correctly?"

One cause of the performance decrease was a mismatch between the subtasks produced by the high-level policy and the actions executed by the low-level policy. Even with the more steerable low-level policy, we observed several scenarios in which the high-level policy generates the correct subtask, but the low-level policy does not execute it properly. This is especially apparent in out-of-distribution scenarios, where policies are more prone to ignoring language commands.

One explanation for this behavior could be that the VLM can see how the incorrect subtask was executed in hindsight but cannot observe how the low-level policy will execute the corrected subtask.

This also explains why FLUTE is complementary to HARP: HARP cannot exert control over the actions selected by the low-level policy, but FLUTE can. This is why the combination of the two methods significantly outperforms each method individually. Another error source is incorrect labels produced by HARP. While we find that HARP generally produces correct labels, we still rely on a VLM to interpret the behavior of the policy, which we recognize is prone to error and hallucination. However, we observe that providing additional trajectory information, such as the vehicle velocity and collisions, helps ground the robot's behavior and, therefore, the correction labels. We will incorporate this discussion into our updated manuscript.

Thank you for the thoughtful review. Please let us know if our responses address your concerns, and we would be happy to clarify anything further!

Thank you again for taking the time to review our work and provide valuable comments and suggestions. As the rebuttal period comes to a close, we would sincerely appreciate you taking the time to review our response and let us know if you have any remaining concerns; we would be happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We sincerely thank all the reviewers for their detailed comments and suggestions and have done our best to address all the concerns in the individual responses. We summarize the main changes made to the manuscript, which are color-coded for each of the reviewers: Kdwk → green, 8shT → blue, and URcM → red.

New experiments:

  • The generalization and retention experiments are included in Section 4.5
  • The ablations of the critic backbone are included in Appendix B.2 and are referenced in the main text
  • The criteria weight sensitivity experiment is included in Appendix B.2 and is referenced in the main text

Details and clarifications:

  • We further discuss the failure modes of HARP in Section 4.2
  • We add latency for the policies and critic to Appendix A.1 and Appendix B and reference them in the main text
  • We include the improvements in driving score with the 95% confidence intervals in Section 4.2

As the rebuttal period comes to a close, please let us know if you have any remaining questions. If you believe we have sufficiently addressed your concerns, please consider increasing your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.