acceptodds
Under review as a conference paper at ICLR 2027

VLAs Don’t Always Do What They Say: Towards Measuring and Improving Alignment in Self-Driving VLAs

Abstract

While Vision-Language-Action (VLA) driving models generate textual reasoning traces alongside planned trajectories, these explanations often fail to align with actual execution. We address this misalignment between the reasoning and trajectory through a two part investigation: First, we conduct a mechanistic analysis on Alpamayo (a SoTA driving policy), applying causal interventions including cross-attention masking, prefix/suffix sweeps, clause reordering, logit-level alternatives, and paraphrasing. Our analysis reveals that reasoning traces are heavily front-loaded with maneuver commitments (where clause reordering induces larger trajectory errors than full reasoning masking) and are often superseded by superficial shortcuts in dense scenarios such as complex intersections and work zones. We synthesize these probing techniques into a standardized multi-axis faithfulness benchmark for VLA driving. Second, to prevent policies from exploiting naive alignment metrics, we propose TRACR (Trajectory-Reasoning Aligned Code Reward), a post-training framework using GRPO with a corpus of faithfully validated reward functions that leverage code subtask generation to enforce scene-grounding and logical consistency. Evaluated on held-out OOD scenarios from our benchmark, TRACR outperforms standard RL baselines (such as deterministic rules and LLM judges), improving reasoning-action consistency by over 17% and faithfulness by over 9% while preserving baseline trajectory accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language-Action, Self-Driving, Consistency Alignment, GRPO

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MMnyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the misalignment between textual reasoning and trajectory execution in Vision-Language-Action (VLA) self-driving models through mechanistic probing on Alpamayo 1.5, establishes a multi-axis faithfulness benchmark, and proposes TRACR (Trajectory-Reasoning Aligned Code Reward) to enforce reasoning-action consistency via GRPO post-training with scene-specific verified executable code rewards.

Critical strengths
  • Tackles an important and timely problem: evaluating and improving the causal faithfulness and reasoning-action consistency of VLA models in autonomous driving.
  • Conducts comprehensive mechanistic probing (including structural/positional sweeps, clause reordering, logit-level alternatives, and cache ablations) to diagnose failure modes like positional shortcutting and post-hoc rationalization.
  • Introduces TRACR, a novel post-training framework leveraging offline-verified, scene-specific executable code rewards to enforce semantic and visual alignment without degrading trajectory accuracy.
Critical weaknesses
  • Limited Scalability and Generalization of Code Rewards: The proposed TRACR framework relies on generating scene-specific executable Python programs and filtering them against counterfactual negatives. It remains unclear how well this automated code generation and verification pipeline scales to highly diverse, open-world long-tail scenarios without human-in-the-loop debugging.
  • Confounding Factors in Mechanistic Probing: While the mechanistic analysis (such as prefix/suffix sweeps and clause reordering) reveals strong sensitivity to token position and surface order, it is difficult to completely decouple whether trajectory shifts stem from true semantic reasoning changes or artifacts of autoregressive attention scaling over modified prompt lengths.
  • Incomplete Computational Overhead Analysis: The paper introduces multiple complex evaluation axes, offline program verification, and GRPO rollout sampling (K=12 candidates per scene), but lacks a detailed computational and latency profile during training and inference compared to standard baselines.
Questions for authors
  1. Could the authors discuss the failure rate and computational cost of the automated code generation and verification pipeline (Section 3.5) when scaled to thousands of diverse long-tail driving scenarios?
  2. To what extent are the trajectory shifts observed in the clause-reordering and prefix-sweeping probes caused by changes in autoregressive attention length and positional encodings rather than pure semantic disruption?
  3. Please provide a computational efficiency comparison (e.g., training time overhead, GPU memory, and inference latency) between TRACR and standard LLM-judge or global-reward GRPO baselines.
1 comment
AuthorsOpenReview ↗

Verifier Scalability, Positional Effects, and Compute Costs

Thank you for taking the time to read our work and for providing feedback. Thank you also for asking us to distinguish the cost of generating reward programs from the cost of using them, and to clarify what the interventional results establish about position and meaning. We do our sincere best to address your questions below:

How does program generation and verification scale? We generate each scene-specific program offline before performing GRPO post-training. We do this in four stages: 1) Identify the relevant event, 2) specify the acceptable commitments and motion, 3) define graded checks and contradiction penalties, and 4) implement these executable checks in Python through code subtask reward generation. We admit a program if it passes all four checks on the expert pair and accepts at most 6 of 100 cross-scene mismatches. Once admitted, it can score all 12 candidates in a GRPO group and be reused each time training revisits the scene.

At 1,000 scenes, evaluating one expert pair and 100 mismatches per scene requires 101,000 pair evaluations before retries. We measured the cost of scoring after authoring. Running 32 existing programs on 384 saved rollouts took 5.84 seconds on CPU, with zero execution errors. We did not measure end-to-end authoring cost or the fraction of programs requiring repair at this scale. Once a program is admitted, it can score additional rollouts without another LLM call. The practical advantage we can demonstrate is reuse (i.e., after a program is accepted, scoring additional rollouts requires only executing its checks). In generating offline reward functions, we achieved a ~93%+ code eligible reward training sample for strong code reward functions that are discriminative enough to reward trajectory-reasoning alignment. Additionally, we hope to curate a dataset that we can contribute to the community for code trajectory-alignment checks that are more expressive than global deterministic code functions with traditional rules.

How much of the trajectory shift comes from position or length? Reordering clauses shifted the predicted path by 0.168 m, compared with 0.0039 m when replaying the original text. This establishes sensitivity to clause order. Moving a maneuver clause also changes its position and preceding context, while a prefix or suffix sweep changes the amount of visible context. Those probes alone cannot isolate a change in meaning. Furthermore, as shown in our initial experiments, we also found that clause reversal, on average, exceeds entire masking the reasoning in terms of ADE, revealing sensitivity to reasoning, which we leverage through the code reward generation alignment rewards, which leverage the GT reasoning trace for rewards.

We also examined interventions that preserve sequence length. In Table 5, masking the emitted reasoning in the action expert’s cache while holding the images, generated sequence, and diffusion seed fixed shifted the trajectory by 0.128 m (95% CI 0.108 to 0.152 m, clustered over 939 clips). In a follow-up, we masked either the first parsed action span or the nearest non-action span of equal token count. Masking the action span displaced the path more, by 0.022 m on average for frozen Alpamayo (123 eligible clips) and 0.069 m for the step-720 checkpoint (110 clips). On the 99 clips eligible for both, the paired difference in this contrast was +0.045 m (95% CI −0.011 to +0.124). The within-model effect persists with token IDs and sequence length fixed, although the two spans still occupy different positions. Stricter equal-length clause swaps and action reversals yielded no eligible edits under our controls.

What training and inference overhead did we measure? In the 100-step pilot on the same eight B200 GPUs, the scene-specific ranked run took 124 minutes (16.5 allocated GPU-hours), compared with 132 minutes (17.6 allocated GPU-hours) for the global ranked verifier. These single-run measurements show comparable training time in this setting. The deployment inference latency and memory have 0% overhead because TRACR modifies only policy weights and does not run code at evaluation. For a peak memory profile, the LLM-judge relies on an external API call which doesn't affect training memory. The process of code reward generation is done entirely offline before training. At inference, both the reward program and the judge are absent from the execution path: the trained policy generates the reasoning trace and trajectory directly.

Reviewer_Nu4hOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Mechanistic analysis of reasoning–trajectory dependence in Alpamayo; an eight-axis alignment benchmark; TRACR, using scene-specific generated reward programs for GRPO post-training.

Critical strengths

Useful distinction between reasoning–action agreement and causal faithfulness.

Varied reasoning interventions, with replay controls and fixed sampling randomness.

Encouraging held-out trajectory results: slightly better ADE/FDE while improving judged alignment.

Critical weaknesses

Unclear GRPO implementation. No specification of trainable modules, policy likelihoods, or gradient flow. Particularly consequential for the diffusion action expert: how are likelihood ratios computed, and does the expert receive any RL updates? Reward equations alone insufficient to reconstruct the training method.

Gap between reward and claimed faithfulness. Code rewards checking agreement between reasoning and trajectory; no direct enforcement of causal dependence or paraphrase invariance. High reward possible even when reasoning has no influence on action generation.

Alignment reward insufficiently specified. Four conceptual checks, but insufficient concrete verifier examples and scoring details. Validation against one expert positive and cross-scene negatives offering limited assurance against subtle contradictions or reward exploitation.

Mixed faithfulness results. Greater reasoning dependence, but paraphrase invariance dropping from 99.0 to 33.9. Potentially increased language sensitivity without reliable semantic dependence. Slightly lower grounding; modest static consistency gains; no clear overall superiority across axes.

Limited experimental support. One rollout per held-out scene, no reported training-seed variability or uncertainty for the main comparisons, and no component ablations isolating the source of TRACR’s gains. Training data and OOD split construction also unclear.

Questions for authors

Which modules are updated? Which outputs enter the GRPO likelihood ratio? Exact objective for the diffusion expert, if trained?

Concrete reward-program examples, including parsing, graded scores, and failure cases?

Explanation for the large loss of paraphrase invariance, given its inclusion in the definition of faithfulness?

Training/evaluation split details and matched reward ablations separating scene-specific verification from reward shaping?

1 comment
AuthorsOpenReview ↗

GRPO, Rewards, and Training

Thank you for taking the time to read our work and provide feedback. Thank you also for asking us to clarify the GRPO update, the verifier, and the evidence for each reward component.

Which outputs receive the GRPO update? We sample 12 completions per scene, each containing a reasoning trace and discrete trajectory tokens. We decode both to compute one reward per completion and form advantages within the group. The clipped policy loss applies each completion’s advantage to its generated reasoning, trajectory, and trajectory-end tokens. Prompt tokens are excluded from the loss. For each scored token, we divide its current probability by its detached probability from the first forward pass on the same minibatch, conditioned on the same inputs and preceding tokens. A separate reference-policy pass supplies the KL term. We use one inner iteration, a token-mean loss, clipping bounds of 0.80 and 1.28, a KL weight of 0.001, and fused AdamW at 2 × 10⁻⁶. We froze the diffusion expert during GRPO training. It received no parameter updates, and we computed no likelihood ratio for its denoising steps.

What does the reward enforce? In a pedestrian-yield program, a trace such as “I will yield to the pedestrian” identifies the pedestrian as the stated reason and yielding as the maneuver commitment. The program checks the predicted speed profile for deceleration, persistence of the yield, and recovery, then combines text and motion checks into a score between 0 and 1. Among completions with ADE at or below 3 m and a valid score, we rank the scores and add a bonus of up to 0.15 to the trajectory and comfort reward. In a held-out comparison, the pre-repair verifier scored the judge-preferred pair at 1.000 and its alternative at 0.992. It ranked them in the preferred order, but the 0.008 gap fell below the required 0.05 margin. A planner can also earn a high score by accurately describing a maneuver it has already selected, which motivates the intervention tests.

Paraphrase invariance. Relative to frozen Alpamayo, TRACR increases full-reasoning dependence from 32.2 to 73.0 and counterfactual action response from 23.8 to 53.3. Paraphrase invariance falls from 99.0 to 33.9. The four other trained baselines also score between 30.5 and 33.7 on invariance. In a separate step-720 test, we changed one connective while preserving token count and the count of each parsed action. We held images and diffusion randomness fixed and compared each edit with forced replay of the original trace. On 147 jointly eligible clips, the paired difference in mean displacement versus frozen was +0.003 m (95% CI −0.009 to +0.014). These narrow edits produced similar average shifts across models. Table 3 shows why we report responsiveness to reasoning edits and robustness to rewording separately.

Training split and matched ablations. We used the Physical AI AV OOD reasoning collection for post-training and evaluation (~1K out of 1.7K clips for training). We ran a 711 step run used 946 training clips and 16 monitor clips, with 3 exclusions. The 211 clips drawn from the official OOD validation split shared no clip IDs with either set. They included 131 clips from a baseline-failure screening pool and 80 from the broad OOD pool.

The separate ablation pilot used 63 training clips and 16 monitor clips, all disjoint from the 211. The ranked scene-specific and global conditions shared the task reward, 3 m ADE gate, and 0.15 rank weight. We averaged four final rollouts per monitor clip and bootstrapped paired differences across the 16 clips. Mean ADE differences in meters (first minus second, 95% CI) were:

  • Scene-specific vs. global verifier, both ranked: −0.112 [−0.477, +0.198].
  • Ranked vs. raw score, same scene-specific code: −0.162 [−0.452, +0.095].
  • Assigned vs. shuffled within-group rank bonuses: −0.146 [−0.332, +0.020].
  • Gate enabled vs. disabled, same global verifier: −0.161 [−0.598, +0.185].

All four ADE point estimates favor the first condition, although each interval includes zero. On the 211-scene benchmark, the archived step-711 candidate reproduces the submitted means. Its blinded Gemma score improved over frozen by 0.313 points out of 5 (95% CI +0.118 to +0.512), and its mean ADE difference was −0.058 m (−0.220 to +0.100). These intervals quantify variation across scenes with one rollout per scene. The clearest held-out gain is in judged trace–trajectory agreement, alongside the lowest observed mean ADE and FDE in Table 2. We plan to further scale these results beyond 16 clips for verification in our paper.

Reviewer_JdnAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether natural-language reasoning in autonomous-driving VLAs agrees with, and causally influences, predicted trajectories. It probes Alpamayo 1.5 through reasoning masking, clause reordering, paraphrasing, alternative decoding, and cache interventions. The authors organize these tests into an eight-axis benchmark covering static alignment and intervention-based behavior. They propose TRACR, which generates scene-specific executable reward programs, validates them against expert demonstrations and mismatched reasoning–trajectory pairs, and freezes them for GRPO post-training. A trajectory-error gate restricts alignment bonuses to eligible candidates. Experiments compare TRACR with fixed-rule and LLM-judge rewards. On 211 held-out OOD scenes, TRACR achieves an ADE of 2.011 m versus 2.069 m for frozen Alpamayo, with higher holistic Gemma scores. However, the multi-axis results reveal substantial trade-offs between reasoning sensitivity and paraphrase invariance.

Critical strengths

The diagnostic study addresses an important distinction between plausible explanations and behaviorally influential reasoning. Combining masking, clause reordering, paraphrasing, and decoding interventions with replay-noise controls makes the analysis more informative than a text-quality or trajectory-only evaluation. The observed sensitivity to semantically equivalent clause reordering is particularly useful evidence about the limitations of the current policy.

TRACR provides a concrete and inspectable approach to alignment-focused post-training. Scene-specific executable rewards, offline validation, and trajectory-quality gating form a practical framework. The held-out results show improved holistic scores alongside slightly lower ADE/FDE than the frozen policy, suggesting that alignment-oriented training need not sacrifice open-loop trajectory accuracy in this evaluation.

Critical weaknesses

The central claim of improved faithfulness is insufficiently supported across the proposed axes. Table 3 shows higher full-reasoning dependence and counterfactual response, but paraphrase invariance falls from 99.0 to 33.9. Increased sensitivity may therefore include undesirable dependence on wording. TRACR also trails the scalar LLM-judge and global-reward baselines on static reasoning–action consistency. Appendix B.2 explicitly states that the Gemma audit does not measure interventional dependence, so its 9.21% improvement cannot independently establish improved causal faithfulness. These trade-offs directly affect the paper’s primary conclusion.

The reliability and distinct contribution of the generated rewards remain insufficiently established. Section 3.5 validates each program using an expert pair and 100 cross-scene mismatches; this does not establish discrimination of subtle within-scene contradictions or acceptance of valid alternative explanations. The reported experiments do not isolate scene-specific generation, verification/repair, and rank/gating through component ablations. Furthermore, the main 211-scene comparison uses one rollout per scene without uncertainty estimates for the reported improvements. Consequently, the robustness of the gains and their attribution to the proposed mechanism remain unclear.

Questions for authors

Clarification: How should the decline in paraphrase invariance from 99.0 to 33.9 be reconciled with the faithfulness claim? Please specify intervention construction and thresholds, and explain whether semantic changes or evaluation artifacts contribute.

Clarification: Which distinct metrics and baselines support the abstract’s “over 17%” consistency and “over 9%” faithfulness improvements? How are the static Gemma audit and intervention-based results distinguished in those claims?

Evidence request: Can you report verifier performance on within-scene minimal contradictions and valid paraphrases or alternative plans, including acceptance/repair rates and representative failures?

Evidence request: Can you provide component ablations under matched reward settings and uncertainty estimates for the held-out gains? These results would materially affect whether the improvements can be attributed to validated scene-specific code rewards.

1 comment
AuthorsOpenReview ↗

Metrics, Verifier, Ablations

Thank you for taking the time to read our work. Thank you also for asking about the distinct metrics/baselines, verifier performance, and component ablations.

Percentage Metrics. In Table 1, TRACR improves internal parser consistency from 0.266 to 0.313 relative to Global Reward–Rank (+17.7%). In Table 2, TRACR improves the blinded holistic Gemma score from 3.398 to 3.711 relative to frozen Alpamayo (+9.21%). Gemma evaluates how well a completed reasoning trace and its trajectory agree. Table 3 measures how the planned trajectory changes when we alter the trace. We will name each metric and baseline in the abstract and describe the 9.21% improvement as a gain in judged trace–trajectory agreement. The scalar LLM judge scores highest on Gemma (3.773), and both scalar baselines score higher on static reasoning–action consistency. TRACR achieves the lowest mean ADE and FDE in Table 2 and the highest full-reasoning dependence and counterfactual action response among the Alpamayo methods in Table 3.

Paraphrase invariance. Relative to frozen Alpamayo, TRACR increases full-reasoning dependence from 32.2 to 73.0 and counterfactual action response from 23.8 to 53.3. Paraphrase invariance falls from 99.0 to 33.9, with all four other trained baselines scoring between 30.5 and 33.7. Reporting these axes separately makes the trade-off visible: stronger responses to reasoning edits can coexist with sensitivity to wording.

Appendix B.1 defines invariance as the fraction of intended meaning-preserving rewrites whose trajectories stay within a replay-calibrated threshold. To isolate a specific wording change, we ran a separate test on 211 OOD clips. We replaced a single connective only when the edit preserved token count and the count of each parsed action. We held images and diffusion randomness fixed. Repeated denoising produced zero drift, while forcing the original text shifted the path by 0.0046 m on average, so we compared each edit with its forced original. Only 2/161 frozen-model edits and 5/170 step-720 edits exceeded the largest of 5 cm, 1% of the control trajectory’s first-to-last waypoint distance, and three times replay drift. On the 147 clips eligible for both models, the paired difference in mean displacement was +0.003 m (95% CI −0.009 to +0.014). Thus, for these connective substitutions, over 97% of edits in each model stayed within the specified threshold. The edits preserved parsed actions but lacked independent semantic labels.

Trajectory accuracy and uncertainty. We recovered all 211 matched clips from the original one-rollout evaluation. The TRACR step-711 candidate reproduces the submitted means. Across 20,000 paired bootstrap resamples over clips, the Gemma gain was +0.313 points on a 0–5 scale (95% CI +0.118 to +0.512). Mean ADE fell from 2.069 to 2.011 m (difference −0.058 m, 95% CI −0.220 to +0.100), and mean FDE fell from 5.854 to 5.627 m (−0.227 m, −0.754 to +0.278). Both trajectory intervals include zero. The clearest held-out gain is in judged agreement, alongside the lowest observed trajectory errors.

Verifier and ablations. On 16 monitor clips, we compared scene-specific and global scoring with the same task reward, 3 m gate, and rank weight. The paired difference in mean ADE favored scene-specific scoring (−0.112 m, 95% CI −0.477 to +0.198), while automated faithfulness slightly favored global scoring (scene-specific minus global, −0.024, 95% CI −0.055 to +0.008). The clearest evidence for TRACR is the improvement in blinded trace–trajectory agreement over frozen Alpamayo on 211 held-out clips, alongside the lowest observed mean ADE and FDE in Table 2.

Comments on the paper

1 comment
Area_Chair_U5rcOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.