acceptodds
Under review as a conference paper at ICLR 2027

AlterVLA: A Vision-Language-Action Model with Latent Temporal Change Prediction for Autonomous Driving

Abstract

Anticipating scene evolution can improve autonomous driving, but predicting complete future representations may devote modeling capacity to visual content already available in current observations. We introduce AlterVLA, a vision-language-action framework that uses compact, multi-horizon change tokens to focus foresight on scene evolution and provide complementary information for trajectory planning. Learning these representations in the pretrained model's own visual space avoids reliance on an external visual teacher. Predicted changes serve as direct planning conditions within a shared backbone, enabling continuous trajectory generation through a lightweight head without requiring future observations at inference. We further fine-tune AlterVLA with reinforcement learning to optimize safety, progress, and comfort beyond trajectory imitation. Accordingly, a four-stage training pipeline is developed to bridge future-supervised foresight and deployable planning. The planning interface is first learned with oracle changes and then preserved when predicted changes are introduced, before the policy is refined through reinforcement learning. Using a single front-view camera, AlterVLA achieves 88.53 PDMS on NAVSIM v1 and 87.82 EPDMS on NAVSIM v2 under supervised fine-tuning. With reinforcement fine-tuning, it achieves 90.94 PDMS on NAVSIM v1. Ablations show that temporal-change conditioning outperforms full-state prediction and the no-change baseline in PDMS. https://anonymous.4open.science/r/Alter_vla-369D/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vison Language Action, Autonomous Driving, Temporal Change Predictor

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ywEpOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AlterVLA, a vision-language-action model for autonomous driving that predicts temporal changes in visual feature space to condition trajectory planning. Built on Qwen3.5-2B, it learns compact change tokens at multiple future horizons using differences between current and future features from a frozen visual encoder. A four-stage pipeline first learns these representations, trains a planner with oracle change tokens, trains a predictor to replace them using current observations and ego information, and finally refines the trajectory head through reinforcement learning. The predicted change tokens directly condition diffusion-based trajectory generation within a shared backbone. Experiments using a single front-view camera on NAVSIM v1 and v2 report competitive planning performance, with ablations comparing temporal-change prediction against full-state prediction, no-change conditioning, and single-horizon variants.

Critical strengths
  1. Well-motivated temporal-change conditioning for planning. Since the planner already receives current visual features, predicting compact changes relative to those features is a reasonable way to represent future scene evolution. The predicted tokens preserve multiple temporal horizons and directly condition trajectory denoising. Learning these representations in the backbone’s own visual feature space also avoids an additional visual teacher.

  2. Relevant ablations and competitive planning performance. Table 3 compares the proposed representation with no-change conditioning, full-state prediction, single-horizon prediction, and an oracle reference. Multi-horizon change conditioning improves PDMS from 86.27 to 88.53 over the no-change baseline and outperforms the single-horizon variants, supporting the usefulness of the design. The method also achieves competitive results on NAVSIM v1 and v2 using a 2B backbone and a single front-view camera.

Critical weaknesses
  1. Limited evidence for the advantage and generality of temporal-change representations. Table 3 provides a useful comparison with full-state prediction, but the additional gain is modest: 88.53 versus 88.20 PDMS. The paper does not clearly specify whether the full-state baseline matches the proposed method in token budget, module capacity, and training procedure. Moreover, all experiments use Qwen3.5-2B. Since the change representation is learned in the backbone’s own visual feature space, comparisons across model families and larger parameter scales would help establish whether its advantage extends beyond this configuration.

  2. The benefit of the four-stage training procedure is not sufficiently isolated. The planner is first trained with oracle change tokens and then frozen while the predictor learns to reproduce those tokens, without trajectory supervision in Stage III. Although Stage IV subsequently adapts the trajectory head through RL, the paper does not compare this procedure with training the planner directly on predicted changes or jointly fine-tuning the predictor and planner after the switch. The oracle-versus-predicted comparison measures the effect of conditioning quality rather than the advantage of the training schedule. Matched-budget comparisons would clarify whether the staged design offers benefits over more direct alternatives.

Questions for authors
  1. In Table 3, does the full-state prediction baseline use the same token budget, module capacity, and training procedure as temporal-change prediction? Please clarify any differences that could affect the 0.33-point PDMS gap.

  2. Does the advantage of temporal-change prediction over full-state prediction persist on another VLM family or a larger backbone? A targeted comparison of no-change, full-state, and temporal-change conditioning would help assess generality.

  3. How does the oracle-first training procedure compare with training the planner directly on predicted changes, or jointly fine-tuning the predictor and planner with trajectory supervision after Stage III? Comparisons under matched training budgets would clarify the benefit of the proposed schedule.

1 comment

Thank you for asking about the full-state baseline, backbone scale, and training schedule. All PDMS results below are on NAVSIM v1.

Q1 — Full-state baseline and experimental controls

In Table 3, full-state and temporal-change prediction both use 128 foresight tokens per scene, with 32 tokens at each of 0.5, 1, 2, and 4 s. The follow-up comparisons also match planner capacity, training stages, and optimization budget.

Temporal-change prediction improves SFT PDMS in all three paired training seeds, with a mean gain of 0.40 points (sample SD 0.07, n = 3).

Training seedFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
4288.2088.53+0.33
13788.0188.48+0.47
82988.0888.49+0.41

Predicted change also improves SFT PDMS at all four tested denoising step counts, with gains of 0.33–0.50 points. The 10-step row is the seed-42 reference from Table 3.

Denoising stepsFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
187.9988.49+0.50
288.0888.47+0.39
588.1188.52+0.41
1088.2088.53+0.33

Q2 — Larger backbones and other VLM families

On Qwen3.5-4B, adding temporal-change conditioning improves SFT PDMS from 89.077 to 89.418 (+0.341 points) relative to the no-change baseline (revised paper, Appendix A.4, Table 10). This result shows a benefit at a larger scale within the Qwen3.5 family. A comparison with full-state prediction at 4B and evaluation on other VLM families remain open.

Q3 — Oracle-first training, direct training, and joint fine-tuning

Oracle-first training leads direct training with predicted change tokens by 4.93 PDMS points after SFT and 3.40 points after RFT. The two runs match planner architecture, hidden-state inputs, planner training budget, and evaluation settings. Both use the same frozen predictor at inference.

Training strategySFT PDMSRFT PDMSRFT gain
Direct training with predicted change tokens83.6087.54+3.94
Original Stage III (oracle-first)88.5390.94+2.41
Joint fine-tuning after Stage III89.0190.07+1.06

For the same oracle-trained planner, switching from oracle to predicted tokens at inference reduces PDMS from 89.10 to 88.53 (−0.57 points).

Joint fine-tuning adds supervised updates to the predictor and planner, improving SFT PDMS by 0.48 points. Applying the same RFT procedure to the original and jointly fine-tuned checkpoints gives 90.94 and 90.07 PDMS, respectively. We use the original Stage III checkpoint to initialize Stage IV because it yields 0.87 points higher final PDMS in this comparison.

Rollout diversity. Before RFT, we sample 8 trajectories per NAVTRAIN scene token from each checkpoint. The jointly fine-tuned checkpoint produces more tightly clustered candidates.

Candidate diversity metricOriginal Stage IIIJoint fine-tuning
Mean pairwise trajectory distance0.311 m0.091 m
Mean pairwise endpoint distance0.465 m0.168 m
Longitudinal endpoint SD0.341 m0.127 m
Lateral endpoint SD0.126 m0.039 m
Scenes with lateral endpoint SD < 10 cm61.50%90.05%
Reviewer_dxG4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AlterVLA, a model of visual temporal change in latent space. It predicts latent differences between adjacent frames and the current frame to help the model learn information about future object motion, velocity, and occupancy changes. The paper explores learning scene evolution through latent representations and demonstrates the effectiveness of this approach.

Critical strengths

The paper explores an approach to learning latent differences. The overall design is reasonable and interesting, offering new insights to the community.

Critical weaknesses

Learning latent changes does not appear fundamentally different from directly predicting the future and seems to share the problems of conventional world models. The paper does not clearly establish the advantages of the proposed approach over conventional methods. The experimental procedure involves too many stages, making it difficult to discern the contribution of each stage. The additional use of reinforcement learning (RL) appears to have limited relevance to the proposed approach and does not seem to better support the paper’s central argument. Its necessity therefore requires explanation.

Questions for authors

Could the authors explain why all four stages are necessary and provide evidence of the gains contributed by each? This would help assess the role of each stage in the overall approach. Could the authors explain why RL is necessary and provide evidence showing how it supports the proposed approach? This would clarify its relevance to the paper’s central argument. Do the gains from RL depend on the change condition, or would applying the same RL procedure to the same planner without change conditioning yield similar gains? Please provide a comparison to establish whether RL supports the paper’s central claim. With the same token count, model capacity, and training budget, can the 0.33-point advantage of change over full-state be consistently reproduced? Evidence of reproducibility would help assess the reliability of this relatively small improvement.

4 comments
AuthorsOpenReview ↗

Response Q1, Q3 and Q4

Thank you for the questions about our training stages, reinforcement fine-tuning, and the reproducibility of the full-state comparison. All PDMS scores below are on NAVSIM v1.

Q1 — What does each training stage contribute?

Thank you for asking us to clarify the training procedure. The purpose of temporal-change prediction is to improve trajectory generation. The four stages learn the change representation, teach the planner to use it, replace oracle inputs with predictions, and refine trajectories using driving rewards.

Stage I — Learn the change representation. The Temporal Change Model compresses differences between current and future visual features into compact codes. A decoder reconstructs future visual features from these codes and current features, providing supervision for representation learning. The reported training MAE is 0.1050.

Stage II — Teach the planner to use change information. We condition trajectory denoising on oracle change tokens extracted from recorded future observations and supervise the output with expert trajectories. This lets the planner learn the relationship between accurate change information and trajectory generation. Oracle conditioning achieves 89.10 PDMS, compared with 86.27 without change conditioning, a gain of 2.83 points.

Stage III — Predict the conditioning tokens from current inputs. We freeze the planner and representation modules and train the predictor to produce compatible change tokens from the current image and ego inputs. At inference, these predicted tokens directly condition trajectory denoising. Switching the same planner from oracle to predicted tokens changes PDMS from 89.10 to 88.53, a reduction of 0.57 points. The resulting planner remains 2.26 points above the no-change baseline.

To test the oracle-first schedule, we also train a planner directly with predicted change tokens, omitting oracle-conditioned planner training. This comparison matches planner architecture, hidden-state inputs, planner training budget, and evaluation settings. Both planners use the same frozen predictor at inference.

Planner configurationConditioning at evaluationSFT PDMS
Without change conditioningNo change tokens86.27
Stage II plannerOracle tokens89.10
Oracle-first planner after Stage IIIPredicted tokens88.53
Planner trained directly with predicted tokensPredicted tokens83.60

Oracle-first training improves PDMS by 4.93 points over direct predicted-token training. This comparison supports first teaching the planner to use accurate change information, then supplying predicted codes through the same conditioning interface.

Stage IV — Refine trajectories using driving rewards. RFT updates only the trajectory head and improves PDMS from 88.53 to 90.94 (+2.41 points). This stage is optional. The supervised result of 88.53 PDMS already demonstrates the benefit of predicted-change conditioning, while RFT further improves driving performance.

Q3 — Does the RFT gain depend on change conditioning?

RFT improves both planners. We apply the same Stage IV procedure and update only the trajectory head. The backbone and upstream modules remain frozen, including the change predictor when present.

Change conditioningSFT PDMSRFT PDMSRFT gain
Without change86.2790.16+3.89
With predicted change88.5390.94+2.41

The planner without change conditioning gains more from RFT. AlterVLA leads by 2.26 PDMS points before RFT and 0.78 points after RFT (revised paper, Appendix A.5, Table 11).

Q4 — Is the advantage over full-state prediction reproducible?

Predicted change improves SFT PDMS in all three paired training seeds. Both variants use 128 foresight tokens per scene, with 32 tokens at each of 0.5, 1, 2, and 4 s. The follow-up runs match planner capacity, training stages, and optimization budget. Given the current features, uncompressed temporal differences and future features determine each other (Appendix A.1). Our comparison measures which target works better under the same compression budget.

The mean paired gain is +0.40 PDMS points (sample SD 0.07, n = 3).

Training seedFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
4288.2088.53+0.33
13788.0188.48+0.47
82988.0888.49+0.41

Predicted change also leads at every tested denoising step count, with gains of 0.33–0.50 PDMS points. The 10-step row is the seed-42 reference from Table 3.

Denoising stepsFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
187.9988.49+0.50
288.0888.47+0.39
588.1188.52+0.41
1088.2088.53+0.33
AuthorsOpenReview ↗

Q2 — Reinforcement Fine-Tuning Improves Ego Progress and PDMS

Q2 — Why use reinforcement learning?

We use reinforcement fine-tuning (RFT) to improve ego progress (EP) while accounting for safety, drivable-area compliance, and comfort. Supervised fine-tuning (SFT) teaches the planner to imitate expert trajectories and provides only indirect supervision for the progress of generated trajectories. RFT scores multiple candidates per scene with a PDMS-based reward, giving the planner a direct signal to favor trajectories that make progress while meeting other driving objectives. RFT updates only the trajectory head.

PlannerStageNCDACEPComfortPDMS
Without change conditioningSFT98.9095.8080.6095.2086.27
Without change conditioningRFT99.2897.1083.5099.9090.16
AlterVLA with predicted changeSFT99.6896.1282.0395.7488.53
AlterVLA with predicted changeRFT99.798.384.9100.090.94

RFT improves EP by about 2.9 points in both planners. In AlterVLA, EP increases from 82.03 to 84.9, alongside improvements in drivable-area compliance and comfort. PDMS rises from 88.53 to 90.94 (+2.41 points). Predicted-change conditioning already contributes +2.26 PDMS points at the supervised stage, and RFT further improves the planner through direct feedback on its generated trajectories.

Thank you for the detailed clarifications and additional experiments. The stage-wise ablations, matched training comparisons, and multi-seed results have addressed my main concerns and strengthened the empirical support for the method. I am therefore raising my score.

Thank you for your careful review and constructive feedback. We appreciate your reconsideration and are glad that the additional experiments addressed your main concerns. We will ensure that the stage-wise ablations, matched training comparisons, and multi-seed results are clearly included in the revised manuscript.

Reviewer_esPSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AlterVLA, a vision-language-action framework for autonomous driving that conditions trajectory generation on predicted temporal changes rather than complete future visual states. It learns compact change tokens from differences between current and future features extracted by a frozen Qwen visual encoder at multiple horizons. A predictor estimates these tokens from current observations and ego information, while a shared backbone incorporates them into continuous trajectory denoising. The main contributions are multi-horizon change modeling in the pretrained model’s native visual space, direct change-conditioned planning, and a four-stage training pipeline comprising representation learning, oracle-conditioned planning, change prediction, and reinforcement fine-tuning. Using a single front-view camera, the method reports 88.53 PDMS on NAVSIM v1 and 87.82 EPDMS on NAVSIM v2 under supervised training, with reinforcement fine-tuning improving v1 PDMS to 90.94. Validation includes comparisons with existing methods, ablations of future representations, prediction horizons, and trajectory generation methods, as well as qualitative trajectory and attention visualizations.

Critical strengths
  1. A well-motivated approach to using future supervision for planning. AlterVLA learns compact, multi-horizon change representations in the pretrained model’s own visual feature space and directly incorporates their predictions into trajectory generation. Reusing the frozen visual encoder avoids an additional visual teacher, while the predictor enables inference without future observations. This provides a coherent mechanism for transferring training-time foresight into deployable planning, making the representation design practically relevant.
  2. Competitive supervised performance with informative internal comparisons. Using a single front-view camera, AlterVLA reports 88.53 PDMS on NAVSIM v1 and 87.82 EPDMS on NAVSIM v2 before reinforcement fine-tuning. Its representation ablation improves PDMS from 86.27 without change conditioning to 88.53 with predicted multi-horizon changes, compared with 89.10 using oracle changes. These results provide encouraging evidence that predicted change conditioning benefits planning and preserves much of the performance attainable with privileged future information, independently of the subsequent reinforcement-learning gains.
Critical weaknesses
  1. The substantive novelty over closely related residual and future-aware planning methods is insufficiently established. ResWorld[1] already motivates temporal residual modeling by avoiding redundant prediction of static scene content and uses predicted future representations to improve trajectory planning. RLA-WM[2] also provides a closely related mechanism for compressing visual feature differences into latent representations. AlterVLA introduces differences through native VLM features, multi-horizon tokens, and direct action conditioning, but does not adequately establish the substantive advance enabled by this combination. In particular, the absence of discussion of ResWorld leaves an important gap in positioning the contribution. This limits my confidence that the paper provides sufficient new insight beyond integrating existing ideas.
  2. The experimental evidence does not convincingly establish the proposed design’s incremental value. The comparison directly supporting change prediction over full-state prediction shows only a 0.33 PDMS improvement (88.53 vs. 88.20). Furthermore, the overall benchmark results provide limited independent evidence of a practical advance: AlterVLA’s 90.94 PDMS after reinforcement fine-tuning is below the 91.4 reported by FLARE[3] under single-camera, single-sample RFT evaluation, while potential efficiency benefits remain unquantified. Together, these limitations leave the claimed representation advantage and its practical significance insufficiently demonstrated. [1] Zhang, Jinqing, et al. "Resworld: Temporal residual world model for end-to-end autonomous driving." arXiv preprint arXiv:2602.10884 (2026). [2] Zhang, Xinyu, et al. "Learning Visual Feature-Based World Models via Residual Latent Action." arXiv preprint arXiv:2605.07079 (2026). [3] Xie, Chengen, et al. "FLARE: Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Driving." arXiv preprint arXiv:2601.05611 (2026).
Questions for authors
  1. Clarification of novelty: How does AlterVLA advance beyond the temporal residual modeling and future-guided planning in ResWorld, and the residual latent encoding mechanism in RLA-WM? Which aspect—native VLM features, multi-horizon representations, or direct action conditioning—constitutes the principal new contribution, and what evidence isolates its benefit?
  2. Clarification of experimental controls: Does the full-state prediction baseline match AlterVLA in token count, model capacity, training stages, and optimization budget? How many tokens are used by each single-horizon baseline? If the budgets differ, a fixed-budget comparison would help establish whether the reported gains come from the proposed representation and temporal coverage.
  3. Clarification of practical advantages: Does the shared-backbone design provide measurable computational benefits relative to approaches with separate planners? Please report end-to-end inference latency, hardware, batch size, and sampling settings, including the change predictor and all denoising steps. These measurements would help assess whether the architecture offers a meaningful efficiency–performance tradeoff.
5 comments

Thank you for asking us to clarify the contribution, experimental controls, and inference cost. All PDMS scores below are on NAVSIM v1.

Q1 — Novelty and supporting evidence

AlterVLA's main contribution is to condition trajectory denoising directly on 128 predicted change tokens across four horizons within a shared VLM backbone. These tokens encode temporal changes in the VLM's own visual features. ResWorld uses predicted future BEV features to refine an initial trajectory, while RLA-WM uses residual latents for action-conditioned future prediction and policy learning.

The no-change ablation measures the overall benefit of adding predicted-change conditioning. The standalone DiT comparison evaluates two planner implementations.

ComparisonBaseline SFT PDMSAlterVLA SFT PDMSGain
No change vs. predicted change86.2788.53+2.26
Standalone DiT vs. shared backbone87.1888.53+1.35

The standalone DiT and AlterVLA use the same batch size and evaluation settings. Their scores are reported after 40 and 8 epochs, respectively. The appendix heatmaps illustrate attention in the oracle change encoder. In the single-camera RFT comparison, AlterVLA reaches 90.94 PDMS, 0.46 points below FLARE's reported single-sample score of 91.40.

Q2 — Experimental controls and temporal coverage

In Table 3, full-state and temporal-change prediction both use 128 foresight tokens per scene, with 32 tokens at each of 0.5, 1, 2, and 4 s. The follow-up comparisons also match planner capacity, training stages, and optimization budget. All scores in this section are SFT PDMS.

Temporal coverage. The 1 s and 2 s baselines each allocate all 128 tokens to one horizon. At the same total token budget, the four-horizon configuration scores 0.90 PDMS points higher than the stronger 2 s baseline.

Horizon configurationTokensPDMS
2 s only12887.63
0.5, 1, 2, and 4 s4 × 3288.53

Paired training seeds. Predicted change leads in all three pairs, with a mean gain of +0.40 PDMS points (sample SD 0.07, n = 3).

Training seedFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
4288.2088.53+0.33
13788.0188.48+0.47
82988.0888.49+0.41

Denoising steps. Predicted change leads at every tested step count, with gains of 0.33–0.50 PDMS points. The 10-step row is the seed-42 reference from Table 3.

Denoising stepsFull state (128 tokens)Predicted change (128 tokens)Δ PDMS
187.9988.49+0.50
288.0888.47+0.39
588.1188.52+0.41
1088.2088.53+0.33
AuthorsOpenReview ↗

W1 — Novelty of Direct Change Conditioning for Trajectory Generation

Thank you for pointing out the overlap with ResWorld and RLA-WM. Both are closely related precedents for residual modeling, and we will make this relationship explicit in the related-work discussion.

What AlterVLA adds. The principal contribution is to use predicted visual changes directly as conditioning for trajectory denoising within a shared VLM backbone. ResWorld predicts future BEV features and uses them to refine an initial trajectory. RLA-WM encodes DINO feature residuals and uses action-conditioned latent predictions to reconstruct future visual features. AlterVLA instead predicts 128 change tokens across four horizons from the current observation and ego state. The planner uses those tokens directly during denoising, with no future-feature reconstruction at inference. Using the VLM's own visual features lets the change model reuse the existing visual encoder.

What the ablations establish. All results below are supervised PDMS on NAVSIM v1, before reinforcement fine-tuning.

ComparisonResultWhat it tests
No change conditioning vs. predicted change conditioning86.27 → 88.53 (+2.26)The planning benefit of adding predicted-change conditioning
Full-state vs. temporal-change prediction, both with 128 tokens+0.40 PDMS on average across three paired seedsThe benefit of representing change at a fixed token budget
2 s only vs. four horizons, both with 128 tokens87.63 → 88.53 (+0.90)The benefit of broader temporal coverage in the tested configurations

These results support the use of compact, multi-horizon change predictions as inputs to trajectory generation. Quantifying the separate effect of using native VLM features would require a matched visual-encoder ablation.

AuthorsOpenReview ↗

Response to Q3

Q3 — Why does the shared backbone reduce inference cost?

Denoising architecture. During training, the diffusion process adds noise to the normalized expert trajectory A0A_0 at diffusion time ss:

\qquad \epsilon \sim \mathcal{N}(0,I).$$ At inference, sampling starts from noise. At each denoising step, AlterVLA embeds the **8 noisy waypoints** and the diffusion time as action tokens. The shared VLM backbone processes these tokens with a prefix $P$ containing visual, ego-state, and predicted-change tokens. A lightweight MLP reads the resulting action hidden states and predicts the clean trajectory: $$\widehat{A}_{0} = g_{\omega}\left(F_{\theta}\left([P; E_{\mathrm{act}}(A_{s},s)]\right)_{\mathrm{act}}\right).$$ DPM-Solver++ uses this clean-trajectory estimate to update the trajectory. In the standalone DiT baseline, a separate transformer denoises the action tokens and reads the conditioning features through cross-attention. **Compute and parameters.** Let $N$ be the number of action tokens, $M$ the number of conditioning tokens, $d$ the DiT hidden width, and $d_c$ the conditioning width. Within a standard DiT block, action self-attention costs $O(N^2d)$ for token interactions. Cross-attention adds $O(NMd)$ for attention scores and value aggregation at each denoising step, even if the conditioning keys and values are cached. Across $K$ denoising steps and $L$ cross-attention layers, this term scales as $O(KLNMd)$. Each cross-attention layer also has $2d^2 + 2d_cd$ projection parameters, excluding biases. AlterVLA handles these action–condition interactions within the shared backbone and uses an MLP to read out the trajectory. **End-to-end measurements.** We benchmarked both implementations on **8 NVIDIA H20 GPUs** with **16 samples per GPU**. Both implementations use **10 DPM-Solver++ denoising steps**. AlterVLA uses one front-view image. End-to-end timing covers input preprocessing, visual encoding, change prediction, all denoising steps, and trajectory postprocessing. | Planner | End-to-end latency | Reported VRAM | Throughput assuming batch latency (samples/s) | |---|---:|---:|---:| | Standalone DiT | 87 ms | 3,971 MB | 1,471 | | AlterVLA (shared backbone) | **61 ms** | **2,365 MB** | **2,098** | In this benchmark, AlterVLA reduces end-to-end latency by **26 ms (29.9%)** and reported VRAM use by **1,606 MB (40.4%)**. The corresponding throughput is **42.6% higher** under the batch-latency assumption.

Comments on the paper

3 comments
Area_Chair_wpCQOpenReview ↗

Discussion

Dear Reviewers, the authors have posted their rebuttal. Please review their responses and engage in the discussion.

Best, AC

AuthorsOpenReview ↗

General Response

Dear ICLR 2027 AC, SAC, PC, and Reviewers (esPS, dxG4, and ywEp),

Thank you for your feedback. During the discussion, we provided additional experiments and methodological clarifications addressing the reviewers’ concerns. The table below summarizes the evidence and the corresponding point-by-point responses. All PDMS results reported here are on NAVSIM v1.

ConcernReviewer(s)Additional Evidence and Clarifications
Novelty and relationship to prior workesPS: W1, Q1; dxG4: weaknessesClarified the relationship to ResWorld and RLA-WM. AlterVLA directly conditions trajectory denoising on 128 predicted temporal-change tokens across four horizons within a shared VLM backbone. Adding predicted-change conditioning improves SFT PDMS from 86.27 to 88.53 (+2.26). Response to esPS Q1.
Incremental value over full-state predictionesPS: W2, Q2; dxG4: Q4; ywEp: Q1Compared full-state and temporal-change prediction with 128 tokens, matched planner capacity, training stages, and optimization budgets. Temporal-change prediction leads across three paired 2B training seeds, with gains of +0.33, +0.47, and +0.41 PDMS and a mean gain of +0.40 (sample SD 0.07). It also leads at 1, 2, 5, and 10 denoising steps, with gains of 0.33–0.50 points. Responses to esPS Q2, dxG4 Q4, and ywEp Q1.
Temporal coverage under a fixed token budgetesPS: Q2Clarified that the 1 s and 2 s single-horizon baselines each use 128 tokens, while the four-horizon configuration uses 32 tokens per horizon. At the same total budget, four-horizon conditioning improves PDMS from the stronger 2 s baseline’s 87.63 to 88.53 (+0.90). Response to esPS Q2.
Role of each training stagedxG4: Q1Explained how Stage I learns compact change representations through future-feature reconstruction, Stage II trains the planner with oracle codes, Stage III trains a predictor to supply compatible codes from current inputs, and optional Stage IV refines trajectories using driving rewards. With the same planner, replacing oracle codes with predictions changes PDMS from 89.10 to 88.53 (−0.57). Response to dxG4 Q1.
Oracle-first training and joint fine-tuningdxG4: Q1; ywEp: Q3Added direct predicted-token training and joint-fine-tuning comparisons. Oracle-first and direct training use the same frozen predictor at inference and achieve 88.53 vs. 83.60 SFT PDMS (+4.93). Joint fine-tuning improves SFT PDMS to 89.01, but subsequent RFT reaches 90.07, compared with 90.94 when initialized from the original Stage III checkpoint. Responses to dxG4 Q1 and ywEp Q3.
Purpose of RFT and dependence on change conditioningdxG4: Q2–Q3Applied the same RFT procedure to both planners, updating only the trajectory head. Without change conditioning, PDMS improves from 86.27 to 90.16. With predicted changes, it improves from 88.53 to 90.94, retaining a 0.78-point advantage. AlterVLA’s ego progress also increases from 82.03 to 84.9, alongside improvements in drivable-area compliance and comfort. Responses to dxG4 Q2–Q3.
Larger-backbone validationywEp: Q2 and follow-upAdded matched 128-token full-state comparisons on Qwen3.5-4B. Temporal-change prediction improves PDMS from 89.17 to 89.42 (+0.25) for seed 42 and from 89.11 to 89.39 (+0.28) for seed 137. These results extend the observed advantage to 4B within the Qwen3.5 family. Other VLM families remain future work. Follow-up response to ywEp.
Practical efficiency and benchmark performanceesPS: W2, Q3Benchmarked both planner implementations on 8 NVIDIA H20 GPUs, with 16 samples per GPU and 10 DPM-Solver++ steps. Timing covers preprocessing, visual encoding, change prediction, all denoising steps, and postprocessing. The shared backbone reduces latency from 87 to 61 ms (29.9%) and reported VRAM from 3,971 to 2,365 MB (40.4%). Our main model uses a 2B backbone, compared with FLARE’s 4B backbone, while its 90.94 RFT PDMS remains 0.46 points below FLARE’s reported 91.40. Response to esPS Q3 and follow-up clarification.

Reviewer ywEp confirmed that the reproducibility and training-schedule concerns were addressed. After receiving the matched 4B comparisons, the reviewer also confirmed that the remaining concern was addressed and indicated that they were raising their rating. Another reviewer maintained their rating, considering the contribution insufficient despite the clarifications and additional evidence. We respect this assessment and submit the above evidence for the Area Chair’s consideration.

We will ensure that the additional experiments and clarifications, including both 4B seed comparisons, are incorporated into the revised manuscript. We sincerely appreciate your careful consideration and welcome any remaining questions.

Best regards, The Authors of AlterVLA

AuthorsOpenReview ↗

Clarification to the Area Chair Regarding Reviewer esPS’s Remaining Novelty Concern

Clarification Regarding Reviewer esPS’s Remaining Novelty Concern

Dear Area Chair,

We would like to summarize Reviewer esPS’s remaining concern and the evidence provided during the discussion, to assist your assessment.

Original concerns and our responses

Reviewer esPS questioned the substantive advance over ResWorld and RLA-WM, the fairness of the representation comparisons, and the practical benefit of the shared-backbone design. We provided the following clarifications and experiments. All PDMS results below are on NAVSIM v1.

ConcernClarification and supporting evidence
Contribution beyond existing residual modelingClarified that AlterVLA predicts compact changes in the VLM’s own visual features and directly conditions trajectory denoising on these predictions within a shared backbone. Adding predicted-change conditioning improves SFT PDMS from 86.27 to 88.53 (+2.26).
Representation advantage under matched budgetsBoth full-state and change prediction use 128 tokens, with matched planner capacity, training stages, and optimization budgets. Change prediction improves PDMS across all three paired 2B seeds by +0.33, +0.47, and +0.41, averaging +0.40 points.
Benefit of multi-horizon modelingAt the same 128-token budget, four-horizon conditioning improves PDMS from the stronger 2 s baseline’s 87.63 to 88.53 (+0.90).
Evidence at a larger backbone scaleMatched 128-token full-state comparisons on Qwen3.5-4B show gains of +0.25 and +0.28 PDMS across two paired seeds.
Practical benefit of the shared backboneWith matched hardware, batch size, and denoising steps, end-to-end latency decreases from 87 to 61 ms, and reported VRAM decreases from 3,971 to 2,365 MB relative to our standalone DiT.

Reviewer’s assessment after the rebuttal

Reviewer esPS stated that the experimental setup, controls, and related-work comparison had largely addressed their concerns, and explicitly confirmed that our response to Q2 had fully resolved their reservations about experimental fairness.

The reviewer nevertheless maintained a score of 2 (Weak Reject) because they considered the distinctions from prior approaches to be implementation techniques and judged the overall novelty insufficient. We understand that this remaining reservation concerns the significance of the contribution.

Our position for the AC’s consideration

Our contribution builds on established residual modeling and studies its use as compact, multi-horizon conditioning for VLM-based trajectory generation. The controlled experiments quantify the benefits of the representation choice and temporal coverage, while the larger-backbone comparisons and inference measurements provide additional evidence of applicability and practical value.

AlterVLA demonstrates competitive planning performance against existing baselines and consistent improvements over full-state prediction across training seeds and backbone scales under matched budgets. The shared-backbone design also reduces inference latency and GPU memory use. We respectfully ask that the contribution be assessed together with these reproducible planning gains and measured efficiency benefits. We will further clarify the contribution and the scope supported by the experimental evidence in the revised manuscript.

Thank you for your careful consideration.

Best regards, The Authors of AlterVLA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.