acceptodds
Under review as a conference paper at ICLR 2027

Spot the Difference: Efficient VLA Inference via Differential Coding

Abstract

Vision-language-action (VLA) models inherit the perception and reasoning of large vision-language models (VLMs), but they pay for it on every policy query: a multi-billion-parameter VLM re-encodes each new observation, even though consecutive frames of a manipulation rollout are highly redundant. We propose Differential VLA, which applies a principle shared by differential coding in communications, predictive coding in cognitive science, and I-frame/P-frame video compression: *encode a reference once, and just spot the difference afterwards*. Differential VLA runs the VLM (System 2) on a reference frame only once every control steps and caches its tokens. In between, a lightweight *differential encoder* lets the features of the current frame attend to those of the reference frame and compresses the residual into a handful of *difference tokens*, which are appended to the cached tokens that condition the diffusion-transformer action head (System 1). Built on GR00T N1.7 with , Differential VLA achieves an average success rate of 93.5% on LIBERO, which is within 1.0 points of the baseline, while calling the VLM 8× less often. An amortized query costs 1.8× fewer FLOPs and runs 1.27× faster than an equally optimized baseline. On SimplerEnv, Differential VLA stays within 6.6 points of the baseline while calling the VLM 16× less often, and on two real-world tasks with an SO-101 arm, it matches the baseline's success rate with 1.22 to 1.27× lower latency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
VLA, vision-language-action models, efficient inference, robot manipulation, dual-system architectures, temporal redundancy, differential coding, predective coding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TqK7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Differential VLA to exploit temporal redundancy in robot observations. Built on GR00T N1.7, the method evaluates the expensive VLM on a reference observation every T environment steps and caches its output. Between refreshes, a DINOv2-small encoder and reference-conditioned cross-attention produce a small number of additional tokens from the current observation. These tokens condition the action head alongside the cached VLM representation.

Critical strengths

Combining infrequent semantic encoding with fresh visual feedback is a sensible way to avoid both repeated VLM computation and purely stale conditioning.

The experimental results are promising.

Critical weaknesses

LIBERO-Goal decreases from 96.2% to 87.7%, including a decrease from 99% to 53% on bowl-on-plate (Table 14). The explanation that similar failures persist with T=8 is plausible, but quantitative results are needed to distinguish fine-tuning effects from stale conditioning. On SimplerEnv, the average decrease of 6.6 percentage points combines large losses, such as carrot-on-plate falling from 65.6% to 31.3%, with a large gain on eggplant-in-sink from 1.0% to 45.5% (Table 7).

The zero and identity interventions in Section 4.6 demonstrate reliance on fresh visual information, but both remove the actual current observation. They do not establish the benefit of conditioning the fast encoder on the reference frame. The most important missing experiment is a trained current-only ablation using the same DINO backbone, pooling, number of tokens, cached VLM context, refresh interval, and training recipe.

A simple feature-subtraction variant would further test the value of learned cross-frame matching. These are targeted mechanism ablations, not a request to reimplement every related system.

Questions for authors

How does a trained current-only visual branch compare with the reference-conditioned encoder under the same backbone, token budget, refresh schedule, and training recipe? Does simple feature subtraction perform comparably? Can a limited T sweep and uncertainty estimates from the existing evaluation structure establish how stable the reported trade-off is?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TqK7 (1/3)

We thank the reviewer for pinpointing the mechanism ablations that were missing, and we apologize for the late response, caused by extensive new experiments with limited resources. We agree with the assessment and have run those ablations. Please see the general response for the changes in the revised PDF.

W1. LIBERO-Goal and SimplerEnv degradation.

The LIBERO-Goal result in the submission is superseded. The drop was specific to the submitted default, i.e., cross-attention injection with the concat-project pooler. The revised PDF uses self-attention injection and reports Goal at 96.5% against 96.6% for the official checkpoint, with the bowl-on-plate task (8) at 100% against 98% (per-task results in Table 18):

MethodSpatialObjectGoalLongAvg.
N1.7 official ckpt., our eval.95.398.396.690.095.1
N1.7 + cache (T=64)0.20.03.00.00.8
Ours (T=64), submitted (cross-attn.)95.799.487.791.393.5
Ours (T=64), revised (self-attn.)95.898.996.592.495.9

In the revised ablation grid, the submitted configuration re-trained reaches 83.9% on Goal, with the loss on three bowl-placing tasks (1, 4, 8 at 73%, 66%, 48%), while the other seven pooler/injection combinations lie between 95.8% and 96.9%. Since all eight configurations use the same T = 64 and the same cached VLM tokens, stale conditioning alone cannot explain a drop that only one of them shows. The qualitative "same failures at T = 8" statement has been removed.

On SimplerEnv the gap (53.1% vs. 61.3%) remains and is uneven across tasks, as the reviewer notes (per-task numbers in general response, item 4), and we do not yet have a clean explanation. One likely factor is the real-to-sim gap: SimplerEnv evaluates policies trained on real BridgeData images in simulation. The VLM is frozen, but DINOv2 is fine-tuned on the real images, so its difference tokens may generalize worse to the rendered frames than the VLM tokens do. LIBERO, which trains and tests in the same simulator, has no such gap. Our models follow the fine-tuning recipe published with the official checkpoints (general response, item 2).

AuthorsOpenReview ↗

Response to Reviewer TqK7 (2/3)

W2. Current-only ablation and feature subtraction.

We agree that the zero and identity interventions show reliance on the difference tokens but not the value of reference conditioning, and that the trained current-only ablation is the most important missing experiment. It should have been in the original submission. We trained both requested models, changing nothing else:

  • Current-only. The reference-conditioned cross-attention is removed and the encoder output is LN(F_c), followed by the same concat-project pooler with m = 1 token per view, the same cached VLM tokens at T = 64, self-attention injection, and the same recipe, including the random reference offsets.
  • Feature subtraction. The cross-attention is replaced by LN(F_c) - LN(F_r), the per-location subtraction discussed in Section 3.2 (shared LayerNorm before subtracting, positional embedding added after), with everything else identical. This tests whether learned cross-frame matching adds anything over comparing patches at the same location.

The current-only model is trained on all four LIBERO suites and SimplerEnv (general response, item 5; Tables 1 and 9), and the feature-subtraction variant on LIBERO-10.

MethodSpatialObjectGoalLongLIBERO Avg.SimplerEnv
Current-only (T=64)95.699.495.992.895.954.1
Feature subtraction (T=64)---87.6--
Differential (T=64)95.898.996.592.495.953.1

We also trained the current-only model with T = 256 on LIBERO-10, to compare it with the T = 256 model of the reference-interval study (Table 5):

MethodLong
Current-only (T=256)92.6
Differential (T=256)92.8

Differential VLA performs better than current-only on LIBERO-Goal and LIBERO-Spatial and at T = 256, whereas current-only performs better on LIBERO-Object, LIBERO-10, and SimplerEnv, by comparable margins. The difference in compute is negligible: the reference cross-attention adds 0.8 of the 500.5 GFLOPs of a System-1 query (0.2%), so both models have the same 1.8x FLOP reduction over GR00T N1.7.

The differential encoder is a generalization of the current-only encoder: setting the output projection of the reference cross-attention to zero recovers the current-only model (Section 3.2). We conjecture that the reference conditioning becomes more useful in two regimes. First, at longer reference update intervals: on LIBERO-10, the gap changes sign from -0.4 points at T = 64 to +0.2 points at T = 256. The margins are small, but the trend is what we would expect: the older the reference frame, the more the scene has changed since the VLM saw it, so encoding that change matters more. Second, with a smaller action head or lower-dimensional tokens: the difference tokens are a structured, compact summary of what has changed. A current-only encoder leaves it to the action head to compare the current frame with the cached VLM tokens, which is harder when the head or the tokens have less capacity. We are running experiments to test both conjectures and will report the results in the camera-ready version.

Feature subtraction is clearly worse: 87.6% on LIBERO-10, 4.8 points below Differential VLA and 5.2 below current-only, and every one of its five checkpoints (85.2-87.6%) is below the worst checkpoint of Differential VLA (89.9%). Comparing patches at the same location is therefore not a substitute for the learned cross-frame matching. A likely reason is that, once the camera or objects move, the n-th token of the reference frame may not be semantically aligned with the n-th token of the current frame, so their difference does not measure the change at that point.

AuthorsOpenReview ↗

Response to Reviewer TqK7 (3/3)

Q. T sweep and uncertainty.

The revised PDF adds the reference-interval study (Table 5): models trained with T in {8, 16, 32, 64, 128, 256} on LIBERO-10 and evaluated at every T'. At matched intervals success stays between 91.0% and 92.8%, with the highest at T = 256. Shortening the interval at test time keeps every model between 89.2% and 93.0%. Lengthening it beyond the training interval degrades the models trained with short T (the T = 8 model drops to 53.5% at T' = 64), while the T = 256 model stays between 91.5% and 93.0% everywhere (bold: matched intervals).

Train T \ Test T'8163264128256
891.089.281.853.528.59.9
1690.392.191.084.873.043.7
3289.289.791.189.487.561.7
6489.291.989.792.491.080.2
12891.790.990.392.692.291.5
25693.091.791.592.592.792.8

On uncertainty: every LIBERO number is an average over 1000 episodes (10 tasks, 100 episodes each, twice on each of 50 initial states). The revised PDF also reports the mean and range over the five evaluated checkpoints (16k-20k steps) next to the best checkpoint that the protocol selects (Table 17), and states that the reported number is the best of five checkpoints, which slightly inflates the numbers.

BenchmarkReportedMeanMin.Max.
LIBERO-Spatial95.895.594.995.8
LIBERO-Object98.998.598.098.9
LIBERO-Goal96.596.195.796.5
LIBERO-1092.491.189.992.4
LIBERO avg.95.995.3--
SimplerEnv53.151.349.453.1
Reviewer_mDFXOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Differential VLA for efficient dual-system VLA inference on GR00T N1.7: the VLM encodes a reference frame only every T steps, while a lightweight DINOv2-based differential encoder summarizes subsequent scene change into a few difference tokens that, together with the cached VLM tokens, condition the DiT action head. On LIBERO it stays within 1.0 point of the re-evaluated official baseline (93.5%) with 8× fewer VLM calls and ~1.27× lower amortized latency (~1.8× fewer FLOPs); on SimplerEnv the gap is 6.6 points at 16× fewer VLM calls; and on two SO-101 real-robot tasks it matches baseline success with 1.22–1.27× lower latency. The paper further supports the design with a temporal-redundancy analysis and ablations showing that naive reuse of stale VLM tokens fails, while the action head relies on the difference tokens.

Critical strengths
  1. The problem is well motivated by a quantitative temporal-redundancy analysis, which shows high short-horizon similarity and faster decorrelation in the wrist view, clearly justifying why neither full VLM re-encoding nor naive reuse of stale tokens is ideal.
  2. Strong controls isolate the proposed mechanism: reusing cached VLM tokens without difference tokens collapses on LIBERO (0.8%) and SimplerEnv (13.5%), and zero/identity replacements of difference tokens drop success to 0% on the tested LIBERO-10 tasks, showing that the action head relies on the differential encoder.
  3. The empirical evaluation is careful and transparent: LIBERO, SimplerEnv, and paired real-robot trials; separate System-1/System-2 and amortized cost reporting; analytic FLOP breakdowns; and an open acknowledgment that a compiled baseline can outperform the method in wall-clock latency.
Critical weaknesses
  1. Positioning relative to closely related efficient / dual-system VLAs is incomplete. Methods such as HiRT and RoboDual already run System 2 sparsely while System 1 consumes fresher perception, and accelerators such as VLA-Cache and EfficientVLA target the same VLM bottleneck; concurrent feature-delta bridging approaches that periodically skip the VLM on GR00T-class models are especially close. The paper claims a first application of differential coding to VLA perception, but without matched VLM-call-budget comparisons it is hard to isolate what the input-space difference tokens uniquely contribute beyond sparse VLM reuse plus cheap current vision.

  2. The practical efficiency gains are modest and unevenly characterized. Amortized latency improves by about 1.27× while FLOPs improve by about 1.8×; System-2 queries are slower than a full baseline query (38.1 vs 31.5 ms); and Appendix E notes that a TensorRT-compiled baseline can be faster in wall-clock than Differential VLA’s eager amortized latency. This weakens the deployability argument, including the suggested remote System-2 offloading scenario, which is not measured under realistic network conditions.

Questions for authors
  1. Please report a same-recipe GR00T N1.7 fine-tune that runs the VLM every query. How do Tables 1 and 7 change?
  2. For LIBERO-Goal task 8 (“bowl on plate”), can you release per-T success rates and failure modes stratified by \Delta t? Does the T=8 claim of “same failures” hold quantitatively?
  3. Have you tried m>1, uneven token budgets favoring the wrist view, or an adaptive refresh rule when difference-token norms / attention entropy exceed a threshold?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mDFX (1/3)

We thank the reviewer for the thorough summary and for recognizing the controls, and we apologize for the late response, caused by extensive new experiments with limited resources. Please see the general response for the changes in the revised PDF.

W1. Positioning and the matched-VLM-budget comparison.

We agree that the submission did not isolate what the difference tokens add over "sparse System 2 plus cheap current vision". HiRT and RoboDual were developed for earlier VLA designs: HiRT predicts one action per step without a diffusion or flow-matching head, and RoboDual builds on OpenVLA, which decodes action tokens autoregressively. Neither carries over directly to recent architectures such as GR00T or the π₀ family, which predict action chunks with a flow-matching head. What they share with Differential VLA, and what the reviewer asks us to isolate, is sparse System-2 calls plus a cheap encoding of the current frame. What Differential VLA adds is a fast encoder conditioned on the reference frame the VLM saw, which can encode the change relative to the cached semantics rather than the current frame on its own. We test this directly with a matched control: a model with the same cached VLM tokens at the same call budget, the same DINOv2-small encoder, pooler, and recipe, but no reference conditioning. It should have been in the original submission. We have trained this current-only model on all four LIBERO suites and SimplerEnv (general response, item 5):

MethodSpatialObjectGoalLongLIBERO Avg.SimplerEnv
Current-only (T=64)95.699.495.992.895.954.1
Differential (T=64)95.898.996.592.495.953.1

We also trained it with T = 256 on LIBERO-10, to compare it with the T = 256 model of the reference-interval study (Table 5):

MethodLong
Current-only (T=256)92.6
Differential (T=256)92.8

On LIBERO, the current-only model averages 95.9%, the same as Differential VLA. Differential VLA performs better than current-only on LIBERO-Goal and LIBERO-Spatial and at T = 256, whereas current-only performs better on LIBERO-Object, LIBERO-10, and SimplerEnv, by comparable margins. The difference in compute is negligible: the reference cross-attention adds 0.8 of the 500.5 GFLOPs of a System-1 query (0.2%), so both models have the same 1.8x FLOP reduction over GR00T N1.7.

The differential encoder is a generalization of the current-only encoder: setting the output projection of the reference cross-attention to zero recovers the current-only model (Section 3.2). We conjecture that the reference conditioning becomes more useful in two regimes. First, at longer reference update intervals: on LIBERO-10, the gap changes sign from -0.4 points at T = 64 to +0.2 points at T = 256. The margins are small, but the trend is what we would expect: the older the reference frame, the more the scene has changed since the VLM saw it, so encoding that change matters more. Second, with a smaller action head or lower-dimensional tokens: the difference tokens are a structured, compact summary of what has changed. A current-only encoder leaves it to the action head to compare the current frame with the cached VLM tokens, which is harder when the head or the tokens have less capacity. We are running experiments to test both conjectures and will report the results in the camera-ready version.

As discussed in the related work (Section 5 and Appendix A), VLA-Cache and EfficientVLA reduce the cost of each VLM query rather than the number of queries, so they are complementary to our method. We will also discuss the concurrent work Latent Bridge in the paper once its authors are de-anonymized. Like Differential VLA, it skips VLM calls on a GR00T model, but it predicts the change in the VLM feature space: a small DiT updates the cached image-token features from those features, the robot state, and the previous action, without seeing the current image. Differential VLA instead encodes the change in the input space from the current frame and leaves the cached VLM tokens unchanged. The two methods also make different training trade-offs: Latent Bridge keeps the pretrained VLA frozen and trains the bridge on simulator rollouts with DAgger refinement, while Differential VLA fine-tunes the model offline on the demonstrations, without a simulator, which is what allows our real-robot training.

AuthorsOpenReview ↗

Response to Reviewer mDFX (2/3)

W2. Size and characterization of the efficiency gains.

We accept the characterization and will not overstate the result. The revised latencies, in eager PyTorch (Table 3) and compiled with TensorRT (Table 8), are as follows (ms, speedup over the baseline in parentheses, amortized at T = 64, n = 8):

QueryEagerTensorRT
GR00T N1.733.521.4
Ours, System 239.4 (0.85x)26.9 (0.80x)
Ours, System 123.3 (1.44x)16.3 (1.31x)
Ours, amortized25.3 (1.32x)17.6 (1.21x)

Three clarifications.

  • The 1.8x FLOP reduction is architectural, whereas the 1.32x wall-clock gain is measured in eager PyTorch for both models. We attribute the gap to per-query costs that do not scale with FLOPs, such as data preprocessing and kernel launches, which skipping the VLM does not remove. With both models compiled with TensorRT (Appendix E, new Table 8; general response, item 6), the amortized speed-up is 1.21x, so the method stays faster than the baseline when both are compiled, by a slightly smaller margin than in eager mode.
  • System-2 queries are slower than a baseline query (39.4 vs. 33.5 ms) because they run both the VLM and the differential encoder. This is the price of the design, and the saving comes entirely from the System-1 queries (23.3 ms, 1.44x). The amortized gain therefore grows with T. For example, the revised PDF shows that a model trained with T = 256 stays between 91.5% and 93.0% on LIBERO-10 at every test interval (Table 5). At T = 256 one query in 32 is a System-2 query, and the amortized latency computed from Table 3 is 23.8 ms, 1.41x (16.6 ms, 1.29x against the compiled baseline).
  • Remote offloading of System 2 is not measured. The paper only presents it in the conclusion (Section 6) as a deployment setting that would make the trade-off more favorable, not as a result. What the paper does measure is the SO-101 setup, where the whole policy runs on a server over a link that adds 32-35 ms per query to both methods. In this setting, the end-to-end round trip drops from 87.5 to 79.0 ms on block placing and from 87.6 to 78.9 ms on cup stacking.

Q1. Same-recipe GR00T N1.7 fine-tune.

The official checkpoints in Tables 1 and 9 are already a same-recipe baseline: our models are fine-tuned from the same pretrained checkpoint with the recipe published with them (number of steps, effective batch size, optimizer, schedule, augmentation, and state dropout; general response, item 2). The only possible remaining difference is checkpoint selection: we report the best of five checkpoints, and we do not know how the official checkpoints were selected. To rule out any difference, we fine-tuned GR00T N1.7 on LIBERO-10 ourselves, running the VLM on every query and selecting the checkpoint as for our models (general response, item 7):

MethodFreq.LIBERO-10
N1.7, official ckpt., our eval.1/890.0
N1.7, our fine-tune1/891.6
Current-only (T=64)1/6492.8
Ours (T=64)1/6492.4

In the camera-ready version, we will add the fine-tune results for the other three LIBERO suites and SimplerEnv, so that Tables 1 and 9 (Tables 1 and 7 of the submission) also include a baseline with the same checkpoint selection. In the real-robot experiments, both methods are fine-tuned by us with the same recipe and evaluated at their final checkpoint (ours vs. baseline: 16/24 vs. 17/24 on block placing and 8/24 vs. 8/24 on cup stacking).

Q2. LIBERO-Goal task 8.

With the revised self-attention default the task is at 100% (general response, item 1), so the per-T analysis of its failures is no longer needed and the "same failures at T = 8" claim has been removed. The drop was specific to cross-attention injection with the concat-project pooler: every other pooler/injection combination reaches 95.8% to 96.9% on LIBERO-Goal (Table 4). The reviewer's underlying question, how success depends on the reference interval, is answered quantitatively on LIBERO-10 by Table 5: for models trained at T in {8, ..., 256} and evaluated at matched T', success stays between 91.0% and 92.8% (bold: matched intervals).

Train T \ Test T'8163264128256
891.089.281.853.528.59.9
1690.392.191.084.873.043.7
3289.289.791.189.487.561.7
6489.291.989.792.491.080.2
12891.790.990.392.692.291.5
25693.091.791.592.592.792.8
AuthorsOpenReview ↗

Response to Reviewer mDFX (3/3)

Q3. m > 1, uneven token budgets, adaptive refresh.

More tokens per view change success rate on LIBERO-10 by at most 0.6 points (Table 16):

Tokens per viewLIBERO-10
m = 1 (default)92.4
m = 292.1
m = 493.0

We have not tried an uneven budget favoring the wrist view, which decorrelates faster in our redundancy analysis. Note that this is not just a change of m, though: all views currently share one pooler, so an uneven budget requires a separate pooler for each view. We also have not implemented an adaptive refresh rule. The reference-interval study does make it attractive: a model trained with a long interval tolerates every shorter test-time interval, so a test-time rule that refreshes early when the scene changes a lot could be added without retraining. Both are discussed as future work in the limitations (Section 6).

Reviewer_hZvUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Differential VLA. It reuses expensive VLM representation across observations while providing fresh vision features through a small encoder. This method shows a useful tradeoff between accuracy and computation cost. The method is built on GR00T N1.7 and runs the VLM on a reference observation. A DINO-v2-small encoder processes the current image between reference updates. Reference-conditioned cross-attention generates features, which are pooled into one difference token per camera view. The encoder and action head are fine-tuned while the VLM is frozen during training. Experimental results on LIBERO show the improvements from the proposed framework.

Critical strengths
  1. This paper designs a mechanism that reuses context semantic information while maintaining access to current visual information. This addresses the limitations of simply caching stale VLM features. The motivation is clear with the temporal redundancy.

  2. The architecture and the training procedure are practical. Reference-conditioned attention handles object motion. Random reference offsets train the action head to work with stale semantics. This design is reasonable.

  3. The evaluation reports FLOPs, preprocessing latency. The 1.27x speedp uses an optimized eager baseline, making the comparison reasonable.

  4. Experiments on two simulation benchmarks and different real hardware make the result solid.

Critical weaknesses
  1. The central differential-coding contribution is insufficiently isolated. The most important missing baseline is cached VLM tokens plus a lightweight encoder of the current frame alone, matched for training and compute.

  2. The deployment speed advantage remains incomplete. Appendix E reports a TensorRT baseline at 20.9 ms, faster than the current Differential VLA implementation’s 24.9 ms. No compiled Differential VLA result is provided.

  3. LIBERO-Goal drops from 96.2% to 87.7%, including a bowl-placement task falling from 99% to 53%. SimplerEnv loses 6.6 percentage points overall. Would the authors provide some explanation on these degradation?

Questions for authors

Why use the concat-project pooler as the default setting? It seems that the projection contains 202M parameters, while CLS and mean pooling achieve comparable success rates.

4 comments
AuthorsOpenReview ↗

Response to Reviewer hZvU (1/2)

We thank the reviewer for the positive assessment and the precise questions, and we apologize for the late response, caused by extensive new experiments with limited resources. Please see the general response for the changes in the revised PDF.

W1. Cached VLM tokens plus a lightweight encoder of the current frame alone.

We agree that this is an important missing baseline and thank the reviewer for pointing it out. It should have been in the original submission. We have now trained exactly the requested model on all four LIBERO suites and SimplerEnv (general response, item 5): the same DINOv2-small, the same concat-project pooler with m = 1 token per view, the same cached VLM tokens at T = 64, the same self-attention injection, and the same recipe, with the reference-conditioned cross-attention removed so that the encoder sees only the current frame. It is the "Current-only" row in Tables 1 and 9 and in the per-task results (Table 18).

MethodSpatialObjectGoalLongLIBERO Avg.SimplerEnv
Current-only (T=64)95.699.495.992.895.954.1
Differential (T=64)95.898.996.592.495.953.1

We also trained the current-only model with T = 256 on LIBERO-10, to compare it with the T = 256 model of the reference-interval study (Table 5):

MethodLong
Current-only (T=256)92.6
Differential (T=256)92.8

Differential VLA performs better than current-only on LIBERO-Goal and LIBERO-Spatial and at T = 256, whereas current-only performs better on LIBERO-Object, LIBERO-10, and SimplerEnv, by comparable margins. The difference in compute is negligible: the reference cross-attention adds 0.8 of the 500.5 GFLOPs of a System-1 query (0.2%), so both models have the same 1.8x FLOP reduction over GR00T N1.7.

The differential encoder is a generalization of the current-only encoder: setting the output projection of the reference cross-attention to zero recovers the current-only model (Section 3.2). We conjecture that the reference conditioning becomes more useful in two regimes. First, at longer reference update intervals: on LIBERO-10, the gap changes sign from -0.4 points at T = 64 to +0.2 points at T = 256. The margins are small, but the trend is what we would expect: the older the reference frame, the more the scene has changed since the VLM saw it, so encoding that change matters more. Second, with a smaller action head or lower-dimensional tokens: the difference tokens are a structured, compact summary of what has changed. A current-only encoder leaves it to the action head to compare the current frame with the cached VLM tokens, which is harder when the head or the tokens have less capacity. We are running experiments to test both conjectures and will report the results in the camera-ready version.

W2. Compiled Differential VLA.

The reviewer is right that the deployment argument is incomplete without a compiled number. With both models compiled with TensorRT and measured under the same protocol as Table 3 (one RTX 4090, batch size 1, end to end including preprocessing, 3200 queries after 64 warm-up), we obtain the following latencies (ms), with the speedup over the baseline in parentheses. The amortized cost is at T = 64, n = 8.

QueryEagerTensorRT
GR00T N1.733.521.4
Ours, System 239.4 (0.85x)26.9 (0.80x)
Ours, System 123.3 (1.44x)16.3 (1.31x)
Ours, amortized25.3 (1.32x)17.6 (1.21x)

So Differential VLA remains faster when both models are compiled, although the gain is somewhat smaller than in eager mode and, as before, well below the 1.8x reduction in operations. The revised PDF reports this in Appendix E (new Table 8).

AuthorsOpenReview ↗

Response to Reviewer hZvU (2/2)

W3. LIBERO-Goal and SimplerEnv degradation.

The LIBERO-Goal drop is resolved in the revised PDF (general response, item 1): it was specific to cross-attention injection with the concat-project pooler. With the self-attention default, Goal is at 96.5% against 96.6% for the official checkpoint, and the bowl-on-plate task is at 100% (Table 1):

MethodSpatialObjectGoalLongAvg.
N1.7 official ckpt., our eval.95.398.396.690.095.1
N1.7 + cache (T=64)0.20.03.00.00.8
Ours (T=64), submitted (cross-attn.)95.799.487.791.393.5
Ours (T=64), revised (self-attn.)95.898.996.592.495.9

The SimplerEnv gap (53.1% vs. 61.3%, 16x fewer VLM calls) remains, and we do not yet have a clean explanation. Our model follows the fine-tuning recipe published with the official Bridge checkpoint (general response, item 2). One likely factor is the real-to-sim gap: SimplerEnv evaluates policies trained on real BridgeData images in simulation. The VLM is frozen, but DINOv2 is fine-tuned on the real images, so its difference tokens may generalize worse to the rendered frames than the VLM tokens do. LIBERO, which trains and tests in the same simulator, has no such gap. The real-robot results are also consistent with this explanation: on SO-101, where training and testing both take place in the real world and both methods are fine-tuned by us with the same recipe and evaluated at their final checkpoint, Differential VLA matches the baseline (16/24 vs. 17/24 on block placing and 8/24 vs. 8/24 on cup stacking, Table 2).

Q. Why concat-project as the default?

The answer depends on the injection method, which the revised PDF changes, so we first explain that choice.

Self-attention injection.

The revised default prepends the difference tokens to the state and action tokens of the DiT instead of appending them to the cached VLM tokens (Section 3.3). We chose it for an architectural reason: it only assumes that the action head consumes a token sequence, so it carries over to action heads without a cross-attention pathway to the VLM tokens, such as the prefix attention of pi_0.5. In the ablation (Table 4), neither injection method dominates the other.

Cost of concat-project.

The 202M figure is correct for the submitted cross-attention default, which projects to the LLM width 2048. Under self-attention injection the projection maps 257 x 384 = 98,688 features to the DiT width 1536, i.e. 152M parameters. Its computational cost is negligible: it is a single matrix-vector product per view, 0.6 GFLOPs per query against 500.5 GFLOPs for a System-1 query (about 0.1%). The cost is therefore memory, not compute.

Choice of pooler.

We agree that CLS and mean pooling are viable: under self-attention injection, the four poolers are within 0.7 points of each other on LIBERO-Goal, and mean pooling is 0.5 points below concat-project on LIBERO-10. With the injection method fixed, we chose concat-project because it was the best-performing pooler under self-attention injection on both suites (92.4% on LIBERO-10 and 96.5% on LIBERO-Goal, Table 4).

Thanks for the detailed reply. Most of my concerns have been resolved. I will keep the rate.

Thank you for your prompt response and positive feedback. We appreciate your time and effort.

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank the reviewers for their careful reading, and we apologize for the late response, caused by extensive new experiments with limited resources. All three reviews raise the same two points: (i) the drop on LIBERO-Goal, in particular on the bowl-on-plate task, and (ii) the lack of a trained control that uses the current frame without conditioning on the reference frame. The revised paper (PDF uploaded) addresses both: (i) in item 1 below, and (ii) with a trained current-only control on all four LIBERO suites (item 5). Below we summarize the changes in the revised PDF.

1. Default injection method.

The submitted paper used cross-attention injection by default and reported self-attention injection as an ablation (92.2% vs. 91.3% on LIBERO-10). Cross-attention injection performed better in our earlier experiments, before we added augmentation to the DINOv2 input; when we found that self-attention injection had overtaken it, there was not enough time before the deadline to rerun all experiments with it. We have since trained all four LIBERO suites with self-attention injection, which prepends the difference tokens to the DiT token sequence, and made it the default. This raises the LIBERO average from 93.5% to 95.9% and LIBERO-Goal from 87.7% to 96.5% (Table 1), against 95.1% and 96.6% for the official checkpoint under our evaluation.

The LIBERO-Goal drop was specific to the combination of cross-attention injection with the concat-project pooler. In the revised ablation grid (Table 4), the other seven pooler/injection combinations on LIBERO-Goal lie between 95.8% and 96.9%, and only cross-attention + concat-project falls to 83.9% (re-run), with the loss concentrated on three bowl-placing tasks (tasks 1, 4, 8 at 73%, 66%, 48%). With the self-attention default, the bowl-on-plate task (Goal task 8) is at 100%, and the weakest Goal task is task 3 at 87%, against 88% for the baseline.

2. Baselines.

In the submission, the official GR00T N1.7 checkpoints were evaluated with only 5 episodes per task (50 per suite). We re-evaluated them under the protocol we use for our models (100 episodes per task, twice on each of the 50 initial states). This changes the baseline averages from 94.5 to 95.1 on LIBERO. Our models are fine-tuned with the recipe published with these checkpoints (number of steps, effective batch size, optimizer, schedule, augmentation, and state dropout), on both LIBERO and SimplerEnv, so the official checkpoints already serve as a same-recipe baseline. The only possible remaining difference is checkpoint selection (item 7).

3. New ablations.

The revised PDF adds (a) the full pooling x injection grid on both LIBERO-10 and LIBERO-Goal (Table 4); (b) a reference-interval study on LIBERO-10 that trains with T in {8, 16, 32, 64, 128, 256} and evaluates every model at every test-time T' (Table 5); (c) m = 2 and m = 4 difference tokens per view, a frozen DINOv2, initialization from the LIBERO-10 checkpoint instead of the pretrained one, and fine-tuning only the frame encoder from either initialization (Appendix, Table 16); and (d) per-task results for every ablation cell. In (b), success at matched intervals stays between 91.0% and 92.8%, and the model trained with T = 256 stays between 91.5% and 93.0% at every test interval.

4. SimplerEnv.

We reran SimplerEnv (WidowX) with the self-attention default. The average is 53.1% (submitted: 54.7%), 8.2 points below the official checkpoint under our evaluation (61.3%), with 16x fewer VLM calls. The loss is concentrated on the spoon, carrot, and cube-stacking tasks (44.8%, 34.4%, 16.7% against 79.2%, 65.6%, 37.5%); the eggplant-in-sink task, on which every baseline succeeds at most 2% of the time, is at 20.2% (submitted: 45.5%). Table 9 keeps the cross-attention model as a separate row; over the five checkpoints from 16k to 20k steps, the two models overlap (51.8% to 54.7% vs. 49.4% to 53.1%, Table 17).

AuthorsOpenReview ↗

General response to all reviewers (2/2)

5. Current-only control.

We trained a current-only model on all four LIBERO suites and SimplerEnv: the same DINOv2-small, pooler, m = 1, T = 64, cached VLM tokens, and recipe, but with an encoder that sees only the current frame, without reference-frame conditioning. It is a new row in Tables 1 and 9 and in the per-task results (Table 18). On LIBERO, it averages 95.9%, almost the same as Differential VLA (95.5% vs. 95.3% averaged over the five checkpoints instead of the best one, Table 17). On SimplerEnv, it reached 54.1% against 53.1% for ours (52.5% vs. 51.3% averaged over the five checkpoints). We also trained the current-only model with T = 256 on LIBERO-10, next to the T = 256 model of the reference-interval study (Table 5): it reaches 92.6% against 92.8% for ours (91.8% vs. 92.0% averaged over the five checkpoints). Differential VLA performs better than current-only on LIBERO-Goal and LIBERO-Spatial and at T = 256, whereas current-only performs better on LIBERO-Object, LIBERO-10, and SimplerEnv by comparable margins. The difference in compute is negligible: the reference cross-attention adds 0.8 of the 500.5 GFLOPs of a System-1 query (0.2%), so both models have the same 1.8x FLOP reduction over GR00T N1.7.

The differential encoder is a generalization of the current-only encoder: setting the output projection of the reference cross-attention to zero recovers the current-only model (Section 3.2). We conjecture that the reference conditioning becomes more useful in two regimes. First, at longer reference update intervals: on LIBERO-10, the gap changes sign from -0.4 points at T = 64 to +0.2 points at T = 256. The margins are small, but the trend is what we would expect: the older the reference frame, the more the scene has changed since the VLM saw it, so encoding that change matters more. Second, with a smaller action head or lower-dimensional tokens: the difference tokens are a structured, compact summary of what has changed. A current-only encoder leaves it to the action head to compare the current frame with the cached VLM tokens, which is harder when the head or the tokens have less capacity. We are running experiments to test both conjectures and will report the results in the camera-ready version.

6. Latency.

We reran the eager PyTorch measurements (Table 3) with the self-attention model, and the new Appendix E (Table 8) compiles both models with TensorRT. Both use the same protocol (one RTX 4090, batch size 1, end to end including preprocessing, 3200 queries after 64 warm-up). At T = 64, n = 8, the amortized latency is 25.3 ms against 33.5 ms for GR00T N1.7 in eager mode (1.32x; submitted: 1.27x), and 17.6 ms against 21.4 ms with both models compiled (1.21x).

7. Checkpoint selection.

We report the best of the five checkpoints from 16k to 20k steps. We do not know how the official checkpoints were selected. They may well have been chosen the same way. However, to rule out every possible difference, we fine-tuned the base GR00T N1.7 model on LIBERO-10 ourselves and selected its checkpoint in the same way. It reaches 91.6% on LIBERO-10, against 90.0% for the official checkpoint, and 92.4% for Differential VLA, so Differential VLA stays 0.8 points above the baseline with the same recipe and checkpoint selection (0.4 points averaged over the five checkpoints: 91.1% vs. 90.7%), while calling the VLM 8x less often. In the camera-ready version, we will also add the fine-tune results for the other three LIBERO suites and SimplerEnv.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.