- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Differential VLA to exploit temporal redundancy in robot observations. Built on GR00T N1.7, the method evaluates the expensive VLM on a reference observation every T environment steps and caches its output. Between refreshes, a DINOv2-small encoder and reference-conditioned cross-attention produce a small number of additional tokens from the current observation. These tokens condition the action head alongside the cached VLM representation.
Critical strengths
Combining infrequent semantic encoding with fresh visual feedback is a sensible way to avoid both repeated VLM computation and purely stale conditioning.
The experimental results are promising.
Critical weaknesses
LIBERO-Goal decreases from 96.2% to 87.7%, including a decrease from 99% to 53% on bowl-on-plate (Table 14). The explanation that similar failures persist with T=8 is plausible, but quantitative results are needed to distinguish fine-tuning effects from stale conditioning. On SimplerEnv, the average decrease of 6.6 percentage points combines large losses, such as carrot-on-plate falling from 65.6% to 31.3%, with a large gain on eggplant-in-sink from 1.0% to 45.5% (Table 7).
The zero and identity interventions in Section 4.6 demonstrate reliance on fresh visual information, but both remove the actual current observation. They do not establish the benefit of conditioning the fast encoder on the reference frame. The most important missing experiment is a trained current-only ablation using the same DINO backbone, pooling, number of tokens, cached VLM context, refresh interval, and training recipe.
A simple feature-subtraction variant would further test the value of learned cross-frame matching. These are targeted mechanism ablations, not a request to reimplement every related system.
Questions for authors
How does a trained current-only visual branch compare with the reference-conditioned encoder under the same backbone, token budget, refresh schedule, and training recipe? Does simple feature subtraction perform comparably? Can a limited T sweep and uncertainty estimates from the existing evaluation structure establish how stable the reported trade-off is?
3 comments
Response to Reviewer TqK7 (2/3)
W2. Current-only ablation and feature subtraction.
We agree that the zero and identity interventions show reliance on the difference tokens but not the value of reference conditioning, and that the trained current-only ablation is the most important missing experiment. It should have been in the original submission. We trained both requested models, changing nothing else:
- Current-only. The reference-conditioned cross-attention is removed and the encoder output is LN(F_c), followed by the same concat-project pooler with m = 1 token per view, the same cached VLM tokens at T = 64, self-attention injection, and the same recipe, including the random reference offsets.
- Feature subtraction. The cross-attention is replaced by LN(F_c) - LN(F_r), the per-location subtraction discussed in Section 3.2 (shared LayerNorm before subtracting, positional embedding added after), with everything else identical. This tests whether learned cross-frame matching adds anything over comparing patches at the same location.
The current-only model is trained on all four LIBERO suites and SimplerEnv (general response, item 5; Tables 1 and 9), and the feature-subtraction variant on LIBERO-10.
| Method | Spatial | Object | Goal | Long | LIBERO Avg. | SimplerEnv |
|---|---|---|---|---|---|---|
| Current-only (T=64) | 95.6 | 99.4 | 95.9 | 92.8 | 95.9 | 54.1 |
| Feature subtraction (T=64) | - | - | - | 87.6 | - | - |
| Differential (T=64) | 95.8 | 98.9 | 96.5 | 92.4 | 95.9 | 53.1 |
We also trained the current-only model with T = 256 on LIBERO-10, to compare it with the T = 256 model of the reference-interval study (Table 5):
| Method | Long |
|---|---|
| Current-only (T=256) | 92.6 |
| Differential (T=256) | 92.8 |
Differential VLA performs better than current-only on LIBERO-Goal and LIBERO-Spatial and at T = 256, whereas current-only performs better on LIBERO-Object, LIBERO-10, and SimplerEnv, by comparable margins. The difference in compute is negligible: the reference cross-attention adds 0.8 of the 500.5 GFLOPs of a System-1 query (0.2%), so both models have the same 1.8x FLOP reduction over GR00T N1.7.
The differential encoder is a generalization of the current-only encoder: setting the output projection of the reference cross-attention to zero recovers the current-only model (Section 3.2). We conjecture that the reference conditioning becomes more useful in two regimes. First, at longer reference update intervals: on LIBERO-10, the gap changes sign from -0.4 points at T = 64 to +0.2 points at T = 256. The margins are small, but the trend is what we would expect: the older the reference frame, the more the scene has changed since the VLM saw it, so encoding that change matters more. Second, with a smaller action head or lower-dimensional tokens: the difference tokens are a structured, compact summary of what has changed. A current-only encoder leaves it to the action head to compare the current frame with the cached VLM tokens, which is harder when the head or the tokens have less capacity. We are running experiments to test both conjectures and will report the results in the camera-ready version.
Feature subtraction is clearly worse: 87.6% on LIBERO-10, 4.8 points below Differential VLA and 5.2 below current-only, and every one of its five checkpoints (85.2-87.6%) is below the worst checkpoint of Differential VLA (89.9%). Comparing patches at the same location is therefore not a substitute for the learned cross-frame matching. A likely reason is that, once the camera or objects move, the n-th token of the reference frame may not be semantically aligned with the n-th token of the current frame, so their difference does not measure the change at that point.
Response to Reviewer TqK7 (3/3)
Q. T sweep and uncertainty.
The revised PDF adds the reference-interval study (Table 5): models trained with T in {8, 16, 32, 64, 128, 256} on LIBERO-10 and evaluated at every T'. At matched intervals success stays between 91.0% and 92.8%, with the highest at T = 256. Shortening the interval at test time keeps every model between 89.2% and 93.0%. Lengthening it beyond the training interval degrades the models trained with short T (the T = 8 model drops to 53.5% at T' = 64), while the T = 256 model stays between 91.5% and 93.0% everywhere (bold: matched intervals).
| Train T \ Test T' | 8 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| 8 | 91.0 | 89.2 | 81.8 | 53.5 | 28.5 | 9.9 |
| 16 | 90.3 | 92.1 | 91.0 | 84.8 | 73.0 | 43.7 |
| 32 | 89.2 | 89.7 | 91.1 | 89.4 | 87.5 | 61.7 |
| 64 | 89.2 | 91.9 | 89.7 | 92.4 | 91.0 | 80.2 |
| 128 | 91.7 | 90.9 | 90.3 | 92.6 | 92.2 | 91.5 |
| 256 | 93.0 | 91.7 | 91.5 | 92.5 | 92.7 | 92.8 |
On uncertainty: every LIBERO number is an average over 1000 episodes (10 tasks, 100 episodes each, twice on each of 50 initial states). The revised PDF also reports the mean and range over the five evaluated checkpoints (16k-20k steps) next to the best checkpoint that the protocol selects (Table 17), and states that the reported number is the best of five checkpoints, which slightly inflates the numbers.
| Benchmark | Reported | Mean | Min. | Max. |
|---|---|---|---|---|
| LIBERO-Spatial | 95.8 | 95.5 | 94.9 | 95.8 |
| LIBERO-Object | 98.9 | 98.5 | 98.0 | 98.9 |
| LIBERO-Goal | 96.5 | 96.1 | 95.7 | 96.5 |
| LIBERO-10 | 92.4 | 91.1 | 89.9 | 92.4 |
| LIBERO avg. | 95.9 | 95.3 | - | - |
| SimplerEnv | 53.1 | 51.3 | 49.4 | 53.1 |
Response to Reviewer TqK7 (1/3)
We thank the reviewer for pinpointing the mechanism ablations that were missing, and we apologize for the late response, caused by extensive new experiments with limited resources. We agree with the assessment and have run those ablations. Please see the general response for the changes in the revised PDF.
W1. LIBERO-Goal and SimplerEnv degradation.
The LIBERO-Goal result in the submission is superseded. The drop was specific to the submitted default, i.e., cross-attention injection with the concat-project pooler. The revised PDF uses self-attention injection and reports Goal at 96.5% against 96.6% for the official checkpoint, with the bowl-on-plate task (8) at 100% against 98% (per-task results in Table 18):
In the revised ablation grid, the submitted configuration re-trained reaches 83.9% on Goal, with the loss on three bowl-placing tasks (1, 4, 8 at 73%, 66%, 48%), while the other seven pooler/injection combinations lie between 95.8% and 96.9%. Since all eight configurations use the same T = 64 and the same cached VLM tokens, stale conditioning alone cannot explain a drop that only one of them shows. The qualitative "same failures at T = 8" statement has been removed.
On SimplerEnv the gap (53.1% vs. 61.3%) remains and is uneven across tasks, as the reviewer notes (per-task numbers in general response, item 4), and we do not yet have a clean explanation. One likely factor is the real-to-sim gap: SimplerEnv evaluates policies trained on real BridgeData images in simulation. The VLM is frozen, but DINOv2 is fine-tuned on the real images, so its difference tokens may generalize worse to the rendered frames than the VLM tokens do. LIBERO, which trains and tests in the same simulator, has no such gap. Our models follow the fine-tuning recipe published with the official checkpoints (general response, item 2).