- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Z-sight is a three-stream MoT (Qwen3.5-2B VLM + foresight expert + action expert). The foresight expert flow-matches future spatial tokens in the VLM's own video-extended vision encoder space against an EMA target. The action expert attends to these tokens during joint denoising. Training has three stages: ~2k h of egocentric human video, then ~2.75k h of cross-embodiment robot data, then 219 G1 demos. On 5 real-world Unitree G1 tasks (10 trials each), it reports 62% SR vs. 46% for Fast-WAM and 32% for π0.5 / GR00T N1.7, at 110 ms per chunk.
Critical strengths
- Predicting the future in the policy's own perception space is a clean idea: it avoids a pixel decoder and an external target encoder. The causal space-time encoder adds no parameters.
- The anti-collapse design is careful: an EMA target, knowledge insulation so the encoder sees only the FAST loss, and a frozen encoder in post-training.
- The test-time interventions (Noise/Copy) are a useful probe of whether the policy actually reads the predicted stream.
Critical weaknesses
- The novelty is overstated and key prior work is missing. Several earlier systems already use the "VLM + foresight expert + action expert" MoT design with one-way attention from foresight to action: F1-VLA, InternVLA-A1 and UniJEPA. Predicting future latents for VLAs is also established: FLARE, VLA-JEPA and DreamVLA. Only FLARE is cited. GR00T N1.5 already trains with the FLARE future-latent objective, so labelling GR00T as "no future prediction" needs checking for N1.7. What remains new is mainly that the target space is the VLM's own encoder, and that choice is never isolated experimentally. Other deltas (EMA target instead of frozen encoder, flow-matching instead of regression, joint denoising, space-time encoder) are incremental and none is ablated.
- The ablations are too thin to support the causal claims.
- All ablations run on Dog alone, the easiest short task, with 10 trials each. The headline claim is about long-horizon gains (Chair/Book/Fruit), and that claim is never ablated.
- "No foresight branch" drops to 40%, below both VLAs. The authors attribute this to their own frozen-VLM recipe, which means the "standard VLA" control is broken rather than fair. A properly trained no-foresight model is needed, built from mid-training onward with identical data and compute.
- The second contribution, human video giving dense supervision, has no ablation at all: none without human pre-training, none without the 20% human replay.
- The Copy intervention does not show that the policy uses the prediction's content. Copy feeds a deterministic, out-of-distribution trajectory with a fixed ξ, so the 0% may reflect distribution shift rather than semantic reliance. Better tests would be: swapping in a prediction from another episode or timestep, an oracle using ground-truth , and a measure of prediction quality (error vs. a copy-current baseline, retrieval, or a probe). Without these, it is unclear whether the expert predicts dynamics or largely copies the present, given that is zero-initialised.
- The evidence base is narrow. It uses one robot, 5 tasks, and 10 trials per task. There are no simulation benchmarks (LIBERO, RoboTwin 2.0, SimplerEnv) and no tests of generalisation (unseen objects, scenes or instructions). Baseline budgets also differ: Fast-WAM gets 7.5k steps vs. 20k for the others, and GR00T is trained with a frozen backbone. Z-sight's mid-training additionally includes Humanoid Everyday G1 data. These differences confound the comparison.
Questions for authors
-
Why does removing only in post-training collapse Dog from 90% to 30%, when the mid-trained foresight expert should already encode dynamics? What happens if the foresight stream is simply detached from the action expert's attention at test time (vs. Noise)?
-
The recipe uses three stages and ~4.7k h of mixed data. Can you ablate the stages (no human pre-training; no mid-training) and the data sources? Without this, the gains cannot be separated from the training pipeline and data scale, which the baselines do not share.
7 comments
Prediction-design controls. On RoboTwin and SIMPLE, we trained five variants from the same mid-trained initialization, with the same VLM backbone, demonstrations, and adaptation recipe. DINOv2-B target retains the foresight stream and joint denoising but replaces Z-sight's targets with frozen DINOv2-B/14 features through a linear target projection. FLARE-style alignment is a controlled auxiliary-alignment variant within our architecture. It removes the denoising expert and aligns future-query tokens to the same DINOv2-B features with a cosine loss. This auxiliary branch is used only during training. No foresight branch removes the branch, while no view loss retains it without future-prediction supervision during adaptation.
RoboTwin 2.0: 100k steps at global batch 256 on clean demonstrations, evaluated on 50 tasks with 20 episodes per task under clean (C2C) and randomized (C2R) conditions, giving 1,000 episodes per condition. SIMPLE: 50k steps at global batch 256, evaluated on 12 teleoperation tasks at the Hard randomization level with ten episodes per task, giving 120 episodes per entry. In every adaptation variant, the vision tower, language model, state projection, and action expert are trainable, with action gradients flowing into the VLM. Success rates are in percent.
| Design | SIMPLE | RoboTwin C2C | RoboTwin C2R |
|---|---|---|---|
| No view loss | 25.0 | 60.9 | 11.6 |
| No foresight branch | 68.3 | 76.3 | 22.0 |
| FLARE-style alignment | 67.5 | 74.6 | 20.9 |
| DINOv2-B target + joint denoising | 70.8 | 76.7 | 26.6 |
| Z-sight | 73.1 | 82.4 | 40.1 |
With joint denoising retained, Z-sight improves over frozen DINOv2-B targets by 5.7 points in RoboTwin C2C, 13.5 in C2R, and 2.3 on SIMPLE. With the same external DINOv2-B target, inference-time feature prediction outperforms auxiliary alignment by 2.1 C2C and 5.7 C2R points. These controls support the specific target-and-prediction formulation.
Joint versus sequential inference. Sequential inference first completes future-feature denoising and then denoises actions with that estimate fixed. Each pair uses identical learned weights.
| Evaluation | Joint | Sequential |
|---|---|---|
| SIMPLE | 73.1 | 64.2 |
| RoboTwin C2C | 82.4 | 82.6 |
| RoboTwin C2R | 40.1 | 37.9 |
| LIBERO | 97.70 | 97.05 |
| Latency, ms/chunk | 114.8 | 208.0 |
Joint inference improves SIMPLE success by 8.9 points, with smaller differences on LIBERO and RoboTwin. Sequential inference takes ten future-denoising steps followed by ten action-denoising steps, versus ten joint steps. On the same RTX 4090 and implementation, sequential inference takes 1.81× as long.
2. What the foresight stream predicts and how actions use it
Content, uncertainty, and attention. We agree that Copy alone cannot distinguish dependence on future content from sensitivity to its fixed-noise trajectory. We therefore added the Swap and attention-removal tests you suggested, alongside a matched-label noise control. All use the same SIMPLE checkpoint and 120 episodes.
| Intervention | Future-stream label | SIMPLE Hard (%) |
|---|---|---|
| Model's own prediction | running | 73.1 |
| Noise | 61.7 | |
| Noise | running | 16.7 |
| Swap | running | 11.7 |
| Copy | running | 5.8 |
| Detach | running | 7.5 |
Swap supplies the same model's predicted trajectory from a different episode or timestep, preserving the token layout, noise schedule, and label. Its 11.7% success, compared with 73.1% for the observation-matched prediction, supports dependence on observation-specific future content. On RoboTwin, Swap reduces success from 82.4% / 40.1% to 7.5% / 4.0% under C2C / C2R, respectively. Changing only the Gaussian noise label from to the running reduces success from 61.7% to 16.7%, showing that the uncertainty cue also matters.
Detach directly answers your attention question. It blocks action tokens from attending to foresight tokens while the foresight stream continues denoising. Success falls to 7.5%, compared with 61.7% when the attention route remains available but receives noise labeled . Detach removes a route used during training, whereas Noise preserves that route and signals an unreliable input. In the post-training experiments reported here, the action noise level and a separate visual noise level are sampled subject to , exposing the action expert to future streams with varying uncertainty.
Prediction quality against the present-frame baseline. On held-out episodes from all six G1 tasks, we evaluate 11,676 observation-to-future windows with valid future frames. Let be the predicted future, the true future from the EMA target encoder, the present-frame features, and the true future of an unrelated window.
| Metric | Value | |
|---|---|---|
| Predicted future vs true future | 0.955 | |
| Present frame vs true future | 0.921 | |
| Unrelated window vs true future | 0.850 | |
| Predicted-over-present cosine gain | +0.034 | |
| Predicted change vs true change | 0.646 |
Predictions are closer to the true future than the present-frame baseline is, with a positive gain on every task. The change cosine of 0.646 directly measures agreement between predicted and observed feature changes relative to the present. The zero-initialized is a learnable projection for current-frame conditioning. It does not constrain the predicted output to copy the present, and future-feature generation starts from Gaussian noise.
Oracle future and action quality. On held-out episodes from the same six tasks, we fix the initial action noise and vary only the future features supplied to the action expert. We measure normalized MSE of the sampled action chunk against the demonstration.
| Future-feature input | Action MSE | Relative change |
|---|---|---|
| The model's own prediction | 0.0495 | Reference |
| True future , noised to the running level | 0.0436 | −11.9% |
| Clean present frame , label | 0.0635 | +28.3% |
The ground-truth future, noised to the running level, lowers action error on every task, with an overall 11.9% reduction. Supplying the clean present frame raises error by 28.3%. This offline present-frame input uses , distinct from the running-label Copy intervention. Together, the prediction measurements, Swap, and oracle experiment provide evidence that the stream predicts changes beyond temporal persistence and that its content affects generated actions.
To address your question of whether the foresight expert predicts useful changes or largely copies the present, we conducted the Swap, oracle, and attention-removal experiments you suggested, along with held-out prediction measurements. We also broaden the evidence to six G1 tasks and three simulation benchmarks, including randomized evaluation on RoboTwin and SIMPLE. Below, we first clarify the relationship to the predictive policies you identify, then present the design controls and content diagnostics.
1. Unifying VLA and latent world modeling
Z-sight is an end-to-end trainable model that bridges vision-language-action modeling and latent world modeling. Drawing on the joint future-and-action generation paradigm of WAMs, we model future observations in the same visual representation space that grounds the VLM's perception and semantic reasoning. A VLM backbone, foresight expert, and action expert are integrated through a unified MoT and trained together. At inference, future visual features and actions are jointly denoised, making world modeling part of the policy's action-generation process within a shared vision-language foundation.
The works you identify are directly relevant to this positioning. F1-VLA, InternVLA-A1, and UniJEPA already couple predictive and action experts with a VLM. Their prediction formulations differ: F1-VLA generates discrete visual tokens across scales, InternVLA-A1 regresses continuous video latents, and UniJEPA regresses future features against a frozen encoder based on PaliGemma's native SigLIP representation. Z-sight jointly generates future features and actions through flow matching, with an EMA target that follows the policy's own visual encoder.
Future-representation learning also has clear precedents. FLARE aligns intermediate action-DiT representations with EMA future targets, VLA-JEPA learns latent actions through prediction in a frozen V-JEPA 2 representation, and DreamVLA uses predictive dynamics, depth, and semantic supervision. Our contribution is the unified generative formulation above, with explicit future-feature generation coupled to action generation in the VLM's visual space. The experiments below examine the value of this formulation through alternative targets, auxiliary alignment, and sequential conditioning.
We will add the missing references and expand the related-work discussion to cover F1-VLA, InternVLA-A1, UniJEPA, FLARE, VLA-JEPA, and DreamVLA, explicitly comparing their prediction targets and action-generation mechanisms with Z-sight.
You are correct that GR00T N1.5 incorporates FLARE-style predictive supervision. FLARE aligns intermediate action-network representations with future observation embeddings. The released GR00T N1.7 architecture conditions a separate action DiT on VLM outputs. Z-sight integrates vision-language understanding, a generative foresight expert, and an action expert within one MoT. Future spatial tokens are denoised from noise in the VLM's native visual space under their own prediction objective, and the action expert attends to the evolving future stream during joint denoising. This couples an explicit generative model of future observations to control, extending future-representation alignment to joint latent-world-and-action generation. The matched alignment, target-space, and sequential controls below test this formulation under a shared backbone and adaptation recipe.