acceptodds
Withdrawn from ICLR 2027

-sight: Unifying Vision-Language-Action and Latent World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce -sight, a unified robot foundation model that brings vision-language-action modeling and world modeling together through feature-space future prediction. Recent video-based world action models (WAMs) adapt pretrained video generators for robot control, combining visual dynamics modeling with action learning at substantial computational cost. -sight brings future prediction into the native visual representation space of a pretrained vision-language model (VLM), allowing the policy to perceive the present and predict the future within a shared visual foundation. We couple a dedicated foresight expert and a continuous action expert to the VLM through a mixture-of-transformers architecture. The foresight expert denoises future spatial visual tokens, while the action expert attends to the prediction stream to generate actions. This design supports joint learning from robot trajectories and human egocentric video without requiring pixel reconstruction or a pretrained video generator. During inference, -sight efficiently generates future visual features and continuous actions through joint denoising. Evaluated on a real humanoid robot, -sight achieves a 62% average success rate across five manipulation tasks, outperforming VLA and WAM baselines by a large margin. Furthermore, it maintains real-time inference speed of 110ms per chunk without the latency bottleneck of pixel-space generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Foundation Models, Vision-Language-Action Models, World Models, Latent World Modeling, Humanoid Robotics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jP8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Z-sight is a three-stream MoT (Qwen3.5-2B VLM + foresight expert + action expert). The foresight expert flow-matches future spatial tokens in the VLM's own video-extended vision encoder space against an EMA target. The action expert attends to these tokens during joint denoising. Training has three stages: ~2k h of egocentric human video, then ~2.75k h of cross-embodiment robot data, then 219 G1 demos. On 5 real-world Unitree G1 tasks (10 trials each), it reports 62% SR vs. 46% for Fast-WAM and 32% for π0.5 / GR00T N1.7, at 110 ms per chunk.

Critical strengths
  1. Predicting the future in the policy's own perception space is a clean idea: it avoids a pixel decoder and an external target encoder. The causal space-time encoder adds no parameters.
  2. The anti-collapse design is careful: an EMA target, knowledge insulation so the encoder sees only the FAST loss, and a frozen encoder in post-training.
  3. The test-time interventions (Noise/Copy) are a useful probe of whether the policy actually reads the predicted stream.
Critical weaknesses
  1. The novelty is overstated and key prior work is missing. Several earlier systems already use the "VLM + foresight expert + action expert" MoT design with one-way attention from foresight to action: F1-VLA, InternVLA-A1 and UniJEPA. Predicting future latents for VLAs is also established: FLARE, VLA-JEPA and DreamVLA. Only FLARE is cited. GR00T N1.5 already trains with the FLARE future-latent objective, so labelling GR00T as "no future prediction" needs checking for N1.7. What remains new is mainly that the target space is the VLM's own encoder, and that choice is never isolated experimentally. Other deltas (EMA target instead of frozen encoder, flow-matching instead of regression, joint denoising, space-time encoder) are incremental and none is ablated.
  2. The ablations are too thin to support the causal claims.
    • All ablations run on Dog alone, the easiest short task, with 10 trials each. The headline claim is about long-horizon gains (Chair/Book/Fruit), and that claim is never ablated.
    • "No foresight branch" drops to 40%, below both VLAs. The authors attribute this to their own frozen-VLM recipe, which means the "standard VLA" control is broken rather than fair. A properly trained no-foresight model is needed, built from mid-training onward with identical data and compute.
    • The second contribution, human video giving dense supervision, has no ablation at all: none without human pre-training, none without the 20% human replay.
  3. The Copy intervention does not show that the policy uses the prediction's content. Copy feeds a deterministic, out-of-distribution trajectory with a fixed ξ, so the 0% may reflect distribution shift rather than semantic reliance. Better tests would be: swapping in a prediction from another episode or timestep, an oracle using ground-truth zt+z_t^+, and a measure of prediction quality (error vs. a copy-current baseline, retrieval, or a probe). Without these, it is unclear whether the expert predicts dynamics or largely copies the present, given that WcW_c is zero-initialised.
  4. The evidence base is narrow. It uses one robot, 5 tasks, and 10 trials per task. There are no simulation benchmarks (LIBERO, RoboTwin 2.0, SimplerEnv) and no tests of generalisation (unseen objects, scenes or instructions). Baseline budgets also differ: Fast-WAM gets 7.5k steps vs. 20k for the others, and GR00T is trained with a frozen backbone. Z-sight's mid-training additionally includes Humanoid Everyday G1 data. These differences confound the comparison.
Questions for authors
  1. Why does removing Lview\mathcal{L}_{\text{view}} only in post-training collapse Dog from 90% to 30%, when the mid-trained foresight expert should already encode dynamics? What happens if the foresight stream is simply detached from the action expert's attention at test time (vs. Noise)?

  2. The recipe uses three stages and ~4.7k h of mixed data. Can you ablate the stages (no human pre-training; no mid-training) and the data sources? Without this, the gains cannot be separated from the training pipeline and data scale, which the baselines do not share.

7 comments

To address your question of whether the foresight expert predicts useful changes or largely copies the present, we conducted the Swap, oracle, and attention-removal experiments you suggested, along with held-out prediction measurements. We also broaden the evidence to six G1 tasks and three simulation benchmarks, including randomized evaluation on RoboTwin and SIMPLE. Below, we first clarify the relationship to the predictive policies you identify, then present the design controls and content diagnostics.

1. Unifying VLA and latent world modeling

Z-sight is an end-to-end trainable model that bridges vision-language-action modeling and latent world modeling. Drawing on the joint future-and-action generation paradigm of WAMs, we model future observations in the same visual representation space that grounds the VLM's perception and semantic reasoning. A VLM backbone, foresight expert, and action expert are integrated through a unified MoT and trained together. At inference, future visual features and actions are jointly denoised, making world modeling part of the policy's action-generation process within a shared vision-language foundation.

The works you identify are directly relevant to this positioning. F1-VLA, InternVLA-A1, and UniJEPA already couple predictive and action experts with a VLM. Their prediction formulations differ: F1-VLA generates discrete visual tokens across scales, InternVLA-A1 regresses continuous video latents, and UniJEPA regresses future features against a frozen encoder based on PaliGemma's native SigLIP representation. Z-sight jointly generates future features and actions through flow matching, with an EMA target that follows the policy's own visual encoder.

Future-representation learning also has clear precedents. FLARE aligns intermediate action-DiT representations with EMA future targets, VLA-JEPA learns latent actions through prediction in a frozen V-JEPA 2 representation, and DreamVLA uses predictive dynamics, depth, and semantic supervision. Our contribution is the unified generative formulation above, with explicit future-feature generation coupled to action generation in the VLM's visual space. The experiments below examine the value of this formulation through alternative targets, auxiliary alignment, and sequential conditioning.

We will add the missing references and expand the related-work discussion to cover F1-VLA, InternVLA-A1, UniJEPA, FLARE, VLA-JEPA, and DreamVLA, explicitly comparing their prediction targets and action-generation mechanisms with Z-sight.

You are correct that GR00T N1.5 incorporates FLARE-style predictive supervision. FLARE aligns intermediate action-network representations with future observation embeddings. The released GR00T N1.7 architecture conditions a separate action DiT on VLM outputs. Z-sight integrates vision-language understanding, a generative foresight expert, and an action expert within one MoT. Future spatial tokens are denoised from noise in the VLM's native visual space under their own prediction objective, and the action expert attends to the evolving future stream during joint denoising. This couples an explicit generative model of future observations to control, extending future-representation alignment to joint latent-world-and-action generation. The matched alignment, target-space, and sequential controls below test this formulation under a shared backbone and adaptation recipe.

Prediction-design controls. On RoboTwin and SIMPLE, we trained five variants from the same mid-trained initialization, with the same VLM backbone, demonstrations, and adaptation recipe. DINOv2-B target retains the foresight stream and joint denoising but replaces Z-sight's targets with frozen DINOv2-B/14 features through a linear target projection. FLARE-style alignment is a controlled auxiliary-alignment variant within our architecture. It removes the denoising expert and aligns future-query tokens to the same DINOv2-B features with a cosine loss. This auxiliary branch is used only during training. No foresight branch removes the branch, while no view loss retains it without future-prediction supervision during adaptation.

RoboTwin 2.0: 100k steps at global batch 256 on clean demonstrations, evaluated on 50 tasks with 20 episodes per task under clean (C2C) and randomized (C2R) conditions, giving 1,000 episodes per condition. SIMPLE: 50k steps at global batch 256, evaluated on 12 teleoperation tasks at the Hard randomization level with ten episodes per task, giving 120 episodes per entry. In every adaptation variant, the vision tower, language model, state projection, and action expert are trainable, with action gradients flowing into the VLM. Success rates are in percent.

DesignSIMPLERoboTwin C2CRoboTwin C2R
No view loss25.060.911.6
No foresight branch68.376.322.0
FLARE-style alignment67.574.620.9
DINOv2-B target + joint denoising70.876.726.6
Z-sight73.182.440.1

With joint denoising retained, Z-sight improves over frozen DINOv2-B targets by 5.7 points in RoboTwin C2C, 13.5 in C2R, and 2.3 on SIMPLE. With the same external DINOv2-B target, inference-time feature prediction outperforms auxiliary alignment by 2.1 C2C and 5.7 C2R points. These controls support the specific target-and-prediction formulation.

Joint versus sequential inference. Sequential inference first completes future-feature denoising and then denoises actions with that estimate fixed. Each pair uses identical learned weights.

EvaluationJointSequential
SIMPLE73.164.2
RoboTwin C2C82.482.6
RoboTwin C2R40.137.9
LIBERO97.7097.05
Latency, ms/chunk114.8208.0

Joint inference improves SIMPLE success by 8.9 points, with smaller differences on LIBERO and RoboTwin. Sequential inference takes ten future-denoising steps followed by ten action-denoising steps, versus ten joint steps. On the same RTX 4090 and implementation, sequential inference takes 1.81× as long.

2. What the foresight stream predicts and how actions use it

Content, uncertainty, and attention. We agree that Copy alone cannot distinguish dependence on future content from sensitivity to its fixed-noise trajectory. We therefore added the Swap and attention-removal tests you suggested, alongside a matched-label noise control. All use the same SIMPLE checkpoint and 120 episodes.

InterventionFuture-stream labelSIMPLE Hard (%)
Model's own predictionrunning τ\tau73.1
Noiseσ=1\sigma=161.7
Noiserunning τ\tau16.7
Swaprunning τ\tau11.7
Copyrunning τ\tau5.8
Detachrunning τ\tau7.5

Swap supplies the same model's predicted trajectory from a different episode or timestep, preserving the token layout, noise schedule, and label. Its 11.7% success, compared with 73.1% for the observation-matched prediction, supports dependence on observation-specific future content. On RoboTwin, Swap reduces success from 82.4% / 40.1% to 7.5% / 4.0% under C2C / C2R, respectively. Changing only the Gaussian noise label from σ=1\sigma=1 to the running τ\tau reduces success from 61.7% to 16.7%, showing that the uncertainty cue also matters.

Detach directly answers your attention question. It blocks action tokens from attending to foresight tokens while the foresight stream continues denoising. Success falls to 7.5%, compared with 61.7% when the attention route remains available but receives noise labeled σ=1\sigma=1. Detach removes a route used during training, whereas Noise preserves that route and signals an unreliable input. In the post-training experiments reported here, the action noise level τ\tau and a separate visual noise level σ\sigma are sampled subject to σ≥τ\sigma \geq \tau, exposing the action expert to future streams with varying uncertainty.

Prediction quality against the present-frame baseline. On held-out episodes from all six G1 tasks, we evaluate 11,676 observation-to-future windows with valid future frames. Let α\alpha be the predicted future, yy the true future from the EMA target encoder, β\beta the present-frame features, and γ\gamma the true future of an unrelated window.

MetricValue
Predicted future α\alpha vs true future yycos⁡(α,y)\cos(\alpha, y)0.955
Present frame β\beta vs true future yycos⁡(β,y)\cos(\beta, y)0.921
Unrelated window γ\gamma vs true future yycos⁡(γ,y)\cos(\gamma, y)0.850
Predicted-over-present cosine gaincos⁡(α,y)−cos⁡(β,y)\cos(\alpha, y) - \cos(\beta, y)+0.034
Predicted change vs true changecos⁡(α−β, y−β)\cos(\alpha - \beta,\, y - \beta)0.646

Predictions are closer to the true future than the present-frame baseline is, with a positive gain on every task. The change cosine of 0.646 directly measures agreement between predicted and observed feature changes relative to the present. The zero-initialized WcW_c is a learnable projection for current-frame conditioning. It does not constrain the predicted output to copy the present, and future-feature generation starts from Gaussian noise.

Oracle future and action quality. On held-out episodes from the same six tasks, we fix the initial action noise and vary only the future features supplied to the action expert. We measure normalized MSE of the sampled action chunk against the demonstration.

Future-feature inputAction MSERelative change
The model's own prediction α\alpha0.0495Reference
True future yy, noised to the running level τ\tau0.0436−11.9%
Clean present frame β\beta, label σ=0\sigma=00.0635+28.3%

The ground-truth future, noised to the running level, lowers action error on every task, with an overall 11.9% reduction. Supplying the clean present frame raises error by 28.3%. This offline present-frame input uses σ=0\sigma=0, distinct from the running-label Copy intervention. Together, the prediction measurements, Swap, and oracle experiment provide evidence that the stream predicts changes beyond temporal persistence and that its content affects generated actions.

Reviewer_pr7COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Z-sight integrates a pretrained VLM, a future-feature denoising expert, and a continuous action expert through a mixture-of-transformers architecture. Future targets use the VLM's native visual representation with an EMA teacher during joint training. The pipeline combines human and robot data and is evaluated on five real humanoid manipulation tasks, with foresight interventions and post-training ablations on the Dog task.

Critical strengths
  1. The full system demonstrates a substantial practical result on a real humanoid: 62% average success compared with 46% for Fast-WAM and 32% for the two VLA baselines, with model-compute latency comparable to the evaluated VLAs. The integration of dense feature prediction and continuous control without video decoding is technically useful, even if its broad motivation has precedents.
  2. The paper does not rely solely on benchmark comparisons. Test-time modifications establish sensitivity to the foresight stream, and removing the view loss while preserving the architecture reduces Dog success from 90% to 30%. The latter is a meaningful control for continued predictive supervision within the inherited training regime. The authors also disclose the adaptation disadvantage of the no-foresight variant.
Critical weaknesses
  1. The native-space novelty is insufficiently differentiated from closely related, already-cited work. FLARE [1] uses future-feature alignment, an EMA target tied to the policy encoder, and human-video learning. More directly, UniJEPA [2] combines a VLM, a future-feature expert, and an action expert through MoT, and explicitly discusses the advantage of SigLIP as PaliGemma's native visual encoder. These are not identical systems: Z-sight's dense spatial targets, shared/EMA design, joint denoising, and humanoid integration remain possible contributions. However, predicting in the VLM's visual space is not sufficient by itself to distinguish the method. No matched native-versus-external-target or comparable prediction-branch control establishes the value of the specific representational choice.
  2. The mechanism evidence has a narrower scope than the main explanation. Noise intervention marks the stream as unavailable with sigma=1, whereas copy-current intervention follows the running noise label and asserts a static future; content and uncertainty cues therefore differ. Their 60% versus 0% success demonstrates different responses to these interventions, not uniquely accurate dynamics or native-space superiority. Both post-training variants inherit a foresight-trained checkpoint with a largely frozen VLM. The no-view-loss control is informative within this regime, but does not establish the same advantage under matched training from a common pre-foresight initialization. All these studies use ten trials on Dog, although the main benefit is emphasized on longer tasks.

[1] Zheng, Ruijie, et al. "Flare: Robot learning with implicit world modeling." arXiv preprint arXiv:2505.15659 (2025).

[2] Zhang, Jianke, et al. "UniJEPA: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning." arXiv preprint arXiv:2510.10642 (2025).

Questions for authors
  1. Which design specifically separates Z-sight from UniJEPA and FLARE, beyond the target being a pretrained visual feature? A focused native-versus-external-target or alternative-branch comparison with the same data and backbone would directly test the stated motivation.
  2. Does the prediction benefit persist when compared from a common initialization before foresight-specific training, with an adaptation recipe suitable for both models? Please distinguish this question from the already-useful no-view-loss post-training result.
  3. Is any intervention evidence available with matched noise labels, or on one of the longer tasks? Such evidence would clarify whether the observed sensitivity supports the claimed long-horizon mechanism rather than only a local dependence on the stream.
5 comments

We appreciate your distinction between the practical value of the system and the evidence needed to establish its specific contribution. Our response focuses on the differences from FLARE and UniJEPA, the role of initialization in the supervision controls, and interventions that match the noise label while changing future content. We address these with controlled comparisons and extend the evidence beyond Dog.

1. Unifying VLA and latent world modeling

Z-sight is an end-to-end trainable model that bridges vision-language-action modeling and latent world modeling. Drawing on the joint future-and-action generation paradigm of WAMs, we model future observations in the same visual representation space that grounds the VLM's perception and semantic reasoning. A VLM backbone, foresight expert, and action expert are integrated through a unified MoT and trained together. At inference, future visual features and actions are jointly denoised, making world modeling part of the policy's action-generation process within a shared vision-language foundation.

FLARE organizes future supervision around an action DiT conditioned on vision-language embeddings. Its alignment objective, trained against EMA targets and also applicable to human video, shapes intermediate DiT representations to anticipate future observations. Z-sight brings a dedicated generative foresight expert into the VLM-based architecture, coupling visual understanding, future generation, and action generation within the same model. The future is represented in the visual tokens used by the VLM, and its generation proceeds jointly with action generation.

UniJEPA also integrates a VLM, a prediction expert, and an action expert through MoT. It learns future features by direct regression against a frozen target encoder based on PaliGemma's native SigLIP representation, and uses the predictive representations to condition action denoising. Z-sight draws on WAMs to jointly generate future features and actions through flow matching, with an EMA target that follows the policy's own visual encoder. This unifies the VLA's semantic foundation with generative world modeling in an end-to-end trainable policy. The controls below examine this formulation through alternative prediction targets, an alignment branch, and sequential future-to-action conditioning.

We will expand the paper's discussion of FLARE and UniJEPA, with explicit citations and a direct comparison of their prediction targets, training objectives, and inference procedures with Z-sight.

On RoboTwin and SIMPLE, we trained five models from the same mid-trained initialization, with the same VLM backbone, demonstrations, and adaptation recipe. DINOv2-B target retains the foresight stream and joint denoising but replaces Z-sight's targets with frozen DINOv2-B/14 future-frame features through a linear target projection. FLARE-style alignment is a controlled auxiliary-alignment variant within our architecture: it removes the denoising expert and aligns future-query tokens to the same frozen DINOv2-B features with a cosine loss. This auxiliary branch is used only during training. No foresight branch is the action-only variant, while no view loss keeps the branch but removes future-prediction supervision during adaptation.

RoboTwin 2.0: 100k steps at global batch 256 on clean demonstrations, evaluated on all 50 tasks with 20 episodes per task under clean (C2C) and randomized (C2R) conditions, giving 1,000 episodes per condition. SIMPLE: 50k steps at global batch 256, evaluated on the 12 teleoperation tasks at the Hard randomization level with ten episodes per task, giving 120 episodes per entry. In every adaptation variant, including the no-branch model, the vision tower, language model, state projection, and action expert are trainable, with action gradients flowing into the VLM. Success rates are in percent.

DesignSIMPLERoboTwin C2CRoboTwin C2R
No view loss25.060.911.6
No foresight branch68.376.322.0
FLARE-style alignment67.574.620.9
DINOv2-B target + joint denoising70.876.726.6
Z-sight73.182.440.1

With joint denoising retained, Z-sight's target formulation improves over frozen DINOv2-B targets by 5.7 points in RoboTwin C2C, 13.5 in C2R, and 2.3 on SIMPLE. With the same external DINOv2-B target, inference-time feature prediction outperforms auxiliary alignment by 2.1 C2C and 5.7 C2R points. These results support the specific target-and-prediction formulation: the tested forms of dense future supervision do not yield interchangeable results.

We also tested whether actions need to read the evolving prediction or can instead condition on one final estimate. Sequential inference first completes future-feature denoising and then denoises actions with that estimate fixed. Each joint/sequential pair uses identical learned weights.

EvaluationJointSequential
SIMPLE73.164.2
RoboTwin C2C82.482.6
RoboTwin C2R40.137.9
LIBERO97.7097.05
Latency, ms/chunk114.8208.0

Joint denoising improves SIMPLE success by 8.9 points. Differences are smaller on LIBERO and RoboTwin, including a 0.2-point C2C advantage for sequential inference. Sequential inference requires ten future-denoising steps followed by ten action-denoising steps, versus ten joint steps. On the same RTX 4090 and implementation, it takes 1.81× as long as joint inference.

2. Continued supervision and the initialization question

The new adaptation controls broaden the supervision comparison to three simulation settings, with both vision and language backbones trainable. LIBERO uses one policy jointly trained on all four suites for 80k steps at global batch 128, evaluated for 50 episodes per task, 500 per suite and 2,000 per variant. RoboTwin and SIMPLE follow the settings above.

Matched adaptationFull modelNo view lossNo foresight branch
LIBERO97.7074.6595.35
RoboTwin C2R40.111.622.0
SIMPLE Hard73.125.068.3

Within each benchmark, the variants share initialization, demonstrations, steps, batch size, and optimization schedule. Removing the view loss preserves architecture and parameter count, and the foresight branch continues to receive action-loss gradients. The degradation across all three settings supports continued predictive supervision beyond a parameter-count explanation. The no-branch model can adapt its vision and language backbones, addressing the frozen-backbone restriction you identified.

The controls above start from a shared checkpoint after foresight mid-training. A comparison from a common pre-foresight initialization would require repeating the complete 300,000-step mid-training stage and downstream evaluation for each branch. Given the time limit and available compute resources, we prioritized matched adaptation controls across three benchmarks with both vision and language backbones trainable. These experiments consistently support the benefit of continued future-prediction supervision under matched adaptation conditions.

3. Separating prediction content from the noise label

We agree that the original Noise and Copy interventions changed both content and uncertainty cues. Noise supplied Gaussian noise with the label σ=1\sigma=1, declaring the stream unreliable. Copy supplied a static-future trajectory with the running label τ\tau. We now vary content and label separately while keeping the learned weights fixed.

Stream contentLabelSIMPLE hard (%)RoboTwin C2C / C2R (%)
Model's own predictionrunning τ73.182.4 / 40.1
Gaussian noiseσ = 1 (declared unreliable)61.779.0 / 33.9
Gaussian noiserunning τ16.750.5 / 15.3
Model's own prediction for a different episode and timestep (Swap)running τ11.77.5 / 4.0
Present frame (Copy)running τ5.87.8 / 2.0

On SIMPLE, changing only the noise label from σ=1\sigma=1 to the running τ\tau reduces success from 61.7% to 16.7%, showing that the uncertainty cue affects how the policy uses the stream. Under the same running label, the model's own prediction achieves 73.1%, compared with 16.7% for noise, 11.7% for Swap, and 5.8% for Copy. Swap uses predictions from the same model with the same token layout, noise schedule, and label, but from a different episode or timestep. Its large drop supports dependence on observation-specific future content. It preserves the stream's format and schedule while disrupting consistency with the current observation. On RoboTwin, Swap reduces success from 82.4% / 40.1% to 7.5% / 4.0% under C2C / C2R, respectively.

In the post-training experiments reported here, we sample the action noise level τ\tau and a separate visual noise level σ\sigma subject to σ≥τ\sigma \geq \tau, exposing the action expert to future streams with varying uncertainty. The retained performance under Noise (σ=1\sigma=1) is consistent with that exposure. The matched-label results establish sensitivity to both content and uncertainty. Prediction accuracy is measured separately below, and the target-design comparison is given in the first section.

4. Evidence beyond the short Dog task

Interventions across three benchmarks. We applied the same running-label noise intervention to SIMPLE Hard, RoboTwin 2.0, and LIBERO. Within each benchmark, all variants use the same checkpoint and evaluation protocol. The learned prediction and Gaussian-noise replacement both use the running label τ\tau. SIMPLE covers 12 humanoid tasks with 120 episodes per entry, and RoboTwin covers 50 bimanual tasks with 1,000 episodes per condition. LIBERO uses 2,000 episodes in total, including 500 on the Long suite. Success rates are in percent.

VariantSIMPLE HardRoboTwin C2CRoboTwin C2RLIBERO Avg.LIBERO Long
Joint denoising73.182.440.197.7096.6
Sequential64.282.637.997.0594.8
Noise (σ = τ)16.750.515.375.1550.8

With the noise label held fixed, replacing the predicted content reduces success by 56.4 points on SIMPLE Hard, 31.9 points on RoboTwin C2C, and 24.8 points on C2R. On LIBERO-Long, success falls from 96.6% to 50.8%, a 45.8-point drop. These results extend the intervention evidence across humanoid and bimanual manipulation, alongside an explicit long-horizon suite. The physical foresight ablations remain on Dog.

Prediction quality across the G1 tasks. We evaluate prediction quality on held-out episodes from all six G1 tasks, including Book, Chair, Fruit, and Trash.

Over 11,676 observation-to-future windows with valid future frames, we compare the predicted future features α\alpha with the true future features yy from the EMA target encoder. The present-frame features β\beta provide a temporal-persistence baseline, and the true future of an unrelated window γ\gamma provides a second reference. All metrics are pooled over these valid windows.

MetricValue
Predicted future α\alpha vs true future yycos⁡(α,y)\cos(\alpha, y)0.955
Present frame β\beta vs true future yycos⁡(β,y)\cos(\beta, y)0.921
Unrelated window γ\gamma vs true future yycos⁡(γ,y)\cos(\gamma, y)0.850
Predicted-over-present cosine gaincos⁡(α,y)−cos⁡(β,y)\cos(\alpha, y) - \cos(\beta, y)+0.034
Predicted change vs true changecos⁡(α−β, y−β)\cos(\alpha - \beta,\, y - \beta)0.646

The prediction is closer to the true future than the present frame is, with a positive gain on every task. The change cosine compares predicted and observed changes relative to the present frame, providing evidence of predicted feature change beyond temporal persistence.

Relevance to sampled actions. On held-out episodes from the same six tasks, we fix the initial action noise and change only the future features supplied to the action expert. The metric is normalized MSE of the sampled action chunk against the demonstration.

Future-feature inputAction MSERelative change
The model's own prediction α\alpha0.0495Reference
True future yy, noised to the running level τ\tau0.0436−11.9%
Clean present frame β\beta, label σ=0\sigma=00.0635+28.3%

Supplying the true future at the appropriate noise level lowers action error on every task, with an overall reduction of 11.9%. Supplying the clean present frame raises it. This offline input uses σ=0\sigma=0, distinct from the running-label Copy intervention above. These measurements provide direct evidence of future-feature prediction and its relevance to actions, complementing the closed-loop interventions.

Reviewer_gBK7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Z-sight couples a VLA policy with a latent world model in one mixture-of-transformers. A pretrained Qwen3-5.2B VLM encodes the observation history, state, and instruction; a foresight expert denoises future visual tokens in the VLM's own encoder space, supervised against an EMA copy of that encoder; and an action expert predicts a 30-step chunk by conditional flow matching while attending to the evolving foresight stream. At inference both expert streams are denoised jointly. Training is three-stage: FAST-token pretraining on about 2.0k hours of egocentric human video, mid-training on about 2.75k hours of robot data, and post-training on 219 G1 demonstrations.

On a Unitree G1 with SONIC whole-body control, over five real-world tasks and 10 trials per task, Z-sight reaches 62% average success against 46% for Fast-WAM and 32% for both pi0.5 and GR00T N1.7, at a median 109.5 ms per chunk on an RTX 4090. Post-training ablations reduce Dog success from 90% to 30% without the future-feature loss and to 40% without the foresight branch; at test time, Noise gives 60% and Copy 0%.

Critical strengths
  1. Credible system-level evidence on a real humanoid The evaluation is on a real G1 doing locomotion, grasping, and multi-stage manipulation, with shared demonstrations, interfaces, and deployment stack across methods, as detailed in Appendix F. Z-sight matches or exceeds every baseline on every task, and Table E.2 shows it is the only method to complete the last two stages of Fruit. It keeps inference-time future prediction while running faster than pi0.5, which takes 120.5 ms, and Fast-WAM, which takes 155.9 ms; GR00T is faster still at 60 ms but does not predict. These results characterize Z-sight with SONIC, which supplies balance and low-level whole-body coordination.
  2. An architecture-preserving ablation that rules out a capacity explanation Removing only the future-feature loss at post-training, with architecture and parameter count unchanged, drops Dog success from 90% to 30%, with p = 0.02. That weakens the most obvious alternative account, that the extra branch helps by adding capacity, and the Noise and Copy interventions point the same way.
Critical weaknesses
  1. The two distinctive design choices are never isolated The paper's claim is not merely that pixel generation is avoided: FLARE aligns policy features with future-observation embeddings, and Zhang et al.'s 2025 DreamVLA supervises with future dynamic-region, depth, and semantic targets while dropping its decoders at inference. What is specific here is that the target is the token space the policy consumes, and that the action expert reads it during joint denoising. Neither is compared against a plausible alternative: FLARE-style alignment, prediction in a separate space, or predict-then-condition without joint denoising. Without such a control, the gains are equally consistent with 'any dense future supervision helps'. Section 4.4 also tests retention rather than introduction. Every variant starts from a checkpoint that already had future-prediction mid-training, and No foresight branch removes a trained path under a recipe the paper says was designed around it, with the VLM and encoder frozen except for the state projection. A mid-training-level control, or a no-foresight variant with architecture-appropriate tuning, would settle this; one long task would be enough. Noise and Copy establish dependence, but they also move the stream off its training distribution, so they do not show that the prediction is accurate or task-relevant. No held-out foresight metric is reported, such as cosine similarity to the EMA target or a copy-current-features baseline, and nothing tests whether predicted changes track object displacement rather than head-camera ego-motion; Copy at 0% is equally consistent with brittleness to any large perturbation. The human-video claim is untested for the same reason: Table A.1 shows that pretraining uses FAST tokens only, and the view loss enters at mid-training, where human data is 20% replay.

  2. The long-horizon claim rests on 10 trials per task and ablations only on the shortest task Every ablation is on Dog, whose demonstrations average 16 s, while the mechanism is claimed for tasks of 39, 51, and 85 s, where the baseline gap is largest. On Fruit, Z-sight succeeds in 1/10 trials against 0/10, with progress 17.5% against 15.0%; Table E.2 shows it alone completes the last two stages, which is suggestive but n = 1. The text's 'only model to achieve full success' reads more strongly than one trial supports. With 10 trials per task and no seed replication, one trial moves a task's rate by 10 points. The paper's two-sided Fisher test on 31/50 against 23/50 gives p = 0.16, so the margin over Fast-WAM is not significant, although the margins over the two VLAs are significant, with p = 0.005. The Dog comparison behind the main ablation is not significant on its own either: 9/10 against 6/10 gives p = 0.30, even though the 90%-vs-30% ablation difference is significant. The 455 G1 demonstrations comprise 219 for the five evaluated tasks and 236 for five tasks collected but not evaluated, with a 10% holdout. The paper does not give task-selection criteria, whether objects, layouts, and scenes are disjoint across train, validation, and test, or checkpoint-selection rules. Appendix F also shows that Fast-WAM was trained for 7,500 steps at batch 192 against 20,000 steps at batch 128 for the others, about half the samples; unless that budget was validation-selected, the strongest baseline is under-trained. That appendix otherwise matches data and interfaces more carefully than is common, which is why this needs a direct answer. Minor: the per-stage losses and trainable modules must be assembled from Sections 3.1–3.2 and Table A.1; this affects my Presentation score only. These are missing controls rather than errors, and most are addressable, which is why I record only minor soundness concerns; but they bear on the central claim, which is the basis for my rating.

Questions for authors

Clarifications of existing experiments and implementation What were the criteria for the five evaluated tasks, the per-task splits, and the separation of objects, layouts, and scenes? How are checkpoints selected, and how was Fast-WAM's 7,500-step budget chosen against 20,000 elsewhere? What was tuned for the Table 2 variants beyond the full model's settings, and did the no-foresight variant receive architecture-appropriate tuning? Please give trial-level outcomes, and explain what supports the Noise-robustness claim given that noise is sampled only during mid-training. How many of the 30 predicted steps are executed before replanning, and are inference and execution asynchronous? What is the closed-loop latency including acquisition, communication, and control? The 109.5 ms measures model compute, not the control rate.

Evidence that could materially change the assessment Can the two choices be separated? One control would replace native-space prediction with FLARE-style alignment or prediction in a separate space; the other would condition the action expert on a single final estimate instead of joint denoising. Either control on one or two tasks, reported with latency, would change how I read the contribution. What is the held-out foresight accuracy, such as cosine similarity to the EMA target, compared with copying current features, and does it correlate with stage-level success? Does the view loss help on Book or Chair? What happens if it is disabled only for human samples? These questions concern existing claims; additional platforms and action-conditioned planning are future work, not prerequisites here.

5 comments
AuthorsOpenReview ↗

Response to Q1

Thank you for specifying the comparisons that would materially change your assessment. We have carried out the alternative-target and predict-then-condition controls you proposed, together with held-out prediction measurements. We present these first, then report the expanded G1 and simulation evaluation and answer your questions on training fairness, task selection, and deployment.

1. Separating the future-target design from the prediction mechanism

On RoboTwin and SIMPLE we trained five models from the same mid-trained initialization under an identical adaptation recipe, changing the foresight design. Z-sight is our formulation: a dense target in the policy’s own visual-encoder space, updated by EMA, denoised jointly with the actions. DINOv2-B target keeps the foresight stream and joint denoising but replaces the native target with frozen DINOv2-B/14 features of the future frames through a linear target projection. FLARE-style alignment removes the denoising expert and aligns future-query tokens on the policy to the same frozen DINOv2-B features with a cosine loss. This auxiliary branch is used only during training. No foresight branch is the action-only variant, and no view loss keeps the branch and its tokens but removes future-prediction supervision during adaptation. The latter still receives action-loss gradients.

Success rates are in percent. LIBERO: we jointly train one policy on the demonstrations from all four suites for 80k steps at global batch 128. We evaluate 50 episodes per task, giving 500 per suite and 2,000 per variant. RoboTwin 2.0: 100k steps at global batch 256 on the clean demonstrations, evaluated on all 50 tasks with 20 episodes per task under the clean (C2C) and randomized (C2R) conditions. SIMPLE: 50k steps at global batch 256, evaluated on the 12 teleoperation tasks at the Hard randomization level with 10 episodes per task, 120 episodes per entry.

In every adaptation variant, including the no-branch model, the policy's vision tower, language model, state projection, and action expert are trainable, with action gradients flowing into the VLM.

The results are shown below:

DesignSIMPLERoboTwin C2CRoboTwin C2R
No view loss25.060.911.6
No foresight branch68.376.322.0
FLARE-style alignment67.574.620.9
DINOv2-B target + joint denoising70.876.726.6
Z-sight73.182.440.1

On RoboTwin, replacing ZZ-sight's EMA target formulation with frozen DINOv2-B targets while keeping joint denoising (DINOv2-B target) costs 5.7 points in C2C and 13.5 in C2R, and the FLARE-style variant lands where the model without a foresight branch lands (20.9 vs 22.0 C2R). Relative to the FLARE-style variant, keeping joint denoising on the same external target (DINOv2-B target) recovers 2.1 C2C and 5.7 C2R points. The ordering on C2R is therefore ZZ-sight 40.1 > external target (DINOv2-B) with joint denoising 26.6 > no branch 22.0 ≈ FLARE-style 20.9, with the no-view-loss variant (11.6) far below all of them. ZZ-sight also achieves the highest success on RoboTwin C2C and SIMPLE Hard.

We agree that avoiding pixel decoding does not by itself distinguish Z-sight from FLARE, DreamVLA, and related work. These controls show that the tested forms of dense future supervision are not interchangeable. With frozen DINOv2-B targets, inference-time feature prediction outperforms auxiliary alignment. Retaining joint denoising and adopting Z-sight's target formulation gives a further gain. The results support the specific formulation of a dense target in the policy’s own encoder space, updated by EMA, read by the action expert during joint denoising.

We will explicitly cite and discuss FLARE and DreamVLA in the revised paper, clarifying how Z-sight differs in its prediction targets and the use of future features during action generation.

2. Conditioning actions on a single final prediction

We implemented the proposed sequential intervention: first complete future-feature denoising, then denoise actions with that final estimate fixed. Each joint/sequential pair uses identical learned weights.

EvaluationJointSequential
SIMPLE73.164.2
RoboTwin C2C82.482.6
RoboTwin C2R40.137.9
LIBERO97.7097.05
Latency, ms/chunk114.8208.0

The evolving stream improves success most clearly on SIMPLE, by 8.9 points. Differences are smaller on LIBERO and RoboTwin, including a 0.2-point C2C advantage for sequential inference.

Beyond performance differences, sequential inference requires ten future-denoising steps followed by ten action-denoising steps, versus ten joint steps. On the same RTX 4090 and implementation, sequential inference takes 1.81× as long as joint inference, which further justifies our design choice of joint denoising.

AuthorsOpenReview ↗

Response to Q1

3. Prediction accuracy, content, and relevance to actions

Does the foresight expert predict the future, and does the policy use what it predicts? We now measure these two questions separately.

Prediction accuracy (offline). On held-out episodes from all six physical G1 tasks, we compare the expert’s predicted future features α\alpha with the true future features yy from the EMA target encoder over 11,676 observation-to-future windows with valid future frames. Two reference points make the numbers interpretable: the features of the present frame β\beta (the prediction of a model that assumes nothing changes) and the true future of an unrelated window γ\gamma. All metrics are pooled over these valid windows.

MetricValue
Predicted future α\alpha vs true future yycos⁡(α,y)\cos(\alpha, y)0.955
Present frame β\beta vs true future yycos⁡(β,y)\cos(\beta, y)0.921
Unrelated window γ\gamma vs true future yycos⁡(γ,y)\cos(\gamma, y)0.850
Predicted-over-present cosine gaincos⁡(α,y)−cos⁡(β,y)\cos(\alpha, y) - \cos(\beta, y)+0.034
Predicted change vs true changecos⁡(α−β, y−β)\cos(\alpha - \beta,\, y - \beta)0.646

The prediction is closer to the true future than the present frame is, and both exceed the unrelated-future reference. The gain cos⁡(α,y)−cos⁡(β,y)\cos(\alpha, y) - \cos(\beta, y) is positive on every one of the six tasks individually. The last row measures direction: it compares the predicted change α−β\alpha - \beta with the true change y−βy - \beta, both relative to the present frame. Its positive value provides evidence of predicted feature change beyond temporal persistence.

Does the policy use the predicted content (closed loop)? During joint denoising the policy receives, together with the future stream, a scalar label stating how much noise the stream currently carries. The Noise intervention in the submission replaced the stream with Gaussian noise and set this label to σ = 1 ("this stream is pure noise"), so it changed the content and the policy's trust in it at the same time. We now vary the two separately on SIMPLE and RoboTwin.

Stream contentLabelSIMPLE hard (%)RoboTwin C2C / C2R (%)
Model's own predictionrunning τ73.182.4 / 40.1
Gaussian noiseσ = 1 (declared unreliable)61.779.0 / 33.9
Gaussian noiserunning τ16.750.5 / 15.3
Model's own prediction for a different episode and timestep (Swap)running τ11.77.5 / 4.0
Present frame (Copy)running τ5.87.8 / 2.0

As shown in the table above: (1) Under the "unreliable" label the policy tolerates Gaussian noise much better than under the running label. This shows that both the content and its declared uncertainty affect control. (2) Swap further tests the concern that Copy's collapse merely reflects brittleness to any large perturbation. The swapped stream is a prediction of the same model, with the same token layout, noise schedule and label, but depicts a different episode or timestep. The large success drop supports observation-specific dependence on future content. Swap preserves the stream's format and schedule, while disrupting its conditional consistency with the current observation. On RoboTwin, Swap reduces success from 82.4% / 40.1% to 7.5% / 4.0% under C2C / C2R, respectively.

We agree that the Noise-versus-Copy ordering in the submission mixed content and label and cannot by itself establish that the predicted dynamics are accurate. The prediction-accuracy measurements above address that question directly.

Does a better prediction help the actions (offline)? On held-out episodes from the same six tasks, with the same initial action noise, we measure the error of the sampled action chunk against the demonstration (normalised MSE) while changing only what the action expert receives as the future stream.

Future-feature inputAction MSERelative change
The model's own prediction α\alpha0.0495Reference
True future yy, noised to the running level τ\tau0.0436−11.9%
Clean present frame β\beta, label σ=0\sigma=00.0635+28.3%

Supplying the true future at the appropriate noise level lowers the error on every task, showing that more accurate future information can improve sampled actions. Supplying the clean present frame raises it. This offline input differs from the running-label Copy intervention above. These measurements establish feature prediction and action relevance.

AuthorsOpenReview ↗

Response to Q1

4. Fairness and scope of the foresight ablations

The G1 ablations in Table 2 used the full model's learning rate and training-step budget, without a separate search for either hyperparameter.

The new controls show that continued predictive supervision matters across three simulation settings, with both vision and language backbones trainable:

Matched adaptationFull modelNo view lossNo foresight branch
LIBERO97.7074.6595.35
RoboTwin C2R40.111.622.0
SIMPLE Hard73.125.068.3

Within each benchmark, these variants share the mid-trained initialization, demonstrations, steps, batch size, and optimization schedule. Removing the view loss preserves architecture and parameter count, but the foresight branch continues to receive action-loss gradients. It is therefore not a fixed copy of the mid-trained predictor. The consistent degradation supports continued predictive supervision during adaptation, beyond a parameter-count explanation.

5. Human-video supervision and the human-only loss question

The future-feature loss is applied to all mid-training samples, including the 20% human-video replay: human observations receive the same dense future-feature supervision as robot observations. With 300,000 optimization steps and a global batch size of 1,024, this sampling fraction corresponds to approximately 300,000 × 1,024 × 20% = 61.44 million human sample presentations. Human pretraining itself uses the FAST objective. Future-feature supervision on human video enters during mid-training.

We agree that this exposure does not by itself establish the downstream performance gain attributable specifically to human video. Our adaptation ablations establish the value of continued predictive supervision, but do not isolate the contribution of human data from robot data or the preceding training stages. Our claim is therefore that the architecture incorporates dense future-feature supervision from human observations without requiring corresponding robot action labels.

The direct control would disable the future-feature loss only on human samples, retaining the same replay mixture, robot supervision, and other objectives. A separate control would remove human replay while matching the total mid-training budget, testing the contribution of the replay mixture as a whole. Each full-scale branch requires repeating the complete 300,000-step mid-training stage, followed by downstream adaptation and evaluation. These experiments exceed the compute budget available for this rebuttal, so we have not performed them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.