acceptodds
Desk rejected at ICLR 2027

AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to /, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS ( versus 0.122), and correct-versus- shuffled language changes trajectory error by only 2.8–3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot video prediction, world models, object-centric learning, particle dynamics, language-conditioned video prediction, robotic manipulation, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_CZ4zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AcrossVAM1.0 predicts future robot video frames from four context frames and a language instruction, treating motion and appearance separately. A frozen SAM3-DLP encoder turns each frame into three particles, one each for the robot, arm and gripper, plus a background latent. A small 0.28M-parameter Transformer then rolls these particles forward, with the instruction injected through FiLM from a frozen OpenCLIP text encoder. To produce the five future frames, the decoder renders the predicted particle motion, takes appearance from the last context frame, and uses a learned mask to blend the result with a copy of that frame. On their own VRS benchmark, the authors report a 21% lower trajectory error than simply keeping the particles still, and a gain of +0.6 dB PSNR and +0.004 SSIM over copying the last frame.

Critical strengths

The idea of separating particle-based motion from context-only appearance is reasonable, but without any learned baseline it is impossible to tell whether it offers any advantage over existing approaches.

Critical weaknesses

I believe this submission is not ready for review. No baseline has been run: all five methods in Table 2 are marked "queued" or "in progress", and the only measured comparison is copying the last frame, against which LPIPS is still worse and the PSNR gain is only +0.6 dB. Core components are left undefined or unused. Together with phrasing such as "below our 10% target", this makes the paper read like an internal progress report rather than a completed study, and I recommend rejection.

Questions for authors

NA

Reviewer_dyaMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AcrossVAM1.0, a text-conditioned robot video prediction model that separates motion from appearance. A frozen encoder maps the context frames to a few semantic particles (robot, arm, gripper). A small (0.28M-parameter) Transformer, conditioned on a text embedding via FiLM, predicts how these particles move. Against a persistence baseline (copying the last frame), the method gives small improvements in PSNR/SSIM and motion-region PSNR, but worse LPIPS. Language has only a small effect on the predictions.

This is an interesting research direction, but the paper reads like an early draft. In particular, Table 2 lists the baselines the authors plan to compare against, but none of these comparisons have been run.

Critical strengths
  1. Honest evaluation. The authors use a proper train/validation/test protocol, evaluating on test only once. They also report their negative results openly: LPIPS is worse than persistence, the language effect is small, and some robots perform worse than persistence.

  2. A compact dynamics model. The dynamics model has very few trainable parameters (0.28M). This makes it interesting to investigate, as it could be a powerful and interpretable interface for robot dynamics.

Critical weaknesses
  1. No baselines. The only comparison is persistence (copying the last frame). Table 2 lists five published methods as "queued" or "in progress" rather than reporting results. Without these comparisons, it is impossible to judge whether the particle-based approach offers any advantage over existing methods.

  2. Small gains. Compared to persistence, the improvement is +0.6 dB PSNR, and LPIPS is worse. Much of the gain appears to come from the learned mask blending the prediction back towards the last frame.

Other issues

  • Method clarity. Section 3 is hard to follow. I suggest starting it with one block of equations covering all learned components, then walking the reader through them. An algorithm box would also help.
  • Undefined terms. What is the "DLP" in SAM3-DLP? "DiffPrior", the model variant reported as the main result in Table 1, is also never described in the method section.
  • Notation.
    • In Eq. 6, y^TF\hat{\mathbf{y}}^{\text{TF}} and y^AR\hat{\mathbf{y}}^{\text{AR}} are not defined.
    • Eqs. 6 and 9 would be clearer if written as expectations, so it is clear which variables come from the data and which from the model.
    • h\mathbf{h} is indexed by the block ℓ\ell in Eq. 5 but by time tt in Eq. 7, which is confusing.
  • Motion PSNR. This metric is only described as "PSNR in a dilated motion/foreground region". How is this region computed?
  • Metrics. PSNR, SSIM and LPIPS are frame-level metrics. I suggest also reporting FVD (and FID).
  • Table 5. It is unclear what the reader should take from the "Floats" column and the results in this table.
  • Reproducibility. There are not enough details to reproduce the experiments; Appendix A is very brief.
  • Citation (line 86). Dreamer (Hafner et al., 2023) does reconstruct observations. Did the authors mean to cite a decoder-free method such as TD-MPC?
  • Figure 2. The text and images are too small; I can barely see the differences between the rows.
Questions for authors
  1. Can you provide results for the baselines in Table 2?
  2. What is SAM3-DLP, and what is the DiffPrior model?
  3. How is the motion region for Motion PSNR defined?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.