acceptodds
Under review as a conference paper at ICLR 2027

EvoScene-VLA: Evolving Scene Beliefs Inside the Action Decoder for Chunked Robot Control

Abstract

Chunked vision–language–action (VLA) policies generate multi-step actions from one observation and typically re-observe after executing the action chunk. Earlier actions change the scene conditions for later steps, while occlusion can compromise visual feedback. The previous scene estimate is typically not propagated to the next control call. Although spatial and temporal VLAs enhance geometric modeling and maintain temporal memory, respectively, their internal representations may become stale as robot actions alter object poses and contact states. We introduce EvoScene-VLA, which uses compact scene tokens to unify within-chunk scene prediction, cross-chunk state propagation, and observation-based correction. The action expert jointly denoises actions and future scene states in a single flow-matching process, allowing predicted scene changes to inform action generation. The final scene state serves as the next prior for subsequent correction. At the next control call, the vision–language model (VLM) fuses the incoming visual input with this prior. To preserve control-relevant geometry, two-level geometric anchoring applies local depth supervision to per-view scene features and global 3D feature supervision to the fused scene state; a shared geometric decoder imposes the same constraints on current and future states. Since demonstrations do not directly provide future scene states, we train a scene predictor using 3D teacher features from future images. Its predictions provide targets for the action expert's future scene states. At deployment, we remove both Scene Predictor and Geometric Anchor. EvoScene-VLA improves average success over the matched baseline by 2.4 and 2.2 percentage points on 31 RoboTwin tasks under fixed and randomized settings, respectively, and by 2.7 points on LIBERO. Across three real-robot tasks, average success rises from 37.3% to 42.0%. Ablations show that a recurrent scene state alone does not necessarily improve performance, whereas state propagation can improve control when combined with geometric and future scene supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Action Model, 3D understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YTTtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoScene-VLA, a recurrent scene representation framework for chunked vision-language-action (VLA) policies. Built upon LingBot-VLA, the method jointly generates action chunks and future scene states through flow matching. The final predicted scene state is propagated to the next control chunk as a prior, where it is fused with new visual observations. During training, geometric supervision and a separate Scene Predictor provide learning signals for the scene representations, while these auxiliary modules are removed at inference.

The method is evaluated on RoboTwin, LIBERO, and three real-world cleaning tasks using Galaxea R1-Lite. Compared with a matched LingBot-VLA baseline, it improves average success rates by 2.4 and 2.2 percentage points on RoboTwin under Clean and Rand settings, respectively, by 2.7 points on LIBERO, and from 37.3% to 42.0% on real-robot tasks.

The central contribution is the integration of action-conditioned future scene prediction, cross-chunk state propagation, and observation-based correction within a unified VLA architecture.

Critical strengths

S1. Meaningful formulation and reasonably controlled evaluation.

The paper addresses a relevant limitation of chunked VLA policies: scene representations may become outdated as robot actions change the environment. Integrating future scene prediction and recurrent state propagation into the action decoder is an interesting design.

The experimental evaluation includes a matched LingBot-VLA* baseline with the same backbone, action chunk length, sampling schedule, and training protocol. The RoboTwin results include three independent training seeds, with 100 rollouts per task, setting, and seed. The improvements are observed across multiple tasks rather than being restricted to a few selected examples. The inclusion of real-robot experiments further strengthens the empirical evaluation.

S2. Useful ablations investigating recurrent scene representations.

The paper provides more than standard performance comparisons. Table 6 shows that introducing recurrence without appropriate scene supervision can reduce performance, while recurrence becomes beneficial when combined with scene supervision. Table 3 additionally investigates the effect of matched versus mismatched scene priors during closed-loop execution.

These experiments provide useful evidence that the recurrent representation carries information relevant to control, although they do not fully establish the necessity of the proposed mechanism.

Critical weaknesses

W1. The necessity and independent contribution of the proposed recurrent scene mechanism remain insufficiently established.

The paper argues that conventional temporal memories may become outdated after robot actions and proposes action-conditioned scene prediction to address this problem. However, existing VLA approaches already incorporate temporal memory, future-state prediction, or world-model representations. The paper does not sufficiently establish why the proposed action-updated recurrent state is necessary compared with simpler alternatives.

For example, an informative comparison would be to retain the previous corrected scene state across chunks without explicitly predicting future scene states, or to use an equally sized historical-memory representation. Although the paper discusses MemoryVLA, CUT3R, FLARE, UP-VLA, and other related methods, it lacks direct mechanism-level comparisons under matched conditions.

Furthermore, the ablations do not fully disentangle the contributions of geometric supervision, future-scene supervision, and joint action–scene generation. In Table 6, the future-prediction and representation losses are grouped together. It therefore remains unclear whether the improvements primarily arise from better auxiliary representation learning or from the proposed action-conditioned scene generation and propagation mechanism.

This limitation directly affects the assessment of the paper's central contribution, because the experiments establish the effectiveness of the complete system more convincingly than the necessity of its individual design choices.

W2. The claimed dynamic scene correction mechanism lacks direct experimental validation.

A central motivation is that the model should maintain an evolving scene state and correct inaccurate predictions using new observations. However, the experiments do not directly evaluate the accuracy of future scene predictions or the model's ability to correct incorrect priors when actual scene evolution differs from the predicted evolution.

Table 3 demonstrates that matched scene priors improve closed-loop performance compared with mismatched priors. Nevertheless, demonstrating that prior information affects action generation is not equivalent to demonstrating reliable scene-state correction. In particular, the camera-view replacement experiments introduce severe observation inconsistencies and do not directly establish robustness to natural occlusions, unexpected object movements, or execution disturbances.

An additional concern is the difference between training and deployment. During training, recurrent priors for subsequent chunks are supplied by the Scene Predictor using demonstration actions, while during inference they are generated by the action expert itself. The paper does not sufficiently clarify whether the same normalization is applied to these priors or investigate the consequences of possible prediction-distribution differences over repeated chunks. This is a potential source of error accumulation rather than an established implementation error.

Without direct measurements of prediction accuracy, correction accuracy, and recovery under scene changes, the evidence for the complete recurrent prediction-and-correction loop remains incomplete.

Questions for authors
  1. Necessity of action-conditioned recurrence: How does the proposed mechanism compare with simply propagating the previous corrected scene state, or maintaining an equally sized historical-memory representation? These comparisons would help establish whether explicitly predicting future scene states provides an independent benefit.

  2. Relationship to existing world-model approaches: Can the authors clarify the specific advantages of the proposed mechanism over relevant action-conditioned future-prediction and world-model approaches, such as FLARE, UP-VLA, and Motus? The published Motus results in Table 1 use a different training task set and therefore do not constitute a controlled comparison.

  3. Contribution of joint scene generation: If all geometric and future-scene supervision is retained, but action tokens are prevented from attending to predicted future scene tokens, how does performance change?

  4. Dynamic scene correction and training–inference consistency: How accurately do predicted scene states reflect actual future observations? Under unexpected object movement, execution disturbances, or incorrect priors, can the authors quantify prediction errors, correction accuracy, and recovery performance? Additionally, are recurrent priors normalized consistently during training and inference, and does training with action-expert-generated priors affect long-horizon stability?

  5. Reliability of the reported improvements: Can the authors provide multi-seed uncertainty for the LIBERO component ablations and uncertainty estimates for the real-robot performance differences between methods? The reported Wilson intervals quantify individual success rates but do not directly characterize the uncertainty of the observed improvements.

1 comment

Dear Reviewer YTTt,

Thank you once again for taking the time to review our paper and for your constructive feedback.

We sincerely hope that the additional experiments and our response have addressed your concerns.

If you have any further questions or suggestions, we would be glad to offer further clarifications.

Thank you for your time and consideration!

Reviewer_AVTkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper adds compact recurrent scene tokens to a chunked LingBot-VLA policy. The action expert uses flow matching to generate an action chunk together with future scene states. The last predicted state then serves as the prior for the next observation, and the VLM updates it using the current images. Training combines local depth supervision, supervision on current/future 3D teacher features, and a Scene Predictor used only during training to provide latent targets. The auxiliary predictor and the geometry decoders are removed at deployment. Against a matched baseline with depth supervision, the paper reports improvements on 31 RoboTwin tasks and three real-robot cleaning tasks, plus LIBERO component ablations. In these ablations, recurrence helps when combined with scene supervision but can hurt without it.

Critical strengths
  1. The paper gives a concrete way to carry a predicted scene state between action chunks. The action expert generates the action chunk and future scene states together. The final state is passed to the next VLM call and updated with the new observation (pp. 4–6; Algorithm 2, p. 16). The extra Scene Predictor and geometry decoders are only used during training, which keeps the deployed architecture relatively simple.

  2. The matched-baseline evaluation is solid, and the recurrence result is informative. On RoboTwin, the 2.4/2.2-point gains are backed by three training seeds, 100 rollouts per task/setting/seed, and higher means on 22/23 of the 31 tasks (Table 10, p. 18). Table 6 (p. 8) shows that recurrence improves LIBERO by 1.10 points with full scene supervision but lowers it by 2.10 points without scene supervision or local depth. This is useful evidence that the benefit of recurrence depends on the scene supervision in the tested setup.

Critical weaknesses
  1. I am not convinced that the experiments establish the main claimed benefit of within-chunk scene prediction. The introduction links action–scene mismatch to later actions being conditioned on the initial observation (pp. 1–2). But a decoder conditioned on that observation can still learn to anticipate the effects of earlier actions. Fixed visual input does not by itself imply a static internal scene representation. Algorithm 2 (p. 16) also executes the full action chunk before re-observing. Explicit prediction may improve planning under this constraint, but it does not add visual feedback during execution. The key question is therefore whether it improves on implicit prediction under comparable conditions. Tables 4 and 6 (p. 8) switch current-state alignment and future prediction together (Lrep/Lpred), and do not isolate scene-to-action interaction while keeping history, supervision and token capacity matched. The gains could come from better geometric features, auxiliary supervision or memory. The recurrence ablation is useful, but it does not answer this within-chunk question. Since this mechanism is central to the paper, I find the evidence insufficient.

  2. The geometry and prior diagnostics give limited support for the proposed scene-state explanation. Table 2 and Appendix J (pp. 7, 22) measure current-state depth relative to the teacher on training episodes, with per-image log-depth scale/shift alignment. The paper states these conditions clearly, but the readout does not independently validate the future states generated by the action expert. Table 3 compares matched priors with priors from other episodes, possibly other tasks, under intact or fully replaced camera views. This shows that prior content matters. It does not separate the benefit of a useful prior from the harm of an incorrect one, or establish the benefit over a cold prior under realistic occlusion. The seed and trial counts for Tables 3 and 6 should also be reported. These diagnostics leave the future-state and occlusion interpretations uncertain, although the main RoboTwin gains remain supported.

Questions for authors
  1. With the same observation frequency, recurrent history, geometry/future supervision and comparable token capacity, does blocking future-scene-to-action interaction remove the gain? Can you separate current-state Lrep alignment from future Lpred supervision? This would help determine whether the improvement needs the proposed within-chunk mechanism.

  2. Figure 1 (p. 2) gives qualitative examples of improved execution. Is there a controlled quantitative test showing less action–scene mismatch later in a chunk? How accurately do the generated future states predict scene changes on held-out episodes? The current-state depth readout and overall task success do not directly test this explanation.

  3. Is there a matched-versus-cold-prior closed-loop comparison under the same view intervention, or another control that separates useful prior information from harmful mismatched information? Please also give the training-seed and rollout counts, and the uncertainty of the relevant differences in Tables 3 and 6.

1 comment

Thank you for the clarification. My main concerns have been addressed, and I am raising my score to weak accept.

Reviewer_qAvPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoScene-VLA, a framework which maintains and evolves latent scene representation in VLA policies. The key idea is to jointly generate action chunks and future scene states through flow matching. The motivation is to use predicted scene changes to inform action generation. another important key design is that the predicted scene state is propagated to the next action step and fused with new visual observations. The author introduce geometric supervision using depth and 3d foundation models with a scene predictor that provides supervision targets for predicted scene states. Experiments on both simulation environment and real-robot environment demonstrate improvements over baselines.

Critical strengths

I think the paper has the following strength:

  1. well-motivated problem. Scene changes during action execution is a well known challenge in robotic manipulation. The author's approach of maintaining a scene representation is a reasonable and meaningful direction.
  2. coherent method design. The use of geometric supervision and a training-only scene predictor provides a practical way to learn these latent representations without requiring a separate scene prediction model at inference time.
  3. reasonably comprehensive experimental evaluation. the author evaluated their method on robotwin tasks, libero, and three real-robot tasks. the comparisons with matched baseline, component ablations and prior intervention experiments provide useful evidence for the effectiveness of the proposed framework.
Critical weaknesses

Thank you for the awesome work! I have the following concerns and hope the authors may clarify them.

  1. Insufficient evidence for the necessity of joint action scene generation. While the framework improves performance, it is still unclear whether the gains come from joint action-scene generation or simply from the additional geometric supervision and recurrent scene memory. it would be helpful to compare against a VLA using the frozen 3d features and similar recurrent memory, but without joint scene generation, to better justify the proposed design and motivation.
  2. limited evaluation under realistic occlusion. one of the main motivation of the paper is to address partial observations caused by occlusion. however, the prior intervention experiment replace an entire camera view with an image from another episode, which differs significantly from realistic occlusion, where only parts of the scene are temporarily hidden. and under this setup, proposed method's performance also dropped sharply from 85% to 14% and 13% raising concerns.
Questions for authors

Here are my questions for the author to clarify further:

  1. how much of the performance gain comes specifically from joint action-scene generation, rather than the additional geometric supervision and recurrent scene memory? Could the author compare against a VLA using pretrained 3D features and similar recurrent memory, but without joint scene generation?
  2. could the authors evaluate the framework under more realistic occlusion scenarios, where objects or parts of the scene are temporarily hidden, rather than replacing an entire camera view with an unrelated image?

Comments on the paper

6 comments
Area_Chair_jnGiOpenReview ↗

Reminder: Author–Reviewer Discussion Deadline Approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns, if you haven't already. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

Thank you for the additional ablations! In particular, the attention masking control (row 5) is very helpful and has largely addressed some of my original concern.

I have one small follow-up question regarding the Scene Predictor.

Since the Scene Predictor is already trained and used to provide recurrent scene priors during training, I wonder whether it could also be retained at inference.

Specifically, could you keep the full model unchanged, but instead of carrying the scene state generated by the Action Expert to the next chunk, use the trained Scene Predictor to update the memory based on the current scene and the generated actions?

I think this could be a relatively simple inference-only ablation, potentially without retraining. Even a small-scale test would be helpful to compare the two sources of scene memory.

Of course, if time is limited, a brief explanation of why the Scene Predictor is removed at inference or what you expect this experiment's result to be would also be appreciated. No additional experiment is necessary if it is inconvenient.

Thank you again for the detailed response!

Dear Reviewer qAvP,

Thank you for the constructive follow-up. We’re glad the additional controls helped address your earlier concerns.

1. Why is the Scene Predictor used only during training?

Our goal is for the Action Expert itself to learn how the scene changes as actions are executed, and to use these predictions for both action generation and scene memory updates across chunks.

A training challenge is that demonstrations contain future images but do not directly provide the compact scene states needed by the Action Expert. We therefore introduce the Scene Predictor and supervise its predictions with geometric features extracted from future images. Its predicted states serve both as learning targets for the Action Expert and as recurrent priors during training. This provides explicit geometric and temporal supervision for learning the scene representation.

With this supervision, the Action Expert learns to generate future scene states jointly with actions. Its predicted end-of-chunk state can then serve as the prior for the next control call, where a new observation corrects it. Our intention is therefore to retain the learned scene prediction capability within the action decoder, so that deployment does not require a separate model to update the scene memory. This is the main reason we remove the Scene Predictor at inference.

2. Does retaining it at inference help?

We used the same checkpoint without retraining and kept joint action–scene denoising unchanged. After denoising, we fed the current scene state and the generated action chunk to the Scene Predictor. Its output replaced only the next-chunk prior.

In a single-seed LIBERO evaluation on an RTX 4090:

Source of the next-chunk priorSuccess rate (%)  Inference latency per chunk (ms)
Scene Predictor (suggested variant)85.75338.0
Action Expert (original EvoScene-VLA)86.60311.9

In this run, retaining the Predictor did not improve success and added 26.1 ms per chunk and 42.1M parameters at deployment.

This result supports our design choice: the scene predictions learned and generated by the Action Expert can support control across chunks without retaining the auxiliary Predictor at deployment.

We appreciate the suggestion and hope this comparison addresses your question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.