acceptodds
Under review as a conference paper at ICLR 2027

Resolving State‑Representation Mismatch: State‑Space Visual Reasoning for Open‑Loop VLA Planning

Abstract

Despite rapid progress in vision-language-action (VLA) models, existing reasoning paradigms still face a fundamental state-representation mismatch in open-loop planning. Given only an initial observation, models must internally simulate action-conditioned state transitions, whereas text-, pixel-, and latent-space reasoning can suffer from lossy spatial compression, error-accumulating visual generation, and bypass of intermediate latent tokens, respectively, undermining reliable long-horizon planning. We propose State-Space Visual Reasoning (SSVR), which decouples static visual context, language constraints, and a recurrent latent state. SSVR encodes the initial image and instruction once, then conditions each action prediction on the latent state and updates it with an action-conditioned GRU. Using Qwen2.5-VL as the backbone, SSVR achieves 99.5/99.6, 96.3/98.0, and 83.9/90.6 EM/PR on FrozenLake, Maze, and MiniBehavior, substantially outperforming prior methods. Extensive experiments support the effectiveness of recurrent state modeling for VLA open-loop planning across input transformations and transfer settings. By reusing static visual-textual context and updating a compact recurrent state, SSVR supports efficient multi-step inference, achieving up to faster Maze decoding rollouts than the evaluated baselines with the prefix cache prebuilt.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Reasoning, VLA Planning, State‑Representation Mismatch

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JrYROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the state-representation mismatch between largely static scene context and action-evolving execution state in open-loop planning for vision-language-action models. It proposes State-Space Visual Reasoning, which reuses the static visual-textual context and maintains a recurrent latent state updated by an action-conditioned GRU. Empirically, SSVR improves long-horizon planning and enables efficient multi-step inference, achieving up to 98.58× faster Maze decoding rollouts than baselines.

Critical strengths
  • S1. Strong empirical performance on the evaluated open-loop tasks. SSVR consistently outperforms the compared baselines on FrozenLake, Maze, and MiniBehavior. Gains are especially large on Maze and MiniBehavior, suggesting that an action-conditioned latent state is effective for these open-loop planning problems.
  • S2. Efficient inference. SSVR substantially reduces reported rollout latency compared with other reasoning methods, and its design is simple and practical.
Critical weaknesses
  • W1. The evaluated environments are too narrow to support broad VLA claims. FrozenLake and Maze are highly controlled: static visuals, deterministic transitions, and discrete actions. Even MiniBehavior is much simpler than realistic VLA settings. In real manipulation, observations change during execution due to camera motion, object motion, occlusion, and stochasticity. The initial image therefore may not be a sufficient persistent visual context. It remains unclear whether SSVR generalizes beyond cases where the state can be largely inferred from the initial observation and action history.

  • W2. The GRU state may be a bottleneck in long-horizon planning or detail-sensitive tasks. SSVR compresses the entire history into a single GRU state, so it is also lossy. The paper does not analyze long-horizon error accumulation or compare against Transformer long-context or external memory. The large performance drop on Maze Level 6 when the image is downscaled to 0.7 (Figure 4) suggests sensitivity to visual detail and state precision, but GRU capacity is not examined.

Questions for authors

Q1. What happens when the visual observation changes during execution? SSVR caches the initial visual context and only updates a latent state. How does it handle new visual information that cannot be recovered from the initial image and action history, such as camera motion, occlusion, or newly revealed objects?

Q2. How does performance scale to more realistic action spaces and longer horizons? The current experiments use mostly discrete actions and structured environments. Does the proposed representation still work for continuous actions, longer-horizon manipulation, stochastic transitions, and settings where observations change substantially over time?

Q3. How does SSVR scale beyond single-token discrete actions? Can it handle continuous, parameterized, or combinatorial actions while still reusing the LM head? How are tokenization and probability normalization handled then?

Q4. How important is the initial state design? The initial state is initialized from mean-pooled visual features. Does this lose spatial information? Have the authors compared attention pooling, learnable queries, or spatial token aggregation?

8 comments
AuthorsOpenReview ↗

Response to Reviewer JrYR

We thank Reviewer JrYR for recognizing the planning performance, inference efficiency, and simplicity of the design. To address your concerns, we have added long-horizon state information retention analysis (App E.2), state-capacity (App E.2) and module comparisons (App E.1), and initialization ablations (App E.3), as well as MazeFlip (App E.6), continuous navigation (App E.7), and restricted-history generalization experiments (App E.5). We respond to each point below.


Q1: Clarification of the exclusion of partially observable settings and additional experiments with dynamic scenes

First, we clarify that this work studies “pre-execution planning” in VLA: making full use of available visual information before acting to simulate action consequences and generate a complete plan. The scenarios raised in the review that require receiving new observations, identifying unexpected outcomes, and revising plans examine “adaptation during execution.” These correspond to different stages of a task. Accordingly, SSVR uses a protocol with an initial observation and no execution feedback to directly evaluate the model's capacity for internal visual simulation.

Pre-execution planning can also encompass dynamic changes governed by known rules. To this end, we add the MazeFlip experiment: LEFT/RIGHT actions trigger a vertical flip of the subsequent view, while UP/DOWN actions trigger a horizontal flip. The model receives only the initial image and performs internal simulation based on action history and known rules, without intermediate observations. As shown below, SSVR achieves 86.20% EM and 94.82% PR, demonstrating its ability to plan in dynamic scenes without intermediate observations.

Training epochEM (%)PR (%)
133.754.55
260.580.02
37288.58
480.492.25
586.294.82

Therefore, the original evaluation directly addresses the research objective of pre-execution planning in VLA, and the additional MazeFlip experiment further demonstrates SSVR's planning ability under predictable dynamic views. Online adaptation driven by new observations constitutes closed-loop adjustment during execution and lies beyond the scope of this work.


Q2: Validation of generalization to longer planning sequences

SSVR's learned state updates can extend to longer planning processes. We add a restricted-history training experiment, restricting training action prefixes to at most 5 steps, retraining the model, and evaluating it on 166 Maze tasks requiring 6–8-step plans, with a model trained on the full history range as a reference. Plans of 7–8 steps require continued decisions after action prefixes longer than 5 steps. The results are as follows:

SSVR training configurationEM on 6–8-step tasks (%)PR on 6–8-step tasks (%)
Restricted history: action prefix ≤5 steps78.31392.958
Full history86.74792.871

The restricted-history model generates complete optimal plans on 78.31% of the test tasks and achieves 92.96% PR.

These results show that the model can effectively generalize its learned state updates to longer action histories not covered by training supervision, supporting the generation of complete longer plans.


Q2/Q3: Clarification of the single-token restriction, LM head reuse, and extension to continuous actions.

SSVR's state update takes a vector encoding of the action, so it does not require each environment action to correspond to a single token. A single token is the encoding choice for the discrete-action interface in the paper; parameterized or combinatorial actions can be extended to multi-token representations to reuse the LM head, while continuous actions can also be supported by discretizing them into tokens or using a regression head for output.

The encoding extension for parameterized or combinatorial actions can retain the LM head prediction and normalization mechanism already used in the paper. Specifically, an action can be represented as “action type + parameters or action components + end marker,” with continuous parameters first discretized. The LM head predicts each token sequentially, applying softmax over the candidate set permitted by the current format; the probability of the complete action is the product of the conditional probabilities at each step. Once the entire action has been generated, it is encoded and the state is updated once.

To further examine the extension from discrete-token output to continuous-value regression, we add a two-dimensional continuous navigation experiment. The model predicts displacement through a two-dimensional regression head, encodes the displacement for state updates, and is trained accordingly. During validation, it plans from the initial image under a common 32-step budget without execution feedback; success requires reaching the goal region without collisions, and path efficiency measures motion cost relative to the shortest feasible path, with failed samples assigned zero.

Additional experimentSuccess rate (%)Path efficiency (%)
Two-dimensional continuous navigation, validation set (128 tasks)79.6977.04

This result demonstrates that the same state-modeling architecture can extend from discrete action selection to continuous motion planning by adapting the action output and encoding interfaces.


Q4: Clarification of the state-initialization strategy and an additional attention-pooling comparison.

First, we clarify that the initialization method is a replaceable implementation component. Mean pooling was chosen to keep the implementation and overall framework simple, highlighting SSVR's core design: decoupling the high-dimensional scene context that remains valid throughout execution from the low-dimensional execution state that evolves with actions, thereby addressing the “state-representation mismatch.”

In SSVR, mean pooling is responsible only for constructing the initial execution state; the complete visual tokens and their spatial information remain in the cache and are continuously available for decision-making. Spatial details therefore do not need to be independently encoded in the pooled vector, and the model can reread the relevant visual information according to its current state.

To address the reviewer's question about initialization design, we further add a learnable-query attention-pooling comparison. The results on Maze are as follows:

State initializationEM (%)PR (%)
Mean pooling96.397.98
Learnable-query attention pooling97.499.18

Attention pooling improves EM and PR by 1.10 and 1.20 percentage points, respectively, indicating that more selective information aggregation can further improve planning accuracy.

Therefore, mean pooling provides an effective and simple default initialization, while attention pooling demonstrates the benefits of module optimization; our decoupling paradigm accommodates different initialization strategies and does not depend on a particular pooling method.


W1: Clarification of evaluation scenario selection and additional experiments extending VLA open-loop planning

We first wish to emphasize that this work studies pre-execution planning in VLA, whose core challenge is to make full use of available visual information and generate optimal action plans through successive internal simulation. We select evaluation scenarios around this challenge to examine whether scene–state decoupling can address the failure of existing paradigms to perform multi-step reasoning and planning in visual scenes with complex topology due to “state-representation mismatch.”

Using VPRL's FrozenLake, Maze, and MiniBehavior both ensures baseline comparability and allows the evaluation to directly address the principal failure modes of three existing paradigms:

  • Text space: jointly compressing scene and state leads to loss of spatial relationships. These tasks require precise preservation of holes, walls, and passage connectivity; omitting or blurring key topological relationships in a language summary directly affects subsequent planning.

  • Pixel space: repeatedly reconstructing the entire scene leads to redundant computation and accumulation of generation errors. The scene is largely persistent, while action-induced changes in position and interaction state are relatively compact, allowing these tasks to effectively examine the difference between “regenerating the entire image” and “reusing the scene while updating the state.”

  • Latent space: insufficient association between intermediate representations and task-state evolution leads to visual-representation bypass. Navigation, picking up, carrying, and placing require the intermediate state to be continuously updated with actions and to participate in subsequent visual readout and decision-making. These tasks can therefore directly examine whether intermediate representations actually serve a multi-step reasoning function.

To address questions about broader scenarios, we further add dynamic-view and continuous-action experiments, which, together with the original cross-task transfer results, examine the method's capacity for extension:

Task or experimentCapability examinedResults
FrozenLake, Maze, MiniBehaviorRisk avoidance, topological path finding, and phased object interactionEM/PR (%): 99.5/99.6, 96.3/98.0, and 83.9/90.6, respectively
FrozenLake → MazeTransfer across map structures without target-task trainingEM/PR (%): SSVR 33.6/38.3; VPRL 6.8/7.2
MazeFlip experimentSimulation of predictable view changes without intermediate observations86.20% EM, 94.82% PR
Two-dimensional continuous navigation experimentObstacle-avoiding planning with continuous displacements, 128 validation tasks79.69% success rate, 77.04% path efficiency

We reiterate that “pre-execution planning” and “adaptation during execution” are capability requirements at different stages of VLA. Focusing on pre-execution planning, the subject of this work, the current experiments already provide direct evidence at three levels:

  1. The original tasks have already effectively demonstrated SSVR's ability to perform multi-step reasoning and planning in visual scenes with complex topology through scene–state decoupling.

  2. The additional MazeFlip experiment further demonstrates SSVR's planning ability on dynamic maps.

  3. The additional two-dimensional continuous navigation experiment also provides further validation of SSVR's planning ability in continuous action spaces.

Therefore, the original evaluation directly addresses the research question of this work, while the additional experiments further broaden the conditions of applicability; together, they support scene–state decoupling as an effective reasoning paradigm for pre-execution open-loop planning in VLA.


Reviewer_Tva4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces "State-Space Visual Reasoning" (SSVR), a method designed for open-loop vision-language-action planning. The authors argue that existing reasoning paradigms fail to adequately distinguish between "persistent scene context" and the "dynamically evolving execution state." SSVR requires only a single encoding of the initial image and instruction, utilizing an action-driven Gated Recurrent Unit (GRU) to update a recurrent latent state, which in turn guides subsequent action predictions. Experiments conducted in FrozenLake, Maze, and MiniBehavior environments demonstrate that this approach significantly outperforms baseline models in planning capability.

Critical strengths
  1. The primary contribution of this paper lies in its method to open-loop vision-language-action (VLA) planning, which decouples persistent vision-language context from the evolving execution state. This design maintains access to the original scene while preserving action-relevant information through a recurrent state.

  2. This method achieves significant improvements across all three tasks (FrozenLake, Maze, and MiniBehavior), with an average increase of 12.6 points in the EM metric compared to VPRL, and a 21.8 point increase in the Maze EM metric.

Critical weaknesses
  1. The contribution of the action-conditioned GRU to the reported planning gains is not sufficiently isolated by comparisons. While the ablations examine the soft-target coefficient and the action-output interface, they do not compare SSVR with an action-history-conditioned alternative trained using the same soft-target supervision.

  2. The setting of experiments is not described in sufficient detail. The paper fails to specify the number of maps in the training and test sets, the number of supervised decision states, the method used to partition the dataset at the map level, or the specific task rules that define "safe actions".

Questions for authors
  1. Could you compare SSVR with an action-history-conditioned alternative under the same soft-target supervision to isolate the contribution of the action-conditioned GRU?

  2. Could you specify the dataset sizes, map-level train/test splits, and task-specific definitions of the safe-action sets?

  3. What training data and supervision are used for the text-SFT, VPFT, and VPRL baselines, and are their evaluation protocols identical to SSVR’s?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tva4

We thank Reviewer Tva4 for recognizing the scene–state decoupling design and the planning gains. To address your concerns, we have added LSTM and action-history Transformer comparisons under the same soft-target supervision (App E.1), and provided dataset sizes, splits, safe-action definitions, and baseline training and evaluation protocols. We respond to each point below.

Q1/W1: Isolation of the contribution of the action-conditioned GRU and comparisons under the same supervision

We add LSTM and action-history Transformer comparisons under the same soft-target supervision, directly examining the role of the GRU relative to alternative implementations by fixing supervision and replacing the history-modeling module. Table entries report EM / PR (%):

TaskGRULSTMAction-history Transformer
FrozenLake99.50 / 99.5598.00 / 98.8197.50 / 98.95
Maze96.30 / 97.9896.30 / 97.4992.50 / 95.25
MiniBehavior83.87 / 90.5673.70 / 85.9666.25 / 84.49

In the additional comparisons, LSTM and the action-history Transformer achieve 96.30% and 92.50% EM on Maze, respectively, demonstrating that this state-modeling architecture is compatible with both recurrent updates and explicit history modeling. In the paper, we instead fix the architecture and change the supervision coefficient α\alpha from 1.0 to 0.7, improving Maze EM by 4.5 percentage points.

The two sets of experiments examine the effects of history modeling and the contribution of soft-target supervision through “fixing supervision and replacing modules” and “fixing the architecture and varying supervision,” respectively.


Q2/W2: Additional details on dataset sizes, splits, and safe-action definitions

SSVR uses VPRL's established training/test files, with no overlapping task instances between training and test sets in any of the three datasets. The same physical layout is counted as one map, while different configurations, such as start and goal positions, on the same map count as different task instances; training decision states are deduplicated within each task, with carrying status also considered for MiniBehavior. Both the effective supervised sample counts and the deduplicated decision-state counts refer to the training sets before EOS fine-tuning. The dataset sizes are as follows:

TaskTraining/test tasksUnique maps (training/test)Effective supervised samplesUnique supervised decision states
FrozenLake4,000 / 1,0002,719 / 797161,71162,282
Maze4,000 / 1,0003,175 / 883139,72671,863
MiniBehavior566 / 403566 / 14290,67829,347

Safe actions must first satisfy each task's legality constraints:

TaskLegality constraints
FrozenLakeMoves must stay within the grid and avoid holes.
MazeMoves must stay within the grid and cannot pass through walls.
MiniBehaviorMoves must follow traversability rules; PICK and DROP must satisfy the interaction-neighborhood and task-phase conditions for the printer and table, respectively.

Among legal actions, those that reach the goal, perform a valid PICK, or change the known distance value are included in the safe set.


Q3/W2: Clarification of baseline training supervision and consistency of evaluation protocols

text-SFT, VPFT, and VPRL each use the corresponding training data and supervision scheme from the VPRL paper. text-SFT is supervised using action sequences from optimal trajectories; VPFT is supervised using visual transitions from optimal trajectories; VPRL is first initialized through visual-transition training and then optimizes planning using GRPO and progress rewards. The specific training data sizes for each method are provided in Appendix E.1, Table 4 of VPRL.

All methods generate plans from the initial observation without actual execution feedback, using a budget of H=H∗H=H^\ast. EM evaluates whether any legal shortest solution is generated; PR measures the length of the longest consecutive optimal prefix from the starting point as a fraction of the shortest-solution length.

The revised manuscript has been uploaded, with the additional experiments, details, and visualizations included in the appendix. We hope this evidence addresses your concerns and that you will consider revising your rating upward. Thank you again for your time and constructive feedback.

Thank you for the detailed response and the additional experiments. I appreciate the new comparisons with LSTM and action-history Transformer under the same supervision, as well as the clarifications regarding dataset statistics and evaluation protocols. These additions address several of my original concerns.

However, some concerns regarding dataset splits and the fairness of baseline comparisons remain. While the rebuttal improves the paper, I do not find the additional evidence sufficient to change my overall assessment. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Request for Clarification of Remaining Concerns

We sincerely thank Reviewer Tva4 for acknowledging that our additional comparisons and clarifications have addressed several of the original concerns.

Regarding dataset splits, SSVR uses the exact training and test files released by VPRL [1], as can be verified in our anonymously released code.

[1] Yi Xu et al. Visual Planning: Let’s Think Only with Images. ICLR 2026.

To address the remaining concerns, we would appreciate clarification on which specific aspect of the dataset splits or which baseline comparison remains in question. This would help us provide targeted explanations and verifiable evidence.

We thank the reviewer again for the continued discussion.

Best regards,
The Authors

Reviewer_FfpdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors propose State-Space Visual Reasoning (SSVR) for open-loop planning from an initial image and instruction. It uses a single recurrent state token to condition action prediction by a Qwen2.5-VL backbone. After each predicted action, an action-conditioned GRU updates the state token for the next step, without receiving new observations. Experiments on FrozenLake, Maze, and MiniBehavior report improved planning accuracy over VPRL and other baselines, with faster decoding through context reuse and compact state updates.

Critical strengths
  • The method is simple and reasonably designed. Reusing the initial visual context while updating a compact GRU state avoids repeatedly generating the whole scene, and gives a concrete way to track action history.

  • The reported gains over VPRL are substantial across all three tasks, especially on Maze, where EM improves from 74.5% to 96.3%. The paper also measures decoding latency, though the headline speedup excludes prefix-cache construction.

  • The experiments go beyond in-distribution accuracy. Prompt and image transformations, FrozenLake-to-Maze transfer, and VQA retention provide useful checks of robustness and generalization. The direct transfer result is particularly encouraging.

  • The method is clearly specified, with equations, training and inference pseudocode, and implementation details that make the recurrent update easy to trace.

Critical weaknesses
  • The novelty is moderate. The action-conditioned GRU loop is interesting, but recurrent state tracking is established, and ete_t is still a latent representation. The results support this particular design more clearly than they establish a fundamental distinction between “latent-space” and “state-space” reasoning.

  • The comparisons do not isolate the proposed mechanism. SSVR receives task-specific training, while LVR and Monet are evaluated without it. The ablations vary supervision and the action head, but do not compare the GRU against an equally trained action-history Transformer or alternative recurrent update. Architecture and supervision remain mixed together in the reported gains.

  • The evaluation is narrow for the broader VLA framing. All three tasks are deterministic gridworlds where the initial image provides enough information to plan the full sequence, and the rollout budget is supplied from the ground-truth shortest solution length. The setting does not cover partial observability, where new observations are needed to revise the plan, or unexpected action outcomes. The transfer results are encouraging, but generalization to longer unseen plans and structurally different maps also remains unclear.

Questions for authors

Please address the weaknesses in the rebuttal.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ffpd

We thank Reviewer Ffpd for the careful assessment and recognition of our method design, planning performance, and transfer results. To address your concerns, we have added module comparisons under the same supervision (App E.1), autonomous termination (App E.4), length generalization (App E.5), MazeFlip (App E.6), and continuous navigation experiments (App E.7), as well as measurements of the one-time setup latency for preprocessing, prefix-cache construction, and state initialization (App F). We respond to each point below.


W1: Clarification of novelty and the distinction between “latent space” and “state space”

First, we clarify that SSVR's core innovation is to decouple the high-dimensional scene context that remains valid throughout execution from the low-dimensional execution state that evolves with actions, addressing the “state-representation mismatch” in open-loop planning. These two types of information play different temporal roles: scene information must be persistently retained, while the execution state must be updated with actions. Treating them jointly causes the model to repeatedly compress or reconstruct visual evidence that remains valid as it advances the state.

This mismatch manifests differently across paradigms: 1) text-space reasoning jointly compresses scene topology and dynamic state, severely degrading spatial relationships; 2) pixel-space reasoning repeatedly and inefficiently generates the entire scene, recomputing static content and propagating generation errors; 3) latent-space reasoning does not explicitly connect intermediate representations to task-state evolution, allowing subsequent decoding to shift toward linguistic context and bypass visual representations.

Compared with latent-space reasoning paradigms, SSVR's key distinction lies in an explicit division of information roles: the scene continuously provides visual evidence, while the state represents current task progress and guides visual readout. The recurrent state does not need to independently encode the entire scene or all prior reasoning content; instead, it serves as a dynamic query updated by actions to retrieve the visual information needed for the current decision. This division concentrates the state's representational demands on dynamic task information while preserving continuous access to the complete scene, yielding a reasoning mechanism of “persistent scene retention and state-guided readout.”

Therefore, SSVR and existing latent-space reasoning cannot be regarded as the same mechanism merely because ete_t is also a continuous latent representation; the innovation lies in scene–state decoupling and a coordinated reasoning architecture in which dynamic task states guide visual readout.


W2-1: Explanation of why LVR and Monet in Table 1 were not trained on the tasks

First, the comparison using officially released models directly demonstrates the failure of existing latent reasoning mechanisms. Taking Monet as an example, despite fine-tuning on large-scale image–text reasoning data, its average EM across the three tasks is 2.4%, even below native Qwen2.5-VL CoT's 3.7%. The mechanism analysis in the paper further shows that Monet stops producing latent tokens and switches to text-only reasoning on these tasks; during output generation, LVR allocates 3.5% and 3.8% of its attention to image and intermediate latent tokens, respectively. Together, these results indicate that their learned reasoning patterns fail to effectively support the open-loop, multi-step planning and reasoning tasks considered here.

Second, training both methods requires intermediate supervision absent from the current datasets, making such training infeasible. LVR and Monet require ROI visual reconstruction and interleaved image–text CoT supervision, respectively. We attempted direct fine-tuning with the available action labels, but the training loss did not converge and latent-token representations collapsed. Table 1 therefore retains and explicitly labels the direct-transfer results of the officially released models.

Third, the comparison between SSVR and VPRL is sufficient to establish effectiveness. Compared with VPRL (ICLR 2026) under the same training conditions, average EM increases from 80.6% to 93.2%, a gain of 12.6 percentage points, which is sufficient to demonstrate the advantages of SSVR's reasoning paradigm.


W2-2: Isolation of architectural and supervision contributions, and additional history-modeling comparisons under equal training conditions

We have added LSTM and action-history Transformer comparisons under the same training conditions and soft-target supervision, directly addressing the concern that architectural and supervision contributions are confounded. The two sets of experiments control different variables:

  1. Fixing the architecture to examine the contribution of supervision. In the paper, changing the supervision coefficient α\alpha from 1.0 to 0.7 improves Maze EM from 91.8% to 96.3%, a gain of 4.5 percentage points.

  2. Fixing supervision to examine the update module. The additional experiments keep the training data and soft-target supervision identical, replacing the GRU with an LSTM or an action-history Transformer to compare planning performance across history-modeling approaches.

Table entries report EM / PR (%):

TaskGRULSTMAction-history Transformer
FrozenLake99.50 / 99.5598.00 / 98.8197.50 / 98.95
Maze96.30 / 97.9896.30 / 97.4992.50 / 95.25
MiniBehavior83.87 / 90.5673.70 / 85.9666.25 / 84.49

These two sets of comparisons separately quantify the gains from supervision and the effects of the update module. LSTM achieves 96.30% EM on Maze, matching the GRU, and the action-history Transformer also supports this planning architecture, further demonstrating that the modeling principle of scene–state decoupling accommodates different implementations.


W3-1: Clarification of the exclusion of partially observable settings and additional experiments with dynamic scenes

First, we clarify that this work studies “pre-execution planning” in VLA: making full use of available visual information before acting to simulate action consequences and generate a complete plan. The scenarios raised in the review that require receiving new observations, identifying unexpected outcomes, and revising plans examine “adaptation during execution.” These correspond to different stages of a task. Accordingly, SSVR uses a protocol with an initial observation and no execution feedback to directly evaluate the model's capacity for internal visual simulation.

Pre-execution planning can also encompass dynamic changes governed by known rules. To this end, we add the MazeFlip experiment: LEFT/RIGHT actions trigger a vertical flip of the subsequent view, while UP/DOWN actions trigger a horizontal flip. The model receives only the initial image and performs internal simulation based on action history and known rules, without intermediate observations. As shown below, SSVR achieves 86.20% EM and 94.82% PR, demonstrating its ability to plan in dynamic scenes without intermediate observations.

Training epochEM (%)PR (%)
133.754.55
260.580.02
37288.58
480.492.25
586.294.82

Therefore, the original evaluation directly addresses the research objective of pre-execution planning in VLA, and the additional MazeFlip experiment further demonstrates SSVR's planning ability under predictable dynamic views. Online adaptation driven by new observations constitutes closed-loop adjustment during execution and lies beyond the scope of this work.


W3-2: Clarification of the use of ground-truth shortest-solution lengths and additional evaluation of autonomous termination

The paper uses ground-truth shortest-solution lengths to set the planning budget in order to maintain the same evaluation protocol as VPRL and ensure comparability with the baseline.

To examine whether SSVR can plan and stop autonomously without being given this length, we add an autonomous termination experiment. Specifically, we fine-tune the trained models for five additional epochs with EOS termination supervision to jointly learn action prediction and stopping decisions. At test time, instead of using each instance's ground-truth shortest-solution length, we apply a common maximum budget of 64 decisions, including EOS: generation ends when the model emits EOS; reaching the limit without emitting EOS is recorded as “no stop” and counted as a failure. We use “generating an optimal action sequence to complete the task and immediately emitting EOS” as the strict criterion for “optimal plan and stop.” The results are as follows:

TaskOptimal plan and stop (%)Correct stop (%)Early stop (%)Late stop (%)No stop (%)
FrozenLake92.292.51.360.2
Maze96.8971.90.20.9
MiniBehavior87.59390.3233.2266.2030.248

Therefore, the length budget in the paper serves baseline comparison under a unified protocol; the additional experiment directly demonstrates that, with termination supervision, SSVR can generate optimal plans and stop autonomously without externally supplied instance-specific shortest-solution lengths.


W3-3: Additional validation of long-plan generalization beyond the training history range

To address generalization beyond the action-prefix range covered by training supervision, we add a restricted-history training experiment to examine whether SSVR can continue planning effectively beyond its training history range. Specifically, we restrict the action prefixes in supervised training samples to at most 5 steps, retrain SSVR, and evaluate it on 166 Maze tasks requiring 6–8-step plans, using a model trained with the full history range as a reference. Plans of 7–8 steps require continued decisions after action prefixes longer than 5 steps. The results are as follows:

SSVR training configurationEM on 6–8-step tasks (%)PR on 6–8-step tasks (%)
Restricted history: action prefix ≤ 5 steps78.31392.958
Full history86.74792.871

The restricted-history model generates complete optimal plans on 78.31% of the test tasks and achieves 92.96% PR. These results show that SSVR's learned state updates can extend to longer action histories not covered by training supervision, supporting the generation of complete longer plans.


W3-4: Validation of transfer across map structures and extension to continuous actions

To address generalization across changes in map structure, Table 5 of the paper already reports direct transfer of the full model from FrozenLake to Maze. We directly apply the backbone adaptation parameters and state module trained on FrozenLake to Maze, without training on the target task. This transfer requires the model to shift its planning basis from avoiding holes to determining connectivity through walls; SSVR improves EM and PR over VPRL by 26.8 and 31.1 percentage points, respectively, demonstrating transfer across map structures and connectivity cues.

To further examine whether the method extends beyond discrete action settings, we add a two-dimensional continuous navigation experiment. We replace the action output with two-dimensional continuous displacement regression, encode the predicted displacements for state updates, and train the continuous-action model accordingly. During validation, the model uses only the initial image to plan obstacle-avoiding paths under a common 32-step budget without execution feedback. Success requires reaching the goal region within the budget without collisions; path efficiency measures motion cost relative to the shortest feasible path, with failed samples assigned zero. The results are as follows:

Evaluation settingSuccess rate (%)Path efficiency (%)
Two-dimensional continuous navigation, validation set (128 tasks)79.6977.04

Therefore, the original results demonstrate SSVR's ability to transfer across map structures without target-task training, while the additional experiment further demonstrates its ability to extend from discrete action selection to continuous motion planning by adapting the action interface.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We sincerely thank Reviewers Ffpd, Tva4, and JrYR for their rigorous, thorough reviews and highly valuable suggestions. We are delighted that all three reviewers recognize SSVR’s strong advantages in planning performance\color{green}{\text{SSVR's strong advantages in planning performance}} and acknowledge our work's strengths from different perspectives: Ffpd and Tva4 recognize the core design rationale of decoupling\color{green}{\text{core design rationale of decoupling}} high-dimensional scene context\color{green}{\text{high-dimensional scene context}} from low-dimensional state\color{green}{\text{from low-dimensional state}}; JrYR appreciates the method’s simplicity, efficiency,\color{green}{\text{the method's simplicity, efficiency,}} and practical applicability\color{green}{\text{and practical applicability}}; and Ffpd also acknowledges the comprehensive experimental evaluation\color{green}{\text{comprehensive experimental evaluation}}, including robustness\color{green}{\text{robustness}}, cross-environment transfer\color{green}{\text{cross-environment transfer}}, and VQA capability retention\color{green}{\text{VQA capability retention}}, as well as the clear presentation of implementation details\color{green}{\text{clear presentation of implementation details}} in the paper.

We also take the reviewers' questions very seriously and have responded point by point:

  • Reviewer Ffpd: (1) Novelty over latent-space reasoning:\color{blue}{\text{(1) Novelty over latent-space reasoning:}} we clarify that the innovation lies in decoupling high-dimensional scene context from low-dimensional execution state, thereby establishing a coordinated mechanism that integrates persistent scene retention, action-driven state updates, and state-guided visual readout; (2) Lack of task-specific training for LVR and Monet:\color{blue}{\text{(2) Lack of task-specific training for LVR and Monet:}} we explain that incompatible training protocols prevent such training and that the current results suffice to support our conclusions; (3) Confounding of architectural and supervision contributions:\color{blue}{\text{(3) Confounding of architectural and supervision contributions:}} we add comparisons of alternative architectures under the same supervision to validate the contributions; (4) Absence of partially observable settings:\color{blue}{\text{(4) Absence of partially observable settings:}} we clarify that adaptation during execution falls outside pre-execution planning, add MazeFlip to demonstrate support for predictable changes, and add experiments on MiniBehavior to validate SSVR’s ability to use local feedback during execution under stochasticity and partial observability; (5) Dependence on ground-truth planning lengths:\color{blue}{\text{(5) Dependence on ground-truth planning lengths:}} we add autonomous termination evaluations to demonstrate SSVR's independence from these lengths; (6) Length generalization:\color{blue}{\text{(6) Length generalization:}} we add restricted-history generalization experiments to demonstrate generalization; (7) Extension to different map structures or action spaces:\color{blue}{\text{(7) Extension to different map structures or action spaces:}} we add navigation experiments in a continuous action space with obstacle avoidance to demonstrate extensibility; (8) Cache construction latency:\color{blue}{\text{(8) Cache construction latency:}} we add cache construction time measurements to demonstrate inference efficiency.

  • Reviewer Tva4: (1) Comparisons across architectures:\color{blue}{\text{(1) Comparisons across architectures:}} we add comparisons of alternative architectures under the same supervision to demonstrate compatibility; (2) Dataset details and baseline training/evaluation protocols:\color{blue}{\text{(2) Dataset details and baseline training/evaluation protocols:}} we provide further details to resolve ambiguities.

  • Reviewer JrYR: (1) Changing visual observations:\color{blue}{\text{(1) Changing visual observations:}} we distinguish open-loop planning from closed-loop adaptation and use MazeFlip to demonstrate support for predictable changes, adding MiniBehavior experiments with local feedback for newly revealed obstacles; (2) Longer plans:\color{blue}{\text{(2) Longer plans:}} we add restricted-history generalization experiments to demonstrate support for longer plans; (3) The single-token restriction and complex actions:\color{blue}{\text{(3) The single-token restriction and complex actions:}} we explain how to support multi-token encoding and reuse the LM head, and add continuous navigation experiments to validate support for continuous action spaces; (4) Information loss from mean pooling:\color{blue}{\text{(4) Information loss from mean pooling:}} we clarify that all visual information remains cached, while the low-dimensional state obtained by mean pooling only guides queries; (5) Broader VLA scenarios:\color{blue}{\text{(5) Broader VLA scenarios:}} we clarify the pre-execution planning objective, add MazeFlip to demonstrate support for predictable changes, add continuous navigation experiments to validate continuous action spaces, and add MiniBehavior experiments with local feedback to demonstrate practical applicability under stochasticity and partial observability; (6) GRU capacity and long-horizon bottlenecks:\color{blue}{\text{(6) GRU capacity and long-horizon bottlenecks:}} we clarify the low capacity requirements of the low-dimensional execution state and add long-horizon state information retention analysis and state-capacity ablations for further validation.

We thank the reviewers for their constructive suggestions, which improved the paper. All these additional experiments and clarifications are incorporated into the revised manuscript. We believe we have sufficiently addressed the main concerns and respectfully ask the reviewers to reassess our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.