acceptodds
Under review as a conference paper at ICLR 2027

PLATO: A Unified World-State Representation for VLA Perception and Prediction

Abstract

As in Plato’s allegory of the cave, vision-language-action (VLA) models learn from 2D visual projections, while the underlying physical world state remains implicit. We introduce PLATO (Physical Language for Action through Tokenized Observations), a unified world-state representation for VLA perception and prediction. Learned from multi-view observations with geometric, semantic, and forward–inverse dynamics supervision, PLATO preserves 3D structure, semantics, and interaction-relevant information in a compact form. Continuous features or embeddings of discrete codes provide current-state context, and 128 discrete state tokens serve as autoregressive future-state targets alongside action supervision. Both roles leverage the VLM’s native conditioning and next-token prediction paradigm, without specialized dense prediction branches. Across VLA pretraining and post-training on three simulation benchmarks and four real-world tasks, PLATO improves manipulation success and robustness. Future-state supervision raises average real-world success by 11.7 and 25.4% under nominal and randomized conditions, respectively, and current-state context further improves viewpoint robustness, with a 40% gain over supervision alone in flower insertion. Beyond performance, our analyses show that PLATO reshapes VLA representations: geometry becomes more linearly accessible and RGB embeddings more consistent across viewpoints, revealing that explicit world-state inputs and targets substantially alter the physical structure learned by the VLA from robot data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Vision-Language-Action Models, World-State Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ieW3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PLATO, a unified world-state representation for Vision-Language-Action (VLA) models. It compresses multi-view visual observations into 128 discrete state tokens using geometric reconstruction, semantic supervision, and forward/inverse dynamics objectives.

The learned representation is incorporated into VLAs in two ways: (1) as an auxiliary target for autoregressive future-state prediction during training, and (2) as additional current-state context for action generation. The authors evaluate the framework on RoboTwin2.0, LIBERO-Plus, RoboCasa Human-50, and four real-world manipulation tasks. They also investigate geometric decodability and cross-view feature consistency.

The results show promising improvements in certain real-world precision-manipulation and viewpoint-shift settings, although gains on several simulation benchmarks are modest.

Critical strengths

1. A unified interface for VLA perception and prediction.

The most compelling contribution is the use of a shared world-state representation for both current-state conditioning and future-state supervision. This provides a coherent alternative to incorporating multiple heterogeneous geometric and semantic prediction targets. In particular, the supervision-only approach does not require running the tokenizer during policy inference, which is a practical advantage.

2. Promising real-world results and representation analysis.

The paper evaluates four real-world manipulation tasks on three robotic platforms. PLATO supervision improves average success from 73.75% to 85.42% under nominal conditions and from 47.92% to 73.33% under task-specific perturbations. The cable-tie threading results are particularly encouraging. The geometric probing experiments further provide evidence that the auxiliary supervision changes the information encoded by the VLM, rather than merely adding an additional input modality.

Critical weaknesses

1. Insufficient isolation of the proposed design choices.

The action-only comparisons support the effectiveness of the overall PLATO training approach. However, the alternative prediction-target comparisons in Table 4 vary multiple factors, including view coverage, prediction length, loss formulation, and auxiliary weight. This makes it difficult to isolate the respective benefits of 3D geometry, discretization, and the prediction interface.

Furthermore, Figure 5 evaluates high-dimensional pre-MLP features rather than the recovered quantized representation. Although the tokenizer's reconstruction and dynamics objectives operate through the discrete bottleneck, direct evidence of information preservation after quantization remains limited.

2. Limited characterization of performance reliability and generalization.

The improvements vary substantially across benchmarks. Gains are relatively small on RoboTwin2.0 (+1.64/+0.58 percentage points), while LIBERO-Plus and RoboCasa show more meaningful improvements. Additional current-state context improves overall LIBERO-Plus performance by only +1.06 points, although it substantially improves viewpoint robustness in real-world flower insertion.

The real-world results are encouraging, but several comparisons involve limited trial counts. Without uncertainty estimates or repeated policy-training runs, it is difficult to assess the consistency and broader generality of the reported improvements.

Questions for authors
  1. Statistical reliability: Could the authors provide uncertainty estimates or repeated-seed results for the main improvements, particularly the small RoboTwin2.0 gains and the real-world flower-insertion viewpoint-shift results? Also, is the reported 30-minute cable-tie experiment identical to the 29/30 nominal evaluation, and how was the 1–2 mm alignment requirement assessed?

  2. Fairness and data separation: Can the authors clarify the additional training compute required for PLATO and the overlap between tokenizer training data and downstream evaluation benchmarks? Are evaluation episodes completely excluded from tokenizer training? How much of the gain remains when controlling for additional teacher supervision or training resources?

  3. Contribution of the discrete representation: Can the authors provide a more controlled comparison between discrete PLATO targets and continuous/2D alternatives, particularly with matched view coverage and auxiliary training budgets? Also, how much geometric and semantic information is preserved in the quantized tokens, rather than only in the pre-quantization features evaluated in Figure 5?

  4. Complementarity of supervision and context: Can the authors provide a context-only baseline or otherwise clarify whether jointly using context and supervision provides benefits beyond independently adding either component? Given the relatively small overall LIBERO-Plus context gain, what evidence supports broader applicability beyond viewpoint-shift scenarios?

7 comments

Thank you for your careful assessment of PLATO’s shared perception/prediction interface and its deployment advantages. Your questions about controlled comparisons, statistical reliability, and the standalone contribution of context helped us focus the revision on the evidence needed to assess the method. In response, we have added more closely matched representation comparisons, additional downstream training seeds, rollout-level uncertainty estimates, and a context-only baseline. We address your questions below.

[W1/Q3: View- and token-matched Cosmos controls]

R1: We add a three-view Cosmos DI16×16 control with 126 tokens, closely matching PLATO's 128 tokens. Each future view is resized to 96 × 112, producing a 6 × 7 code grid. All policies use the same 40k-step training recipe and action objective.

Table R3.1. View- and token-matched Cosmos comparison.

Future targetFuture viewsPredicted tokensClean (%)Randomized (%)Combined (%)
None, action-only——69.8070.6270.21
Cosmos, original125673.7270.2071.96
Cosmos, compact three-view312671.8471.8471.84
PLATO discrete312873.4873.0273.25

PLATO improves combined success over the three-view Cosmos control by 1.41 percentage points, providing stronger evidence for the benefit of its 3D-aware future-state supervision.

[W1/Q3: Auxiliary-loss sensitivity]

R2: We sweep the auxiliary-loss weight while keeping each objective's internal formulation fixed. Although the best-performing weight varies across objectives, PLATO achieves the highest reported success at both tested nonzero weights.

Table R3.2. Auxiliary-loss weight sweep: combined success (%).

Auxiliary targetλ = 0λ = 0.1λ = 1
PLATO discrete70.2173.2572.74
Continuous pre-VQ70.2172.7370.17
Depth–DINO70.2166.5270.89

[W1/Q3: Actual quantized representation]

R3: We recover features from the actual 128 discrete token IDs using the frozen codebook and recovery network, and evaluate them with the same probe splits and training budget as Figure 5.

Table R3.3. Representation fidelity before and after quantization.

Tokenizer readoutShapeDepth AbsRel ↓Instruction accuracy (%) ↑
Pre-MLP, Figure 5128 × 10240.03368 ± 0.0006498.77 ± 0.40
Recovered post-VQ128 × 10240.03208 ± 0.0008296.80 ± 0.26

The recovered representation largely preserves Figure 5's geometric and semantic fidelity, with comparable depth accuracy and 96.80% instruction-matching accuracy after quantization.

[W2/Q1: Additional policy-training seeds]

R4: The submitted results in Tables 1, 2, and 4 use one policy-training seed per condition. Your emphasis on statistical reliability is especially helpful for interpreting the smaller RoboTwin gains. To examine their stability across downstream training runs, we add two training seeds, 10241 and 10242, alongside the submitted seed.

Table R3.4. RoboTwin policy-training seed results.

Training seedAction-only: Clean / Randomized / Combined (%)PLATO: Clean / Randomized / Combined (%)
1024088.08 / 88.60 / 88.3489.72 / 89.18 / 89.45
1024188.72 / 88.86 / 88.7989.80 / 89.08 / 89.44
1024289.76 / 89.02 / 89.3990.36 / 89.68 / 90.02
Three-seed mean ± SD88.85 ± 0.85 / 88.83 ± 0.21 / 88.84 ± 0.5389.96 ± 0.35 / 89.31 ± 0.32 / 89.64 ± 0.33

Across seeds 10240, 10241, and 10242, PLATO achieves 89.64 ± 0.33% combined success, compared with 88.84 ± 0.53% for action-only. The seemingly modest RoboTwin gain is an average over 50 tasks, many of which already have little room for improvement: 19 action-only tasks and 22 PLATO tasks reach at least 95% success in the original clean evaluation. This average therefore combines near-saturated tasks with more challenging ones, making the larger gains on lower-baseline tasks less visible. On these tasks, PLATO-initialized policies show consistent improvements across all three available trained seeds, for example:

Table R3.5. Task-level success gains across training seeds.

TaskSeed10240 gain (pp)Seed10241 gain (pp)Seed10242 gain (pp)
turn switch+9+10+6
scan object+6+3+5
place bread skillet+5+5+3

[W2/Q1: Evaluation rollouts and confidence intervals]

R5: We report rollout counts and 95% confidence intervals for the gains across the main comparisons, focusing on the two cases you highlighted. The Table 2 LIBERO-Plus context comparison uses 10,030 rollouts per policy and yields a 95% CI of [+0.20, +1.91] percentage points; the real-world flower-insertion comparison under viewpoint shift uses 15 trials per method and yields a 95% CI of [+8.16, +63.85] percentage points. Both intervals exclude zero, indicating statistically significant gains in the rollout-level analysis.

Table R3.6. Evaluation rollout counts and confidence intervals for gains.

ComparisonRollouts per policyScope / methods comparedGain (pp)95% CI for gain (pp)
Table 1, RoboTwin (submitted)10,00050 tasks × 100 episodes × 2 environment configurations+1.11[+0.24, +1.98]
Table 1, LIBERO-Plus10,030PLATO supervision vs. action-only+7.58[+6.60, +8.56]
Table 1, RoboCasa Human-501,200PLATO supervision vs. action-only+6.33[+2.34, +10.30]
Table 2, LIBERO-Plus context10,030Context vs. supervision-only initialization+1.06[+0.20, +1.91]
Table 4(a), prediction interfaces10,000PLATO supervision vs. action-only at 40k+3.04[+1.79, +4.29]
Table 15, real-world example15 per methodShifted-view flower: context vs. supervision+40.00[+8.16, +63.85]

Intervals use an unpaired, rollout-level Newcombe/Wilson approximation.

[W2/Q1: Cable-tie task and precision description]

R6: The nominal cable-tie result is 29 successful insertions in 30 trials during the continuous 30-minute run. The 1–2 mm description refers to the cable-tie hole diameter measured with a ruler and specifies the task's physical alignment constraint.

[Q2: Additional training costs]

R7: We quantify the RoboTwin policy-training cost for 40k steps on eight A800 GPUs, with a global batch size of 128.

Table R3.7. RoboTwin policy-training GPU-hours at 40k steps.

PolicyGPU-hours for 40k stepsIncrease over action-onlyRelative increase
Action-only217.51——
PLATO supervision-only249.29+31.78+14.61%
PLATO context-only221.71+4.20+1.93%

The shared tokenizer has a separate one-time pretraining cost of approximately 834 GPU-hours. Supervision-only deployment requires no additional tokenizer computation; the context variant encodes the current observations once per action-chunk query.

[Q2: Training/evaluation separation]

R8: Tokenizer training uses RoboTwin, LIBERO, and RoboCasa training data, together with the other sources listed in Table 7. Policy-evaluation trajectories and observations are excluded. All four real-world task datasets are also excluded; supplementary videos illustrate strong depth and pointmap reconstruction on these unseen task settings.

[Q2: Teacher-supervision controls]

R9: Continuous pre-VQ and Depth–DINO provide teacher-informed controls that supervise similar types of geometric and semantic information through different target representations. The loss-weight sweep shows that PLATO achieves higher reported success at both tested nonzero weights, supporting the benefits of its discrete supervision interface over these alternatives.

[Q4: Context-only baseline and benefits beyond viewpoint shifts]

R10: In LIBERO-Plus, we already ablated context during post-training without auxiliary PLATO supervision. The real-world ablation adds context alongside PLATO supervision and improves viewpoint robustness. We further evaluate context-only training on RoboTwin to test benefits beyond viewpoint shifts.

We compare action-only, supervision-only, and context-only policies using the same Qwen initialization, 40k policy steps, action losses, and evaluation protocol.

Table R3.8. RoboTwin action-only, supervision-only, and context-only comparison.

VariantCurrent contextFuture-state auxiliary weightClean (%)Randomized (%)Combined (%)
Action-onlyNo069.8070.6270.21
Supervision-onlyNo0.173.4873.0273.25
Context-onlyYes071.2071.0871.14

Context-only improves combined success from 70.21% to 71.14% (+0.93 points), showing gains of +10.5 points on place A to B's left and +5.5 points on stacking two blocks. These results support context benefits beyond viewpoint shifts, including tasks requiring spatial-relation reasoning and precise object placement.

Reviewer_gQR9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PLATO learns a compact scene representation from multi-view RGB using a frozen VGGT-Omega backbone, learned scene queries, and a 16,384-entry vector-quantized codebook producing 128 state tokens. Geometry/semantic reconstruction, forward-inverse dynamics, and augmentation consistency train the tokenizer. The frozen representation is then used either as autoregressive future-state supervision, with attention isolation from the action branches, or as current-state context. Supervision alone adds no tokenizer or future-token generation at deployment. Experiments cover three simulation benchmarks, four real-world tasks, and three policy initializations. Reported real-world task-average gains are 11.67 percentage points under nominal conditions and 25.41 under task-specific perturbations; current-state context additionally improves shifted-view flower insertion from 8/15 to 14/15 successes. Frozen-feature probes show improved geometric decodability, including against simulator ground truth on RoboCasa.

Critical strengths
  1. A useful shared interface with convincing practical benefits. The same compact state vocabulary supports both current-scene conditioning and future-state supervision within an existing VLM. Attention isolation and removal of the auxiliary branch at deployment make supervision particularly attractive in practice. The matched comparisons show gains across distinct initializations and training stages (Tables 1 and 12), with especially substantial improvements for real-world cable-tie threading (19/30 to 29/30 nominal; 9/20 to 17/20 randomized, Table 13). This breadth and the absence of added deployment computation for supervision-only policies are the main reasons I lean toward acceptance.

  2. The representation analysis adds evidence beyond task success. Matched frozen-VLM probes use episode-disjoint splits and show consistent improvement across all six readout levels and all three benchmarks (Table 8). Importantly, RoboCasa uses simulator ground truth rather than only the tokenizer teacher's pseudo-labels. The appendix also reports original-feature cross-view silhouette scores, rather than relying exclusively on t-SNE. These analyses support the narrower conclusion that the training changes accessible geometric information and view dependence, even if they do not establish the full causal explanation for improved control.

Critical weaknesses
  1. The alternative-target experiments support the complete PLATO recipe more strongly than they isolate the contributions of 3D information or discretization. Table 10 changes several factors together: Cosmos receives one future view and 256 slots, while PLATO receives three views and 128 slots; continuous pre-VQ regression uses a different loss and outer weight; and Depth-DINO uses only eight VLM query slots with separate resampling heads. Different loss units mean the numerical weights alone cannot establish equal optimization pressure. Likewise, Figure 5 probes high-dimensional pre-MLP features (128 x 1024), rather than directly evaluating the discrete bottleneck or the 128 x 16 context representation, and compares multi-layer aggregation against final-layer backbone features. These are transparently described, useful comparisons, but they leave uncertainty about which part of the proposed interface is responsible for the gains. This limits the methodological conclusion of Section 4.4.2.

  2. The uncertainty and generalization scope of several central claims remain insufficiently characterized. Table 13 helpfully supplies rollout counts, but each real-world setting has only 15-30 trials, and policy results are not accompanied by variation across independent training seeds or confidence intervals. Consequently, the smaller gains (e.g., +0.60 points for dynamics in Table 4(b), +0.58 on randomized RoboTwin in Table 1, and +1.06 overall for context in Table 2) are difficult to distinguish from variability. Moreover, the tokenizer corpus includes all three simulation benchmark families (Table 6); exact exclusions of evaluation episodes/assets are not specified, so these experiments should not be interpreted as transfer to unseen benchmark domains. The four excluded real-world tasks provide valuable complementary evidence, but the strongest context result uses one task and one shifted camera configuration. These limitations temper the strength of the broad robustness claim, without negating the larger matched gains.

Questions for authors
  1. Clarification of data separation: For RoboTwin, LIBERO/Plus, and RoboCasa-Human50, which tokenizer-training episodes, scenes, assets, and task configurations are shared with or excluded from policy evaluation? In particular, were evaluation trajectories or their observations ever used to train the tokenizer? Please distinguish ordinary in-domain representation pretraining from genuinely held-out evaluation.

  2. Clarification of uncertainty: How many independently trained policy seeds underlie Tables 1, 2, and 4, and how many evaluation rollouts are used for the simulation results? Can you provide seed-level results/uncertainty for the small dynamics and context gains, and explain whether real-world initial conditions were matched or randomized across policies? The counts in Table 13 are already useful; the missing issue is uncertainty and reproducibility across training runs.

  3. Evidence for the interface explanation: Are view-matched Cosmos results and sensitivity to auxiliary-loss weighting available? A comparison using the same three views and comparable prediction capacity would materially strengthen the claim that geometry/discrete targets, rather than target coverage or optimization, explain the advantage. Please also clarify how much of Figure 5's geometric/semantic fidelity survives in the actual quantized representation.

  4. Clarification of the cross-view inference: Appendix A.2.2 uses 82 progress-matched pairs from only two episodes that may differ in layout. Can you distinguish reduced camera dependence from episode/layout effects or reduced task discrimination, for example with same-scene cross-view pairs and a check of task/state separability? Table 9 already provides quantitative original-space metrics; the concern is the sampling/control of the comparison, not an absence of metrics.

Overall, I lean toward weak acceptance because the shared interface and larger matched control gains constitute a useful empirical contribution. My support is for the complete method; the stronger causal and broad generalization interpretations need qualification. Answers establishing clean evaluation separation and stability would strengthen this assessment.

5 comments

Thank you for your careful assessment and for recognizing the practical value of PLATO’s shared interface. We particularly appreciate your distinction between evidence for the complete method and evidence isolating individual design choices. Your comments helped us identify where more closely matched comparisons and clearer evaluation controls were needed. Following your suggestions, we have added target comparisons, uncertainty analyses, and same-scene cross-view controls to address the specific questions you raised.

[W1/Q3: View- and token-matched Cosmos controls]

R1: We add a Cosmos control using all three future views and the same DI16×16 tokenizer. Each view is resized to 96 × 112, producing a 6 × 7 code grid and 3 × 42 = 126 tokens, closely matching PLATO's 128-token budget.

Table R2.1. View- and token-matched Cosmos comparison.

Future targetFuture viewsPredicted tokensClean (%)Randomized (%)Combined (%)
None, action-only——69.8070.6270.21
Cosmos, original125673.7270.2071.96
Cosmos, compact three-view312671.8471.8471.84
PLATO discrete312873.4873.0273.25

PLATO exceeds the compact three-view control by 1.41 percentage points. Matching view coverage and closely matching the token budget provides stronger evidence for the benefit of PLATO's 3D-aware future-state supervision.

[W1/Q3: Auxiliary-loss sensitivity]

R2: The original outer weights were chosen to bring the initial auxiliary-loss magnitudes to a comparable scale. Since similar loss magnitudes do not guarantee similar gradient contributions, we add a weight sweep while keeping each objective's internal formulation fixed. Although the best-performing weight varies across objectives, PLATO achieves the highest reported success at both tested nonzero weights.

Entries are combined success (%). The λ = 0 entries share one action-only reference.

Table R2.2. Auxiliary-loss weight sweep: combined success (%).

Auxiliary targetλ = 0λ = 0.1λ = 1
PLATO discrete70.2173.2572.74
Continuous pre-VQ70.2172.7370.17
Depth–DINO70.2166.5270.89

[W1/Q3: Actual quantized representation]

R3: Your question highlights an important distinction between the informativeness of the pre-MLP features and the information retained by the actual discrete representation. To examine this directly, we evaluate features recovered from the 128 discrete token IDs using the frozen codebook and recovery network.

Table R2.3. Representation fidelity before and after quantization.

Tokenizer readoutShapeDepth AbsRel ↓Instruction accuracy (%) ↑
Pre-MLP, Figure 5128 × 10240.03368 ± 0.0006498.77 ± 0.40
Recovered post-VQ128 × 10240.03208 ± 0.0008296.80 ± 0.26

The recovered representation largely preserves the geometric and semantic fidelity shown in Figure 5, with comparable depth accuracy and 96.80% instruction-matching accuracy after quantization.

[W2/Q1: Training/evaluation overlap and the scope of generalization]

R4: The tokenizer uses RoboTwin, LIBERO, and RoboCasa training data, along with the other sources listed in Table 7. Policy-evaluation trajectories and their observations are excluded from tokenizer training. All four real-world task datasets are also excluded; the supplementary reconstruction videos illustrate the tokenizer's transfer to these previously unseen task settings.

[W2/Q2: Additional policy-training seeds]

R5: The submitted results in Tables 1, 2, and 4 use one policy-training seed per condition. With limited training resources, we focus additional runs on RoboTwin to examine the smaller aggregate gain you highlighted, adding downstream training seeds 10241 and 10242.

Table R2.4. RoboTwin policy-training seed results.

Training seedAction-only: Clean / Randomized / Combined (%)PLATO: Clean / Randomized / Combined (%)
1024088.08 / 88.60 / 88.3489.72 / 89.18 / 89.45
1024188.72 / 88.86 / 88.7989.80 / 89.08 / 89.44
1024289.76 / 89.02 / 89.3990.36 / 89.68 / 90.02
Three-seed mean ± SD88.85 ± 0.85 / 88.83 ± 0.21 / 88.84 ± 0.5389.96 ± 0.35 / 89.31 ± 0.32 / 89.64 ± 0.33

The seemingly modest RoboTwin gain is an average over 50 tasks, many of which already have little room for improvement: 19 action-only tasks and 22 PLATO tasks reach at least 95% success in the original clean evaluation. This average therefore combines near-saturated tasks with more challenging ones, making the larger gains on lower-baseline tasks less visible. On these tasks, PLATO-initialized policies show consistent improvements across all three available trained seeds, for example:

Table R2.5. Task-level success gains across training seeds.

TaskSeed10240 gain (pp)Seed10241 gain (pp)Seed10242 gain (pp)
turn switch+9+10+6
scan object+6+3+5
place bread skillet+5+5+3

These task-level results show repeatable benefits on more challenging individual tasks.

[W2/Q2: Evaluation rollouts and confidence intervals]

R6: We report rollout counts and 95% confidence intervals for the gains across the main comparisons. Simulation evaluations follow the benchmark-specific protocols, and real-world evaluations include as many trials as the available operator time permits. For the smaller context gain you highlighted, the Table 2 LIBERO-Plus comparison uses 10,030 rollouts per policy. Its +1.06-point gain has a 95% CI of [+0.20, +1.91] points, excluding zero and supporting a positive effect at the rollout level.

Table R2.6. Evaluation rollout counts and confidence intervals for gains.

ComparisonRollouts per policyScope / methods comparedGain (pp)95% CI for gain (pp)
Table 1, RoboTwin (submitted)10,00050 tasks × 100 episodes × 2 environment configurations+1.11[+0.24, +1.98]
Table 1, LIBERO-Plus10,030PLATO supervision vs. action-only+7.58[+6.60, +8.56]
Table 1, RoboCasa Human-501,200PLATO supervision vs. action-only+6.33[+2.34, +10.30]
Table 2, LIBERO-Plus context10,030Context vs. supervision-only initialization+1.06[+0.20, +1.91]
Table 4(a), prediction interfaces10,000PLATO supervision vs. action-only at 40k+3.04[+1.79, +4.29]
Table 15, real-world example15 per methodShifted-view flower: context vs. supervision+40.00[+8.16, +63.85]

Intervals use an unpaired, rollout-level Newcombe/Wilson approximation.

[W2/Q2: Matched real-world initial states]

R7: All policies are evaluated using the same set of real-world initial configurations. Corresponding trials use identical object positions and orientations and the same robot starting state. Before each trial, the scene is restored to its designated initial configuration, and we record the setups to support reproducibility.

[Q4: Same-scene cross-view control and task/state separability]

R8: Your suggestion to use same-scene pairs is particularly helpful for separating camera dependence from episode and layout differences. Following your suggestion, we evaluate the two Table 2 checkpoints on paired views of exactly the same physical scene on LIBERO and measure both task and coarse-state discrimination.

We use Bowl to plate and Two moka pots to stove, with three episodes per task and 20 time points per episode. Each of the 120 physical states is rendered at 0° and +30° external-camera azimuth. Simulator time, joint positions, and velocities remain unchanged between paired views, and wrist-camera images are pixel-identical. Both checkpoints receive the same 240 observations.

The visualization in Appendix A.2.2 shows one row per checkpoint, with points colored by camera, coarse state, and task across the columns. PLATO context produces greater overlap between camera views while preserving state and task separability. The following metrics are computed in the original feature space:

Table R2.7. Same-scene cross-view analysis in the original feature space.

Original-feature metricAction-only checkpoint+PLATO-context checkpoint
Between-task balanced accuracy100.00%100.00%
Bowl within-task state balanced accuracy97.87%97.87%
Moka within-task state balanced accuracy86.48%90.72%
Joint task × state balanced accuracy91.90%93.29%
Bowl within-task camera silhouette0.15960.1252
Moka within-task camera silhouette0.18630.1548

PLATO context reduces camera-related clustering while preserving task and coarse state discrimination.

Reviewer_6pwVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Physical Language for Action through Tokenized Observations (PLATO), a unified world-state representation for VLA. The authors first propose a world-state tokenizer, where the multi-view observations are fed into a frozen VGGT-Omega to extract the multi-layer patch features. Then 128 learnable queries cross-attend to the VGGT-Omega patch features layer by layer, followed by several self-attention blocks, and are quantized by a VQ codebook with 16,384 learned codewords into 128 discrete tokens. The selected codebook vectors are decoded by a camera-conditioned decoder to reconstruct the depth, point map and DINO features of each view. In this way, the trained tokenizer compresses the multi-view observations into a compact representation, which is then frozen and used for the VLA.

Then, during training, the VLA predicts the tokens of the future observations alongside the actions, which are supervised by the tokens obtained by tokenizing the future observations with the frozen tokenizer. Besides, the tokens obtained by the world-state tokenizer can also serve as additional context, which are fed into the VLA together with the camera observations and language instructions.

The authors conduct experiments on four real-world manipulation tasks and three simulation benchmarks (RoboTwin2.0, LIBERO-Plus and RoboCasa Human-50). The experiments and ablation studies show the effectiveness of PLATO as both the future-state supervision and the current-state context for VLA.

Critical strengths
  1. The proposed PLATO is simple and effective.

The world-state tokenizer compresses the multi-view observations into only 128 tokens. This representation can be used as both the future-state supervision and the current-state context, based on the native next-token prediction and conditioning of the VLM, without additional dense prediction branches. Besides, the future-state prediction is removed at inference time, so it brings no extra cost for deployment.

  1. The experiments are comprehensive and convincing.

The authors conduct experiments on four real-world tasks under both nominal and randomized environments, and three simulation benchmarks with large-scale pretraining. The PLATO supervision improves the average real-world success rate by 11.7 and 25.4% under nominal and randomized conditions, respectively. Besides, the representation analysis shows that PLATO makes the VLA features more 3D aware and 3D consistent across viewpoints.

Critical weaknesses
  1. The reason for adopting VGGT-Omega is not well justified. In PLATO, VGGT-Omega is used as both the feature extractor and the teacher during the training of the tokenizer. However, the authors do not explain why VGGT-Omega is chosen, and there is no ablation study on other foundation models, e.g., DINOv3 or Depth Anything 3. As a result, it is unclear whether the improvements come from the proposed tokenizer or from the strong VGGT-Omega features.

  2. The computational cost of PLATO is not reported. When PLATO is used as context, VGGT-Omega and the tokenizer need to be inferenced at every inference step, which brings significant extra computational cost. However, the inference time and computational cost are not reported in the paper. I consider that the comparison between the context variant and the baselines is unfair, since the context variant uses an additional large model at inference time.

Questions for authors
  1. Why do the authors choose VGGT-Omega as the feature extractor and the teacher of the tokenizer? What about other foundation models, e.g., DINOv3 or Depth Anything 3?

  2. What are the inference time and computational cost of PLATO when it is used as context?

3 comments

Thank you for your careful assessment and for recognizing the value of PLATO’s shared interface. Your questions about backbone choice and inference cost helped us identify two aspects that needed more direct experimental evidence. Following your suggestions, we have added alternative-backbone comparisons and measurements of context inference latency and memory usage. We address your questions below.

[W1/Q1: Why VGGT-Omega, and what about DINOv3 or Depth Anything 3?]

R1: Your suggestion to examine alternative foundation models is particularly helpful for assessing how the tokenizer depends on its input backbone. We chose VGGT-Omega as the feature extractor and teacher because it provides strong multi-view 3D features together with depth and pointmap estimates that directly support our geometric reconstruction objectives. Following your suggestion, we compare it with the two alternatives, DINOv3 and Depth Anything 3, to evaluate PLATO across different backbones and assess our choice of VGGT-Omega.

Setting. We train the student tokenizer modules from scratch on the same RoboTwin training data for 20k steps, using frozen pretrained input backbones. We vary the input backbone while keeping the VGGT reconstruction teacher and tokenizer objectives fixed. All variants share 128 scene slots, a 16-dimensional VQ bottleneck, a 16,384-entry codebook, a 1024-dimensional recovery representation, and the same reconstruction, dynamics, and consistency objectives. Each tokenizer then provides future-state supervision to a Qwen-initialized policy trained for 40k steps with auxiliary weight 0.1.

Results.

Table R1.1. RoboTwin policy performance with alternative tokenizer backbones.

Input backbone of RoboTwin-only tokenizerTokenizer stepsPolicy stepsClean (%)Randomized (%)Combined (%)
VGGT-Omega20k40k73.7472.5873.16
DINOv320k40k73.2672.3272.79
Depth Anything 320k40k72.2270.2871.25
Action-only reference, no auxiliary target—40k69.8070.6270.21

The DINOv3- and Depth Anything 3-based tokenizers improve combined success over action-only by 2.58 and 1.04 percentage points, respectively. These gains show that PLATO's tokenizer architecture and supervision protocol remain effective across alternative input backbones under shared teacher supervision. VGGT-Omega achieves the highest combined success at 73.16%, supporting its selection as the default backbone.

[W2/Q2: Inference time and computational overhead of PLATO context]

R2: You raise an important practical point: the benefits of adding PLATO context should be assessed alongside the cost of running an additional model at inference. We therefore report both prediction latency and peak GPU memory for the Qwen3.5-2B RoboTwin configuration. The current-state representation is computed once per action-chunk query, in parallel with RGB encoding, and reused throughout that query's flow-denoising steps. We benchmark the action-only Qwen3.5-2B VLA and its PLATO-context variant independently on the same RTX5090, using the 40k-step RoboTwin checkpoints, batch size 1, 10 Euler denoising steps, and a 32-step action chunk. Each row reports 100 predictions after compilation and warm-up, with CUDA synchronization. Timing covers GPU-ready model inference, including RGB and context encoding.

Table R1.2. Prediction latency and peak GPU memory on RTX5090.

Inference configurationMean prediction (ms)Median prediction (ms)P95 prediction (ms)Peak allocated memory (GiB)
Qwen3.5-2B VLA (action-only)56.8656.7157.185.25
+ PLATO context78.6478.4979.0210.51

PLATO context offers an accuracy–cost trade-off: it adds 21.78 ms per prediction for the reported gains in policy robustness and performance, while keeping mean model-prediction latency at 78.64 ms, below 100 ms.

Dear Reviewer 6pwV:

Thank you again for your thoughtful feedback! We really appreciate your recognition of PLATO’s simple design and comprehensive experiments.

Following your suggestions, we’ve added alternative-backbone comparisons and reported the latency and memory overhead of PLATO context.

We hope these updates address your concerns. Please don’t hesitate to let us know if you have any further questions. We’d greatly appreciate any feedback!

Best,

Paper42832 Authors

Dear Reviewer 6pwV:

With just one day remaining in the discussion period, we would be very grateful to hear your thoughts on our rebuttal. Any further questions, suggestions, or feedback on our responses would be most welcome. We would also be very grateful if you could let us know whether our responses have addressed your remaining concerns, or if there is anything further you would like us to clarify.

Thank you again for your time, thoughtful review, and valuable suggestions. We very much look forward to hearing from you!

Best,

Paper42832 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank the reviewers for their thoughtful comments. We have expanded our experiments and clarified the evaluation protocol. The main additions are:

  1. Controlled comparisons with 2D and continuous baselines. We add a three-view Cosmos baseline with 126 tokens, closely matching PLATO's 128-token budget, and compare PLATO, continuous pre-VQ, and Depth–DINO supervision across multiple loss weights.
  2. Information preserved after quantization. Features recovered from the 128 discrete token IDs achieve depth AbsRel of 0.03208 ± 0.00082 and instruction-matching accuracy of 96.80 ± 0.26%, compared with 0.03368 ± 0.00064 and 98.77 ± 0.40% before the quantization MLP. These results show that the discrete representation retains strong geometric and semantic fidelity.
  3. Training and deployment costs. On an RTX5090, adding context increases mean prediction latency from 56.86 ms to 78.64 ms. Supervision-only deployment requires no additional tokenizer computation. We also clarify PLATO's additional training costs.
  4. Alternative foundation models. We train RoboTwin-only tokenizers using VGGT-Omega, DINOv3, and Depth Anything 3 backbones. VGGT-Omega achieves the highest downstream success in this comparison at 73.16%. The DINOv3- and Depth Anything 3-based tokenizers achieve 72.79% and 71.25% success, respectively, compared with 70.21% for action-only. These results support the benefits of PLATO's tokenizer architecture and supervision objectives across alternative backbones.
  5. Additional training seeds and evaluation statistics. We add two RoboTwin policy-training seeds and report seed-level results, rollout counts, and uncertainty estimates for simulation and real-world evaluations.
  6. Controlled cross-view VLM feature analysis. We render identical physical states in LIBERO from two camera viewpoints and evaluate both checkpoints on the same observations. PLATO context reduces camera-related clustering while preserving task and coarse state discrimination.
  7. Context-only benefits beyond viewpoint shifts. We add a context-only baseline under the same 40k-step RoboTwin training recipe. It achieves 71.14% success, compared with 70.21% for action-only, providing additional evidence of context benefits beyond the viewpoint-shift setting.
  8. Training data and real-world protocol. We clarify that tokenizer training uses simulation benchmark training data and excludes evaluation trajectories and observations. All four real-world task datasets are excluded from tokenizer training. We also clarify that corresponding real-world trials use identical initial object poses and robot starting states across policies, with recorded setups and trial accounting.

The revised manuscript includes the post-quantization results in Section 4.4.1, the updated continuous baseline in Section 4.4.2, the backbone ablation in Section 4.4.3, the same-scene cross-view analysis in Appendix A.2.2, and training and deployment costs in Appendix A.7. Revisions are highlighted in blue. We address the individual concerns below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.