- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PLATO, a unified world-state representation for Vision-Language-Action (VLA) models. It compresses multi-view visual observations into 128 discrete state tokens using geometric reconstruction, semantic supervision, and forward/inverse dynamics objectives.
The learned representation is incorporated into VLAs in two ways: (1) as an auxiliary target for autoregressive future-state prediction during training, and (2) as additional current-state context for action generation. The authors evaluate the framework on RoboTwin2.0, LIBERO-Plus, RoboCasa Human-50, and four real-world manipulation tasks. They also investigate geometric decodability and cross-view feature consistency.
The results show promising improvements in certain real-world precision-manipulation and viewpoint-shift settings, although gains on several simulation benchmarks are modest.
Critical strengths
1. A unified interface for VLA perception and prediction.
The most compelling contribution is the use of a shared world-state representation for both current-state conditioning and future-state supervision. This provides a coherent alternative to incorporating multiple heterogeneous geometric and semantic prediction targets. In particular, the supervision-only approach does not require running the tokenizer during policy inference, which is a practical advantage.
2. Promising real-world results and representation analysis.
The paper evaluates four real-world manipulation tasks on three robotic platforms. PLATO supervision improves average success from 73.75% to 85.42% under nominal conditions and from 47.92% to 73.33% under task-specific perturbations. The cable-tie threading results are particularly encouraging. The geometric probing experiments further provide evidence that the auxiliary supervision changes the information encoded by the VLM, rather than merely adding an additional input modality.
Critical weaknesses
1. Insufficient isolation of the proposed design choices.
The action-only comparisons support the effectiveness of the overall PLATO training approach. However, the alternative prediction-target comparisons in Table 4 vary multiple factors, including view coverage, prediction length, loss formulation, and auxiliary weight. This makes it difficult to isolate the respective benefits of 3D geometry, discretization, and the prediction interface.
Furthermore, Figure 5 evaluates high-dimensional pre-MLP features rather than the recovered quantized representation. Although the tokenizer's reconstruction and dynamics objectives operate through the discrete bottleneck, direct evidence of information preservation after quantization remains limited.
2. Limited characterization of performance reliability and generalization.
The improvements vary substantially across benchmarks. Gains are relatively small on RoboTwin2.0 (+1.64/+0.58 percentage points), while LIBERO-Plus and RoboCasa show more meaningful improvements. Additional current-state context improves overall LIBERO-Plus performance by only +1.06 points, although it substantially improves viewpoint robustness in real-world flower insertion.
The real-world results are encouraging, but several comparisons involve limited trial counts. Without uncertainty estimates or repeated policy-training runs, it is difficult to assess the consistency and broader generality of the reported improvements.
Questions for authors
-
Statistical reliability: Could the authors provide uncertainty estimates or repeated-seed results for the main improvements, particularly the small RoboTwin2.0 gains and the real-world flower-insertion viewpoint-shift results? Also, is the reported 30-minute cable-tie experiment identical to the 29/30 nominal evaluation, and how was the 1–2 mm alignment requirement assessed?
-
Fairness and data separation: Can the authors clarify the additional training compute required for PLATO and the overlap between tokenizer training data and downstream evaluation benchmarks? Are evaluation episodes completely excluded from tokenizer training? How much of the gain remains when controlling for additional teacher supervision or training resources?
-
Contribution of the discrete representation: Can the authors provide a more controlled comparison between discrete PLATO targets and continuous/2D alternatives, particularly with matched view coverage and auxiliary training budgets? Also, how much geometric and semantic information is preserved in the quantized tokens, rather than only in the pre-quantization features evaluated in Figure 5?
-
Complementarity of supervision and context: Can the authors provide a context-only baseline or otherwise clarify whether jointly using context and supervision provides benefits beyond independently adding either component? Given the relatively small overall LIBERO-Plus context gain, what evidence supports broader applicability beyond viewpoint-shift scenarios?
7 comments
[W2/Q1: Evaluation rollouts and confidence intervals]
R5: We report rollout counts and 95% confidence intervals for the gains across the main comparisons, focusing on the two cases you highlighted. The Table 2 LIBERO-Plus context comparison uses 10,030 rollouts per policy and yields a 95% CI of [+0.20, +1.91] percentage points; the real-world flower-insertion comparison under viewpoint shift uses 15 trials per method and yields a 95% CI of [+8.16, +63.85] percentage points. Both intervals exclude zero, indicating statistically significant gains in the rollout-level analysis.
Table R3.6. Evaluation rollout counts and confidence intervals for gains.
| Comparison | Rollouts per policy | Scope / methods compared | Gain (pp) | 95% CI for gain (pp) |
|---|---|---|---|---|
| Table 1, RoboTwin (submitted) | 10,000 | 50 tasks × 100 episodes × 2 environment configurations | +1.11 | [+0.24, +1.98] |
| Table 1, LIBERO-Plus | 10,030 | PLATO supervision vs. action-only | +7.58 | [+6.60, +8.56] |
| Table 1, RoboCasa Human-50 | 1,200 | PLATO supervision vs. action-only | +6.33 | [+2.34, +10.30] |
| Table 2, LIBERO-Plus context | 10,030 | Context vs. supervision-only initialization | +1.06 | [+0.20, +1.91] |
| Table 4(a), prediction interfaces | 10,000 | PLATO supervision vs. action-only at 40k | +3.04 | [+1.79, +4.29] |
| Table 15, real-world example | 15 per method | Shifted-view flower: context vs. supervision | +40.00 | [+8.16, +63.85] |
Intervals use an unpaired, rollout-level Newcombe/Wilson approximation.
[W2/Q1: Cable-tie task and precision description]
R6: The nominal cable-tie result is 29 successful insertions in 30 trials during the continuous 30-minute run. The 1–2 mm description refers to the cable-tie hole diameter measured with a ruler and specifies the task's physical alignment constraint.
[Q2: Additional training costs]
R7: We quantify the RoboTwin policy-training cost for 40k steps on eight A800 GPUs, with a global batch size of 128.
Table R3.7. RoboTwin policy-training GPU-hours at 40k steps.
| Policy | GPU-hours for 40k steps | Increase over action-only | Relative increase |
|---|---|---|---|
| Action-only | 217.51 | — | — |
| PLATO supervision-only | 249.29 | +31.78 | +14.61% |
| PLATO context-only | 221.71 | +4.20 | +1.93% |
The shared tokenizer has a separate one-time pretraining cost of approximately 834 GPU-hours. Supervision-only deployment requires no additional tokenizer computation; the context variant encodes the current observations once per action-chunk query.
[Q2: Training/evaluation separation]
R8: Tokenizer training uses RoboTwin, LIBERO, and RoboCasa training data, together with the other sources listed in Table 7. Policy-evaluation trajectories and observations are excluded. All four real-world task datasets are also excluded; supplementary videos illustrate strong depth and pointmap reconstruction on these unseen task settings.
[Q2: Teacher-supervision controls]
R9: Continuous pre-VQ and Depth–DINO provide teacher-informed controls that supervise similar types of geometric and semantic information through different target representations. The loss-weight sweep shows that PLATO achieves higher reported success at both tested nonzero weights, supporting the benefits of its discrete supervision interface over these alternatives.
[Q4: Context-only baseline and benefits beyond viewpoint shifts]
R10: In LIBERO-Plus, we already ablated context during post-training without auxiliary PLATO supervision. The real-world ablation adds context alongside PLATO supervision and improves viewpoint robustness. We further evaluate context-only training on RoboTwin to test benefits beyond viewpoint shifts.
We compare action-only, supervision-only, and context-only policies using the same Qwen initialization, 40k policy steps, action losses, and evaluation protocol.
Table R3.8. RoboTwin action-only, supervision-only, and context-only comparison.
| Variant | Current context | Future-state auxiliary weight | Clean (%) | Randomized (%) | Combined (%) |
|---|---|---|---|---|---|
| Action-only | No | 0 | 69.80 | 70.62 | 70.21 |
| Supervision-only | No | 0.1 | 73.48 | 73.02 | 73.25 |
| Context-only | Yes | 0 | 71.20 | 71.08 | 71.14 |
Context-only improves combined success from 70.21% to 71.14% (+0.93 points), showing gains of +10.5 points on place A to B's left and +5.5 points on stacking two blocks. These results support context benefits beyond viewpoint shifts, including tasks requiring spatial-relation reasoning and precise object placement.
Thank you for your careful assessment of PLATO’s shared perception/prediction interface and its deployment advantages. Your questions about controlled comparisons, statistical reliability, and the standalone contribution of context helped us focus the revision on the evidence needed to assess the method. In response, we have added more closely matched representation comparisons, additional downstream training seeds, rollout-level uncertainty estimates, and a context-only baseline. We address your questions below.
[W1/Q3: View- and token-matched Cosmos controls]
R1: We add a three-view Cosmos DI16×16 control with 126 tokens, closely matching PLATO's 128 tokens. Each future view is resized to 96 × 112, producing a 6 × 7 code grid. All policies use the same 40k-step training recipe and action objective.
Table R3.1. View- and token-matched Cosmos comparison.
PLATO improves combined success over the three-view Cosmos control by 1.41 percentage points, providing stronger evidence for the benefit of its 3D-aware future-state supervision.
[W1/Q3: Auxiliary-loss sensitivity]
R2: We sweep the auxiliary-loss weight while keeping each objective's internal formulation fixed. Although the best-performing weight varies across objectives, PLATO achieves the highest reported success at both tested nonzero weights.
Table R3.2. Auxiliary-loss weight sweep: combined success (%).
[W1/Q3: Actual quantized representation]
R3: We recover features from the actual 128 discrete token IDs using the frozen codebook and recovery network, and evaluate them with the same probe splits and training budget as Figure 5.
Table R3.3. Representation fidelity before and after quantization.
The recovered representation largely preserves Figure 5's geometric and semantic fidelity, with comparable depth accuracy and 96.80% instruction-matching accuracy after quantization.
[W2/Q1: Additional policy-training seeds]
R4: The submitted results in Tables 1, 2, and 4 use one policy-training seed per condition. Your emphasis on statistical reliability is especially helpful for interpreting the smaller RoboTwin gains. To examine their stability across downstream training runs, we add two training seeds, 10241 and 10242, alongside the submitted seed.
Table R3.4. RoboTwin policy-training seed results.
Across seeds 10240, 10241, and 10242, PLATO achieves 89.64 ± 0.33% combined success, compared with 88.84 ± 0.53% for action-only. The seemingly modest RoboTwin gain is an average over 50 tasks, many of which already have little room for improvement: 19 action-only tasks and 22 PLATO tasks reach at least 95% success in the original clean evaluation. This average therefore combines near-saturated tasks with more challenging ones, making the larger gains on lower-baseline tasks less visible. On these tasks, PLATO-initialized policies show consistent improvements across all three available trained seeds, for example:
Table R3.5. Task-level success gains across training seeds.