- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Complementary Predictive Representations (CPR), a two-stage framework for injecting predictive geometric information into VLA policies and world action models.
In pretraining, dense features from a frozen VGGT are extracted from multi-view video clips and compressed by a hierarchical query-based tokenizer, first across views and then over time. A causal Slot-Temporal Mixer produces a predictive geometric representation G, trained with feature reconstruction, masked slot prediction, and latent transition prediction. A motor representation Z is then extracted from latent transitions ΔG and supervised with ground-truth action chunks.
In post-training, the frozen model provides G and Z, computed from current and future frames, as privileged targets. The student policy regresses G via added geometry queries and aligns an intermediate layer to Z, so no extra input is required at inference.
Experiments cover LIBERO, LIBERO-Plus, SimplerEnv-WidowX, four real-world tasks, three VLA backbones, and FastWAM.
Critical strengths
- The method adds no inference-time cost and is architecture-agnostic, and the evaluation is broad and includes extensive ablations, probing analyses, and real-world OOD perturbations.
Critical weaknesses
- The high-level idea is established in published work, and the paper's framing overstates the contribution. Prior work already uses predictive geometry for action generation, with future information serving only as privileged training-time supervision:
- FLARE [1] pretrains an action-aware observation embedding and aligns additional policy tokens with future observation embeddings at no inference cost. This is structurally close to CPR's motor-representation alignment.
- GeoPredict [2] uses predictive future 3D geometry purely as training-time supervision.
- GAP [3] predicts future latents of a frozen 3D geometric foundation model as an auxiliary target for manipulation.
- Xu et al. [4] jointly predict compressed future latents with actions and drop the future branch at inference.
- Spatial Forcing [5] aligns VLA features with VGGT representations.
Of these, GAP, GeoPredict, FLARE and Xu et al. are not cited. Spatial Forcing is cited but categorized as "prediction-centric", although it aligns current-frame features; it is neither discussed as a direct precedent nor compared.
Against this background, presenting "how predictive geometry should be represented for action generation" as an open question that the paper identifies and answers overstates the contribution. What remains distinctive is the decomposition into complementary G and Z targets with separate transfer pathways. However, the direct evidence for this decomposition is limited: unified vs. complementary in Table 5 is 67.7 vs. 71.9, a difference of four episodes out of 96.
- Design complexity, with unclear attribution of which components matter:
-
The pipeline stacks many components: a hierarchical view-then-time query tokenizer; an anchor-frame-plus-4-frame slot scheme borrowed from video VAEs; a causal slot mixer; three pretraining objectives with a variance-covariance regularizer; a four-stage curriculum; a cosine-decayed alignment weight; and a specific alignment layer. Several of these choices (the tokenizer hierarchy, slot grouping, causal masking, and the curriculum) are not ablated at all.
-
The others are supported only by differences of a few points on a 24-trial-per-task benchmark, and the alignment layer and λ_geo are selected based on results on that same benchmark (Tables 11 and 13).
A minimal variant would clarify whether the added structure is necessary.
[1] FLARE: Robot Learning with Implicit World Modeling. CoRL 2025.
[2] GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation. CVPR 2026.
[3] Action–Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation (GAP). CVPR 2026.
[4] Diffusion-Based Imaginative Coordination for Bimanual Manipulation (Xu et al.). ICCV 2025.
[5] Spatial Forcing: Implicit Spatial Representation Alignment for Vision-Language-Action Model. ICLR 2026.
Questions for authors
-
The gain attributable to geometry is not significant. In Table 12, the VGGT teacher achieves 71.9% versus 69.8% for DINO, which has no explicit 3D structure. On WidowX (24 trials per task, 96 per model), this corresponds to 69 versus 67 successes, a difference of two episodes. The per-task results are also mixed: VGGT wins on Stack Cube, while DINO wins on Put Carrot and Put Eggplant. Consequently, most of the 10.4-point improvement over the baseline is obtained without a geometric teacher, and the evidence does not support attributing the gains specifically to geometry, which is the central claim of the title and narrative.
-
The motor representation is derived from ΔG alone, which is questionable. Z is computed by cross-attending only to ΔG_k = G_{k+1} − G_k. Inverse dynamics is naturally a function of both the current state and the transition, i.e., a = f(s_t, s_{t+1}), or equivalently f(G_k, ΔG_k). The same latent change can correspond to different actions depending on the absolute configuration (e.g., gripper pose, object position, gripper open/close state). A difference vector alone is sufficient only under strong assumptions about the latent space, such as approximate translation invariance. Table 10 compares only G versus ΔG (68.8 vs. 71.9, which is itself within noise). The natural G + ΔG variant is neither tested nor discussed.
-
Missing key related work. plz refer to the Critical Weaknesses
-
VGGT is applied independently to each frame-view pair (Sec. 3.1, App. C.1), which forgoes its multi-view joint reasoning. Was this choice ablated?
-
Fig. 8 says "Make Sandwich" while the text says "Make a Burger".
-
The "Dense Geometry Feature" illustration in Fig. 1(c) should be revised
-
The supplementary real-robot videos are a welcome addition. Accompanying them with a short textual description would make them more informative, for example: the difficulty and success criteria of each task, typical failure modes of the baselines and of CPR, and whether the shown rollouts are randomly selected or representative.
13 comments
Rebuttal by Authors (2/7)
2. Further Analysis for Complementary Predictive Representation
W-2:What remains distinctive is the decomposition into complementary G and Z targets with separate transfer pathways. However, the direct evidence for this decomposition is limited: unified vs. complementary in Table 5 is 67.7 vs. 71.9, a difference of four episodes out of 96.
We thank the reviewer for this comment. The WidowX results in Tab.5 are not based on a single evaluation of 96 episodes. Specifically, for each model, we evaluate four tasks with 24 episodes each across five random seeds (0, 1, 2, 3, and 2026). Reported success rates are averaged across the five runs. Thus, the 4.2 percentage-point improvement reflects the difference between repeated evaluation averages, rather than a single 96-episode result.
Beyond downstream policy performance, Fig.6 provides direct evidence that G and Z capture distinct functional information. We conduct zero-shot depth and action probing on LIBERO-90. Although Unified captures both geometric and action-aware information, it underperforms separately learned PGR and MR on their respective geometry- and action-related probing tasks.
To further assess the benefits of assigning predictive geometric and action-aware information to complementary representations, we extend the evaluation beyond the experiments reported in Tab.5 and Fig. 6 of the main paper. Specifically, we train on LIBERO and evaluate zero-shot on LIBERO-PLUS, which comprises 10,030 trials. On LIBERO, CPR improves the average success rate from 97.6% to 98.6%. More importantly, on the more challenging LIBERO-PLUS benchmark, CPR achieves 81.7%, compared with 77.7% for Unified. Notably, Camera and Noise robustness improve by 7.7 and 6.1 percentage points, respectively. CPR also improves performance across all seven evaluation dimensions, further supporting the generalization benefits of complementary geometric and motor representations.
| LIBERO | Object | Spatial | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Unified | 99.0 | 97.8 | 97.0 | 96.8 | 97.6 |
| Ours | 100.0 | 99.0 | 98.2 | 97.0 | 98.6 |
| LIBERO-PLUS | Camera | Robot | Language | Light | Background | Noise | Layout | Total |
|---|---|---|---|---|---|---|---|---|
| Unified | 69.8 | 56.6 | 85.9 | 84.2 | 88.3 | 89.5 | 74.4 | 77.7 |
| Ours | 77.5 | 60.8 | 88.6 | 86.1 | 89.1 | 95.6 | 77.2 | 81.7 |
Taken together, these results support our motivation that predictive geometric modeling benefits from preserving coherent scene structure and its evolution, whereas action generation benefits from emphasizing action-relevant geometric variations. CPR explicitly captures these complementary properties through separate G and Z representations and their role-specific transfer pathways.
We will incorporate these additional results and clarifications into the revised manuscript (Line.1395->1442).
Rebuttal by Authors (3/7)
3. More Components Ablations(1/2)
W-3:The pipeline stacks many components: a hierarchical view-then-time query tokenizer; an anchor-frame-plus-4-frame slot scheme borrowed from video VAEs; a causal slot mixer; three pretraining objectives with a variance-covariance regularizer; a four-stage curriculum; a cosine-decayed alignment weight; and a specific alignment layer. Several of these choices (the tokenizer hierarchy, slot grouping, causal masking, and the curriculum) are not ablated at all.
W-4: The alignment layer and λ_geo are selected based on results on that same benchmark (Tables 11 and 13).
W-5: A minimal variant would clarify whether the added structure is necessary.
We thank the reviewer for this constructive comment. CPR is built around a central motivation: organizing predictive geometric information while separating structured geometric states from action-aware transitions. We have conducted extensive ablations of its core components, including complementary PGR/MR pathways (Tab.4–5), pretraining objectives (Tab.9), motor representation inputs (Tab.10), alignment layers (Tab.11), and geometry-guidance weights (Tab.13). To further address the reviewer's concerns, we conduct additional controlled ablations on tokenizer design, temporal grouping, causal masking, and training curriculum.
Hierarchical View-then-Time Tokenization. We compare our hierarchical tokenizer with Flat Joint Tokenization, which jointly compresses multi-view and temporal features. Our design achieves 71.9% average WidowX success, compared with 69.8% for flat tokenization. We further evaluate both variants through zero-shot generation using FastWAM on LIBERO-90. These results support explicitly separating view-level and temporal aggregation. We will incorporate these additional results and clarifications into the revised manuscript (Line.1443->1449).
| WidowX | Put Spoon | Put Carrot | Stack Cube | Put Eggplant | Avg. |
|---|---|---|---|---|---|
| Flat Joint | 87.5 | 66.7 | 70.8 | 54.3 | 69.8 |
| Ours | 95.8 | 70.8 | 70.8 | 50.0 | 71.9 |
| LIBERO-90 | PSNR↑ | SSIM↑ | LPIPS↓ | Action L1↓ |
|---|---|---|---|---|
| Flat Joint | 21.08 | 0.795 | 0.193 | 0.091 |
| Ours | 22.91 | 0.803 | 0.186 | 0.075 |
Temporal Grouping. We investigate temporal grouping granularity by comparing group sizes of 2, 4, and 8 while keeping the anchor-frame configuration fixed.
| Group Size | Put Spoon | Put Carrot | Stack Cube | Put Eggplant | Avg. |
|---|---|---|---|---|---|
| 2 | 91.7 | 75.0 | 70.8 | 54.2 | 72.9 |
| 4 | 95.8 | 70.8 | 70.8 | 50.0 | 71.9 |
| 8 | 91.7 | 66.7 | 70.8 | 50.0 | 69.8 |
Although group size 2 achieves a slightly higher average success rate, the adopted 4-frame configuration offers a favorable trade-off between performance and temporal grouping granularity(0.36s vs.0.41s). Larger groups (8) lead to noticeable performance degradation. We will incorporate these additional results and clarifications into the revised manuscript (Line.1451->1455).
Causal Slot-Temporal Mixer. We replace causal attention with bidirectional attention while keeping the remaining architecture unchanged and conduct the experiments in WidowX and LBERO-90. Causal attention improves average WidowX success from 68.8% to 71.9%. Consistently, FastWAM's zero-shot generation also improves compared with bidirectional attention.
| WidowX | Put Spoon | Put Carrot | Stack Cube | Put Eggplant | Avg. |
|---|---|---|---|---|---|
| Bidirectional | 91.7 | 70.8 | 66.7 | 45.8 | 68.8 |
| Causal | 95.8 | 70.8 | 70.8 | 50.0 | 71.9 |
| LBERO-90 | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Action L1 ↓ |
|---|---|---|---|---|
| Bidirectional | 20.95 | 0.787 | 0.201 | 0.096 |
| Causal | 22.91 | 0.803 | 0.186 | 0.075 |
These results support causal temporal modeling for constructing history-conditioned predictive geometric states. We will incorporate these additional results and clarifications into the revised manuscript (Line.1457->1491).
Variance-Covariance Regularizer. We investigate the role of the variance-covariance regularizer while retaining all three pretraining objectives. Removing this regularizer leads to representation collapse by 28.6k training steps, characterized by a progressive decrease in feature variance toward zero. In contrast, the model trained with the regularizer maintains a healthy level of feature variance at the same training stage.
Rebuttal by Authors (1/7)
1. More Compaired with Related worksWe sincerely thank the reviewer for identifying these important related works. Our central motivation, as introduced in Lines 81–90 in main paper, is not simply to investigate how future geometric information can supervise action generation. Rather, we aim to address the different representational requirements of predictive geometric modeling and action generation, and how these requirements can be jointly accommodated through complementary representations.
The cited works explore several closely related directions, but their representation construction and transfer mechanisms differ from ours:
Therefore, compared with above methods, our contribution lies in explicitly distinguishing two complementary representational objectives: preserving coherent scene geometry and its temporal evolution, and extracting action-aware information from geometric transitions.
We will expand the discussion of related work and refine the taxonomy and presentation of our contributions in the revised manuscript and Appendix to improve clarity and reproducibility (Line.140–>147 and 2050–>2077).