acceptodds
Withdrawn from ICLR 2027

From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Acton model, Predictive Geometric Representation, Motor Representation, Representation Guidances

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AzfmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Complementary Predictive Representations (CPR), a two-stage framework for injecting predictive geometric information into VLA policies and world action models.

In pretraining, dense features from a frozen VGGT are extracted from multi-view video clips and compressed by a hierarchical query-based tokenizer, first across views and then over time. A causal Slot-Temporal Mixer produces a predictive geometric representation G, trained with feature reconstruction, masked slot prediction, and latent transition prediction. A motor representation Z is then extracted from latent transitions ΔG and supervised with ground-truth action chunks.

In post-training, the frozen model provides G and Z, computed from current and future frames, as privileged targets. The student policy regresses G via added geometry queries and aligns an intermediate layer to Z, so no extra input is required at inference.

Experiments cover LIBERO, LIBERO-Plus, SimplerEnv-WidowX, four real-world tasks, three VLA backbones, and FastWAM.

Critical strengths
  • The method adds no inference-time cost and is architecture-agnostic, and the evaluation is broad and includes extensive ablations, probing analyses, and real-world OOD perturbations.
Critical weaknesses
  1. The high-level idea is established in published work, and the paper's framing overstates the contribution. Prior work already uses predictive geometry for action generation, with future information serving only as privileged training-time supervision:
  • FLARE [1] pretrains an action-aware observation embedding and aligns additional policy tokens with future observation embeddings at no inference cost. This is structurally close to CPR's motor-representation alignment.
  • GeoPredict [2] uses predictive future 3D geometry purely as training-time supervision.
  • GAP [3] predicts future latents of a frozen 3D geometric foundation model as an auxiliary target for manipulation.
  • Xu et al. [4] jointly predict compressed future latents with actions and drop the future branch at inference.
  • Spatial Forcing [5] aligns VLA features with VGGT representations.

Of these, GAP, GeoPredict, FLARE and Xu et al. are not cited. Spatial Forcing is cited but categorized as "prediction-centric", although it aligns current-frame features; it is neither discussed as a direct precedent nor compared.

Against this background, presenting "how predictive geometry should be represented for action generation" as an open question that the paper identifies and answers overstates the contribution. What remains distinctive is the decomposition into complementary G and Z targets with separate transfer pathways. However, the direct evidence for this decomposition is limited: unified vs. complementary in Table 5 is 67.7 vs. 71.9, a difference of four episodes out of 96.

  1. Design complexity, with unclear attribution of which components matter:
  • The pipeline stacks many components: a hierarchical view-then-time query tokenizer; an anchor-frame-plus-4-frame slot scheme borrowed from video VAEs; a causal slot mixer; three pretraining objectives with a variance-covariance regularizer; a four-stage curriculum; a cosine-decayed alignment weight; and a specific alignment layer. Several of these choices (the tokenizer hierarchy, slot grouping, causal masking, and the curriculum) are not ablated at all.

  • The others are supported only by differences of a few points on a 24-trial-per-task benchmark, and the alignment layer and λ_geo are selected based on results on that same benchmark (Tables 11 and 13).

A minimal variant would clarify whether the added structure is necessary.

[1] FLARE: Robot Learning with Implicit World Modeling. CoRL 2025.

[2] GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation. CVPR 2026.

[3] Action–Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation (GAP). CVPR 2026.

[4] Diffusion-Based Imaginative Coordination for Bimanual Manipulation (Xu et al.). ICCV 2025.

[5] Spatial Forcing: Implicit Spatial Representation Alignment for Vision-Language-Action Model. ICLR 2026.

Questions for authors
  1. The gain attributable to geometry is not significant. In Table 12, the VGGT teacher achieves 71.9% versus 69.8% for DINO, which has no explicit 3D structure. On WidowX (24 trials per task, 96 per model), this corresponds to 69 versus 67 successes, a difference of two episodes. The per-task results are also mixed: VGGT wins on Stack Cube, while DINO wins on Put Carrot and Put Eggplant. Consequently, most of the 10.4-point improvement over the baseline is obtained without a geometric teacher, and the evidence does not support attributing the gains specifically to geometry, which is the central claim of the title and narrative.

  2. The motor representation is derived from ΔG alone, which is questionable. Z is computed by cross-attending only to ΔG_k = G_{k+1} − G_k. Inverse dynamics is naturally a function of both the current state and the transition, i.e., a = f(s_t, s_{t+1}), or equivalently f(G_k, ΔG_k). The same latent change can correspond to different actions depending on the absolute configuration (e.g., gripper pose, object position, gripper open/close state). A difference vector alone is sufficient only under strong assumptions about the latent space, such as approximate translation invariance. Table 10 compares only G versus ΔG (68.8 vs. 71.9, which is itself within noise). The natural G + ΔG variant is neither tested nor discussed.

  3. Missing key related work. plz refer to the Critical Weaknesses

  4. VGGT is applied independently to each frame-view pair (Sec. 3.1, App. C.1), which forgoes its multi-view joint reasoning. Was this choice ablated?

  5. Fig. 8 says "Make Sandwich" while the text says "Make a Burger".

  6. The "Dense Geometry Feature" illustration in Fig. 1(c) should be revised

  7. The supplementary real-robot videos are a welcome addition. Accompanying them with a short textual description would make them more informative, for example: the difficulty and success criteria of each task, typical failure modes of the baselines and of CPR, and whether the shown rollouts are randomly selected or representative.

13 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/7)

1. More Compaired with Related works

W-1: The high-level idea is established in published work, and the paper's framing overstates the contribution. Prior work already uses predictive geometry for action generation, with future information serving only as privileged training-time supervision:

  • FLARE [1] pretrains an action-aware observation embedding and aligns additional policy tokens with future observation embeddings at no inference cost. This is structurally close to CPR's motor-representation alignment.
  • GeoPredict [2] uses predictive future 3D geometry purely as training-time supervision.
  • GAP [3] predicts future latents of a frozen 3D geometric foundation model as an auxiliary target for manipulation.
  • Xu et al. [4] jointly predict compressed future latents with actions and drop the future branch at inference.
  • Spatial Forcing [5] aligns VLA features with VGGT representations.

Of these, GAP, GeoPredict, FLARE and Xu et al. are not cited. Spatial Forcing is cited but categorized as "prediction-centric", although it aligns current-frame features; it is neither discussed as a direct precedent nor compared.

Against this background, presenting "how predictive geometry should be represented for action generation" as an open question that the paper identifies and answers overstates the contribution.

Q-1: Missing key related work. plz refer to the Critical Weaknesses.

We sincerely thank the reviewer for identifying these important related works. Our central motivation, as introduced in Lines 81–90 in main paper, is not simply to investigate how future geometric information can supervise action generation. Rather, we aim to address the different representational requirements of predictive geometric modeling and action generation, and how these requirements can be jointly accommodated through complementary representations.

The cited works explore several closely related directions, but their representation construction and transfer mechanisms differ from ours:

  • FLARE pretrains action-aware observation embeddings and aligns future policy tokens with future observation embeddings. CPR further distinguishes geometry-preserving representations from action-aware representations derived explicitly from geometric transitions.
  • GeoPredict leverages predictive kinematics and future 3D Gaussian geometry as auxiliary supervision for manipulation. CPR additionally constructs a transition-derived motor representation to capture action-aware geometric variations.
  • GAP jointly predicts future actions and 3D geometric information for bimanual manipulation. CPR instead explicitly separates predictive geometric states and action-aware geometric transitions into complementary representation targets.
  • Xu et al. jointly predicts compressed future latents and actions, removing the future prediction branch at inference. CPR focuses on the complementary organization of geometric and motor representations and their distinct downstream transfer pathways.
  • Spatial Forcing aligns current-time VLA features with pretrained VGGT representations. We acknowledge that our original categorization of this work as prediction-centric was inaccurate. We will reclassify it as a current-observation geometric feature alignment approach and clarify how CPR further models evolving geometric states and transition-derived motor representations.

Therefore, compared with above methods, our contribution lies in explicitly distinguishing two complementary representational objectives: preserving coherent scene geometry and its temporal evolution, and extracting action-aware information from geometric transitions.

We will expand the discussion of related work and refine the taxonomy and presentation of our contributions in the revised manuscript and Appendix to improve clarity and reproducibility (Line.140–>147 and 2050–>2077).

AuthorsOpenReview ↗

Rebuttal by Authors (2/7)

2. Further Analysis for Complementary Predictive Representation

W-2:What remains distinctive is the decomposition into complementary G and Z targets with separate transfer pathways. However, the direct evidence for this decomposition is limited: unified vs. complementary in Table 5 is 67.7 vs. 71.9, a difference of four episodes out of 96.

We thank the reviewer for this comment. The WidowX results in Tab.5 are not based on a single evaluation of 96 episodes. Specifically, for each model, we evaluate four tasks with 24 episodes each across five random seeds (0, 1, 2, 3, and 2026). Reported success rates are averaged across the five runs. Thus, the 4.2 percentage-point improvement reflects the difference between repeated evaluation averages, rather than a single 96-episode result.

Beyond downstream policy performance, Fig.6 provides direct evidence that G and Z capture distinct functional information. We conduct zero-shot depth and action probing on LIBERO-90. Although Unified captures both geometric and action-aware information, it underperforms separately learned PGR and MR on their respective geometry- and action-related probing tasks.

To further assess the benefits of assigning predictive geometric and action-aware information to complementary representations, we extend the evaluation beyond the experiments reported in Tab.5 and Fig. 6 of the main paper. Specifically, we train on LIBERO and evaluate zero-shot on LIBERO-PLUS, which comprises 10,030 trials. On LIBERO, CPR improves the average success rate from 97.6% to 98.6%. More importantly, on the more challenging LIBERO-PLUS benchmark, CPR achieves 81.7%, compared with 77.7% for Unified. Notably, Camera and Noise robustness improve by 7.7 and 6.1 percentage points, respectively. CPR also improves performance across all seven evaluation dimensions, further supporting the generalization benefits of complementary geometric and motor representations.

LIBEROObjectSpatialGoalLongAvg.
Unified99.097.897.096.897.6
Ours100.099.098.297.098.6
LIBERO-PLUSCameraRobotLanguageLightBackgroundNoiseLayoutTotal
Unified69.856.685.984.288.389.574.477.7
Ours77.560.888.686.189.195.677.281.7

Taken together, these results support our motivation that predictive geometric modeling benefits from preserving coherent scene structure and its evolution, whereas action generation benefits from emphasizing action-relevant geometric variations. CPR explicitly captures these complementary properties through separate G and Z representations and their role-specific transfer pathways.

We will incorporate these additional results and clarifications into the revised manuscript (Line.1395->1442).

AuthorsOpenReview ↗

Rebuttal by Authors (3/7)

3. More Components Ablations(1/2)

W-3:The pipeline stacks many components: a hierarchical view-then-time query tokenizer; an anchor-frame-plus-4-frame slot scheme borrowed from video VAEs; a causal slot mixer; three pretraining objectives with a variance-covariance regularizer; a four-stage curriculum; a cosine-decayed alignment weight; and a specific alignment layer. Several of these choices (the tokenizer hierarchy, slot grouping, causal masking, and the curriculum) are not ablated at all.

W-4: The alignment layer and λ_geo are selected based on results on that same benchmark (Tables 11 and 13).

W-5: A minimal variant would clarify whether the added structure is necessary.

We thank the reviewer for this constructive comment. CPR is built around a central motivation: organizing predictive geometric information while separating structured geometric states from action-aware transitions. We have conducted extensive ablations of its core components, including complementary PGR/MR pathways (Tab.4–5), pretraining objectives (Tab.9), motor representation inputs (Tab.10), alignment layers (Tab.11), and geometry-guidance weights (Tab.13). To further address the reviewer's concerns, we conduct additional controlled ablations on tokenizer design, temporal grouping, causal masking, and training curriculum.

Hierarchical View-then-Time Tokenization. We compare our hierarchical tokenizer with Flat Joint Tokenization, which jointly compresses multi-view and temporal features. Our design achieves 71.9% average WidowX success, compared with 69.8% for flat tokenization. We further evaluate both variants through zero-shot generation using FastWAM on LIBERO-90. These results support explicitly separating view-level and temporal aggregation. We will incorporate these additional results and clarifications into the revised manuscript (Line.1443->1449).

WidowXPut SpoonPut CarrotStack CubePut EggplantAvg.
Flat Joint87.566.770.854.369.8
Ours95.870.870.850.071.9
LIBERO-90PSNR↑SSIM↑LPIPS↓Action L1↓
Flat Joint21.080.7950.1930.091
Ours22.910.8030.1860.075

Temporal Grouping. We investigate temporal grouping granularity by comparing group sizes of 2, 4, and 8 while keeping the anchor-frame configuration fixed.

Group SizePut SpoonPut CarrotStack CubePut EggplantAvg.
291.775.070.854.272.9
495.870.870.850.071.9
891.766.770.850.069.8

Although group size 2 achieves a slightly higher average success rate, the adopted 4-frame configuration offers a favorable trade-off between performance and temporal grouping granularity(0.36s vs.0.41s). Larger groups (8) lead to noticeable performance degradation. We will incorporate these additional results and clarifications into the revised manuscript (Line.1451->1455).

Causal Slot-Temporal Mixer. We replace causal attention with bidirectional attention while keeping the remaining architecture unchanged and conduct the experiments in WidowX and LBERO-90. Causal attention improves average WidowX success from 68.8% to 71.9%. Consistently, FastWAM's zero-shot generation also improves compared with bidirectional attention.

WidowXPut SpoonPut CarrotStack CubePut EggplantAvg.
Bidirectional91.770.866.745.868.8
Causal95.870.870.850.071.9
LBERO-90PSNR ↑SSIM ↑LPIPS ↓Action L1 ↓
Bidirectional20.950.7870.2010.096
Causal22.910.8030.1860.075

These results support causal temporal modeling for constructing history-conditioned predictive geometric states. We will incorporate these additional results and clarifications into the revised manuscript (Line.1457->1491).

Variance-Covariance Regularizer. We investigate the role of the variance-covariance regularizer while retaining all three pretraining objectives. Removing this regularizer leads to representation collapse by 28.6k training steps, characterized by a progressive decrease in feature variance toward zero. In contrast, the model trained with the regularizer maintains a healthy level of feature variance at the same training stage.

Reviewer_mPDXOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes complementary predictive representations (CPR), a two-stage framework for guiding manipulation policies with predictive geometry and action-relevant transitions. During pretraining, frozen VGGT features are compressed into temporal geometric states using reconstruction, masked prediction, and transition prediction objectives, while an action-supervised motor representation is learned from differences between these states. During policy post-training, future-informed geometric targets supervise dedicated geometry queries, and motor targets align intermediate action features; deployment uses current-time inputs without the representation teacher. The authors evaluate the framework across LIBERO, LIBERO-Plus, SimplerEnv, real robots, and several VLA/WAM architectures, reporting a 10.4-percentage-point improvement on WidowX.

Critical strengths
  1. Relative to the same downstream baseline, CPR improves LIBERO from 96.6% to 98.6%, LIBERO-Plus from 75.0% to 81.7%, and WidowX from 61.5% to 71.9% (Tables 1–2). The real-robot results improve on all four tasks, including stopper removal from 33.3% to 60.0% and liquid transfer from 13.3% to 46.7% (Fig. 4), while the additional unseen visual/spatial perturbations improve average success from 21.7% to 50.0% (Appendix A.1, Fig. 7), supporting practical value beyond nominal simulation performance.

  2. The experiments test meaningful representation alternatives, including relevant prior methods under a shared downstream framework.

  3. The privileged-target formulation preserves current-input action generation, and Fig. 5 reports improvements from 62.5% to 74.0% for the GR00T-style policy and from 59.4% to 68.8% for the π0-style policy. On LIBERO-90, explicitly excluded from representation pretraining in Appendix B.1, FastWAM’s PSNR improves from 19.67 to 22.91, LPIPS decreases from 0.229 to 0.186, and action L1 error decreases from 0.124 to 0.075, supporting usefulness for both world prediction and action modeling.

Critical weaknesses
  1. FutureVLA (Xu et al., 2026, Secs. 3.1–3.2) already separates visual reconstruction and motor supervision and aligns future-informed representations with intermediate VLA features, so CPR’s distinguishing contribution is its evolving geometric states, transition-derived motor targets, and separate transfer pathways.

  2. Several central margins are only 2.1–4.2 percentage points, including VGGT versus DINO, transitions versus states, CPR versus RepWAM, and complementary versus unified representations (Tables 5, 8, 10, and 12), without reported training-seed variation or confidence intervals. Appendix E cites the CogAct protocol but does not explicitly state the actual WidowX or real-robot evaluation counts. Moreover, Table 3’s CPR entries average 98.3%, not 98.5%, and Table 1’s 3D-CAVLA entries average approximately 97.3%, not 98.1%. These discrepancies require correction and reduce confidence in the small reported FastWAM success gain, although they do not negate the larger improvements elsewhere.

  3. Please introduce essential implementation details and resource controls. Appendix B.2 discloses substantial teacher pretraining, but the paper does not specify important aspects of video/action synchronization, heterogeneous action-space handling, or the token/dimension mappings used to align teacher motor representations with different policy architectures. Tables 8 and 14 standardize the downstream framework, but do not sufficiently document the representation teachers’ pretraining data, optimization budgets, and adaptation procedures, limiting both reproducibility and attribution of the comparative gains specifically to CPR’s representation design.

Questions for authors
  1. What are the actual evaluation-trial counts and independent training-seed results for the central WidowX comparisons and the real-robot experiments, and were the alignment-layer and loss-weight choices in Tables 11 and 13 selected on separate validation instances?

  2. For the unified variant and the methods in Tables 8 and 14, which teacher encoders, pretraining corpora, optimization budgets, and representation capacities were used?

  3. Please specify the video sampling interval and prediction horizon, action-chunk synchronization and normalization across embodiments, and the projections or token mappings used for motor alignment in each policy architecture.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

  1. Distinguishing contribution of CPR

W-1:FutureVLA (Xu et al., 2026, Secs. 3.1–3.2) already separates visual reconstruction and motor supervision and aligns future-informed representations with intermediate VLA features, so CPR’s distinguishing contribution is its evolving geometric states, transition-derived motor targets, and separate transfer pathways.

We thank the reviewer for this insightful characterization. As you have stated, CPR builds upon future-guided direction but differs in terms of representation construction and role-specific transfer. Specifically, CPR explicitly models evolving geometric states through hierarchical geometric tokenization and predictive objectives. It further derives a complementary motor representation from consecutive geometric transitions under action supervision, capturing action-relevant changes rather than absolute scene geometry. Thus, CPR's novelty lies not in visual-motor decoupling itself, but in constructing complementary evolving geometric and transition-derived motor representations and transferring them through role-specific policy pathways. Thank you once again for recognizing the methodological novelty of CPR.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

  1. The Details of Evaluation Protocols, Statistical Reliability, and Validation Protocol

W-2:Several central margins are only 2.1–4.2 percentage points, including VGGT versus DINO, transitions versus states, CPR versus RepWAM, and complementary versus unified representations (Tables 5, 8, 10, and 12), without reported training-seed variation or confidence intervals. Appendix E cites the CogAct protocol but does not explicitly state the actual WidowX or real-robot evaluation counts.

Q-1:What are the actual evaluation-trial counts and independent training-seed results for the central WidowX comparisons and the real-robot experiments, and were the alignment-layer and loss-weight choices in Tables 11 and 13 selected on separate validation instances?

We thank the reviewer for raising this important concern. We clarify the evaluation protocols, provide confidence intervals for the central WidowX comparisons, and specify the validation protocol used for hyperparameter selection.

Statistical Results and Hyperparameter Selection. To quantify evaluation variability, We further report the mean performance and the corresponding 95% confidence interval (CI) across these three training seeds in Tables 5, 8, 10, and 12.

Complementary vs. Unified Representations (Table 5):

MethodPut SpoonPut CarrotStack CubePut EggplantAvg.
Baseline83.3 ± 5.7958.3 ± 5.9654.2 ± 6.7450.0 ± 4.2261.5 ± 2.81
Unified91.7 ± 5.4470.8 ± 7.4462.5 ± 7.1645.8 ± 5.7967.7 ± 3.44
Ours95.8 ± 3.6670.8 ± 6.3470.8 ± 7.3250.0 ± 7.3271.9 ± 2.05

CPR vs. Previous Methods (Table 8):

MethodPut SpoonPut CarrotStack CubePut EggplantAvg.
WoG83.3 ± 6.7462.5 ± 7.4437.5 ± 5.6275.0 ± 5.7964.6 ± 3.00
FutureVLA79.2 ± 6.7470.8 ± 6.5941.7 ± 6.5979.2 ± 7.0367.7 ± 2.53
RepWAM87.5 ± 5.2675.0 ± 6.2854.2 ± 6.7458.3 ± 5.4468.8 ± 2.96
Ours95.8 ± 3.6670.8 ± 6.3470.8 ± 7.3250.0 ± 7.3271.9 ± 2.05

Geometric States vs. Transitions (Table 10):

MethodPut SpoonPut CarrotStack CubePut EggplantAvg.
Predictive States G87.5 ± 6.2866.7 ± 6.2866.7 ± 6.5954.2 ± 6.1268.8 ± 3.33
Geometric Transitions95.8 ± 3.6670.8 ± 6.3470.8 ± 7.3250.0 ± 7.3271.9 ± 2.05

VGGT vs. DINO (Table 12):

MethodPut SpoonPut CarrotStack CubePut EggplantAvg.
DINO87.5 ± 7.0375.0 ± 5.0754.2 ± 4.6662.5 ± 6.4469.8 ± 3.10
VGGT95.8 ± 3.6670.8 ± 6.3470.8 ± 7.3250.0 ± 7.3271.9 ± 2.05

For the alignment-layer configurations in Table 11 and the loss-weight configurations in Table 13, specifically, for Table 11, we first fixed the geometry loss weight at 0.5 and evaluated different alignment-layer configurations. We then fixed the alignment-layer configuration selected from Table 11 and varied the geometry loss weight in Table 13. Importantly, we directly transferred the selected alignment-layer configuration and geometry loss weight to LIBERO without further parameter tuning, and obtained promising results. Due to computational constraints, we did not perform an extensive grid search.

Evaluation Protocols. Following the evaluation protocol of CogAct, for each model, we evaluate four WidowX tasks (Spoon, Carrot, Cube, and Eggplant), with 24 test episodes per task, yielding 96 trials per evaluation run. We repeat the evaluation across five random seeds (0, 1, 2, 3, and 2026). For real-world experiments, we conduct 60 physical executions per task across four tasks.

We will include the above in the revised manuscript and Appendix to improve statistical transparency and reproducibility (Line.1869>1871 and Line.2004->2005).

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

  1. Writing Corrections

W-3: Moreover, Table 3’s CPR entries average 98.3%, not 98.5%, and Table 1’s 3D-CAVLA entries average approximately 97.3%, not 98.1%. These discrepancies require correction and reduce confidence in the small reported FastWAM success gain, although they do not negate the larger improvements elsewhere.

We sincerely apologize for the transcription errors in the reported results. We have rechecked the original evaluation records and corrected the corresponding numerical entries in the revised manuscript.

Specifically, the average success rate of 3D-CAVLA in Table 1 has been corrected from 98.1% to 97.3% (Object Task suit 96.8 -> 99.8). For Table 3, we have also corrected the incorrectly transcribed suite-level results for FastWAM with CPR (Long Task suit 96.8 -> 97.8), while retaining its overall average of 98.5%, which is verified against the original evaluation results. We have further rechecked the reported averages and corresponding numerical entries throughout the manuscript to ensure consistency between individual benchmark results, aggregate performance, and the descriptions in the main text.

We thank the reviewer for identifying these errors, which helped us improve the accuracy and reliability of the experimental reporting.

  1. Implementation Details of Model Settings

W-4: Please introduce essential implementation details and resource controls. Appendix B.2 discloses substantial teacher pretraining, but the paper does not specify important aspects of video/action synchronization, heterogeneous action-space handling, or the token/dimension mappings used to align teacher motor representations with different policy architectures.

Q-2: Please specify the video sampling interval and prediction horizon, action-chunk synchronization and normalization across embodiments, and the projections or token mappings used for motor alignment in each policy architecture.

We provide the following clarifications on temporal synchronization, heterogeneous action-space handling, motor representation alignment, and implementation efficiency.

Video Sampling and Action Synchronization. We sample one video frame every two native dataset frames, with an observation window spanning 16 native-frame intervals. Correspondingly, we extract 16 consecutive action steps following the observation index as the prediction horizon.

Heterogeneous Action-Space Handling. To support joint training across embodiments with different action dimensions, we adopt a unified 32-dimensional action target. Lower-dimensional actions are zero-padded, with min-max normalization applied across embodiments. A dimension-wise mask excludes padded dimensions from the regression loss. We further employ a single shared action prediction head across all embodiments, without embodiment-specific heads.

Motor Representation Alignment. We directly align intermediate policy features with teacher motor representations using a cosine similarity objective. Both representations share the same 2560-dimensional feature space; therefore, no additional dimensionality projection is required. This enables direct feature-level supervision while preserving temporal correspondence.

Implementation and Resource Considerations. To reduce downstream training overhead, teacher motor representations are pre-extracted offline and reused as fixed supervision targets, eliminating repeated teacher inference during policy optimization.

We will incorporate these implementation details and configurations into the revised manuscript and Appendix to improve reproducibility (Line.1709->1720).

Reviewer_1qJAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for learning complementary predictive representations. It first learns structured geometric representations from multi-view videos, then learns motor representations from geometric state transitions with action supervision. These two representations provide privileged supervision for the downstream policy's geometry prediction and action generation pathways, respectively. Extensive experiments demonstrate the effectiveness of the pretrained representations.

Critical strengths
  1. The framework is well motivated: it distinguishes scene geometry from action-oriented motor representations and transfers them to the policy through dedicated supervision pathways.
  2. Learning motor representations from latent geometric transitions with action supervision provides a useful connection between predictive representation learning and action generation.
  3. The effectiveness of both representations is supported by experiments across different model backbones and multiple settings.
Critical weaknesses
  1. The main experiment on representation organization does not sufficiently isolate the contribution of the complementary design. The "Unified" baseline also removes the motor branch, simultaneously changing representation roles and supervision pathways. This makes it difficult to attribute the performance gains specifically to the complementary organization.
  2. The architectures of several key modules and the training details across stages remain insufficiently specified, including the predictor architectures and the allocation of training steps to each stage.
  3. The handling of action representations and supervision across embodiments remains unclear, particularly the adaptations for different action dimensions.
Questions for authors
  1. In Table 5, "Complementary" outperforms "Unified" by 4.2 percentage points, but "Unified" also removes the motor branch. Could the authors clarify whether this improvement stems from the representation organization or the accompanying architectural changes?
  2. Could the authors provide the detailed architectures of the Slot-Temporal Mixer, Masked Predictor, and Transition Predictor, together with the training configurations and number of training steps for each stage?
  3. During pretraining, how does the framework handle differences in action dimensionality and sampling frequency across robot datasets? Are the action prediction heads shared?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

1. Complementarity Beyond Dual-Pathway Supervision

W-1: The main experiment on representation organization does not sufficiently isolate the contribution of the complementary design. The "Unified" baseline also removes the motor branch, simultaneously changing representation roles and supervision pathways. This makes it difficult to attribute the performance gains specifically to the complementary organization.

Q-1: In Table 5, "Complementary" outperforms "Unified" by 4.2 percentage points, but "Unified" also removes the motor branch. Could the authors clarify whether this improvement stems from the representation organization or the accompanying architectural changes?

To disentangle representation organization from the number of supervision pathways, we conduct an additional controlled experiment, termed Dual-Unified. It retains Unified's pretraining setup but introduces two post-training supervision pathways: the shared representation supervises both the geometry-query pathway and the intermediate features of the action-generation pathway. Importantly, the representation targets remain identical to Unified.

MethodsPut SpoonPut CarrotStack CubePut EggplantAvg.
Baseline83.358.354.250.061.5
Unified91.770.862.545.867.7
Dual-Unified95.866.766.745.868.8
Complementary 95.870.870.850.071.9

Adding a second supervision pathway improves Unified from 67.7% to 68.8%, but Dual-Unified still falls 3.1 percentage points short of CPR. This indicates that the complementary organization, which assigns distinct representations to preserve predictive geometric structure and capture action-aware transitions , provides additional benefits. We will incorporate this controlled comparison and clarify the interpretation of the ablation in the revised manuscript (Line.460->Line.485).

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

  1. The Details of Architectures and Training Configurations

W-2: The architectures of several key modules and the training details across stages remain insufficiently specified, including the predictor architectures and the allocation of training steps to each stage.

Q-2: Could the authors provide the detailed architectures of the Slot-Temporal Mixer, Masked Predictor, and Transition Predictor, together with the training configurations and number of training steps for each stage?

We will expand the Method and Appendix sections to clarify the module architectures, training configurations, and stage-wise optimization schedule.

Slot-Temporal Mixer. Given geometric states G∈RB×K×S×DG\in\mathbb{R}^{B\times K\times S\times D}, where K=3K=3, S=98S=98, and D=512D=512, we reshape the input to R(B⋅S)×K×D\mathbb{R}^{(B\cdot S)\times K\times D}, treating each geometric slot as an independent temporal sequence. The mixer comprises four causal Transformer layers with 8 attention heads, a hidden dimension of 512, and an FFN dimension of 2048. Each layer adopts Pre-Norm, residual connections, and zero dropout. Causal self-attention operates exclusively along the temporal slot dimension, enabling each slot to aggregate information from current and preceding slots without additional slot-wise attention. Learnable slot positional embeddings are added before temporal mixing.

Masked Geometric Representation Prediction. We adopt slot-level masking, replacing each masked slot with a learnable mask-query embedding. The target and context branches share the same Slot-Temporal Mixer parameters: the target branch encodes the complete sequence to produce detached target representations, while the context branch processes masked inputs. Before prediction, masked context features are explicitly overwritten with learnable mask-query embeddings to prevent information leakage from potentially reconstructed features. The masked predictor comprises two bidirectional Pre-Norm Transformer layers with 8 attention heads, a hidden dimension of 512, an FFN dimension of 2048, and zero dropout. It uses independent learnable slot positional embeddings, separate from those of the Slot-Temporal Mixer. A prediction head (LayerNorm → Linear(512, 512)) projects the outputs into the target representation space. Smooth L1 loss is applied exclusively at masked slot positions against detached targets, establishing a slot-level predictive learning objective rather than individual-slot or pixel-level reconstruction.

Geometric Transition Prediction. An auxiliary geometric transition prediction module consisting of LayerNorm → Linear(512, 512) → GELU → Linear(512, 512). Given current geometric states GG, it predicts latent transition, supervised by detached next-slot targets using Smooth L1 loss.

Stage-wise Training Configurations. We use a four-stage progressive training schedule with a total of 120K optimization steps, which is consist of Geometry Reconstruction(0-25k), Masked Geometric Representation Prediction(25k-50k), Geometric Transition Prediction(50k-70k), Motor Representation Learning(70k-120k). Pretraining uses 32 NVIDIA H200 GPUs with a global batch size of 512. We employ a base learning rate of 1×10−51\times10^{-5}, betas of (0.9,0.95)(0.9, 0.95), weight decay of 1×10−81\times10^{-8}. The loss weights are λmask=0.5\lambda_{\text{mask}}=0.5, λtrans=0.5\lambda_{\text{trans}}=0.5, and λvc=0.06\lambda_{\text{vc}}=0.06, with gradient clipping set to 1.0. We use BF16 mixed-precision training with DeepSpeed ZeRO-2, while keeping both the pretrained 3D foundation encoder and VGGT teacher frozen throughout training.

We will incorporate these architectural specifications (Line.1769->1797) and training configurations (Line.1709->1720) into the revised manuscript. Furthermore, upon acceptance, we will publicly release the source code to facilitate the reproducibility of our work.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

  1. The Details of Datasets Setting

W-3:The handling of action representations and supervision across embodiments remains unclear, particularly the adaptations for different action dimensions.

Q-3:During pretraining, how does the framework handle differences in action dimensionality and sampling frequency across robot datasets? Are the action prediction heads shared?

We will expand the Method and Appendix sections to clarify the action dimensionality and sampling frequency across robot datasets.

Action dimensionality. All pretraining datasets have action dimensions di≤32d_i\leq 32. We therefore adopt a unified 32-dimensional action target, zero-padding dimensions when di<32d_i<32. A dimension-wise mask excludes padded dimensions from the action regression loss.

Temporal sampling frequency. We preserve each dataset's native action sampling frequency without resampling trajectories to a common frequency. For visual observations, we sample one frame every two native dataset frames.

Shared action prediction head. We employ a single shared action prediction head across all robot embodiments during representation pretraining, without embodiment-specific heads.

Data composition. We follow the dataset mixture specified in Appendix B.1 (Table 26), which reports the sampling ratio of each pretraining dataset.

We will clarify the action dimensionality, temporal sampling conventions, and shared prediction head in the revised manuscript (Line.1709->1720) .

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates for Paper 13394

We sincerely thank all reviewers for the thoughtful and constructive feedback. A revised paper has been uploaded, with the changes marked in blue. We would like to provide a brief factual summary of our contributions and the updates made during the rebuttal period.

Summary of Contributions

Our work, Complementary Predictive Representations (CPR), addresses the distinct representational needs of predictive geometric modeling and action generation: the former preserves coherent scene geometry and its temporal evolution, while the latter captures geometric variations relevant to motor commands. Accordingly, CPR learns two complementary representations: a predictive geometric representation of evolving scene states and an action-aware motor representation derived from geometric transitions. These representations guide downstream policy learning through dedicated geometry-query and motor-alignment pathways, respectively.

Reviewer Consensus

We are encouraged that reviewers recognized the value of this work, highlighting:

  • Strong Empirical Performance: Consistent improvements across simulation benchmarks and challenging real-world manipulation tasks (1qJA, mPDX, Azfm).
  • Cross-architecture Transferability: The applicability of CPR across different VLA/WAM architectures without additional representation-teacher inference at deployment (1qJA, mPDX, Azfm).
  • Extensive Experimental Evaluation: The use of multiple benchmarks, representation probing, and component-level ablations to investigate the effectiveness of CPR (1qJA, mPDX, Azfm).
  • Methodological Novelty: The motivation behind CPR's design, which is to learn complementary predictive representations and transfer them to the policy through dedicated supervision pathways (1qJA, mPDX).

Rebuttal Updates

In our responses, we have addressed all reviewer questions. A summary of the key updates follows:

  • Further Analysis for Complementary Representations (1qJA, Azfm): We introduced a dual-unified baseline that uses the same unified representation to supervise both downstream guidance pathways, further providing stronger evidence for the benefit of distinct representation targets. Additional LIBERO-to-LIBERO-PLUS evaluations show that CPR outperforms unified, further demonstrating its generalization advantages.

  • Implementation Details (1qJA, mPDX): We expanded the architectural specifications of the Slot-Temporal Mixer, Masked Predictor, Transition Predictor, and motor representation modules. We further clarified stage-wise optimization, video/action synchronization, cross-embodiment action handling, shared action prediction, and motor feature alignment.

  • Extensive Architectural and Training Ablations (Azfm): We conducted additional controlled experiments on hierarchical versus flat joint tokenization, causal versus bidirectional temporal modeling, temporal grouping sizes, and progressive versus joint training. We further evaluated architectural variants using FastWAM's zero-shot generation on LIBERO-90.

  • Additional Motor and Geometric Representation Analyses (Azfm): We introduced the reviewer-suggested G + ΔG motor representation baseline. We also evaluated joint multi-view VGGT encoding.

  • Additional Real-world Evidence for Geometric Priors (Azfm): To further investigate the contribution of explicit geometric information, we conducted additional real-robot comparisons between DINO-based and VGGT-based CPR. These results provide additional evidence for the practical benefits of geometric priors in spatially demanding manipulation.

  • Clarification on Statistical Significance (mPDX, Azfm): We clarified our evaluation protocols. These evaluation procedures provide more reliable performance estimates and help reduce sensitivity to individual trials.

  • Presentation and Related Work (mPDX, Azfm): We corrected the writing errors, revised figure terminology, and added textual descriptions for the supplementary real-robot videos. We further distinguished CPR from FLARE, GeoPredict, GAP, FutureVLA, DBIC-BM, and Spatial Forcing, and revised the related-work discussion.

We appreciate the reviewers’ engagement and the opportunity to clarify our contributions. We believe these additional results and clarifications provide a more comprehensive empirical foundation for CPR and further substantiate its contribution to predictive representation learning for embodied manipulation.

I thank the authors for their extensive engagement. The revisions resolved several of my concerns, including the missing related work, the evidence on LIBERO-Plus and on real robots, and the additional ablations. However, the per-seed WidowX results added in the second response look unusual. For most alternative variants, every task's success rate increases steadily with seed order and is spread evenly around the previously reported mean, a pattern that the proposed method's runs do not show, so the ablations do not clearly support the design choices. In addition, the closest published baseline, Spatial Forcing, performs on par with CPR on LIBERO (98.5 vs. 98.6). This leaves the benefit of the substantially more complex pipeline unclear. Taken together with the remaining inconsistencies in the manuscript, I do not think the paper is ready for ICLR in its current form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.