acceptodds
Withdrawn from ICLR 2027

Learning Maximally Computable Visual Representations for Controllable Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D reconstruction provides explicit geometry and accurate transformation of observed content, but requires dense inputs and often degrades under large viewpoint changes. In contrast, generative world models can synthesize realistic future observations, yet their data-driven dynamics make precise geometric control and long-horizon consistency difficult to guarantee. We introduce Maximally Computable Visual Representation (MCVR), which combines explicit reconstruction with generative modeling. Rather than predicting the entire future visual state, MCVR learns a persistent representation whose physically supported content is deterministically propagated under known camera and object transformations, while learned generation is reserved for unrecoverable content. This enables accurate object manipulation and geometry-aware generation while reducing unnecessary prediction. Experiments on autonomous-driving scenes show that MCVR achieves a strong trade-off between visual sufficiency and physical computability, improving controllability and geometric consistency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Controllable Generation, Persistent Visual Representation, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hWRJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MCVR, a visual representation trained for both image reconstruction and consistency under explicit geometric transport. Features from a frozen DINOv3 encoder are adapted into persistent states attached to static world coordinates or object-local coordinates. At a requested camera/object configuration, supported features are transported, unsupported regions are completed by a feature generator, and a view-conditioned renderer produces RGB. The evaluation uses nuScenes and includes representation diagnostics, transport-coverage curves, depth/segmentation evaluation, and a 150-pair target-frame synthesis benchmark. The reported synthesis results improve appearance metrics over MagicDrive-V2.

Critical strengths
  1. The distinction between representation inconsistency and absent scene support is a useful formulation of the problem (Sections 3.1–3.3 and 4.3). Anchoring static and dynamic content in appropriate coordinate frames, preserving supported features, and completing unsupported regions forms a coherent approach to controllable generation. This makes the proposed contribution more specific than simply adding geometric conditioning to an image generator.

  2. The paper reports a substantial appearance-quality improvement in its target-frame setting: PSNR increases from 14.21 to 21.07 and LPIPS decreases from 0.519 to 0.286 relative to MagicDrive-V2 (Table 2). The benchmark is constructed from held-out validation scenes, uses the same target pairs, and reports scene-balanced estimates with scene-level bootstrap intervals (Appendix A.3). These are encouraging results for the complete pipeline, while not isolating the effect of the representation objective.

Critical weaknesses
  1. The central representation-learning claim lacks its decisive comparison. Table 1 reports only MCVR (Recon-only), which Section 4.1 defines as removing L_comp; it contains no full MCVR row. Thus it shows why reconstruction alone may be insufficient, but not whether the proposed objective resolves that problem or what visual-sufficiency trade-off it achieves. Table 4 explicitly changes inference operators for a fixed checkpoint and is not a training ablation, so it cannot establish the claimed causal chain from the learned representation to synthesis quality. This remains the case after checking Appendix A.1–A.3. Moreover, Table 1 says its metrics use negative-calibrated compatibility rather than raw cosine, whereas Equations (8)–(9) define raw-cosine metrics without specifying the calibration. A matched full-MCVR/Recon-only comparison with a consistent metric definition is necessary to substantiate the main contribution.

  2. The evidence does not establish the claimed improvement in controllability and long-horizon consistency. In Table 2, MCVR has worse point estimates for all three control measures than MagicDrive-V2: CenterErr 2.10 versus 1.75, BoxIoU 0.517 versus 0.556, and DetRecall 0.809 versus 0.986. These figures do not establish a significant difference for every metric, but they do not support a quantitative control advantage. Appendix A.3 specifies that this benchmark consists of individual target frames approximately one second from the source. Figure 3 measures transport coverage up to four seconds, not generated-sequence consistency; the video-metric paragraph supplies no corresponding rollout results in the supplied PDF. Consequently, appearance gains at one second cannot establish that memory write-back improves long-horizon generation. The exact MagicDrive-V2 conditioning is also left as “Native” rather than enumerated, limiting interpretation of the comparison.

Questions for authors
  1. Can you provide the full MCVR versus Recon-only results under the same geometry, renderer capacity, and training budget, including both Table 1 metrics and target-frame synthesis? Please also clarify whether Table 1 uses raw cosine or negative-calibrated compatibility and give the calibration definition if applicable. This would directly determine whether the proposed representation objective is supported.

  2. What generated-rollout evidence supports the claimed long-horizon consistency benefit, and was memory write-back enabled in those evaluations? In the paper, I could locate the approximately one-second single-frame benchmark and four-second transport-coverage curves, but not sequence-consistency results. This distinction materially affects the scope of the contribution.

  3. What precise image history, camera coverage, map information, and agent-state conditioning were used for MagicDrive-V2 in Table 2, and how should its stronger reported control point estimates be reconciled with the claimed controllability advantage of MCVR?

Reviewer_reptOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MCVR, a visual representation, which essentially fine-tunes the DINOv3 plus an adapter, meant to stay informative while being explicitly transportable under known motion. Given one source image, the method predicts depth and movable object masks. It then moves the features with the target ego pose and with ground-truth 3D box poses, splats them into the target view, inpaints uncovered regions and decodes to RGB. Evaluation uses nuScenes front-camera pairs and show improved results.

Critical strengths

Some strengths:

  1. Transporting geometrically determined content instead of regenerating it is a valid and well-motivated idea for controllable driving simulation, as shown in the paper's results, not "controllable world model".
  2. Some results, for example, the analysis of transportable area versus time horizon is a useful diagnostic, and the visibility is implemented carefully.
Critical weaknesses

There are quite a few weaknesses of the paper:

  1. The central claim is untested. Table 1 reports only MCVR (Recon-only), the variant without the computability objective, and it is less transportable than off-the-shelf DA3 and DINOv3, the full model's performance is then never shown.
  2. Some results contradict the abstract. MCVR falls behind MagicDrive-V2 on every control metric in Table 2, despite the claim of improved controllability in the abstract. Also no long-horizon results are reported. Fig. 3 shows only about 20–25% of content is transportable after 1 s, which weakens the premise of “reducing unnecessary prediction.”
  3. Limited novelty and missing comparisons. The method is essentially single-image novel view synthesis by depth-based feature warping plus inpainting, which is neither cited nor compared. Street Gaussians, RGB and VAE are named as baselines but are missing from the tables. Key ablations are absent: warping RGB directly, using DA3 or DINOv3 features in the same pipeline, and removing L_comp end to end.
  4. Presentation and completeness. Table entries are still marked “pending,” and training coefficients and schedules are omitted.
Questions for authors
  1. What are the Table 1 results for the full MCVR, with L_comp?
  2. How does MCVR compare with RGB warping plus inpainting, and with feature warping, given the same geometry?
  3. What supports the controllability claim, given the Table 2 control metrics?
  4. Could author provide any of the long-horizon results?
  5. How robust is the method with predicted rather than ground-truth 3D boxes?
Reviewer_K5EDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MCVR aims to learn visual features that retain enough information for rendering while remaining consistent under explicit camera and object transformations. A trainable adapter on frozen DINOv3 features is optimized for reconstruction and transported-feature consistency. Predicted depth and movable-object segmentation provide geometric support; annotated object poses enable transport in canonical object coordinates. A feature-completion network fills unsupported regions, and a shared renderer produces target RGB. Experiments on nuScenes examine representation reconstruction and transportability, geometric support over time, and target-frame synthesis under recorded ego and actor motion. The intended claim is that explicitly transporting persistent state reduces prediction burden and improves geometric control and consistency.

Critical strengths
  1. The distinction between visual sufficiency and transport consistency is a useful research question (§§3.1–3.3). Requiring both prevents a transportable but visually impoverished state from counting as success. The proposed common transport operator and same-capacity probe renderers are sensible tools for studying that trade-off.

  2. The target-frame results show a substantial image-quality advantage, and the evaluation discloses important limitations (Table 2; Appendix A.3). MCVR reports 21.07 versus 14.21 dB PSNR and 0.286 versus 0.519 LPIPS against MagicDrive-V2 on the selected 150 pairs. Scene-level confidence intervals, missed-target treatment, and the distinction between fixed-weight diagnostics and retrained ablations are useful reporting choices.

Critical weaknesses
  1. The principal representation-learning claim is not evaluated by the presented comparison (§4.2; Table 1; Table 4). Table 1 contains MCVR Recon-only but no full MCVR row trained with the proposed computability objective. It therefore motivates the objective without demonstrating that the objective improves the trade-off. Table 4 changes inference operators under a fixed checkpoint and explicitly cannot substitute for that comparison. In addition, Table 1 says transportability uses negative-calibrated compatibility, whereas Appendix A.3, Equations (8)–(9), defines raw cosine-based metrics without the calibration. The missing central comparison and inconsistent metric specification prevent verification of the paper's claimed causal chain.

  2. The reported evidence does not support the stronger control, long-horizon, and prediction-cost conclusions (§§4.3–4.5; Table 2; Appendix A.3). All three reported control point estimates favor MagicDrive-V2: CenterErr is 1.75 versus 2.10, BoxIoU is 0.556 versus 0.517, and detection recall is 0.986 versus 0.809. The appearance advantage should therefore be distinguished from improved control. The quantitative synthesis benchmark contains individual pairs separated by approximately one second; the coverage curve out to four seconds is not a recurrent-generation consistency evaluation. Likewise, the fraction of supported pixels does not establish FLOP or latency savings when the completion network and renderer still run. These are central gaps between the demonstrated results and the advertised contribution, rather than requests for additional peripheral benchmarks.

Questions for authors
  1. Was full MCVR evaluated in the Table 1 protocol? If so, please provide the existing row and identify the checkpoint and losses used; otherwise, which reported result directly measures the effect of the computability objective?
  2. What is the exact negative-calibration formula referenced in Table 1, how is it fitted, and does it replace the raw-cosine definitions in Appendix A.3?
  3. Please reconcile the control claims with Table 2's three control metrics. What exact source-image/history and geometric conditioning did the evaluated MagicDrive-V2 configuration receive?
  4. Which already completed results, if any, evaluate recurrent memory write-back, long-horizon generated-image consistency, or computational savings? Please distinguish those measurements from the source-to-target coverage analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.