acceptodds
Under review as a conference paper at ICLR 2027

RM3D: A TRAINING-FREE RETRIEVAL-AND-MORPHING PIPELINE FOR SINGLE IMAGE 3D GENERATION

Abstract

We present RM3D, a training-free retrieval-and-morphing framework for single-image 3D generation that uses a retrieved 3D asset without per-instance optimization, task-specific fine-tuning, or additional trainable weights. Existing retrieval-augmented methods provide external 3D evidence at inference time, but still optimize each input or train a reference-conditioned network. RM3D instead reuses a frozen native-3D generator as both a shape prior and a mechanism for latent morphing. Given one image, it retrieves a semantically compatible mesh, loads its precomputed structured latent, and morphs that latent toward the input image inside the frozen sampler. To preserve the retrieved geometry while adapting its appearance, RM3D aligns source and target condition tokens through Hungarian matching, uses correspondence-matched key/value memory in early transformer blocks, and switches to target-only key/value memory in later blocks. Across the quantitative comparisons, RM3D ranks first among the compared methods on every reported appearance and geometry metric. Qualitative comparisons also show that it better preserves input appearance and cross-view structure.

Primary area:
generative models
Keywords:
3D, Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GavVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RM3D is a retrieval-and-morphing pipeline for single-image 3D generation using frozen OpenShape, DINOv2, and TRELLIS models. It retrieves an asset from a 23,835-object database and uses its cached SLAT as a source reference, while the target endpoint is generated from the input image. The method combines condition-token alignment, SLAT-guided structural matching, correspondence-matched attention, and target-only attention in late blocks. It requires neither task-specific training nor per-instance optimization. Experiments on Toys4K report improvements over TRELLIS-S and other generators, cumulative component ablations, and a database-size scaling study.

Critical strengths
  • Using retrieved 3D assets to supplement a frozen generator is a useful direction. The method incorporates external references without task-specific training or per-instance optimization.

  • The distinction between borrowing source structure and preserving target appearance is well motivated. The correspondence and target-focused attention designs offer a practical way to address the mismatch between a retrieved asset and the input image.

Critical weaknesses
  • Configuration A omits the semantic, structural, and texture alignment already available in full Interp3D. Since configuration C introduces related alignment mechanisms, its improvement over A may partly reflect recovering capabilities removed from the baseline. A comparison with full Interp3D under the same retrieval setup is needed to establish the benefit of RM3D's modifications.

  • The comparisons omit more recent methods such as Direct3D-S2 and TRELLIS.2.

  • The objects selected for the qualitative comparisons are quite simple, and recent 3D generators (TRELLIS.2, Pixel3D, etc.) can already produce reasonable results for such objects. The real challenge lies in complex or visually ambiguous objects. The paper should show cases where existing methods produce implausible details and RM3D makes them more plausible, but such examples are not presented.

  • The three-pass sampler and target generation add computation, but the paper gives no runtime comparison with TRELLIS. Figure 6 reports average gains from a larger database, without showing how many of the 200 test objects improve or get worse.

  • Some descriptions are inconsistent across the paper: (1) Figure 2 suggests multi-view source conditioning, but Section 3.2 specifies cached view-059 tokens. (2) Section 4.2 describes native 3DGS rendering, although Hunyuan3D-2.1 outputs meshes; how these meshes are converted or rendered is not explained.

  • Eq. (6) jointly reorders target K/V before concatenation, which leaves standard unmasked attention unchanged. I cannot tell how the correspondence in Eq. (4) affects structure generation. Any positional reassignment or masking that makes this operation effective needs to be stated explicitly.

Questions for authors

The following questions correspond, in order, to the critical weaknesses above.

  1. Using the same retrieved assets, does RM3D still outperform Interp3D when its original semantic, structural, and texture alignment are all retained?

  2. Could you compare RM3D with Direct3D-S2 and TRELLIS.2 using the same input images? How do their geometry and texture quality compare?

  3. Could you show complex or visually ambiguous objects where TRELLIS produces missing parts or incorrect connections, and RM3D corrects them? Please show the retrieved asset and close-ups of the same details in both outputs.

  4. How long does RM3D take compared with TRELLIS on the same GPU? In Figure 6, does a larger database help most test objects, or do some results get worse?

  5. Does source conditioning use multiple views or only view-059? Please clarify any view aggregation and how mesh-based outputs are rendered or converted for comparison with native 3DGS.

  6. What positional or masking operation makes Eq. (6) sensitive to the K/V permutation? Please specify it and compare outputs with and without reordering while holding other inputs fixed.

Reviewer_soFPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RM3D uses the cached structured latent of a retrieved 3D asset to guide single-image 3D generation without additional trainable parameters or per-instance optimization. It combines this source with a target latent generated from the input image through grouped Hungarian alignment, correspondence-aware attention, and target-only attention in later transformer blocks.

Critical strengths

None

Critical weaknesses
  • Clarity of exposition. The presentation is difficult to follow because several key implementation choices are insufficiently explained. In particular, the mapping of “view-059” to a camera pose (Sec. 3.2), the criterion for selecting the “nearest nonempty sampled support” (Sec. 3.3), and the rationale for using 25 Euler steps (Sec. 3.4) are left unclear. The method description requires substantial revision to make the pipeline and experimental setup clear and reproducible.
  • Limited incremental contribution over prior work. RM3D appears to be an incremental extension of Interp3D, primarily modifying the source SLAT endpoint and the key/value (K/V) fusion strategy. Although the cumulative ablations support some additions, they do not fully isolate the effects of all proposed changes. The technical novelty appears limited, and the specific contributions beyond Interp3D require clearer justification.
Questions for authors

None

Reviewer_US3WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RM3D is a training-free method for deforming 3D assets. It leverages spatial priors from Trellis, combined with custom retrieval-matching algorithms, to achieve these deformations. The authors acknowledge that the techniques employed are derived from existing methods, making this work more of an integration-focused engineering effort. Furthermore, while the paper claims to be the first training-free approach for this task, a quite similar method, MorphAny3D (CVPR 2026), has already been proposed; yet, this paper fails to provide any comparison against it.

Critical strengths

Practical training-free setting. The method reuses pretrained models and existing assets without additional fine-tuning or per-instance optimization, making it potentially useful when retraining is impractical.

Explicit use of retrieved 3D structure. A complete reference asset provides additional geometric information beyond the conditioning image, offering a plausible way to constrain otherwise ambiguous single-view reconstruction.

Critical weaknesses

The methodological contribution appears incremental and primarily integrative. The pipeline relies on existing components: OpenShape for retrieval, TRELLIS for the representation and generative prior, and Interp3D for the three-stream morphing framework. The attention-control mechanisms also draw on established techniques, as acknowledged by the authors. Integrating these components can be useful, but the paper does not sufficiently establish which new technical insight is essential to the improvements.

The omission of MorphAny3D leaves the novelty positioning incomplete. MorphAny3D (CVPR 2026) already introduces training-free 3D morphing through source–target feature fusion within a structured-latent generator’s attention mechanisms, and its official implementation builds on TRELLIS. Despite this substantial methodological relevance, RM3D neither discusses nor compares against it.

The experimental controls do not convincingly establish the source of the improvements. The ablation baseline uses a simplified Interp3D configuration that omits several of its alignment mechanisms. Consequently, improvements over this baseline do not establish an advantage over the complete predecessor. The experiments also lack a source-free control that retains target-only refinement, leaving open whether the gains mainly come from the retrieved asset, additional computation, or stronger target-side processing.

Questions for authors
  1. Compared with MorphAny3D, what is the central methodological contribution beyond integrating retrieval with an existing training-free morphing framework?
  2. Can the authors independently ablate condition-token alignment, geometry correspondence, K/V matching, and target-only refinement to establish which components are necessary?
  3. How does RM3D differ technically from MorphAny3D beyond reference retrieval and the final reconstruction objective?
  4. Can the authors clarify the precise scope of the “first” claim and explain what additional capability cannot be obtained by combining retrieval with an existing training-free morphing method?
  5. Can RM3D be compared with complete Interp3D and an adapted MorphAny3D using the same retrieved references, input images, and comparable inference budgets?
  6. How much improvement remains when the retrieved source is removed or replaced with a random reference, while target-side refinement and the inference budget are retained?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.