acceptodds
Under review as a conference paper at ICLR 2027

UMI3D: Robust 3D Generation on Unconstrained Multi-Image Inputs via Simultaneous Focusing Cross-Attention Routing

Abstract

Recent 3D foundation models enable 3D asset generation from a single image, but degrade substantially when extended to unconstrained multi-image inputs, which comprise collections of images depicting the same object yet differing in pose, style, illumination, and local details. This degradation often results in distorted geometry, over-smoothed textures, and inconsistent colors. Our analysis suggests that a key limitation is the mismatch between single-image cross-attention and multi-image conditioning: existing models lack an explicit mechanism for selecting a reference image for each voxel at each denoising step. Based on this observation, we propose UMI3D, a training-free framework that restructures cross-attention by explicitly routing each voxel to its most informative image, which prevents conflicting visual evidence from being indiscriminately mixed during generation, thereby unlocking strong performance on inconsistent inputs. To realize this routing, UMI3D introduces Simultaneous Focusing Cross-Attention (SFC-Attn), which activates all conditioning images at each denoising step while allowing each voxel to focus on the single image that best explains it. We further derive the Voxel Reference Score (VRS), a model-intrinsic metric for voxel–image affinity that requires no external models. Extensive experiments show that UMI3D unlocks the multi-image potential of single-image 3D generation frameworks across diverse tasks. Code will be released.

Primary area:
generative models
Keywords:
3D Generation, Unconstrained Input, Training Free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nV6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UMI3D, a training-free framework that restructures cross attention by explicitly routing each voxel to its most informative image, which prevents conflicting visual evidence from being indiscriminately mixed during generation, thereby unlocking strong performance on inconsistent inputs. To realize this routing, UMI3D introduces Simultaneous Focusing Cross-Attention (SFC-Attn), which activates all conditioning images at each denoising step while allowing each voxel to focus on the single image that best explains it. Extensive experiments show that UMI3D unlocks the multi-image potential of single-image 3D generation frameworks across diverse tasks.

Critical strengths

A training-free framework that adapts single-image 3D foundation models to unconstrained multi-image inputs without updating the backbone parameters. A new benchmark for unconstrained multi-image 3D generation with realistic cross-view inconsistencies.

Critical weaknesses

How to verify the consistency of multi-image 3D generation in the experiment? The corresponding quantitative results and qualitative results should be provided. Why the results of UMI3D and other methods in terms of other metrics (LPIPS, DINO, CLIP) present near performance while User Study results of UMI3D has significant gain compared other methods?

Questions for authors

There are many generation designs are proposed for 3D generation, such as [1] DyFusion: Cross-attention 3D object detection with dynamic fusion. [2] Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention [3] Ground-to-Satellite Localization in Unconstrained Image Collections for 3D Scene Reconstruction [4] Wonder3d++: Cross-domain diffusion for high-fidelity 3D generation from a single image [5] Svim3d: Stable video material diffusion for single image 3d generation [6] MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models There methods should be compared and discussed. The similar designs should be ablated in the experiments. How to evaluate the performance of multi-view 3D generation enhancement? Some artifacts are generated in results of UMI3D in figure 8. Figure 2 should be modified and improved. The core designs should be highlighted. How to achieve the Simultaneous Focusing Cross-Attention (SFC-Attn), how to prove the Simultaneous design in this module. The corresponding motivation should be improved. Why use the Voxel Reference Score to route each voxel? Are there any other routing methods? What are the corresponding results compared with other methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nV6m (1/2)

W1: Evaluation of multi-image consistency

We evaluate reference-view fidelity by rendering each generated 3D asset from the input viewpoints and comparing the renderings with their corresponding references using CLIP, DINO, and LPIPS.

Our benchmark is created by independently editing views rendered from source meshes. Because these edits may alter geometry and appearance, the original mesh is not a valid common ground truth for all edited references. We therefore do not separately quantify cross-view consistency. Instead, our user study includes renderings from additional viewpoints and provides complementary evidence of perceived structural and visual coherence. We have clarified the scope and this limitation in the revised appendix.

W2: Automatic metrics versus user preference

Two factors may contribute to this difference.

  1. Although the input views are unconstrained, they depict the same object. As a result, most methods preserve its broad semantic identity and may receive similar semantic-similarity scores.

  2. The evaluations cover different viewpoints. As described in Appendix A.4, our user study includes additional viewpoints, allowing participants to assess structural defects, missing details, and the overall coherence of the 3D asset—qualities that reference-view metrics may not fully capture.

To illustrate this limitation, Fig. 17 shows representative cases in which outputs with visibly different perceptual quality receive similar CLIP, DINO, and LPIPS scores. These examples suggest that semantic similarity and the restricted viewpoints evaluated by the metrics can obscure perceptual quality differences. Thus, metric scores may not fully reflect the quality gap between outputs, motivating our use of a user study as a complementary assessment.

Q1: Related work

Ultra3D [2], Wonder3D++ [4], and Svim3D [5] are single-image 3D generation methods and do not support multi-image input. In addition, the official code for Ultra3D and Svim3D is unavailable, precluding direct comparisons in our setting. DyFusion [1] focuses on 3D detection, [3] on 3D scene reconstruction, and MICON-Bench [6] on in-context image generation; these works address different tasks and are not directly comparable to our multi-image 3D generation method.

We have added [2], [4], and [5] to the Related Work section.

Q2: Performance of multi-view 3D generation enhancement

We evaluate SFC-Attn on 20 randomly selected Sketchfab assets. For each asset, we render three input images from the same fixed viewpoints, uniformly distributed around the asset. The complete GLB model serves as the geometric ground truth, and both generated and reference geometries use the same coordinate system. We report CD (×10³), F-score at a threshold of 0.1, and the CLIP, DINO, and LPIPS metrics used in our paper. The baseline and SFC-Attn-enhanced TRELLIS.2 use identical input views.

As shown below, SFC-Attn improves all five metrics over the baseline, indicating better geometric accuracy and reference-view fidelity.

Multi-view generation results

MethodCD (×10³) ↓F-score (0.1) ↑CLIP ↑DINO ↑LPIPS ↓
TRELLIS.2 baseline86.8787.4289.8378.720.2977
TRELLIS.2 + SFC-Attn84.0387.9889.9979.190.2943

Q3: Artifacts in Fig. 8

We acknowledge that artifacts remain in Fig. 8, including inaccurate text on the watch screen. This is an auxiliary application of VRS-based cross-attention routing on TRELLIS.2, rather than our primary unconstrained multi-image generation task. Compared with the TRELLIS.2 baseline, SFC-Attn improves specific regions, including the knight’s armor texture and SpongeBob’s mouth, but does not eliminate all artifacts. We will explore further improvements to multi-view 3D generation with VRS and SFC-Attn in future work.

Q4: Advice on Fig. 2.

The core components are shown in Fig. 2(b) and (c): panel (b) illustrates how VRS derives voxel–image assignments from attention, while panel (c) shows how SFC-Attn uses these assignments for grouped cross-attention and feature merging. Panel (a) presents the overall backbone pipeline. We will emphasize the labels of panels (b) and (c) and clarify this distinction in the caption.

AuthorsOpenReview ↗

Response to Reviewer nV6m (2/2)

Q5: Implementation of SFC-Attn.

SFC-Attn performs VRS-guided voxel grouping and group-wise cross-attention, as detailed in Sec. 3.3 and Fig. 2(c). Within each denoising step and DiT block, VRS assigns each voxel to a reference image. Each voxel group then attends to its assigned reference, and the group-wise outputs are merged into a unified voxel feature tensor. Thus, all references contribute within the same denoising step, while each voxel selectively uses the reference assigned to it.

As illustrated in Supp. A.1, switching references across denoising steps can cause conditioning oscillation and detail loss, whereas indiscriminately combining reference-conditioned outputs can mix conflicting evidence. SFC-Attn avoids both by keeping all references available within each step while routing their evidence selectively. Here, “simultaneous” refers to using all references within the same step, and “focusing” refers to voxel-wise reference selection.

Q6: Why use the Voxel Reference Score (VRS)?

Different voxels may rely on different input images. VRS measures the attention mass allocated to local image tokens and provides a model-intrinsic proxy for voxel–image support, enabling reference selection without an additional matching model or trained router.

We compare VRS routing with random, opposite-VRS, and soft VRS-weighted routing in Table 4 and Fig. 8. Among these evaluated strategies, VRS routing achieves the best CLIP, DINO, and LPIPS scores.

Routing strategyCLIP ↑DINO ↑LPIPS ↓
Baseline85.9572.990.3413
Random CA Routing85.6172.360.3423
Opposite VRS Routing84.4269.630.3439
Soft VRS-Weighted Fusion85.7772.350.3435
VRS Routing (Ours)86.4274.620.3356

I have read the authors' response and authors generally addressed my main concerns. I decide to maintain my original score.

Reviewer_4MwyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes UMI3D, a training-free framework for extending single-image 3D generators to multiple potentially inconsistent reference images. It introduces a Voxel Reference Score (VRS), defined as the cross-attention mass assigned to local image tokens, to select a reference image for each voxel. The assignments are refined through 3D KNN and used to route cross-attention during geometry and texture generation.

Critical strengths
  1. I like the problem setting and the training-free approach. Reference images collected in practice can differ in viewpoint, pose, and appearance, so extending existing generators to these inputs is useful. The method is straightforward and does not require camera estimation or an external correspondence model.

  2. The analysis of global and local tokens is interesting, and the token-replacement experiment provides useful motivation for VRS. The qualitative results show promising improvements in detail preservation, particularly in the multi-view examples in Figure 8.

Critical weaknesses
  1. I wonder if VRS reliably identifies useful references when the input poses differ. In Figure 5, several inputs have substantially different body poses, rather than just different camera viewpoints. The method relies on the pretrained model’s image-voxel attention to associate relevant regions across these inputs. However, high total attention on local tokens does not necessarily mean that a voxel attends to the correct body part. The main paper also does not clearly explain how the initial sparse voxel coordinates are generated or whether SFC-Attn is applied during sparse-structure generation. These details would help clarify how the method incorporates complementary geometry across different poses, beyond transferring appearance cues.

  2. The experiments do not sufficiently establish the benefit of hard reference selection. Table 3 compares VRS routing with random and opposite routing, but these comparisons do not show whether selecting a single image is preferable to combining information from multiple images using the same VRS scores. A comparison with soft VRS-weighted fusion on the same backbone would help isolate the contribution of hard selection. The geometry, texture, and KNN ablations in the main paper are also only qualitative, making their individual contributions difficult to assess.

Questions for authors
  1. How are the initial sparse voxel coordinates generated in UMI3D? Is SFC-Attn applied during sparse-structure generation as well as the subsequent geometry and texture stages?

  2. For the pose-inconsistent examples in Figure 5, could the authors visualize the selected reference for each voxel together with the corresponding image-patch attention? Comparisons with single-image generation using each input individually would also help identify the complementary geometry contributed by the additional references.

  3. Could the authors compare the proposed VRS-based routing with averaging each voxel’s per-image cross-attention outputs using normalized VRS weights? Quantitative results for the geometry, texture, and KNN ablations would also help assess their individual contributions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Mwy

W1(a) & Q2(a): Visualization of voxel–image correspondence

We add Fig. 9 in the main paper to visualize voxel–image associations during geometry generation (Stage 2) and texture generation (Stage 3).

For each voxel, we average its VRS across transformer blocks at an intermediate denoising step; the voxel is colored according to the reference image with the highest mean VRS. For example, the dragon’s front-facing body is primarily associated with the blue reference, while its wing and rear-facing regions are mostly associated with the orange reference. These spatially coherent assignments qualitatively illustrate how VRS routes different regions to the images that provide relevant visual evidence.

W1(b) & Q1: Sparse voxel coordinate generation

As described in Supp. A.3.1, sparse structure generation follows the TRELLIS.2 baseline conditioning strategy for the first 75% of denoising steps, using only the anchor image. During the remaining 25%, we replace every cross-attention layer with SFC-Attn to incorporate multi-view cues.

This schedule helps maintain a globally coherent structure aligned with the anchor image while allowing other views to refine the structure.

W2(a) & Q3(a): Comparison with soft VRS-weighted fusion

We evaluate soft routing by normalizing VRS scores across reference images and using them as fusion weights. As shown in the fourth row in the following table, this strategy performs similarly to random routing and is outperformed by hard VRS routing on all three metrics. A possible reason is that attention scores cannot be directly interpreted as calibrated fusion weights. Simply normalizing them may therefore misestimate each reference image’s contribution.

Hard VRS routing avoids weighted fusion and achieves the best results among the evaluated strategies. Qualitative comparisons are provided in Supp. A.5.2.

Routing strategyCLIP ↑DINO ↑LPIPS ↓
Baseline85.9572.990.3413
Random CA Routing85.6172.360.3423
Opposite VRS Routing84.4269.630.3439
Soft VRS-Weighted Fusion (New)85.7772.350.3435
VRS Routing (Ours)86.4274.620.3356

W2(b) & Q3(b): Quantitative ablation results

We add quantitative ablations of geometry routing, texture routing, and 3D-KNN to Tab. 2. The full model achieves the best results across all three metrics.

VariantCLIP ↑DINO ↑LPIPS ↓
Baseline85.9572.990.3413
w/o Geometry Routing86.0673.290.3401
w/o Texture Routing86.3373.840.3405
w/o 3D-KNN86.2973.920.3393
Full Model86.4274.620.3356

Q2(b): Comparison with single-image generation

We add Fig. 6 to compare UMI3D with single-image generation and the multi-image baseline. Single-image generation preserves details visible in its reference, but the baseline may disrupt them when combining multiple inputs. UMI3D retains these details while incorporating complementary information from other views.

As the discussion period is coming to a close, we wanted to kindly ask whether you have any further comments or concerns. We have added 3 new figures and 2 new tables into the revised manuscript according to your review, please find if them can help you for the concern.

If you feel that our rebuttal has addressed your concerns, we would be grateful for any feedback and would appreciate it if you would consider updating your score.

Thank you again for your time and consideration.

Reviewer_bdhhOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method that addresses 3D asset generation from multiple images of the same object with inconsistent appearance, pose, or local details. It adapts frozen image-to-3D models through voxel-wise image routing. The proposed Voxel Reference Score (VRS) measures attention mass assigned to local image tokens. SFC-Attn uses this score, followed by spatial KNN refinement, to select one conditioning image per voxel. Evaluation is conducted on a synthetic benchmark. The ablations, a user study, and qualitative demonstrations of additional applications in morphing demonstrate the effectiveness of the designs.

Critical strengths

This work addresses an important problem in 3D asset generation with a training-free method. In my own experiments, achieving 3D consistency with multi-image conditioning often requires carefully designed data augmentation during training, while training-free strategies that alternate the conditioning image tend to lose detail and cross-view consistency. I'm convinced by the visual results.

  • The paper is clear and easy to follow. From the problem formulation and analysis to the proposed method, the presentation is well organized, and I enjoyed reading it.

  • The pipeline and component designs are intuitive and reasonable. The attention analysis reveals an interesting finding: global tokens receive disproportionately high attention per token in TRELLIS.2. Using the distribution of attention between global and local tokens to guide image selection is an interesting direction, although its functional interpretation needs further validation.

  • The results provide meaningful evidence of improved output quality. The qualitative comparisons and the reported the best preference votes are encouraging. The random and opposite-routing ablations also suggest that VRS captures useful information for selecting conditioning images.

Critical weaknesses
  • The functional interpretation motivating VRS is insufficiently supported. The analysis shows that "global tokens receive disproportionately high attention per token in voxel-based models". However, Fig. 4 shows that replacing the knight's global tokens with those from a semantically unrelated desktop computer largely preserves the object's overall appearance. Thus, high global-token attention alone does not establish that these tokens primarily determine overall appearance, and the statement in Section 3.2 that “the backbone primarily consults global tokens to reason about the object's overall structure and appearance” appears insufficiently supported. Since VRS selects images based on their local-token attention, could the authors directly test what these local tokens contribute to appearance (by adding the following experiment)? For example, keep the knight's global tokens in Fig. 4 and replace its local tokens with those from the desktop computer. Do the reference regions pick up the computer’s colors or other appearance cues, and are they affected more than the non-reference regions? This would more directly test the assumption motivating VRS: regions visible in an input image rely more on its local tokens.
  • The relationship to ROAR-3D needs clarification (minor issue). Both ROAR-3D and UMI3D use token-wise view routing: each 3D token selects one reference image for cross-attention. UMI3D's main distinction is its training-free VRS, but the paper does not discuss with ROAR-3D. A clearer discussion and comparison would help establish what VRS adds to the existing routing idea.

If the authors can convincingly address my concerns about the motivation behind VRS, I will raise my score.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer bdhh

W1(a): Effect of global tokens on overall appearance

We agree that disproportionately high per-token attention to global tokens does not, by itself, establish that they primarily determine an object's overall structure and appearance. We have revised Insight 1 to clarify this distinction:

“Global tokens receive disproportionately high per-token attention, making them a prominent conditioning reference during generation.”

This revision highlights the role of global tokens in attention allocation without attributing specific semantic content to them. It does not change our core VRS formulation: VRS uses the attention mass allocated to local tokens as a proxy for visual support, rather than relying on their stored semantic information. The routing ablations in Tab. 4 and Fig. 8 provide empirical support for its effectiveness.

W1(b): Additional experiment on Insight 2

We add a more comprehensive token-swapping experiment using an image of a knight, a computer, and another image of the same knight with a different appearance. The knight image is used to condition Stage 1 voxel generation, keeping the sparse structure fixed. The results are shown in the updated Fig. 4, with further analysis in Supp. A.5.

  • Rows 1 and 2: Replacing the knight’s local tokens with the computer’s local tokens creates a mismatch between the sparse structure and the conditioning image. The resulting discrepancies across both front and back views suggest that local-token information is important for matching the reference to the generated structure.
  • Rows 1 and 3: Replacing only the local tokens with those from the red knight changes the front-facing color and geometry toward the source, while leaving much of the back region less affected. This provides qualitative evidence that local tokens have a stronger influence on regions represented in the reference than on regions without direct reference evidence.
  • Rows 3 and 4: Keeping the red knight’s local tokens fixed while replacing the global tokens primarily changes the non-reference regions, including the highlighted rear strap and back panel. This suggests that global tokens have a stronger influence on regions lacking direct reference evidence.

Overall, these comparisons provide qualitative support for the regional-dependence hypothesis underlying Insight 2 and the motivation for VRS.

W2: Relationship to ROAR-3D

ROAR-3D is related to our work because both methods assign different input images to different 3D tokens. The methods differ in the routing criteria: ROAR-3D trains a router based on the affinity between 3D tokens and mean-pooled image features, whereas VRS uses local-token attention mass from the frozen backbone, without router training or additional trainable attention modules.

A direct experimental comparison is currently unavailable because the official ROAR-3D code has not been released. We have added the discussion between ROAR-3D and our work in the Related Work section.

Thanks for the author's reply. I have read the authors' response and compared the revised manuscript with the original submission. I agree that the revised Fig. 4 provides some evidence that the attention mass allocated to local tokens could serve as a proxy for visual support. However, the second row of the original Fig. 4 also shows the limits of this support, as the generated appearance collapses even in the reference region when the global tokens carry no meaningful information.

I see VRS (Sec. 3.2) as the paper's main contribution, and the authors now provide a fairly thorough discussion of what the score represents and why it may reflect visual support. I still have some reservations about the practical applicability of VRS beyond the settings evaluated in the paper. Overall, I see this as a borderline case for acceptance.

Thank you for your response. We would like to clarify the second row of the original Fig. 4.

As illustrated in Insight 1, global tokens account for a substantial share of the attention mass, this phenomenon exists in both reference and non-reference regions. (But their shares are comparable across different regions, as further proved in the newly added Fig. 9.) This suggests that neither token type can determine global or local information by itself. Removing either type therefore distorts the overall attention distribution and leads to generation failure.

This does not contradict our insight: VRS and our method do not assume that local tokens exclusively encode local information or global tokens exclusively encode global information. Instead, they use the comparable attention scores to determine the routing targets for different regions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All the Reviewers

We thank all reviewers for their careful and constructive feedback. We are encouraged that they recognized the strengths of our work:

  • 1. The importance of the problem and the value of a training-free approach (Reviewers bdhh, 4Mwy, and nV6m).
  • 2. The clarity of the presentation, the intuitive method design, and the promising results (Reviewers bdhh and 4Mwy).
  • 3. The contribution of the benchmark for unconstrained multi-image 3D generation (Reviewer nV6m).

We have carefully addressed the reviewers’ comments in our point-by-point responses and revised the manuscript accordingly. The appendix has been included after the main paper, and all changes in the revised manuscript are highlighted in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.