acceptodds
Under review as a conference paper at ICLR 2027

SUPRA-Tex: Super-Aligned 3D Texture Generation

17 views

Abstract

Single-image 3D texture generation aims to dress a given 3D mesh with relightable materials while faithfully preserving the visual details of a reference image. Existing methods either introduce artifacts when fusing inconsistent multi-view predictions or struggle to preserve fine details through implicit image-feature conditioning. To resolve these challenges, we introduce SUPRA-Tex, built on the principle of decoupling visible-material estimation, surface alignment, and observation-guided 3D completion. This division of labor allows the 3D generator, trained on limited 3D data, to focus its capacity on preserving visible evidence and completing unobserved surfaces. Specifically, we first train an MLLM-based image editing model to predict lighting-separated PBR maps in the reference view. We then project these maps onto the visible surface and encode them with the pretrained material VAE encoder into native observation latents that share the generation target's latent space. An interleaved flow transformer holds these latents as fixed, read-only anchors while alternating unobserved-region and full-surface updates to generate complete material latents. This design achieves super-alignment with the reference, delivering What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces. Trained solely on public datasets, SUPRA-Tex achieves SOTA performance in both geometry-conditioned texturing and end-to-end asset generation, matching or even exceeding leading commercial systems on real photographs. Code and models will be released.

Primary area:
generative models
Keywords:
3D Generation, Texture Generation, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Zf4xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SUPRA-Tex decomposes reference-faithful 3D texturing into visible-material estimation, surface alignment, and full-surface completion. An image-editing model predicts reference-view PBR maps, which are filtered, projected onto the mesh, and encoded with a frozen material VAE. A trained 3D flow transformer completes the materials using read-only observation memory and alternating unobserved-cell/full-surface updates. The paper evaluates given-geometry texturing and a multi-stage image-to-3D pipeline, and demonstrates additional-view conditioning and material editing.

Critical strengths
  1. Clear and compelling insight. I particularly appreciate and agree with the task decomposition. With limited 3D training data, using a specialized image model to estimate visible PBR materials, explicit projection to establish surface correspondence, and a 3D model to complete unseen regions is a clear and sensible division of labor.
  2. Well-motivated methodological design. The projection pipeline includes foreground-mask erosion and local depth-discontinuity filtering, which help reject unreliable correspondences near surface boundaries and should reduce contamination from mixed pixels and projection errors. Within the flow transformer, alternating updates between unobserved cells and the full surface is also a reasonable design for preserving fine details while maintaining global consistency.
  3. Convincing experiments and useful applications. The quantitative evaluation and extensive visual results provide convincing evidence of the method’s effectiveness. The examples generally show high-quality materials and faithful reference details. The framework supports texturing a given mesh, image-to-3D generation without a user-provided mesh, additional-view conditioning, and image-based material editing.
Critical weaknesses
  1. Training–inference mismatch in observations. Appendix B.3 uses GT PBR observations during 3D training, while inference uses frontend predictions. Whole-condition dropout does not reproduce structured prediction errors. The sensitivity to this mismatch, and the benefit of training with predicted or mixed observations, remain insufficiently characterized.
  2. Recovery from partial-encoding distortion is not directly quantified. E.2 documents reconstruction degradation when encoding partial rather than complete materials. Since the VAE remains frozen, the relevant question is how much of this degradation the trained completion model compensates for. E.4 offers encouraging early-learning evidence, but not a matched comparison of direct partial decoding and final completion on identical observed surfaces.
  3. Metadata contributions are not isolated. Coverage and boundary distance are well motivated, but their individual and complementary benefits are not established by separate ablations, especially for partially observed cells.
Questions for authors

My current assessment is weak accept. The following points are the main remaining uncertainties in my assessment. Focused evidence addressing them could move my recommendation toward a clear accept.

  1. Training–inference mismatch. How sensitive is the method to using GT PBR observations during training but predicted observations at inference? In particular, does training with predicted or mixed GT/predicted observations improve material quality when evaluated with predicted observations? A controlled comparison would help establish whether the current training strategy adequately handles this mismatch.
  2. Recovery from partial-encoding distortion. To what extent does the trained completion model compensate for the reconstruction degradation associated with partial encoding in E.2? A paired comparison of direct partial decoding and final completion on identical observed surface voxels, using matched material metrics, would address this concern. Consistent improvement on a small representative set would be sufficient; matching complete-input reconstruction is not expected.
  3. Contribution of observation metadata. Do coverage and boundary distance provide measurable, complementary improvements? Matched ablations removing each descriptor and both together would help establish their contribution, particularly for partially observed cells.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Zf4x (1/4)

Thank you for the careful review and for your support of the task decomposition. We ran a dedicated experiment for each of your three questions. The revised version is uploaded, with changes marked in blue. Table numbers below follow the revised version, where the ablation table is split into Tables 4 and 5 and later tables shift by one.

Metrics. Except for Table C3, the tables below compare generated materials with GT point by point on voxels, which allows per-region statistics: albedo is reported as PSNR in Q1 (dB, higher is better) and as MAE elsewhere (0–255 scale, lower is better; MAE in the main-text tables of the paper is on a 0–1 scale). These voxel-level values are not on the same footing as the rendering metrics of Table 1 and are meant only for comparing rows within a table.


Q1 / W1: Training–inference mismatch in observations

A1: Replacing GT observations with predicted ones lowers full-surface albedo PSNR by 4.20 / 2.19 dB (DTC / Complex). This gap comes mainly from frontend prediction error; the projection pipeline itself accounts for only 0.09 / 0.18 dB. When training is continued from the paper model for 16K steps, none of the three schemes that introduce predicted observations is a net improvement: predicted-only training is clearly worse; mixed observations are on par under predicted observations and worse under GT observations; continuous interpolation gains up to 0.20 dB under predicted observations but loses 0.82 dB under GT observations on DTC. We therefore keep training with GT observations only. Replacing the 2D frontend with a stronger one improves both visible and unseen surfaces (Table C3).

Design rationale. SUPRA-Tex assigns the estimation of visible materials to a specialized image model; the 3D model only needs to follow these observations and complete the rest of the surface. Training with predicted observations would require the 3D model to learn, from limited 3D data, to doubt and correct the observations, which departs from this division of labor and is harder to learn. We therefore train with GT observations only (Appendix B.3). As you point out, whole-condition dropout does not reproduce structured prediction errors, and whether this choice is adequate has to be tested.

(1) Effect of the mismatch. For the same paper model, replacing GT observations with frontend-predicted observations lowers full-surface albedo PSNR by 4.20 dB on DTC-Tex and 2.19 dB on Complex-Tex (first rows of Tables C1 and C2). To separate the sources of error, we render the GT materials into images and pass them through the same projection path as at inference; the drop is then only 0.09 and 0.18 dB. The projection pipeline itself therefore causes little loss, and the gap comes mainly from frontend prediction error.

(2) Controlled comparison. Starting from the paper model, we continue training three branches:

  • Mixed: each view uses GT observations with probability one half and predicted observations otherwise.
  • Predicted only: predicted observations only.
  • Continuous interpolation: following the error-recycling idea of Stable Video Infinity (Li et al., ICLR 2026), the observations of each view are interpolated continuously between GT and prediction, with the interpolation strength drawn at random from 0–1.

The predicted observations are obtained by running the frontend on rendered reference images of the training assets; these assets are not in the frontend's training data. The control branch starts from the same paper model and continues training with GT observations only. Data, number of steps (16K), batch size, and learning rate are identical for the four branches. The original paper model is listed as the reference before continued training. Table C1 evaluates with predicted observations, the setting you ask about; Table C2 evaluates with GT observations.

Table C1: Evaluation with predicted observations (full surface; Alb.: albedo PSNR in dB; Met./Rough.: MAE on a 0–255 scale; Cplx: Complex-Tex)

Observations used in trainingDTC Alb. ↑DTC Met. ↓DTC Rough. ↓Cplx Alb. ↑Cplx Met. ↓Cplx Rough. ↓
GT only (paper model)14.9210.1923.4314.5730.0929.39
GT only (continued training, control)14.909.9223.2114.4429.8629.18
Mixed14.919.6823.1514.5430.5630.43
Predicted only14.547.2636.7613.9433.0838.13
Continuous interpolation14.939.8523.2014.6429.4129.14
AuthorsOpenReview ↗

Response to Reviewer Zf4x (2/4)

(Q1 / W1, continued)

Table C2: Evaluation with GT observations (full surface; columns as in Table C1)

Observations used in trainingDTC Alb. ↑DTC Met. ↓DTC Rough. ↓Cplx Alb. ↑Cplx Met. ↓Cplx Rough. ↓
GT only (paper model)19.122.098.8816.768.3510.25
GT only (continued training, control)19.282.598.1916.718.3710.78
Mixed18.373.2211.3516.758.7111.01
Predicted only15.826.7034.1214.4328.6431.88
Continuous interpolation18.462.4911.3316.928.9110.91

Each branch is compared with the control branch:

  • Predicted only: clearly worse. Evaluated with predicted observations, albedo drops by 0.36 / 0.50 dB and roughness error rises by 13.55 / 8.95 (DTC / Complex); only the metallic error on DTC is lower. Evaluated with GT observations, all six metrics are worse than the control.
  • Mixed: evaluated with predicted observations, it is close to the control, but metallic and roughness errors on Complex rise by 0.70 / 1.25. Evaluated with GT observations, albedo on DTC drops by 0.91 dB, and metallic and roughness errors rise on both datasets.
  • Continuous interpolation: evaluated with predicted observations, all six metrics are slightly better than the control (albedo +0.03 / +0.20 dB; metallic and roughness errors lower by at most 0.45). Evaluated with GT observations, albedo on DTC drops by 0.82 dB and roughness error rises by 3.14.

No scheme is a net improvement: the only gain under predicted observations is small (continuous interpolation, at most 0.20 dB) and comes with lower fidelity under GT observations. We therefore keep GT-only observations as the default training setting.

(3) Replacing the 2D frontend. Table C3 uses the rendering metric of the paper (16 views, shaded PSNRfg, the column of the same name in Table 1) and splits the surface by visibility in the reference image. The visible and unseen parts are each computed per view and then averaged, and the views entering the average differ, so the "All" column is not a weighted average of the two and can be lower than both; please compare within a column.

Table C3: Shaded PSNRfg by visibility in the reference image (paper setting, 25 sampling steps; dB, higher is better)

MethodDTC All ↑DTC Visible ↑DTC Unseen ↑Cplx All ↑Cplx Visible ↑Cplx Unseen ↑
Pixal3D* (materials generated directly by the 3D generator)20.4721.5220.1919.4420.2219.48
Ours, paper frontend22.3423.9821.4220.6421.9920.09
Ours, Qwen frontend (3D model unchanged)23.6825.9722.4721.3223.2120.52

The Qwen frontend replaces the base model, FireRed-Image-Edit-1.1, with Qwen-Image-2.1, with the same training objective and data as the paper frontend. Table C3 shows three things:

  1. Our pipeline is more accurate on the surfaces visible in the reference image: we are 2.46 / 1.77 dB above Pixal3D* on the same geometry (DTC / Complex).
  2. A stronger frontend gives better results, and not only on visible surfaces: with the 3D model unchanged, the Qwen frontend adds a further 1.99 / 1.22 dB on visible surfaces and 1.05 / 0.43 dB on unseen surfaces.
  3. The remaining headroom has a clear upper bound: with GT observations the full surface reaches 24.65 / 21.76 dB, which is 2.31 / 1.12 dB above the paper frontend and only 0.97 / 0.44 dB above the Qwen frontend.

We will continue to improve the frontend and investigate training strategies that help the 3D generator adapt to imperfect predicted observations, including the continuous interpolation above and forcing-based approaches (Chen et al., NeurIPS 2024; Huang et al., NeurIPS 2025).

Revision. See the revised version: Appendix J.3 (Tables 16 and 17) gives this comparison of training observations, Appendix J.2 (Tables 14 and 15) gives the visibility-split results, and Table 5 includes the Qwen frontend. The limitations (Appendix G.2) now state that the 3D model is trained with GT observations only and that inference quality is bounded by frontend accuracy.

AuthorsOpenReview ↗

Response to Reviewer Zf4x (3/4)

Q2 / W2: Recovery from partial-encoding distortion

A2: Most of it. As you suggested, we ran a paired comparison on identical observed voxels: on both test sets and all three channels, completion removes 66–82% of the partial-encoding distortion (in MAE).

Setup. We follow the protocol of Appendix E.2: GT geometry, cameras, and GT observations, the same visibility filtering, and the frozen material VAE. E.2 shows three examples on Complex-Tex only; here we report statistics over the two full test sets, DTC-Tex and Complex-Tex. Three settings are compared: complete-material encoding followed by decoding (reference), direct decoding of the partial latents, and decoding of the completion model's output. All three use the same objects and GT materials and are scored on the same observed voxels.

Table C4: MAE on observed voxels (0–255 scale, lower is better; "After completion" is the mean over 3 sampling seeds; distortion removed = (direct partial decoding − after completion) / (direct partial decoding − complete encoding))

Test setChannelComplete encoding, decoded (reference)Direct partial decodingAfter completionDistortion removed
DTC-TexAlbedo1.988.454.0268.5%
Metallic0.170.710.2781.5%
Roughness1.615.712.9168.3%
Complex-TexAlbedo3.1010.385.6065.7%
Metallic1.233.461.8671.7%
Roughness1.566.642.6977.8%

Results. On both test sets and all three channels, the error after completion is clearly lower than that of direct partial decoding, and the 95% confidence intervals of the reductions exclude 0. In albedo PSNR, the metric used in E.2, DTC-Tex rises from 28.1 to 33.1 dB (40.1 dB for complete encoding) and Complex-Tex from 25.8 to 28.4 dB (34.1 dB for complete encoding). The conclusion is unchanged when the observations go through the 2D projection path used at inference: on Complex-Tex the albedo error after completion is 5.69, close to the 5.60 in Table C4.

Why the model can compensate. The completion model generates all surface cells, including the observed ones: the training loss covers all surface tokens, the target is the latent of the complete-material encoding, and the observations are not pasted back at inference. During generation, the observed tokens are therefore refined from the partial encoding toward the complete encoding. This reduces encoding distortion; the model has no objective for correcting the material estimate itself. The end of E.2 notes that partial encoding leaves "room for refinement during full-surface generation"; Table C4 quantifies this refinement.

Revision. See Appendix J.4 (Table 18) of the revised version; Appendix E.2 refers to it as the cohort-level statistics for its three examples.

Reviewer_d9JzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SUPRA-Tex generates PBR textures for a mesh from a reference image. It predicts 2D material maps, projects them onto the visible surface, and encodes these observations as conditions for a 3D texture generator. The observation features stay fixed, while successive blocks alternate between updating the unseen regions and the whole surface. Experiments show better rendering and appearance consistency than the compared baselines on both ground-truth and generated meshes, as well as better material-attribute scores in the ground-truth mesh setting.

Critical strengths

The results look good, especially the lettering and small patterns transferred from the reference image. The method also works on generated meshes, which makes the results more useful in practice. The material-content and oracle experiments provide further evidence about how the input observations affect the output. The architecture keeps the observation features fixed and interleaves updates to the unobserved regions with updates to the full generated surface.

Critical weaknesses

The gains over joint-update attention in Table 4 are marginal. A baseline that keeps observations fixed and updates the full generated surface at every block is missing, making it unclear whether alternating the updates accounts for the improvement.

I also have concerns about novelty. The observation-conditioning design appears straightforward, and its contribution beyond the 2D PBR frontend is not sufficiently demonstrated. Much of the improvement could come from better 2D PBR predictions. The fixed/mean-MR and oracle tests are helpful, but they do not show how much the proposed completion architecture adds when the quality of those predictions changes.

Questions for authors
  1. With observations fixed, how does updating the full generated surface at every block compare with the proposed alternating updates? Please use the same frontend, observed regions, model depth, training schedule, and sampling settings, and report both quality and runtime. The current timing comparison uses 10K and 30K checkpoints, so it does not answer this question.

  2. How much of the improvement comes from the 2D PBR predictor? Please compare the proposed completion model with the baseline using the same PBR inputs, and repeat this comparison with another predictor. Also, how are conflicting material estimates from overlapping views combined? Can the model correct an incorrect observation?

  3. Please report errors separately for observed and unseen surfaces. Please release the test asset IDs and describe the category, material, and texture distribution. For metallic and roughness errors, please report the fraction of the ground-truth foreground covered by the intersecting masks used for evaluation.

  4. In Table 3, SUPRA-Tex uses an additional synthesized back view, while the commercial systems receive only the original reference image. Please add a single-reference comparison for SUPRA-Tex, or provide the same back view to the other methods where their interfaces support it.

Please also report end-to-end texturing time and peak memory against a representative baseline on the same hardware and resolution. List frontend and sampling costs separately, and clarify whether the reported total includes the full pipeline.

9 comments
AuthorsOpenReview ↗

Response to Reviewer d9Jz (1/5)

Thank you for the careful review and for your positive assessment of the result quality, in particular the preservation of lettering and small patterns from the reference image and the fact that the method also works on generated meshes. The controls you requested are the experiments needed to separate the contribution of each component, and we have run all of them. The revised version is uploaded, with changes marked in blue. Table numbers below follow the revised version, where the ablation table is split into Tables 4 and 5 and later tables shift by one.

Quality metrics in the tables below follow Tables 4 and 5 of the paper; MR is the mean of metallic and roughness MAE, and "shaded PSNR" in the text refers to the Shad. column.


Q1 / W1: Full-surface update at every block vs. alternating updates

A1: Alternating (interleaved) updates do not improve quality; their role is to save computation. Under matched training, the three designs that keep the observation stream have similar quality (shaded PSNR within 0.1 dB, comparable MR MAE), and alternating updates sample fastest: 5% less time than the new control and 16% less than joint-update attention. What is saved is the observation-state updates and, in half of the blocks, the queries from observed cells; unobserved cells are still updated in every block.

Control setup. The new control changes one thing: observations stay fixed, and alternating updates are replaced by a full-surface update at every block. We also trained residual injection under the same settings (no observation stream; the same observation latents are added to the observed cells at every layer) and include it in Table B1. The four variants share the frontend, observed regions, model depth, training data and schedule, and sampling settings. To limit training time, all are trained for 10k steps. The architecture rows of Table 4 in the original submission were 30k-step results, so their numbers differ slightly.

Table B1: Quality and cost of each variant under matched training (timing: 15 objects drawn at a fixed interval from each test set, 30 in total; for each object and architecture, 2 warmups and 3 timed runs, median, then mean over objects; time per step is the time of one 25-step sampling run divided by 25; the four variants are timed in alternation on the same H800 and the same objects; peak sampling memory is the mean over objects of the per-object peak)

VariantDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓Time/step ↓Peak mem. ↓
Residual injection (no observation stream)21.3218.630.07620.3117.210.1470.252 s12.6 GiB
Joint-update attention (observations updated)22.2819.490.04420.5418.060.0960.340 s12.5 GiB
Read-only observations, full-surface update at every block (new control)22.2819.490.04220.6118.040.0960.302 s12.4 GiB
Ours: read-only observations + interleaved updates22.3419.580.04320.5917.980.0950.287 s12.1 GiB

Quality. With observations fixed, updating the full surface at every block matches joint-update attention. Relative to the new control, alternating updates change shaded PSNR by +0.06 / −0.02 dB and albedo PSNR by +0.09 / −0.06 dB (DTC / Complex), with comparable MR MAE. Residual injection is 1.0 / 0.3 dB below ours in shaded PSNR, and its MR MAE is 1.5–1.8 times ours. For reference, the standard deviation across sampling seeds of the same model is 0.08 / 0.06 dB in shaded PSNR and 0.10 / 0.08 dB in albedo PSNR. The update scheme therefore does not affect quality; the larger quality differences come from native observation conditioning itself and from observation quality (Tables B2 and B3 under Q2).

Runtime. With observations fixed, alternating updates are 5% faster than a full-surface update at every block, because each forward pass evaluates about 17% fewer query tokens (Appendix D.3 of the paper). Relative to joint-update attention, fixing the observations saves 11%, and adding alternating updates brings the total saving to 16%. Residual injection has no observation tokens and samples fastest (12% less time than ours). Peak sampling memory is similar across the four variants and lowest for ours. The object set differs from Table 10 of the paper (all 210 objects), so absolute values are not directly comparable.

Revision. See Tables 4 and 5 and Sec. 4.3 of the revised version. Table 4 is Table B1 (matched 10k-step training, with sampling time); Table 5 keeps the observation ablation with the final 30k-step model. Sec. 4.3 drops the 0.15 / 0.16 dB albedo statement and reports the sampling-seed standard deviation. The introduction, the contribution list, and Sec. 3.4 now present the fixed observation stream and the alternating updates as an efficiency design.

AuthorsOpenReview ↗

Response to Reviewer d9Jz (2/5)

Q2 / W2: Contribution of the 2D predictor and of the completion model

A2: The gain does not come from the 2D frontend alone. With the same input and the same training, residual injection is 1.0 / 0.3 dB below ours in shaded PSNR and its MR MAE is 1.5–1.8 times ours (Table B1). With the 3D model unchanged, a better frontend gives better overall results (Table B3).

Comparison with the same predicted albedo as input. Pixal3D* and TRELLIS.2 accept only RGB images, so we give them the albedo predicted by the frontend; metallic and roughness are still generated by each method (official weights, no retraining). We use the same prediction. As requested, we repeated the comparison with Pixal3D* using another predictor: the Qwen frontend replaces the base model of the 2D frontend, FireRed-Image-Edit-1.1, with Qwen-Image-2.1, with the same training objective and data as the paper frontend.

Table B2: Comparison with the same predicted albedo as input (the two rows with the original reference image as input are from Table 1 of the paper)

MethodInputDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓
Pixal3D*Original reference image20.4717.600.06419.4417.250.133
Pixal3D*Albedo predicted by the paper frontend17.5816.090.11417.4216.200.216
Pixal3D*Albedo predicted by the Qwen frontend17.7516.350.11617.5716.250.211
TRELLIS.2Original reference image19.7316.900.06118.4116.170.121
TRELLIS.2Albedo predicted by the paper frontend16.5314.960.12715.9014.650.229
OursPrediction of the paper frontend22.3419.550.04220.6418.120.096
OursPrediction of the Qwen frontend23.6821.140.04121.3218.460.095

With the same predicted albedo as input, our shaded PSNR is higher than that of Pixal3D* by 4.8 / 3.2 dB (paper frontend) and 5.9 / 3.8 dB (Qwen frontend), and higher than that of TRELLIS.2 by 5.8 / 4.7 dB. Given the predicted albedo, the baselines score 1.9–3.2 dB lower than with the original reference image: off-the-shelf generators conditioned on image features cannot directly use pixel-aligned material values. When the predictor improves, Pixal3D* gains only 0.17 / 0.15 dB, while we gain 1.34 / 0.68 dB.

A control with the same input and matched training. Pixal3D* and TRELLIS.2 were not trained for albedo input. Residual injection in Table B1 provides a control with the same input and the same training: the same 3D model and the same observation latents, with only the conditioning changed to layer-wise residual injection. Its shaded PSNR is 1.0 / 0.3 dB below ours and its MR MAE is 1.5–1.8 times ours; this difference comes from how the observations are injected.

Why native observation encoding and generative completion are both needed. More direct alternatives do not work either. Encoding and decoding the projected observations directly, without generation, gives an albedo MAE of 8.45 / 10.38 on the observed voxels; after completion it is 4.02 / 5.60 (0–255 scale, GT observations). Replacing the observed part of the generated result with the encoded observations removes this improvement (on a subset of DTC-Tex: 3.04 after completion, 8.22 after replacement). Native observation encoding and observation-guided completion therefore work as a whole: without the former, PBR cannot enter the generator; without the latter, partial observations do not become complete, clean materials.

When prediction quality changes. In Table B3 our 3D model is unchanged and only the quality of the observations varies. The first row, Pixal3D* with the original reference image, is given for reference.

Table B3: Observations of different quality (the Pixal3D* row is from Table 1 of the paper and the GT observation row from Table 5)

MethodObservation sourceDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓
Pixal3D*Original reference image20.4717.600.06419.4417.250.133
OursPaper frontend, run at 512 resolution21.4718.820.06020.1517.580.124
OursPaper frontend22.3419.550.04220.6418.120.096
OursQwen frontend23.6821.140.04121.3218.460.095
OursGT observations24.6522.170.01121.7620.630.030

Better observations give better results. Even at the lowest prediction quality, all three of our metrics are better than those of Pixal3D*.

AuthorsOpenReview ↗

Response to Reviewer d9Jz (3/5)

(Q2 / W2, continued)

On novelty. The novelty lies in the division of labor and in the interface it requires, not in the complexity of a module. A 2D model estimates the visible materials, projection places them on the surface, and the 3D model completes the rest. For this to work, the interface must (i) deliver the predicted values to the surface pixel by pixel and (ii) complete the unseen surface consistently with them. As shown above, handing the prediction to an existing generator as an image condition does not achieve (i) (Table B2), and decoding the projected observations directly, or pasting them back into the result, fails (ii). Encoding the observations into the latent space of the generation target, aligned with the surface, and letting the same model complete the whole surface meets both. Reviewers Zf4x and aneJ single out these two points as strengths: "a clear and sensible division of labor", and the surface-aligned observation latent as "an interesting design".

Combining overlapping views. Among the quantitative comparisons in the paper, only the commercial comparison in Table 3 uses two views (the reference image and a synthesized back view). Cells seen by both views keep the observation from the reference image; the back view fills only what the reference image does not see. The two views barely overlap: over the 45 in-the-wild cases, cells seen by both views account on average for 0.12% of the union of cells observed by either view (median 0.003%).

Can the model correct an incorrect observation? It is not designed to. The 3D model follows the observations and does not re-judge whether a material estimate is right, which is also why text and patterns in the reference image are preserved precisely; Appendix G.2 lists, as a limitation, that frontend material errors can be retained in the final asset. We also tried to make the 3D model adapt to erroneous observations: when evaluated with predicted observations, training with predicted observations was worse, and training with a mix of GT and predicted observations was no better (Appendix J.3, Table 16 of the revised version). Errors in the observations are therefore addressed in the frontend, and this route is effective: with the 3D model unchanged, a better frontend narrows the gap to GT observations from 2.31 / 1.12 dB to 0.97 / 0.44 dB in shaded PSNR (Table B3).

Revision. Table 5 of the revised version includes the Qwen-frontend row.


Q3: Observed vs. unseen surfaces, test set, region for material errors

A3: We lead on both observed and unseen surfaces. On surfaces not visible in the reference image (about 58% of the evaluated surface), our shaded PSNR and MR MAE remain the best among all methods, so the overall metrics are not carried by the visible region. The list of test assets is released through an anonymous repository, and the region used for metallic and roughness errors covers more than 99% of the GT foreground.

Observed and unseen surfaces. For every evaluation view of Table 1, we split the foreground pixels by whether the corresponding surface point is visible in the reference image; visibility is determined with the same filtering as our projection. The observed part accounts for about 41–42% and the unseen part for about 58–59%. Each part is computed per view and then averaged, and the views entering the average differ between the two parts, so the overall numbers in Table 1 are not a weighted average of the two and can be lower than both; please compare within a column. Tables B4 and B5 list our method and the four baselines that are best on the individual metrics; in every column, the remaining baselines are below the best baseline listed.

Table B4: Surfaces visible in the reference image

MethodDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓
TEXGen21.34––20.30––
LumiTex19.9518.960.22518.2216.830.217
TRELLIS.220.0717.830.06018.3416.960.116
Pixal3D*21.5219.640.06620.2218.730.133
Ours, paper frontend23.9822.520.04421.9920.790.094
Ours, Qwen frontend25.9725.110.04123.2121.830.091
Reviewer_aneJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  1. This paper studies the generation of complete, relightable PBR materials from a single reference image and a 3D mesh, focusing on preserving visible text/patterns and completing materials on unseen surfaces.
  2. The method first predicts albedo, metallic, and roughness maps with a 2D model, projects and encodes them as surface-aligned 3D observation latents, and then generates full-surface materials with a flow transformer using a read-only observation stream and interleaved updates.
  3. In the given-geometry and end-to-end settings, this paper reports better detail preservation, multi-view rendering quality, and PBR metrics than the listed baselines.
Critical strengths
  1. The paper decouples 2D material estimation, 3D surface alignment, and unseen-region completion. The motivation is clear, and the surface-aligned observation latent is an interesting design.
  2. The experiments cover shaded appearance, albedo, metallic, roughness, novel views, runtime, and architectural ablations, providing relatively broad evaluation.
  3. The videos show particularly strong visual alignment results and provide an attractive demonstration of preserving text, logos, and fine-grained patterns.
Critical weaknesses
  1. The attribution of the end-to-end results is limited. In Table 2, SUPRA-Tex and Pixal3D* share the same predicted geometry, while the other methods use their own geometry-generation pipelines. Therefore, this comparison more directly supports the material-generation advantage of SUPRA-Tex under shared geometry, rather than the overall capability of an independent end-to-end system.

  2. The contributions of the core components could be further disentangled. Table 4 generally supports the effectiveness of the proposed conditioning architecture, and the appendix states that the ablations use matched training settings. However, it remains unclear how much of the improvement comes from the read-only observation stream, the interleaved update schedule, or the quality of the PBR observations. The paper should distinguish architectural gains from observation-quality gains and report the stability or variance of the results where possible.

  3. Figure 2 mainly presents a module-level pipeline and does not sufficiently explain the observation stream, generation stream, read-only observation anchors, or interleaved update schedule. Although Figure 1 in Appendix A provides these details, the method remains difficult to understand from the main text alone. A simplified two-stream diagram and update schedule in the main paper would improve readability.

  4. The core idea of the paper is reasonable and well motivated: the 2D frontend directly predicts PBR maps from the 1024×1024 reference image, these maps are projected onto the 3D surface and encoded as native observed conditions, and a TRELLIS.2-style 3D generator then completes the full texture latent over unobserved regions. This provides an intuitive mechanism for achieving accurate alignment on observed regions while producing plausible completion on unseen surfaces. However, the source of the claimed “super-alignment” improvement is not fully isolated. The paper does not report a controlled 512×512 versus 1024×1024 resolution ablation, while the baselines follow method-specific resolution settings. Since higher-resolution material prediction may itself improve the preservation of small text, logos, and fine patterns, it remains unclear how much of the observed alignment gain comes from the proposed native observation conditioning and how much comes from using 1024×1024 PBR prediction. A resolution-controlled ablation, or at least a clearer discussion of this limitation, would make the “super-alignment” claim more convincing.

Questions for authors
  1. Is Table 2 intended primarily to demonstrate a material-generation advantage under shared geometry, or an overall end-to-end system advantage? Could the corresponding claim be stated more precisely?

  2. Could the authors further separate the contributions of the read-only observation stream, the interleaved update schedule, and the quality of the PBR observations in Table 4?

  3. Could the authors add a a controlled 512×512 versus 1024×1024 resolution ablation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aneJ (1/3)

Thank you for the careful review, and for your positive assessment of the decoupled design and the surface-aligned observation latents. We respond point by point below, with Q1–Q3 merged into the corresponding weaknesses. The revised version is uploaded, with changes marked in blue. Table numbers below follow the revised version, where the ablation table is split into Tables 4 and 5 and later tables shift by one.


W1 / Q1: What Table 2 supports

A1: Table 2 primarily shows a material-generation advantage under shared geometry, i.e., our controlled comparison with Pixal3D*. The revised version narrows the claim accordingly.

SUPRA-Tex and Pixal3D* use the same predicted geometry (produced by Pixal3D*; we claim no contribution in geometry) and differ only in how the texture is generated, so this pair is a controlled comparison. Pixal3D* is also the strongest baseline in the table overall: it ranks ahead of the other baselines on all 9 metrics. We improve on it on all 9, with four-view PSNR higher by 1.77 / 2.49 dB and LPIPS lower by 65% / 59% (DTC-Tex / Complex-Tex). The other methods generate their own geometry; those rows serve as system-level references.

Revision. See Sec. 4.2 of the revised version: the results paragraph first presents the controlled comparison with Pixal3D* under shared geometry and then lists the other methods as system-level references. The introduction and the conclusion now state that the geometry comes from Pixal3D*.


W2 / Q2: Contribution of each component

A2: The observation stream itself matters: under matched training, residual injection, which has no observation stream, is 1.0 / 0.3 dB below ours in shaded PSNR, with an MR MAE 1.5–1.8 times ours. Making the stream read-only and then interleaving the updates does not change quality (the three variants that keep the observation stream are within 0.1 dB in shaded PSNR); their role is to save computation: our sampling takes 16% less time than joint-update attention. Beyond that, quality mainly follows observation quality: with the 3D model unchanged, better observations give better results.

Two tables separate the three factors. The upper half of Table A1 fixes the input and changes the conditioning; the lower half fixes the 3D model and changes the observations. Table A2 fixes the observations and changes how they are injected and updated. Quality metrics follow Tables 4 and 5 of the paper: Shad. and Alb. are shaded and albedo PSNRfg, MR is the mean of metallic and roughness MAE, and Cplx is Complex-Tex; "shaded PSNR" in the text refers to the Shad. column.

Table A1: Effect of the conditioning and of observation quality (the Fixed MR, paper frontend, and GT observation rows are from Table 5 of the paper)

VariantDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓
Same predicted albedo as input, different conditioning
Pixal3D* (image-feature conditioning)17.5816.090.11417.4216.200.216
TRELLIS.2 (image-feature conditioning)16.5314.960.12715.9014.650.229
Same 3D model, different observations
Ours, Fixed MR21.9019.530.06419.9518.150.189
Ours, paper frontend (native observation conditioning)22.3419.550.04220.6418.120.096
Ours, Qwen frontend23.6821.140.04121.3218.460.095
Ours, GT observations24.6522.170.01121.7620.630.030
AuthorsOpenReview ↗

Response to Reviewer aneJ (2/3)

(W2 / Q2, continued)

(1) Native observation conditioning (Table 1 and upper half of Table A1). Pixal3D* conditions on image features. On the same given geometry and with its original RGB input, its shaded PSNRfg is 1.2–1.9 dB below ours (Table 1). Passing our frontend's predicted albedo to existing generators does not close this gap (upper half of Table A1; official weights, with metallic and roughness still generated by each method): Pixal3D* and TRELLIS.2 are 3.2–5.8 dB below ours in shaded PSNR, their MR MAE is more than twice ours, and both are below their own results with RGB input. Neither baseline was retrained for albedo input; the control with the same input and the same training is residual injection in Table A2.

(2) Observation quality (lower half of Table A1). With the 3D model unchanged, shaded PSNR increases in the order Fixed MR, paper frontend, Qwen frontend, GT observations. Fixed MR replaces only metallic and roughness with constants and keeps the albedo of the paper frontend. The Qwen frontend replaces the base model of the 2D frontend, FireRed-Image-Edit-1.1, with Qwen-Image-2.1, with the same training objective and data as the paper frontend.

(3) Read-only observation stream and interleaved updates (Table A2). The four variants use the same predicted PBR observations and are trained with the same data and settings; to limit training time and cost, all are trained for 10k steps. The architecture rows of Table 4 in the original submission were 30k-step results, so their numbers differ slightly.

Table A2: Four variants under matched training (sampling time per step is measured on 30 objects, with the four variants timed in alternation on the same H800; absolute values are not directly comparable to Table 10 of the paper, which uses all 210 objects)

VariantDTC Shad. ↑DTC Alb. ↑DTC MR ↓Cplx Shad. ↑Cplx Alb. ↑Cplx MR ↓Time/step ↓
Residual injection (no observation stream)21.3218.630.07620.3117.210.1470.252 s
Joint-update attention (observations updated)22.2819.490.04420.5418.060.0960.340 s
Read-only observations, full-surface update at every block22.2819.490.04220.6118.040.0960.302 s
Ours: read-only observations + interleaved updates22.3419.580.04320.5917.980.0950.287 s
  • Read-only observation stream: making the observation stream of joint-update attention read-only leaves quality unchanged and reduces sampling time per step by 11%.
  • Interleaved updates: switching further to interleaved updates changes shaded and albedo PSNR by less than 0.1 dB, with comparable MR MAE, and reduces sampling time by a further 5%, for 16% in total relative to joint-update attention.
  • Residual injection: there is no observation stream; the same observation latents are added to the observed cells at every layer through a shared linear layer. It samples fastest, but its shaded PSNR is 1.0 / 0.3 dB below ours and its MR MAE is 1.5–1.8 times ours.

Stability. Across 4 sampling seeds of the same model, the standard deviation of the dataset-mean shaded PSNR is 0.08 / 0.06 dB (DTC / Complex) and that of albedo PSNR is 0.10 / 0.08 dB, far smaller than the shaded PSNR differences in Table A1.

Revision. See Tables 4 and 5 and Sec. 4.3 of the revised version. The ablation table is split in two: Table 4 is the architecture ablation, which is Table A2 (four variants under matched 10k-step training, with sampling time), and Table 5 is the observation ablation with the final 30k-step model. Sec. 4.3 is rewritten accordingly: the statement that albedo PSNRfg improves by 0.15 / 0.16 dB is removed, and the sampling-seed standard deviation is reported. The introduction, the contribution list, Sec. 3.4, and the conclusion now present the read-only observation stream and the interleaved updates as an efficiency design: quality on par with joint-update attention, with 16% less sampling time.

AuthorsOpenReview ↗

Response to Reviewer aneJ (3/3)

W3: Two-stream structure and update schedule

A3: The revised version adds a simplified two-stream diagram to Sec. 3.4 (new Figure 3, simplified from the diagram in Appendix A), with the update schedule in its caption; the caption of Figure 2 points to it.

The diagram conveys two points.

Two streams. The observation stream O holds the observation latents of the observed cells and stays fixed throughout: each block only reads keys and values from it, and it issues no queries and receives no updates. The generation stream X covers all surface cells and is the stream that is updated.

Update schedule. There are 60 blocks, alternating between even and odd:

BlockQueries and updatesReads keys/values from
Even blocks (30)Unobserved cells onlyAll of O and X
Odd blocks (30)All surface cellsAll of O and X

W4 / Q3: Effect of the 1024 resolution

A4: Higher resolution helps our method; with both the reference image and the frontend reduced to 512, our reference-view PSNR is still about 2 dB above Pixal3D* at 1024, and still above TEXGen, which runs at 512.

Resolution of each method. All methods receive the same 1024 reference image and use it under their official settings. The resolution at which the reference image enters each network is:

  • 1024: Pixal3D*, TRELLIS.2, and MV-Adapter, the same as ours (MV-Adapter generates its multi-view images at 768).
  • 768: Step1X-3D and Ink3D.
  • 512: Hunyuan3D 2.1, LumiTex, MVPainter, UniTEX, and TEXGen.

After generation, MV-Adapter and Hunyuan3D 2.1 additionally upsample the generated views to 1536 and 2048 with RealESRGAN; the other methods apply no super-resolution under their official defaults. Super-resolution acts on the generated output and does not change the resolution at which the reference image is used.

Controlled experiment. The reference image is downsampled to 512 and the 2D frontend also runs at 512; geometry, 3D model, and sampling are unchanged.

Table A3: Shaded PSNRfg (dB, higher is better; rows 1, 2, and 4 are from Table 8 and Table 1 of the paper)

SettingReference view, DTC ↑Reference view, Cplx ↑16 views, DTC ↑16 views, Cplx ↑
Pixal3D*, 102421.8420.0620.4719.44
Ours, 1024 (paper setting)25.3722.9322.3420.64
Ours, reference image and frontend at 51223.7722.1521.4720.15
TEXGen, 51222.8121.9718.3318.25
  • Versus Pixal3D* (the strongest baseline overall): at 1024 for both, we are higher by 3.5 / 2.9 dB (reference view) and 1.9 / 1.2 dB (16 views). With ours at 512 and Pixal3D* still at 1024, we remain higher by 1.9 / 2.1 dB and 1.0 / 0.7 dB.
  • Versus TEXGen (the strongest baseline on the reference view, running at 512): at 512 for both, we are higher by 1.0 / 0.2 dB (reference view) and 3.1 / 1.9 dB (16 views).
  • Contribution of resolution itself: going from 1024 to 512 lowers our reference-view PSNR by 1.6 / 0.8 dB. Part of this drop may come from the 2D frontend, which was trained only at 1024 and predicts less accurately at another resolution. Even if the whole drop is attributed to resolution, it is less than half of our margin over Pixal3D* (3.5 / 2.9 dB).

Revision. See Appendix J.1 and Table 13 of the revised version, which contain this controlled experiment and the resolution settings of all methods.


Thank you again for your comments. We are happy to continue the discussion if anything remains unclear.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We are glad that the reviews recognize "a clear and sensible division of labor" (Zf4x), the surface-aligned observation latent as "an interesting design" (aneJ), and the preservation of "the lettering and small patterns" of the reference image (d9Jz).

In response, we conducted a substantial set of new experiments over the discussion period, including newly trained models, new training data, a re-evaluation of all methods, and a new human study. The revised version is uploaded, with changes in blue (new Figure 3, Tables 4–5, and Appendix J). In summary:

  • We trained four conditioning architectures under identical settings, including the control asked for by d9Jz: the observation stream accounts for the quality gain, and keeping it read-only with interleaved updates gives the same quality with 16% less sampling time (Table 4).
  • We swapped in a stronger 2D frontend and left the 3D model unchanged: shaded PSNR rises by 1.34 / 0.68 dB on DTC-Tex / Complex-Tex (Table 5).
  • We re-evaluated our method and all baselines separately on surfaces visible and not visible in the reference image: ours is the best on both (J.2).
  • We reran our pipeline at 512 resolution: it stays about 2 dB above Pixal3D⋆ at 1024 on the reference view (J.1).
  • We gave the five commercial systems the same synthesized back view and collected new human ratings: our results are still preferred over all five (J.6).
  • We continued training the 3D generator with three schemes that use predicted observations: none is a net improvement (J.3). A paired evaluation shows that completion removes 66–82% of the partial-encoding distortion (J.4).

Point-by-point responses are under each review. We are happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.