- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SUPRA-Tex decomposes reference-faithful 3D texturing into visible-material estimation, surface alignment, and full-surface completion. An image-editing model predicts reference-view PBR maps, which are filtered, projected onto the mesh, and encoded with a frozen material VAE. A trained 3D flow transformer completes the materials using read-only observation memory and alternating unobserved-cell/full-surface updates. The paper evaluates given-geometry texturing and a multi-stage image-to-3D pipeline, and demonstrates additional-view conditioning and material editing.
Critical strengths
- Clear and compelling insight. I particularly appreciate and agree with the task decomposition. With limited 3D training data, using a specialized image model to estimate visible PBR materials, explicit projection to establish surface correspondence, and a 3D model to complete unseen regions is a clear and sensible division of labor.
- Well-motivated methodological design. The projection pipeline includes foreground-mask erosion and local depth-discontinuity filtering, which help reject unreliable correspondences near surface boundaries and should reduce contamination from mixed pixels and projection errors. Within the flow transformer, alternating updates between unobserved cells and the full surface is also a reasonable design for preserving fine details while maintaining global consistency.
- Convincing experiments and useful applications. The quantitative evaluation and extensive visual results provide convincing evidence of the method’s effectiveness. The examples generally show high-quality materials and faithful reference details. The framework supports texturing a given mesh, image-to-3D generation without a user-provided mesh, additional-view conditioning, and image-based material editing.
Critical weaknesses
- Training–inference mismatch in observations. Appendix B.3 uses GT PBR observations during 3D training, while inference uses frontend predictions. Whole-condition dropout does not reproduce structured prediction errors. The sensitivity to this mismatch, and the benefit of training with predicted or mixed observations, remain insufficiently characterized.
- Recovery from partial-encoding distortion is not directly quantified. E.2 documents reconstruction degradation when encoding partial rather than complete materials. Since the VAE remains frozen, the relevant question is how much of this degradation the trained completion model compensates for. E.4 offers encouraging early-learning evidence, but not a matched comparison of direct partial decoding and final completion on identical observed surfaces.
- Metadata contributions are not isolated. Coverage and boundary distance are well motivated, but their individual and complementary benefits are not established by separate ablations, especially for partially observed cells.
Questions for authors
My current assessment is weak accept. The following points are the main remaining uncertainties in my assessment. Focused evidence addressing them could move my recommendation toward a clear accept.
- Training–inference mismatch. How sensitive is the method to using GT PBR observations during training but predicted observations at inference? In particular, does training with predicted or mixed GT/predicted observations improve material quality when evaluated with predicted observations? A controlled comparison would help establish whether the current training strategy adequately handles this mismatch.
- Recovery from partial-encoding distortion. To what extent does the trained completion model compensate for the reconstruction degradation associated with partial encoding in E.2? A paired comparison of direct partial decoding and final completion on identical observed surface voxels, using matched material metrics, would address this concern. Consistent improvement on a small representative set would be sufficient; matching complete-input reconstruction is not expected.
- Contribution of observation metadata. Do coverage and boundary distance provide measurable, complementary improvements? Matched ablations removing each descriptor and both together would help establish their contribution, particularly for partially observed cells.
5 comments
Response to Reviewer Zf4x (2/4)
(Q1 / W1, continued)
Table C2: Evaluation with GT observations (full surface; columns as in Table C1)
| Observations used in training | DTC Alb. ↑ | DTC Met. ↓ | DTC Rough. ↓ | Cplx Alb. ↑ | Cplx Met. ↓ | Cplx Rough. ↓ |
|---|---|---|---|---|---|---|
| GT only (paper model) | 19.12 | 2.09 | 8.88 | 16.76 | 8.35 | 10.25 |
| GT only (continued training, control) | 19.28 | 2.59 | 8.19 | 16.71 | 8.37 | 10.78 |
| Mixed | 18.37 | 3.22 | 11.35 | 16.75 | 8.71 | 11.01 |
| Predicted only | 15.82 | 6.70 | 34.12 | 14.43 | 28.64 | 31.88 |
| Continuous interpolation | 18.46 | 2.49 | 11.33 | 16.92 | 8.91 | 10.91 |
Each branch is compared with the control branch:
- Predicted only: clearly worse. Evaluated with predicted observations, albedo drops by 0.36 / 0.50 dB and roughness error rises by 13.55 / 8.95 (DTC / Complex); only the metallic error on DTC is lower. Evaluated with GT observations, all six metrics are worse than the control.
- Mixed: evaluated with predicted observations, it is close to the control, but metallic and roughness errors on Complex rise by 0.70 / 1.25. Evaluated with GT observations, albedo on DTC drops by 0.91 dB, and metallic and roughness errors rise on both datasets.
- Continuous interpolation: evaluated with predicted observations, all six metrics are slightly better than the control (albedo +0.03 / +0.20 dB; metallic and roughness errors lower by at most 0.45). Evaluated with GT observations, albedo on DTC drops by 0.82 dB and roughness error rises by 3.14.
No scheme is a net improvement: the only gain under predicted observations is small (continuous interpolation, at most 0.20 dB) and comes with lower fidelity under GT observations. We therefore keep GT-only observations as the default training setting.
(3) Replacing the 2D frontend. Table C3 uses the rendering metric of the paper (16 views, shaded PSNRfg, the column of the same name in Table 1) and splits the surface by visibility in the reference image. The visible and unseen parts are each computed per view and then averaged, and the views entering the average differ, so the "All" column is not a weighted average of the two and can be lower than both; please compare within a column.
Table C3: Shaded PSNRfg by visibility in the reference image (paper setting, 25 sampling steps; dB, higher is better)
| Method | DTC All ↑ | DTC Visible ↑ | DTC Unseen ↑ | Cplx All ↑ | Cplx Visible ↑ | Cplx Unseen ↑ |
|---|---|---|---|---|---|---|
| Pixal3D* (materials generated directly by the 3D generator) | 20.47 | 21.52 | 20.19 | 19.44 | 20.22 | 19.48 |
| Ours, paper frontend | 22.34 | 23.98 | 21.42 | 20.64 | 21.99 | 20.09 |
| Ours, Qwen frontend (3D model unchanged) | 23.68 | 25.97 | 22.47 | 21.32 | 23.21 | 20.52 |
The Qwen frontend replaces the base model, FireRed-Image-Edit-1.1, with Qwen-Image-2.1, with the same training objective and data as the paper frontend. Table C3 shows three things:
- Our pipeline is more accurate on the surfaces visible in the reference image: we are 2.46 / 1.77 dB above Pixal3D* on the same geometry (DTC / Complex).
- A stronger frontend gives better results, and not only on visible surfaces: with the 3D model unchanged, the Qwen frontend adds a further 1.99 / 1.22 dB on visible surfaces and 1.05 / 0.43 dB on unseen surfaces.
- The remaining headroom has a clear upper bound: with GT observations the full surface reaches 24.65 / 21.76 dB, which is 2.31 / 1.12 dB above the paper frontend and only 0.97 / 0.44 dB above the Qwen frontend.
We will continue to improve the frontend and investigate training strategies that help the 3D generator adapt to imperfect predicted observations, including the continuous interpolation above and forcing-based approaches (Chen et al., NeurIPS 2024; Huang et al., NeurIPS 2025).
Revision. See the revised version: Appendix J.3 (Tables 16 and 17) gives this comparison of training observations, Appendix J.2 (Tables 14 and 15) gives the visibility-split results, and Table 5 includes the Qwen frontend. The limitations (Appendix G.2) now state that the 3D model is trained with GT observations only and that inference quality is bounded by frontend accuracy.
Response to Reviewer Zf4x (3/4)
Q2 / W2: Recovery from partial-encoding distortion
A2: Most of it. As you suggested, we ran a paired comparison on identical observed voxels: on both test sets and all three channels, completion removes 66–82% of the partial-encoding distortion (in MAE).
Setup. We follow the protocol of Appendix E.2: GT geometry, cameras, and GT observations, the same visibility filtering, and the frozen material VAE. E.2 shows three examples on Complex-Tex only; here we report statistics over the two full test sets, DTC-Tex and Complex-Tex. Three settings are compared: complete-material encoding followed by decoding (reference), direct decoding of the partial latents, and decoding of the completion model's output. All three use the same objects and GT materials and are scored on the same observed voxels.
Table C4: MAE on observed voxels (0–255 scale, lower is better; "After completion" is the mean over 3 sampling seeds; distortion removed = (direct partial decoding − after completion) / (direct partial decoding − complete encoding))
| Test set | Channel | Complete encoding, decoded (reference) | Direct partial decoding | After completion | Distortion removed |
|---|---|---|---|---|---|
| DTC-Tex | Albedo | 1.98 | 8.45 | 4.02 | 68.5% |
| Metallic | 0.17 | 0.71 | 0.27 | 81.5% | |
| Roughness | 1.61 | 5.71 | 2.91 | 68.3% | |
| Complex-Tex | Albedo | 3.10 | 10.38 | 5.60 | 65.7% |
| Metallic | 1.23 | 3.46 | 1.86 | 71.7% | |
| Roughness | 1.56 | 6.64 | 2.69 | 77.8% |
Results. On both test sets and all three channels, the error after completion is clearly lower than that of direct partial decoding, and the 95% confidence intervals of the reductions exclude 0. In albedo PSNR, the metric used in E.2, DTC-Tex rises from 28.1 to 33.1 dB (40.1 dB for complete encoding) and Complex-Tex from 25.8 to 28.4 dB (34.1 dB for complete encoding). The conclusion is unchanged when the observations go through the 2D projection path used at inference: on Complex-Tex the albedo error after completion is 5.69, close to the 5.60 in Table C4.
Why the model can compensate. The completion model generates all surface cells, including the observed ones: the training loss covers all surface tokens, the target is the latent of the complete-material encoding, and the observations are not pasted back at inference. During generation, the observed tokens are therefore refined from the partial encoding toward the complete encoding. This reduces encoding distortion; the model has no objective for correcting the material estimate itself. The end of E.2 notes that partial encoding leaves "room for refinement during full-surface generation"; Table C4 quantifies this refinement.
Revision. See Appendix J.4 (Table 18) of the revised version; Appendix E.2 refers to it as the cohort-level statistics for its three examples.
Response to Reviewer Zf4x (1/4)
Thank you for the careful review and for your support of the task decomposition. We ran a dedicated experiment for each of your three questions. The revised version is uploaded, with changes marked in blue. Table numbers below follow the revised version, where the ablation table is split into Tables 4 and 5 and later tables shift by one.
Metrics. Except for Table C3, the tables below compare generated materials with GT point by point on voxels, which allows per-region statistics: albedo is reported as PSNR in Q1 (dB, higher is better) and as MAE elsewhere (0–255 scale, lower is better; MAE in the main-text tables of the paper is on a 0–1 scale). These voxel-level values are not on the same footing as the rendering metrics of Table 1 and are meant only for comparing rows within a table.
Q1 / W1: Training–inference mismatch in observations
A1: Replacing GT observations with predicted ones lowers full-surface albedo PSNR by 4.20 / 2.19 dB (DTC / Complex). This gap comes mainly from frontend prediction error; the projection pipeline itself accounts for only 0.09 / 0.18 dB. When training is continued from the paper model for 16K steps, none of the three schemes that introduce predicted observations is a net improvement: predicted-only training is clearly worse; mixed observations are on par under predicted observations and worse under GT observations; continuous interpolation gains up to 0.20 dB under predicted observations but loses 0.82 dB under GT observations on DTC. We therefore keep training with GT observations only. Replacing the 2D frontend with a stronger one improves both visible and unseen surfaces (Table C3).
Design rationale. SUPRA-Tex assigns the estimation of visible materials to a specialized image model; the 3D model only needs to follow these observations and complete the rest of the surface. Training with predicted observations would require the 3D model to learn, from limited 3D data, to doubt and correct the observations, which departs from this division of labor and is harder to learn. We therefore train with GT observations only (Appendix B.3). As you point out, whole-condition dropout does not reproduce structured prediction errors, and whether this choice is adequate has to be tested.
(1) Effect of the mismatch. For the same paper model, replacing GT observations with frontend-predicted observations lowers full-surface albedo PSNR by 4.20 dB on DTC-Tex and 2.19 dB on Complex-Tex (first rows of Tables C1 and C2). To separate the sources of error, we render the GT materials into images and pass them through the same projection path as at inference; the drop is then only 0.09 and 0.18 dB. The projection pipeline itself therefore causes little loss, and the gap comes mainly from frontend prediction error.
(2) Controlled comparison. Starting from the paper model, we continue training three branches:
The predicted observations are obtained by running the frontend on rendered reference images of the training assets; these assets are not in the frontend's training data. The control branch starts from the same paper model and continues training with GT observations only. Data, number of steps (16K), batch size, and learning rate are identical for the four branches. The original paper model is listed as the reference before continued training. Table C1 evaluates with predicted observations, the setting you ask about; Table C2 evaluates with GT observations.
Table C1: Evaluation with predicted observations (full surface; Alb.: albedo PSNR in dB; Met./Rough.: MAE on a 0–255 scale; Cplx: Complex-Tex)