- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents UVRelief, a surface codec for representing human avatars in a compact latent space. The proposed method represents clothed human geometry as surface displacements relative to a fitted SMPL-X model in the UV space. A VAE is then trained to encode this representation into a compact latent space, followed by an image-conditioned flow-matching model for avatar generation. Experiments on reconstruction and generation demonstrate the effectiveness of the proposed representation.
Critical strengths
-
Simple and intuitive human-specific representation. The proposed method effectively leverages a human prior by representing complex clothed human geometry as displacements relative to the SMPL-X surface. Compared with directly modeling absolute 3D positions, UVRelief provides a more structured and compact representation.
-
Solid performance on both reconstruction and generation. The paper evaluates the proposed representation on both codec-level reconstruction and downstream image-conditioned avatar generation. The proposed codec achieves better reconstruction performance than TRELLIS and its human-specific finetuned variant, while the full generation pipeline also achieves strong quantitative performance, particularly in terms of PSNR and SSIM.
Critical weaknesses
-
The motivation and scalability of the proposed representation are not fully convincing. One of the main motivations of this work is that existing image-based codecs implicitly need to solve 3D reconstruction when learning the latent representation. However, UVRelief instead assumes access to complete textured 3D meshes and fitted SMPL-X models during training. In particular, part of the training data from MVHumanNet++ is first converted into 3D meshes using a multi-view reconstruction method, followed by SMPL-X fitting with OmniFit. Therefore, the proposed approach appears to move a significant part of the reconstruction difficulty from representation learning to data preprocessing. While this design indeed makes the latent representation easier to learn, it is unclear whether this trade-off is favorable in terms of data scalability.
-
The experiments do not sufficiently demonstrate the advantage of UVRelief itself for generative modeling. The paper argues that the proposed relative representation leads to a more learnable latent space. However, in the controlled encoder-side ablation, the reconstruction performance of the position-map baseline is already very close to that of UVRelief (23.35 vs. 23.46 PSNR). The much larger improvement is observed over the SIGMAN-like cross-attention baseline, which needs to obtain its latent representation from multi-view images, whereas UVRelief is constructed from complete 3D geometry. Therefore, this comparison does not fully isolate the benefit of the proposed relative representation. More importantly, a controlled quantitative generation comparison between UVRelief and the position-map representation is missing. Such an experiment would provide more direct evidence that UVRelief indeed results in a substantially more learnable latent space for downstream generative modeling.
Questions for authors
-
Could the authors provide more details about how TRELLIS was fine-tuned on the human-specific dataset, including the training setup and supervision?
-
Could the authors provide a quantitative generation comparison between UVRelief and the position-map baseline?
-
How sensitive is UVRelief to errors in SMPL-X fitting?
3 comments
Part 2
Q3. Sensitivity to SMPL-X fitting errors
We perturb the fit at three levels, rebuild the body consistently, re-bake the maps and re-run reconstruction on the 100 X-Humans subjects:
| level | shape sigma | pose sigma | PSNR | SSIM | LPIPS |
|---|---|---|---|---|---|
| 0 | -- | -- | 28.12 | 0.9770 | 0.0354 |
| 1 | 0.3 | 1.5 deg | 26.22 | 0.9719 | 0.0435 |
| 2 | 0.6 | 3.0 deg | 23.47 | 0.9604 | 0.0557 |
| 3 | 1.0 | 5.0 deg | 20.34 | 0.9434 | 0.0736 |
The codec indeed inherits the accuracy of the fit — a body-relative representation cannot do otherwise — and we now quantify this instead of only stating it in the limitations. Note that both the relief map and the position map receive the same perturbed body, so the comparison between encodings is unaffected by this dependence.
Ethics
We will add a short data statement: all scans come from public research datasets with consent for research use; our method produces avatars of the scanned subjects and could be misused for impersonation, so we will discuss the need for consent when applying it to new identities.
Changes promised for the revision
- Sec. 4.1: baseline grouping, SMPL-X provenance, TRELLIS fine-tuning details.
- Sec. 4.5: the controlled generation comparison.
- New section: out-of-distribution evaluation with voxel statistics.
- Limitations: preprocessing trade-off, data scale, sensitivity to fit errors.
- Ethics statement.
It's a little late for response, but it addresses my several concerns. Thanks for authors' effort on these extra experiments, and I decide to raise my rating to 3. I hope this method can be applied to larger dataset.
Part 1
Response to Reviewer Z3xP
We thank the reviewer for the constructive report and for pointing at the exact experiment that was missing. We have run it, together with an out-of-distribution evaluation and a fit-sensitivity study.
W1. Reconstruction-based preprocessing: does it just move the difficulty?
The reviewer is right that our pipeline needs an explicit surface before learning the latent space, and that this is a design choice with a cost. We would argue it is a favourable trade-off, for a reason that can be measured.
An explicit surface supplies geometric supervision: the latent is tied to a plausible 3D structure rather than to view-dependent appearance. A codec trained directly from multi-view images without such a reconstruction has no constraint on the unobserved geometry, so it tends to fit the input views and may produce shapes that are only consistent from those views. Our approach moves part of the difficulty into preprocessing, but the resulting latent is well defined in 3D, and the generation prior only has to model a bounded residual over the SMPL-X manifold.
On scalability, we note that the reconstruction step is exactly what recent human-specific datasets provide at scale, and that the cost is one-off per asset rather than per training run. We will state this trade-off explicitly in the limitations, including the option of using the million-scale reconstructed bodies released with SIGMAN.
W2 / Q2. The controlled generation comparison
Agreed, and now provided. One flow-matching prior per encoding, identical architecture, data, supervision and steps; only the encoder input changes:
The reviewer's observation that "the position map is already close" is exactly what we see on X-Humans (27.08 vs 25.99 for generation, 28.12 vs 27.72 for reconstruction). The new 4D-DRESS set shows where the two diverge: +2.51 dB for reconstruction and better FID for generation, on frames whose surface leaves the volume covered by the training bodies (0.82% of the surface for held-out X-Humans bodies versus 9.19% for the selected 4D-DRESS frames). This is the direct evidence that the relative encoding gives a more learnable latent space once the query leaves the training distribution.
We also agree that the cross-attention comparison confounds two factors (it must recover geometry from images), which is why we do not use it as the main evidence for the representation claim; it is reported as a different design point.
Q1. How TRELLIS was fine-tuned
We fine-tune only the second-stage VAE of TRELLIS, i.e. the sparse-voxel decoder that produces voxel-attached 3DGS from aggregated multi-view DINOv2 features. The first stage is left untouched: the voxel grid that conditions the decoder is computed directly from the GT geometry, exactly as in the public release. The fine-tuning uses the same 8k human meshes, the same renderer and the same image supervision as our model. The baseline therefore receives perfect geometry at its input, which is a favourable setting for it, and any remaining gap is due to the latent representation rather than to the input. We will state the optimisation details (steps, batch size, learning rate) in the appendix so that the comparison is reproducible.