acceptodds
Withdrawn from ICLR 2027

Structure-Detail Decoupled Autoregressive Generation for Fast and High-Fidelity Virtual Try-On

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Virtual try-on (VTON) is a bi-conditional image generation problem that requires not only accurate person preservation but also faithful garment deformation and detail synthesis. Diffusion-based VTON methods can jointly model these factors in a compressed latent space, but suffer from high-frequency detail loss due to inherent latent compression, even with costly multi-step denoising. Recent visual autoregressive (VAR) models offer a promising alternative for high-quality generation with faster inference, yet remain unexplored for VTON due to the lack of effective bi-conditioning mechanisms. To bridge this gap, we first introduce VAR-VTON, a VAR-based VTON model that incorporates garment conditioning and structural guidance for efficient latent-space VTON. Despite its efficacy, latent-space generation still struggles to preserve fine-grained garment details. We argue that different VTON sub-tasks should be addressed in different representation spaces: structural synthesis such as garment warping and person layout is suited to the latent space, whereas fine-grained detail recovery should be tackled in the pixel space. Motivated by this insight, we further propose STAR-VTON, a Two-Stage AutoRegressive framework that builds upon VAR-VTON by decoupling latent-space structural synthesis from pixel-space detail recovery. Our idea is to resort to a matching-informed refiner to establish dense correspondences between the stage-one generation and the source garment to directly map fine-grained pixel-space details. Extensive experiments show that STAR-VTON achieves an impressive efficiency-fidelity trade-off: VAR-VTON runs at least faster than diffusion-based counterparts without degrading quality, and the pixel-space refiner effectively restores fine details and acts as a plug-and-play module that can benefit existing VTON approaches.

Primary area:
generative models
Keywords:
Virtual Try-On, Conditional Image Generation, Visual Autoregressive Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_723vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STAR-VTON, a two-stage try-on (subset of image editing with reference) pipeline, which mainly focuses on better trade-offs between quality and speed. Generally, stage 1 utilizes a VAR-based generator to produce a try-on "preview" and accomplish the detail with stage 2's refiner (based on an ROI extractor and a simple UNet). Overall, compared to the existing model, the proposed VAR-VTRON achieves better fidelity and faster inference. The refiner further improves garment-detail preservation and serves as a general, plug-in module for other virtual try-on models.

Critical strengths
  1. The entire pipeline is solid and effective in engineering applications. The refiner module indeed provides an extra rollback for complex detail areas, and it provides space to continuously simplify/accelerate the editing model.
Critical weaknesses
  1. Despite its intuitive design and empirical effectiveness, the proposed pipeline offers limited insight. In particular, the paper does not distinguish the cause of detail degradation (VAE compression, errors introduced by the generative editing model, and input resolution). It also assumes that reliable correspondences can be established despite degraded garment appearance, without sufficiently examining when this assumption holds or how matching errors affect subsequent refinement.

  2. VAR is efficient, but I can hardly find the internal relations between the VAR-based model and the detail degradation problem. It makes the whole system faster, while the associated VQ-VAE exacerbates the detail loss. From my perspective, the PiT-based method is sounder, which gives consideration to both efficiency and fidelity.

  3. Compared to the try-on model, it does perform competitively. However, comparisons with general editing models (FLUX klein, Lance, ...) are required as well.

Questions for authors

See Weaknesses.

Reviewer_eMwZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a visual auto-regressive modelling (VAR)-based virtual try-on (VTON) approach dubbed VAR-VTON. It leverages the superior inference speed of VAR along with its on-par generative performance to latent diffusion models, to achieve high-quality try on results. VAR-VTON utilizes joint attention over unchanged garment tokens and also uses a cloth-agnostic representation token. However, due to VAE compression, fine details are lost in the resulting try-on outputs. Thus, the authors propose a subsequent stage designed for restoring lost details from the garment, giving rise to STAR-VTON. The second stage achieves this using keypoint matching and ROI extraction to correct relevant fine details. STAR-VTON achieves state-of-the-art results in VTON.

Critical strengths
  1. State-of-the-art performance with fast inference: The method improves state-of-the-art in VTON significantly. Especially, the qualitative results in Fig. 4 show substantial improvements in text preservation, which could have wide-ranging implications in practical settings. Additionally, the inference speed is very fast.

  2. Extensive experiments: The ablations are quite extensive and show the importance of the proposed components empirically.

Critical weaknesses
  1. Why VAR specifically?: Currently, the motivation for why VAR is not very clear other than simply fast inference speed. It seems the major gain comes from the STAR-VTON framework, i.e. the incorporation of the refiner stage (Fig. 1). However, this refiner can be plugged into any existing framework, and given that other VTON frameworks in Fig. 1 are well ahead of VAR-VTON, the stage 2 refiner can substantially improve their performance too (as shown in Table 8). Thus, to me, more than using VAR, the two-stage approach seems to be the key contribution, again raising the question why VAR other than inference speed?

  2. Current reported metrics do not capture specific attributes of the generates VTON results, such as identity, pose, etc.

Questions for authors
  1. Line 48 states one of the major challenges in VTON as deforming the garment to align with the target body pose and shape, and preserving the person's attributes. Current metrics in Table 1 are more global, i.e. FID, KID, CLIP-I. Are there any metrics that specifically measure these objectives?

  2. How is the proposed two-stage refiner specific to VAR, given that a majority of the improvement stems from it?

  3. Can the authors provide comparisons with accelerated diffusion model approaches such as CAT-DM [1], with and without the stage 2 refiner?

[1] Zeng, Jianhao, et al. "Cat-dm: Controllable accelerated virtual try-on with diffusion model." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Reviewer_pgBROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a two-stage pipeline for fast and high-fidelity virtual try-on. Stage one adapts a VAR model to generate a coarse structure through decoupled garment conditioning and scale-wise injection of garment-agnostic person representations. Stage two employs a U-Net to recover garment details in pixel space, using visual correspondence between the garment and the stage-one output obtained from a pre-trained image matcher. Experiments on VITON-HD and DressCode demonstrate the effectiveness of the proposed method.

Critical strengths

1.Adapting the VAR model to virtual try-on is non-trivial.

2.The full model STAR-VTON achieves better performance than the compared baselines.

3.The paper includes an ablation study on the VAR adaptation designs for VTON.

Critical weaknesses

Major concerns:

1.Train-test discrepancy in the VAR adaptation. The proposed decoupled garment conditioning and scale-wise garment-agnostic representation injection may be vulnerable to the train-test discrepancy in VAR: during inference, the hidden states of the generated image at smaller scales are not ground-truth features as in training. The authors do not discuss this issue. Please analyze its impact and discuss possible mitigation strategies.

2.Marginal contribution of the stage-two refiner. Table 1 shows that the stage-two refiner brings only marginal improvement when comparing STAR-VTON with VAR-VTON without the refiner. Moreover, the most significant gain appears in CLIP-I only, which is not sufficient to convincingly measure garment detail preservation. Please provide stronger evidence, such as additional metrics, region-specific evaluations, qualitative comparisons.

3.Impact of the image matcher. The authors merely discuss how the choice of image matcher affects visual correspondence and the final try-on quality. Please compare the performances among different image matcher.

4.Missing user study. A user study is needed to substantiate perceptual quality and garment detail preservation.

5.Limited generalization assessment. In-the-wild results are not reported. Since performance on VITON-HD and DressCode is already saturated, in-the-wild evaluations would better assess generalization ability.

Minor concerns:

1.Comprehensive results on fine-grained categories of DressCode are not reported in the Appendix. Please include them.

2.Architecture choice of the stage-two refiner. A U-Net is used in the stage-two refiner, but the representation bottleneck in its middle blocks may hinder the preservation of person and garment details. Please justify this choice over Pixel-DiT, or provide an ablation/comparison to support it.

3.Some SOTA baselines are missing, e.g., MOFA-VTON and PROMO (CVPR 2026). This is not critical for the assessment, but including them would strengthen the comparison.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.