acceptodds
Under review as a conference paper at ICLR 2027

ProxyFit: Point Correspondences as Proxy Representations for Wearing-Style Control in Virtual Try-On

Abstract

Diffusion-based virtual try-on transfers garment appearance with high fidelity, but offers limited control over how a garment is worn. Text instructions leave exact target positions ambiguous, while existing spatial interfaces restrict garment-part coverage or provide insufficient control adherence. We introduce ProxyFit, which represents the intended wearing state through editable proxy point mappings from a flat garment to the body, enabling precise interaction by dragging target points. Our first contribution is an automated data pipeline that matches mesh-derived garment and person points, filters unreliable mappings and images, and constructs contrastive try-on pairs with the same person and garment but different wearing states. Our second contribution is a dedicated proxy representation branch: Fold extracts person-point features, attention retrieves local garment features, and ReFold writes them back at the specified body locations. This explicit routing improves adherence to user-specified point targets while preserving local garment appearance. Experiments demonstrate strong performance in both precise point control and standard virtual try-on. Across front opening, sleeve placement, hem placement, and fit, ProxyFit achieves 82.8% mean spatial control accuracy, exceeding WAW-style by 9.5 percentage points. It reduces point-placement error by 32.1% relative to Qwen-Image-Edit, while achieving LPIPS of 0.201 and FID of 44.35 on the standard VTON task.

Primary area:
generative models
Keywords:
Diffusion model, Virtual try-on, Conditional generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6C6xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ProxyFit, which uses proxy point mappings between garment-side source points and person-side target points to enable precise wearing-style control in virtual try-on, including front opening, sleeve placement, hem placement, and fit. The authors develop an automated data pipeline that combines image editing, mesh-based correspondence matching, and quality filtering to construct paired images and point mappings for different wearing states of the same person and garment. Built on an SDXL-based dual-UNet backbone, ProxyFit introduces a dedicated proxy representation branch that performs Fold–Attention–ReFold operations to retrieve local features around garment source points and write them to specified person-side target locations. The model uses paired denoising training with shared noise and timesteps, while mapping dropout supports ordinary VTON. Experiments evaluate spatial control accuracy, point-placement error, and standard VTON quality, with ablation studies examining the contributions of the proxy branch and paired-training strategy.

Critical strengths
  1. The authors develop an automated data pipeline that produces paired images and point correspondences for learning wearing-style control. The pipeline combines image editing, mesh-based correspondence matching, rule-based filtering, and multimodal pair review to construct samples with different wearing states under shared person and garment conditions, while identifying the corresponding locations of garment parts in person images. It provides supervision for both appearance changes and spatial mappings, making it well suited to training precise wearing-style control.

  2. The authors introduce ProxyFit to enable precise control over how garments are worn while also supporting ordinary virtual try-on. Users can drag target points to coordinate changes in front opening, sleeve placement, and hem placement. Its proxy representation branch uses Fold–Attention–ReFold to extract queries at person-side target locations, retrieve local features around the corresponding garment-side source points, and write the retrieved features back at the specified locations. This well-motivated design directly incorporates garment–person correspondences into feature routing. Matched-noise paired training helps the model learn responses to wearing-state changes, while mapping dropout enables the same model to perform ordinary VTON without point mappings.

  3. The experiments demonstrate strong control precision and generation quality, with ablations supporting the effectiveness of the main design choices. Under the same backbone, training data, and optimization budget, ProxyFit improves mean spatial control accuracy from 0.733 to 0.828 and reduces point-placement error from 51.40 to 40.13 pixels compared with WAW-style. The proxy branch also outperforms RGB- and embedding-based injection, while paired sampling and shared noise provide further improvements. Qualitative results demonstrate coordinated control of multiple garment parts. On standard VTON, the model achieves the best LPIPS, FID, and KID among the compared methods, indicating that improved controllability is compatible with strong appearance fidelity.

Critical weaknesses
  1. The baseline comparisons are not sufficiently comprehensive. Limited qualitative comparisons and implementation details for WAW-style and MOFA-style make it difficult to assess the fidelity of these reimplementations. The evaluation also lacks comparisons with stronger closed-source commercial image editing models. Moreover, providing general-purpose editors with text instructions alone, without visual drag annotations such as those in Figure 4, leaves their ability to follow explicit spatial guidance underexplored.

  2. Table 2 evaluates control accuracy, while Table 3 measures image quality only for ordinary VTON. Thus, appearance fidelity under point-based control lacks sufficient quantitative validation.

Questions for authors
  1. Could the authors provide more qualitative comparisons and additional implementation details for WAW-style and MOFA-style? I believe these additions would strengthen the credibility of the paper’s findings.

  2. Could the authors compare ProxyFit with stronger closed-source commercial models? Additionally, could general-purpose image editing models be given both text prompts and visual drag instructions, similar to the point-movement annotations shown in the source images of Figure 4?

  3. Could the authors evaluate both control accuracy and image quality on the same point-controlled test set to assess whether ProxyFit preserves appearance fidelity during wearing-state changes?

Reviewer_gE1oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ProxyFit for fine-grained wearing-style control in diffusion-based virtual try-on. It represents the desired wearing state through correspondences between source garment points and target locations in person images. An automated pipeline uses DINOv3-based matching and quality filtering to construct correspondence-annotated training data, while a Fold–Attention–ReFold branch retrieves local garment features and writes them back at the specified target locations.

Experiments on proprietary data report higher spatial control accuracy than a same-backbone point-attention baseline and lower point-placement error than Qwen-Image-Edit, alongside competitive results on standard VTON quality metrics..

Control evaluation relies on Gemini 3.5 Flash and DINO-based matching, which are also used for data filtering and correspondence construction, respectively. This overlap raises concerns about evaluation independence, while the proprietary dataset and commercial evaluator complicate independent reproduction. Although the paper reports 87% agreement with human judgments, the validation protocol is insufficiently detailed to assess evaluator reliability.

Critical strengths
  1. The paper proposes a sound framework for controlling the placement of individual garment parts. The Fold–Attention–ReFold branch directly incorporates point correspondences by retrieving local garment features and writing feature residuals at the paired target locations.

  2. The automated pipeline constructs paired examples with shared person and garment conditions but different wearing states, providing supervision that links changes in point mappings to changes in garment placement.

Critical weaknesses
  1. Although the proposed method is reasonable, its improvement over Qwen-Image-Edit in mean control accuracy is modest (82.8% vs. 81.5%), despite a larger reduction in point-placement error. The qualitative examples show that Qwen-Image-Edit can alter regions outside the intended edit. However, the Qwen baseline receives text instructions and is not adapted to the proposed interaction setting, so these differences may partly reflect differences in conditioning and training. Could Qwen-Image-Edit match or exceed the proposed method if fine-tuned on the same generated paired data with comparable point conditioning? Such a comparison would help establish whether the proposed method retains an advantage over an image editor trained for the same task.

  2. The rationale for choosing SDXL as the backbone is not sufficiently discussed. Could a more recent backbone, such as FLUX or Qwen-Image, make better use of the constructed training data? It would be helpful to explain the considerations behind this choice and investigate whether the proposed mechanism provides similar benefits on another backbone.

Questions for authors
  1. Beyond the attribute-level results in Table 2, are there systematic differences in editing precision across garment parts, such as sleeves, collars, and hems? How does performance vary with the difficulty of the requested edit, for example, larger point displacements or greater occlusion?

  2. Given the same person and garment images and identical point mappings, how much do the results vary across random seeds? For example, when the same sleeve-raising edit is repeated, how consistent are the resulting sleeve position and the preservation of regions outside the intended edit?

  3. Consistency during sequential editing. When users make successive edits, how reliably does the method follow each new request while preserving unrelated regions and earlier edits that should remain unchanged? Does visual drift accumulate over multiple editing steps?

4 comments
AuthorsOpenReview ↗

Response 2/3

The core operations of Proxy Branch are reading, matching, and writing back within spatial features and do not depend on UNet-specific convolutional structures. In the new Qwen experiments, we connect it to 15 DiT layers, with each person proxy point still querying only the 3×33\times3 neighborhood of its paired garment point. Compared with the same LoRA settings, adding this branch improves mAcc by approximately 3.2 percentage points and reduces point error by 9.91 pixels (21.55%). Thus, SDXL is an implementation choice for establishing controlled comparisons in this paper; the new Qwen experiments support transferring the proxy routing design to another backbone.

Q1: Edits of Different Parts and Different Difficulty Levels

Editing difficulty is mainly determined jointly by displacement magnitude, visibility changes, and the distribution of similar scenarios in the training set, rather than solely by the name of the garment part. Small movements within a visible surface are relatively easy, such as slightly adjusting the visible hem height, cuff placement, or front-opening spacing. In these cases, the source part and target location are clear, and the model mainly needs to adjust local structure while preserving the existing appearance.

Edits involving large movements, occlusion changes, or the generation of previously invisible content are usually more challenging. For example, rolling sleeves up substantially to the upper arms requires forming new folds and exposing the arms. We examine this difference in editing difficulty by comparing different sleeve-roll magnitudes.

For this purpose, we manually selected 50 images from the dataset and used SDXL-ProxyFit to perform two edits: rolling sleeves up to the forearms and rolling sleeves up to the upper arms. Each state produces 50 evaluation images, giving 100 images in total. Accuracy is calculated as the number of successful images for each state divided by 50, without averaging accuracies over multiple random seeds.

Table R6. Control precision at different sleeve-roll magnitudes.

Editing stateSleeve Acc ↑dEucd_{\mathrm{Euc}} ↓
Sleeves rolled up to the forearms0.8633.46
Sleeves rolled up to the upper arms0.7845.84

In addition, we manually selected 50 images from the dataset, used SDXL-ProxyFit to perform small- and large-magnitude front-opening edits, and report their control precision.

Table R7. Control precision at different front-opening magnitudes.

MethodOpen Acc ↑dEucd_{\mathrm{Euc}} ↓
Small-magnitude front opening0.8442.97
Large-magnitude front opening0.8445.02

In Table R6, rolling sleeves up to the upper arms has an accuracy 8 percentage points lower than rolling them up to the forearms, and point error increases by 12.38 pixels (approximately 37.00%). In Table R7, both front-opening magnitudes have an accuracy of 0.84, while the larger front opening increases point error by 2.05 pixels (approximately 4.77%). This shows that, in these selected examples, the effect of displacement magnitude on control results varies with the editing task.

Q2: Stability across Random Seeds

Content outside the intended editing region is consistently well preserved, with differences mainly occurring in the editing results within the target region. We conducted an uncertainty test to report its performance. We fix the person, garment, proxy point mappings, and inference parameters and change only the random seed. On top of the original seed of 42, we add 43, 44, 45, and 46, for a total of five random seeds (42–46) to evaluate control stability. Table R3 first computes the aggregate test-set metrics for each seed separately and then reports the mean and standard deviation across the five seeds.

Table R3. Control stability with fixed inputs and point mappings and varying random seeds. Each entry is the mean ± standard deviation under random seeds 42, 43, 44, 45, and 46. MOFA-style reports only Hem Acc on the 113-pair hem subset; this column is reported separately from the four-task mAcc on the full test set.

MethodEvaluation scopeFour-task mAcc ↑Hem Acc ↑Point error ↓
WAW-styleFull point-control test set0.726 ± 0.017--50.99 ± 2.983
MOFA-style113-pair hem subset--0.750 ± 0.021--
ProxyFit: SDXLFull point-control test set0.834 ± 0.013--39.87 ± 2.389

In the results reported in Table R3, ProxyFit's cross-seed mAcc standard deviation is 0.013, lower than WAW-style's 0.017; its point-error standard deviation is 2.389 pixels, lower than WAW-style's 2.983 pixels. This supports that ProxyFit exhibits smaller cross-seed fluctuations in aggregate test-set control metrics.

Dear Reviewer gE1o,

Thank you again for your valuable feedback and for recognizing our explicit correspondence mechanism and paired-data supervision. As the discussion period is approaching its end, we would like to briefly summarize the key additions made in response to your concerns:

  • Fair Qwen comparisons and transfer across backbones: We added Qwen-Image-Edit-2509 experiments with visual point annotations, same-data LoRA fine-tuning, and LoRA fine-tuning with Proxy Branch. Same-data LoRA reaches an mAcc of 0.843, exceeding SDXL-ProxyFit’s 0.828. Under matched LoRA settings and training budgets, adding Proxy Branch further improves mAcc to 0.875 and reduces point-placement error from 45.98 to 36.07 pixels. These results demonstrate that our training data and proxy routing brach are also works well on another, more powerful backbone. We also clarified our choice of SDXL for cost in practical scenarios, and fair comparisons with academic try-on models.

  • Editing difficulty and displacement magnitude: We added controlled comparisons of sleeve-roll and front-opening magnitudes. On 50 selected inputs, rolling sleeves to the forearms and upper arms achieves accuracies of 0.86 and 0.78, respectively, with point errors of 33.46 and 45.84 pixels. We also report small- and large-magnitude front-opening results, showing that the effect of displacement magnitude varies with the editing task.

  • Stability across random seeds: We evaluated fixed inputs and point mappings under five random seeds, 42–46. ProxyFit achieves an aggregate mAcc of 0.834 ± 0.013 and point-placement error of 39.87 ± 2.389 pixels, with smaller cross-seed fluctuations than WAW-style in both metrics.

  • Sequential editing and visual drift: We added qualitative comparisons of three successive edits, using each output as the next input, for ProxyFit-Qwen with LoRA and Proxy Branch versus the original Qwen. Both exhibit some visual drift, while the ProxyFit version shows less drift in the presented comparisons. We also discuss the remaining color shifts and oversaturation in the garment region.

  • Evaluation independence and human-validation details: We added third-party GPT evaluation for each method, which preserves the method ranking observed in the original full-set Gemini evaluation. We also clarified the original 87% human-agreement protocol, including three annotators blinded to the generation method, independent judgments, and majority voting.

We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be more than happy to provide additional clarification. We truly value the opportunity to discuss our work with you.

Thank you again for your time and consideration.

Best regards, The Authors

I thank the authors for their detailed responses and additional experiments.

The rebuttal has addressed most of my concerns. In particular, the Qwen experiments under matched training conditions provide meaningful evidence that the proposed approach transfers across backbones and that the Proxy Branch offers benefits beyond task-specific fine-tuning. The additional random-seed stability tests and evaluation using another model also alleviate my concerns about robustness and evaluation reliability.

While some limitations remain in the scope of the stability analysis and independent evaluation, the additional evidence strengthens the paper sufficiently to support acceptance. I am therefore raising my score to 3.

Dear Reviewer gE1o,

Thank you for your careful consideration of our rebuttal and additional experiments, and for your support of our work. We appreciate your recognition of the evidence for cross-backbone transfer, the Proxy Branch’s benefits beyond task-specific fine-tuning, and the additional validation of robustness and evaluation reliability.

Your constructive feedback has helped us strengthen the experimental evidence and clarify our contributions.

Thank you again for your support! We wish you all the best!

Best regards, The Authors

Reviewer_eYtJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new development in virtual try-ons, enabling drag-and-drop controls to garments. Specifically, it proposes a dataset curation approach and a dataset, as well as the method. The method uses custom unet and representation branch, and leverages contrastive learning from their custom datasets.

Critical strengths
  • A more fine-grained control for virtual try-on, inspirational for generative models.
  • a dataset tool for the task
  • technical pipeline is well documented
Critical weaknesses
  • insufficient experiment evidence, The benchmark is curated by authors, and the baselines are not aimed for the same task. Even still, the metrics are not strong.
Questions for authors

What other existing baselines could be assessed?

2 comments

Dear Reviewer eYtJ,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Dear Reviewer eYtJ,

Thank you again for your valuable feedback and for recognizing the value of fine-grained try-on control and our data construction tools. As the discussion period is approaching its end, we would like to briefly summarize the key additions made in response to your concerns:

  • Additional task-matched baselines: We added comparisons with Qwen-Image-Edit-2509 under visual point conditioning, same-data LoRA fine-tuning, and LoRA fine-tuning with Proxy Branch, together with two Gemini configurations using visual point annotations. These comparisons directly examine the spatial control task addressed by our paper and provides more baselines. We also clarify that the WAW-style and MOFA-style baselines target the same wearing-style control task.

  • Explicit proxy routing works well on Qwen-Image-Edit backbone: Same-data Qwen LoRA fine-tuning achieves an mAcc of 0.843, exceeding SDXL-ProxyFit’s 0.828. Under matched LoRA settings and training budgets, adding Proxy Branch further improves mAcc to 0.875 and reduces point-placement error from 45.98 to 36.07 pixels. These results demonstrate that our training method and proxy routing are also works well on another, more powerful backbone.

  • Validation on public benchmarks: We added standard VTON evaluations on VITON-HD and the upper-body subset of DressCode, using LaDI-VTON’s test scripts and the corresponding dataset splits and evaluation protocols. These experiments provide complementary evidence of garment fidelity and overall generation quality beyond our point-control benchmark, with ProxyFit achieving the best results among the compared methods.

We would greatly appreciate any further feedback you may have. If any questions or concerns remain, we would be more than happy to provide additional clarification. We truly value the opportunity to discuss our work with you.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.