ProxyFit: Point Correspondences as Proxy Representations for Wearing-Style Control in Virtual Try-On

Diffusion-based virtual try-on transfers garment appearance with high fidelity, but offers limited control over how a garment is worn. Text instructions leave exact target positions ambiguous, while existing spatial interfaces restrict garment-part coverage or provide insufficient control adherence. We introduce ProxyFit, which represents the intended wearing state through editable proxy point mappings from a flat garment to the body, enabling precise interaction by dragging target points. Our first contribution is an automated data pipeline that matches mesh-derived garment and person points, filters unreliable mappings and images, and constructs contrastive try-on pairs with the same person and garment but different wearing states. Our second contribution is a dedicated proxy representation branch: Fold extracts person-point features, attention retrieves local garment features, and ReFold writes them back at the specified body locations. This explicit routing improves adherence to user-specified point targets while preserving local garment appearance. Experiments demonstrate strong performance in both precise point control and standard virtual try-on. Across front opening, sleeve placement, hem placement, and fit, ProxyFit achieves 82.8% mean spatial control accuracy, exceeding WAW-style by 9.5 percentage points. It reduces point-placement error by 32.1% relative to Qwen-Image-Edit, while achieving LPIPS of 0.201 and FID of 44.35 on the standard VTON task.