acceptodds
Under review as a conference paper at ICLR 2027

Point-Conditioned Drag Editing for In-Context Diffusion Transformers

Abstract

Point-based drag editing enables intuitive spatial control by allowing users to specify sparse source-target point pairs. However, converting such sparse geometric directives into coherent image edits remains challenging, especially when the edited content must move accurately while preserving identity, background, and local visual details. Existing drag-editing methods often rely on iterative point tracking, hand-crafted motion propagation, region-level assumptions, or mask-guided inpainting, which can introduce additional user burden and limit generalization to complex motion. We propose PiDiT, a point-conditioned drag-editing framework built on an in-context diffusion transformer. PiDiT renders sparse drag pairs into spatial point maps, encodes them with a lightweight point encoder, and injects the resulting token-aligned point features directly into the transformer's hidden states. This design treats point pairs as internal spatial conditions for the target generation branch while preserving the source image as in-context visual evidence. The framework is trained with a flow-matching objective and requires only the source image and sparse point pairs at inference time, without user-provided masks or test-time optimization. Experiments on DragBench and ReD-Bench show that PiDiT improves point reachability while maintaining competitive visual fidelity and identity preservation.

Primary area:
generative models
Keywords:
Image Editing, Diffusion Models, Diffusion Transformers, Controllable Image Generation, Spatial Conditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b2EjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PiDiT, a drag-based image editing framework built on an in-context diffusion transformer. It represents source–target point pairs as sparse index maps, encodes them into spatially aligned features, and injects these features into the image-token representations. The point encoder and LoRA parameters are trained using flow matching, while inference requires only a source image and point pairs without user-provided masks or test-time optimization. Experiments on DragBench and ReD-Bench report improved point reachability, alongside qualitative examples of joint drag-and-text editing.

Critical strengths
  1. Simple and useful interaction interface. PiDiT requires only the source image and sparse point pairs, avoiding the need for users to specify editable masks. Its formulation integrates spatial control into the generation process without iterative point tracking or test-time optimization. This makes the approach practically appealing for interactive editing.

  2. Consistent reported improvements in point reachability. Tables 1 and 2 report the lowest mean distance among the evaluated methods on DragBench and across the operation categories of ReD-Bench. The qualitative results also cover varied objects, geometric transformations, and non-rigid edits. Subject to clarification of the sampling protocol, these results provide promising evidence of effective spatial control.

Critical weaknesses
  1. Table 3 explicitly reports the best seed for each sample selected by mean distance, and its default LPIPS and mean-distance values match those in Tables 1 and 4. This does not establish that the main tables use the same selection rule. However, the supplied PDF leaves the candidate counts, output-selection criteria, and aggregation across seeds for the main comparisons unclear. The authors should specify these details for each method and distinguish averages over all outputs from prespecified seeds from best-of-N results. Any best-of-N comparison should use documented, comparable selection criteria and candidate budgets where applicable.

  2. The paper describes the point encoder and several training settings, but the supplied PDF does not identify the exact pretrained checkpoint or the LoRA rank and target modules. Figure 2 places point-feature addition before the DiT blocks, but its relation to the intermediate hidden states described in Section 4.4 and to the input projection needs clarification. Section 5.1 identifies video frame pairs and generated image edits, but not the specific source collections or editing generator. The supplied PDF also leaves the curation procedure under-specified and does not state whether overlap with the evaluation benchmarks was checked. These omissions hinder faithful reproduction and make it difficult to assess the provenance of the training data and their independence from the evaluation benchmarks.

  3. The G_PQ margins over the strongest reported alternatives in Tables 1 and 2 are small, while the supplied PDF gives neither uncertainty estimates nor the VIEScore evaluator configuration. This quality score does not by itself establish identity preservation. The authors correctly acknowledge that LPIPS can favor conservative edits, so its results alone cannot establish which method better preserves identity while completing the requested motion. The abstract and conclusion appropriately describe visual quality as competitive, whereas the stronger preservation wording in the introduction and Figure 3 needs more direct support. Documenting the evaluator and providing a targeted preservation analysis that accounts for achieved drag accuracy would strengthen this narrower claim.

Questions for authors
  1. How many candidates are generated per sample for each method, and which tables use selection by mean distance? What criteria select the reported outputs, and how are metrics aggregated across samples and seeds? Can the authors distinguish averages over all outputs from prespecified seeds from any best-of-N results, documenting comparable selection criteria and candidate budgets where applicable?

  2. Which backbone checkpoint, LoRA rank, and LoRA target modules are used? Where exactly are point features added relative to the input projection and transformer blocks? Please identify the specific source collections and editing generator, and explain whether and how overlap with the evaluation benchmarks was checked.

  3. Which evaluator model, version, and prompting configuration are used for VIEScore? Can the authors provide uncertainty estimates for the reported quality scores? What evidence supports identity and background preservation after accounting for differences in achieved drag accuracy?

2 comments

Thank you for your careful review. We address each weakness together with its corresponding question.

[B1] Evaluation protocol. For every method in Tables 1–4, we generate two candidates per test sample and select the one with the lowest MD. MD, LPIPS, and GPQG_{PQ} come from that same output and are averaged equally across test samples. These are best-of-2 by MD results, rather than averages over all candidates. Within each benchmark, all test samples use the same prespecified seeds.

[B2] Implementation and data provenance.

Backbone and LoRA. We use black-forest-labs/FLUX.1-Kontext-dev. Base weights, VAE, and text encoders are frozen; we train the point-map encoder, point-feature projection, and LoRA parameters. LoRA rank/alpha/dropout are 32/32/0.05. Targets are:

  • 19 double-stream blocks: attn.to_q, attn.to_k, attn.to_v, attn.to_out.0, ff.net.0.proj, and ff.net.2.
  • 38 single-stream blocks: attn.to_q, attn.to_k, and attn.to_v.

Injection. Packed 64-dimensional point features are projected to 3072 dimensions and added to corresponding image tokens after x_embedder and before the first DiT block. These projected tokens are the “hidden states” in §4.4, consistent with Figure 2.

Data and curation. The 2,213 paired records contain 1,366 video-frame pairs (1,232 from videos collected from the Pexels website; 134 from OpenVid-1M) and 847 generated edits. Video processing includes motion filtering, CoTracker tracking/frame-pair selection, foreground-point/displacement filtering, and manual review.

Generated edits use Pexels source images. Qwen2.5-VL-7B-Instruct assesses suitability and generates geometric instructions; Qwen-Image-Edit-2511 generates targets. Point-pair annotation, filtering, and manual review follow.

Benchmark overlap. We compared DINOv2 embeddings of training source/target images against DragBench and ReD-Bench reference images using cosine similarity. Manual review of high-similarity candidates identified no duplicates or near-duplicates.

[B3] Quality evaluation and preservation.

Evaluator. We use VIEScore's PQ component [3], GPT-4o (gpt-4o-2024-11-20), and its official zero-shot PQ prompt. Naturalness and artifacts are each scored 0–10; their minimum is the per-image PQ. The full evaluation prompt is provided in the appendix.

The additional 95% confidence intervals for GPQG_{PQ} are:

Benchmark95% confidence interval
DragBench[8.6066, 8.8590]
ReD-Bench[8.5667, 8.9083]

Preservation. Following ConsistentID [1], we use CLIP-I as complementary evidence of source-content consistency. Following BrushEdit [2], we compute background masked LPIPS outside official DragBench editing masks.

MethodCLIP-I ↑Background masked LPIPS ↓
Inpaint4Drag0.95810.0458
DragNeXt0.96140.1415
LightningDrag0.94640.0806
DragFlow0.96020.0851
FastDrag0.93870.1022
DragDiffusion0.95230.0827
PiDiT0.96850.0882

PiDiT has the highest CLIP-I in this comparison. Its background LPIPS is numerically close to LightningDrag, DragDiffusion, and DragFlow, and lower than FastDrag and DragNeXt. We interpret these preservation metrics jointly with MD, global LPIPS, GPQG_{PQ}, and qualitative results to assess motion completion, content consistency, background preservation, and visual quality. We describe visual quality as competitive.

[1] Huang et al. ConsistentID: Portrait Generation With Multimodal Fine-Grained Identity Preserving. TPAMI, 2026. [2] Li et al. BrushEdit: All-in-One Image Inpainting and Editing. TPAMI, 2026. [3] Ku et al. VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation. ACL, 2024.

Thanks for the authors' response. My main concerns have been addressed and thus I keep my positive score.

Reviewer_tABpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel point-conditioned framework for drag-based image editing. Specifically, sparse drag pairs are first rendered as spatial point maps and then encoded by a dedicated point encoder to obtain point features, which serve as spatial conditioning signals for the in-context diffusion transformer during image generation. The proposed method is evaluated on two benchmarks to assess its performance.

Critical strengths

1.User-friendly and practical framework. The proposed method enables drag-based image editing using only sparse point pairs, eliminating the need for user-provided masks or test-time optimization.

2.Promising performance. Experimental results on two benchmarks demonstrate competitive performance compared with state-of-the-art approaches.

Critical weaknesses

1.Insufficient evaluation. (1)The strong MD and G_PQ scores show PiDiT's better point reachability and visual quality, but LPIPS performance is noticeably worse than several baselines. The paper explains that LPIPS measures source similarity but may favor conservative or mask-assisted edits, despite this, the evaluation of visual fidelity and identity preservation remains insufficient. (2)The results of the qualitative comparison are not discussed in detail.

2.Insufficient ablation. (1)LPIPS has been argued in the paper that it may favor conservative or mask-assisted edits, so the supported conclusion of the ablation results in Table 4 becomes less convincing. (2)The used pretrained backbone is unclear, and the effectiveness of the proposed method is not sufficiently isolated from the backbone. (3)The paper emphasizes the use of a compact, distribution-balanced training dataset of approximately 2.2K samples, but provides limited details about its balancing strategy. And there is no further discussion about whether further increasing the training data size would lead to additional improvements.

Questions for authors

1.Which specific pretrained model is used as the backbone of PiDiT? And to what extent do the reported improvements originate from the pretrained backbone rather than the proposed method?

2.How robust is the proposed method with respect to the number, order, and spatial distribution of the drag points?

1 comment

Thank you for your constructive suggestions.

[T1a] Preservation evaluation. Following ConsistentID [1], CLIP-I complements our evaluation of source-content consistency. Following BrushEdit [2], background masked LPIPS measures changes outside the official DragBench editing masks.

MethodCLIP-I ↑Background masked LPIPS ↓
Inpaint4Drag0.95810.0458
DragNeXt0.96140.1415
LightningDrag0.94640.0806
DragFlow0.96020.0851
FastDrag0.93870.1022
DragDiffusion0.95230.0827
PiDiT0.96850.0882

PiDiT achieves the highest CLIP-I and competitive background preservation while requiring only source images and point pairs, without a user mask. We interpret preservation jointly with MD, global LPIPS, GPQG_{PQ}, and qualitative results: unchanged content alone does not establish motion completion, and point reachability alone does not guarantee visual quality.

[T1b] Qualitative discussion. The main paper presents qualitative comparisons in Figure 3 and discusses editing distortions and artifacts in §5.2. We further analyze the examples method by method, considering motion completion, structural coherence, and background preservation:

  • LightningDrag: the car remains oblique and enlarged (row 3); the lamp has severe artifacts (row 2).
  • Inpaint4Drag: artifacts appear near the duck's original location (row 4) and the lamb's leg-torso junction (row 1).
  • DragNeXt: the duck remains near the hedge (row 4); background contrast changes around the spinning top (row 8).
  • FastDrag: the car background blurs (row 3); the duck moves insufficiently left (row 4); grass is distorted (row 7).
  • DragFlow: an unnatural bright region remains above the moved duck (row 4); lampshade edges stretch (row 2).
  • DragDiffusion: the car front and bumper distort (row 3), and the duck deforms severely (row 4).

PiDiT more faithfully realizes the intended edits in these examples while preserving coherent structures.

[T2a] Injection ablation. We supplement Table 4 with background masked LPIPS:

InjectionBackground masked LPIPS ↓MD ↓
Hidden-state0.088217.0644
QK0.129317.2702

The regional metric provides additional evidence for ablation study.

[T2b] Backbone and contribution. PiDiT uses FLUX.1-Kontext-dev, extending its image/text editing interface to explicit source-target point pairs. We add same-backbone comparisons: Coordinate uses Fourier coordinate embeddings; PME uses our point-map encoder. Add is PiDiT's additive injection; Cross is image-to-point cross attention at multiple layers.

MethodMD ↓LPIPS ↓CLIP-I ↑Background masked LPIPS ↓
Coordinate + Add22.81780.14950.95840.0926
Coordinate + Cross55.56170.14740.95410.1029
PME + Cross22.60320.14800.95630.0910
PME + Add (ours)17.06440.14550.96850.0882

These crossed variants provide evidence for both representation and injection choices.

[T2c] Data coverage and scale. The 2,213 pairs comprise 1,366 video-frame pairs and 847 generated edits: pose change 774, deformation 587, rotation 581, translation 161, scale 62, and spread 48. Category-guided collection and round-robin assignment of geometric operations within content categories broaden coverage; final counts reflect filtering rather than equal class quotas. Figure 5 examines 0.5K, 1.0K, 1.5K, and 2.2K pairs. Studying larger datasets and broader generalization is future work.

[TQ1] Please see T2b for the backbone and same-backbone comparisons.

[TQ2] Point-condition robustness. We permute point pairs while fixing correspondences, noise, and inference settings. Count and distance groups stratify existing results. Reordering changes the metrics little, and GPQG_{PQ} remains stable across groups.

ConditionLPIPS ↓MD ↓GPQG_{PQ} ↑
Original Order0.145517.06448.7352
Permuted Order0.143817.13878.7329
Distance <64 px0.133815.49908.7539
Distance ≥64 px0.163819.51128.7059
Point count <30.147916.82648.7338
Point count ≥30.132718.33878.7427

[1] Huang et al. ConsistentID: Portrait Generation With Multimodal Fine-Grained Identity Preserving. TPAMI, 2026. [2] Li et al. BrushEdit: All-in-One Image Inpainting and Editing. TPAMI, 2026.

Reviewer_hnNgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a point-conditioned image-editing framework PiDiT which is built on an in-context diffusion transformer with its main contribution of encoding source–target point pairs as sparse index maps and injecting the resulting spatially aligned features into the transformer’s hidden states. The proposed method learns drag control through flow matching and LoRA adaptation, requiring neither user-provided masks nor test-time optimization. PiDiT was trained on 2.2K pre-processed image pairs and was reported the lowest mean point distance on DragBench and all three ReD-Bench operation categories. In general, the paper addresses a useful and existing problem. However, the claim of stronger identity preservation is less convincing, and the methodological contribution is not sufficiently separated from the pretrained backbone and carefully designed training data.

Critical strengths
  1. Spatially aligned point features provide a straightforward way to add drag control to an in-context editing model without introducing a dense-flow prediction stage. The design is simple yet effective.
  2. The results on DragBench and ReD-Bench are promising, with decreased MD compared with the best baselines.
  3. The paper is well written and easy to read.
Critical weaknesses
  1. The approach is technically straightforward, and the methodological contribution appears incremental, primarily integrating point-map conditioning into an existing in-context DiT. Stronger same backbone comparisons, such as coordinate embeddings, point condition token concatenation, and simple patch-wise point map projection, would help clarify the contribution of proposed encoding and injection design.
  2. The claim of stronger identity and background preservation is not solid. I can notice some artifacts regarding identity preservation in Figure 12, i.e., Iron Man's left hand and the girl's legs.
  3. The quantitative results are marginal, with very small perceptual quality gains, for example, the values on LPIPS and G_PQ in Table 1 are not superior.
  4. The 2.2K examples in the training dataset combine video-frame pairs and generated image edits, which is inspiring. However, the manuscript provides limited information about source selection, the editing generator, filtering, annotation procedures, or category proportions.
Questions for authors

See Critical Weaknesses above.

1 comment

Thank you for recognizing our simple and effective point conditioning, improved point reachability, and clear presentation.

[H1] Same-backbone comparisons. To clarify the roles of our point encoding and injection design, we add three baselines built on the same FLUX.1-Kontext-dev backbone:

  • Coordinate + Add Injection: Fourier embeddings of source-target point pair coordinates, incorporated through the same additive injection mechanism used in PiDiT.
  • Coordinate + Cross Attention: The same coordinate embeddings, incorporated through image-to-point cross attention at multiple layers.
  • PME + Cross Attention: Our points map encoder (PME), with additive injection replaced by image-to-point cross attention at multiple layers.
MethodMD ↓LPIPS ↓CLIP-I ↑Background masked LPIPS ↓
Coordinate + Add22.81780.14950.95840.0926
Coordinate + Cross55.56170.14740.95410.1029
PME + Cross22.60320.14800.95630.0910
PME + Add (ours)17.06440.14550.96850.0882

These crossed comparisons examine point representations under fixed injection and injection mechanisms under a fixed representation, supporting our encoding and injection design.

[H2] Figure 12 and preservation. We acknowledge the local inconsistencies in Iron Man's left hand and the girl's legs. With at most two additional samples per case, we obtained outputs that better preserved these regions while following the editing instructions; see Appendix Figure 13.

Regarding background preservation, §4.2 states the objective “to preserve the background, identity, and visual consistency of the source image,” without claiming stronger background preservation.

Following ConsistentID [1], we add CLIP-I as complementary evidence of source-content consistency. PiDiT achieves the highest score in this comparison:

MethodCLIP-I ↑
Inpaint4Drag0.9581
DragNeXt0.9614
LightningDrag0.9464
DragFlow0.9602
FastDrag0.9387
DragDiffusion0.9523
PiDiT0.9685

[H3] Quantitative gains. Table 1 reports the lowest MD and highest GPQG_{PQ} for PiDiT. Our main advantage is accurate drag control with competitive visual quality. LPIPS measures source similarity and can penalize geometric changes required by an edit, including rotation and deformation. It should therefore be interpreted jointly with motion completion (MD), perceptual quality (GPQG_{PQ}), and the complementary consistency results in H2.

[H4] Data composition and construction. The 2,213 training pairs comprise 1,366 video-frame pairs and 847 generated edits. Primary drag intents are pose change (774), deformation (587), rotation (581), translation (161), scale (62), and spread (48).

The video branch contains 1,232 pairs from videos collected from the Pexels website and 134 from OpenVid-1M. Construction includes motion filtering, CoTracker tracking and frame-pair selection, foreground-point and displacement filtering, and manual review.

For generated edits, Pexels images are assessed by Qwen2.5-VL-7B-Instruct, which also generates geometric editing instructions. Qwen-Image-Edit-2511 produces target images, followed by point-pair annotation, filtering, and manual review.

[1] Huang et al. ConsistentID: Portrait Generation With Multimodal Fine-Grained Identity Preserving. TPAMI, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.