acceptodds
Withdrawn from ICLR 2027

Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reference-based diffusion models enable highly controllable image generation by leveraging elements from input images to guide prompt-driven synthesis. However, these models are computationally expensive in runtime, and their cost scales severely with the number of input references. While the efficiency of diffusion models has been extensively studied in the context of prompt-driven generation, it remains largely under-explored in the realm of reference-based models. This setting presents unique challenges not addressed by methods focusing solely on generation. In particular, the wasteful representation of references as dense token grids offers significant opportunities for improvement. In this work, we present SparseContext, a method for constructing sparse reference representations by retaining only a reduced subset of reference tokens. We observe that even without modifying the model, dropping a significant portion of reference tokens at inference time largely preserves its generation capabilities. To fully realize this potential, we fine-tune the model with random token dropping at varying ratios, encouraging robustness to partial reference representations. Crucially, this training strategy decouples the model from any specific token selection rule, allowing flexible control at inference time. At inference time, instead of random dropping, we apply task-aware token selection strategies that prioritize the most informative regions of the reference images, adapting the token budget to the input and task requirements. Extensive experiments show our method achieves a increase in inference speed for multi-reference generation and an for single reference generation. Importantly, this efficiency is achieved without compromising visual quality across both spatially-aligned editing and subject-driven generation.

Primary area:
generative models
Keywords:
Generative models, Efficient Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xwFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the high inference cost of DiT-based reference-conditioned generation. The authors find that the model can still produce reasonable results after some reference tokens are dropped, even without additional training. In Sparse Context, the model randomly drops reference tokens to keep 5-25% of them and is then fine-tuned with LoRA using the remaining tokens. At inference time, the model selects tokens using Canny edges for image editing and saliency maps for personalization. The results show better quality than token dropping without fine-tuning in image editing, single-reference and multi-reference generation, and video generation. The method also improves inference efficiency over the original model, especially in multi-reference settings.

Critical strengths
  1. The method is easy to understand. It randomly drops some reference tokens and then fine-tunes the model with LoRA using the remaining tokens. The authors evaluate this approach on FLUX.2-Klein-4B, FLUX.2-Klein-9B, and a video generation model. It can also be combined with KV caching and token merging.

  2. Training with random token dropping avoids tying the model to a particular token selection rule, allowing the selection strategy to be adapted to the task at inference time.

Critical weaknesses
  1. Image editing is tested only on PIE-Bench, and single-reference personalization only on a subset of Subjects-200K. Multi-reference generation shows the largest speedups, but quantitative evaluation of identity preservation in this setting is limited.

  2. The paper needs to explain more clearly how the method relates to and differs from ToPi [1] and OminiControl2 [2]. The image experiments are also limited to FLUX.2-Klein (9B and 4B).

  3. The presentation and experimental details need to be clearer. For single-reference personalization, the main paper reports 20K training samples, but Appendix A.11 reports 30K. Model names are misspelled and written inconsistently, such as "FLUX.2-Klien" and "Flux-Klien2". It would also be helpful to check the GPU name "RTX-6000 Pro" in A.12 and correct any typos. KID scores are reported without explaining the metric.

Questions for authors
  1. OmniContext [3] and XVerseBench [4] include multi-reference generation tasks and provide evaluation protocols. Please report results on these benchmarks and on ImgEdit [5] for image editing.

  2. Please explain how the method differs from ToPi [1] and OminiControl2 [2] and what advantages it offers. If comparison results are available, please include them.

  3. Please provide results of applying the proposed method to other reference-conditioned generation models, such as FLUX.2 [dev] or Qwen-Image-Edit [6].

  4. A comparison with encoding reference images at a lower resolution would help show the benefit of selecting which tokens to keep.


[1] Lin, J., et al. Token Pruning for In-Context Generation in Diffusion Transformers. arXiv (2026).

[2] Tan, Z., et al. OminiControl2: Efficient Conditioning for Diffusion Transformers. CVPR Findings (2026).

[3] Wu, C., et al. OmniGen2: Towards Instruction-Aligned Multimodal Generation. CVPR (2026).

[4] Chen, B., et al. XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation. NeurIPS (2025).

[5] Ye, Y., et al. ImgEdit: A Unified Image Editing Dataset and Benchmark. NeurIPS (2025).

[6] Wu, C., et al. Qwen-Image Technical Report. arXiv (2025).

6 comments
AuthorsOpenReview ↗

1. Benchmarking on ImgEdit[5] and XVerseBench[4]

We thank the reviewer for this suggestion. In the following results we evaluate our trained checkpoint on ImgEdit-Bench[5] (editing) and XVerseBench [4] (multi reference generation). We skip OmniContext[3] as its official evaluation protocol requires GPT 4.1 as a judge. (API Access) Editing: ImgEdit-Bench [5] We evaluate on ImgEdit-Bench (all 737 single-turn edits spanning 9 edit types) with the same evaluation setup we used in Table1 in paper.

Table — ImgEdit-Bench (737 samples).

MethodTokens fLPIPS ↓CLIP-I ↑DINO ↑KID ↓ (×10⁻³)
Full-reference1.000.3610.8280.7891.9
Naive0.100.7120.6940.5535.2
Ours0.100.5850.7890.7412.6
Naive0.150.6930.7230.6124.3
Ours0.150.5560.8030.7642.2
Naive0.200.6760.7440.6473.8
Ours0.200.5310.8120.7782.3

These results show that same observations hold on ImgEdit-Bench[5] also.

Multi-Reference Generation: XVerseBench [4]

We run all 300 XVerseBench prompts (1-3 subjects each) on Flux2-Klein9B and score them with benchmark’s official metrics:

  1. DPG for prompt following
  2. ID for face identity
  3. IP for identity preservation
  4. AES for aesthetics

Table — XVerseBench (300 prompts), FLUX.2-Klein-9B. For Ours the LoRA is fixed; only the inference-time selection rule differs (random vs. segmentation).

MethodTokens fDPG ↑ID ↑IP ↑AES ↑AVG ↑
Full-reference1.0095.347.474.555.568.2
Naive (random)0.1095.814.660.658.257.3
Ours (random)0.1089.920.164.352.856.8
Ours (segmentation)0.1091.423.169.055.359.7
Naive (random)0.1595.218.862.357.658.5
Ours (random)0.1590.322.165.652.457.6
Ours (segmentation)0.1591.926.670.654.260.8
Naive (random)0.2094.121.864.457.059.3
Ours (random)0.2090.125.266.952.458.7
Ours (segmentation)0.2091.728.970.853.861.3

Since the selection rule at inference time is pluggable, we also sample tokens only from the segmented subject region, as XVerseBench subjects are cut-out images on white a white background. Choosing tokens in this way gives further improvement over random token selection at each keep fraction.

Furthermore, ours preserves identity and appearance better than naive dropping at every keep fraction. From visual inspection we observe that Naive scores are higher on DPG and AES because with only f% of the tokens, it barely extracts any information from the references and follows text mainly.

AuthorsOpenReview ↗

2. Evaluation for Multi-Reference Case

To score multi-reference generation on our evaluation set, we add a per-object subject fidelity metric. Each reference object is segmented using SAM3 in both output and ground truth image and we report the CLIP and DINO similarities between the two segmented crops. We use random token selection in this case for all reference images on our trained checkpoint and we preserve identity better than the naive (no-LoRA) checkpoint at all keep fractions:

Table — Multi-reference subject fidelity (output vs. dataset GT), 200 samples.

MethodfracCLIP ↑DINO ↑
Baseline (full tokens)1.000.91840.6761
Naive0.100.86900.4655
Ours0.100.88660.5428
Naive0.150.88020.5171
Ours0.150.89020.5732
Naive0.200.88750.5446
Ours0.200.89660.5816

Across every drop fraction, our method preserves per-object subject identity substantially better than naive dropping and approaches full token baseline as budget increases.

AuthorsOpenReview ↗

3. Comparison with ToPi[1]

We thank the reviewer for pointing to this related work. ToPi[1] is a training free, inference time method which scores the reference tokens by attention based importance and keeps an adaptive subset with the selection mask recomputed periodically during the whole denoising process. We note three points:

  1. ToPi[1] is not evaluated on the Flux.2-Klein family of models that we use, and has no publicly available code. Hence a faithful reimplementation is not feasible during this discussion period.

  2. Because its mask is updated periodically across timesteps, the retained token set is not fixed through the denoising trajectory, hence the method has to periodically process a full length sequence to update the mask due to which it does not yield a consistent reduction in peak memory.

  3. In contrast, we assign position ids / RoPE a single time, select tokens once, then physically remove the rest of them and the kept tokens retain their original RoPE. This shortened sequence is then used for the full denoising process.

We will add [1] to our related work section, and mention this in revision.

Reviewer_ggTGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Sparse Context, which accelerates reference-conditioned generation through LoRA adaptation with randomly dropped reference tokens. At inference, task-specific heuristics select tokens under a prescribed budget. Experiments cover image editing, personalization, multi-reference generation, and video generation, demonstrating substantial speedups and improved quality over naïve token dropping, with remaining trade-offs relative to full-reference generation.

Critical strengths
  • The method is simple and straightforward to implement, providing a lightweight approach to accelerating reference-conditioned generation.
  • Experiments span multiple tasks and models, providing useful evidence of the effectiveness and scalability of sparse reference conditioning.
Critical weaknesses
  • Token dropping will inevitaly discard information necessary to preserve foreground appearance and background details, as illustrated by the qualitative results in this paper and the videos in the zip file. This limits confidence in fidelity-sensitive applications such as product-conditioned generation and consistent character creation.
  • Quality-preservation claims are overstated. In Table 1, the 2.06× speedup at (f=0.05) accompanies an increase in editing LPIPS from 0.433 to 0.665 and a decrease in DINO from 0.760 to 0.660. These results indicate a quality–efficiency trade-off rather than acceleration “without compromising visual quality.” Table 3 also reports lower acceleration at (f=0.10) than at (f=0.20), which requires explanation.
  • Inference relies on task-specific heuristics. Token budgets and selection rules are externally specified rather than automatically determined from the input and instruction. Appendix A.13 shows that a selection rule suitable for one operation can fail for another. Reliable defaults across diverse inputs are not established; additional tuning and repeated generation could reduce the practical efficiency gains.
Questions for authors
  • Are dropped tokens physically removed from the sequence? Do retained tokens preserve their original spatial coordinates for RoPE, or are positions reassigned?
  • Could the authors provide more visualizations and evaluations of multi-reference generation?
  • Do reported speedups include preprocessing, saliency/segmentation prediction, and sampling overhead?
  • Training randomly selects reference tokens, whereas inference selects them using prior-based rules. Does this introduce a training–inference mismatch, and how does it affect generation quality?
4 comments
AuthorsOpenReview ↗

1. Token Dropping and Selection

We thank the reviewer for this point, and we would like to clarify that our method is also well suited to fidelity-sensitive applications because Sparse Context does not remove tokens uniformly: its inference-time selection retains the tokens that carry the subject’s identity and discards the redundant/background context.

  1. For product-conditioned generation, this means foreground patches are preserved using our inference-time selection strategy, while background tokens that do not define the product’s appearance are dropped.

  2. Furthermore, this trade-off is completely controllable since the token budget is a free parameter. A user can simply use a slightly higher keep fraction. It can also be adaptive depending on the size and proportion of the image covered by the object.

Different token keep fractions can be selected accordingly, but this is still an inference-time selection strategy that requires no retraining.

AuthorsOpenReview ↗

2. Speed-Quality Trade-off

We agree with the reviewer. This is indeed a speed-quality trade-off. We will correct this wording in the revision so the claims accurately reflect this trade-off.

As for the f=0.05 setting, we agree that it is an overly aggressive setting, but that is not representative of how our method is intended to be used. We will instead report f=0.15 along with f=0.10 and f=0.20 as shown below:

------Table T1-----------------------Personalization------------|---------Image Editing---------------

MethodClip IDino IClip TIKID|LPIPSClip IDino IClip TIKID
Baseline (100% tokens)0.7500.4510.29021.2|0.4330.8330.7600.3151.5
Naive 10%0.6810.3600.28727.8|0.7190.7230.5530.3093.5
Ours 10%0.7480.4520.27422.7|0.6280.7960.7030.3113.4
Naive 15%0.6910.3800.28726.9|0.7090.7470.5980.3133.1
Ours 15%0.7540.4570.27423.4|0.6060.8060.7200.3132.9
Naive 20%0.6920.3900.28827.1|0.6870.7620.6310.3163.0
Ours 20%0.7540.4550.27423.0|0.5760.8170.7380.3142.4

KID is reported ×10⁻³ in all results

Regarding Table 3, there is a typo and those speedups are reported in the opposite order. Correct speedups are 1.50x at f=0.10 and 1.42x at f=0.20. We will also correct this.

AuthorsOpenReview ↗

3. Flexible Token Selection at Inference Time

The key insight is that random dropping during training is what enables flexible token selection at inference. The model is never trained to rely on a fixed subset of tokens; hence, any token selection strategy can be plugged in at inference time depending on the use case.

In addition, selection rules are not strict. In our experiments, we used a canny edge based selection rule for editing because it is cheap and intuitive to use, not a hard constraint. To demonstrate this we also show results obtained when token selection is done using segmentation-mask in which we retain tokens from every segment instead of edges. We can see this better token selection strategies like this outperforms canny based selection at the same f=0.20 keep fraction without any retraining.

Table — Image editing (PIE-Bench), keep-fraction = 20%. Token-selection rule swapped at inference; training unchanged.

SettingLPIPS ↓CLIP-I ↑DINO-I ↑CLIP-TI ↑KID ↓
Baseline (full tokens)0.4330.8330.7600.3151.50
Naive 20% (no LoRA)0.6870.7620.6310.3162.97
Ours 20% (Canny selection)0.5760.8170.7380.3142.37
Ours 20% (Segmentation selection)0.5100.8240.7780.3161.26

With 20% of the tokens kept, segmentation-based selection matches or exceeds the full-token baseline the full-token baseline on DINO-I (0.778 vs 0.760) and KID (1.26 vs 1.50), while CLIP-TI (text alignment) stays close — i.e. the intended edit is still performed faithfully with only a fraction of the tokens.

AuthorsOpenReview ↗

4. Questions

Q1 - Yes, the positional ids are assigned first and the tokens are dropped afterwards, so the retained tokens keep their original RoPE.

Q2 - To evaluate multi-reference generation, we add a per-object subject fidelity metric. Each reference object is segmented using SAM3 in both output and ground truth image and we report the CLIP and DINO similarities between the two segmented crops. We use random token selection in this case for all reference images on our trained checkpoint and we preserve identity better than the naive (no-LoRA) checkpoint at all keep fractions:

Table — Multi-reference subject fidelity (output vs. dataset GT), 200 samples.

MethodfracCLIP ↑DINO ↑
Baseline (full tokens)1.000.91840.6761
Naive0.100.86900.4655
Ours0.100.88660.5428
Naive0.150.88020.5171
Ours0.150.89020.5732
Naive0.200.88750.5446
Ours0.200.89660.5816

We will add more qualitative results in the revised draft.

Q3 - The currently reported speedups exclude only the time required to generate a Canny edge map (0.4 ms) and a saliency map (0.7s). We will state these times and update the speedups in the revision to include them.

Q4 - This is a key insight because our model is trained with random token dropping, so it easily adapts to different token selection strategies at inference. Training with random dropping makes the model robust to different sets of tokens, and better token selection strategies can result in better generation

Reviewer_yjNmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Sparse Context, a method that accelerates reference-conditioned image and video generation by reducing the number of reference-image tokens diffusion transformers process. The method fine-tunes the model using random token dropping to improve robustness to sparse inputs, then applies task-aware token selection at inference time, using Canny edges for image editing and saliency maps for subject personalization.

Critical strengths
  • The proposed method is simple and effective. Parse Context uses lightweight LoRA fine-tuning with random token dropping, avoiding major architectural modifications to the underlying generative model.
  • The method supports different token budgets and task-aware selection strategies, allowing users to trade off inference speed and generation quality.
  • Sparse Context consistently outperforms naive token dropping and several token-reduction baselines on multiple image-quality and identity-preservation metrics.
Critical weaknesses
  • The novelty of the paper is limited. The method mainly combines random token dropping during LoRA fine-tuning with heuristic selection strategies such as Canny edges and saliency maps. The methodological contribution beyond existing token-pruning approaches is not sufficiently clear.
  • In the paper, edge is used for the editing task, and saliency is used for the personalization task. However, the justification of this insight is limited. In the supplementary material's limitations section, random sampling sometimes outperforms Canny edge for the editing task. In this regard, it is unclear whether we should use a fixed token-selection strategy across all samples.
  • The experiments for quality evaluation are limited. Several metrics degrade compared with the full-token baseline, particularly at aggressive token reduction ratios. The evaluation also lacks direct measures of instruction-following accuracy or human judgments to verify that the intended edits are correctly performed.
  • The ablation study is also limited. The experiments do not fully disentangle the benefits of LoRA fine-tuning from those of task-aware token selection. Additional controlled comparisons are needed to establish each component's contribution.
  • There is similar work with a training-free approach to address a similar problem [1]. This method selectively prunes reference tokens to improve image editing. Although the problem definition is similar, this work has not been compared in this paper. It is unclear whether the proposed training-based approach outperforms the training-free approach.

[1] Token Pruning for In-Context Generation in Diffusion Transformers. arXiv2026.

  • It would be better to evaluate the proposed method in other image editing scenarios, such as large pose change, face expression change, or complex object deformation. It is unclear whether the proposed method still matches the full-reference model's performance and successfully preserves identity and details.
  • Multi-reference and video evaluation is limited. Multi-reference generation lacks quantitative evaluation of individual object identity and compositional consistency. The video experiments also do not adequately assess temporal coherence or motion consistency.
  • Although this work prioritizes efficiency, the overhead of saliency- or segmentation-based token selection is not clearly accounted for in the reported speedups.
Questions for authors

Please refer to the weaknesses.

8 comments
AuthorsOpenReview ↗

1. Integration with Existing Token Reduction Approaches

Our contribution is a training recipe that is complementary to existing inference time token reduction techniques, rather than a complete merging heuristic. As discussed in Sec 4.3, all three baselines can be readily combined with our trained checkpoint at inference time for further improvements as shown below where all configurations use a keep fraction of f=0.20:

Table — Integrating existing token-reduction methods with our checkpoint (keep fraction 0.20).

PersonalizationImage Editing
MethodCLIP-I ↑DINO ↑CLIP-TI ↑KID ↓|LPIPS ↓CLIP-I ↑DINO ↑CLIP-TI ↑KID ↓
Baseline (100% tokens)0.7500.4510.29021.24|0.4330.8340.7600.3151.48
Naive 20%0.6920.3900.28827.10|0.6870.7620.6310.3162.97
Ours 20%0.7540.4550.27423.01|0.5760.8170.7380.3142.37
Token Avg 20%0.6730.3550.28828.68|0.6620.7350.5980.3132.44
Token Avg + Ours 20%0.7350.4270.27425.19|0.6050.7910.7120.3141.48
ToMe 20%0.6870.3770.28827.50|0.6370.7670.6530.3151.48
ToMe + Ours 20%0.7470.4480.27425.12|0.5290.8180.7670.3161.36

These results show that existing techniques integrated with our checkpoints consistently improves performance.

AuthorsOpenReview ↗

2. Token Selection Strategies

The key insight is that random token dropping during training is what enables flexible token selection at inference. Because the model is never trained to rely on a fixed set of tokens, any token selection strategy can be plugged in at inference time depending on the use case.

We use canny edge for editing and saliency maps for editing and personalization respectively as both are intuitive and cheap options. Canny captures the structural boundaries that editing should preserve and saliency highlights the subject for personalization, but these selection strategies are not hard constraints.

This also explains observation in supplementary limitations that random sampling sometimes outperforms Canny Edge for editing, this is because the model is trained with random dropping which makes random token selection itself a candidate token selection strategy. So canny is default choice in most of the editing cases, but not universally optimal choice and this rule can be adjusted accordingly.

Because the rule is pluggable, a stronger selection method at inference time directly improves results with no change in training. To demonstrate this, we evaluate segmentation mask based token selection strategy which retains tokens from every segment rather than only along edges and it gives better results than canny edge based sampling at the same keep fraction of 20%.

Table — Image editing (PIE-Bench), keep-fraction = 20%. Token-selection rule swapped at inference; training unchanged.

SettingLPIPS ↓CLIP-I ↑DINO-I ↑CLIP-TI ↑KID ↓
Baseline (full tokens)0.4330.8330.7600.3151.50
Naive 20% (no LoRA)0.6870.7620.6310.3162.97
Ours 20% (Canny selection)0.5760.8170.7380.3142.37
Ours 20% (Segmentation selection)0.5100.8240.7780.3161.26
AuthorsOpenReview ↗

3. Comparison with full-token baseline

We agree that several metrics degrade at the most aggressive token reduction ratio. The f=0.05 setting is an overly aggressive operating point that is not representative of how our method works. We will remove f=0.05 and report f=0.15 instead.

To directly measure whether the intended edit is correctly performed, we report Clip-TI (text image alignment) where CLIP-TI remains close to full token baseline and comparable to naive dropping at every keep fraction.

------Table-----------------------Personalization------------|---------Image Editing---------------

MethodClip IDino IClip TIKID|LPIPSClip IDino IClip TIKID
Baseline (100% tokens)0.7500.4510.29021.2|0.4330.8330.7600.3151.5
Naive 10%0.6810.3600.28727.8|0.7190.7230.5530.3093.5
Ours 10%0.7480.4520.27422.7|0.6280.7960.7030.3113.4
Naive 15%0.6910.3800.28726.9|0.7090.7470.5980.3133.1
Ours 15%0.7540.4570.27423.4|0.6060.8060.7200.3132.9
Naive 20%0.6920.3900.28827.1|0.6870.7620.6310.3163.0
Ours 20%0.7540.4550.27423.0|0.5760.8170.7380.3142.4

KID is reported ×10⁻³ in all results

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.