acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Attention-Based Token Pruning for Efficient Multi-Image Synthesis in UMMs

Abstract

Unified Multimodal Models (UMMs) enable powerful multi-image synthesis but incur substantial computation from long visual contexts. While token pruning offers a direct route to accelerating inference, existing methods are primarily adapted from understanding tasks, suffering from insufficient reference image exploration, confused conditional guidance, and pooling-induced ambiguity that degrades generation quality. To address these issues, we propose CATP, the first training-free context-compression method tailored to UMM generation, which **C**alibrates the discrepancies of general **A**ttention-based **T**oken **P**runing for better quality-efficiency trade-off. Concretely, building on independent layer-wise pruning during denoising, we introduce Guidance-Consistent Pruning (GCP) to synchronize pruning decisions across conditioning branches, and a Taylor-Calibrated Attention Metric (TCAM) that derives a theoretically grounded second-order correction for pooling-induced bias. In addition, a Fast Scoring Operator (FSO) fuses the calibration into tiled online Softmax to reduce pruning cost. Extensive experiments across representative UMMs and benchmarks demonstrate that CATP consistently outperforms state-of-the-art token-reduction methods, achieving up to 2.09 attention-core speedup. Code will be released upon paper acceptance.

Primary area:
generative models
Keywords:
Efficient Inference, Unified Multimodal Model, Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vv59OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CATP, a training-free visual-context pruning framework for multi-image synthesis in unified multimodal models. Guidance-Consistent Pruning (GCP) synchronizes reference-token selections across CFG branches to avoid inconsistent visual conditioning. The Taylor-Calibrated Attention Metric (TCAM) addresses the bias introduced by query pooling through a second-order correction, while the Fast Scoring Operator (FSO) integrates this calibration into an efficient tiled computation. Experiments across different model architectures and benchmarks demonstrate favorable generation quality under aggressive context compression, together with meaningful attention acceleration.

Critical strengths

1. Well-motivated analysis of generation-specific pruning problems. The paper identifies how independently selected visual contexts across CFG branches can introduce unintended components into conditional guidance. GCP addresses this issue through a simple and effective synchronization strategy, yielding substantial improvements in generation quality, particularly under aggressive pruning. The connection between the identified problem, its mathematical formulation, and the proposed solution is clear and convincing.

2. An elegant connection between principled scoring and efficient implementation. TCAM derives a meaningful correction for pooling-induced attention bias from local query variation and the distribution of reference keys. FSO further reformulates this correction into statistics that can be efficiently accumulated during tiled attention computation. The components complement each other naturally, providing a favorable balance between scoring accuracy and computational efficiency. The consistent improvements across different models and benchmarks further demonstrate the practical value of the proposed framework.

Critical weaknesses

1. Standalone inference savings could be reported more directly. The main efficiency analysis focuses on attention-core latency, while the appendix provides whole-denoising speedups when CATP is combined with other acceleration methods. The standalone end-to-end denoising benefit of CATP, including the overhead of periodic scoring and token selection, is less explicit. Reporting this would provide a clearer picture of its practical acceleration.

2. The computational matching of adapted baselines could be clarified. The paper states that competing methods are adapted to equivalent computational budgets, but the specific configurations and matching criteria are not fully described. Since these methods reduce computation at different positions within the model, a concise explanation of their operating points would help readers interpret the comparisons more precisely.

Questions for authors
  1. Could the authors report the standalone whole-denoising latency of CATP under the existing efficiency evaluation setting, including scoring and token-selection overhead? A brief addition to the current efficiency results would suffice.

  2. Could the authors clarify how computational budgets were matched across the adapted baselines, particularly SDTM and ToPi? A compact description of the relevant configurations and budget calculation would be helpful.

4 comments
AuthorsOpenReview ↗

(1/2) W1&Q1: On Efficiency Results.

Dear Reviewer, we sincerely thank you for the insightful comments. We organize our responses into 2 sections to address your concerns regarding efficiency results (W1&Q1) and baseline configurations (W2&Q2).

(1/2) W1&Q1: On Efficiency Results.

Thanks for raising this concern. We would like to first clarify that the attention-core latency reported in Fig. 4(a) already includes all scoring, token selection, and pruning overhead, rather than only the reduced attention computation. While model inference involves multiple computational components, CATP specifically targets the costly attention induced by long visual contexts. Therefore, we initially follow prior practices [1,2] in primarily focusing on attention-core latency to directly assess the efficiency gains of the optimized component.

Furthermore, we fully agree with the reviewer’s suggestion and provide the end-to-end denoising speedup of standalone CATP, as shown in Tab. R8 below. Although the overall speedup is inevitably reduced when accounting for unoptimized components such as FFNs, CATP still achieves up to 1.44× acceleration over the entire denoising process. As similarly explored in other domains [3,4], this represents a meaningful practical efficiency gain for such a sequence-level optimization approach.

We have added these results to Fig. 4(b) and clearly discussed the scope of the reported acceleration in Lines 465-473 of the revised manuscript.

Table R8: End-to-end denoising latency (s) and speedup of CATP, across varying numbers of reference images.

Image Number 14710
Full Model34.8541.4748.3155.05
CATP (Ours)32.0533.9536.3238.15
Speedup1.09×1.22×1.33×1.44×

[1] Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR’25.

[2] Xattention: Block sparse attention with antidiagonal scoring. ICML’25.

[3] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. EMNLP’24.

[4] Attend to not attended: Structure-then-detail token merging for post-training dit acceleration. CVPR’25.

AuthorsOpenReview ↗

(2/2) W2&Q2: On Baseline Configurations.

(2/2) W2&Q2: On Baseline Configurations.

Thanks for your valuable suggestion. As described in Lines 376–377 (also Lines 374-376 of the original submission), we restrict all compared methods to reduce attention-core computation in image-conditioned branches, and adjust their configurations to ensure comparable computational budgets under each target retention ratio (i.e. 25% and 10%) by following common practices [1,2]. For Text-Aligned and VisionZip, we directly apply the same per-layer token retention ratio as our CATP. For ToPi and SDTM, whose official implementations retain full computation at certain layers or timesteps, we accordingly reduce the retention ratios in the compressible components to match overall budgets. In order to make fair comparisons, the details are specified as below:

1) ToPi. We keep the first of 28 layers unpruned in both settings. At 25% retention, we follow ToPi’s original schedule, performing full-context computation and updating pruning decisions every 10 timesteps. Ultimately, 5 timesteps retain full computation, while the remaining 44 timesteps retain 17% of reference tokens in 27 of 28 layers, yielding an average retention ratio of (5×28+44×(1+27×17(5\times28+44\times(1+27\times17\\%))/(49\times28)\approx 28\\%. At 10% retention, we extend the pruning interval to 25 timesteps, resulting in 2 full-computation steps and 47 steps retaining 7% of reference tokens in 27 of 28 layers. This yields an average retention ratio of (2×28+47×(1+27×7(2\times28+47\times(1+27\times7\\%))/(49\times28)\approx 14\\%. For clarity, the detailed configurations are summarized in Tab. R9 below.

Table R9: Detailed token retention ratios of the adapted ToPi baseline.

Target Retention Layer 1 (All steps) Layers 2–28 (Full context) Layers 2–28 (Pruned context) Avg. Retention
25%100%5 steps ×\times 100%44 steps ×\times 17%28%
10%100%2 steps ×\times 100%47 steps ×\times 7%14%

2) SDTM. Unlike context token pruning, SDTM reduces the denoising tokens, which also decreases projection computations. Therefore, we adjust its compression settings to match the overall attention-core GFLOPs, including both projection and attention matrix multiplications. Given its sensitivity to compression in the last several denoising steps, we follow the official configuration and retain full computation in the final 5 timesteps for both settings. At 25% retention, full computation is additionally performed every 8 timesteps, while the other steps retain 30% of denoising tokens per layer. At 10% retention, we perform full computation every 10 timesteps and retain 20% of denoising tokens otherwise. The resulting average attention-core GFLOPs for the Full Model, SDTM, and CATP are summarized in Tab. R10 below.

Table R10: Detailed configurations of the adapted SDTM baseline and average attention-core GFLOPs comparison with CATP.

Target Retention Full-computation Reduced-computation SDTM GFLOPs CATP (Ours) GFLOPs
25%4 + 6 steps39 steps (retain 30% tokens) 391.8311.4
10%4 + 5 steps40 steps (retain 20% tokens) 315.9198.0

Notably, across all the above settings, CATP consistently incurs no more computational cost than the compared baselines, while still delivering notable performance gains, as shown in Tabs. 1 and 2, further demonstrating the effectiveness of our method. These detailed configurations have been added to Appendix C of the revised manuscript.

[1] Visionzip: Longer is better but not necessary in vision language model. CVPR’25.

[2] Collaborative multi-mode pruning for vision-language models. CVPR’26.

Thank you for the detailed rebuttal. The additional standalone denoising results and clarification of baseline configurations adequately address my concerns. The response further strengthens my positive assessment of the paper, and I maintain my acceptance recommendation.

Dear Reviewer,

Thank you very much for your positive feedback and continued support of our work. We sincerely appreciate your time and thoughtful consideration throughout the review process.

 

Best regards,

The Authors

Reviewer_EpiDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CATP, a training-free method for accelerating multi-reference image synthesis in unified multimodal models by pruning reference-image KV tokens during denoising. Its main components are layer-wise token selection, guidance-consistent pruning across conditioning branches, a Taylor-based correction to attention scores computed with pooled queries, and a fused scoring operator to reduce overhead. On Bagel and SenseNova-U1, CATP generally preserves generation quality better than the evaluated token-reduction baselines at matched retention ratios across three benchmarks. Ablations support the contributions of its components. The reported maximum 2.09× speedup is for the attention core, rather than the full generation pipeline.

Critical strengths
  1. A well-motivated contribution beyond prior dynamic pruning. Unlike ToPi, CATP addresses pruning inconsistencies between UMM conditioning branches. The GCP ablation is consequential: at 10% token retention, it raises the MacroBench average from 6.91 to 8.07, indicating that this generation-specific design accounts for a substantial part of the quality gain.

  2. Broad and reasonably robust empirical validation. CATP is evaluated on two UMMs and three multi-image benchmarks at different retention ratios. Additional evaluators and a five-seed analysis support the reported quality advantage. This makes the improvement more persuasive than a result confined to one model or evaluation metric.

Critical weaknesses
  1. The practical efficiency gain is not fully established. The headline 2.09× speedup measures the attention core, not complete image generation. Although the appendix reports denoising speedups when CATP is combined with other methods, it does not clearly report CATP alone with all scoring and pruning overhead included. This limits my ability to judge the paper’s central quality–latency trade-off.

  2. The independent benefit of TCAM is modest relative to GCP. In Table 3, TCAM adds only 0.16 and 0.08 MacroBench points at 25% and 10% retention, respectively, after GCP is applied. Without evidence that these incremental gains are stable—or that the calibrated scores better approximate the unpooled attention scores—it is difficult to assess how much the proposed calibration, a central technical contribution, improves upon the simpler GCP-based approach.

Questions for authors
  1. Efficiency clarification: What is CATP’s standalone end-to-end generation latency, including scoring and pruning overhead, relative to the full model and the adapted ToPi baseline at comparable output quality? The reported 2.09× figure measures only the attention core, so this result would materially affect my assessment of the practical speedup.

  2. TCAM contribution: How stable is TCAM’s incremental gain over GCP alone across random seeds, and does TCAM measurably improve agreement with the unpooled attention-based token ranking? These results would help establish whether the proposed calibration provides a reliable benefit beyond the simpler GCP-based method.

4 comments
AuthorsOpenReview ↗

(1/2) W1&Q1: On Efficiency Clarification.

Dear Reviewer, we sincerely thank you for the insightful comments. We organize our responses into 2 sections to address your concerns regarding efficiency clarification (W1&Q1) and TCAM’s contribution (W2&Q2).

(1/2) W1&Q1: On Efficiency Clarification.

We thank the reviewer for raising this concern. We would like to first clarify that the attention-core latency reported in Fig. 4(a) already includes all scoring, token selection, and pruning overhead, rather than only the reduced attention computation. While model inference involves multiple computational components, CATP specifically targets the costly attention induced by long visual contexts. Therefore, we initially follow prior practices [1,2] in primarily focusing on attention-core latency to directly assess the efficiency gains of the optimized component.

Furthermore, we fully agree with the reviewer’s suggestion and additionally evaluate the end-to-end generation efficiency, as reported in Tab. R5 below. While the overall speedup is inevitably lower when accounting for unoptimized components such as FFNs, CATP still achieves up to 1.44× speedup of overall generation at 25% token retention, outperforming the adapted ToPi (up to 1.35×) while achieving better generation quality (8.46 v.s. 8.15 on MacroBench Avg.).

Table R5: End-to-end generation latency (s) and speedup with 10 reference images by various methods, along with the overall MacroBench average score.

Image NumberLatency (s) Speedup MacroBench Avg. 
Full Model55.051.00×8.14
ToPi (25% retention)40.821.35×8.15
CATP (Ours, 25% retention) 38.151.44×8.46
CATP (Ours, 10% retention) 34.971.57×8.15

Moreover, Tab. R5 also indicates a more pronounced efficiency advantage of CATP at a directly comparable generation quality on MacroBench. Specifically, CATP at 10% retention achieves up to 1.57× speedup, compared with 1.35× for ToPi at 25% retention and 1.00× for the full model, while all three exhibit comparable MacroBench average scores (8.15, 8.15, and 8.14, respectively).

These results demonstrate that CATP offers a more favorable and practical quality–latency trade-off. Given that it specifically targets redundancy along the sequence dimension, this achieved speedup represents a practical efficiency gain for such an optimization scheme [3,4]. We have added these results to Fig. 4(b) and clearly discussed the scope of the reported acceleration in Lines 465-473 of the revised manuscript.

[1] Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR’25.

[2] Xattention: Block sparse attention with antidiagonal scoring. ICML’25.

[3] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. EMNLP’24.

[4] Attend to not attended: Structure-then-detail token merging for post-training dit acceleration. CVPR’25.

AuthorsOpenReview ↗

(2/2) W2&Q2: On TCAM.

(2/2) W2&Q2: On TCAM.

Thanks for the insightful concern on TCAM’s contribution. We would like to address it from the following three aspects:

Firstly, TCAM provides stable incremental gains beyond GCP alone. Following the reviewer’s suggestion, we supplement ablation experiments over 5 random seeds. As shown in Tab. R6 below, incorporating TCAM improves the mean MacroBench Avg. by 0.144 and 0.112 points at 25% and 10% token retention, respectively. These gains exceed the corresponding standard deviations, suggesting that TCAM provides consistent and stable incremental benefits beyond the GCP-only baseline, rather than merely random variations in generation. We have included this result in Tab. G of the revised manuscript.

Table R6: MacroBench scores (mean ± std) w/o and with TCAM on Bagel at 25% and 10% token retention ratios, with GCP enabled in all settings.

Method25%Retention  10% Retention
1–56–10Avg.1–56–10Avg.
w/o TCAM8.651 ± 0.0477.976 ± 0.0448.314 ± 0.0218.302 ± 0.0557.863 ± 0.0588.084 ± 0.009
with TCAM8.727 ± 0.0338.189 ± 0.0348.458 ± 0.0208.434 ± 0.0487.955 ± 0.0458.196 ± 0.027

Secondly, TCAM indeed improves agreement with the unpooled attention-based token ranking. Following the reviewer’s suggestion, we additionally evaluate the Top-K overlap with the unpooled attention scores. As reported in Tab. R7 below, TCAM improves the overlap over the baseline pooled scores by 0.38% and 0.54% at 25% and 10% retention, respectively. This provides direct evidence that our second-order calibration improves the fidelity of attention-based token selection, which is consistent with the objective of our theoretical derivation. We have incorporated this result and the corresponding discussion in Tab. 5 and Lines 500-503 of the revised manuscript.

Table R7: Top-K overlap with dense attention scores at different retention ratios.

Method Top-25% Top-10% 
Pooling92.98%89.32%
Pooling + TCAM 93.36%89.86%
Improvement+0.38%+0.54%

Finally, we would like to further highlight our contribution. TCAM’s primary contribution lies in investigating a new problem of pooling-induced bias in attention-based importance estimation, explicitly modeling this bias, and preliminarily validating the effectiveness of calibration with a practical mechanism. Given the remaining approximation gap, our empirical experiments show that the exact second-order correction can further improve MacroBench Avg. to 8.31 at 10% retention, yet is at the cost of substantial scoring overhead that is difficult to eliminate in practice. Therefore, we consider more accurate bias estimation, particularly without compromising computational efficiency, to remain a promising direction for future research. We have revised in Lines 508-514 of the manuscript to better clarify the contributions and limitations of TCAM.

Reviewer_EpiDOpenReview ↗

Post-Rebuttal Assessment: Maintain Weak Acceptance

Thank you for the additional experiments. I see two main contributions: guidance-consistent pruning and a practical balance between generation quality and speed. The new end-to-end measurements largely address my efficiency concern. With ten reference images, CATP achieves a 1.57× speedup. Its MacroBench average remains close to that of the full model. The five-seed ablation also provides stronger evidence for a small but stable gain from TCAM beyond GCP.

My remaining concern is that TCAM’s gains are modest. This applies to both generation quality and agreement with unpooled attention rankings. The reported speedup also applies to the tested setting. It should not be assumed to hold for all reference-image counts.

Overall, the new experiments largely address my original concerns. The practical speedup and generation-specific pruning design support acceptance. I therefore maintain my rating of 3 (Weak acceptance).

Dear Reviewer,

Many thanks for your recognition of the contributions and practical value of our work. We sincerely appreciate your valuable time and continued support throughout the review process!

We are also actively exploring further research questions inspired by this work and look forward to making continued progress.

 

Best regards,

The Authors

Reviewer_rYxxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CATP, a training-free token pruning method that speeds up multi-image synthesis in unified multimodal models. It prunes the reference-image KV cache layer by layer at a few denoising steps, scoring tokens by the attention they receive. It shares pruning decisions across CFG branches to keep the guidance clean, corrects the bias of scoring with pooled queries using a second-order Taylor term, and implements the scoring as a fused kernel. On Bagel and SenseNova-U1, CATP outperforms adapted pruning baselines across three multi-image benchmarks, with up to 2.09× attention-core speedup.

Critical strengths
  1. The observation behind GCP is interesting and useful. When the CFG branches prune independently, they keep different reference tokens, and that mismatch leaks into the guidance and gets amplified. The paper explains this clearly, the fix is simple, and it gives the largest gain, especially at low retention ratios.
  2. The experiments are run carefully for this kind of paper: two UMMs, several benchmarks, multiple MLLM judges, multiple seeds, and a fused kernel that keeps the scoring cost low.
Critical weaknesses
  1. The speedup is not convincing enough. The paper only reports attention-core speedup, and the headline 2.09× comes from the most favorable setting (10 reference images at 1024 resolution). With fewer images the gain drops quickly (around 1.2× with one image), and end-to-end latency for CATP alone isn’t reported. All experiments also use 50 sampling steps. With fewer steps, as in distilled models, the pruned KV cache is reused much less, so the benefit may shrink further. For an acceleration paper, end-to-end results across image counts and step counts are essential.
  2. The contribution of TCAM is overclaimed. TCAM gets the most theoretical treatment in the paper, but its gain is very small (e.g., +0.08 at 10% retention), which is hard to call a real improvement on MLLM-judged generation benchmarks. It also needs a heuristic scaling factor to work, and the “exact” second-order correction actually performs worse. The paper never checks directly whether the calibrated scores are closer to the dense scores.
  3. “Better than the full model” is not convincing. Several lossy acceleration methods, including SDTM and TaylorSeer on their own, also score higher than the full model. This suggests the benchmark and judge setup may not be very sensitive to dropping reference information, which makes the claimed quality gains, and the small differences between methods, hard to trust.
Questions for authors
  1. What is the end-to-end latency and memory saving of CATP alone, with fewer reference images (e.g., 1–3) and with fewer sampling steps (e.g., a distilled few-step model)?
  2. Does TCAM actually bring the pooled scores closer to the dense scores, for example in top-K overlap? Given the small gains and the need for a heuristic λ, how much does the second-order correction matter in practice?
  3. Why do several lossy methods (SDTM, TaylorSeer) score higher than the full model? Can you add a human evaluation or a non-MLLM consistency metric to confirm that the gains are real?
8 comments
AuthorsOpenReview ↗

(1/3) W1&Q1: On Efficiency Results.

Dear Reviewer, we sincerely thank you for the insightful comments. We organize our responses into 3 sections to address your concerns regarding efficiency results (W1&Q1), TCAM’s contribution (W2&Q2) and evaluation metrics (W3&Q3).

(1/3) W1&Q1: On Efficiency Results.

1) End-to-end latency across image numbers.

We would like to first clarify the scope of our method and evaluation. While model inference involves multiple computational components, CATP specifically targets the costly attention induced by long visual contexts. Therefore, following prior practices [1,2], our original experiments primarily focused on attention-core latency to directly assess the efficiency gains of the optimized component.

By valuing the reviewer’s suggestion, we evaluate the end-to-end latency of CATP across varying numbers of reference images, as reported in Tab. R1 below. Due to the inclusion of unoptimized components such as FFNs, the overall speedup inevitably decreases, while CATP still achieves up to 1.44× with 10 images and 1.09×~1.18× with 1~3 images.

Table R1: End-to-end latency (s) and speedup of CATP across varying numbers of reference images.

Images12345678910
Full Model34.8536.9539.1541.4743.7346.0348.3150.6152.8755.05
CATP32.0532.6433.2533.9534.7835.7736.3236.9537.5938.15
Speedup1.09×1.13×1.18×1.22×1.26×1.29×1.33×1.37×1.41×1.44×

Besides, we also would like to politely highlight the practical value of CATP from two perspectives:

On the one hand, multi-image synthesis involving numerous images is currently a common and practically relevant scenario (e.g. for multi-subject composition and fashion design), where we target the increasingly challenging quality–efficiency trade-off with growing images numbers. Therefore, we tend to place more emphasis on the efficiency gains under longer visual contexts, while considering the fewer-image results as complementary evidence of its broad applicability.

On the other hand, as CATP specifically targets redundancy along the sequence dimension, the achieved 1.44× end-to-end speedup represents a meaningful gain for this scheme [3,4]. Moreover, CATP can be readily integrated with methods optimizing other computational components, providing complementary context compression. As already validated in Tab. H (also Tab. G in the original submission) of Appendix, such combination further achieves 3.06× end-to-end speedup.

In the revised manuscript, we have incorporated the detailed end-to-end latency results into Fig. 4(b) and Tab. I, along with the corresponding discussion in Lines 465-473.

2) End-to-end latency across sampling steps.

We first politely clarify that the original 50-step setting follows the default inference configuration of the evaluated UMMs for multi-image synthesis, ensuring reliable generation quality and fair comparisons under their native inference settings.

Following the reviewer's suggestion, we conduct evaluation under fewer sampling steps, including the commonly adopted 20- and 8-step settings. As shown in Tab. R2 below, CATP maintains up to 1.38× speedup even with only 8 sampling steps, remaining practical in efficiency. The modest reduction (compared to 1.44× under 50 steps) is expected, as the scoring and pruning overhead is amortized over fewer steps. Nevertheless, further acceleration under extremely compressed few-step generation remains a widely recognized challenge in the community [4]. We have added this result into Fig. 4(c), along with a discussion in Lines 474-480 of the revised manuscript.

Table R2: End-to-end latency (s) and speedup of CATP across varying sampling steps with 10 reference images.

Steps82050
Full Model7.7320.9855.05
CATP5.6014.7538.15
Speedup1.38×1.42×1.44×

3) Memory saving.

We politely clarify that CATP primarily targets inference acceleration rather than memory saving, because in this multi-image synthesis scenario, the context is not sufficiently long to make GPU memory a major bottleneck. For instance, Bagel’s model parameters occupy ≈30\approx 30 GB of GPU memory, whereas its full KV Cache requires only ≈2.5\approx 2.5 GB even with 10 reference images. Although CATP reduces the active KV Cache footprint to ≈0.7\approx 0.7 GB, the potential overall savings are inherently bounded by the KV Cache’s small share of GPU memory. Therefore, we do not emphasize memory reduction as a primary advantage of CATP.

[1] Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR’25.

[2] Xattention: Block sparse attention with antidiagonal scoring. ICML’25.

[3] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. EMNLP’24.

[4] Attend to not attended: Structure-then-detail token merging for post-training dit acceleration. CVPR’25.

AuthorsOpenReview ↗

(2/3) W2&Q2: On TCAM Contribution.

(2/3) W2&Q2: On TCAM Contribution.

Thanks for the insightful concern on TCAM’s contribution. We would like to address it from the following four aspects:

Firstly, for the modest gain at 10% retention, we politely clarify that under highly constrained token budgets, the inevitable loss of substantial reference information is hard to be fully compensated by improved importance estimation alone, making further quality improvements rather challenging. Notably, TCAM still achieves a 0.16-point gain with 8–10 reference images at 10% retention, demonstrating its practical value in our target multi-image scenarios.

Secondly, TCAM indeed brings the calibrated scores closer to dense scores. Following the reviewer’s suggestion, we additionally evaluate the Top-K overlap with dense attention scores. As reported in Tab. R3 below, TCAM improves the overlap over the original pooled scores by 0.38% and 0.54% at 25% and 10% retention, respectively. This provides direct evidence that our second-order calibration improves the fidelity of attention-based token selection, which is consistent with the objective of our theoretical derivation. We have incorporated this result and the corresponding discussion in Tab. 5 and Lines 500-503 of the revised manuscript.

Table R3: Top-K overlap with dense attention scores at different retention ratios.

MethodTop-25% Top-10% 
Pooling92.98%89.32%
Pooling + TCAM 93.36%89.86%
Improvement+0.38%+0.54%

Thirdly, the heuristic factor λ\lambda provides a simple yet optional adjustment to the approximate correction. As already discussed in Lines 790-792 (Lines 682-685 in the original submission), our second-order derivation involves approximations, such that the resulting calibration term captures the correction trend of pooling-induced bias rather than its exact value. Thus, λ\lambda is introduced to regulate the correction strength and mitigate potential overcorrection. As shown in Tab. B of Appendix, the correction remains effective even without heuristic scaling (λ=1\lambda=1), while adjusting it yields additional gains. In practice, performance remains relatively stable across λ∈[0.5,1.0]\lambda\in[0.5,1.0], requiring no fine-grained tuning, and we therefore empirically fix it to 0.5 across all experiments. We have further clarified this in the revision.

Finally, we would like to further highlight our contribution. TCAM’s primary contribution lies in investigating a new problem of pooling-induced bias in attention-based importance estimation, explicitly modeling this bias, and preliminarily validating the effectiveness of calibration with a practical mechanism. Given the remaining approximation gap, our empirical experiments show that the exact second-order correction can further improve MacroBench Avg. to 8.31 at 10% retention, yet is at the cost of substantial scoring overhead that is difficult to eliminate in practice. Therefore, we consider more accurate bias estimation, particularly without compromising computational efficiency, to remain a promising direction for future research. We have revised in Lines 508-514 of the manuscript to better clarify the contributions and limitations of TCAM.

AuthorsOpenReview ↗

(3/3) W3&Q3: On Evaluation Metrics.

(3/3) W3&Q3: On Evaluation Metrics.

We sincerely thank the reviewer for raising concerns about the reliability of current evaluation metrics. We would like to address it from four aspects below.

1) Attribution of potential quality gains over the full model.

In multi-image synthesis, conditioning inputs often contain substantial redundant or irrelevant information. Consequently, moderate computational reduction may help mitigate interference from such redundancy and enable better focus on critical contents, potentially improving generation quality in certain cases. Similar observations have been reported in related LLM and MLLM studies [1,2], particularly in long-context scenarios. We have also illustrated such improvements, e.g. recovering missing subjects, through qualitative examples in Fig. B of Appendix.

2) Justification of the adopted MLLM-based evaluation.

Typically, multi-image synthesis involves diverse and complex transformations, including facial expression / clothing / pose changes, as well as subject interactions, spatial composition, etc., which are difficult to comprehensively assess using fixed metrics. Therefore, existing benchmarks commonly adopt MLLM-based evaluation to capture these diverse aspects of generation quality. Given this, we respectfully clarify that our evaluation strictly follows the official evaluation procedures and metrics of the respective benchmarks, with consistent settings across all methods. To mitigate potential evaluation biases, we have already provided qualitative comparisons in Fig. 3, cross-evaluator judges in Appendix A.3, and statistical stability analysis across random seeds in Appendix A.4. These results consistently support the reliability of CATP’s performance advantages.

3) Additional non-MLLM consistency evaluation.

Nevertheless, we fully value the reviewer's suggestion and further conduct a non-MLLM consistency evaluation. Following [4,5], we compute the DINOv2-based Image Similarity (DINOv2-I) [3] between corresponding reference and generated regions on MacroBench for all compared methods at 25% token retention.

As shown in Tab. R4 below, CATP still consistently outperforms compared acceleration methods on DINOv2-I, surpassing the second-best method by 1.53 points on average. This provides further evidence of its effectiveness in preserving reference information. This evaluation is now presented in Tab. 3 and Lines 417-429 of the revised manuscript.

Table R4: Comparison of DINOv2-I scores by various methods on MacroBench using Bagel at 25% token retention.

Method1–34–56–78-10Avg.
Full Model60.6152.4749.1643.5349.04
Text-Aligned54.8150.1346.8241.5446.46
VisionZip51.4247.6344.2438.9343.80
SDTM52.1545.3840.5033.7940.36
ToPi54.0449.5846.7141.2046.09
CATP (Ours)56.8851.4848.4342.9747.99

Notably, such feature similarity alone cannot fully capture spatial composition, subject interactions, or instruction following [6], which alternatively can be assessed by MLLM-based evaluation. Therefore, we consider this supplemented evaluation as complementary to, rather than a replacement for, the original MLLM-based assessment.

4) Modification of quality claims.

Finally, we acknowledge that existing metrics may not fully capture quality differences in complex multi-image synthesis, which remains an inherent challenge for this task across the broader community [6,7]. Accordingly, we have moderated our claims of outperforming the full model and instead focus on CATP's favorable quality-efficiency trade-off and compatibility in revision.

[1] Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. ICLR’25.

[2] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. EMNLP’24.

[3] Dinov2: Learning robust visual features without supervision. TMLR’24.

[4] Identity decoupling for multi-subject personalization of text-to-image models. NeurIPS’24.

[5] Xverse: Consistent multi-subject control of identity and semantic attributes via dit modulation. NeurIPS’25.

[6] ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing. ICCV’25.

[7] Mico-150k: A comprehensive dataset advancing multi-image composition. CVPR’26.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.