acceptodds
Withdrawn from ICLR 2027

Twilight: Thinking with Generated Images via Lightweight KV Cache

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Thinking with Generated Images (TwGI) has emerged as a promising paradigm that enables vision-language models to actively synthesize novel visual content as intermediate reasoning steps, transcending the limitations of static input images. However, due to the inclusion of numerous image tokens and multiple classifier-free guidance (CFG) samples, TWGI's reasoning process incurs significant memory pressure, limiting scalability to long-sequence generation. By analyzing attention patterns, we reveal that TwGI's KV cache contains significant redundancy, as some heads maintain low attention scores on previous tokens. Therefore, we propose Twilight, the first training-free KV cache eviction framework tailored for TwGI. Specifically, we introduce a novel normalized standard deviation metric to identify modality-specific retrieval heads that attend globally to historical tokens. Leveraging the distinctive attention patterns of retrieval heads and streaming heads across text and image generation stages, we design targeted KV cache eviction strategies and sparse attention mechanisms. Our approach selectively retains critical tokens while discarding redundant ones, significantly reducing memory consumption without compromising generation quality. Extensive experiments validate the effectiveness of our approach. Twilight maintains or even improves generation performance through a training-free approach, while reducing KV-cache memory by over 60% on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache eviction, Thinking with images, Twilight

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C3X6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Twilight is a training-free KV-cache compression framework for interleaved text–image generation. It identifies modality-specific retrieval heads using normalized attention standard deviation and applies retention policies tailored to head types and classifier-free guidance branches. Experiments on TwGI report substantial cache reductions while maintaining generation performance, supported by component ablations and decoding-efficiency measurements.

Critical strengths
  1. Meaningful efficiency gains. Twilight retains approximately 29–35% of the original KV cache on DPG-Bench while maintaining comparable scores. This is a practically useful result without retraining the underlying model.
  2. A coherent design for interleaved generation. The modality-specific head assignments and boundary- and CFG-aware retention policies address features that a uniform text-oriented policy would miss. The ablations support several of these distinctions.
Critical weaknesses

The incremental contribution is not sufficiently isolated. The proposed head-identification statistic measures attention dispersion, which is not necessarily equivalent to the need for long-range retrieval. The existing ablations establish the importance of head counts and modality-specific assignments, but do not directly compare selection criteria under the same retention policy and budget. Table 10 uses Twilight’s own identification method, so it does not resolve this question. Together with limited positioning against head-aware image-generation methods such as SSD and HeadKV, this leaves uncertainty about the contribution beyond adapting known head-aware compression principles to TwGI.

Questions for authors
  1. Clarification: When switching modalities, how is historical context supplied to heads that change from streaming to retrieval? Is the sequence re-prefilled, and which transition or prefill costs are included in Figure 6?
  2. Evidence that could change my assessment: With retention rules and cache budget fixed, how does normalized standard deviation compare with a simple alternative, such as attention mass outside the retained window? This would help establish the benefit of the proposed selector.
Reviewer_vLRROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Twilight is a training-free KV-cache eviction framework for interleaved text–image generation. It identifies modality-specific retrieval heads using normalized attention standard deviation and applies different retention policies across heads and CFG branches. Experiments report substantial KV-cache reductions while largely preserving generation quality, alongside improved per-token decoding latency at long context lengths.

Critical strengths
  1. The paper addresses the substantial memory overhead introduced by intermediate images and multiple CFG branches in multimodal reasoning.
  2. Distinguishing text and image generation, preserving modality boundaries, and tailoring retention to CFG conditions are reasonable adaptations to this setting.
  3. The evaluation includes image generation, supplementary visual understanding tasks, and ablations examining head allocation, attention sinks, modality boundaries, and context retention.
Critical weaknesses
  1. The overall framework closely follows DuoAttention. The proposed normalized standard deviation measures attention concentration rather than dependence on distant information: a head attending sharply to a distant critical token may still require global context. Direct comparisons with alternative identification criteria are missing.
  2. Text and image generation use different retrieval-head sets, while the appendix states that evicted entries are permanently removed. It remains unclear how a head that switches from streaming to retrieval accesses previously discarded history, or whether additional retention or recomputation is required.
  3. KeyDiff and R-KV suffer severe quality degradation, but their adaptation details and budget–quality curves are insufficiently documented. The DuoAttention-style baseline uses Twilight’s head identification, so it does not establish the advantage of the proposed identification method.
  4. Reported speedups primarily concern per-token decoding at long context lengths. End-to-end task latency, throughput under specified batch sizes, and generation quality at these long-context operating points are needed to establish practical benefits.
Questions for authors
  1. Could you compare the proposed head-identification metric with attention entropy, attention mass outside the retained window, and an output-error-based criterion?
  2. How are caches managed when a head changes from streaming to retrieval across modalities? Please clarify any additional storage or recomputation costs.
  3. Could you report end-to-end latency and throughput, together with generation quality at the context lengths used for speed measurements?
Reviewer_nuCXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents itself as a novel method for KV caching, specially tailored for the Thinking with Generated Images (TwGI) approach. In essence, it is more of an analysis report with interesting insights, with a first implementation based on these insights. The content of the paper investigates attention patterns in TwGI processes. The analysis is interesting, and the experiments suggest that the resulting insights can reduce memory usage with minimal impact on generation quality. That said, the paper's base motivation, improving upon the already existing KV cache mechanism in the TwGI approach, starts from a method that is not peer reviewed and with little academic traction. This, combined with limited methodological novelty and unclear exposition, raises concerns regarding the merit of this investigation unfortunately.

Critical strengths
  • The attention analysis is the strongest part of the paper. It provides interesting observations about the TwGI process, and the roles of different heads in it.
  • The experiments indicate that the resulting insights can indeed be leveraged for better memory consumption with minimal degradation in output quality (or sometimes even improve it).
Critical weaknesses
  • First and foremost, the paper addresses a niche approach, and improves a niche problem in it. This renders the paper's merits dubious.
  • In addition, the contribution of the paper is minimal. The main contribution is arguably the analysis of attention patterns in TwGI. Although interesting, it is still rather limited, and does not discuss alternative approaches. It would have been more meaningful if it could show fundamental limitations of existing eviction strategies, or explain why those strategies could not be adapted according to the revealed insights.
  • For the proposed method itself, the approach largely follows established VLM head-classification practices. The main novelty introduced, seems to be a natural extension of STD to sequences, and no alternatives to justifications are offered to suggest otherwise. The paper neither establishes why existing metrics are inadequate nor convincingly distinguishes its novel contributions from the application of known techniques. Specifically, the paper simply states that eviction approaches designed for text are inappropriate, and cites two papers from over two years ago, which are irrelevant currently.
  • Lastly and with less significance, the exposition is unclear. The introduction devotes substantial space to establishing the importance of TwGI, instead of motivating the underlying eviction problem, explaining which existing strategies are inadequate and why, ect. Figure 1 is largely an infographic that contributes little substantive information, and the intended takeaways from Figure 5 are unclear. The method section also lacks justification for its design choices throughout and a clear distinction between established components as apposed to novel contributions is also required.
Questions for authors

For future revisions, I would encourage the authors to reframe the paper as an analysis report paper, with the technical method offered as a secondary contribution. In order to do so, perhaps another approach or a family of models can be investigated similarly. In addition, I would encourage the authors to focus more of justification for design choices, and to explain more clearly why different approaches could not or should not be considered for said choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.