- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how attention heads in unified vision-language models respond to changes in conditioning. By measuring activation differences between the positive and negative branches of classifier-free guidance (CFG), the authors identify heterogeneous yet largely stable head-level condition-change awareness patterns in Janus and Muddit. Based on these observations, they propose HeadTuned-CFG, a training-free method that selectively interpolates or extrapolates attention-head activations to improve conditional generation. A KL-projection interpretation is provided under a local Gaussian assumption. Experiments demonstrate substantial improvements in text-to-image (T2I) generation, particularly on Muddit, and more modest gains in image-to-text (I2T) generation.
Critical strengths
1. Elegant connection between mechanistic observations and practical intervention. The paper establishes a compelling connection between head-level condition sensitivity and inference-time guidance tuning. The proposed method is simple, training-free, and directly reuses the positive and negative branches already available in CFG. The cross-sample stability of the head rankings and the targeted selection ablations provide meaningful support for this design.
2. Strong T2I performance with meaningful experimental validation. The improvements on Muddit are substantial, particularly the reduction in FID from 20.90 to 16.29, accompanied by improvements across several complementary metrics. The method also improves Janus despite its different generation paradigm. Comparisons against tuned CFG and ablations involving layer-based, random, and reversed head selection strengthen the practical contribution.
Critical weaknesses
1. The novelty relative to closely related attention-level guidance methods is insufficiently established. Prior work such as HeadHunter [1] has investigated head selection for attention-based guidance, while Normalized Attention Guidance (NAG)[2] has explored extrapolation between positive and negative attention features. The proposed condition-change-based ranking and selective interpolation strategy appear to offer meaningful distinctions, but these differences are not sufficiently articulated. In particular, the paper does not clearly establish which insights are new beyond existing head-level intervention and feature-level guidance methods. A more focused comparison would substantially strengthen the contribution.
2. The mechanistic interpretation does not yet fully explain the proposed intervention strategy. While the probing results establish stable heterogeneity in head responses, the connection between response magnitude and the usefulness of condition-specific information remains somewhat indirect. The strong performance of the layer-prior baseline also suggests that a substantial portion of the improvement may come from layer-level structure rather than precise head-level specialization. Furthermore, the Gaussian analysis explains the interpolation/extrapolation form but does not directly motivate why T2I benefits from suppressing low-response heads whereas I2T benefits from amplifying high-response heads. Clarifying this connection would make the paper's central insight more convincing.
3. The I2T results provide relatively limited support for the broader applicability of the method. The I2T improvements over VCD on Janus are modest, while the method does not improve Muddit on the same task. Moreover, the reported MME improvement is obtained after selecting hyperparameters on the evaluated benchmark itself, limiting its value as independent validation. These observations do not undermine the strong T2I results, but they leave the effectiveness of the proposed framework across both generation directions less convincing than the paper's overall framing suggests.
References
[1] D. Ahn, J. Kang, S. Lee, et al. Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models. Advances in Neural Information Processing Systems (NeurIPS), 2025.
[2] D.-Y. Chen, H. Bandyopadhyay, K. Zou, and Y.-Z. Song. Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models. Advances in Neural Information Processing Systems (NeurIPS), 2025.
Questions for authors
-
Could the authors clarify the key conceptual and technical differences from HeadHunter and NAG, particularly regarding the head-selection criterion and the interpolation/extrapolation mechanism?
-
What motivates the asymmetric intervention strategies across T2I and I2T? Specifically, why should low condition-change awareness indicate responses worth suppressing in T2I, while high awareness indicates responses worth amplifying in I2T? How much of this behavior is attributable to head-level specialization beyond layer-level structure?
-
Could the authors clarify the interpretation of the I2T results, particularly the lack of improvement on Muddit and the evaluation implications of selecting MME hyperparameters on the reported benchmark? Also, for Janus, what motivates using head rankings obtained during decoding to guide interventions applied only during prefill?
5 comments
Question2 & Weakness2
Why interpolation in T2I and extrapolation in I2T. A unified VLM serves both generation and understanding, and the two tasks ask different things of the guidance. The direction of tuning was chosen empirically for each task; our interpretation is as follows.
T2I is a generation task. The output is a whole image, and its quality relies on the generative ability of the pretrained model, which the tuning should preserve. Interpolation keeps the activation of a head between the two branches of CFG, so it does not move the activations away from what the model itself produces. We apply it only to the agnostic heads, whose difference between the two branches carries little condition information. The guidance is thus weakened only where it mostly amplifies noise, and the generation quality of the pretrained model is maintained.
I2T, in the tasks we study, is an understanding task. The output is a short answer to a question about the image (yes or no in POPE and MME) rather than a long sequence, so what matters is not the quality of a long generation but whether the answer follows the semantic content of the image. The answer therefore needs stronger guidance from the image, and we extrapolate the aware heads, which respond most strongly to it.
Head level beyond layer level. Layer-level structure accounts for part of the gain, and selecting individual heads by adds to it consistently. We conducted additional experiments that compare the -ranked selection with the layer prior on all ten MJHQ categories, and over ten runs with different seeds on one category (Section 5.3, Appendix H). Note that the late-layer prior entry of the submitted Muddit ablation was corrected from 30.71 to 31.97 (General Response, point 1).
FID on each MJHQ category and on MJHQ-full (Tables 6 and 24):
| animals | art | fashion | food | indoor | landscape | logo | people | plants | vehicles | full | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muddit, late-layer prior | 31.97 | 37.23 | 34.45 | 34.06 | 36.10 | 35.16 | 36.98 | 37.61 | 44.15 | 31.24 | 16.73 |
| Muddit, -ranked (ours) | 30.41 | 36.70 | 34.02 | 33.60 | 36.17 | 34.56 | 36.65 | 36.88 | 43.01 | 30.93 | 16.29 |
| Janus, early-layer prior | 26.55 | 30.37 | 29.41 | 31.01 | 27.28 | 25.03 | 39.24 | 33.32 | 30.29 | 23.19 | 10.15 |
| Janus, -ranked (ours) | 25.95 | 29.93 | 28.83 | 31.37 | 27.03 | 24.81 | 37.60 | 32.02 | 29.37 | 23.16 | 9.92 |
FID on MJHQ-art over ten runs with different seeds (Tables 25 and 26):
| Layer prior | -ranked (ours) | Difference | Runs in which ours is lower | |
|---|---|---|---|---|
| Muddit | 10 / 10 | |||
| Janus | 8 / 10 |
We would also like to note that the correct direction of the layer prior is itself determined by the probe: in the opposite direction, the layer prior is worse than vanilla CFG on both models.
FID on MJHQ-animals (Tables 5 and 23):
| CFG | Layer prior, probed direction | Layer prior, opposite direction | -ranked (ours) | |
|---|---|---|---|---|
| Muddit | 33.91 | 31.97 | 38.84 | 30.41 |
| Janus | 27.32 | 26.55 | 27.63 | 25.95 |
We have added the head-selection results to the revised manuscript, in Section 5.3 (Tables 5 and 6) and Appendix H (Tables 23–26), highlighted in blue.
Question3 & Weakness3
MME hyperparameters. We agree that the originally submitted MME result was not an independent validation. We now report MME-P, the ten perception subtasks, as in the Janus paper, with still selected on the hallucination subset only. HeadTuned-VCD also improves over VCD on the six subtasks outside the search. POPE follows the same protocol, as in the original manuscript: the setting is selected on the random split and applied unchanged to the popular and adversarial splits (Table 3).
MME score of Janus (Table 4; per subtask in Table 22):
| Hallucination subset (used for the search) | Other six perception subtasks | MME-P | |
|---|---|---|---|
| VCD | 466.67 | 520.73 | 987.40 |
| HeadTuned-VCD | 475.00 | 533.51 | 1008.51 |
Size of the gains on Janus. We repeat the comparison between VCD and HeadTuned-VCD over five runs with different seeds; HeadTuned-VCD is higher than VCD in every run on both benchmarks.
Janus over five runs with different seeds (Table 27):
| VCD | HeadTuned-VCD | Gain | Runs in which ours is higher | |
|---|---|---|---|---|
| POPE, overall accuracy (%) | 5 / 5 | |||
| MME-P | 5 / 5 |
Muddit. We reported this case as a limitation in the original manuscript (Section 5.2) and see two reasons for it. First, VCD itself barely helps Muddit: on the POPE random split its accuracy is almost unchanged (from 67.13 to 67.70, against 84.17 to 87.77 on Janus), and the number of invalid answers rises from 193 to 305 out of 3,000. There is thus little guidance signal for head tuning to refine. Second, the I2T head ranking of Muddit is unstable across samples: the mean pairwise Spearman correlation stays above 0.90 only in the first 9 of the 32 denoising steps and falls to 0.67 at the last, against 0.962 for Janus (Section 3.2, Figure 6). The heads selected by the probe are therefore unreliable for an individual sample. HeadTuned-CFG relies on a stable ranking and an effective base guidance, and neither holds for Muddit in I2T.
Decode ranking and prefill intervention on Janus. Where the tuning operator is applied depends on the model (Appendix E of the original manuscript). For Janus we arrived at this design experimentally; a possible explanation is as follows. Janus decodes autoregressively with a KV cache: the condition tokens are processed once in the prefill pass, and every decode step accesses the condition by attending to their cached keys and values. Tuning the selected heads in the prefill pass therefore changes, once and consistently, how the condition is presented to all decode steps, whereas an intervention at a decode step acts on one token at a time. The ranking is measured at the decode positions because the awareness describes how a head responds to the change of condition while the output is generated. In Muddit, every denoising step recomputes the full sequence, so the tuning is applied directly at the generation positions; in both models, the heads are selected by the awareness measured at the generation positions.
We have revised the MME evaluation in Section 5.2 (Table 4) and Appendix G.2 (Table 22) of the revised manuscript, and added the repeated runs in Appendix H.2 (Table 27); the changes are highlighted in blue.
Response to Reviewer YLKL
We thank the reviewer for the comments. We answer the questions below; the weaknesses are addressed together with the corresponding questions.
Question1 & Weakness1
HeadHunter [1] and NAG [2] each share one component with HeadTuned-CFG: HeadHunter also selects attention heads, and NAG also extrapolates between positive and negative attention features. The differences lie in what the selection is based on and in what the interpolation/extrapolation acts on.
Scope. Both methods are developed for continuous diffusion and flow models that generate images or videos. We study unified VLMs, one autoregressive and one masked-diffusion, in T2I generation and in specific I2T generation tasks.
Head-selection criterion. HeadHunter selects heads by a greedy search: it generates samples with each candidate head and scores them with an external objective such as PickScore. NAG has no selection step and is applied uniformly. We rank heads by the condition-change awareness δ, which is read from the activations of ordinary CFG generations on a probing set. It involves no search over candidate heads and no external objective, and it is an interpretable quantity of the model itself.
Interpolation/extrapolation mechanism. In HeadHunter, SoftPAG interpolates the attention map of the selected heads toward the identity; this builds a perturbed branch, and the prediction is guided away from it. NAG extrapolates the attention output under the positive prompt away from that under the negative prompt, with one strength for all heads, to restore negative prompting in few-step sampling where CFG fails. HeadTuned-CFG builds no perturbed branch and acts on the two branches of CFG: for a selected head, it moves the positive-branch activation toward the negative branch (s<1) if the head is agnostic, or away from it (s>1) if the head is aware. Different heads thus receive different guidance strengths.
New insight. The main contribution of the paper is a finding that neither method addresses: the attention heads of unified VLMs differ consistently in their response to the change of condition, in both generation directions and on two paradigms. HeadTuned-CFG is a simple inference-time intervention that exploits this property.
References