acceptodds
Under review as a conference paper at ICLR 2027

Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

Abstract

Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named HeadTuned-CFG, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.

Primary area:
generative models
Keywords:
Unified Vision-Language Model, Multimodal, Interpretability, Inference-time Intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YLKLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how attention heads in unified vision-language models respond to changes in conditioning. By measuring activation differences between the positive and negative branches of classifier-free guidance (CFG), the authors identify heterogeneous yet largely stable head-level condition-change awareness patterns in Janus and Muddit. Based on these observations, they propose HeadTuned-CFG, a training-free method that selectively interpolates or extrapolates attention-head activations to improve conditional generation. A KL-projection interpretation is provided under a local Gaussian assumption. Experiments demonstrate substantial improvements in text-to-image (T2I) generation, particularly on Muddit, and more modest gains in image-to-text (I2T) generation.

Critical strengths

1. Elegant connection between mechanistic observations and practical intervention. The paper establishes a compelling connection between head-level condition sensitivity and inference-time guidance tuning. The proposed method is simple, training-free, and directly reuses the positive and negative branches already available in CFG. The cross-sample stability of the head rankings and the targeted selection ablations provide meaningful support for this design.

2. Strong T2I performance with meaningful experimental validation. The improvements on Muddit are substantial, particularly the reduction in FID from 20.90 to 16.29, accompanied by improvements across several complementary metrics. The method also improves Janus despite its different generation paradigm. Comparisons against tuned CFG and ablations involving layer-based, random, and reversed head selection strengthen the practical contribution.

Critical weaknesses

1. The novelty relative to closely related attention-level guidance methods is insufficiently established. Prior work such as HeadHunter [1] has investigated head selection for attention-based guidance, while Normalized Attention Guidance (NAG)[2] has explored extrapolation between positive and negative attention features. The proposed condition-change-based ranking and selective interpolation strategy appear to offer meaningful distinctions, but these differences are not sufficiently articulated. In particular, the paper does not clearly establish which insights are new beyond existing head-level intervention and feature-level guidance methods. A more focused comparison would substantially strengthen the contribution.

2. The mechanistic interpretation does not yet fully explain the proposed intervention strategy. While the probing results establish stable heterogeneity in head responses, the connection between response magnitude and the usefulness of condition-specific information remains somewhat indirect. The strong performance of the layer-prior baseline also suggests that a substantial portion of the improvement may come from layer-level structure rather than precise head-level specialization. Furthermore, the Gaussian analysis explains the interpolation/extrapolation form but does not directly motivate why T2I benefits from suppressing low-response heads whereas I2T benefits from amplifying high-response heads. Clarifying this connection would make the paper's central insight more convincing.

3. The I2T results provide relatively limited support for the broader applicability of the method. The I2T improvements over VCD on Janus are modest, while the method does not improve Muddit on the same task. Moreover, the reported MME improvement is obtained after selecting hyperparameters on the evaluated benchmark itself, limiting its value as independent validation. These observations do not undermine the strong T2I results, but they leave the effectiveness of the proposed framework across both generation directions less convincing than the paper's overall framing suggests.

References

[1] D. Ahn, J. Kang, S. Lee, et al. Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] D.-Y. Chen, H. Bandyopadhyay, K. Zou, and Y.-Z. Song. Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models. Advances in Neural Information Processing Systems (NeurIPS), 2025.

Questions for authors
  1. Could the authors clarify the key conceptual and technical differences from HeadHunter and NAG, particularly regarding the head-selection criterion and the interpolation/extrapolation mechanism?

  2. What motivates the asymmetric intervention strategies across T2I and I2T? Specifically, why should low condition-change awareness indicate responses worth suppressing in T2I, while high awareness indicates responses worth amplifying in I2T? How much of this behavior is attributable to head-level specialization beyond layer-level structure?

  3. Could the authors clarify the interpretation of the I2T results, particularly the lack of improvement on Muddit and the evaluation implications of selecting MME hyperparameters on the reported benchmark? Also, for Janus, what motivates using head rankings obtained during decoding to guide interventions applied only during prefill?

5 comments

Response to Reviewer YLKL

We thank the reviewer for the comments. We answer the questions below; the weaknesses are addressed together with the corresponding questions.

Question1 & Weakness1

HeadHunter [1] and NAG [2] each share one component with HeadTuned-CFG: HeadHunter also selects attention heads, and NAG also extrapolates between positive and negative attention features. The differences lie in what the selection is based on and in what the interpolation/extrapolation acts on.

Scope. Both methods are developed for continuous diffusion and flow models that generate images or videos. We study unified VLMs, one autoregressive and one masked-diffusion, in T2I generation and in specific I2T generation tasks.

Head-selection criterion. HeadHunter selects heads by a greedy search: it generates samples with each candidate head and scores them with an external objective such as PickScore. NAG has no selection step and is applied uniformly. We rank heads by the condition-change awareness δ\delta, which is read from the activations of ordinary CFG generations on a probing set. It involves no search over candidate heads and no external objective, and it is an interpretable quantity of the model itself.

Interpolation/extrapolation mechanism. In HeadHunter, SoftPAG interpolates the attention map of the selected heads toward the identity; this builds a perturbed branch, and the prediction is guided away from it. NAG extrapolates the attention output under the positive prompt away from that under the negative prompt, with one strength for all heads, to restore negative prompting in few-step sampling where CFG fails. HeadTuned-CFG builds no perturbed branch and acts on the two branches of CFG: for a selected head, it moves the positive-branch activation toward the negative branch (s<1s < 1) if the head is agnostic, or away from it (s>1s > 1) if the head is aware. Different heads thus receive different guidance strengths.

New insight. The main contribution of the paper is a finding that neither method addresses: the attention heads of unified VLMs differ consistently in their response to the change of condition, in both generation directions and on two paradigms. HeadTuned-CFG is a simple inference-time intervention that exploits this property.

References

  • [1] Ahn et al. Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models. arXiv:2506.10978.
  • [2] Chen et al. Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models. arXiv:2505.21179.

Question2 & Weakness2

Why interpolation in T2I and extrapolation in I2T. A unified VLM serves both generation and understanding, and the two tasks ask different things of the guidance. The direction of tuning was chosen empirically for each task; our interpretation is as follows.

T2I is a generation task. The output is a whole image, and its quality relies on the generative ability of the pretrained model, which the tuning should preserve. Interpolation keeps the activation of a head between the two branches of CFG, so it does not move the activations away from what the model itself produces. We apply it only to the agnostic heads, whose difference between the two branches carries little condition information. The guidance is thus weakened only where it mostly amplifies noise, and the generation quality of the pretrained model is maintained.

I2T, in the tasks we study, is an understanding task. The output is a short answer to a question about the image (yes or no in POPE and MME) rather than a long sequence, so what matters is not the quality of a long generation but whether the answer follows the semantic content of the image. The answer therefore needs stronger guidance from the image, and we extrapolate the aware heads, which respond most strongly to it.

Head level beyond layer level. Layer-level structure accounts for part of the gain, and selecting individual heads by δ\delta adds to it consistently. We conducted additional experiments that compare the δ\delta-ranked selection with the layer prior on all ten MJHQ categories, and over ten runs with different seeds on one category (Section 5.3, Appendix H). Note that the late-layer prior entry of the submitted Muddit ablation was corrected from 30.71 to 31.97 (General Response, point 1).

FID on each MJHQ category and on MJHQ-full (Tables 6 and 24):

animalsartfashionfoodindoorlandscapelogopeopleplantsvehiclesfull
Muddit, late-layer prior31.9737.2334.4534.0636.1035.1636.9837.6144.1531.2416.73
Muddit, δ\delta-ranked (ours)30.4136.7034.0233.6036.1734.5636.6536.8843.0130.9316.29
Janus, early-layer prior26.5530.3729.4131.0127.2825.0339.2433.3230.2923.1910.15
Janus, δ\delta-ranked (ours)25.9529.9328.8331.3727.0324.8137.6032.0229.3723.169.92

FID on MJHQ-art over ten runs with different seeds (Tables 25 and 26):

Layer priorδ\delta-ranked (ours)DifferenceRuns in which ours is lower
Muddit37.38±0.1337.38 \pm 0.1336.87±0.28\mathbf{36.87} \pm 0.28−0.51±0.32-0.51 \pm 0.3210 / 10
Janus30.28±0.1430.28 \pm 0.1430.17±0.20\mathbf{30.17} \pm 0.20−0.11±0.19-0.11 \pm 0.198 / 10

We would also like to note that the correct direction of the layer prior is itself determined by the probe: in the opposite direction, the layer prior is worse than vanilla CFG on both models.

FID on MJHQ-animals (Tables 5 and 23):

CFGLayer prior, probed directionLayer prior, opposite directionδ\delta-ranked (ours)
Muddit33.9131.9738.8430.41
Janus27.3226.5527.6325.95

We have added the head-selection results to the revised manuscript, in Section 5.3 (Tables 5 and 6) and Appendix H (Tables 23–26), highlighted in blue.

Question3 & Weakness3

MME hyperparameters. We agree that the originally submitted MME result was not an independent validation. We now report MME-P, the ten perception subtasks, as in the Janus paper, with (w,k,s)(w, k, s) still selected on the hallucination subset only. HeadTuned-VCD also improves over VCD on the six subtasks outside the search. POPE follows the same protocol, as in the original manuscript: the setting is selected on the random split and applied unchanged to the popular and adversarial splits (Table 3).

MME score of Janus (Table 4; per subtask in Table 22):

Hallucination subset (used for the search)Other six perception subtasksMME-P
VCD466.67520.73987.40
HeadTuned-VCD475.00533.511008.51

Size of the gains on Janus. We repeat the comparison between VCD and HeadTuned-VCD over five runs with different seeds; HeadTuned-VCD is higher than VCD in every run on both benchmarks.

Janus over five runs with different seeds (Table 27):

VCDHeadTuned-VCDGainRuns in which ours is higher
POPE, overall accuracy (%)85.86±0.0785.86 \pm 0.0786.43±0.07\mathbf{86.43} \pm 0.07+0.57±0.09+0.57 \pm 0.095 / 5
MME-P988.04±1.60988.04 \pm 1.601011.50±3.09\mathbf{1011.50} \pm 3.09+23.46±3.72+23.46 \pm 3.725 / 5

Muddit. We reported this case as a limitation in the original manuscript (Section 5.2) and see two reasons for it. First, VCD itself barely helps Muddit: on the POPE random split its accuracy is almost unchanged (from 67.13 to 67.70, against 84.17 to 87.77 on Janus), and the number of invalid answers rises from 193 to 305 out of 3,000. There is thus little guidance signal for head tuning to refine. Second, the I2T head ranking of Muddit is unstable across samples: the mean pairwise Spearman correlation stays above 0.90 only in the first 9 of the 32 denoising steps and falls to 0.67 at the last, against 0.962 for Janus (Section 3.2, Figure 6). The heads selected by the probe are therefore unreliable for an individual sample. HeadTuned-CFG relies on a stable ranking and an effective base guidance, and neither holds for Muddit in I2T.

Decode ranking and prefill intervention on Janus. Where the tuning operator is applied depends on the model (Appendix E of the original manuscript). For Janus we arrived at this design experimentally; a possible explanation is as follows. Janus decodes autoregressively with a KV cache: the condition tokens are processed once in the prefill pass, and every decode step accesses the condition by attending to their cached keys and values. Tuning the selected heads in the prefill pass therefore changes, once and consistently, how the condition is presented to all decode steps, whereas an intervention at a decode step acts on one token at a time. The ranking is measured at the decode positions because the awareness describes how a head responds to the change of condition while the output is generated. In Muddit, every denoising step recomputes the full sequence, so the tuning is applied directly at the generation positions; in both models, the heads are selected by the awareness measured at the generation positions.

We have revised the MME evaluation in Section 5.2 (Table 4) and Appendix G.2 (Table 22) of the revised manuscript, and added the repeated runs in Appendix H.2 (Table 27); the changes are highlighted in blue.

Reviewer_jzKVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a shared vision-language backbone responds to conditioning in text-to-image and image-to-text generation. It compares attention-head activations between positive and negative guidance branches, then uses the resulting ranking to adjust guidance at selected heads. This provides insights for understanding how different attention heads use conditioning information and whether these differences can guide improvements in generation without additional training.

Critical strengths
  • Studying both generation directions within unified models is a worthwhile way to examine how a shared backbone uses different conditions. The proposed head-level intervention is also a reasonable design to treat functionally distinct components with different guidance scales. The method requires no additional training.
  • The main experimental results support the usefulness of the proposed head-level intervention, with the MJHQ-full showing substantial FID improvements from 20.90 to 16.29 on Muddit.
Critical weaknesses
  • The probe identifies heads that respond differently to conditioning, but does not establish that these differences reflect semantic specialization. It measures the absolute activation difference between the positive and negative branches without normalising for each head’s activation magnitude. Heads with naturally larger activations could therefore receive higher scores even if their relative sensitivity to conditioning is modest. If activation scales also vary systematically across layers, a stable ranking across prompts could reflect network depth rather than distinct semantic roles. The layer-prior control makes this concern consequential: on MJHQ-animals, selecting heads by layer alone achieves 30.71 FID, compared with 30.41 for the proposed probe. Without uncertainty estimates, it is unclear whether this difference demonstrates a meaningful advantage from identifying individual heads.
  • The paper should articulate more clearly the novelty about using the probe to select and adjust attention heads. Head-level interventions and guidance adjustment are already established approaches, so the contribution depends on whether the proposed statistic provides a better basis for deciding where and how to intervene. The local Gaussian analysis put forward in the paper does not provide guidance for how strongly a selected head should be adjusted.
Questions for authors
  • What is the main technical advance over existing head-level intervention and guidance methods? Does it lie in how heads are selected, how their activations are modified, or the combination of the two?
  • Are the gains over layer-based head selection, and the smaller improvements on Janus, MME and POPE, consistent across repeated runs? Please report variability or confidence intervals to show whether these gains exceed evaluation noise.
5 comments

Response to Reviewer jzKV

We thank the reviewer for the comments. We answer the questions below; the weaknesses are addressed together with the corresponding questions.

Question1

The main advance lies in how the heads are selected and in the finding behind this selection; the modification of the activations is deliberately simple. Among existing methods, HeadHunter [1] is the closest to ours, since it also intervenes on selected attention heads, so we compare with it directly.

Scope. HeadHunter is developed for DiT-based text-to-image diffusion and flow models. We study unified VLMs, one autoregressive and one masked-diffusion, in T2I generation and in specific I2T generation tasks.

Head selection. HeadHunter selects heads by a greedy search: it generates samples with each candidate head and scores them with an external objective such as PickScore. We rank heads by the condition-change awareness δ\delta, which is read from the activations of ordinary CFG generations on a probing set. It involves no search over candidate heads and no external objective, and it is an interpretable quantity of the model itself: the attention heads of unified VLMs differ consistently in this response, in both generation directions and on two paradigms. This finding is the main contribution of the paper.

Activation modification. HeadHunter replaces the attention maps of the selected heads with the identity, or interpolates toward it (SoftPAG), to build a perturbed branch, and guides the prediction away from that branch. HeadTuned-CFG builds no perturbed branch: it keeps the two branches of CFG and rescales their difference head by head, so that different heads receive different guidance strengths.

We discussed the relation to perturbation guidance methods, to which HeadHunter belongs, in the original manuscript (Section 2.4).

References

  • [1] Ahn et al. Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models. arXiv:2506.10978.

Question2 & Weakness1

We conducted additional experiments: the gain over layer-based selection is evaluated on all ten MJHQ categories and over repeated runs with different seeds, the comparison with VCD on POPE and MME is repeated over five runs, and a normalized score is added to the ablation. Note that the value 30.71 mentioned in W1 is one of the two entries corrected in the revision (General Response, point 1); it is 31.97 in the revised Table 5.

Gain over layer-based selection across MJHQ categories. The δ\delta-ranked selection has a lower FID than the layer prior in 9 of 10 categories on both models, and on MJHQ-full.

FID on each MJHQ category and on MJHQ-full (Tables 6 and 24):

animalsartfashionfoodindoorlandscapelogopeopleplantsvehiclesfull
Muddit, late-layer prior31.9737.2334.4534.0636.1035.1636.9837.6144.1531.2416.73
Muddit, δ\delta-ranked (ours)30.4136.7034.0233.6036.1734.5636.6536.8843.0130.9316.29
Janus, early-layer prior26.5530.3729.4131.0127.2825.0339.2433.3230.2923.1910.15
Janus, δ\delta-ranked (ours)25.9529.9328.8331.3727.0324.8137.6032.0229.3723.169.92

Gain over layer-based selection across seeds. Over ten runs with different seeds on MJHQ-art, the δ\delta-ranked selection is lower than the layer prior in all ten runs on Muddit and in eight on Janus.

FID on MJHQ-art over ten runs with different seeds (Tables 25 and 26):

Layer priorδ\delta-ranked (ours)DifferenceRuns in which ours is lower
Muddit37.38±0.1337.38 \pm 0.1336.87±0.28\mathbf{36.87} \pm 0.28−0.51±0.32-0.51 \pm 0.3210 / 10
Janus30.28±0.1430.28 \pm 0.1430.17±0.20\mathbf{30.17} \pm 0.20−0.11±0.19-0.11 \pm 0.198 / 10

Gains on POPE and MME across seeds. HeadTuned-VCD is higher than VCD in all five runs on both benchmarks. Note that the MME evaluation has been revised: we now report MME-P, with the hyperparameters still selected on the hallucination subset only (General Response, point 3).

Janus over five runs with different seeds (Table 27):

VCDHeadTuned-VCDGainRuns in which ours is higher
POPE, overall accuracy (%)85.86±0.0785.86 \pm 0.0786.43±0.07\mathbf{86.43} \pm 0.07+0.57±0.09+0.57 \pm 0.095 / 5
MME-P988.04±1.60988.04 \pm 1.601011.50±3.09\mathbf{1011.50} \pm 3.09+23.46±3.72+23.46 \pm 3.725 / 5

Normalization by the activation magnitude. We agree with the concern that δ\delta measures the absolute activation difference without normalizing for the activation magnitude of each head. We therefore added the normalized score δ~=δ/∥hpos∥\tilde{\delta} = \delta / \lVert \mathbf{h}_{\mathrm{pos}} \rVert as a selection rule in the ablation: heads are ranked by δ~\tilde{\delta} instead of δ\delta and tuned with the same kk and ss (Section 5.3; Tables 5, 6, 23, and 24).

FID on MJHQ-full (Tables 6 and 24):

CFGLayer priorδ~\tilde{\delta}-rankedδ\delta-ranked (ours)
Muddit20.9016.7316.4816.29
Janus10.5110.159.899.92

The two rules give similar results, and both are better than the layer prior, which selects heads by depth alone. Removing the activation scale from the score therefore does not remove the gain, which indicates that the differences between heads captured by δ\delta reflect their response to the condition rather than differences in activation scale. Since δ\delta is slightly better on Muddit and on par on Janus, we keep it as the selection rule.

We have added these results to the revised manuscript, in Section 5.3 (Tables 5 and 6) and Appendix H (Tables 23–27), and revised the MME evaluation in Section 5.2 (Table 4); the changes are highlighted in blue.

Weakness2

We agree that the local Gaussian analysis explains why the tuning operator takes the form of an interpolation or extrapolation between the two branches, but does not prescribe how strongly each selected head should be adjusted. In practice, kk and ss are selected by grid search (Appendix G). Setting the head-wise strength from the probed awareness is left to future work, as noted in the original manuscript (Section 6).

Reviewer_LVYfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how attention heads in unified vision-language models respond to changes in conditioning information during multimodal generation. The authors introduce a probing method that measures differences in attention-head activations between the positive and negative branches of classifier-free guidance (CFG). Through experiments on Janus and Muddit, they find that attention heads vary in how strongly they respond to changes in conditioning information. They also observe that the relative rankings of these heads remain fairly consistent across different input samples in several experimental settings. Based on these findings, the authors propose HeadTuned-CFG, an inference-time method that selectively adjusts attention-head activations according to their sensitivity to conditioning changes. They evaluate the method on both text-to-image (T2I) and image-to-text (I2T) generation tasks. The results show improvements over standard CFG for T2I generation, while the gains on I2T hallucination benchmarks are more limited.

Critical strengths
  1. Interesting connection between attention analysis and inference-time guidance. One strength of this paper is that it goes beyond analyzing attention heads and uses the findings to develop a practical generation method. HeadTuned-CFG adjusts guidance at the attention-head level without requiring additional model training. I find this connection between understanding how attention heads respond to conditioning changes and using that information to improve generation particularly interesting. The authors also test their approach on both autoregressive and masked-diffusion models, which suggests that the method could be useful across different unified VLM architectures.
  2. Promising T2I results supported by ablation studies. The paper reports clear improvements over standard CFG in text-to-image generation. For example, on MJHQ-full, HeadTuned-CFG reduces FID from 20.90 to 16.29 for Muddit and from 10.51 to 9.92 for Janus. These results suggest that the proposed method can improve generation quality, although the size of the improvement varies across models. I also appreciate the ablation studies on head selection. The authors compare their probing-based approach with random, reversed, and layer-based selection strategies. The results suggest that identifying heads based on their sensitivity to conditioning changes is more effective than selecting heads arbitrarily.
Critical weaknesses
  1. The probing metric does not fully demonstrate functional specialization of attention heads. My main concern is whether the proposed condition-change awareness metric truly captures how much task-relevant conditioning information each attention head represents. The metric is based on the magnitude of activation differences between the positive and negative CFG branches, but a larger difference does not necessarily mean that a head plays a more important role in processing conditioning information. For example, differences in activation magnitude or representation scaling could also affect the head rankings. While the attention visualizations and head-selection experiments provide some supporting evidence, they do not fully establish a causal relationship between the measured awareness and the functional role of each head. Since functional specialization is one of the main motivations behind the proposed method, I think additional experiments or controls would help make this claim more convincing.
  2. It is unclear how much the method improves over simpler alternatives or how well it generalizes across tasks. Although the T2I results are promising, some of the ablation results suggest that the proposed head-selection strategy may not always offer a substantial advantage over simpler approaches. For example, in the Muddit experiments on MJHQ-animals, the probing-based selection achieves an FID of 30.41, while selecting heads from later layers gives an FID of 30.71. Both approaches perform better than random or reversed selection, but the difference between them is relatively small. This makes it difficult to determine how much of the improvement comes specifically from the proposed awareness-based ranking. The results for image-to-text generation are also less convincing. On POPE, HeadTuned-VCD improves overall accuracy from 85.89% to 86.45% compared with VCD, which is a fairly modest gain. The authors also report that the method does not improve I2T performance on Muddit. These findings suggest that the method's effectiveness may depend on the task or model architecture. I would like to see stronger evidence that awareness-based head selection consistently outperforms simpler strategies and provides meaningful improvements across both generation directions.
Questions for authors
  1. Head selection: In the Muddit ablation study, the proposed probing-based method performs only slightly better than the simpler late-layer selection strategy. Is this improvement consistent across different MJHQ categories and multiple generation seeds?
  2. Mechanistic interpretation: How do the authors determine whether the observed differences between attention heads reflect genuine specialization in processing conditioning information, rather than differences in activation scale? For example, would normalizing the activation differences or conducting controlled causal interventions produce similar head rankings and performance improvements?
  3. Generalization and efficiency: Could the authors provide more details on the computational overhead of the probing process and hyperparameter tuning? Also, why does HeadTuned-VCD fail to improve I2T performance on Muddit?
6 comments

Response to Reviewer LVYf

We thank the reviewer for the comments. We answer the questions below; the weaknesses are addressed together with the corresponding questions.

Question1 & Weakness2

Probe-based selection vs. layer prior (T2I). We agree that the ablation in the submitted version was too limited in scale to show that the proposed head selection is better than the simpler layer prior. We conducted additional experiments that compare the two on all ten MJHQ categories, and over ten runs with different seeds on one category (Section 5.3, Appendix H). Note that the value 30.71 quoted in the review is one of the two entries corrected in the revision (General Response, point 1); it is 31.97 in the revised Table 5.

FID on each MJHQ category and on MJHQ-full (Tables 6 and 24):

animalsartfashionfoodindoorlandscapelogopeopleplantsvehiclesfull
Muddit, late-layer prior31.9737.2334.4534.0636.1035.1636.9837.6144.1531.2416.73
Muddit, δ\delta-ranked (ours)30.4136.7034.0233.6036.1734.5636.6536.8843.0130.9316.29
Janus, early-layer prior26.5530.3729.4131.0127.2825.0339.2433.3230.2923.1910.15
Janus, δ\delta-ranked (ours)25.9529.9328.8331.3727.0324.8137.6032.0229.3723.169.92

FID on MJHQ-art over ten runs with different seeds (Tables 25 and 26):

Layer priorδ\delta-ranked (ours)DifferenceRuns in which ours is lower
Muddit37.38±0.1337.38 \pm 0.1336.87±0.28\mathbf{36.87} \pm 0.28−0.51±0.32-0.51 \pm 0.3210 / 10
Janus30.28±0.1430.28 \pm 0.1430.17±0.20\mathbf{30.17} \pm 0.20−0.11±0.19-0.11 \pm 0.198 / 10

We would also like to note that the correct direction of the layer prior is itself determined by the probe: in the opposite direction, the layer prior is worse than vanilla CFG on both models.

FID on MJHQ-animals (Tables 5 and 23):

CFGLayer prior, probed directionLayer prior, opposite directionδ\delta-ranked (ours)
Muddit33.9131.9738.8430.41
Janus27.3226.5527.6325.95

In summary, the improvement over the layer prior is consistent across MJHQ categories and across generation seeds.

Gains on POPE and MME (I2T). We repeat the comparison between VCD and HeadTuned-VCD over five runs with different seeds; HeadTuned-VCD is higher than VCD in every run on both benchmarks. Note that the MME evaluation has been revised: we now report MME-P, with the hyperparameters still selected on the hallucination subset only (General Response, point 3).

Janus over five runs with different seeds (Table 27):

VCDHeadTuned-VCDGainRuns in which ours is higher
POPE, overall accuracy (%)85.86±0.0785.86 \pm 0.0786.43±0.07\mathbf{86.43} \pm 0.07+0.57±0.09+0.57 \pm 0.095 / 5
MME-P988.04±1.60988.04 \pm 1.601011.50±3.09\mathbf{1011.50} \pm 3.09+23.46±3.72+23.46 \pm 3.725 / 5

We have added these results to the revised manuscript, in Section 5.3 (Tables 5 and 6) and Appendix H (Tables 23–27), and revised the MME evaluation in Section 5.2 (Table 4); the changes are highlighted in blue.

Question2 & Weakness1

We added the normalized score δ~=δ/∥hpos∥\tilde{\delta} = \delta / \lVert \mathbf{h}_{\mathrm{pos}} \rVert as a head-selection rule in the ablation: heads are ranked by δ~\tilde{\delta} instead of δ\delta and tuned with the same kk and ss. We compare it with the δ\delta-ranked selection on all ten MJHQ categories (Section 5.3; Tables 5, 6, 23, and 24).

FID on each MJHQ category and on MJHQ-full (Tables 6 and 24):

animalsartfashionfoodindoorlandscapelogopeopleplantsvehiclesfull
Muddit, CFG33.9142.8238.7939.5343.8240.9340.2743.8549.9336.8920.90
Muddit, δ~\tilde{\delta}-ranked31.2937.2033.7533.2536.2535.7936.7937.0244.1030.4116.48
Muddit, δ\delta-ranked (ours)30.4136.7034.0233.6036.1734.5636.6536.8843.0130.9316.29
Janus, CFG27.3231.3330.1031.0528.0627.5337.9033.9730.2023.6610.51
Janus, δ~\tilde{\delta}-ranked26.4530.5728.2831.2226.1923.8037.5632.9129.6822.779.89
Janus, δ\delta-ranked (ours)25.9529.9328.8331.3727.0324.8137.6032.0229.3723.169.92

The two rules give similar results: tuning the δ~\tilde{\delta}-ranked heads lowers the FID of vanilla CFG by about the same margin as tuning the δ\delta-ranked heads, on both models. Removing the activation scale from the score therefore does not remove the gain, which indicates that the differences between heads captured by δ\delta reflect their response to the condition rather than differences in activation scale. Since δ\delta is slightly better on Muddit and on par on Janus, we keep it as the selection rule.

We have added the normalized selection rule and its results to the revised manuscript, in Section 5.3 (Tables 5 and 6) and Appendix H.1 (Tables 23 and 24), highlighted in blue.

Question3

Probing and inference. Probing is a one-time cost of ∣P∣|\mathcal{P}| ordinary CFG generations: it only reads activations that are already computed, and the hooks add less than 10−410^{-4} of the FLOPs. In our setting, this is about 2–3 GPU-hours per probe on Janus and under 1 GPU-hour on Muddit. At inference, HeadTuned-CFG adds no forward pass: the head-wise scaling adds at most about 10−510^{-5} of the FLOPs of CFG, whereas PAG+CFG needs a third branch.

Hyperparameter tuning. (w,k,s)(w, k, s) are selected by grid search, on MJHQ-animals for T2I and on the POPE random split and the MME hallucination subset for I2T. We provided the full search in the original manuscript (Appendix G; Tables 10–21 in the revised version). A more efficient way of setting the hyperparameters is left to future work (Section 6).

HeadTuned-VCD on Muddit. We reported this case as a limitation in the original manuscript (Section 5.2) and see two reasons for it. First, VCD itself barely helps Muddit: on the POPE random split its accuracy is almost unchanged (from 67.13 to 67.70, against 84.17 to 87.77 on Janus), and the number of invalid answers rises from 193 to 305 out of 3,000. There is thus little guidance signal for head tuning to refine. Second, the I2T head ranking of Muddit is unstable across samples: the mean pairwise Spearman correlation stays above 0.90 only in the first 9 of the 32 denoising steps and falls to 0.67 at the last, against 0.962 for Janus (Section 3.2, Figure 6). The heads selected by the probe are therefore unreliable for an individual sample. HeadTuned-CFG relies on a stable ranking and an effective base guidance, and neither holds for Muddit in I2T.

We have added the cost analysis to the revised manuscript, in Section 5.1 and Appendix I (Table 28), highlighted in blue.

Comments on the paper

1 comment

General Response

We thank all reviewers for their comments. We have uploaded a revised paper, in which the changes are marked in blue; the new material is in Sections 5.1–5.3 and Appendices G.2, H, and I. Here we summarize the changes that address shared concerns.

1. Expand head-selection ablation. The submitted ablation used MJHQ-animals and a single seed, which limits the confidence in its conclusion. We conducted additional experiments:

  • all ten MJHQ categories, for the δ\delta-ranked selection and the layer prior (Muddit: Section 5.3, Table 6; Janus: Appendix H.1, Table 24);
  • repeated runs with different seeds, ten on MJHQ-art for the same comparison (Appendix H.2, Tables 25 and 26);
  • more head sets, namely the layer prior in the opposite direction and the δ~\tilde{\delta}-ranked selection of point 2 (Muddit: Section 5.3, Table 5; Janus: Appendix H.1, Table 23).

Correction. Two entries of the submitted Muddit ablation on MJHQ-animals, the late-layer prior (30.71) and the random set (39.31), had been generated on a different GPU type and software environment from all other results. We re-ran both under the same setup as the other experiments and revised them with the corrected values, 31.97 and 39.40 (Table 5). All results in the revised paper are obtained in the same environment.

2. Normalized delta added as a selection rule. We added the normalized score δ~=δ/∥hpos∥\tilde{\delta} = \delta / \lVert \mathbf{h}_{\mathrm{pos}} \rVert as a selection rule in the ablation: heads are ranked by δ~\tilde{\delta} instead of δ\delta and tuned with the same kk and ss, on all ten MJHQ categories (Muddit: Section 5.3, Tables 5 and 6; Janus: Appendix H.1, Tables 23 and 24).

3. Adjusted I2T evaluation. The submitted version reported the four hallucination subtasks of MME, on which (w,k,s)(w, k, s) had been selected. We now report MME-P, the ten perception subtasks, as in the Janus paper; the setting is still selected on the hallucination subset and applied unchanged to the other six subtasks (Section 5.2, Table 4; per subtask in Appendix G.2, Table 22). We also compare VCD and HeadTuned-VCD on POPE and MME-P over repeated runs with different seeds (Appendix H.2, Table 27).

4. Relation to existing head-level and attention-level guidance. The main contribution of the paper is a finding: the attention heads of unified VLMs differ consistently in their response to the change of condition, in both generation directions and on two paradigms. HeadTuned-CFG is a simple inference-time intervention that exploits this property. It differs from PAG, SEG, HeadHunter [1], and NAG [2] in three respects.

Scope. These methods are developed for continuous diffusion and flow models that generate images or videos. We study unified VLMs, one autoregressive and one masked-diffusion, in T2I generation and in specific I2T generation tasks.

Where to intervene. PAG and SEG act on all heads of a few selected layers, and NAG applies its extrapolation uniformly, without a selection step. HeadHunter selects individual heads by a greedy search: it generates samples with each candidate head and scores them with an external objective. We rank heads by δ\delta, which is read from the activations of ordinary CFG generations on a probing set; it involves no search over candidate heads and no external objective, and it is an interpretable quantity of the model itself.

How to intervene. PAG, SEG, and HeadHunter build a perturbed branch and guide the prediction away from it. NAG extrapolates the attention output under the positive prompt away from that under the negative prompt. HeadTuned-CFG keeps the two branches of CFG and rescales their difference head by head: toward the negative branch for agnostic heads, away from it for aware heads.

References

  • [1] Ahn et al. Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models. arXiv:2506.10978.
  • [2] Chen et al. Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models. arXiv:2505.21179.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.