acceptodds
Under review as a conference paper at ICLR 2027

Commutator-Guided Calibration of Attention-Head Interactions in Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) aggregate attention-head outputs additively without explicitly composing their state-dependent residual operators. We diagnose their non-commutativity (NC), or dependence on composition order, using routing-matrix commutators and changes in predicted clean latents under reversed head-operator composition. Their calibrated comparison defines weighted undershoot graphs: Laplacian traces summarize layer-level mismatch, while edge weights identify pairs with weak functional responses relative to their routing-level reference. We use a two-phase procedure: Phase 1 probes a frozen pretrained model to select compact, layer-specific head-pair sets without generation metrics; Phase 2 fine-tunes the model with a one-sided hinge objective applied to these pairs. Across ImageNet and CUB-200-2011, the results suggest that intervention efficacy reflects both calibrated undershoot and layer position: high diagnostic energy alone does not guarantee larger generation gains. Across continuation seeds, simultaneous NC regularization at layers 10 and 24 achieves ImageNet validation FID , compared with for pretrained DiT-XL/2, and improves Inception Score, and precision over matched denoising-only (no-NC) fine-tuning. On CUB-200-2011, NC achieves FID , compared with for the domain-adapted baseline, with higher precision. No-NC controls support gains beyond continued denoising training, while pair-budget ablations favour compact interventions over broader regularization. These results support efficacy of selective regularization of attention-head interactions while retaining the additive DiT architecture.

Primary area:
generative models
Keywords:
Diffusion transformers, attention-head interactions, non-commutativity, graph Laplacians, layer-wise calibration, generative modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2X68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a non-commutativity diagnostic to the analysis of DiT attention heads, proposing an additive aggregation of heads while noting that the state-dependent residual operators do not commute; experiments are conducted using DiT-XL/2 on ImageNet 256×256, with transfer to CUB-200-2011.

Critical strengths
  1. Introducing non-commutative/operator-order analysis into DiT attention heads is a novel approach.
  2. The design is ingenious: Phase 1 diagnoses and selects head pairs, while Phase 2 applies regularization only to the selected pairs.
  3. The FID improvements at Layer 24 and the joint 10+24 configuration are significant and convincing.
Critical weaknesses
  1. The core diagnosis relies on the sequential application of single-head updates, whereas standard DiT models aggregate outputs via addition after computing all heads in parallel. Does the identified sensitivity to operation order actually reflect non-commutative effects present in truly parallel models?
  2. There is scant discussion regarding the selection of key hyperparameters, potential overfitting, or result sensitivity, eg. concerning the layer-specific scale in Phase 1, the pair scale in Phase 2, and the techniques detailed in Appendix A.6 (such as online calibration, EMA, log-space centering, and clipping).
  3. For Phase 1 graph energy, Layers 3, 10, and 24 were selected, but the NC effect at Layer 3 was nearly identical to the "no-NC" baseline, demonstrating that high graph energy does not guarantee effective intervention. Nevertheless, Layer 3 was still chosen as an intervention layer, resulting in wasted computation and undermining the credibility of the selection criteria.
  4. Table 2 reveals a significant disparity in FID scores between the "no-NC" and NC settings, it is unclear whether the "no-NC" setting also underwent the same partial fine-tuning process.
  5. The scope of the experiments and the assessment of generalizability are too limited.
Questions for authors

see weakness

Reviewer_sEEvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies non-commutative interactions between attention heads in diffusion transformers. It compares routing-matrix commutators with functional order sensitivity measured by reversing hypothetical sequential head compositions. A calibrated undershoot score selects layers and head pairs for subsequent fine-tuning with a one-sided hinge regularizer, while preserving the additive architecture. Experiments on ImageNet and CUB show that selected interventions improve FID and precision over matched denoising-only continuation. Additional analyses examine intervention location, pair budgets, diagnostic robustness, and comparisons with unconditional commutator shrinkage.

Critical strengths
  1. The paper provides complementary diagnostics of attention-head interactions by distinguishing routing non-commutativity from changes in predicted clean latents. The resulting selective regularization procedure preserves the original DiT architecture and inference process.

  2. Matched denoising-only controls and repeated continuation runs support improvements beyond ordinary fine-tuning. Results on ImageNet and CUB demonstrate gains in FID and precision, while analyses across timestep windows and probe batches provide evidence of diagnostic stability.

Critical weaknesses
  1. Calibrated undershoot is insufficiently justified as an intervention criterion. Weak output sensitivity to head order may reflect normal processing by subsequent layers. The paper demonstrates generation improvements, but provides limited evidence that falling below the calibrated reference identifies interactions that particularly benefit from regularization.

  2. In the fixed-controller comparison, plain shrinkage achieves comparable FID and higher recall, while the calibrated hinge improves IS and precision (Table 8). This supports a trade-off between metrics, but provides limited evidence that recomputing functional responses during training offers sufficient additional value over the simpler regularizer.

Questions for authors
  1. Why should functional order sensitivity below the calibrated reference indicate an interaction requiring regularization, rather than normal attenuation by subsequent layers?

  2. How does the evidence distinguish the benefit of undershoot-based pair selection from the benefit of regularizing a small number of pairs?

Reviewer_F14gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates attention-head non-commutativity in Diffusion Transformers through routing-matrix commutators and changes in predicted clean latents under reversed head compositions. It uses these diagnostics to select head pairs and fine-tunes selected blocks with a calibrated hinge regularizer, while retaining the original additive attention architecture. Experiments on ImageNet and CUB-200-2011 show improvements in FID and precision over matched denoising-only continuation, accompanied by lower recall. Additional analyses examine where to intervene, how many pairs to regularize, and the stability of the diagnostic across batches and timestep windows.

Critical strengths
  • The matched denoising-only controls support gains beyond continued training. The later-layer and joint interventions improve ImageNet FID, IS, and precision, and the CUB experiments show improvements in FID and precision after domain adaptation. Reporting variation across continuation seeds and the accompanying recall reductions gives a balanced account of the results.
  • The paper examines the diagnostic beyond the main generation results. Pair rankings are stable across probe batches and routing-aggregation choices, and Layer 24 pair selection remains unchanged across the tested timestep windows. The layer comparisons also show that high undershoot alone does not guarantee a useful intervention, appropriately qualifying the diagnostic’s role.
Critical weaknesses
  • The additional benefit of functional calibration over plain commutator shrinkage is not yet clear. The matched comparison in Table 8 is useful: the hinge gives higher IS and precision, while shrinkage gives slightly lower mean FID and higher recall. The paper acknowledges this trade-off, but the evidence for the calibrated objective is consequently less conclusive than the evidence for commutator regularization itself.
  • The pair-budget ablation does not establish whether calibrated undershoot selects better pairs than simpler alternatives. There is no equal-budget comparison with random selection or selection by routing commutator magnitude alone. Also, because the objective sums pairwise penalties, increasing the pair count changes the overall regularization strength. This leaves some ambiguity in attributing the poorer results with larger sets to which pairs were added.
  • Some calibration details remain unclear despite the implementation appendix. The paper mentions a percentile-based layer scale and optional online pair-scale updates, but does not give an explicit rule for constructing the layer scale or clearly identify which updates were enabled in the main runs. These choices affect the hinge threshold and are needed to reproduce the method.
Questions for authors
  • Table 8 shows different trade-offs for the hinge and plain shrinkage under fixed controllers. How representative is this comparison of the main training configuration, and is there evidence that the response-based threshold contributes to the main results?
  • Is there an equal-budget comparison with randomly selected pairs or pairs ranked by routing commutator magnitude? For the pair-budget ablation, controlling the total regularization strength would also help separate the effect of pair selection from that of a larger penalty.
  • Please specify how the layer calibration scale is computed and which online calibration updates are enabled for the main results. A clear distinction between these settings and the fixed-controller experiment would resolve the ambiguity.

Comments on the paper

3 comments

Dear Area Chair and Reviewers,

We would like to draw your attention to a point of continuity in the discussion. Reviewer 2X68’s Comment 2 raises a concern that we addressed in our earlier response to Reviewer sEEv, which covers the magnitude-ranking comparison, expanded-pool random selection, and the distinction between pair selection and the hinge–shrinkage comparison.

To ensure this prior response is fully considered, we have reiterated the relevant findings in our direct reply to Reviewer 2X68. We respectfully ask the Area Chair to review the linked exchange alongside our latest reply and to draw the other reviewers’ attention to this evidence. If any aspect of the concern remains unresolved after these responses and experiments, we would welcome clarification of the specific point still at issue, so that the discussion can focus on any remaining disagreement.

Kind regards,
The Authors

Dear Area Chair and Reviewers,

We would appreciate your guidance on the remaining shared concern: whether functional-response calibration adds value beyond simpler alternatives. We consolidate the evidence below and clarify how the comparisons relate.

Adding NC regularization versus choosing the NC objective

Table 2 compares the complete NC procedure with matched no-NC continuation, which uses only the denoising loss. At Layer 24, NC improves mean FID from 9.055 to 6.626 and precision from 0.5597 to 0.6133, with a recall trade-off.

Table 10 compares two regularization objectives within our undershoot-guided framework. Both use the same seven Phase 1 undershoot-selected pairs under matched fixed controllers: the Phase 1 layer scale, unit pair scales, and disabled adaptive calibration.

  • Plain commutator shrinkage penalizes nonzero routing commutators unconditionally.
  • The calibrated hinge conditions the penalty on the functional response and becomes inactive when the calibrated constraint is satisfied, potentially at nonzero commutator magnitude.

The simpler shrinkage objective therefore retains our undershoot-based pair selection. This comparison examines how the selected pairs are regularized. Conversely, magnitude ranking changes pair selection while retaining the calibrated hinge and online controllers.

Under fixed controllers, the hinge reaches mean FID 6.850 and shrinkage 6.774, compared with 9.055 for denoising-only continuation. Both regularization variants achieve lower reported FID and higher precision than no-NC training, with recall trade-offs. Thus, the reported gains in these metrics occur with both regularization objectives. This comparison spans Tables 2 and 10; the directly matched comparisons are NC versus no-NC in Table 2 and hinge versus shrinkage in Table 10.

Consolidated evidence

ComparisonMain finding
Complete NC procedure versus matched denoising-only continuationLayer 24 mean FID improves from 9.055 to 6.626, with higher IS and precision and lower recall.
Undershoot selection versus equal-budget random selectionControls cover the 25 under-satisfied candidates and all 34 valid observed candidates, with three random sets per pool and two seeds per set. Undershoot selection achieves better FID, IS and precision than the tested random sets.
Undershoot versus commutator-magnitude rankingThe selections share six of seven pairs (85.7%), with similar mean FID: 6.626 versus 6.593.
Coefficient-normalized pair budgetsComparisons of 3, 7 and 13 pairs control the total pair coefficient before outer weighting.
Calibrated hinge versus plain shrinkageBoth use the same undershoot-selected pairs and achieve lower reported FID and higher precision than no-NC (across Tables 2 and 10). Within the matched comparison, the hinge gives higher IS and precision; shrinkage gives slightly lower FID and higher recall.
Calibration settings and sensitivityAppendices A.5–A.6.1 document training settings, calibration rules, and sensitivity to percentile, EMA, centering and clipping.

Additional evidence on undershoot targeting

We additionally compare seven undershoot-selected with seven overshoot-selected pairs at ImageNet Layer 24. Within each objective, we retain the checkpoint, training budget, controller settings and evaluation protocol, changing the selected pair set. Undershoot and overshoot status are determined from Phase 1 measurements.

Calibrated hinge: mean ± sample standard deviation over continuation seeds 0 and 1, with 50,000 generated images per run.

Pair selectionFID ↓IS ↑Precision ↑Recall ↑
Undershoot6.626 ± 0.188130.53 ± 1.150.6133 ± 0.00390.6658 ± 0.0017
Overshoot8.712 ± 0.167114.60 ± 1.140.5660 ± 0.00360.6916 ± 0.0020

Plain shrinkage: completed evaluation for seed 0, with 50,000 generated images per configuration. The second seed remains pending.

Pair selectionFID ↓IS ↑Precision ↑Recall ↑
Undershoot6.904126.980.59920.6781
Overshoot8.933114.390.56250.6893

The shrinkage undershoot row is the seed-0 result underlying Table 10. Table 10 reports a three-seed mean of 6.774; here we use the corresponding seed-0 reference for a matched comparison.

Under both objectives, the tested undershoot-selected pairs achieve lower FID and higher IS and precision than the overshoot-selected pairs, while overshoot selection yields higher recall. The shrinkage comparison is informative because its penalty is unconditional: the observed difference does not depend on hinge activation. These results support the practical value of targeting the selected undershooting pairs, with the shrinkage finding currently limited to one seed.

Request for guidance

We would be grateful if Reviewer sEEv could indicate whether these additional comparisons address the concern about undershoot targeting. If a concern remains about the specific contribution of functional calibration, identifying the comparison or clarification still needed to guide us in the right direction. We also welcome perspectives from other reviewers 2X68, and F14g wishing to contribute.

We respectfully ask the Area Chair to consider this consolidated evidence alongside the revised manuscript and discussion.

Kind regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.