acceptodds
Under review as a conference paper at ICLR 2027

TSGate: Timestep-aware Gated Attetion for Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have emerged as the dominant architecture for high-fidelity image and video generation. Recent DiT systems increasingly use structured prompts for training, improving caption quality and prompt adherence. However, their generation quality can degrade severely under out-of-domain (OOD) prompts, including the free-form descriptions supplied by users at inference time. Although LLM-based rewriting can convert these prompts into structured formats, it does not guarantee alignment with the training distribution. Our analysis links this degradation to attention sinks and reduced early-step image-to-text attention, and shows that sink suppression alone is insufficient to restore generation quality. Despite effective sink suppression, models trained with standard gated attention exhibit reduced early-step image-to-text attention and suboptimal generation quality. Based on these insights, we propose Timestep-Aware Gated Attention (TSGate), which injects a timestep-conditioned bias into the gate signal so that gating behavior adapts across denoising steps. Extensive experiments show that TSGate consistently outperforms both baseline and standard gated attention across multiple benchmarks, improving raw-prompt DPG by 9.5% over baseline.

Primary area:
generative models
Keywords:
Diffusion Transformers, Attention Sink, Gated Attention, Generative Models, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b4PkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the practical problem of DiTs trained with structured prompts degrading substantially when given free-form prompts from real users. It investigates this issue through attention sinks and early cross-modal interactions during denoising, and proposes a simple method called TSGate.

Critical strengths

1.The paper studies a practically relevant distribution shift in which models are trained with structured descriptions but receive free-form prompts at inference time. 2.TSGate is simple and easy to integrate into existing DiTs. It adds a timestep-dependent bias to the gating logits without directly modifying the attention softmax. 3.The progressive prompt degradation experiment includes several intermediate conditions, providing a more informative analysis than a simple comparison between structured and raw prompts.

Critical weaknesses

1.The evidence linking attention sinks to generation degradation is largely correlational. Since standard gating changes attention more broadly, the current experiments do not establish a direct causal relationship. 2.The main OOD comparison changes prompt format, length, and information content simultaneously. Comparisons using semantically matched prompts that differ only in format would better isolate the effect of format shift. 3.TSGate is not compared with other parameter-matched ways of conditioning the gate on time, such as feature concatenation, FiLM, or AdaLN. Such comparisons are needed to separate the benefit of timestep alignment from that of added capacity.

Questions for authors
  1. How do TSGate and TSGate-L0 affect training speed, inference latency, and memory usage compared with the baseline?
  2. Why does TSGate-head perform substantially worse than element-wise TSGate and slightly below the baseline? Does this suggest that channel-wise gating is essential to the improvement?
3 comments
AuthorsOpenReview ↗

Comment for Reviewer b4Pk

Thank you for your careful reading and for recognizing the value of our problem and experiments. We respond below.

Weak1. Linking attention sinks to generation quality

The quality gain comes from addressing two factors together: attention sinks and early text-image attention interaction. Our argument is a three-step chain, each step supported by a comparison that controls a different factor. (1) Template markers affect generation quality. Section 4.1 identifies recurring heading markers as attention sinks. The w/o section condition in Figure 7 removes these markers but keeps the prose unchanged. Baseline drops from 78.6 to 77.2, while TSGate stays at 79.4 to 79.5. (2) Suppressing sinks helps, but it is not enough. Gated attention, the natural remedy for sinks, reduces the Layer 0 Sink Score from 0.0383 to 0.0204 and improves raw DPG from 55.483 to 58.976. However, it has the lowest early image-to-text attention of the three models (Figure 6), when global semantics are established. (3) Early text interaction is the missing part. As you point out, gating changes attention broadly, so we compare two models with the same content gating that differ only in the time branch (Table 2). TSGate has a higher Layer 0 Sink Score than Gated attention (0.0287 vs. 0.0204), yet its early image-to-text fraction is 15.9 percentage points higher and its raw DPG is also higher (60.749 vs. 58.976; difference 1.773, 95% CI [0.709, 2.870]). Raw-prompt quality is therefore not determined by sink suppression alone; TSGate improves it by addressing both factors.

Weak2. Separating format, length, and information content

The progressive experiment in Figure 7 was designed to separate format, length, and information content.

StepConditionWhat changesTSGate − Baseline
0In domainNothing+0.7
1Shuffle sectionFormat; content preserved+1.8
2W/o sectionFormat; prose preserved+2.3
3W/o stylePart of the description removed+0.9
4Only subjectOnly the subject description kept+1.5
5Raw promptFormat, length, and information+5.3

TSGate leads in all six conditions. With content preserved, the gap grows from +0.7 to +2.3 as the format changes, and is largest when all three shift together. Figure 6 examines length separately: the early image-to-text gain over Gated attention holds for both short and long prompts (16.6 and 15.2 percentage points). We will clarify this in the revision.

Weak3. Other forms of time conditioning, and alignment versus capacity

These methods form one family of gate-logit designs:

ConditioningGate logitsIn the paper
AdaLN / FiLM on the gate inputXs(t)W+bX_s(t)W+b, with Xs(t)X_s(t) modulated by adaLN-ZeroGated attention
Feature concatenation + linear projectionXs(t)W1+ϕ(t)W2+bX_s(t)W_1+\phi(t)W_2+bTSGate (Equation (5), ϕ=SiLU(temb)\phi=\text{SiLU}(t_{emb}))
FiLM shift on the logitsCs(t)+β(t)C_s(t)+\beta(t)TSGate
FiLM scale and shift on the logits(1+γ(t))⊙Cs(t)+β(t)(1+\gamma(t))\odot C_s(t)+\beta(t)A natural extension

Gated attention versus TSGate is thus a comparison between two members of this family, where the additive form gains 1.773 on raw DPG. We chose the shift because a shared offset preserves the content-based ordering of gates across tokens (Appendix A.3). In the revision, we will add comparisons with other parameter-matched forms of time conditioning to the appendix.

Q1. Training speed, inference latency, and memory

TSGate's content gate is the same as in standard gated attention: a widened query projection and an element-wise sigmoid multiplication. Its time branch adds one linear projection per stream and layer, computed once and shared across tokens; this costs the same as projecting one extra token and does not grow with resolution or prompt length. With Baseline's wider shared expert, the two models have nearly equal total parameters (1,250.8M vs. 1,250.7M). TSGate-L0 is 6.2% smaller (1,172.9M). The training procedure and sampling loop are unchanged.

Q2. TSGate-head and channel-wise gating

Yes, fine-grained channel selection appears important on raw prompts, which is why element-wise gating is our default. A head-wise gate applies one value to all 128 channels of a head and has fewer parameters by construction, so it is less able to select features within a head. TSGate-head still performs well elsewhere: it is above Baseline on full-recaption DPG (78.858 vs. 78.637) and GenEval (0.7972 vs. 0.7861), and it has the highest T2I-CompBench mean (0.5264). Its weakness on raw DPG indicates that channel-level selection is especially useful under prompt shift.

Thank you for your time and for raising these concerns, which have been very helpful for improving our paper. Please feel free to let us know if you have any remaining concerns. We would greatly appreciate it if you could reconsider your score in light of our response, should it address your concerns.

Thank you for the clarifications, which alleviate my concern about prompt-format confounding. However, the parameter-matched comparisons with alternative time-conditioning designs remain pending, leaving the specific contribution of the proposed time branch insufficiently established. I am maintaining my current rating.

Thank you for the follow up. Here are something we would like to claim.

(1) Two of the listed alternatives coincide with TSGate. Concatenating ϕ(t)\phi(t) to the gate input followed by a linear projection gives [Xs(t),ϕ(t)]W=Xs(t)W1+ϕ(t)W2[X_s(t), \phi(t)]W = X_s(t)W_1 + \phi(t)W_2, which is exactly Equation (5). A FiLM shift on the gate logits is the same additive term. The remaining untested case is a time-dependent scale on the logits. (2) Alignment versus capacity, with capacity held fixed. The reverse intervention (Table 10) keeps every parameter and every learned bias value of the trained TSGate, and only reverses the order in which these biases are applied across denoising steps. Raw DPG drops by 2.614 (95% CI [−4.281, −0.994]). Since capacity is unchanged, this drop reflects alignment with the denoising stage. (3) Most of the gain does not require the added parameters. TSGate-L0 adds the time branch only at layer 0, which is 7.1M parameters (0.6%) more than Gated attention. It reaches 60.424 raw DPG versus 58.976, recovering 82% of the full TSGate gain, and its difference from full TSGate is −0.325 (95% CI [−1.291, 0.615]). In addition, the branch takes only tt as input, so it cannot add content-dependent capacity. At inference, it reduces to a fixed bias for each sampling step.

We also want to be precise about the scope of our claim. Matched-mean inference changes raw DPG by −0.929 with an interval that includes zero (Table 10). We therefore do not claim that a time-varying schedule is necessary, or that the additive form is the best way to condition the gate on time. Our claim is that an explicit, timestep-aligned offset on the gate logits improves over content-only gating. We hope these clarifications address your remaining concerns on "time-conditioning designs".

Reviewer_ovPwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work studies why the quality of Diffusion Transformers degrades when the model, trained on structured captions, is given out-of-domain raw prompts. The authors show that suppressing attention sinks (via gated attention) is insufficient to maintain model performance, as text attention is reduced during high-noise early sampling steps. Based on this, they claim that a proper gating mechanism should be timestep-aware and introduce TSGate, a mechanism that is characterized by denoising time dependence. Experiments indicate that training a Diffusion Transformer with TSGate consistently outperforms the plain baseline and the Gated Attention variant when the final model is evaluated on OOD prompts, suggesting greater robustness to prompt format.

Critical strengths

The authors provide several mechanistic analyses that mostly support their claims about the root cause of OOD prompt degradation in DiTs. These analyses make the proposed method sound well-motivated. Also, the reported quality gains on raw prompts are well documented, with extensive comparisons often including confidence intervals.

Critical weaknesses

Weakness 1. Motivation in Practice.

I am not convinced by the paper's main motivation: crafting diffusion models that are more robust to out-of-distribution prompts. As the authors already state, practically, this problem is usually solved using prompt engineering.

The authors point out that classical rewriting mechanisms cannot ensure the elimination of distributional shift, but I did not find any convincing evidence of it in the paper. In Tab. 1, TSGate indeed offers a lot for raw prompts, but its advantage is marginal and unconvincing for full re-captions, even though it requires training the Diffusion Model from scratch.

Given that such a computation is considered alongside your method, the comparison misses many baselines that could work here: mixing raw prompts with re-captioned ones during diffusion model training, or using post-training techniques for prompt adherence. Also, a potential user could easily consider fine-tuning an LLM for prompt rewriting to obtain a better distribution of prompts. Another option is using few-shot prompting. I believe that to make your method useful in practice, the authors should demonstrate that it can outperform these promising baselines that also aim to make the Diffusion Model more prompt-robust.

I also believe that stating your method as a "plug-and-play" enhancement is not quite accurate, as the mechanism requires training the model from scratch.

Weakness 2. Causes of prompt degradation

Some observations and experiments on root causes of OOD prompt degradation seem to reiterate already known observations, raising my novelty concerns:

  1. Template tokens largely absorb attention in DiTs [1].
  2. Sinks' severity depends on diffusion noise level and layers [2,3]. However, in Section 2.2, the authors claim that their work fills a gap in the literature by "demonstrating that effective solutions must be timestep-aware".

To me, the authors overstate by claiming that existing studies:

  • characterize sinks in isolation (see [1]),
  • fail to analyze this problem in depth (see [2]).

[1] Li, Maohua, et al. "Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers." arXiv:2607.19139.

[2] Wu, Fangzheng, and Brian Summa. "Attention sinks in diffusion transformers: A causal analysis." ICML 2026.

[3] Wu, Xiaoyu, et al. "Taming Outlier Tokens in Diffusion Transformers." Wu, Xiaoyu, et al. "Taming Outlier Tokens in Diffusion Transformers." arXiv:2605.05206

Questions for authors

Q1. Figure 18 contains contradicting results for recaptioned representations (REC): in absolute DPG score (a) gated attention outperforms TSGate, while (b) the score difference between them is +1.01 advantage for TSGate. Which of these two results is correct?

Q2. It is not clear to me from the text how the rewritten prompts for the DPG recaption dataset were obtained. Could you describe this process more thoroughly than it is done in the AI Use Statement?

Q3. Could you provide some examples of raw prompts and their rewrites? I've only seen structured versions in Appendix D.

Q4. It is an interesting finding in Section 4.1 that attention concentrates on separators, but I couldn't find any analyses supporting this claim. Could you elaborate on that?

4 comments
AuthorsOpenReview ↗

Comment for Reviewer ovPw (Weak1-2)

Thank you for your comments. Some of the concerns conflate our contribution with existing observations, or treat inference-side techniques as substitutes for model-side methods. We clarify these points below.

Weak1. Practical motivation

Prompt engineering is of course standard industrial practice, but it does not eliminate this problem. It is standard precisely because user inputs must be mapped onto the training caption distribution, and this mapping cannot be guaranteed to be accurate. The captioner and the prompt-rewriting model are two independent models that take entirely different inputs. To our knowledge, how to better align their distributions remains an open problem.

Our paper provides evidence for this. The intermediate conditions in Figure 7 are exactly what imperfect rewriting produces: reordered sections, missing headings, and missing fields. TSGate outperforms Baseline under all of these conditions (+1.8, +2.3, +0.9, +1.5), and the gap reaches +5.3 on raw prompts (Table 1). Figures 8–15 show the corresponding qualitative improvements.

The smaller gap under full recaption is expected, since it is in-domain input. This condition is precisely LLM rewriting combined with TSGate, and TSGate still scores higher (79.355 vs. 78.637). TSGate does not compete with rewriting models (whether fine-tuned or few-shot) or with post-training techniques for prompt adherence. They operate on the prompt or a trained model and can be applied on top of TSGate.

Mixing raw and re-captioned prompts. This approach was used in earlier foundation-model training, but it is not a good choice. Training on more detailed structured prompts itself improves DiT generation quality, which is why training captions across the industry have become increasingly long and detailed. Industrial examples abound; e.g., Scaling Properties of Text Conditioning in Visual Generation (ByteDance Seed, 2026) reports that richer structured prompts substantially improve generation quality. We do not believe that mixing in SD1.5-era short prompts and tags improves results, and we verified this long ago. Moreover, mixed training also requires training from scratch, so cost does not distinguish it from TSGate.

Plug-and-play. Training from scratch is inherent to gated attention, as noted by Qiu et al. (2025). TSGate is a plug-and-play component at the level of foundation-model design and requires no other architectural changes. For a model that will be trained anyway, it adds almost no overhead (1,250.8M vs. 1,250.7M total parameters, with an unchanged training procedure).

Weak2. Novelty

We do not claim to be the first to discover attention sinks in DiTs. They were reported years ago, and this is not our contribution. Our contribution is to show that suppressing sinks alone cannot restore generation quality under OOD prompts, and to address this with timestep-aware gating.

[2] studies where sinks occur and their causal role. It does not address how to resolve sinks, as we already note in the paper.

[3] examines, from the perspective of outlier tokens in ViTs, whether a similar phenomenon exists in DiTs. Its solution is register tokens, also common in LLMs, an entirely different route from gated attention.

That sink severity varies with noise level does not imply our conclusion, because our timestep awareness is not derived from sink severity. TSGate has a higher Layer 0 Sink Score than Gated attention (0.0287 vs. 0.0204), yet its early image-to-text fraction is 15.9 percentage points higher and its raw DPG is also higher (60.749 vs. 58.976; difference 1.773, 95% CI [0.709, 2.870]). Neither [2] nor [3] establishes this. We will add a more detailed comparison with [2] and [3] in the revision.

[1] shows a problem that is established industry consensus that template tokens attract a large share of attention. Since OmniGen2, it has been widely recognized that generation quality degrades when the system prompt is not truncated, and the same holds for fixed template tokens. This is precisely why public pipelines such as Qwen-Image discard the fixed template prefix. Point 1 of this weakness therefore does not concern our contribution. Furthermore, [1] goes on to study how semantics are stored in template tokens, a question we consider of limited value. Since a sink means the model dumps attention onto these tokens, the quality drop after removing them is obvious and needs no dedicated explanation. What is truly worth studying is how to resolve the sink problem. We cited [1], but its research goal is entirely different from ours.

AuthorsOpenReview ↗

Comment for Reviewer ovPw (Q1-4)

Q1. Figure 18

Thank you for pointing this out. This was a plotting error on our side, which we will correct in the revision. The data are below (the same 300 DPG-Bench sources; 95% source-bootstrap CIs):

InputBaselineGated attentionTSGateTSGate − Gated attention95% CI
Raw55.37458.62660.918+2.292[0.701, 3.861]
Str57.74361.92761.164−0.763[−2.527, 0.908]
Hdr56.83761.31860.684−0.634[−2.329, 1.056]
Rep55.81658.94361.487+2.544[0.931, 4.088]
Pad55.09458.27859.146+0.868[−0.827, 2.543]
REC78.46378.21779.148+0.931[−0.243, 2.030]

On REC, TSGate − Gated attention is +0.931 (95% CI [−0.243, 2.030]), with TSGate ahead in mean.

Q2. How the rewritten DPG prompts were obtained

The structured format is defined in Appendix D. As stated in the AI Use Statement, training images were re-captioned into this format with Qwen2.5-VL 72B, and the original evaluation prompts were rewritten into the same in-domain format with deepseek-v4.1-flash and Gemini-3.1-pro.

Q3. Examples of raw prompts

Raw prompts for the three structured examples in Appendix D:

(a) Lavender, sample 0439: A dreamy close-up of purple lavender flowers in soft daylight, with a few delicate blooms sharply focused against a warm, creamy bokeh background. The shallow depth of field and gentle lighting create a calm, intimate atmosphere.

(b) Mountain light, sample 0454: A wide landscape photograph of golden sunlight illuminating rolling hills and a forested ridge, with patches of snow in the foreground and distant snow-capped mountains beneath a partly cloudy blue sky. Warm highlights contrast with cool shadows, creating a serene, expansive atmosphere.

(c) Fresh still life, sample 0861: A bright overhead still-life photograph of pink grapefruit slices on a white plate, green grapes in a bowl, and deep red dahlias arranged on a white marble countertop. Soft natural window light, fresh vibrant colors, and gentle shadows create a relaxed morning atmosphere.

Q4. Attention concentrating on separators

We briefly explain the mechanism behind sinks again: Tokens that appear consistently across samples become sink targets, which is also why repeated tokens exhibit this problem. Whatever captions are used, sinks will arise as long as some tokens occur more frequently than others. Under OOD prompts, prompt engineering cannot precisely reproduce the frequency of every token, so an appropriate mechanism is needed inside the model. Quantitatively, averaged over 20 cases:

Sink typeAttention relative to an average token
Padding/EOS42.4×
Markdown #59.8×
Newline \n34.5×

We will add this table in the revision.

Finally, we want to be clear. This paper studies a model-level problem: why DiTs trained on structured captions degrade under OOD prompts, and why sink suppression alone cannot restore generation quality. Prompt rewriting and post-training alignment techniques act at inference time or on an already trained model. They are orthogonal to our work and can be applied on top of it, and we do not see them as a reason to dismiss model-side research. [1], [2], and [3] discuss the existence, location, and causes of sinks. None of them addresses degradation under OOD prompts, and none overlaps with our contribution. You assigned a Soundness score of 1, i.e., "significant concerns about the paper's soundness that do not appear easy to address." However, the only concrete issue raised in the review is the plotting error in Figure 18 (a), which does not affect any conclusion of the paper. The remaining comments concern motivation and positioning relative to related work, not any error in our experiments or derivations. We believe the current score rests on a misreading of our contribution, and the responses above have addressed every point you raised.

I thank the Authors for their hard work during the rebuttal and for the new clarifications.

I now feel convinced by your explanation of the differences from other related work (W2). I hope the authors will describe contrasts to prior papers more thoroughly. I also encourage you to include corrections and examples from Q1, Q3, and Q4 in the final version. With Q2, I was not precise enough, as I expected to see a better description of the rewriting procedure (e.g., prompts to LLM), but I will not take it into account in my final score.

Unfortunately, I still have some concerns about practicality and baseline comparisons. I will write them here and remain open to discussing them with the authors, being aware of the approaching end of the discussion period:

"The intermediate conditions in Figure 7 are exactly what imperfect rewriting produces: reordered sections, missing headings, and missing fields."

To my understanding, the "in domain" scores in Figure 7 show the setup in which prompt conditioning is the output of an external LLM (DeepSeek or Gemini). This scenario is where the Baseline performs best, while its performance deteriorates under conditions prepared by the authors for this study. I am aware that it does not necessarily mean that rewriting does not lead to such errors, but it suggests that the baseline handles this setup well.

"This condition is precisely LLM rewriting combined with TSGate, and TSGate still scores higher (79.355 vs. 78.637)."

I am aware of this difference, but it seems to be a marginal gap (below 1% relative). Do you have results for this exact comparison, including standard deviation or standard error, to demonstrate that the advantage of TSGate is reliable here?

Building on the previous point, I understand that rewriting can be considered an orthogonal approach, but if the benefit is small or not significant in a common deployment setting, the main motivation remains weak.

"We do not believe that mixing in SD1.5-era short prompts and tags improves results, and we verified this long ago. Moreover, mixed training also requires training from scratch, so cost does not distinguish it from TSGate."

An existing large diffusion model can still be fine-tuned, for example with LoRA, on raw and re-captioned prompts, while gated attention cannot be added to such an architecture. I understand this is not a limitation of your method, but it still makes mixed-data training more practically useful. Finally, the mixed training is, for me, a model-side fix, so it should have been compared with your method.

Concluding, I agree that my initial assignment of 1 to Soundness was unfair. I wanted it to reflect my concerns about the paper's practical motivation and the baseline comparison. Nevertheless, I updated it and, accordingly, changed my rating to weak rejection (2).

Thank you for the detailed follow up and for updating your scores! We believe the remaining points are addressed by results already in the paper, and we explain them below.

  1. Figure 7 and rewriting. The intermediate conditions are controlled perturbations by design: each changes one property while holding the others fixed. Their relevance to rewriting does not depend on how often a particular LLM makes a particular error. It depends on which properties a rewriter is structurally unable to guarantee. The training captions are written by a VLM that sees the image, whereas the rewriter sees only the user's text. Section order, heading markup, and whether a style or mood section exists are therefore not determined by the input. When a user specifies no style, the rewriter must either omit that section (W/o style, Only subject) or invent content the user did not ask for; Appendix E shows that full recaption can add or change scene facts. The in-domain condition in Figure 7 is the best case, with strong LLMs instructed to produce exactly the training format. Yet with identical content, simply reordering sections lowers Baseline from 78.6 to 77.1, and removing headings lowers it to 77.2, while TSGate stays at 79.5. A model whose quality depends on the rewriter reproducing heading markup and section order is fragile to any change of rewriter, template, or prompt style, regardless of how well a specific pipeline performs today.

  2. The full-recaption gap. On 1,065 DPG sources with four seeds each, TSGate − Gated attention is +1.028 and TSGate − Baseline is +0.718. On GenEval, which also uses rewritten prompts, TSGate − Baseline is +0.0203, whereas Gated attention alone stays at Baseline level (0.7862 vs. 0.7861). For a method targeting robustness, the requirement in the in-domain setting is that it does not trade away quality. TSGate has a higher mean than Baseline on all three full-recaption benchmarks, with a significant gain on GenEval.

  3. Practical motivation. Rewriting is common but not perfect, that is our original starting point. This idea came from our practical foundation model pre-training. TSGate complements rewriting rather than replacing it. When prompts are rewritten, TSGate still improves the results: full-recaption DPG rises from 78.637 to 79.355. When rewriting is absent or imperfect, TSGate also serves as a fallback, improving raw DPG by 5.266 points on the MoE backbone and by 22.540 points on the Dense backbone. Since both rewriting and TSGate improve generation quality, the two can naturally be used together.

  4. Mixed-caption training and LoRA. Mixed-caption training and TSGate act on different axes: one changes the training data, the other changes the architecture. Architecture studies, including gated attention (Qiu et al., 2025), compare designs under a fixed data recipe, and we follow this protocol. Any mixed-data recipe applies equally to a TSGate model, so the two are complementary rather than competing. Mixing also introduces a mixing ratio and requires anticipating the deployment prompt distribution, whereas TSGate obtains its raw-prompt gain without changing the data and without losing in-domain quality. LoRA fine-tuning of an existing model is a post-hoc adaptation, a different use case from the base-model design we study, and it can likewise be applied to a TSGate model. Moreover, TSGate's time branch is zero-initialized, so it can be inserted into a trained gated-attention model without changing its output at initialization.

We will include the data reported above and the points from this discussion in the revision. We hope this clarifies that the remaining concerns relate to the scope of deployment rather than to the validity of our results, and we would appreciate it if you could take these points into account in your final assessment.

Reviewer_dUDvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

When training large-scale text-to-image DiTs today — especially MMDiT-style architectures with joint attention over concatenated text and image tokens — it is common practice to rewrite training captions into structured long texts with fixed Markdown headings (e.g. # Summary, # Setting). Models trained this way perform well on inputs in the same format, but quality drops noticeably when users enter short, free-form text (OOD prompts).

Through attention-map analysis, the authors identify two phenomena. First, fixed markers in the template, such as headings and newline tokens, become attention sinks during training; once the input switches to natural short sentences, these markers are gone and the attention distribution is disrupted. Second, during early denoising (the high-noise stage), the fraction of image-to-text attention is markedly lower under OOD prompts, so the image does not receive enough textual guidance early on.

Standard gated attention suppresses sinks, but the authors observe that models trained with it show even lower early image–text attention. To address this, they propose TSGate: a channel-wise bias Bts,s(t)B_{ts,s}(t), computed from the timestep embedding through a SiLU and a linear layer, is added before the sigmoid of the gate. The idea is that this bias, which does not pass through the noisy image representation, keeps image–text interaction strong during early denoising and shifts toward image-internal refinement later. They also test a lightweight variant, TSGate-L0, which adds the time branch only at layer 0.

Critical strengths
  • The problem is real and practical. Training DiTs on structured long captions produced by VLM re-captioning is now standard in both industry and open-source work, and overfitting to the template format — with degraded results on raw short prompts — is a commonly encountered pain point.
  • The chain of analysis is interesting. The authors do not simply claim that "removing attention sinks makes things better." Instead, they show that standard gated attention suppresses sinks more strongly yet scores lower than TSGate (Figure 16, Table 1), and attribute the difference to the early image–text attention fraction and to the high-then-low crossover across stages in Figure 6.
  • Solid experimental scale, with careful statistics at the prompt level. The authors train several MoE and Dense DiTs of roughly 1.1–1.5B parameters from scratch on 100M re-captioned LAION images. Prompt-paired bootstrap confidence intervals are reported for the main results, and limitations such as the Dense comparison not being parameter-matched and some intervals crossing zero are stated openly. The fixed-weight time-bias intervention in Appendix F is a good idea; reversing the bias order lowers raw DPG by 2.6 points.
  • The method is simple and easy to implement in existing MMDiT architectures.
Critical weaknesses
  • Location: Table 1 (lines 324–334), Appendix Table 7 (lines 998–1013), Abstract (lines 024–026) The external benchmarks do not cover the paper's core scenario. The paper is about degradation under raw prompts, but only DPG-Bench evaluates raw prompts; Table 7 states "All models use full recaption" (line 1001), meaning GenEval and T2I-CompBench only evaluate structured prompts. Even under those structured prompts, TSGate beats Baseline on CompBench by only 0.0012, with a 95% CI of [−0.0046,0.0067][-0.0046, 0.0067] that crosses zero (it is also below the paper's own TSGate-head variant, 0.5264). The original prompts of these two benchmarks are themselves short free-form texts, so raw prompts should be evaluated on them directly. The claim in the abstract that TSGate "consistently outperforms both the baseline and standard gated attention" should also be toned down.

  • Location: Appendix F Table 10 (lines 1296–1310), Figure 6 A hand-designed time schedule is missing as a control, so it is unclear whether the learned timestep bias is actually necessary. Following the paper's own motivation, a simpler approach would be to add a scalar bias that varies with the timestep before the sigmoid, e.g. b(t)=b0+k⋅t/Tb(t)=b_0+k\cdot t/T, large early and small late. In Appendix F, the reverse intervention suggests that temporal order does matter to some extent, but the matched-mean intervention (applying each channel's average bias at every step) changes raw DPG by only −0.9, with a confidence interval containing zero. Note that a time-constant per-channel bias can be absorbed exactly into the content-gate bias bq,gb_{q,g}, so under matched-mean the TSGate checkpoint is functionally a content-only gated model at inference, only with shifted gate biases. That it loses only 0.9 points suggests that much of TSGate's advantage over Gated attention may come from training with the time branch, rather than from the bias varying across timesteps at inference. The current experiments cannot separate these explanations; a hand-designed schedule trained from scratch would help clarify both whether the bias needs to vary with time and whether it needs to be learned per channel.

  • Location: Section 4.2.3 (lines 445–449), Appendix Tables 3 (lines 864–884) and 5, line 856 The Dense experiment lacks a Gated attention baseline. On Dense, raw DPG rises from 41.6 to 64.2 (+22.5), the largest gain reported anywhere in the paper, and it is used to argue that TSGate applies beyond MoE. But the Dense setting compares only an ungated Baseline against TSGate, with no Gated attention (line 856), so one cannot tell whether these 22.5 points come from gating itself or from the timestep bias. Dense TSGate also has about 340M more parameters than Baseline, half of which are content-gate projections that process tokens directly, and the authors acknowledge the comparison is not parameter-matched (lines 448–449). The Dense Baseline scores 81.8 on full recaption but only 41.6 on raw, i.e. it is unusually sensitive to prompt format, which may further inflate the gain. I suggest adding a Dense Gated attention run, or explicitly positioning the Dense results in the main text as supporting evidence only.

Questions for authors
  1. What are the results on GenEval and T2I-CompBench with raw short prompts? Does TSGate still hold a clear advantage over both Baseline and Gated attention there?
  2. Could you add a hand-designed time-schedule control, i.e. add a scalar bias to standard gated attention that varies linearly (or piecewise) with the timestep, keeping all other settings identical to TSGate, and report raw / full-recaption DPG together with the early image-to-text attention fraction as in Figure 6? If convenient, please also plot the learned Bts(t)B_{ts}(t) as a function of the timestep for several layers and channels.
  3. Once a standard Gated attention baseline is added to the Dense experiment, does the gain come mainly from gating itself or from the timestep bias?
1 comment
AuthorsOpenReview ↗

Comment for Reviewer dUDv

We thank the reviewer for the careful reading and constructive suggestions! We address your comments below.

Weak1 / Q1. GenEval and T2I-CompBench with raw prompts

These two benchmarks were evaluated with full-recaption prompts only (Table 7). The paper's evidence on raw prompts comes from DPG-Bench (1,065 paired sources) and the progressive prompt-degradation conditions in Figure 7. In the current paper, GenEval and T2I-CompBench serve as compositional evaluations in the training format. Within this scope, Table 7 reports four paired contrasts:

ContrastGenEvalT2I-CompBench
TSGate − Gated attention+0.0202 [0.0007, 0.0401]+0.0081 [0.0023, 0.0141]
TSGate − Baseline+0.0203 [0.0025, 0.0389]+0.0012 [−0.0046, 0.0067]

The TSGate − Baseline interval on T2I-CompBench is the only one of the four that contains zero, and TSGate's gain remains visible on average. We agree that "consistently outperforms" is too strong, and we will revise this sentence in the abstract accordingly.

Weak2 / Q2. Whether the learned, time-varying bias is necessary

Training with the time branch is part of the method. Gated attention and TSGate have identical configurations except for the time branch (Table 2; the 84.99M parameter difference is exactly this branch), and their content gates use the same initialization. Gated attention can already express any constant per-channel offset through bq,gb_{q,g}. The distinction between the two models is therefore the explicit time branch, not the ability to represent a constant offset. The −0.929 change is not small relative to the relevant gap. The relevant scale is the TSGate − Gated attention gap of +1.773, not the absolute DPG score. The matched-mean point estimate is about half of this gap, and its interval [−2.455, 0.560] extends beyond the full gap. These figures come from different source sets (200 and 1,065), so the comparison is approximate. An interval containing zero does not establish equivalence, so Table 10 shows neither that most of the advantage comes from training nor that it comes from time variation at inference. The temporal assignment has a measurable effect on raw prompts. The reverse intervention preserves each channel's bias values and mean, changing only their correspondence to denoising stages, yet it lowers raw DPG by 2.614 (95% CI [−4.281, −0.994]). This shows that alignment with the denoising stage matters at inference on raw prompts. It does not show that a time-varying schedule is better than a well-chosen constant bias, and the corresponding interval on structured prompts contains zero. Therefore, we do not claim that a learned per-channel schedule is necessary.

Weak3 / Q3. Dense experiment without a Gated attention baseline

We would like to briefly clarify the role of this experiment. It asks whether the complete TSGate module, compared with no gate, transfers to a Dense backbone. We do not use it to isolate the contribution of the time bias. That question is addressed only in the MoE setting, where Gated attention and the interventions in Appendix F are available. We agree that the paper should state this purpose more clearly.

Thank you for your time and for raising these concerns, which have been very helpful for improving our paper. Please feel free to let us know if you have any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.