- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the practical problem of DiTs trained with structured prompts degrading substantially when given free-form prompts from real users. It investigates this issue through attention sinks and early cross-modal interactions during denoising, and proposes a simple method called TSGate.
Critical strengths
1.The paper studies a practically relevant distribution shift in which models are trained with structured descriptions but receive free-form prompts at inference time. 2.TSGate is simple and easy to integrate into existing DiTs. It adds a timestep-dependent bias to the gating logits without directly modifying the attention softmax. 3.The progressive prompt degradation experiment includes several intermediate conditions, providing a more informative analysis than a simple comparison between structured and raw prompts.
Critical weaknesses
1.The evidence linking attention sinks to generation degradation is largely correlational. Since standard gating changes attention more broadly, the current experiments do not establish a direct causal relationship. 2.The main OOD comparison changes prompt format, length, and information content simultaneously. Comparisons using semantically matched prompts that differ only in format would better isolate the effect of format shift. 3.TSGate is not compared with other parameter-matched ways of conditioning the gate on time, such as feature concatenation, FiLM, or AdaLN. Such comparisons are needed to separate the benefit of timestep alignment from that of added capacity.
Questions for authors
- How do TSGate and TSGate-L0 affect training speed, inference latency, and memory usage compared with the baseline?
- Why does TSGate-head perform substantially worse than element-wise TSGate and slightly below the baseline? Does this suggest that channel-wise gating is essential to the improvement?
3 comments
Thank you for the clarifications, which alleviate my concern about prompt-format confounding. However, the parameter-matched comparisons with alternative time-conditioning designs remain pending, leaving the specific contribution of the proposed time branch insufficiently established. I am maintaining my current rating.
Thank you for the follow up. Here are something we would like to claim.
(1) Two of the listed alternatives coincide with TSGate. Concatenating to the gate input followed by a linear projection gives , which is exactly Equation (5). A FiLM shift on the gate logits is the same additive term. The remaining untested case is a time-dependent scale on the logits. (2) Alignment versus capacity, with capacity held fixed. The reverse intervention (Table 10) keeps every parameter and every learned bias value of the trained TSGate, and only reverses the order in which these biases are applied across denoising steps. Raw DPG drops by 2.614 (95% CI [−4.281, −0.994]). Since capacity is unchanged, this drop reflects alignment with the denoising stage. (3) Most of the gain does not require the added parameters. TSGate-L0 adds the time branch only at layer 0, which is 7.1M parameters (0.6%) more than Gated attention. It reaches 60.424 raw DPG versus 58.976, recovering 82% of the full TSGate gain, and its difference from full TSGate is −0.325 (95% CI [−1.291, 0.615]). In addition, the branch takes only as input, so it cannot add content-dependent capacity. At inference, it reduces to a fixed bias for each sampling step.
We also want to be precise about the scope of our claim. Matched-mean inference changes raw DPG by −0.929 with an interval that includes zero (Table 10). We therefore do not claim that a time-varying schedule is necessary, or that the additive form is the best way to condition the gate on time. Our claim is that an explicit, timestep-aligned offset on the gate logits improves over content-only gating. We hope these clarifications address your remaining concerns on "time-conditioning designs".
Comment for Reviewer b4Pk
Thank you for your careful reading and for recognizing the value of our problem and experiments. We respond below.
Weak1. Linking attention sinks to generation quality
The quality gain comes from addressing two factors together: attention sinks and early text-image attention interaction. Our argument is a three-step chain, each step supported by a comparison that controls a different factor. (1) Template markers affect generation quality. Section 4.1 identifies recurring heading markers as attention sinks. The w/o section condition in Figure 7 removes these markers but keeps the prose unchanged. Baseline drops from 78.6 to 77.2, while TSGate stays at 79.4 to 79.5. (2) Suppressing sinks helps, but it is not enough. Gated attention, the natural remedy for sinks, reduces the Layer 0 Sink Score from 0.0383 to 0.0204 and improves raw DPG from 55.483 to 58.976. However, it has the lowest early image-to-text attention of the three models (Figure 6), when global semantics are established. (3) Early text interaction is the missing part. As you point out, gating changes attention broadly, so we compare two models with the same content gating that differ only in the time branch (Table 2). TSGate has a higher Layer 0 Sink Score than Gated attention (0.0287 vs. 0.0204), yet its early image-to-text fraction is 15.9 percentage points higher and its raw DPG is also higher (60.749 vs. 58.976; difference 1.773, 95% CI [0.709, 2.870]). Raw-prompt quality is therefore not determined by sink suppression alone; TSGate improves it by addressing both factors.
Weak2. Separating format, length, and information content
The progressive experiment in Figure 7 was designed to separate format, length, and information content.
TSGate leads in all six conditions. With content preserved, the gap grows from +0.7 to +2.3 as the format changes, and is largest when all three shift together. Figure 6 examines length separately: the early image-to-text gain over Gated attention holds for both short and long prompts (16.6 and 15.2 percentage points). We will clarify this in the revision.
Weak3. Other forms of time conditioning, and alignment versus capacity
These methods form one family of gate-logit designs:
Gated attention versus TSGate is thus a comparison between two members of this family, where the additive form gains 1.773 on raw DPG. We chose the shift because a shared offset preserves the content-based ordering of gates across tokens (Appendix A.3). In the revision, we will add comparisons with other parameter-matched forms of time conditioning to the appendix.
Q1. Training speed, inference latency, and memory
TSGate's content gate is the same as in standard gated attention: a widened query projection and an element-wise sigmoid multiplication. Its time branch adds one linear projection per stream and layer, computed once and shared across tokens; this costs the same as projecting one extra token and does not grow with resolution or prompt length. With Baseline's wider shared expert, the two models have nearly equal total parameters (1,250.8M vs. 1,250.7M). TSGate-L0 is 6.2% smaller (1,172.9M). The training procedure and sampling loop are unchanged.
Q2. TSGate-head and channel-wise gating
Yes, fine-grained channel selection appears important on raw prompts, which is why element-wise gating is our default. A head-wise gate applies one value to all 128 channels of a head and has fewer parameters by construction, so it is less able to select features within a head. TSGate-head still performs well elsewhere: it is above Baseline on full-recaption DPG (78.858 vs. 78.637) and GenEval (0.7972 vs. 0.7861), and it has the highest T2I-CompBench mean (0.5264). Its weakness on raw DPG indicates that channel-level selection is especially useful under prompt shift.
Thank you for your time and for raising these concerns, which have been very helpful for improving our paper. Please feel free to let us know if you have any remaining concerns. We would greatly appreciate it if you could reconsider your score in light of our response, should it address your concerns.