acceptodds
Withdrawn from ICLR 2027

Spatiotemporal Response Modeling for Diffusion Transformer Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Controlling diffusion-transformer generation requires knowing where an internal edit propagates and how simultaneous edits interact. We introduce ST-Lens, a signed, coordinate-resolved response interface conditioned on residual block and denoising regime. It preserves spatial and video-time structure, revealing regional effects and cross-time interference. ST-Lens reduces response error by 37.15% against a frozen-path readout on 768 held-out fixed-dictionary interventions. We develop response-coupled residual editing (RCRE) to turn this information into coordinated temporal control: a full response Gram matrix jointly allocates a bounded residual write, and native-suffix feedback verifies its effect. Image control attains all 792 prescribed targets; Wan edits execute reversals, pauses and curved return trajectories. Response diagnostics motivate generation-policy candidates, whose final outcomes are measured with complete sampler rollouts. On ImageNet, a standard 50,000-generated-image evaluation per DiT arm against the 50,000 validation images gives 13.73% lower FID and 56.66% lower KID for scheduled guidance; a separate paired 1,000-image run gives a 1.312 sampling-and-decoding speedup. On MSR-VTT, advected sharing reduces mean FVD by 29.35% in a 1,024-pair cohort across three seeds and raises the reported three-seed means in all five VBench dimensions.

Primary area:
generative models
Keywords:
Diffusion Transformers, Video Generation, Spatiotemporal Lens

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8VWDOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose some modification of J-Lens for video models to predict how and where a small write in the residual stream will change the model output.

Critical strengths

None

Critical weaknesses
  • This paper is extremely hard to understand. I suspect it’s fully AI generated. If this is not true I urge the authors to make their writing significantly more clear. I’m happy to revisit the paper if they have majorly improved understanding and flow here.
  • The abstract is very hard to follow. What is a signed, coordinate-resolved response interface? What is signed about it? What is a frozen-path readout? What are held-out fixed-dictionary interventions?
  • The intro, too, is very hard to follow; e.g. “Existing readouts illuminate intermediate content, useful feature directions and attention structure. Editing additionally requires the signed effect of a specified change through the complete remaining network.” what does this mean? It is nowhere introduced what a signed effect means or what a readout means here?
  • L. 079: What is RCRE - it’s not introduced (quickly mentioned in the abstract but not to the point where I’d understand what it means). The authors should not link to Tables and Figures in the intro without explaining why they matter for the problem statement. I basically didn’t fully understand the research problem based on abstract and intro.
  • I really tried hard to understand the method itself but it suffers from the same problems. I think it’s some modification of the J-Lens. Sentences like “The image-editing comparisons use the four-channel epsilon field; the Wan comparisons use the conditional velocity field.” just confuse a reader. It is not introduced what is meant by “epsilon field”, “Wan comparison” or “conditional velocity field”?
Questions for authors

What is a “signed, coordinate-resolved response”? This keeps coming up but I don’t understand it.

Reviewer_HVLvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces ST-Lens, a diffusion transformer probe that estimates the response to a set of residual writes. ST-Lens outperforms direct-head and frozen-path readouts in observation fidelity. It decreases edit leakage compared to other methods, while attaining the same success rate and fewer complete online suffix evaluations. Response-coupled residual editing (RCRE) uses these responses to coordinate temporal writes. The authors claim that accounting for interactions between edits improves video editing accuracy. Finally, the responses observed using ST-Lens motivate two practical applications that improve generation quality: dynamic CFG schedule for images, and static / motion-advected sharing for videos.

Critical strengths

ST-Lens is a carefully elaborated mechanism: it accounts for an edit size budget, regions that should be preserved, and interactions between simultaneous edits. The mathematical derivation is sound. The paper provides ablations on several important axes, comparing controller invocation cost, ST-Lens accuracy and RCRE quality against various baselines.

Critical weaknesses
  1. The main choices are hard to trace. There is very little explanation of how the key hyperparameters (qq, DsD_s) should be chosen. While their meaning is given in the main text, the paper never walks through one concrete choice of either. In RCRE, the head-inverse that produces D\mathcal{D} is a major part of the method, yet its implementation is not explicit. Finally, section 5.4 explores two applications for generation – dynamic CFG for images and motion-advected / static sharing for videos. While the gains are evident, I do not see a direct connection between ST-Lens interface and the proposed techniques. Both already appear in the literature, and ST-Lens does not directly supply their hyperparameter values.
  2. Questionable gains. ST-Lens shows limited Success gains compared to other baselines (Table 2), and its strength seems to be in reduced Leakage, while requiring comparable or fewer Calls. If Success rate is so saturated on this task, could it be better to use a different Success metric or another, more difficult task? Also, there is little evidence that using joint RCRE adds practical benefit compared to Scalar (Table 3; Figures 10, 11). This RCRE visualization example (Figures 10, 11) does not disentangle motion from appearance: the reference already contains the desired ball, background and motion. A more compelling application would be attempting to transfer the reference's motion while preserving a different subject and appearance. Section 5.4 gains, in my opinion, are very loosely related to the proposed mechanism. Thus, I believe there is not enough evidence that ST-Lens / RCRE mechanisms offer significant practical value – more direct practical applications / better task selection could rectify this.
Questions for authors
  1. Hyperparameter clarification. In ST-Lens, how is the proposal dictionary D\mathcal{D} chosen? In 4.1, how is target vector qq chosen? In RCRE, how is D\mathcal{D} derived from the desired conditional velocity change rr?
  2. How exactly does ST-Lens / RCRE motivate using dynamic CFG schedule and static / motion-advected sharing? Is there a direct connection? Could we have arrived at the same techniques without ST-Lens / RCRE?
Reviewer_NrvGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how an edit to the residual stream of a diffusion transformer affects the model output, and how several edits interact. ST-Lens estimates this response from Jacobians averaged over calibration samples for each block and noise level, refreshes them with finite differences when needed, and keeps the spatial and temporal layout of the output. RCRE uses these responses to set edit strengths over time in Wan. Experiments cover response prediction, regional image control, motion commands in Wan, and two generation policies motivated by the analysis (a CFG schedule and shared initial noise across frames).

Critical strengths
  1. Keeping the sign and the spatial layout of the response is a good idea. Figure 4 shows regional effects that cancel in a pooled readout but are still predicted correctly, and in image control ST-Lens keeps edits inside the target region better than the other ways of choosing directions (Table 2).

  2. The calibrated directions are useful in practice. With six measured directions, ST-Lens attains all 792 image targets with about 18% fewer suffix calls than the eight-direction reference (Table 2), and similar call savings hold on new PixArt prompts with frozen calibration (Appendix K). This is why I think the idea is worth developing despite the weaknesses below.

Critical weaknesses
  1. The cached response prediction, which supplies the headline number in the abstract, seems to work well in only one of the four settings where relative error and control success are reported. The 37.15% gain in Table 1 comes from block 21, t = 510. In the other three settings, cached ST-Lens success drops below 70% while measuring at the current state stays above 90% (Tables 10 and 11). On Wan, the cached columns are worse than projected J-Lens (Table 14), so RCRE measures all of them again at each step. The evidence that cached response columns can replace current-state measurements across regimes is therefore thin, and the headline number should not be read as a general result (see Q1).

  2. The baselines for editing and generation are too weak. For editing, the Wan outputs follow a dense reference video (Figure 10), so the comparison with no write does not separate the method from access to the reference, and no outside method is compared for the Wan commands (Table 3). With the same reference, a write right before the output head seems to work as well as the block 28 write at lower cost (Table 15), and Joint RCRE has comparable decoded accuracy to the Scalar fit with one shared coefficient. For generation, the CFG schedule is compared only with constant CFG 4, not with interval guidance (Kynkäänniemi et al., NeurIPS 2024), and the authors themselves write that this diagnostic is distinct from ST-Lens’s residual-suffix Jacobian (Appendix AD). So it is not clear what the joint fit adds over a direct write or the Scalar fit, or what the schedule adds over tuned guidance.

Questions for authors
  1. Is there any setting other than block 21, t = 510 where the cached response can replace measurement at the current state? Relative error or success rate would be more informative than cosine here.

  2. On the Table 3 commands, how does RCRE compare with simple baselines that use the same reference and budget, such as the block 30 write in Table 15? In Figure 10 the edited outputs take on the reference’s appearance rather than the no-write output’s. Can RCRE change the motion while keeping the no-write video’s appearance?

  3. How does the CFG schedule compare with a tuned constant CFG and with interval guidance? How does advected sharing perform on the independent cohort?

Minor: several terms are used before they are defined (e.g., frozen-path, native-suffix, Full8, DSL).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.