acceptodds
Under review as a conference paper at ICLR 2027

Spectral Steering: Local Inverse Geometry for Frozen Diffusion Generators

Abstract

A frozen diffusion generator may need to change several output properties while preserving others. We study when guidance faithfully realizes such a structured request and whether local inverse correction improves its direction. Our account links this benefit to the interaction between the request and the objective-specific remaining-sampler Jacobian: conditioning alone is insufficient. Spectral steering combines a request-specific cosine-gain predictor with damped inverse correction and fixed-time relinearization for finite edits. Scalar nulls and matched-movement controls test the predicted mechanism. Across 8,923 instances in a retrospectively identified local regime, predicted and realized gains correlate at Spearman , versus for condition number. Experiments span images, audio, robotics, and control and planning, including complex character motion generation. On 50 paired inpainting cases, four corrections improve unobserved-pixel PSNR over Adam by 3.25 dB at equal total derivative traversals. Together, the analysis and controls connect request-dependent geometry to directional gains and establish where these gains translate into downstream performance.

Primary area:
generative models
Keywords:
Diffusion Models, Inference-Time Steering, Local Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_uk6bOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to realize structured requests through frozen diffusion generators, changing selected output properties while preserving others. The authors analyze the Jacobian of the requested properties through the complete remaining sampling process and show that first-order guidance can distort the relative components of a request. The method uses damped least-squares correction and repeatedly recomputes the Jacobian and residual at a fixed diffusion time to handle finite edits. The paper also introduces a request-specific score that predicts improvement in cosine alignment over first-order guidance. Experiments cover images, audio, robotics, character motion, and planning.

Critical strengths
  1. The problem is meaningful, and the mechanistic analysis is supported by carefully designed controls. Reducing the overall objective does not necessarily mean accurately realizing multiple requested changes. The paper usefully distinguishes these outcomes. The scalar equivalence test and the experiments matching latent displacement or predicted output movement help separate the benefits of update direction from those of intervention magnitude.

  2. Some experiments demonstrate substantial downstream benefits, and the limitations are discussed openly. In the paired inpainting experiment, four corrections improve missing-pixel PSNR over Adam by 3.25 dB at equal total derivative traversals. The paper also distinguishes request alignment, task outcomes, and executed behavior, and reports failures of local approximation, cases where generic optimizers perform better, and attenuation of improvements during execution. These analyses help clarify the scope of the method.

Critical weaknesses
  1. The significance of the contribution beyond prior work remains unclear. The core correction builds on classical damped least squares and is closely related to existing inverse-correction and constraint-projection methods [1–3]. While the request-specific analysis is useful, the paper needs to explain more clearly what substantial new insight or capability it adds.

  2. The practical value of the prediction score is not sufficiently established. The score predicts directional improvement within a local regime, but it remains unclear whether it can reliably guide steering decisions or anticipate better task outcomes. This limits the significance of one of the paper's central contributions.

  3. The practical advantage over strong optimizers needs clearer characterization. The experiments show useful gains on some tasks, while generic optimizers remain competitive on others. Given the additional computation required for geometric correction, the evidence does not yet clearly establish when this approach is preferable in practice.

References

[1] Weiguo Gao, Ming Li, and Qianxiao Li. Terminally constrained flow-based generative models from an optimal control perspective. arXiv:2601.09474, 2026.

[2] Utkarsh et al. Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints. NeurIPS, 2025. arXiv:2506.04171.

[3] Alexandros Graikos, Nebojsa Jojic, and Dimitris Samaras. Fast constrained sampling in pre-trained diffusion models. NeurIPS, 2025. arXiv:2410.18804.

Questions for authors
  1. How does the core update differ mathematically, and in its assumptions, from GNSolver in the TOCFlow paper [1]? Which differences concern damping, update timing, or implementation, and which constitute an independent methodological or analytical contribution? After accounting for PCFM and Fast Constrained Sampling, how would the authors characterize the main contribution of this work?

  2. Can the reliability of the prediction score be assessed without using post-edit outcomes? How does it perform across the complete diagnostic collection? Is there evidence that it helps select methods or correction depths and thereby improves final target error or task success?

  3. On a few representative tasks, could the authors compare spectral correction with well-tuned Adam and L-BFGS using consistent tuning and stopping rules, reporting end-to-end runtime and memory consumption alongside directional alignment, target error, and task success? This would help clarify the method's practical scope.

1 comment
AuthorsOpenReview ↗

Response to Reviewer uk6b

Thank you for recognizing the value of the work, the questions, and for the references. We clarify the solver relationship and test the predictor on fresh seeds.

Q1. Relation to GNSolver and constrained sampling. With M=JhDΦM=J_hD\Phi, TOCFlow's GNSolver returns M⊤(I+sMM⊤)−1hM^\top(I+sMM^\top)^{-1}h. Up to sign convention and an outer scalar factor, this has the same damped Gauss–Newton form as our J⊤(JJ⊤+λI)−1dJ^\top(JJ^\top+\lambda I)^{-1}d. We acknowledge this algebraic overlap. Relative damping, fixed-time relinearization and differentiation through the implemented remaining sampler are design choices, not a new Gauss–Newton update. TOCFlow, Eq. 148 and Algorithm 3.

PCFM projects using the constraint Jacobian at a predicted endpoint and maps back by interpolation. Fast constrained sampling develops an approximate Newton construction through the one-step denoiser. PCFM, Fast constrained sampling.

The independent contribution is request-specific prediction and controlled evaluation of directional benefit. Request coefficients in the Jacobian's singular basis determine distortion; conditioning alone is insufficient. This yields a scalar null, movement-matched tests and predictions evaluated against nonlinear edits. Holding solver and actual residual fixed, full-sampler differentiation improves K1 cosine on 75–95% of paired vision instances across tested windows. At eight remaining steps, predictor–gain correlation is 0.94 versus 0.68 for the one-step map on inpainting, and 0.80 versus 0.62 on SR. The one-step surrogate nevertheless reconstructs inpainting better and differentiates fewer sampler transitions. This isolates the map's effect without claiming superiority over the complete PCFM or fast-constrained-sampling algorithms.

Q2. Reliability without observing the evaluated outcome. We froze two robotics tasks, calibration seeds 20000–20009, evaluation seeds 20100–20129 and threshold G=0.05 before evaluation. All scales of a seed remain in one split. With no regime filter, the fresh-cohort result is:

TaskRequests / seedsSpearman, exact scoreFour-iteration score
Panda reach + orientation85 / 300.93 [0.83, 0.97]0.93
HalfCheetah target + posture90 / 300.82 [0.63, 0.92]0.82

Intervals bootstrap seeds. Five Panda requests were rejected during task construction. The outcome is K1-minus-FO cosine gain; FO here uses its one-iteration damped least-squares step, not movement matching. Both tasks meet the preregistered correlation ≥0.5 prediction.

The original unfiltered collection gives rho=0.626 over 10,362 instances versus −0.061 for log condition number; after removing population means, 0.599 versus −0.017. The earlier pilot split requires pilot-edit outcomes.

Decisions and overhead. We executed the router, charging its score computation on both branches and reusing the Krylov prefix. It saves only 0–6% of correction-to-trajectory CPU time, below the preregistered 15%; FO routing also fails the request-error noninferiority criterion. With only five or six request coordinates, four predictor iterations cost most of the full solve. Thus directional ranking transfers to these fresh instances, but we claim no routing speed-up. The predictor also does not establish a final-success or optimizer-selection rule. A separate residual-based discrepancy stop gives mean spectral PSNR within 0.02 dB of cross-fitted selection on all four vision operators.

Q3. Tuned optimizers and computational cost. Archived runs use common cross-fitted budgets and disjoint learning-rate tuning; grids and compute differ.

Task (n=50 except deblurring)Spectral PSNRAdam PSNR
Inpainting, missing pixels27.7525.98
4× SR27.9027.79
Deblurring, means only27.9427.92
Composite, target region28.1328.61

Deblurring reports means only, without paired inference; sample count and paired outputs are unavailable.

The paired inpainting gain is +1.77 dB [0.95, 2.67]; cross-fitted bandwidth LSD is 12.874 versus Adam’s 12.525 (paired Wilcoxon p=.15). New profiles cover all ten configurations on a RTX 5090. Medians use three images/task, one run each, including initialization and final sampling. K4 takes 85.3 s versus Adam@64's 49.3 s and DPS's approximately 62 s. With m=8, K4 takes 43.4 s; the separate 30-image quality ablation changes its PSNR by −0.03/+0.08 dB on inpainting/SR. The paired m=16 margin was not re-measured at m=8. Peak spectral allocation is 2.54–2.59 GiB versus Adam's 2.51, read before metrics. Small timing samples do not establish quality-matched frontiers. Full GPU/CPU results are in the appendix.

We have uploaded a revised manuscript incorporating these clarifications and additional experiments. We welcome further questions and would be happy to discuss any remaining concerns.

Reviewer_AiUZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies steering frozen diffusion generators through the local geometry of the map from an intermediate sampling state to the final task output. It replaces first-order gradient guidance with a regularized local inverse correction and uses a spectral analysis to explain how the two approaches respond to different output directions. Experiments cover image restoration and editing, audio, motion generation, and control.

Critical strengths

The evaluation spans several domains and includes both directional alignment and downstream task outcomes.

Critical weaknesses
  • The additional insight specific to diffusion models is unclear.

I am not sure what diffusion-specific insight the SVD analysis provides beyond standard local inverse optimization.

  • The practical benefit relative to computational cost is not sufficiently established

Differentiating through the full remaining sampling trajectory appears expensive. It is unclear whether the improvements justify this cost compared with DPS-style guidance.

Questions for authors
  • How does the runtime compare with DPS-style methods at comparable output quality?

  • What does the spectral analysis tell us specifically about diffusion models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AiUZ

Thank you for your time in reviewing our work and focusing on the diffusion-specific insight and computational value. We have sharpened the contribution statement and added a controlled experiment that directly tests the role of the remaining diffusion sampler.

Q2. What is specific to diffusion here? The SVD identities and damped least-squares update are general mathematics. The empirical question is whether the geometry of the remaining sampling trajectory can be replaced by the one-step denoised estimate when realizing a structured output request. We compare Jfull=∂h(Ft→0(xt))/∂xtJ_{\rm full}=\partial h(F_{t\to0}(x_t))/\partial x_t and Jone=∂h(x^0(xt))/∂xtJ_{\rm one}=\partial h(\hat x_0(x_t))/\partial x_t. Both arms use the same damped solver, fixed target and residual measured through the actual remaining sampler; only the differentiated map changes. This isolates a sampler-specific design choice.

On inpainting and 4× SR, with 20 paired instances per task/window:

Eight remaining DDIM stepsInpaintingSR
K1 cosine gain, full minus one-step+0.012+0.036
Fraction with higher full-map cosine90%95%
K1 PSNR difference−1.18 dB+0.31 dB
Predictor–gain Spearman, full / one-step0.94 / 0.680.80 / 0.62

Across 4/8/12 remaining steps, median agreement between the latent corrections decreases from 0.956 to 0.898 for inpainting and from 0.918 to 0.869 for SR. Audio provides a larger directional effect: +0.110 cosine early in sampling (+0.264 including preservation), versus +0.002 late, on 50 paired instances.

These results show that the remaining trajectory changes the geometry relevant to request alignment, especially earlier in the tested samplers. They also delimit its practical value: the one-step surrogate reconstructs missing pixels better and differentiates fewer sampler transitions per correction. Higher alignment need not mean higher reconstruction PSNR. The full-map predictor's stronger correlation concerns the gain of its own correction, rather than superiority over every alternative algorithm.

The analytical contribution is a request-specific explanation and testable prediction of this distortion. The same Jacobian can preserve one request's direction and distort another's, so condition number alone is insufficient. The scalar null and matched latent/output movement controls distinguish the directional effect from step magnitude. Fresh, unfiltered robotics tests now give predictor–gain correlations of 0.93 and 0.82 on 30 evaluation seeds per task, with the protocol fixed before evaluation; this new test uses an implementable one-iteration least-squares FO step. We present the work as an analysis and controlled evaluation of steering through frozen samplers, with an explicitly classical inverse solver.

Q1. Runtime relative to DPS at comparable quality. We now report measured operating points, separating quality cohorts from timing samples. All ten configurations were profiled on one RTX 5090 after a 60 s idle gate; 5 Hz telemetry detected no competing GPU activity during accepted timed intervals. Times include initialization, correction and final sampling, with loading and metrics excluded. We use the first valid attempt. The cooperative reservation is partial, and the timing cohort is three images/task with one run each, so these are descriptive profiles.

SR operating pointPSNR (n=50)Online time (n=3), s
Spectral K127.3921.7
Spectral K427.7785.3
DPS, T=100025.9962.0

SR K1 exceeds DPS's PSNR at 0.35× its measured time. This PSNR comparison does not imply an LPIPS advantage. On inpainting, DPS remains stronger than m=16 K4: 28.76 versus 28.32 dB on DPS's 50-image cohort, and 61.9 versus 85.3 s in the timing profile.

The new budget experiment provides a practical response to the cost concern: m=8 reduces K4 to 43.4 s, below both DPS and Adam@64 (49.3 s). Separately, the 30-image paired ablation changes K4 PSNR by −0.03/+0.08 dB on inpainting/SR. This establishes near-retention of K4 quality at a lower Krylov budget, not a new paired quality advantage over DPS or Adam. Full-map peak allocation is 2.54–2.59 GiB versus DPS's 2.50–2.51. One-step K2 takes 5.1 s versus full K2's 42.8 s, with the alignment/quality trade-off above. The appendix includes all ten configurations and operation counts. We make no claim of universal dominance or a quality-matched speed frontier.

We have uploaded a revised paper incorporating these clarifications. We welcome further questions and would be happy to discuss any remaining concerns.

We hope these experiments clarify the diffusion-specific contribution and its computational trade-offs. Could you clarify which specific issue in the derivation, assumptions, or experimental methodology underlies the soundness of 1? We would appreciate the opportunity to address any remaining concern and ask you to reconsider the assessment in light of this evidence.

Thank you for the additional experiments.

My concern about soundness is with the problem formulation. I understand the derivation once the paper assumes that steering is performed by differentiating through the entire remaining trajectory. What I still do not understand is why the structured editing task motivates that assumption. The example of moving a wrist while keeping the feet fixed explains why both properties matter, but it leaves open why one would use end-to-end optimization for this task when DPS-style guidance can apply the same objective at each denoising step. The one-step ablation clarifies the effect of the Jacobian choice within the proposed solver. My original question concerns the earlier decision to formulate steering as end-to-end optimization in the first place.

AuthorsOpenReview ↗

On structured editing and per-step guidance

Thank you for engaging with our response and for stating the remaining concern so precisely. It is the central question, and we answer it with direct comparisons against per-step guidance.

Short answer. Per-step guidance and local inversion are good at different things, and we measured both on the same instances. Guidance is a strong constraint satisfier. The inverse step through the remaining sampler is the better realizer of a prescribed change, and a prescribed change is what structured editing asks for.

Where this comes from. The idea of spectral steering came from the control of frozen generators in downstream applications, such as robotics and control, or request-specific multimedia editing. A generator has produced a sample, and a user must change one thing in it by a stated amount while the rest stays as generated. In a trajectory, the end point moves by a few centimetres while posture and style stay. In a character motion, a wrist reaches a target while the motion stays natural. In an image or an audio clip, one region or attribute changes. The principle: change only what the request asks, and only as much as needed.

Why this leads to the remaining sampler.

  • The model stays the prior. We move one intermediate state as little as possible and leave the pretrained sampler untouched. Per-step guidance alters the dynamics at every step.
  • The edit is verified on the real output. Each correction ends with the actual output and its measured residual. DPS applies the objective to the posterior mean x^0(xt)\hat x_0(x_t), so the error on the real output is known only when sampling ends.
  • A vector request keeps its proportions. DPS is the first-order instance of Section 3 with the one-step map: each step applies Jone⊤rJ_{\rm one}^\top r, which weights the request's components by σi2\sigma_i^2. The inverse step removes this weighting.
  • It is cheap where it is needed. Structured requests are low-dimensional, and an explicit Jacobian costs one backward pass per request coordinate. A complete trajectory edit takes 0.7–1.4 s on CPU.

Head-to-head with per-step guidance. The baseline is guidance on the one-step estimate, the DPS mechanism, in its Universal Guidance form with tuned scale, applied from the same intermediate state to the same objective.

  • Trajectory pose editing (6 request coordinates, 79 instances): one inverse step halves the request error of tuned guidance (0.143 vs 0.287), with 6 backward passes against 16. At matched movement, guidance wins only 8–19% of pairs.
  • Executed planning (263 and 315 runs): the inverse step realizes the requested goal change better in 16 of 20 guidance settings (error 0.11 against 0.31–0.40 on the dense task).
  • Audio, matched movement: guidance loses all 360 pairs (error 1.54–1.59 dB against 0.45–0.68).
  • Vision, four operators, 50 images each:
    • Guidance never exceeds our best setting. It is behind on SR (0.13 dB, p<10−7p<10^{-7}) and deblurring (0.14 dB, p=0.002p=0.002).
    • Native DPS (T=1000) is 1.4 dB behind one correction on SR, at about three times the time.
  • Smaller interventions. Where guidance reaches its targets it moves the sample much further: 6–33× in audio, 5× on inpainting and about 12× on dense planning.
  • Stability. No spectral setting ends below the unguided output on any image. In settings that a residual-based selector would choose, guidance does so on 40 of 50 deblurring images and on all 50 composite images, where it reaches a lower measurement residual than ours with 6 dB lower PSNR. The measurement alone therefore cannot select its depth. Our residual-based stop lands within 0.02 dB of the best depth.

What is new. The singular-value account explains these outcomes, and it predicts before the edit how much the inverse step gains over the gradient direction: predicted and realized gains correlate at Spearman 0.93 and 0.82 on fresh seeds with no filtering. We are not aware of a comparable predictor for guidance or for end-to-end optimization.

Concurrent work posted after our submission independently targets similar minimal-intervention formulations and reports better constraint satisfaction, FID and PSNR than DPS on inpainting, SR and deblurring (MintFlow, arXiv:2610.02260). The revised paper states this motivation in the introduction and Section 3, with DPS as the first-order, one-step instance, and will add the head-to-head tables to the appendix. If this resolves the formulation concern, we would be grateful if you would reconsider your assessment and scores in light of these results. We remain happy to answer any further questions or address any remaining concern during the discussion.

AuthorsOpenReview ↗

Summary of the discussion

As the discussion period is closing, we briefly summarize where your concern stands.

You asked why structured editing is formulated through the remaining sampler when DPS-style guidance can apply the same objective at each step. Our reply above answers with direct comparisons on the same instances: per-step guidance is the stronger constraint satisfier baseline, and the inverse step realizes a prescribed change more faithfully. The revised paper now states this motivation in the introduction and Section 3, and a new appendix section reports the full comparison, including where each is better.

One question, so that we address your concern exactly: is there a setting in which you would expect DPS-style guidance to realize a prescribed multi-coordinate change as faithfully as the inverse step? If so, we would be glad to run it. Even a brief indication of whether the formulation concern is now resolved, and if not why, would help us and the Area Chair.

Reviewer_bMLiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why conventional gradient guidance struggles to faithfully realize fine-grained, multi-attribute changes in frozen diffusion generators. The authors identify that different output attributes respond unequally to latent perturbations, leading to distortions in the requested changes. To address this issue, they propose Spectral Steering, which uses Jacobian-based damped inverse correction to better align the resulting changes with the desired objectives. They also introduce a request-specific predictor to estimate when Spectral Steering is expected to provide greater directional improvements over conventional gradient guidance. Experiments across image restoration, audio, robotics, and planning tasks demonstrate improvements in request fidelity and downstream task performance in various settings.

Critical strengths
  1. Insightful Problem Diagnosis and Analysis: The paper provides a clear and insightful analysis of why conventional gradient guidance may fail to faithfully realize fine-grained, multi-attribute requests. Rather than simply demonstrating the limitations of existing methods, the authors explain how unequal responses across the Jacobian's singular modes lead to directional distortion and motivate the proposed correction method.
  2. Extensive Experimental Validation: The paper presents a broad set of experiments across image restoration, audio, robotics, and planning tasks. The inclusion of controlled experiments, such as the scalar null and movement-matched comparisons. These experiments helped validate the proposed analysis beyond simple performance improvements. The evaluation of downstream execution outcomes in robotics further strengthens the empirical evidence.
  3. Meaningful Comparisons with Existing Optimizers: The authors compare Spectral Steering with standard optimization methods, including SGD, Adam, and L-BFGS, rather than relying solely on comparisons with diffusion guidance methods. The compute-matched inpainting experiments provided useful evidence of the effectiveness of iterative spectral correction, making the empirical evaluation more convincing.
Critical weaknesses
  1. Fairness of Hyperparameter and Optimization Budget Selection: The main results report task-specific best-K configurations for Spectral Steering, while the baselines use different hyperparameter-selection procedures and optimization budgets. Although task-specific tuning is reasonable, it is unclear whether all methods receive comparable optimization opportunities. In particular, Appendix G shows that Adam achieves a bandwidth-extension error of 12.525 at 2 steps, compared with 12.730 for Spectral K4, whereas Table 1 reports a substantially worse Adam result of 26.604 at 16 steps. This indicates that the relative performance is sensitive to the choice of optimization budget, potentially making some of the main comparisons appear more favorable to Spectral Steering.

  2. Limited End-to-End Computational Efficiency Evaluation: The paper provides useful comparisons based on VJP/JVP counts and network evaluations, including compute-matched experiments against Adam. However, a comprehensive comparison of end-to-end wall-clock latency and peak GPU memory usage is missing. Since Spectral Steering requires computationally expensive Jacobian-based inverse corrections, the practical efficiency of the proposed method relative to existing optimizers remains unclear.

  3. Limited Demonstration of the Practical Utility of the Request-Specific Predictor: The proposed predictor is an interesting component of the paper, but its practical utility beyond diagnostic analysis remains unclear. The reported correlation is based on a retrospectively identified local regime, and the predictor estimates directional cosine improvement rather than downstream task success or generation quality. Furthermore, computing the predictor requires Jacobian-related operations, which may introduce substantial computational overhead. It remains unclear whether the predictor can reliably and efficiently guide the selection between first-order guidance and Spectral Steering in practice.

Questions for authors
  1. Fairness of Baseline Comparisons: Could the authors further clarify and justify the criteria used to select the task-specific configurations and optimization budgets for all methods, including SGD, Adam, L-BFGS, and Spectral Steering? In particular, why was Adam evaluated at 16 steps for bandwidth extension in Table 1, despite its substantially better performance at 2 steps reported in Appendix G? Could the authors provide additional sensitivity analyses or comparisons under a consistent configuration-selection protocol to demonstrate that the reported performance advantages are robust to these choices?

  2. End-to-End Computational Cost: Could the authors report the end-to-end wall-clock runtime for each method and task directly in Table 1, alongside the existing performance metrics, following the format of Table 1 in CLAMP (Shin et al., 2026)? Additionally, could the authors provide peak GPU memory usage under the same hardware and experimental settings? This would help clarify the performance–computation trade-off of Spectral Steering compared with existing methods.

  3. Practical Utility of the Request-Specific Predictor: Could the authors extend the use of the proposed predictor beyond diagnosing directional improvements? In particular, could it be used to develop an adaptive hybrid strategy that selects between first-order guidance and Spectral Steering based on the predicted directional gain? If so, how would the computational overhead of the predictor affect the overall efficiency of such a strategy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bMLi

Thank you for recognizing the value of the work, for identifying the budget-selection inconsistency and motivating the cost and policy tests.

Q1. Adam at two versus sixteen steps. The submitted comparison mixed budget rules. Adam reaches 12.525 at two steps versus 26.604 at sixteen. The appendix compares all methods with budget curves under the same cross-fitted rule.

For the vision/audio runs, each method uses one two-fold selection rule: select the budget with the best mean endpoint on one fold, evaluate on the other, then swap. Learning rates retain their disjoint tuning-set selections. This equalizes budget selection; original search grids and compute costs remain different.

TaskSpectralAdamSpectral minus Adam
Inpainting, missing PSNR (n=50)27.7525.98+1.77 dB [0.95, 2.67]
4× SR, PSNR (n=50)27.9027.79+0.11 dB [0.08, 0.15]
Deblurring, mean PSNR27.9427.92+0.02 dB
Composite, region PSNR (n=50)28.1328.61−0.47 dB
Bandwidth extension, LSD ↓ (n=25)12.87412.525+0.349 (Wilcoxon p=.15)

The reported unsteered bandwidth LSD is 12.757. On 20 additional paired inpainting images, Adam's 64/128/256-step results are 25.30/24.74/23.85 dB against K4's 27.12. Increasing that optimizer's step budget alone does not remove the inpainting gap.

A separate ground-truth-free discrepancy stop uses the known measurement-noise level. Spectral mean PSNR is within 0.02 dB of cross-fitted selection on all four vision operators. It need not select the same depth: on SR it selects K2 on 27/50 images and deeper rungs on the rest. This is a residual-based rule, separate from the directional predictor.

Q2. Runtime and memory, with load monitoring. The new profiling identifies a useful lower-cost setting: K4 with m=8 takes 43.4 s, 12% less than Adam@64 and about half of K4 at m=16. In the separate 30-image ablation, m=8 changes K4 PSNR by only −0.03/+0.08 dB on inpainting/SR. This is not a paired m=8-versus-Adam quality estimate.

All ten configurations were profiled on one RTX 5090 (fp32, batch 1), after 60 s of idle GPU per run. A cooperative lock and 5 Hz telemetry were used; no competing GPU activity was detected in accepted timed intervals. The partial reservation limits exclusivity evidence to the monitoring resolution. Timers include initialization, solve and final sampling from a loaded model and observed input; loading, scheduler setup and metrics are excluded. Peak allocation is read before metrics. We use the first valid attempt, never the fastest, and exclude earlier shared-load timings.

ConfigurationInpaint / SR sPeak GiB
Spectral K121.6 / 21.72.56–2.58
Spectral K242.8 / 42.82.56–2.59
Spectral K485.3 / 85.32.57–2.59
Spectral K4, m=843.4 / 43.42.54–2.55
Adam@1612.5 / 12.52.51
Adam@6449.3 / 49.32.51
L-BFGS@6498.5 / 98.52.53
DPS, T=100061.9 / 62.02.50–2.51
One-step K25.1 / 5.13.22–3.24
One-step K819.3 / 19.33.22–3.25

Medians use three images/task, one run each; this small profile does not estimate repeat variability. L-BFGS@64 executes 128 VJP closures. Registered-operation counts also fall: 660 versus Adam@64's 768 TFLOP (appendix). These counts do not measure latency.

Together with the 50-image quality results, these profiles show task-dependent trade-offs. K4 at m=16 costs 1.73× Adam@64 for its +1.77 dB inpainting gain; SR K2 costs 3.44× Adam@16 for +0.11 dB. DPS is faster than m=16 K4 and stronger on its inpainting cohort, while SR K1 exceeds DPS's PSNR at 0.35× its time. These are fixed operating points, not quality-matched frontiers.

Q3. Executed adaptive hybrid, including score overhead. We froze the threshold and evaluated on 30 fresh seeds each for Panda and HalfCheetah, disjoint from calibration. Without regime filtering, predicted versus realized directional gain has Spearman 0.93 [0.83, 0.97] and 0.82 [0.63, 0.92] (85/90 requests; seed-bootstrap intervals). FO here uses its one-iteration damped least-squares step.

The policies pay for the score on both branches and either return the four-iteration inverse/FO step or continue the same Krylov basis. Median correction-to-trajectory time falls only 0–6%, below the preregistered 15%; FO routing also fails the request-error noninferiority criterion. With five or six request coordinates, computing the score already costs most of the solve. We therefore claim reliable fresh-instance directional prediction on these tasks, but no operational speed-up from routing. Final task success and optimizer selection remain distinct questions.

We have uploaded a revised manuscript incorporating these clarifications. We welcome further questions and would be happy to discuss any remaining concerns.

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best,

AC

Reviewer_dEcmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Spectral Steering, a method for satisfying multi-attribute requests in frozen diffusion generators by applying a damped local inverse of the Jacobian of the remaining sampling process. The method is evaluated across a range of downstream tasks spanning vision, audio, robotic control and planning, and complex character motion generation.

Critical strengths
  1. The paper provides an in-depth theoretical analysis of how Jacobian geometry distorts structured requests under gradient guidance, offering valuable insights into the underlying mechanism.
  2. The extensive experiments demonstrate the method’s broad applicability across diverse tasks and domains.
  3. Meaningful comparisons, statistical analyses, and an examination of failure modes strengthen the credibility of the empirical findings.
Critical weaknesses
  1. The core method relies on classical damped least squares and repeated linearization. Its novelty relative to existing optimization and diffusion guidance methods, such as D-Flow and OC-Flow, therefore requires clearer articulation.
  2. The predictor is evaluated on retrospectively selected instances, leaving its reliability in unseen nonlinear settings insufficiently established.
  3. Computational comparisons use inconsistent budgets across tasks, making it difficult to assess the method’s practical efficiency.
  4. Performance gains are inconsistent across tasks, and conventional optimizers achieve better results in several experiments. The authors should clarify the conditions under which the proposed method is effective and where its advantages diminish.
Questions for authors
  1. How does reducing the Golub–Kahan iteration budget from 16 to 4 or 8 affect prediction accuracy and guidance performance?
  2. How does the method perform on infeasible requests where the desired displacement lies partially outside the subspace reachable through the Jacobian?
1 comment
AuthorsOpenReview ↗

Response to Reviewer dEcm

Thank you for recognizing the value of the work, the analysis, breadth and controls. Your questions motivated direct tests of reduced Krylov budgets and unreachable requests, now completed.

Novelty relative to D-Flow and OC-Flow. D-Flow differentiates through generation to optimize the source state; OC-Flow optimizes controls along the flow. Our contribution is the request-specific analysis of the remaining-sampler map: for d=∑iaiui+d⊥d=\sum_i a_i u_i+d_\perp, gradient guidance produces gains proportional to σi2ai\sigma_i^2a_i, whereas damped inversion produces σi2ai/(σi2+λ)\sigma_i^2a_i/(\sigma_i^2+\lambda). The modal coefficients matter as well as the spectrum. The predictor, scalar null and movement-matched controls test when this directional correction survives nonlinear sampling. We acknowledge the classical inverse step and its algebraic overlap with TOCFlow's GNSolver. D-Flow, OC-Flow, TOCFlow.

Reliability beyond retrospective regime selection. We froze a fresh robotics test before evaluation: 10 calibration and 30 disjoint evaluation seeds per task, with no regime filter. Spearman correlation between predicted and realized K1-over-FO cosine gain is 0.93 [0.83, 0.97] on Panda and 0.82 [0.63, 0.92] on HalfCheetah, over 85/90 requests; intervals bootstrap seeds. Here FO uses its one-iteration damped least-squares step, rather than movement matching. This complements the original unfiltered collection: rho=0.626 on 10,362 instances, versus −0.061 for condition number. The predictor forecasts directional gain, not final task success; the executed router saves only 0–6% of time, below its 15% criterion.

Budget fairness and scope. With the same cross-fitted budget-selection rule, inpainting improves over Adam by 1.77 dB [0.95, 2.67] and SR by 0.11 dB [0.08, 0.15]. Adam is comparable on deblurring and better on composite restoration. A separate paired 20-image extension gives Adam 25.30/24.74/23.85 dB at 64/128/256 steps versus K4's 27.12. These findings concern the tested grids and budgets, not universal optimizer superiority.

Q1. Reducing Golub–Kahan iterations from 16 to 4 or 8. We evaluated 30 paired images per task, using deterministic prefixes for K1 and separate relinearized K4 runs.

K1 resultm=4m=8m=16
Inpainting missing-pixel PSNR22.4723.0323.11
SR PSNR27.2327.4927.54
Inpainting: predictor–gain Spearman0.9560.9600.960
SR: predictor–gain Spearman0.6650.7220.734

Eight iterations retain 98.6%/96.4% of the K1 PSNR improvement from m=1 to m=16. For K4, m=8 changes PSNR by −0.03/+0.08 dB. In separate profiling (three images/task, one run each; no competing GPU activity detected by 5 Hz telemetry), m=8 reduces K4 from 85.3 to 43.4 s on both tasks, below Adam@64's 49.3 s. Together with the paired quality ablation, this identifies a useful lower-cost setting; m=4 has a measurable SR prediction trade-off.

Q2. Partially unreachable requests: two distinct tests. At a fixed Jacobian, J⊤d⊥=0J^\top d_\perp=0: both the gradient and the exact damped inverse ignore an unreachable component.

Pure contradiction, reachable target fixed. We duplicated one quarter of observed pixels and assigned paired targets ±e/2\pm e/2 around the consistent value, with 10% or 30% unreachable request energy. On five images, J⊤u=0J^\top u=0 in floating point. Every tested K1 prefix (m=1,2,4,8,16), K4 ladder and Adam@16/64 has a relative output L2 change below 0.15% against the consistent request; |ΔPSNR| <0.003 dB. This supports the predicted invariance for both methods.

Conflicting measurements, reachable target also shifted. All 16 planned instances are complete:

Infeasible energy fraction00.10.3
K4 missing-pixel PSNR27.8327.0424.69
Adam@64 missing-pixel PSNR25.0623.8818.67
K4 / Adam@64 latent displacement87 / 20890 / 23499 / 365

The paired K4 advantage is 2.76 [1.06, 4.63], 3.16 [1.82, 4.69] and 6.02 [5.29, 6.75] dB. At fraction 0.3, K4 wins all 16 pairs and its displacement grows only 1.13×, versus Adam's 1.75×. Both degrade: K4 loses 3.13 dB versus Adam's 6.39. The preregistered <1 dB degradation and ≥0.9 reachable-cosine expectations fail (cosine 0.754); the 1.5× displacement bound holds. These results support relative robustness to conflicting measurements, while the fixed-target control isolates the nullspace prediction. Neither establishes nonlinear convergence.

We have uploaded a revised manuscript incorporating these clarifications and additional experiments. We welcome further questions and would be happy to discuss any remaining concerns.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their thoughtful feedback. We address the central concerns with fresh-seed validation, fairer budget selection, measured costs, and additional geometric controls.

Contribution and relation to prior work. Our contribution is a request-specific, testable account of directional distortion through frozen samplers, not a new inverse solver. Steering structured requests requires realizing desired output changes while preserving other properties, yet gradient guidance can distort their relative components. Under the same Jacobian, a single-mode request preserves its direction, while mixed-mode requests can be distorted. Thus, conditioning alone cannot predict when inverse correction improves fidelity; the request's singular-mode coefficients also matter. We turn this insight into a pre-edit predictor validated against nonlinear edits, with scalar-null and movement-matched controls distinguishing directional benefit from step magnitude. Damped least squares is classical and overlaps algebraically with TOCFlow's GNSolver; our contribution is explaining and predicting when and why geometric correction improves the directional fidelity of structured steering requests.

New prospective evidence. With tasks, disjoint seeds and routing threshold fixed before evaluation, the predictor achieves Spearman 0.93 [0.83, 0.97] on Panda and 0.82 [0.63, 0.92] on HalfCheetah (30 fresh seeds/task; 85/90 requests; no regime filtering). FO uses a deployable one-iteration damped least-squares step. Across the original unfiltered 10,362 instances, correlation is 0.626 versus −0.061 for log condition number. The executed router saves only 0–6%, below the preregistered 15% criterion. Prediction transfers to fresh directional outcomes, but does not establish faster routing or final task success.

Comparable configuration selection. Under common cross-fitted budget selection and disjoint learning-rate tuning, spectral steering improves inpainting PSNR over Adam by 1.77 dB [0.95, 2.67] (50 paired images). SR improves by 0.11 dB, while Adam leads composite restoration by 0.47 dB. Bandwidth LSD is 12.874 versus Adam's 12.525 (n=25; Wilcoxon p=.15). Deblurring means are 27.94/27.92 dB, without paired inference or common-selection claims. Budget selection does not equalize tuning effort or compute. On 45 paired executed waypoint cases, K4 reduces Adam's mean error from 7.48 to 3.05 cm, winning 41/45 pairs.

What sampler geometry buys. Holding the damped solver and actual residual fixed, full-sampler differentiation improves K1 cosine on 75–95% of paired vision instances across tested windows. Early audio gains reach +0.110 cosine (+0.264 including preservation). However, the one-step surrogate reconstructs inpainting better and differentiates fewer sampler transitions. Directional fidelity and downstream quality are distinct.

A lower-cost setting. Reducing Krylov iterations from 16 to 8 cuts K4 runtime from 85.3 to 43.4 s, below Adam@64 (49.3 s) and DPS (~62 s). A separate 30-image ablation changes K4 PSNR by only −0.03/+0.08 dB on inpainting/SR, without establishing a new paired advantage over Adam. All ten configurations were profiled on one RTX 5090, with a 60 s idle gate and no competing activity detected at 5 Hz. Timings include initialization and final sampling (three images/task, one run each). Peak spectral allocation is 2.54–2.59 GiB versus Adam's 2.51. At m=16, K4 costs 1.73× Adam@64; DPS remains stronger on its inpainting cohort, while SR K1 exceeds DPS's PSNR at 0.35× its measured time. These small profiles do not establish quality-matched frontiers.

Infeasibility controls. Across 16 conflicting-measurement images, K4 exceeds Adam@64 by 2.76, 3.16 and 6.02 dB as contradiction increases, although both degrade. In a separate five-image fixed-reachable-target control, adding purely unreachable components changes outputs by less than 0.15% in relative L2 and PSNR by less than 0.003 dB. This supports the predicted nullspace invariance without implying nonlinear convergence.

These additions clarify the independent analytical contribution and its task-dependent practical value.

Thank you for the careful review, for recognizing the value of our work, and for raising your score. Your questions led us to clarify the relation to GNSolver and to test the predictor on fresh seeds, and both made the paper better. We remain available for any further question or clarification.

AuthorsOpenReview ↗

On generality, cost and practical value, with new results

Thank you for reading our rebuttal carefully. Your reply names three remaining points. For each, we recall what our first response showed, what you found missing, and what we add now: two new results.

1. "Advantages over existing optimizers remain inconsistent across other evaluated tasks." Our first response gave the common-selection table: +1.77 dB on inpainting, +0.11 dB on SR, and within 0.5 dB of Adam on deblurring and composite restoration. We agree that this margin varies by task, and you note the meaningful gains on inpainting and control; the same holds for motion generation. Two facts put it in context.

  • What does not vary. The paper's three core results are consistent wherever we tested them. At matched movement, one inverse step realizes the request better than the gradient step on 100% of inpainting images, 93% of Panda requests, 85% of audio excerpts and every character-motion pair. The score correlates positively with the realized gain in 29 of 30 populations, and at 0.93 and 0.82 on fresh seeds. No spectral setting ends below the unguided output on any restoration image.
  • Where the margin is largest. Spectral steering is strongest where a goal and preserved properties must be met together: fresh-seed control at comparable time budgets (request error 0.028 against 0.17–0.63 for Adam and L-BFGS; Panda success 67% against 9–53%), executed waypoints (3.05 cm against 7.48 cm for Adam), character motion (1.05 cm bone drift against 5.66–7.20 cm) and inpainting with conflicting measurements (3.2 to 6.0 dB over Adam).

The method proved stable in a broad stress test: one update with one fixed damping coefficient across four domains, against domain-specific baselines, with every outcome reported. The optimizer baselines are strong by design: the end-to-end optimization of DNO and D-Flow, adapted to the same state and sampler with tuned learning rates. We included the standard image benchmarks for generality and fair comparison, and we were positively surprised that a method designed for control and animation is competitive with cv specialized baselines.

2. "The standard K4 configuration is approximately 73% slower than Adam@64, and the faster configuration lacks a direct paired quality comparison against Adam." Both statements are correct for our first response. The 73% is K4 with 16 Krylov iterations (85.3 s against 49.3 s). The same method with 8 iterations took 43.4 s, but we had compared its quality only with the 16-iteration setting. We have now run the comparison you asked for, on 30 images per task with identical inputs and warm starts.

30 paired imagesSpectral K4, m=8Adam@64Difference [95% CI]
Inpainting, missing-pixel PSNR28.3425.35+2.99 dB [1.59, 4.53]
4× SR, PSNR27.9726.36+1.61 dB [1.31, 1.93]
Time (3 images) / TFLOP43.4 s / 66049.3 s / 76812% less time

With 8 iterations, spectral steering is more accurate than Adam@64 at lower cost on both tasks (on 22 and 30 of the 30 images), and it keeps the full margin of 16 iterations on these images. On inpainting, 64 steps is Adam's best budget by PSNR. On SR, against Adam's best budget (16 steps), PSNR ties (27.97 vs 27.92) and Spectral's LPIPS is better (0.132 vs 0.153).

3. "Its practical utility remains limited, as the executed routing achieves only 0–6% runtime savings." Our first response showed that the predictor ranks directional gains on fresh seeds, which you acknowledge, and that routing saves little time. Routing is only one use: with five or six request coordinates the score costs most of the solve, so there was little time to save. We now test a different use: does the score say, before the edit, where the inverse step changes the result? On the same seeds, we compare spectral with a FO-only policy under the same stop rule, split at the threshold frozen before evaluation.

Panda requestsShareSuccess, FO/spectralExecuted error, FO/spectral
Score below threshold33%64% / 64%0.71 / 0.71 cm
Score above threshold67%37% / 68%1.49 / 0.77 cm

Above the threshold, spectral steering nearly doubles success (+32 points, 95% interval [18, 46]). Below it, first-order steering reaches the same success and executed error. On HalfCheetah the executed error follows the same pattern (a gain of 2.7 cm [0.4, 5.4] above the threshold), but success is not separated.

What has changed since our first response. Our faster setting actually beats Adam@64 in a direct paired comparison at lower cost, and the predictor now anticipates task outcomes, not only direction. Both results are in the revised paper. The supplementary videos and audio show these edits in practice. We would be glad to run any further comparison you consider decisive. If this addresses your concerns about the performance–computation trade-off and practical value, we would be grateful if you would reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.