acceptodds
Under review as a conference paper at ICLR 2027

Annealed Backward–Backward Splitting for Training-free Diffusion Guidance

Abstract

Training-free guidance specifies a condition for a pretrained diffusion model via a differentiable loss without additional training, and is therefore applicable across tasks and domains. Most existing methods add to each reverse step a gradient step on the loss, whose step size controls how far the guided iterate moves. Choosing this step size is difficult. The standard rule sets it from the smoothness of the loss, but this smoothness is often unavailable (e.g., when the loss involves a pretrained network), so the step size must be tuned separately for each task. In this paper, we remove the step size from the design. We formulate conditional generation as a sequence of subproblems indexed by the noise level, and we address each subproblem by composing two proximal steps, one for the prior and one for the condition, so that each subproblem (rather than step size) determines how far the guided iterate moves. In training-free guidance, we cannot compute either proximal step in closed form, because the prior is available only through the pretrained diffusion model and the loss may involve a pretrained network. We therefore approximate the proximal step for the prior by one reverse step, whose scale we identify from the noise schedule with a bound on its approximation error, and the proximal step for the condition by an inexact proximal step. We refer to the resulting method as ABBS (annealed backward-backward splitting). In our experiments, ABBS achieves the best guidance validity on three of four image tasks, in data space and latent space, with fidelity comparable to training-free guidance baselines. ABBS also applies to human motion editing, where it achieves editing performance on par with the strongest motion-editing baseline at a substantially lower guidance cost.

Primary area:
generative models
Keywords:
diffusion models, training-free guidance, inverse problems, proximal splitting, controllable motion generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_N8NzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes ABBS, which treats diffusion guidance as a series of optimization problems at different noise levels. It combines approximate proximal steps for the prior and target conditions, mainly to avoid explicit step-size selection. The authors also attempt to derive an error bound and test the method on image guidance and motion editing.

Critical strengths
  1. The proposed guidance framework can be applied to image restoration, style guidance, and motion editing across different pretrained models (Tab 2-3).
Critical weaknesses
  1. One of my major concerns is that ABBS is very close to DiffPIR (cited in line 76, only once). Using the same notation and squared measurement loss, ABBS's Eq. 8-12b with the weight from Sec 3.2 gives

uA≈arg⁡min⁡u[∣y−Au∣2+λx2ctσˉt2∣u−x^0∣2].u_A \approx \arg\min_u \left[\\|y-Au\\|^2 + \frac{\lambda_x}{2c_t\bar{\sigma}_t^2}\\|u-\hat{x}_0\\|^2\right].

DiffPIR's preprint Eq. 12b gives

uD=arg⁡min⁡u[∣y−Au∣2+λyσn2σˉt2∣u−x^0∣2].u_D = \arg\min_u \left[\\|y-Au\\|^2 + \frac{\lambda_y\sigma_n^2}{\bar{\sigma}_t^2}\\|u-\hat{x}_0\\|^2\right].

Setting λx=2λyσn2\lambda_x=2\lambda_y\sigma_n^2 makes the overlap clear. The remaining difference between these subproblems is ABBS's factor 1/ct1/c_t. Without a DiffPIR baseline and separate tests of these changes, benefit of ABBS is very unclear.

  1. Another major concern is the claim in Sec 4.1, stating λ\lambda not as a step size. It is a tradeoff weight in Eq. 9, but it also controls the update size. For an exact solution of Eq. 12b, substituting ρt=λ/νt\rho_t=\lambda/\nu_t from Sec 3.2 gives

xˉt−1=vˉt−1−νtλ∇ft(xˉt−1)\bar{x} _ {t-1} = \bar{v} _ {t-1} - \frac{\nu _ t}{\lambda} \nabla f _ t(\bar{x} _ {t-1})

Therefore νt/λ\nu_t/\lambda actually acts as implicit step size, even though the main motivation of the paper is avoiding step size selection.

  1. In Tab 3 and 6, ABBS-Z starts with a better backbone (FID 0.081 vs 0.301 for MotionLCM). Its better final FID (0.177 vs 0.309) may therefore come from the backbone rather than the proposed method. There needs to be comparisons for the guidance methods using the same backbone and control module.

  2. Remark 2 claims that the approximation is accurate (with low noise) but Eq. B.7 does not establish this. Because its denominator 1−νtLt1-\nu _ t L _ t can approach zero even when νt\nu _ t is small. The paper also acknowledges that the required condition νtLt<1\nu _ t L _ t < 1 cannot be verified (line 308). The authors should therefore justify that, this condition holds within the guidance window, and the resulting bound is small.

Questions for authors
  1. [W1] Can the authors clarify what ABBS adds to DiffPIR through a direct comparison and separate tests of the penalty schedule and sampling update?

  2. [W2] How does tuning λ\lambda avoid step-size selection when νt/λ\nu _ t/\lambda acts as an implicit step size?

  3. [W3] Can the authors compare the guidance methods using the same backbone and control module, in order to isolate ABBS's contribution in Tables 3 and 6?

  4. [W4] Can the authors justify νtLt<1\nu _ t L _ t < 1 holds throughout the guidance window and that the error bound in Eq. B.7 is small enough to support Remark 2?

3 comments
AuthorsOpenReview ↗

## Response to Reviewer N8Nz (1/2): W1

Thank you for the detailed and technical review. Following your comments, we revised the paper on the relation to DiffPIR (new Remark 3), the statements on the step size (Sections 3.2 and 4.1), and Remark 2.

W1. Relation to DiffPIR

Thank you for making the comparison explicit. As you point out, the two subproblems differ only in the weight of the quadratic term, and the weight in ABBS is 1/ct1/c_t times that in DiffPIR. When the re-noising is included, the two updates agree to first order ((a) below).

DiffPIR is designed and evaluated mainly for image restoration, where the data subproblem is often solved in closed form. In contrast, ABBS targets training-free guidance, where the loss involves a pretrained network or a decoder and the subproblem has no closed form. In this setting, ABBS provides the following two points. First, we derived the scale νt\nu_t at which one reverse step is read as a proximal step on the prior, for the samplers of the form of Eq. 5, extending the first-order relation in App. A.1 of DiffPIR, and we gave a bound on the residual (Proposition 1; (b) below). Second, ABBS solves the subproblem iteratively, instead of approximating it by the single gradient step of Eq. 13 of DiffPIR ((c) below). We examined both in Appendix D.3 ((d) below).

(a) DiffPIR also works at the scale of Proposition 1, to first order. The re-noising of DiffPIR (Eq. 15 of DiffPIR) is the case ωt2=(1−αˉt−1)ζ\omega_t^2=(1-\bar\alpha_{t-1})\zeta of Eq. 5, for which ct=1−(σˉt−1/σˉt)1−ζc_t=1-(\bar\sigma_{t-1}/\bar\sigma_t)\sqrt{1-\zeta}. By the identity in Proposition 1 and Eq. 8, the two updates can be written exactly as follows. ABBS:  xˉt−1=proxft/ρt(vˉt−1),DiffPIR:  xˉt−1=vˉt−1+ct(proxft/(ctρt)(vˉt−1)−vˉt−1).\text{ABBS:}\ \ \bar x_{t-1}=\mathrm{prox}_{f_t/\rho_t}(\bar v_{t-1}),\qquad \text{DiffPIR:}\ \ \bar x_{t-1}=\bar v_{t-1}+c_t\bigl(\mathrm{prox}_{f_t/(c_t\rho_t)}(\bar v_{t-1})-\bar v_{t-1}\bigr). The factor 1/ct1/c_t in the weight of ABBS appears in DiffPIR as the coefficient ctc_t in the update. To first order, both updates are vˉt−1−(νt/λ)∇ft(vˉt−1)\bar v_{t-1}-(\nu_t/\lambda)\nabla f_t(\bar v_{t-1}), and this holds for every ζ\zeta. The two updates differ only in directions where the curvature of fSf_\mathcal{S} is comparable to or larger than λ/σˉt2\lambda/\bar\sigma_t^2.

(b) Difference in viewpoint. DiffPIR plugs a denoiser into HQS and returns to the sampler by re-noising. ABBS builds the method on reading one reverse step itself as the proximity operator of the smoothed prior. For DDPM, App. A.1 of DiffPIR shows this reading as a first-order approximation with the same scale. Proposition 1 extends it, with a bound on the residual, to all samplers of Eq. 5.

(c) What ABBS adds. In training-free guidance, the main case is that the loss involves a pretrained network or a decoder, and the data subproblem has no closed form. In this case, DiffPIR approximates the subproblem by one gradient step (Eq. 13 of DiffPIR). Under the same sampler, this corresponds to ABBS whose inner solver is one gradient descent step with step size 1/ρt1/\rho_t, whereas ABBS solves the subproblem with KK iterations. When the subproblem is solved in closed form, the only difference is where ctc_t enters, so we do not claim that ABBS improves on DiffPIR in that case.

(d) Experiments (new Appendix D.3; excerpt of Table 8 below). On Task (iv), we compared the components in your question separately (all with λ=5×10−2\lambda=5\times10^{-2}).

  • Penalty schedule (ABBS vs. V1 (no ctc_t)): If we use the weight λ/σˉt2\lambda/\bar\sigma_t^2 of DiffPIR and do not multiply by ctc_t of the re-noising, the CLIP score becomes much lower than that of ABBS with a similar style score (λ=1×10−2\lambda=1\times10^{-2}, Figure 2) (20.79 vs. 25.78). This gap remains at K=80K=80.
  • Sampling update (V1 vs. V2 (ctc_t in the update)): If we keep the same weight and multiply the update by ctc_t, the results return to almost those of ABBS. That is, ctc_t can enter either the weight or the update, and the performance drops if it enters neither.
  • Inner solver (ABBS vs. V3 (one gradient step)): Even with the same first-order step, approximating the subproblem by the single gradient step of Eq. 13 of DiffPIR gives a much worse style score.

Table 8 (excerpt). "Corr." is the factor on ut−x^0∣t\mathbf{u}_t-\hat{\mathbf{x}}_{0|t} in the update.

Variantρt\rho_tCorr.Inner solverStyle ↓\downarrowCLIP ↑\uparrow
ABBSλ/νt\lambda/\nu_t11Adam331.2527.12
(V1) No ctc_tλ/σˉt2\lambda/\bar\sigma_t^211Adam219.9820.79
(V2) ctc_t in the updateλ/σˉt2\lambda/\bar\sigma_t^2ctc_tAdam327.6627.26
(V3) One gradient step (DiffPIR, Eq. 13)λ/νt\lambda/\nu_t111 gradient step678.2026.86

Note. V3 is not a tuned gradient-step baseline: it keeps the scale of ABBS to isolate the effect of the single gradient step of Eq. 13 of DiffPIR. MPGD, which has the same form with its own step sizes (Ye et al., 2024, Thm. 3.2), is compared in Table 2, and ABBS improves on it in both scores.

AuthorsOpenReview ↗

Response to Reviewer N8Nz (2/2): W2–W4

W2. Role of λ\lambda

You are right. The exact solution of Eq. 12b is an implicit gradient step with step size νt/λ\nu_t/\lambda, and λ\lambda sets the overall scale of this step size. The statement "λ\lambda is not a step size" in Section 4.1 was inaccurate on this point, so we rewrote it. We also withdrew "remove the step size from the design" in the Abstract and Section 1.

Since λ\lambda is the strength of guidance and sets the trade-off between validity and fidelity (Figure 2), we choose it for each task (Table 4). The revised paper therefore makes a narrower claim (Section 3.2). ABBS does not remove the need to choose the strength of guidance, but it removes two design choices of explicit gradient-step guidance: (i) an upper bound on the step size from the smoothness of the loss, and (ii) a hand-designed schedule of the step size over tt.

For (i), since Ft(vˉt−1)=ft(vˉt−1)F_t(\bar v_{t-1})=f_t(\bar v_{t-1}), every xˉt−1\bar x_{t-1} with Ft(xˉt−1)≤Ft(vˉt−1)F_t(\bar x_{t-1})\le F_t(\bar v_{t-1}), in particular the exact solution of Eq. 12b, satisfies ft(xˉt−1)≤ft(vˉt−1)−ρt2∥xˉt−1−vˉt−1∥2f_t(\bar x_{t-1})\le f_t(\bar v_{t-1})-\tfrac{\rho_t}{2}\|\bar x_{t-1}-\bar v_{t-1}\|^2 for every λ\lambda. In contrast, for an explicit gradient step with the same step size, the standard guarantee that ftf_t decreases requires νt/λ<2/Lf\nu_t/\lambda<2/L_f, and LfL_f is unknown for losses that involve a pretrained network (Section 1).

(ii) comes from Proposition 1, not from the implicit step. If we set the weight of Problem 9 to λt=λ\lambda_t=\lambda for all tt, the dependence of the step size on tt is determined by νt\nu_t (ρt=λ/νt\rho_t=\lambda/\nu_t). Thus, the schedule over tt is induced by the sampler through νt\nu_t, rather than hand-designed. The end σˉmax⁡2\bar\sigma^2_{\max} of the guidance window remains a task-level hyperparameter, and we now report its value in Appendix D.

Consistently, V1 in Table 8 (W1), which replaces νt\nu_t by σˉt2\bar\sigma_t^2, lies far below the trade-off curve of ABBS.

The argument above concerns the exact solution of Eq. 12b, as does your derivation. In the implementation, we compute the prox inexactly by KK Adam iterations, and the results change little when we increase KK from 30 to 80 (Appendix D.3).

W3. Comparison with the same backbone and control module

We agree that Table 3 is an end-to-end comparison of systems with different backbones and representations, not a controlled comparison of guidance algorithms, and the revised Section 4.2 states this. Table 3 uses bold only for the editing metrics, and we do not claim that ABBS-Z\mathcal{Z} has a better FID.

The controlled evidence for ABBS in motion is Table 7, which compares the results with and without ABBS under the same backbone and the same control module. On both backbones, adding ABBS greatly reduces the average error (0.268 to 0.004 m and 0.253 to 0.019 m), and on the latent-space backbone it slightly increases the FID (0.146 to 0.177). There, the FID without guidance is already smaller than that of MotionLCM (0.309), so, as you point out, the difference in FID comes from the backbone.

A controlled comparison with the motion-specific guidance of the baselines would require reimplementing it on the same backbone with the same control module, and we did not include this experiment. We therefore base the contribution of ABBS itself on Table 7 and read Table 3 as a comparison of complete systems.

W4. Validity of νtLt<1\nu_t L_t < 1

As you point out, the statement in Remark 2 that the bound is tight at low noise levels does not follow from Eq. (B.7), so we removed it.

By the second-order Tweedie formula, ∇2gt=σˉt−2(I−Jt)\nabla^2 g_t=\bar\sigma_t^{-2}(I-J_t), where Jt⪰0J_t\succeq0 is the Jacobian of E[x0∣xˉt]\mathbb E[x_0\mid\bar x_t]. If JtJ_t has a direction with an eigenvalue close to 0, then νtLt\nu_tL_t is approximately ctc_t or larger. Since c1=1c_1=1 for every sampler, (A2) need not hold near t=1t=1. The first term of the bound is νtLt\nu_tL_t times an upper bound on the distance that prox⁡νtgt\operatorname{prox}_{\nu_tg_t} moves xˉt\bar x_t (added to Remark 2), so the bound is small compared with this distance only when νtLt≪1\nu_tL_t\ll1. Thus, we do not claim that (A2) can be verified or that the bound is uniformly small over all guided steps, and the revised Remark 2 states this.

Algorithm 1 does not require checking (A2). The assumptions (A1) and (A2) are used only to read one reverse step as an inexact proximal step and to bound its residual. The dependence of ρt=λ/νt\rho_t=\lambda/\nu_t on tt is determined by the noise schedule and the sampler, whether or not (A2) holds.

Changes to the paper

  • We added a new Remark 3, Appendix D.3, and Table 8 (W1).
  • We withdrew "λ\lambda is not a step size" and "remove the step size from the design," and we rewrote the Abstract and Sections 1, 3.2, and 4.1 (W2).
  • We stated in Section 4.2 that Table 3 compares complete systems (W3).
  • We rewrote Remark 2 (W4).

Thank you again for the careful review. We hope that the revision addresses your concerns.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer N8Nz,

Thank you again for your thoughtful and constructive feedback.

As the discussion period approaches its end, we would appreciate any feedback on whether our responses and the revised manuscript have adequately addressed your concerns. We apologize for following up only two days after posting our responses, and we appreciate your time.

Best regards,

The Authors

Reviewer_Q2urOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new training-free guidance technique ABBS by decomposing conditional generation into subproblems. At each noise level, they have two proximal updates, one for the prior and one for the condition. The prior step, is approximated by one reverse step with a bounded error. The condition step, is solved within closed form or K steps of Adam. They are able to bypass the need for step size, as the displacement is determined by the scale of the proximity operator. They show results across multiple tasks like segmentation, deblurring, inversion, etc.

Critical strengths
  1. The problem is well-motivated where each reverse step becomes a backward-backward split, and there is no need for step size tuning. It is clearly formulated in the paper.
  2. Strong results across various image tasks, ABBS has the best validity on all four tasks, with comparable fidelity. On motion, ABBS-X beats or matches MaskControl’s editing errors while being faster.
  3. The evaluation covers a broad range of tasks, to showcase different settings like inversion, segmentation , deblurring and super-resolution.
Critical weaknesses
  1. Fig 3 shows the effect of K iterations of Adam for the conditional step, but the learning rate of Adam is not mentioned, and there is no mention of sensitivity to this, and if the saturation of K would be dependent on this. This would ensure that the step size tuning has not shifted into the inner solver.
  2. Fig. 2 shows that λ cause a clear trade-off between validity and fidelity only on the style task, but this is not tested out on other tasks.
Questions for authors
  1. ABBS-Z is the fastest method but has much lower performance compared to MaskControl and ABBS-X, where does the gap in latent space come from?
  2. What value of σ̄²_max is used for each task, and how was it chosen and ablated?
  3. Address weaknesses
1 comment
AuthorsOpenReview ↗

Response to Reviewer Q2ur

Thank you for the positive evaluation and the specific questions. Your questions on the learning rate and σˉmax⁡2\bar\sigma^2_{\max} pointed out details that were missing from the paper. We added them to Appendix D of the revised paper.

Implementation details (learning rate of Adam and σˉmax⁡2\bar\sigma^2_{\max})

We added both settings to Appendix D. As noted in the general response, following the comments of another reviewer, we withdrew the statement "remove the step size from the design." However, the displacement is still determined by the scale of the proximity operator, and what ABBS removes is the upper bound based on smoothness and the design of the step size for each tt. In Section 3.2 of the revised paper, we state that the exact solution of Eq. 12b is an implicit gradient step, and that the guarantee that this step decreases ftf_t needs no upper bound based on the smoothness of the loss. Like KK, the learning rate does not change the subproblem in Eq. 12b itself, and it affects the results through how accurately the inner solver solves that subproblem within KK iterations. For σˉmax⁡2\bar\sigma^2_{\max}, we tried a few candidate values for each task and chose the value that gave good validity and fidelity. We did not perform a systematic ablation.

Trade-off over λ\lambda on other tasks

As you point out, we have examined the trade-off over λ\lambda only on style guidance. In Problem 9, λ\lambda balances the condition term against the prior term, and it sets the overall scale of the step size νt/λ\nu_t/\lambda of the condition prox. Since this mechanism does not depend on the task, we expect a similar qualitative trend on the other tasks: a smaller λ\lambda strengthens guidance and tends to increase validity at the cost of fidelity.

Gap of ABBS-Z\mathcal{Z}

The gap lies mainly in the editing accuracy. Since the generation quality (R-precision and FID) depends on the backbone as well as on the guidance procedure (Table 6), Table 3 does not rank the methods by these metrics.

Since the editing accuracy of ABBS-X\mathcal{X} is comparable to that of MaskControl, we discuss below the gap between ABBS-Z\mathcal{Z} and ABBS-X\mathcal{X}. Table 7 suggests that the gap in editing accuracy arises when guidance is added, and not from the control module. In Table 7, with the control module alone, the errors on the two backbones are almost the same, whereas with ABBS alone under the same settings (Table 4), the error in the latent space is larger. Since this comparison does not involve the control module, the difference between the control modules does not cause this gap. In the latent space, the condition is evaluated through the decoder Dθ\mathcal{D}_\theta. Possible causes are therefore that the subproblem in Eq. 12b becomes more nonlinear, so that its solution with the same KK can be less accurate, and that the generated motion is limited to what the decoder can output. However, the two settings differ not only in the space where guidance is applied but also in the diffusion model itself (MDM v2 and MLD++), so we cannot fully separate the effect of the latent space from that of the model. Thus, we read ABBS-Z\mathcal{Z} as showing that the same guidance procedure applies efficiently in a latent space, and we do not claim that the accuracy gap to ABBS-X\mathcal{X} is intrinsic to latent-space guidance.

Changes to the paper

We added the learning rate and σˉmax⁡2\bar\sigma^2_{\max} for all tasks to Appendix D, and we stated in Section 3.2 the role of the step size νt/λ\nu_t/\lambda of the condition prox.

Thank you again for the positive evaluation and the comments.

Reviewer_YPPSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ABBS guides a pretrained diffusion model by alternating a reverse diffusion step with a regularized condition update. The condition update works on a clean-sample estimate without further denoiser calls. The paper derives the proximal scale nu_t = c_t * bar_sigma_t^2 and bounds the error of one prior update under stated assumptions. It evaluates four image tasks and motion path following in data and latent spaces. ABBS improves condition satisfaction on three image tasks and ties on deblurring. The motion experiments use trained control modules and also test their contributions separately.

Critical strengths
  1. ABBS makes a useful connection between sampling and condition optimization, even though its ingredients have precedents. The quadratic penalty can adjust corrections to the curvature of the condition loss (Section 3, Eqs. 12-15; Algorithm 2). Computing these corrections without more denoiser calls is practical. The derivation also identifies the proximal map that the reverse step approximates and gives an explicit error bound. Together, these provide a more principled way to combine sampling and loss optimization.

  2. The experiments provide several useful checks. Image comparisons use the same backbone. Segmentation distance improves from 1690 to 1483 with almost unchanged FID (Table 2, p. 7). For style guidance, K = 10 gives style/CLIP scores of 362.15/27.31 in 58.8 seconds, compared with TFG's 370.70/27.07 in 55.7 seconds (Table 5, p. 17). This shows a benefit at similar cost. Figures 2-3 already examine the guidance trade-off and diminishing gains from more inner iterations. The motion ablation is also informative: average path error is 0.268 m with the control module alone, 0.008 m with ABBS alone, and 0.004 m with both on the data-space backbone (Table 7, p. 19). ABBS clearly helps tracking, while the control module helps preserve generation quality. As the authors acknowledge, speed comparisons between motion methods also reflect different backbones.

Critical weaknesses
  1. My main concern is why the regularized update works well. The benefit could come from the derived c_t scale, from quadratic regularization more generally, or from its interaction with finite Adam iterations (Remark 1; Appendix C). The image results support the complete method end-to-end, the existing K sweep shows that style results settle as K increases, but keeps the scale fixed.

One useful experiemnt for this could compare nu_t = c_t * bar_sigma_t^2 against nu_t = bar_sigma_t^2 at the already studied K = 30 and K = 80. For each K, keep the backbone, prompts, style references, initial seeds, optimizer, and guided sampling steps the same. Try the same number of lambda values for each scale on development cases, then compare style loss and CLIP alignment on held-out cases at similar runtime. If the proposed scale still helps as this residual falls, that would support the calibration. Similar trade-offs would suggest that generic regularized guidance accounts for most of the benefit. A gain seen only with less accurate inner solutions would suggest that early stopping matters. These outcomes would leave the reported results intact, but change the explanation of the method's contributions.

Questions for authors

Point 1 from Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YPPS

Thank you for the careful review and the positive comments, and in particular for designing a concrete additional experiment. Based on your suggestion, we added the experiment to Appendix D.3 (Table 8) of the revised paper.

Setting

On style guidance (Task (iv)), we compared two choices of νt\nu_t in the weight ρt=λ/νt\rho_t=\lambda/\nu_t, namely ctσˉt2c_t\bar\sigma_t^2 (ABBS) and σˉt2\bar\sigma_t^2, at K=30K=30 and K=80K=80. As you suggested, all settings other than νt\nu_t are the same for both. For the same KK, both evaluate ∇fS\nabla f_\mathcal{S} the same number of times, so their runtimes are also the same. Since the style score and the CLIP score trade off against each other, choosing one λ\lambda for each scale can place the two resulting points at different positions on the trade-off, and then we cannot tell which is better. Therefore, instead of tuning λ\lambda for each scale, we ran νt=σˉt2\nu_t=\bar\sigma_t^2 with the same λ=5×10−2\lambda=5\times10^{-2} as in Table 2, and we compared its CLIP score with that of the point of ABBS with almost the same style score (λ=1×10−2\lambda=1\times10^{-2} in Figure 2). In this way, we can judge with a single metric whether the two scales lie on the same trade-off.

Results

νt\nu_tKKStyle ↓\downarrowCLIP ↑\uparrow
ctσˉt2c_t\bar\sigma_t^2 (ABBS)30331.2527.12
ctσˉt2c_t\bar\sigma_t^2 (ABBS)80325.4627.01
σˉt2\bar\sigma_t^230219.9820.79
σˉt2\bar\sigma_t^280212.8421.39
ctσˉt2c_t\bar\sigma_t^2 (ABBS, λ=1×10−2\lambda=1\times10^{-2})30221.2925.78

Interpretation

The results are consistent with the first of the three outcomes you described. Compared with ABBS at λ=1×10−2\lambda=1\times10^{-2}, which has almost the same style score, νt=σˉt2\nu_t=\bar\sigma_t^2 has a much lower CLIP score and lies below the trade-off curve of ABBS. This gap is unlikely to be due to a less accurate inner solution. Here, we use KK as a proxy for the accuracy of the inner solver. When we increase KK from 30 to 80, the CLIP score of νt=σˉt2\nu_t=\bar\sigma_t^2 increases only slightly, and the gap remains. The results of ABBS also change little beyond K=30K=30 (Figure 3). These results suggest that, at least on the style-guidance task, the gain of ABBS is not explained solely by generic quadratic regularization or by early stopping of the inner solver. Instead, the dependence on tt through ctc_t appears to be important. Note that this dependence also appears, to first order, in the re-noising of DiffPIR (Remark 3 and V2 in Table 8).

Changes to the paper

We added Appendix D.3 and Table 8. The comparison above corresponds to V1 (no ctc_t) in Table 8. Table 8 also includes two other variants (V2 and V3), which answer questions from other reviewers.

Thanks to your suggestion, the revised paper now shows the role of ctc_t concretely. We hope that these results address your concern.

Thanks for the clear response and the additional experiments. These directly address my concern on isolating the role of c_t, atleast for the style-guidance task. I will update my score accordingly.

Thank you very much for the positive feedback. We sincerely appreciate you taking the time to read our response and consider the additional experiments. We are glad that they helped clarify the role of ctc_t.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their careful reading and helpful comments. The comments allowed us to improve the paper in several respects. In the revised PDF, the changes are shown in blue. The main changes are as follows.

  1. We made the relation to DiffPIR explicit in the Introduction and in a new Remark 3 in Section 3 (mainly N8Nz W1).
  2. We added a comparison of variants of ABBS on Task (iv) (new Appendix D.3). In addition to the comparison with νt=σˉt2\nu_t=\bar\sigma_t^2 proposed by Reviewer YPPS, we examined the effects of the weight and of the update separately, and we also compared ABBS with Eq. 13 of DiffPIR (mainly YPPS, N8Nz W1).
  3. We made the statements on the step size more precise. We withdrew statements such as "λ\lambda is not a step size" and "remove the step size from the design." In Section 3.2, we now state that the exact condition prox is an implicit gradient step with step size νt/λ\nu_t/\lambda, and that the guarantee that it decreases ftf_t needs no upper bound based on the smoothness of the loss (mainly N8Nz W2).
  4. We revised Remark 2 on the error bound (mainly N8Nz W4).
  5. We added the setting of Adam and σˉmax⁡2\bar\sigma^2_{\max} to Appendix D (mainly Q2ur).
  6. We stated in Section 4.2 that Table 3 is an end-to-end comparison of systems (mainly N8Nz W3).

We thank the reviewers again for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.