- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether frozen pre-trained text-to-image/video diffusion models (PTDMs) can perform restoration when steered only through their native conditioning pathway. Natural-language prompts and input-token optimization fail. A “conditioning prompt” optimized at the text-encoder output (in practice a gated residual added to the null-text context at every block) does elicit restoration. Training this prompt naively on noisy degraded states leads to train–test trajectory mismatch, so the authors learn it on a monotone noisy-to-clean diffusion bridge (EBR). They add partial inversion for an image-specific start state and a normalized velocity composition for mixed degradations. The approach is instantiated on FLUX (images) and WAN (videos), with one prompt per degradation. It is evaluated on in-distribution, OOD, mixed and unseen (UDC) benchmarks against all-in-one restoration methods and backbone-matched LoRA variants.
Critical strengths
Competitive all-in-one restoration from a fully frozen backbone with very few trainable parameters. The approach beats backbone-matched LoRA variants that train 50–180× more parameters.
The trajectory-mismatch analysis is clear and supported by ablations: naive training < DDBM < EBR, and partial inversion helps on both backbones. The evaluation covers many real-world OOD datasets for both images and videos.
Critical weaknesses
The “elicitation, not adaptation” claim is overstated. The prompt is a learned residual injected into every transformer block with per-block gates. Functionally this is deep prompt/prefix tuning with 0.34–1.57M parameters per task. No experiment separates an “intrinsic” capability from what the optimized prompt learns. There is no comparison with prefix tuning or LoRA at a matched parameter budget, and no control with a weaker or less pre-trained prior. The bridge (EBR) is adopted from prior work, which limits methodological novelty.
The fairness and rigor of the comparisons are questionable. The method is non-blind: the prompt is chosen with the degradation label, and for UDC it is chosen manually (low-light+blur). Several baselines are blind. The LoRA “Concat” baseline seems under-tuned. No variance or seeds are reported, and several test sets are very small.
Fidelity and cost trade-offs. In-distribution PSNR/SSIM trail lightweight methods. FLUX inference costs 5.51 s and 348 TFLOPs per image, against ~0.14 s for DCPT/DFPIR.
Questions for authors
How does the method compare with prefix/deep prompt tuning and with LoRA at the same parameter budget, all trained on the same EBR bridge with partial inversion?
How do the baselines perform when given the same degradation label, and how sensitive is UDC performance to the manual choice of prompt?
Please report variance across seeds and the hyperparameter tuning effort spent on the LoRA “Concat” baseline.
How much of the gain over the “Bridge (LoRA)” variant comes from partial inversion rather than from conditioning-space prompting?
7 comments
Response to Reviewer wYYi Contd.
- CW1, Q1, Q4; No comparison with prefix tuning or LoRA at matched budget; Gain over LoRA Bridge with partial-inversion.
As clarified above, our objective is to investigate restoration capabilities accessible through the frozen PTDM's conditioning pathway, rather than establish superiority over other adaptation methods. Nevertheless, we provide controlled comparisons to evaluate practical effectiveness. We acknowledge that the original LoRA Bridge baseline did not use partial inversion and thank the reviewer for pointing this out. We now evaluate it under identical inference settings (sampling steps, bridge schedule, and partial inversion), alongside parameter-matched LoRA and prefix-tuning baselines (Table R2).
Comparison with LoRA. Full LoRA, using substantially more trainable parameters, generally performs better in-distribution (ID), while our approach achieves stronger OOD performance on average, suggesting better generalization. At matched parameter budgets, our approach outperforms LoRA across most ID and OOD metrics, demonstrating the effectiveness of our approach while preserving the pretrained backbone weights.
Comparison with prefix tuning. As discussed above, our framework is closely related to prefix/deep prompt tuning, but uses a single conditioning prompt shared across transformer blocks, motivated by our investigation of a task-level restoration instruction. We now evaluate prefix-tuning (K/V embeddings at each block) which is more expressive, on WAN. Prefix tuning achieves stronger ID performance, including higher PSNR (23.51 vs. 23.30 dB), while our shared-prompt approach achieves higher OOD PSNR (24.94 vs. 24.28 dB), with competitive performance across the remaining metrics. These results indicate that more expressive prompt parameterizations can offer performance improvements, consistent with the extensions discussed in Sec. B.3. However, our central observation is that restoration behavior can be elicited through a shared conditioning signal in a frozen PTDM, without requiring independently learned block-wise prompts.
We have incorporated the updated LoRA changes along with the parameter-matched version in Table 4 and across the supplementary of the revised version. We have also discussed the comparisons with prefix tuning in Secs. B.1 and E.
Table R2: Comparison with LoRA bridges under identical inference settings (N=6 sampling steps and partial inversion), including parameter-matched baselines. The table also provides updated numbers for the Concat baseline and comparisons with prefix tuning. Best results are in bold among Ours, LoRA, and Concat baselines.
(a) In-distribution (mean)
| Method | Total (M) | Train. (M) | P↑ | S↑ | L↓ | D↓ | C↑ | M↑ | Do↑ |
|---|---|---|---|---|---|---|---|---|---|
| FLUX + Bridge (LoRA) | 13,419.7 | 1,434.45 | 25.70 | 0.855 | 0.110 | 0.082 | 0.616 | 66.98 | -- |
| FLUX + Bridge (LoRA, matched) | 11,993.2 | 7.94 | 23.24 | 0.817 | 0.149 | 0.111 | 0.583 | 65.60 | -- |
| FLUX + Concat (LoRA) | 13,419.7 | 1,434.46 | 19.96 | 0.537 | 0.382 | 0.231 | 0.412 | 52.44 | -- |
| FLUX (Ours) | 11,994.7 | 7.87 | 25.39 | 0.843 | 0.118 | 0.090 | 0.571 | 64.91 | -- |
| WAN + Bridge (LoRA) | 1,633.4 | 87.49 | 22.84 | 0.782 | 0.143 | 0.095 | 0.583 | 65.49 | 0.637 |
| WAN + Bridge (LoRA, matched) | 1,547.6 | 1.74 | 21.82 | 0.760 | 0.179 | 0.122 | 0.539 | 62.00 | 0.586 |
| WAN + Concat (LoRA) | 1,633.4 | 87.50 | 19.11 | 0.616 | 0.275 | 0.181 | 0.492 | 60.41 | 0.491 |
| WAN (Ours) | 1,548.0 | 1.74 | 23.30 | 0.783 | 0.148 | 0.105 | 0.560 | 64.26 | 0.627 |
| WAN + Prefix tuning (matched) | 1,548.1 | 1.84 | 23.51 | 0.789 | 0.137 | 0.094 | 0.572 | 64.67 | 0.632 |
(b) Out-of-distribution (mean)
| Method | Total (M) | Train. (M) | P↑ | S↑ | L↓ | D↓ | C↑ | M↑ | Do↑ |
|---|---|---|---|---|---|---|---|---|---|
| FLUX + Bridge (LoRA) | 13,419.7 | 1,434.45 | 20.36 | 0.732 | 0.252 | 0.179 | 0.545 | 58.73 | -- |
| FLUX + Bridge (LoRA, matched) | 11,993.2 | 7.94 | 17.95 | 0.687 | 0.296 | 0.206 | 0.531 | 57.81 | -- |
| FLUX + Concat (LoRA) | 13,419.7 | 1,434.46 | 18.34 | 0.514 | 0.468 | 0.297 | 0.388 | 46.73 | -- |
| FLUX (Ours) | 11,994.7 | 7.87 | 20.60 | 0.748 | 0.235 | 0.174 | 0.534 | 60.14 | -- |
| WAN + Bridge (LoRA) | 1,633.4 | 87.49 | 21.80 | 0.784 | 0.198 | 0.136 | 0.541 | 59.92 | 0.593 |
| WAN + Bridge (LoRA, matched) | 1,547.6 | 1.74 | 21.38 | 0.757 | 0.210 | 0.154 | 0.519 | 58.82 | 0.570 |
| WAN + Concat (LoRA) | 1,633.4 | 87.50 | 19.38 | 0.605 | 0.350 | 0.237 | 0.461 | 55.12 | 0.471 |
| WAN (Ours) | 1,548.0 | 1.74 | 24.94 | 0.788 | 0.191 | 0.140 | 0.541 | 60.12 | 0.599 |
| WAN + Prefix tuning (matched) | 1,548.1 | 1.84 | 24.28 | 0.788 | 0.184 | 0.133 | 0.547 | 59.96 | 0.599 |
Response to Reviewer wYYi contd.
- CW1; EBR bridge adopted from prior work, limiting methodological novelty.
We agree that the E-Bridge schedule is adopted from prior work and do not claim the bridge formulation itself as our contribution, as explicitly acknowledged in Sec. 3.2 (L259--261). Our contribution lies in identifying how an appropriate training trajectory enables the restoration behavior observed in Fig. 2 to be effectively leveraged for multistep sampling with a frozen PTDM. Specifically, we identify train-test trajectory mismatch as a key obstacle to naive conditioning prompt learning. While conventional DDBM provides a degraded-to-clean transition, its low noise near the degraded endpoint limits the restoration response (Fig. 3, Table 3). This motivates a bridge that maintains sufficiently noisy degraded states while progressively transitioning toward clean states with decreasing noise. E-Bridge satisfies these requirements, and our experiments establish its effectiveness in eliciting restoration from a frozen PTDM. Thus, our contribution is not a new bridge schedule, but identifying and addressing the trajectory requirements for eliciting restoration through a frozen PTDM's conditioning pathway.
- CW2, Q2; Fairness of comparisons is questionable; The method is non-blind while several baselines are blind; How do the baselines perform when given the degradation label?
Our primary backbone-matched LoRA baselines (Bridge and Concat) have access to the same degradation-type information through text instructions, enabling controlled comparisons. Among the broader SOTA baselines, DFPIR, PixWizard, and AutoDIR support text-based task instructions, which we provide during evaluation. Other methods, including DCPT, RestoreVAR, and SLAIR, operate without explicit degradation labels. Nevertheless, we include these approaches as they are state-of-the-art methods and serve as restoration-quality benchmarks for our approach. Thus, our backbone-matched experiments isolate the effectiveness of conditioning-based restoration under comparable task information, while the SOTA comparisons contextualize its practical performance.
- CW2, Q2; UDC degradation is manually chosen.
Since UDC is an unseen task, we compose existing degradation-specific prompts without additional training. Our choice of low-light+blur is motivated by UDC characteristics which include light attenuation and optical blur (L366), consistent with AutoDIR, which similarly identified UDC restoration as brightening and deblurring. We further investigate the sensitivity of UDC restoration to prompt selection in Fig. 12 in the revised version. The low-light prompt improves exposure but fails to correct blur, while mismatched prompts (haze or rain) provide limited to no restoration as expected. In contrast, composing the low-light and blur prompts improves both exposure and sharpness, demonstrating the importance of selecting appropriate degradation-specific prompts. We have incorporated this discussion in Sec. I.4 of the revised version.
- CW2, Q3; Hyper-parameter tuning effort spent on the LoRA concat baseline.
We carefully tuned LoRA-Concat to ensure stable training and strong performance for both backbones, using bf16 with learning rates of for FLUX and for WAN. We also explored alternative learning rates and numerical precisions; higher learning rates with fp16 caused training divergence on FLUX, while lower learning rates yielded inferior performance on WAN. To assess convergence, we extended training substantially beyond the original budgets. On FLUX, increasing training from 50K to 125K steps improved PSNR from to dB. Further training to 162.5K steps yielded no meaningful improvement. Similarly, WAN improved from dB at 40K steps to dB at 100K steps, with no significant improvements beyond 100K. Thus, we now use the checkpoint trained at K steps for FLUX and the checkpoint at K steps for WAN, for our evaluations as they showed best performance.
Despite these extended training budgets, LoRA-Concat remains substantially below our approach (Table R2 above) on most metrics. Overall, our approach achieves stronger restoration with -- fewer trainable parameters and substantially lower training cost, supporting the effectiveness of eliciting restoration from the frozen PTDM using our conditioning prompts. We have updated the results for the Concat baseline throughout the paper.
Response to Reviewer wYYi
We thank the reviewer for their constructive feedback and for recognizing the effectiveness of our approach, the trajectory-mismatch analysis, and the breadth of our evaluation. We address the concerns below.
We agree that our conditioning prompt can be viewed as a form of prefix/deep prompt tuning, as both optimize conditioning representations while keeping backbone weights frozen. We have now clarified this in the revised version (Sec. B.1). However, our objective is not to introduce a new prompt-tuning mechanism, but to investigate whether restoration-relevant capabilities of a pretrained diffusion model (PTDM) can be accessed through its conditioning pathway.
As motivated in Sec. 1 (L48--53), diffusion-based editing demonstrates that frozen PTDMs can steer noisy image states toward desired outputs through language conditioning. We therefore ask whether a frozen PTDM can similarly steer noisy degraded states toward clean outputs, which would reveal restoration-relevant capability. However, natural-language instructions fail to elicit this behavior, motivating our investigation in Fig. 2. Crucially, Fig. 2 demonstrates that directly optimizing the text-encoder output enables restoration from the unaltered PTDM without diffusion-bridge training, residual conditioning, or learnable gates, establishing our central observation independently of these components.
Thus, we use prompt learning as a tool to probe and elicit restoration behavior from a frozen PTDM, analogous to how natural-language instructions can be used to guide generative diffusion models. While our final framework shares mechanisms with deep prompt tuning, we learn a single degradation-specific prompt shared across inputs and reused across blocks, rather than independent layer-specific prompts. We discuss block-wise prompts (similar to deep prompt tuning) as potential extensions in Sec. B.3. Thus, our distinction from prefix/deep prompt tuning lies primarily in our objective which is investigating whether restoration-relevant capabilities can be accessed through conditioning a frozen PTDM.
We conduct two experiments to support whether learned prompts leverage restoration-relevant capabilities of the PTDM rather than independently learning the restoration mapping.
Elicitation with limited conditioning capacity. We repeat the elicitation experiment of Fig. 2 on REVIDE (on WAN) using only 8 learnable tokens (32,768 parameters), without residual conditioning, learnable gates, or bridge training. As shown in Table R1 and Fig. 8 of the revised version, this achieves 20.92 dB PSNR, outperforming natural language (14.99 dB) and text-encoder input optimization (15.58 dB), while approaching the 226-token text-encoder output prompt (22.02 dB) with ∼28× fewer parameters. Since this compact prompt is shared across videos of the same degradation, it suggests that the prompt elicits restoration-relevant capabilities of the frozen backbone rather than independently learning the entire restoration mapping.
Dependence on the pretrained backbone. As suggested by the reviewer, we investigate the role of the pretrained prior by fine-tuning WAN on animated video clips, thereby altering its original prior. We then relearn the conditioning prompt under the same Fig. 2 setting (226 tokens in text-encoder output). Despite retraining the prompt, REVIDE PSNR drops from 22.02 to 18.95 dB with the haze not removed effectively (see Fig. 8 of the revised version). This suggests that the restoration behavior elicited through conditioning depends on the original pretrained representations and was not recovered by prompt optimization alone under the same training protocol, indicating restoration-relevant capabilities in the pre-trained backbone.
Together, these experiments support our hypothesis that our prompts access restoration-relevant capabilities of the PTDM, rather than independently learning the complete restoration behavior. We have now clarified that ``intrinsic restoration capabilities'' refers to capabilities accessible through the conditioning interface of the PTDM (L78-80 of the revised version).
Table R1: Effect of conditioning capacity and pretrained backbone on REVIDE using WAN under the Fig. 2 setting, without residual conditioning, gates, or bridge training.