- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes RefReg, a training-free approach to image inverse problems using pre-trained image/text-to-image generators. A reference image is constructed from the measurement and supplied to the generator alongside a text prompt. The method then schedules image-conditioning strength and suppresses selected generation-to-text attention connections during early sampling. These interventions are combined with data-consistency updates to the predicted clean image. The motivation is that reference information should establish the observed scene structure early, while later steps refine its details.
However, I currently lean towards rejection. My main reservation is that the paper has not yet demonstrated a sufficiently clear advance over simpler, carefully tuned conditioning strategies.
Critical strengths
-
The reconstruction results are strong across a substantial evaluation. RefReg achieves the lowest LPIPS in 15 of the 16 dataset–task combinations in Table A6. The improvements are not restricted to one degradation, and the additional-backbone experiments show consistent perceptual gains. This is the strongest aspect of the paper: there is useful empirical evidence that the proposed combination works well for image restoration
-
The attention-timing experiment provides some support for the design. I found the comparison between early, middle, and late attention blocking useful. Applying the same intervention early improves reconstruction, whereas applying it later gives little benefit. This goes beyond simply removing a component and provides evidence that the timing of conditioning matters.
Critical weaknesses
-
I am not convinced that the methodological contribution is sufficiently established. The contribution is primarily a particular way of configuring and combining reference preprocessing, classifier-free guidance, and attention control around a clean-estimate correction procedure. Equation (7) explicitly adopts an existing two-scale guidance formulation, while the reference construction uses TV regularisation and measurement updates. Combining established ingredients can certainly be valuable, but the paper needs a stronger comparison with simpler alternatives to establish the significance of its particular design.
-
Several choices in the baseline comparisons make the reported advantages difficult to interpret. Appendix D.3 states that the MPC baseline omits the control-energy penalty from the original formulation, but no comparison is provided to establish the effect of that change. In the partially blind experiments, RefReg receives processed references and image-specific captions, whereas GDP and DMPlug receive raw measurement references and a generic prompt. These conditioning differences are particularly consequential in a paper whose central premise is that conditioning substantially affects reconstruction. The comparison therefore does not cleanly separate improvements in the reconstruction and operator-estimation algorithms from improvements in their inputs.
-
The methodological advance over existing sampling and conditioning techniques is not sufficiently demonstrated. Algorithm 1 follows the familiar pattern of predicting a clean image, correcting it for measurement consistency, and using the corrected estimate to continue sampling. Closely related constructions appear in DDS [1] and, more directly for flow models, FlowDPS [2]. ECI [3] explicitly develops an extrapolate–correct–interpolate procedure, while PCFM [4] extends constraint correction to nonlinear constraints in pre-trained flows. These methods differ in their correction operators and treatment of noise, but the basic sampling structure is not new. The conditioning components also have clear precedents. The separate image and text guidance scales in Equation (7) follow InstructPix2Pix [5], as the authors acknowledge. Moreover, Ge et al. [6] already investigate how reference information passes through image and text tokens in FLUX.2 using interventions on attention connections. RefReg applies these ideas differently but I would like to see stronger evidence that this specific intervention offers a substantial advantage over simpler conditioning strategies. My concern is therefore not that the paper fails to cite all relevant work, but that the additional contribution appears to be a particular combination and scheduling of established components. The reconstruction results are encouraging, but the attention intervention is not compared with simply reducing or removing prompt-content conditioning during the same early window.
References to look: [1]–[6] are the strongest references.
[1] Chung, H., Lee, S., and Ye, J. C. (2024). Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems. ICLR. [2] Kim, J., Kim, B. S., and Ye, J. C. (2025). FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems. ICCV. [3] Cheng, C., et al. (2025). Gradient-Free Generation for Hard-Constrained Systems. ICLR. [4] Utkarsh, U., et al. (2025). Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints. NeurIPS. [5] Brooks, T., Holynski, A., and Efros, A. A. (2023). InstructPix2Pix: Learning to Follow Image Editing Instructions. CVPR. [6] Ge, C., Gandikota, R., Torralba, A., and Rott Shaham, T. (2026). Vision-Language Binding in In-Context Image Generation. arXiv:2605.24624. [7] Yılmaz, M. A., Bilican, A., Biner, B. C., and Tekalp, A. M. (2026). Edit2Restore: Few-Shot Image Restoration via Parameter-Efficient Adaptation of Pre-trained Editing Models. arXiv:2601.03391. [8] Song, R., et al. (2026). Training-Free Image Editing with Visual Context Integration and Concept Alignment. arXiv:2604.04487. [9] Erbach, J., et al. (2025). Solving Inverse Problems with FLAIR. arXiv:2506.02680. [10] Webber, G., Denker, A., Barbano, R., and Reader, A. J. (2026). Solving Inverse Problems with Flow-based Models via Model Predictive Control. arXiv:2601.23231.
Questions for authors
-
Reference regulation versus simpler conditioning. Could the authors compare the full method with a baseline using the same processed reference, data-consistency updates, and continuation rule, but a tuned constant image-guidance scale and no attention modifications? For the attention component, how does early blocking compare with simply reducing text guidance or removing the prompt-content tokens during the same window?
-
The framing of the contribution is not clear to me.
6 comments
W2. "… the MPC baseline omits the control-energy penalty from the original formulation, but no comparison is provided … RefReg receives processed references and image-specific captions, whereas GDP and DMPlug receive raw measurement references and a generic prompt. … does not cleanly separate improvements in the reconstruction and operator-estimation algorithms from improvements in their inputs."
RefReg remains better than MPC in all metrics with or without the control-energy penalty (Table R3), and remains best among partially blind methods when GDP and DMPlug receive the same captions and reference and a matched initialization (Table R4).
MPC. Table 1 reports MPC with . We restored the penalty and swept , which spans the range of the original paper's experiment. For each dataset–task pair we selected the with the lowest mean LPIPS on the first 20 evaluation images (/ for SR×8, and / for Gaussian deblurring on DIV2K/FFHQ). Table R3 compares the selected with the control on the same 100 images.
Table R3. MPC without () and with the control-energy penalty vs. RefReg, 100 images per dataset.
| Setting | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| DIV2K SR×8 | MPC () | 22.273 | 0.5555 | 0.4028 |
| MPC () | 22.645 | 0.5835 | 0.4198 | |
| RefReg | 23.018 | 0.6048 | 0.2626 | |
| DIV2K GD | MPC () | 20.758 | 0.4981 | 0.3969 |
| MPC () | 20.763 | 0.4985 | 0.3967 | |
| RefReg | 22.152 | 0.5664 | 0.3213 | |
| FFHQ SR×8 | MPC () | 28.521 | 0.7681 | 0.3093 |
| MPC () | 27.411 | 0.7737 | 0.2671 | |
| RefReg | 29.844 | 0.8075 | 0.1651 | |
| FFHQ GD | MPC () | 26.919 | 0.7555 | 0.2564 |
| MPC () | 26.919 | 0.7555 | 0.2564 | |
| RefReg | 28.940 | 0.7871 | 0.2042 |
Partially blind. The references were already identical across the three methods: all use the measurement clipped to . RefReg's construction starts from it and returns the lowest-residual iterate, which under the identity initial operator (a delta kernel; ) is this starting point (now stated in Appendix D.4). We re-ran GDP and DMPlug with RefReg's image-specific captions, the same measurement and initial noise, and matched initialization, keeping their guidance scales and learning rates fixed to the Appendix D.4 values, and RefReg remains best in every metric (Table R4); since DMPlug's softmax kernel cannot be an exact delta, we report both a near-delta start (99% center mass) and its native random start.
Table R4. Partially blind reconstruction, matched inputs, 100 DIV2K images.
| Task | Method | PSNR↑ | LPIPS↓ | Operator error↓ |
|---|---|---|---|---|
| GD | GDP | 19.08 | 0.322 | 1.983 |
| DMPlug (random init) | 19.17 | 0.397 | 0.894 | |
| DMPlug (near-delta init) | 20.29 | 0.370 | 0.931 | |
| RefReg | 23.71 | 0.283 | 0.175 | |
| HDR | GDP | 20.38 | 0.170 | 0.524 |
| DMPlug | 20.55 | 0.163 | 0.636 | |
| RefReg | 24.67 | 0.077 | 0.170 |
The Illusion of Beating All Baselines
It is often NOT a good sign that "We ran both requested comparisons, and RefReg outperforms every simpler alternative we tested." It could mean the following: 1. benchmarks and settings were biased towards your method, 2. the baselines were not tuned appropriately, 3. I don't understand what simpler implies here. I would expect baselines not to be simple but to be competitive versus your method. Also, I am personally not a huge fan of papers that are phrased in the way, we are the best by far, there's often not true.
On a different note, do the authors ablate against wrong prompts to show the deteriorations?
Can also the author comments on the following works, it seems that something similar is done here:
- TReg — Regularization by Texts for Latent Diffusion Inverse Solvers · ICLR 2025
- ContextMRI — Enhancing Compressed Sensing MRI through Metadata Conditioning · Medical Image Analysis, online 2026
- CFG++ — Manifold-constrained Classifier Free Guidance for Diffusion Models · ICLR 2025 Look at Section 4.3
- P2L — Prompt-tuning Latent Diffusion Models for Inverse Problems · ICML 2024
Are those works cited and commented in the paper and evaluated against?
We thank the reviewer for the careful assessment, and for recognizing the lowest LPIPS in 15 of the 16 dataset–task combinations, the consistent perceptual gains on additional backbones, and the attention-timing experiment as evidence that the timing of conditioning matters.
Our contribution is the use of pretrained IT2I models as multimodal priors for inverse problems, and RefReg, a training-free framework that regulates reference influence during sampling. It lies in this conditioning, not in the basic sampling structure shared with DDS, FlowDPS, ECI, and PCFM. The insight beyond combining known components is when regulation must act: in IT2I priors, whether a generation uses the reference is decided in the first sampling steps (Sec. 3.1; three further backbones in Appendix B), so supplying a reference, or regulating it later, is not enough (Table A8).
A measurement-derived reference supplies spatial structure that text prompts typically leave unspecified, but how its influence should be regulated alongside text conditioning for inverse problems remains underexplored. The design follows from this observation: text-dominated runs show weaker reference influence from the first steps, which motivates stronger early image guidance (IGS) and early generation-to-text blocking (EAB). Ge et al. analyze attention pathways in generation; we use a timed intervention for reconstruction, where only early blocking helps (Table A8). The added value of this design over simpler conditioning is measured directly in Q1 below (Tables R1, R2).
We ran both requested comparisons, and RefReg outperforms every simpler alternative we tested. The full method improves all three metrics over every tested constant image-guidance scale with no attention modifications (Table R1). During the same window, EAB outperforms reducing text guidance (sT∈{0.5,1,2} instead of 4) and removing the prompt-content tokens (Table R2).
Table R1. Comparison with constant image guidance and no attention modifications on random inpainting (90%), 100 DIV2K images.
Table R2. Comparison with text-conditioning alternatives during the same window on motion deblurring. [DIV2K, 100 images]