acceptodds
Under review as a conference paper at ICLR 2027

Reference Regulation in In-Context Image Generation for Inverse Problems

Abstract

Pretrained in-context image generation models, which we refer to as image/text-to-image (IT2I) models, accept a reference image alongside a text prompt. Given a reference derived from a degraded measurement, such a model acts as an amortized, measurement-dependent prior that supplies the spatial structure a text prompt leaves unspecified, and data-consistency updates correct it with the exact likelihood. The measurement enters twice, and how much the prior is trusted becomes a design choice. In native IT2I generation from degraded references, we observe early structure preservation, where the observed layout appears at the first step and later steps refine details, and reduced reference influence, where generations that depart from the observed layout respond less to the reference from the start. A reference alone is thus not enough. We propose RefReg, a posterior sampler that regulates reference influence at three levels: constructing a measurement-derived reference, scheduling reference guidance to emphasize structure early and detail later, and suppressing generation-to-text attention during early structure formation. To compare priors on a common footing, we derive an upper bound on the average posterior approximation error of a conditional prior under data-consistency updates. By this criterion, image conditioning lowers the error bound throughout sampling in generations that preserve the observed layout on IT2I backbones. These results make a case for building inverse solvers on IT2I rather than text-only priors. Across linear, nonlinear, and partially blind inverse problems against flow-based solvers, RefReg improves perceptual quality with competitive fidelity and remains robust with as few as eight sampling steps.

Primary area:
generative models
Keywords:
image/text-to-image (IT2I) models, inverse problems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gw8WOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1(−1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RefReg, a training-free approach to image inverse problems using pre-trained image/text-to-image generators. A reference image is constructed from the measurement and supplied to the generator alongside a text prompt. The method then schedules image-conditioning strength and suppresses selected generation-to-text attention connections during early sampling. These interventions are combined with data-consistency updates to the predicted clean image. The motivation is that reference information should establish the observed scene structure early, while later steps refine its details.

However, I currently lean towards rejection. My main reservation is that the paper has not yet demonstrated a sufficiently clear advance over simpler, carefully tuned conditioning strategies.

Critical strengths
  1. The reconstruction results are strong across a substantial evaluation. RefReg achieves the lowest LPIPS in 15 of the 16 dataset–task combinations in Table A6. The improvements are not restricted to one degradation, and the additional-backbone experiments show consistent perceptual gains. This is the strongest aspect of the paper: there is useful empirical evidence that the proposed combination works well for image restoration

  2. The attention-timing experiment provides some support for the design. I found the comparison between early, middle, and late attention blocking useful. Applying the same intervention early improves reconstruction, whereas applying it later gives little benefit. This goes beyond simply removing a component and provides evidence that the timing of conditioning matters.

Critical weaknesses
  1. I am not convinced that the methodological contribution is sufficiently established. The contribution is primarily a particular way of configuring and combining reference preprocessing, classifier-free guidance, and attention control around a clean-estimate correction procedure. Equation (7) explicitly adopts an existing two-scale guidance formulation, while the reference construction uses TV regularisation and measurement updates. Combining established ingredients can certainly be valuable, but the paper needs a stronger comparison with simpler alternatives to establish the significance of its particular design.

  2. Several choices in the baseline comparisons make the reported advantages difficult to interpret. Appendix D.3 states that the MPC baseline omits the control-energy penalty from the original formulation, but no comparison is provided to establish the effect of that change. In the partially blind experiments, RefReg receives processed references and image-specific captions, whereas GDP and DMPlug receive raw measurement references and a generic prompt. These conditioning differences are particularly consequential in a paper whose central premise is that conditioning substantially affects reconstruction. The comparison therefore does not cleanly separate improvements in the reconstruction and operator-estimation algorithms from improvements in their inputs.

  3. The methodological advance over existing sampling and conditioning techniques is not sufficiently demonstrated. Algorithm 1 follows the familiar pattern of predicting a clean image, correcting it for measurement consistency, and using the corrected estimate to continue sampling. Closely related constructions appear in DDS [1] and, more directly for flow models, FlowDPS [2]. ECI [3] explicitly develops an extrapolate–correct–interpolate procedure, while PCFM [4] extends constraint correction to nonlinear constraints in pre-trained flows. These methods differ in their correction operators and treatment of noise, but the basic sampling structure is not new. The conditioning components also have clear precedents. The separate image and text guidance scales in Equation (7) follow InstructPix2Pix [5], as the authors acknowledge. Moreover, Ge et al. [6] already investigate how reference information passes through image and text tokens in FLUX.2 using interventions on attention connections. RefReg applies these ideas differently but I would like to see stronger evidence that this specific intervention offers a substantial advantage over simpler conditioning strategies. My concern is therefore not that the paper fails to cite all relevant work, but that the additional contribution appears to be a particular combination and scheduling of established components. The reconstruction results are encouraging, but the attention intervention is not compared with simply reducing or removing prompt-content conditioning during the same early window.

References to look: [1]–[6] are the strongest references.

[1] Chung, H., Lee, S., and Ye, J. C. (2024). Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems. ICLR. [2] Kim, J., Kim, B. S., and Ye, J. C. (2025). FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems. ICCV. [3] Cheng, C., et al. (2025). Gradient-Free Generation for Hard-Constrained Systems. ICLR. [4] Utkarsh, U., et al. (2025). Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints. NeurIPS. [5] Brooks, T., Holynski, A., and Efros, A. A. (2023). InstructPix2Pix: Learning to Follow Image Editing Instructions. CVPR. [6] Ge, C., Gandikota, R., Torralba, A., and Rott Shaham, T. (2026). Vision-Language Binding in In-Context Image Generation. arXiv:2605.24624. [7] Yılmaz, M. A., Bilican, A., Biner, B. C., and Tekalp, A. M. (2026). Edit2Restore: Few-Shot Image Restoration via Parameter-Efficient Adaptation of Pre-trained Editing Models. arXiv:2601.03391. [8] Song, R., et al. (2026). Training-Free Image Editing with Visual Context Integration and Concept Alignment. arXiv:2604.04487. [9] Erbach, J., et al. (2025). Solving Inverse Problems with FLAIR. arXiv:2506.02680. [10] Webber, G., Denker, A., Barbano, R., and Reader, A. J. (2026). Solving Inverse Problems with Flow-based Models via Model Predictive Control. arXiv:2601.23231.

Questions for authors
  1. Reference regulation versus simpler conditioning. Could the authors compare the full method with a baseline using the same processed reference, data-consistency updates, and continuation rule, but a tuned constant image-guidance scale and no attention modifications? For the attention component, how does early blocking compare with simply reducing text guidance or removing the prompt-content tokens during the same window?

  2. The framing of the contribution is not clear to me.

6 comments

We thank the reviewer for the careful assessment, and for recognizing the lowest LPIPS in 15 of the 16 dataset–task combinations, the consistent perceptual gains on additional backbones, and the attention-timing experiment as evidence that the timing of conditioning matters.


Q2. "The framing of the contribution is not clear to me."

W1. "… the paper needs a stronger comparison with simpler alternatives to establish the significance of its particular design."

W3. "… the basic sampling structure is not new. … Ge et al. [6] already investigate … using interventions on attention connections. … the additional contribution appears to be a particular combination and scheduling of established components. …"

Our contribution is the use of pretrained IT2I models as multimodal priors for inverse problems, and RefReg, a training-free framework that regulates reference influence during sampling. It lies in this conditioning, not in the basic sampling structure shared with DDS, FlowDPS, ECI, and PCFM. The insight beyond combining known components is when regulation must act: in IT2I priors, whether a generation uses the reference is decided in the first sampling steps (Sec. 3.1; three further backbones in Appendix B), so supplying a reference, or regulating it later, is not enough (Table A8).

A measurement-derived reference supplies spatial structure that text prompts typically leave unspecified, but how its influence should be regulated alongside text conditioning for inverse problems remains underexplored. The design follows from this observation: text-dominated runs show weaker reference influence from the first steps, which motivates stronger early image guidance (IGS) and early generation-to-text blocking (EAB). Ge et al. analyze attention pathways in generation; we use a timed intervention for reconstruction, where only early blocking helps (Table A8). The added value of this design over simpler conditioning is measured directly in Q1 below (Tables R1, R2).


Q1. "Could the authors compare the full method with a baseline using the same processed reference, data-consistency updates, and continuation rule, but a tuned constant image-guidance scale and no attention modifications? … how does early blocking compare with simply reducing text guidance or removing the prompt-content tokens during the same window?"

We ran both requested comparisons, and RefReg outperforms every simpler alternative we tested. The full method improves all three metrics over every tested constant image-guidance scale with no attention modifications (Table R1). During the same window, EAB outperforms reducing text guidance (sT∈{0.5,1,2}s_T\in\{0.5, 1, 2\} instead of 4) and removing the prompt-content tokens (Table R2).

Table R1. Comparison with constant image guidance and no attention modifications on random inpainting (90%), 100 DIV2K images.

MethodsIs_IPSNR ↑SSIM ↑LPIPS ↓
Constant0.522.160.5830.333
1.022.740.6160.299
1.523.380.6520.276
2.023.600.6660.280
2.523.610.6680.297
3.023.610.6710.301
IGS+EAB(Full)3.5→024.660.7440.165

Table R2. Comparison with text-conditioning alternatives during the same window on motion deblurring. [DIV2K, 100 images]

SettingPSNR ↑SSIM ↑LPIPS ↓
No intervention (w/o EAB, Table 4)27.180.7100.191
Reduced text guidance (sT=2s_T=2)26.010.6200.280
Reduced text guidance (sT=1s_T=1)25.220.5690.337
Reduced text guidance (sT=0.5s_T=0.5)24.980.5540.354
Removing prompt-content tokens27.420.7200.177
EAB (RefReg)27.630.7350.156

W2. "… the MPC baseline omits the control-energy penalty from the original formulation, but no comparison is provided … RefReg receives processed references and image-specific captions, whereas GDP and DMPlug receive raw measurement references and a generic prompt. … does not cleanly separate improvements in the reconstruction and operator-estimation algorithms from improvements in their inputs."

RefReg remains better than MPC in all metrics with or without the control-energy penalty (Table R3), and remains best among partially blind methods when GDP and DMPlug receive the same captions and reference and a matched initialization (Table R4).

MPC. Table 1 reports MPC with ρ=0\rho=0. We restored the penalty and swept ρ∈{10−10,10−9,10−8,10−7,10−6,10−5,10−4,5×10−4}\rho\in\{10^{-10},10^{-9}, 10^{-8},10^{-7},10^{-6},10^{-5},10^{-4},5\times10^{-4}\}, which spans the range of the original paper's experiment. For each dataset–task pair we selected the ρ\rho with the lowest mean LPIPS on the first 20 evaluation images (10−610^{-6}/10−510^{-5} for SR×8, and 10−910^{-9}/10−1010^{-10} for Gaussian deblurring on DIV2K/FFHQ). Table R3 compares the selected ρ\rho with the ρ=0\rho=0 control on the same 100 images.

Table R3. MPC without (ρ=0\rho=0) and with the control-energy penalty vs. RefReg, 100 images per dataset.

SettingMethodPSNR ↑SSIM ↑LPIPS ↓
DIV2K SR×8MPC (ρ=0\rho=0)22.2730.55550.4028
MPC (ρ=10−6\rho=10^{-6})22.6450.58350.4198
RefReg23.0180.60480.2626
DIV2K GDMPC (ρ=0\rho=0)20.7580.49810.3969
MPC (ρ=10−9\rho=10^{-9})20.7630.49850.3967
RefReg22.1520.56640.3213
FFHQ SR×8MPC (ρ=0\rho=0)28.5210.76810.3093
MPC (ρ=10−5\rho=10^{-5})27.4110.77370.2671
RefReg29.8440.80750.1651
FFHQ GDMPC (ρ=0\rho=0)26.9190.75550.2564
MPC (ρ=10−10\rho=10^{-10})26.9190.75550.2564
RefReg28.9400.78710.2042

Partially blind. The references were already identical across the three methods: all use the measurement clipped to [−1,1][-1,1]. RefReg's construction starts from it and returns the lowest-residual iterate, which under the identity initial operator (a delta kernel; s=1s=1) is this starting point (now stated in Appendix D.4). We re-ran GDP and DMPlug with RefReg's image-specific captions, the same measurement and initial noise, and matched initialization, keeping their guidance scales and learning rates fixed to the Appendix D.4 values, and RefReg remains best in every metric (Table R4); since DMPlug's softmax kernel cannot be an exact delta, we report both a near-delta start (99% center mass) and its native random start.

Table R4. Partially blind reconstruction, matched inputs, 100 DIV2K images.

TaskMethodPSNR↑LPIPS↓Operator error↓
GDGDP19.080.3221.983
DMPlug (random init)19.170.3970.894
DMPlug (near-delta init)20.290.3700.931
RefReg23.710.2830.175
HDRGDP20.380.1700.524
DMPlug20.550.1630.636
RefReg24.670.0770.170
Reviewer_gw8WOpenReview ↗

The Illusion of Beating All Baselines

It is often NOT a good sign that "We ran both requested comparisons, and RefReg outperforms every simpler alternative we tested." It could mean the following: 1. benchmarks and settings were biased towards your method, 2. the baselines were not tuned appropriately, 3. I don't understand what simpler implies here. I would expect baselines not to be simple but to be competitive versus your method. Also, I am personally not a huge fan of papers that are phrased in the way, we are the best by far, there's often not true.

On a different note, do the authors ablate against wrong prompts to show the deteriorations?

Can also the author comments on the following works, it seems that something similar is done here:

  1. TReg — Regularization by Texts for Latent Diffusion Inverse Solvers · ICLR 2025
  2. ContextMRI — Enhancing Compressed Sensing MRI through Metadata Conditioning · Medical Image Analysis, online 2026
  3. CFG++ — Manifold-constrained Classifier Free Guidance for Diffusion Models · ICLR 2025 Look at Section 4.3
  4. P2L — Prompt-tuning Latent Diffusion Models for Inverse Problems · ICML 2024

Are those works cited and commented in the paper and evaluated against?

Reviewer_VLWMOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on solving inverse problems with multimodal diffusion priors, namely in-context image/text-to-image (IT2I) models such as FLUX.2 and FLUX.1 Kontext, which accept a reference image (which for inverse problems would be the measurement) together with a text prompt. In particular, the paper proposes RefReg, which (i) builds the reference by a TV-regularized data-fidelity optimization, (ii) schedules the image guidance (strong early, weak later), and (iii) blocks generation-to-text attention during the first steps. At every noise level both the clean image and the noise are predicted from the velocity, the clean image is corrected with a few gradient steps on a data-consistency loss, and the two are re-interpolated. The paper also derives a W2 upper bound on the posterior approximation error (Proposition 1), used to argue that IT2I priors are better than their text-only counterparts. Numerical experiments on eight linear and nonlinear tasks (DIV2K and FFHQ), partially blind problems, and four additional backbones validate the method.

Critical strengths
  1. The paper does a good job exploring multimodal priors for inverse problems; while using Flux is not new, exploiting both text and reference image is underexplored in general.
  2. Numerical results are strong and nicely presented. RefReg obtains the best LPIPS and FID in most settings, and the gains hold across additional backbones, a distilled model, and with few NFEs.
  3. The analysis of native IT2I generation in Section 3.1 (early structure preservation, reference- vs. text-dominated runs) is interesting, and it motivates each component of the method well. The ablations (Table 4, Tables A7–A8) support each component.
Critical weaknesses
  1. The theory oversells the claim. Proposition 1 uses the synchronous coupling of Dalalyan [1], later used for W2 bounds of diffusion models (e.g., [2]). In this technique, the sampler is coupled (same Brownian motion) with the exact reverse process, whose drift depends only on the current state, i.e., atx−ct E[X0∣Xt=x,Y]a_t x - c_t\,\mathbb E[X_0 \mid X_t = x, Y] in the notation of the paper (in Dalalyan's analysis, the reference is the Langevin diffusion, whose drift is the score and which starts at its stationary distribution, the target). Lipschitz and log-concavity assumptions are then needed to control how the two trajectories diverge, and for an exact sampler the bound vanishes (up to discretization).

    Here, the reference process ZZ of Lemma A.1 has drift atx−ctX0a_t x - c_t X_0: the posterior mean is replaced by the ground truth X0X_0. ZZ has the correct marginals, but it is pinned to X0X_0 (in Corollary 1, it is just the interpolant (1−t)X0+tX1(1-t)X_0 + tX_1). Since this drift is affine, the difference of the two processes solves a linear ODE (A.8), and this is the only reason why no Lipschitz assumption is needed. I believe this comes at a high price. To put it more concretely, Theorem 5 in [1] reads

    W2(νK,π)≤(1−mh)K W2(ν0,π)⏟initialization+O(h) terms⏟discretization+O(δ,σ) terms⏟gradient error,W_2(\nu_K,\pi) \le \underbrace{(1-mh)^K\, W_2(\nu_0,\pi)}_{\text{initialization}} + \underbrace{O(h)\ \text{terms}}_{\text{discretization}} + \underbrace{O(\delta, \sigma)\ \text{terms}}_{\text{gradient error}},

    Every term vanishes with exact gradients, a small step size, and enough iterations, so the bound is informative about the sampler. This is not the case for Proposition 1. Because the reference is pinned to X0X_0, the bound measures how far the sampler is from the ground truth X0X_0. But the sampler never sees X0X_0, only YY (Assumption A.2), so even a perfect posterior sampler outputs a sample that differs from X0X_0 by about the posterior spread.

Formally, RM(t)≥E tr Cov(X0∣Y)R_M(t) \ge \mathbb E\,\mathrm{tr}\,\mathrm{Cov}(X_0 \mid Y) for every sampler and every tt, so in Corollary 1 even an exact sampler gets a bound of at least (1−τ0)E tr Cov(X0∣Y)(1-\tau_0)\sqrt{\mathbb E\,\mathrm{tr}\,\mathrm{Cov}(X_0 \mid Y)}.

Furthermore, the bound rewards mode collapse: a sampler that always outputs E[X0∣Y]\mathbb E[X_0 \mid Y] has RM=tr Cov(X0∣Y)R_M = \mathrm{tr}\,\mathrm{Cov}(X_0 \mid Y), while a perfect posterior sampler, which outputs an independent posterior sample, has about twice this value. So the mode-collapsed sampler gets the better bound (Remark 1). Remark A.1 acknowledges that the coupling is not optimal, but the issue is more severe: a smaller RMR_M (Remark 1, Figure 4) does not mean a better posterior approximation.

Finally, Figure 4 does not test the proposition. Proposition 1 holds on average over all runs, but Figure 4 uses a biased split: a run is reference-dominated when IT2I is already at least 1 dB more consistent with yy than T2I. A lower RMR_M for IT2I in that group is therefore expected by construction, and it says nothing about the average over all runs, which is what the proposition and Remark 1 compare (in text-dominated runs, the gain is "much smaller or absent"). Could the authors report ΔRrel(t)\Delta R_{\mathrm{rel}}(t) averaged over all runs?

Overall, Proposition 1 is technically correct, but it does not support the claim that IT2I priors yield better posterior sampling than T2I priors: Figure 4 essentially compares the denoising errors of the two priors, which does not require the proposition. I would ask the authors to clarify this point, and, if my reading is correct, to either remove the proposition or present it as a heuristic motivation rather than a posterior-sampling guarantee. I am happy to be corrected if I missed something.

  1. Early generation-to-text attention control is heavy engineering. I believe this is a weakness of the method, as it relies on many design choices: a bias of −30 on selected prompt keys, an exclusion of the quality-suffix and padding embeddings, text-to-text attention control, and an extra bias for box inpainting. It is also backbone-specific, as acknowledged in the Limitations. This seems difficult to apply in practice in plug-and-play methods, where one of the keys is that prior is easy to combine with a data term for several degradations H.

Also, it would be good to have the running times of all methods. Currently, it's difficult to understand if RefReg is improving performance at the cost of larger running time.

Questions for authors
  1. Why is the relative velocity change rtr_t a relevant metric to assess whether an IT2I model is a better prior for restoration than a T2I model? A large rtr_t only says that the reference changes the velocity, which could also mean copying the degradation. Similarly, why is the attention share ρt\rho_t a measure of the influence of the reference?

  2. In Algorithm 1 (line 6), the noise endpoint x^1∣t=xt+(1−t)v~\hat x_{1|t} = x_t + (1-t)\tilde v is computed from the velocity and kept fixed, while the clean endpoint is replaced by the corrected estimate x~0∣t\tilde x_{0|t}. After the correction, the pair (x~0∣t,x^1∣t)(\tilde x_{0|t}, \hat x_{1|t}) is no longer consistent with the model prediction. Why is this preferable to re-noising with fresh or partially fresh noise, as in FlowDPS and TriPS? Could the authors ablate this choice? Note that it also makes the sampler deterministic given the initial noise.

  3. The measurement enters twice, through the reference cI=RA(y)c_I = R_A(y) and through the data-consistency updates. This is not typical in plug-and-play methods, so what is the algorithm targeting: the posterior, a MAP estimate, or something else?

  4. PnP-Flow and OT-ODE [3, 4] are missing, and also some of the baselines in the paper are cheaper to run. How do the baselines perform with a matched budget? This is mainly to understand where RefReg is paying a cost.

[1] A. S. Dalalyan and A. Karagulyan. User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient. Stochastic Processes and their Applications, 129(12):5278–5311, 2019.

[2] X. Gao, H. M. Nguyen, and L. Zhu. Wasserstein convergence guarantees for a general class of score-based generative models. Journal of Machine Learning Research, 26, 2025.

[3] S. Martin, A. Gagneux, P. Hagemann, and G. Steidl. PnP-Flow: Plug-and-play image restoration with flow matching. In ICLR, 2025.

[4] A. Pokle, M. J. Muckley, R. T. Q. Chen, and B. Karrer. Training-free linear image inverses via flows. Transactions on Machine Learning Research, 2024.

5 comments

We thank the reviewer for the careful review, and for recognizing the exploration of multimodal priors for inverse problems, the strong numerical results across backbones and few NFEs, and the analysis of native IT2I generation in Sec. 3.1.


W1-1. "The theory oversells the claim. … Because the reference is pinned to X0X_0, the bound measures how far the sampler is from the ground truth X0X_0. … Formally, RM(t)≥E tr Cov(X0∣Y)R_M(t)\ge\mathbb{E}\,\mathrm{tr}\,\mathrm{Cov}(X_0\mid Y) for every sampler and every tt, so in Corollary 1 even an exact sampler gets a bound of at least (1−τ0)E tr Cov(X0∣Y)(1-\tau_0)\sqrt{\mathbb{E}\,\mathrm{tr}\,\mathrm{Cov}(X_0\mid Y)}."

The bound indeed has this floor, but it is the posterior variance, common to the IT2I and T2I priors: it cancels in the difference of the risks, ΔR\Delta R, so the floor itself favors neither prior. The comparison still orders upper bounds, not posterior errors, as Remark 1 states. This limit of the bound stands; separately, the decomposition below (W1-2) reports what the observed reduction consists of. Since the sampler is independent of X0X_0 given YY, RM(t)=bM(t)+VM(t)+σpost2R_M(t)=b_M(t)+V_M(t)+\sigma^2_{\text{post}}: the squared distance between the mean of the corrected estimates and the posterior mean (bias), their variance over initial noises, and the posterior variance σpost2=E tr Cov(X0∣Y)\sigma^2_{\text{post}}=\mathbb{E}\,\mathrm{tr}\,\mathrm{Cov}(X_0\mid Y).


W1-2. "Furthermore, the bound rewards mode collapse: … a smaller RMR_M (Remark 1, Figure 4) does not mean a better posterior approximation."

The reviewer's example shows a limit of the criterion that holds in general, and our analysis does not remove it; it shows that our runs are not such a case. At the final sampling step, the smaller bias accounts for 69% (FLUX.2) and 57% (FLUX.1 Kontext) of the reduction in RMR_M, and the estimated ratio VM/(bM+σpost2)V_M/(b_M+\sigma^2_{\text{post}}), which lower-bounds VM/σpost2V_M/\sigma^2_{\text{post}}, is 0.96 and 0.60 (Table R1, last column). In the example, by contrast, bM=0b_M=0 for both samplers and only VMV_M decreases, down to 0. Because σpost2\sigma^2_{\text{post}} cancels between the priors, the bias difference is obtained as Δb=ΔR−ΔV\Delta b=\Delta R-\Delta V, without estimating the posterior mean. We report this as a description of the observed reduction and claim no posterior calibration from it.

Table R1. All-run comparison at the final sampling step. Each difference is IT2I minus T2I, normalized by RT2IR_{\mathrm{T2I}}; the bias and variance differences sum to the total. The last column is a lower bound on VM/σpost2V_M/\sigma^2_{\text{post}}.

BackboneΔRrel\Delta R_{\mathrm{rel}}Δbrel\Delta b_{\mathrm{rel}} (bias)ΔVrel\Delta V_{\mathrm{rel}} (variance)VM/(bM+σpost2)V_M/(b_M+\sigma^2_{\text{post}}), IT2I
FLUX.2−0.166−0.115−0.0510.96
FLUX.1 Kontext−0.354−0.202−0.1520.60

W1-3. "Finally, Figure 4 does not test the proposition. … Figure 4 uses a biased split: … Could the authors report ΔRrel(t)\Delta R_{\mathrm{rel}}(t) averaged over all runs?"

We added ΔRrel(t)\Delta R_{\mathrm{rel}}(t) averaged over all 500 runs, without the split, to Fig. 4. It is negative at every sampling time on both backbones, so the sufficient condition of Remark 1 holds on the whole sampling grid and the IT2I prior has the smaller bound over all runs.


W1-4. "Overall, Proposition 1 is technically correct, but it does not support the claim that IT2I priors yield better posterior sampling than T2I priors: Figure 4 essentially compares the denoising errors of the two priors, which does not require the proposition. … either remove the proposition or present it as a heuristic motivation rather than a posterior-sampling guarantee. …"

We keep Proposition 1: it is a proved bound that links the corrected-estimate error, evaluated along each sampler's own trajectory, to the posterior approximation error, and we use it to compare priors, not as a posterior-sampling guarantee. Measuring RM(t)R_M(t) does not require the proposition; this link does. Our claim consists of two statements: at every point of the sampling grid, the IT2I prior has the smaller RM(t)R_M(t) over all runs (W1-3), and at the final step this reduction consists mainly of a smaller bias (W1-2). The first is the theoretical contribution, a proved relation between the corrected-estimate risk and a Wasserstein upper bound; the second is empirical and does not remove the limits of the first.

Revision. Sec. 1 (L93–94, L106–107) and Sec. 3.2 (L260–261) state this role of the bound. In the Empirical validation paragraph, "for posterior sampling, the IT2I prior has a clear advantage over the T2I prior throughout sampling" (L286–287) now reads "which gives the IT2I prior the smaller upper bound in these runs." The decomposition follows Remark 1, the all-run curve is in Fig. 4, and the results are in the Empirical validation paragraph of Sec. 3.2.

W2. "Early generation-to-text attention control is heavy engineering. I believe this is a weakness of the method, as it relies on many design choices: a bias of −30-30 on selected prompt keys, an exclusion of the quality-suffix and padding embeddings, text-to-text attention control, and an extra bias for box inpainting. It is also backbone-specific, as acknowledged in the Limitations. This seems difficult to apply in practice in plug-and-play methods, where one of the keys is that prior is easy to combine with a data term for several degradations HH."

Early generation-to-text attention control follows one rule, blocking prompt-content tokens early, with the same setting for all eight degradations HH (first three steps, all layers and heads; Appendix D.2.2), and it leaves the data term unchanged, so the prior is combined with it as in plug-and-play methods. The listed choices implement this rule: −30-30 is a fixed masking constant, not tuned; the auxiliary text controls (embedding replacement, text-to-text bias) keep the blocked content from reaching generation tokens through other text tokens (Appendix C.2); and the box-inpainting bias, the only task-specific addition, uses only the known mask, as inpainting methods do [R7, R8]. Such control is widely used in frozen models [R1–R5], and its dependence on the backbone is not specific to RefReg; for example, injection layers are identified separately for FLUX and SD3 [R6]. Only the implementation differs; the same principle applies across IT2I backbones (Table A13).

References

[R1] A. Hertz et al. Prompt-to-Prompt image editing with cross-attention control. In ICLR, 2023.

[R2] M. Cao et al. MasaCtrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.

[R3] M. Geva et al. Dissecting recall of factual associations in auto-regressive language models. In EMNLP, 2023.

[R4] Y. Tewel et al. Add-it: Training-free object insertion in images with pretrained diffusion models. In ICLR, 2025.

[R5] C. Ge et al. Vision-language binding in in-context image generation. In Mechanistic Interpretability Workshop at ICML, 2026.

[R6] O. Avrahami et al. Stable Flow: Vital layers for training-free image editing. In CVPR, 2025.

[R7] A. Lugmayr et al. RePaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.

[R8] L. Rout et al. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In NeurIPS, 2023.


W3. "Also, it would be good to have the running times of all methods. Currently, it's difficult to understand if RefReg is improving performance at the cost of larger running time."

Q4. "PnP-Flow and OT-ODE [3, 4] are missing, and also some of the baselines in the paper are cheaper to run. How do the baselines perform with a matched budget? This is mainly to understand where RefReg is paying a cost."

RefReg attains the best reconstruction quality at a running time within the range of the baselines (Table R2, now including PnP-Flow and OT-ODE), and remains best when cheaper baselines are given a matched FLOPs budget (Table R3). The cost is paid in image conditioning: at 50 steps, adding the reference-conditioned branches raises the time from 24.1 s to 42.0 s and the FLOPs from 2,339 T to 5,323 T (Table A9).

Table R2. Quality and running time on Gaussian deblurring (σ=5\sigma=5), 50 sampling steps, 100 DIV2K images at 768 × 768. Time is per image on an A100 GPU.

MethodTime(s)PSNR↑SSIM↑LPIPS↓
FlowChef42.621.590.5430.460
Flower45.021.320.4780.475
TriPS35.020.790.4920.378
MPC30.920.760.4980.396
FLAIR41.619.840.4920.383
FlowDPS20.519.180.5170.651
PnP-Flow18.419.480.4970.658
OT-ODE58.520.890.5360.631
RefReg42.022.150.5660.321

Table R3. FLOPs-matched comparison on Gaussian deblurring (σ=5\sigma=5). Each baseline uses approximately the FLOPs of RefReg at 50 steps.

Method (steps)PSNR↑SSIM↑LPIPS↓
FlowDPS (87)19.120.5120.657
MPC (77)20.820.5010.390
PnP-Flow (89)19.550.4940.660
RefReg (50)22.150.5660.321

Q1. "Why is the relative velocity change rtr_t a relevant metric to assess whether an IT2I model is a better prior for restoration than a T2I model? A large rtr_t only says that the reference changes the velocity, which could also mean copying the degradation. Similarly, why is the attention share ρt\rho_t a measure of the influence of the reference?"

rtr_t and ρt\rho_t are not our criteria for a better prior; they only diagnose how much native IT2I generation uses the reference. We agree that a large rtr_t may also reflect copying the degradation, which is why RefReg regulates reference influence rather than maximizing it (strong early image guidance, reduced later). ρt\rho_t is not a causal measure of influence; it only describes how much attention generation queries place on reference tokens relative to text. Influence is measured by rtr_t, which removes the reference at the same state, and both are lower in text-dominated runs from the first step (Fig. 3(b)).


Q2. "In Algorithm 1 (line 6), the noise endpoint x^1∣t=xt+(1−t)v~\hat{x}_{1|t} = x_t + (1-t)\tilde{v} is computed from the velocity and kept fixed, … Why is this preferable to re-noising with fresh or partially fresh noise, as in FlowDPS and TriPS? Could the authors ablate this choice? Note that it also makes the sampler deterministic given the initial noise."

We ablated this choice, and keeping the noise endpoint fixed is best on all three metrics (Table R4). Our rationale is to keep the layout formed in the early steps (Sec. 3.1): with fresh noise, the next state in the early steps is almost pure new noise (its weight is t≈1t\approx1), so the layout has to be formed again at every step, whereas the retained endpoint carries it over; partially fresh noise lies in between. The sampler is then deterministic given the initial noise, which supplies its diversity.

Table R4. Motion deblurring, 100 DIV2K images, 50 NFEs; only the noise endpoint differs.

Noise endpointPSNR ↑SSIM ↑LPIPS ↓
Fresh noise23.880.5360.375
Partially fresh noise (FlowDPS)26.880.7070.172
Kept fixed (ours)27.630.7350.156

Q3. "The measurement enters twice, through the reference cI=RA(y)c_I = R_A(y) and through the data-consistency updates. This is not typical in plug-and-play methods, so what is the algorithm targeting: the posterior, a MAP estimate, or something else?"

The algorithm targets the posterior of the original observation model, πy=L(X0∣Y=y)\pi_y=\mathcal{L}(X_0\mid Y=y), not a MAP estimate. In typical plug-and-play methods, the prior velocity is unconditional or text-conditioned, and the measurement enters only through data consistency. RefReg additionally conditions the velocity on a measurement-derived reference, so the IT2I model acts as a multimodal (image and text) prior, while the data-consistency updates enforce consistency with the original measurement.

Reviewer_EEjaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RefReg applies frozen image-and-text-conditioned flow models to inverse imaging. It constructs a measurement-derived reference, schedules image guidance, suppresses selected generation-to-text attention early, and corrects predicted clean latents for measurement consistency. A Wasserstein bound relates approximation error to trajectory-wise reconstruction MSE. Eight-task and multi-backbone experiments support a practical reference-regulation procedure; the main contribution is inference-time control, rather than a new posterior-sampling principle.

Critical strengths
  1. The timing claim has direct experimental support. Table A8 holds intervention duration, guidance, and other controls fixed while comparing early, middle, and late attention blocking. Early blocking improves both PSNR and LPIPS on two tasks; later blocking produces negligible gains. Table 4 independently tests components, and Table A7 isolates reference optimization. These interventions support a specific inference-time design insight beyond the established conditioning, guidance, and correction ingredients [R1-R4].

  2. The end-to-end gains survive meaningful controls. The main six-baseline comparisons share the FLUX.2 backbone, resolution, prompts, and measurement-derived reference. Across Tables 1/A6, improvements extend beyond a single task; for DIV2K SR $\times$8, LPIPS improves from FLAIR's 0.348 to 0.266 with PSNR increasing from 22.73 to 23.01 dB. Table A11 extends perceptual gains to nine additional backbone-task combinations. Table A9 reports actual runtime/FLOPs, not only sampling steps.

[R1] Black Forest Labs et al. (2025). FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space.

[R2] Brooks, Holynski, and Efros (CVPR 2023; preprint 2022). InstructPix2Pix: Learning to Follow Image Editing Instructions.

[R3] Kim, Kim, and Ye (ICCV 2025). FlowDPS: Flow-Driven Posterior Sampling for Inverse Problems.

[R4] Ge et al. (2026). Vision-Language Binding in In-Context Image Generation.

Critical weaknesses
  1. The contribution is narrower than its posterior-sampling framing. Reference conditioning, dual guidance, attention intervention, and clean-estimate correction are established; novelty lies mainly in their timing and integration. Proposition 1 orders MSE-driven upper bounds, not actual posterior errors, as the authors acknowledge. Under Appendix A's independent sampling setup, that MSE rewards proximity to the posterior mean and need not reward calibrated posterior diversity. Moreover, Appendix C.3 uses unsquared back-projection/residual norms rather than Gaussian negative log-likelihood, contradicting the abstract's 'exact likelihood' wording. The bound is not thereby invalid, but cannot substantiate improved posterior accuracy.

  2. Remaining controls limit attribution. Table 4 compares scheduled image guidance only with sI=1s_I=1, leaving its advantage over a tuned constant scale unresolved. Appendix D.3 omits MPC's control-energy penalty [R5], a substantive objective change requiring validation. For partially blind tasks, RefReg receives image-specific captions and a constructed reference, whereas GDP/DMPlug receive a generic prompt and raw measurement (D.4). These differences prevent attributing Table 3's gains solely to the solver. They qualify specific claims rather than invalidating all matched main comparisons or the independent attention-timing evidence.

[R5] Webber, Denker, Barbano, and Reader (ICML 2026). Solving Inverse Problems with Flow-based Models via Model Predictive Control.

Questions for authors
  1. Does scheduled image guidance outperform the best constant scale with the same reference, correction budget, and EAB settings? This would distinguish the proposed timing effect from guidance strength alone.

  2. Can the authors validate the MPC adaptation with its control penalty restored and compare partially blind methods using identical captions, reference preparation, and available initialization information? Which performance conclusions survive these controls?

  3. What posterior target is intended, and how should 'exact likelihood' be reconciled with Eq. (C.18)? Please distinguish reconstruction-risk evidence from posterior calibration, and report how conditional branches and inner corrections enter NFE/compute accounting.

4 comments

We thank the reviewer for the constructive review, and for recognizing that our controlled interventions support a specific inference-time design insight and that the end-to-end gains survive meaningful controls.


W1-1. "The contribution is narrower than its posterior-sampling framing. Reference conditioning, dual guidance, attention intervention, and clean-estimate correction are established; novelty lies mainly in their timing and integration."

The individual ingredients are established, but the contribution is not limited to their timing and integration: it is the use of pretrained IT2I models as multimodal priors for inverse problems, which remains underexplored. We analyze how such a prior uses a degraded reference (Sec. 3.1), derive a bound for comparing it with its text-only counterpart (Proposition 1; see W1-2), and regulate the reference accordingly (Sec. 4.2), with the timing validated by controlled interventions (Tables 4, A8).


W1-2. “Proposition 1 orders MSE-driven upper bounds, not actual posterior errors, as the authors acknowledge. Under Appendix A’s independent sampling setup, that MSE rewards proximity to the posterior mean and need not reward calibrated posterior diversity. … The bound is not thereby invalid, but cannot substantiate improved posterior accuracy.”

At the final sampling step, the lower RMR_M of the IT2I prior comes for the larger part from a smaller error of the mean of the estimates and for the smaller part from reduced variance, and the estimates do not concentrate on the posterior mean (decomposition below). Proposition 1 itself orders upper bounds, as Remark 1 states: its role is to relate the corrected-estimate error RM(t)R_M(t) to a Wasserstein bound. The decomposition describes the observed reduction; it does not assess calibrated diversity (Q3-3).

Since the sampler is independent of X0X_0 given YY, RM=bM+VM+σpost2R_M=b_M+V_M+\sigma^2_{\text{post}}. Here bMb_M is the expected squared distance between the mean of the corrected estimates and the posterior mean (at the final step, zero for an exact sampler), VMV_M is the variance of the corrected estimates over initial noises (their diversity), and σpost2=E tr Cov(X0∣Y)\sigma^2_{\text{post}}=\mathbb{E}\,\mathrm{tr}\,\mathrm{Cov}(X_0\mid Y) is the posterior variance, common to both priors. Because σpost2\sigma^2_{\text{post}} cancels between the priors, the bias difference is obtained as Δb=ΔR−ΔV\Delta b=\Delta R-\Delta V, without estimating the posterior mean. Over all 500 runs of the revised Fig. 4, at the final sampling step, the decrease in bMb_M accounts for 69% (FLUX.2) and 57% (FLUX.1 Kontext) of the reduction in RMR_M. Moreover, the estimated ratio VM/(RM−VM)V_M/(R_M-V_M) is 0.96 and 0.60; this ratio lower-bounds VM/σpost2V_M/\sigma^2_{\text{post}} since RM−VM=bM+σpost2≥σpost2R_M-V_M=b_M+\sigma^2_{\text{post}}\ge\sigma^2_{\text{post}}, whereas concentrating estimates on the posterior mean would drive it to 0. The estimates therefore do not collapse onto the posterior mean.

Revision. The decomposition follows Remark 1, and the results are in the Empirical validation paragraph of Sec. 3.2. There, "for posterior sampling, the IT2I prior has a clear advantage over the T2I prior throughout sampling" now reads "which gives the IT2I prior the smaller upper bound in these runs" (L286–287).


W1-3. "Moreover, Appendix C.3 uses unsquared back-projection/residual norms rather than Gaussian negative log-likelihood, contradicting the abstract's 'exact likelihood' wording."

Q3-2. "and how should 'exact likelihood' be reconciled with Eq. (C.18)?"

We corrected the wording: "exact likelihood" was imprecise, since Appendix C.3 uses unsquared back-projection/residual norms, as the reviewer notes. The sentence now reads "… while data-consistency updates correct its estimates against the measurement itself" (L37–38).


W2-1. "Remaining controls limit attribution. Table 4 compares scheduled image guidance only with sI=1s_I = 1, leaving its advantage over a tuned constant scale unresolved. ... They qualify specific claims rather than invalidating ... the independent attention-timing evidence."

Q1. "Does scheduled image guidance outperform the best constant scale with the same reference, correction budget, and EAB settings? This would distinguish the proposed timing effect from guidance strength alone."

Yes. Scheduled image guidance outperforms every tested constant scale in every metric (Table R1). The reference, correction budget, and EAB settings are fixed, and the best constant scales (1.5–2.0) lie inside the tested range.

Table R1. Random inpainting (90%), 100 DIV2K images.

Image guidance sIs_IPSNR ↑SSIM ↑LPIPS ↓
1.024.310.7160.188
1.524.500.7310.187
2.024.500.7400.197
2.524.310.7380.219
3.024.070.7290.245
3.523.880.7190.261
3.5 → 0 (scheduled, ours)24.660.7440.165

W2-2. "Appendix D.3 omits MPC's control-energy penalty [R5], a substantive objective change requiring validation. For partially blind tasks, RefReg receives image-specific captions and a constructed reference, whereas GDP/DMPlug receive a generic prompt and raw measurement (D.4). These differences prevent attributing Table 3's gains solely to the solver. They qualify specific claims rather than invalidating all matched main comparisons ..."

Q2. "Can the authors validate the MPC adaptation with its control penalty restored and compare partially blind methods using identical captions, reference preparation, and available initialization information? Which performance conclusions survive these controls?"

The performance advantage holds under both additional controls: RefReg remains better than MPC with the control penalty restored (Table R2), and remains best among partially blind methods under identical captions, references, and matched initialization (Table R3).

MPC. Table 1 reports MPC with ρ=0\rho=0. We restored the penalty and swept ρ∈{10−10,10−9,10−8,10−7,10−6,10−5,10−4,5×10−4}\rho\in\{10^{-10},10^{-9}, 10^{-8},10^{-7},10^{-6},10^{-5},10^{-4},5\times10^{-4}\}, which spans the range of the original paper's experiment. For each dataset–task pair we selected the ρ\rho with the lowest mean LPIPS on the first 20 evaluation images (10−610^{-6}/10−510^{-5} for SR×8, and 10−910^{-9}/10−1010^{-10} for Gaussian deblurring on DIV2K/FFHQ). Table R2 compares the selected ρ\rho with the ρ=0\rho=0 control on the same 100 images.

Table R2. MPC without (ρ=0\rho=0) and with the control-energy penalty vs. RefReg, 100 images per dataset.

SettingMethodPSNR ↑SSIM ↑LPIPS ↓
DIV2K SR×8MPC (ρ=0\rho=0)22.2730.55550.4028
MPC (ρ=10−6\rho=10^{-6})22.6450.58350.4198
RefReg23.0180.60480.2626
DIV2K GDMPC (ρ=0\rho=0)20.7580.49810.3969
MPC (ρ=10−9\rho=10^{-9})20.7630.49850.3967
RefReg22.1520.56640.3213
FFHQ SR×8MPC (ρ=0\rho=0)28.5210.76810.3093
MPC (ρ=10−5\rho=10^{-5})27.4110.77370.2671
RefReg29.8440.80750.1651
FFHQ GDMPC (ρ=0\rho=0)26.9190.75550.2564
MPC (ρ=10−10\rho=10^{-10})26.9190.75550.2564
RefReg28.9400.78710.2042

Partially blind. The references were already identical across the three methods: all use the measurement clipped to [−1,1][-1,1]. RefReg's construction starts from it and returns the lowest-residual iterate, which under the identity initial operator (a delta kernel; s=1s=1) is this starting point (now stated in Appendix D.4). We re-ran GDP and DMPlug with RefReg's image-specific captions, the same measurement and initial noise, and matched initialization, keeping their guidance scales and learning rates fixed to the Appendix D.4 values, and RefReg remains best in every metric (Table R3); since DMPlug's softmax kernel cannot be an exact delta, we report both a near-delta start (99% center mass) and its native random start.

Table R3. Partially blind reconstruction, matched inputs, 100 DIV2K images.

TaskMethodPSNR ↑LPIPS ↓Operator error ↓
GDGDP19.080.3221.983
DMPlug (random init)19.170.3970.894
DMPlug (near-delta init)20.290.3700.931
RefReg23.710.2830.175
HDRGDP20.380.1700.524
DMPlug20.550.1630.636
RefReg24.670.0770.170

Q3-1. "What posterior target is intended,"

The target is πy=L(X0∣Y=y)\pi_y=\mathcal L(X_0\mid Y=y), the posterior of the clean image given the measurement under the observation model, as in Proposition 1. It is defined independently of the sampler. RefReg is an approximate sampler of πy\pi_y, neither an exact posterior sampler nor a MAP estimator.


Q3-3. "Please distinguish reconstruction-risk evidence from posterior calibration,"

RMR_M (Fig. 4) is reconstruction-risk evidence; the decomposition in W1-2 adds the decrease in bMb_M and a lower bound on VMV_M, and we claim no posterior calibration beyond these.


Q3-4. "… report how conditional branches and inner corrections enter NFE/compute accounting."

NFE counts sampling steps. Each step evaluates the guided-velocity of Eq. (7). FLOPs and sampling time include all conditional branches and inner corrections (Table A9).

Thank you for the detailed response and additional controlled experiments. The fixed-guidance sweep, restoration of the MPC penalty, and matched-input comparisons substantially address my original experimental concerns. I therefore maintain my rating.

Thank you for your positive evaluation and encouraging feedback. We are pleased that our additional controlled experiments have substantially addressed your concerns, and we sincerely appreciate your careful consideration of our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

Dear AC and Reviewers,

We thank all reviewers for their constructive reviews. We answered every point individually and uploaded a revision. Below we summarize the recognized strengths and how each main concern was addressed (R1 = EEja, R2 = VLWM, R3 = gw8W; R3-C1/C2 = R3's follow-up comments).


Strengths

  • Novelty: IT2I models as multimodal priors for inverse problems, with an analysis of how native IT2I generation uses the reference (R2).
  • Validated timing insight: controlled interventions support the design insight (R1); the attention-timing experiment shows that conditioning timing matters (R3).
  • Strong experimental results: lowest LPIPS in 15/16 dataset–task pairs (R3), across backbones with few NFEs (R2, R3), with gains that survive meaningful controls (R1).

Concerns and responses

1. Scope of Proposition 1 and the posterior-sampling claim (R1-W1-2, W1-3, Q3; R2-W1, Q3)

  • What it is: a proved bound linking the corrected-estimate risk RM(t)R_M(t) to a Wasserstein upper bound, used to compare IT2I and T2I priors.
  • What it is not: a posterior-sampling guarantee or calibration evidence.
  • All runs: Fig. 4 now adds ΔRrel(t)\Delta R_{\mathrm{rel}}(t) over all 500 runs; it is negative at every sampling time on both backbones.
  • No collapse: at the final step, reduced bias accounts for 69%/57% (FLUX.2/FLUX.1 Kontext) of the reduction in RMR_M, and a lower bound on VM/σpost2V_M/\sigma^2_{\mathrm{post}} is 0.96/0.60 (collapse would give 0).
  • Revision: the claim now reads "the smaller upper bound in these runs" (L286–287), Sec. 1 and 3.2 state the role of the bound, and "exact likelihood" is corrected (L37–38). The target is πy=L(X0∣Y=y)\pi_y=\mathcal{L}(X_0\mid Y=y), which RefReg approximates.

2. Contribution beyond combining known components (R1-W1-1; R3-Q2, W3, C1)

Reference conditioning, dual guidance, and attention intervention are established; the novelty seems to lie in their combination and scheduling.

  • Contribution: the use of pretrained IT2I models as multimodal priors for inverse problems, and RefReg, a training-free framework that regulates reference influence during sampling.
  • Insight beyond the combination: in IT2I priors, whether a generation uses the reference is decided in the first sampling steps (Sec. 3.1; three further backbones in App. B), so supplying a reference is not enough (Table A8). We use a timed intervention for reconstruction, where early blocking helps.
  • Text-conditioned solvers (R3-C1): TReg, P2L, CFG++, and ContextMRI act on the text condition of a T2I prior, whereas RefReg feeds a measurement-derived reference into an IT2I prior. On the same backbone (SR×8), TReg, P2L, and PSLD + CFG++ reach LPIPS 0.535–0.592 vs. 0.263 for RefReg (R3, Table R7).

3. Simpler alternatives and engineering (R1-W2-1, Q1; R2-W2, Q2; R3-W1, Q1)

Do the schedule and early blocking beat simpler alternatives? Is attention control over-engineered?

  • Image-guidance schedule: with EAB, 3.5 → 0 beats every constant scale (1.0–3.5) on all metrics (R1, Table R1); without attention modifications, constants reach at best 23.61 dB/0.276 LPIPS vs. 24.66/0.165 (R3, Table R1).
  • Early blocking (EAB): reduced text guidance falls below no intervention; dropping prompt-content tokens gives LPIPS 0.177 vs. 0.156 (R3, Table R2).
  • Fixed noise endpoint (R2-Q2): beats fresh and FlowDPS-style re-noising on all metrics (R2, Table R4).
  • Engineering (R2-W2): one rule (block prompt-content tokens for three steps) for all eight degradations; −30 is untuned; the data term is unchanged (plug-and-play).

4. Fairness of comparisons (R1-W2-2, Q2; R3-W2, C1, C2)

MPC lacked its control penalty; partially blind baselines got different inputs.

  • MPC with its penalty (ρ = 1/λ, swept over 10−1010^{-10}–5×10−45\times10^{-4}): RefReg remains better in all metrics on all four settings. In MPC's official FLUX.2-dev (32B) pipeline with 23 tuned configurations, PSNR-selected MPC exceeds RefReg (4B) at 28 steps in PSNR/SSIM (25.80/0.726 vs. 24.85/0.699), but RefReg has lower LPIPS than every MPC setting (0.160 vs. 0.255) and leads in all three at 50 steps.
  • Partially blind, identical inputs: with RefReg's captions, the same reference, and matched initialization, GDP and DMPlug still trail in every metric (e.g., GD operator error ≥ 0.894 vs. 0.175).
  • Claims: we claim lower LPIPS and FID, not best everywhere: RefReg is best in 15/16 (LPIPS), 12/16 (FID), and 10/16 (PSNR) pairs, and second otherwise.

5. Running time and compute (R2-W3, Q4)

  • Running time: 42.0 s/image, within the baselines' 18.4–58.5 s (now including PnP-Flow, OT-ODE; R2, Table R2).
  • Matched budget: at matched FLOPs (baselines at 77–89 steps), RefReg (50 steps) stays best (R2, Table R3).

All new results will be included in the final version.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.