acceptodds
Withdrawn from ICLR 2027

Making Every Step Count: Spatio-Temporal Information Allocation for Imaging Inverse Problems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based generative models have emerged as powerful image priors for training-free inverse problem solving, capturing coherent semantics and fine-grained structure. Despite these strengths, existing flow-based inverse solvers primarily focus on the design of individual updates, largely overlooking spatio-temporal information allocation under a fixed number of function evaluations (NFEs). Temporally, insufficient early exploration can trap the flow trajectory in an incorrect semantic basin, whereas excessive allocation of NFEs to early stages leaves little budget for late-stage refinement. Spatially, data consistency provides direct constraints only within observed regions, whereas the recovery of missing regions relies mainly on the generative prior. To address these two issues, we introduce two complementary and training-free components, i.e., Spectrum-Adaptive Scheduling (SAS) and Measurement-Prioritized Attention (MPA). For temporal allocation, SAS distributes the available NFEs over flow time according to the degradation spectrum and logSNR geometry, thus better balancing semantic exploration and detail refinement. For spatial propagation, MPA exploits data-prior conflicts to guide information toward weakly constrained regions, thereby enhancing semantic and structural fidelity. Extensive experiments on standard image inverse problems, e.g., super-resolution, motion deblurring, and inpainting, demonstrate that the proposed components can be integrated into existing flow-based inverse solvers in a plug-and-play manner without retraining or additional flow-model evaluations, and can also significantly improve the restoration quality of existing solvers.

Primary area:
generative models
Keywords:
Diffusion Models, Inverse Problems, Adaptive Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KzecOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper investigates training-free image inverse problem solvers based on pretrained Flow Matching. It identifies limitations in existing methods under limited function evaluation (NFE) budgets, particularly in time-step allocation and spatial information propagation.

  2. The authors propose Spectrum-Adaptive Scheduling (SAS), which adaptively allocates NFE based on the degradation operator and flow dynamics, and Measurement-Prioritized Attention (MPA), which uses a conflict heatmap to guide information flow from strongly measured regions to weakly constrained areas.

  3. Experiments integrate the proposed methods into FlowLPS and FLAIR using Stable Diffusion 3.5 Medium for super-resolution (8×/12×), motion deblurring, and inpainting on FFHQ and DIV2K. Results demonstrate improved reconstruction fidelity and semantic stability without additional NFE.

Critical strengths
  1. Novel Perspective and Sound Motivation: The paper reframes the bottleneck of flow-based inverse solvers as spatiotemporal information allocation. SAS leverages the degradation operator’s SVD spectrum to adaptively schedule time steps, improving efficiency without additional NFE.

  2. Efficient Design and Generalizability: MPA encodes prior–measurement conflicts as a lightweight rank-1 attention bias, avoiding explicit n×nn \times n attention matrix construction while supporting fused attention. Its plug-and-play design is validated on state-of-the-art frameworks, including FLAIR and FlowLPS.

Critical weaknesses
  1. Insufficient Evidence for MPA’s Spatial Guidance: Shuffled and uniform heatmaps achieve performance comparable to Correct MPA, weakening the claim that MPA’s gains stem from fine-grained spatial guidance. The improvements may instead arise from global attention scaling or additive-bias regularization.
  2. Hyperparameter Sensitivity and Heuristic Task Adaptation: MPA relies on multiple hyperparameters and task-specific configurations, raising concerns about its robustness and adaptability as a general-purpose plug-and-play module.
  3. Unresolved Placeholder and Presentation Issues: Figure 7 retains an explicit placeholder caption indicating that experimental results have yet to be inserted, suggesting insufficient proofreading and raising concerns about the manuscript’s overall presentation quality.
Questions for authors
  1. Clarifying the Spatial Effectiveness of MPA: Please explain why the Shuffled and Uniform heatmaps in Table 6 achieve performance comparable to the Correct heatmap. Could more discriminative spatial metrics, such as localized consistency or gradient errors computed specifically around missing-region boundaries, provide stronger evidence that spatially selective attention effectively directs information flow as intended?

  2. Nonlinear Operators and Practical Noise Robustness: The current spectral analysis of SAS relies heavily on the analytical singular value decomposition of linear operators, such as those represented in the Fourier domain or through permutation matrices. How could SAS’s temporal scheduling strategy be generalized to nonlinear inverse problems, such as non-blind deblurring, phase retrieval, or nonlinear photometric distortions?

Reviewer_6g32OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper uses pretrained diffusion models for image restoration using text conditional diffusion models. They introduce two core mechanisms, SAS and MPA. SAS is a way of adapting the schedule depending on the forward operator. The simplest example is inpainting. Depending on the size of the mask, we want to invest different amounts of "work" into the different times. MPA adapts the attention mechanism to account for which regions need modulation and which ones do not. Modulation in this case means a bias added to the logits of the attention mechanism, which again in the inpainting case, depends upon whether the pixel is known or unknown. The algorithm is then benchmarked against several image restoration algorithms.

Critical strengths
  1. The idea of differentiating forward operators into ones where little change is needed and ones where the prior needs heavy supervision is good and very useful. The effects of this seem very strong numerically.

  2. Similarly changing the attention mechanism to account for more consistency is a novel and fresh idea.

  3. The method helps a lot for inpainting results.

Critical weaknesses
  1. The paper was for me very hard to read. Many sentences use very unusual words and their meaning was not clear for me after many passes.

  2. It has almost impossible to reconstruct the actual used algorithm from the paper. The paper talks about a data consistency step, but i am not sure when and where it is even applied? Are we in a PnP kind of framework? As far as I can tell both steps are somehow inserted into FlowLPS and Flair, but please provide a full algorithm for both into the appendix so one can gauge how general the algorithm is.

  3. For SAS i like the idea but the correction is very coarse. Usually different pixels like need different schedules, as some contain almost perfect info and some need full prior guidance.

  4. For MPA, the correction mechanism of the bias is unclear. Thinking about superresolution, is the bias always on or not? I think it is always included, but depending on Superresolution strength (how much we downsample) we have different amounts of information provided by the observation. The whole construction of cjc_j seems rather ad hoc to me and introduces additional hyperparameters which I am unsure justify their own cost. The uniform control in table 6 seems to perform good as well.

  5. Numerically the method is outperformed by FlowChef in SR experiments. Why is your method marked fat?

My rating could be improved a lot if the paper was written more clearly. I actually really like the idea of the paper.

Questions for authors
  1. MPA only applies to models which use attention as their main mechanism. Many pretrained diffusion models use U-nets. How would you adapt your algorithm for such?

  2. How much could the text embedding account for data consistency? For the FFHQ it seems to be a generic prompt for all images, however for Div2K it seems to be image specific which could confound this claim.

Reviewer_Fm3QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes two training-free modules that can be implemented on top of already existing flow-based image inverse solvers. Spectrum-Adaptive Scheduling (SAS) uses the rank of the degradation operator, together with logSNR-derived temporal bases, to distribute a fixed evaluation budget (50 steps) along flow time. Measurement-Prioritized Attention (MPA) instead converts differences between prior and data-consistent latent estimates into a key-side attention bias. Both components are integrated into FLAIR and FlowLPS, and experiments on FFHQ and DIV2K cover SR, motion deblurring and box inpainting. In particular, with FlowLPS on FFHQ, the proposed additions report strong gains.

Critical strengths
  1. The two additions, while simple in their principles and implementations, prove to produce substantial improvements in several tested tasks. Table 1 reports FlowLPS FFHQ gains of 2.47 dB for SRx8, 2.56 dB for SRx12, and 2.63 dB for inpainting. FLAIR also improves PSNR in every listed setting. These results suggest that the choice of the timesteps is indeed a sensible design element of many flow-based methods, and an automatic way to choose them improves results without retraining.
  2. The paper includes analysis conducted on two solvers, two datasets, ablations of the two components, shuffled vs uniform attention maps and runtime measurements. It is appreciated that Appendix G acknowledges that the scheduling model is a proxy and that attention reweighting does not guarantee better decoded images.
Critical weaknesses
  1. The optimality statement discussed in Appendix B.3 holds only under the assumption that the square root of local trajectory difficulty has the SAS form, but the paper does not measure and/or validate this assumption for either of the solvers tried. Concerning the experiments, the direct comparison of SAS against alternative nonuniform schedules is restricted to FFHQ SRx8. No tests check whether the proposed late-stage allocation helps deblurring, or whether the operator-specific allocation is preferable for inpainting. The paper also does not compare SAS with TORS. As a consequence, the results support the usefulness of reallocating timesteps, but provide limited evidence that using the degradation spectrum to choose that allocation is better than a simpler scheduling rule.
  2. Concerning the MPA module, in Table 6 the correct vs shuffled maps give very similar metrics, and Table 2(a) also shows that adding MPA to SAS slightly worsens PSNR and FID. In particular, the claimed semantic stabilization is supported mainly by selected samples rather than by metrics or more extended qualitative analysis. I believe that, since MPA introduces measurable overhead, establishing an advantage over simple observed-key reweighting is important to the contribution.
Questions for authors
  1. Can the authors provide the exact handcrafted timestep allocations for Table 2(b), and matched SAS-versus-tuned schedule comparisons for deblurring and inpainting as well as SR? Which measured property of the solvers' trajectories supports the assumed difficulty profile in Appendix B.3?
  2. Is there a measurable semantic or structural improvement, across multiple images and seeds, from the spatially correct map, beyond generic upweighting of observed keys? The existing quantitative controls do not establish this distinction.
  3. Please provide complete pseudocode for both solvers. Is Eq. (43) intended only as a pixel-space illustration? In the implemented latent solver, how does the decoder affect the conflict map's support and interpretation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.