acceptodds
Withdrawn from ICLR 2027

Unpaired All-in-One Weather Restoration via Structured Stochastic Updates in Schrödinger Bridges

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unpaired all-in-one weather image restoration requires a single generative model to learn restoration mappings for multiple degradations from non-corresponding degraded and clean images. Unpaired Neural Schrödinger Bridge progressively constructs restoration paths by predicting clean endpoints at different bridge positions. However, its original training estimates gradients using only one complete stochastic realization at a randomly sampled bridge position, without integrating conditional gradient information from multiple stochastic realizations in the joint parameter space of the generator and the trainable feature network. To this end, we propose Structured Stochastic Update Schrödinger Bridge (SSUSB). Conditional Gradient Ensembling (CGE) estimates joint gradients of the generator and feature network from multiple complete stochastic realizations and constructs a consensus using their mean. Bridge Stratified Coupling (BSC) further implements joint sampling without replacement while preserving the uniform marginal distribution of the bridge position for each view. Consensus Tangent-space Projection (CTP) uses the multi-view consensus to calibrate a prespecified candidate gradient. In a frozen Adam-derived metric, it replaces the candidate’s axial component with the consensus while preserving its orthogonal component. Extensive experiments demonstrate that SSUSB can effectively improve unpaired all-in-one weather image restoration performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unpaired learning, All-in-one weather image restoration, Schrödinger bridges, Stochastic gradient optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EU4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Structured Stochastic Update Schrödinger Bridge (SSUSB), a training-time extension of UNSB for unpaired all-in-one weather restoration. SSUSB evaluates multiple complete stochastic bridge realizations at the same batch and model state. Conditional Gradient Ensembling (CGE) averages the joint gradients of the generator and feature network; Bridge Stratified Coupling (BSC) samples bridge positions uniformly without replacement; and Consensus Tangent-space Projection (CTP) replaces the candidate gradient’s consensus-axis component while retaining its tangential component under a frozen Adam-derived metric. The paper proves unbiasedness and conditional variance-reduction properties for CGE and BSC under fixed-state sampling assumptions. Experiments on six weather-restoration benchmarks and three unseen datasets report improvements over eleven unpaired restoration baselines. Additional ablations study the number of views, BSC, CTP, projection scope, and projection metric.

Critical strengths
  1. The distinction between gradient averaging, bridge-position stratification, and consensus projection is clearly stated. The covariance decomposition and finite-population analysis for BSC are self-contained and mathematically plausible under the stated assumptions.

2.The paper evaluates six in-domain datasets and three unseen datasets using full-reference and perceptual metrics, together with qualitative comparisons, view-count analysis, and component ablations. If the protocol is verified to be fair and reproducible, the reported gains suggest practical value.

Critical weaknesses

1.Under the frozen-state objective, mnm_n is the natural unbiased gradient estimator, whereas CTP uses g∗=mn+As−1w⊥g^\ast=m_n+A_s^{-1}w_\perp, adding a candidate-specific component. The paper does not show that g∗g^\ast remains unbiased, has lower variance, or improves expected descent; Appendix A.5 establishes only a geometric projection property. Selecting g(1)g^{(1)} also makes the update order-dependent, while the variance analysis concerns mnm_n, not the actual update g∗g^\ast. Moreover, the frozen second-moment metric does not fully represent Adam updates, which also depend on momentum, bias correction, and separate optimizer states for GG and FF.

2.Tables 1–3 are presented as unpaired comparisons, but the supervision protocol for each baseline is unspecified. It is unclear whether all methods were retrained from scratch using only unpaired degraded and clean pools, whether paired correspondences or clean targets were used, and how originally paired methods were adapted. The statement that no degradation labels or task-specific fine-tuning were used does not establish that paired supervision was removed. Paired baselines may receive additional information, whereas naively adapting them to unpaired training may unfairly disadvantage them. The authors should separate strictly unpaired methods from supervised upper bounds and report the exact protocol for each baseline. The omission of recent methods such as [1-5] further weakens the paper’s state-of-the-art claims.

[1] AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation, ICLR2025

[2] Unifying Heterogeneous Degradations: Uncertainty-Aware Diffusion Bridge Model for All-in-One Image Restoration, ICML2026

[3] Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration, TPAMI2026

[4] StarIR: Convolutional Image Restoration With Spatial-Frequency Fusion, TPAMI2026

[5] Gradient as conditions: Rethinking HOG for all-in-one image restoration, AAAI2026

Questions for authors

1.Can the authors compare SSUSB with three-view IID gradient averaging, gradient accumulation, and a single-view baseline under the same total number of gradient evaluations and the same wall-clock budget? Please report CGE-only, CGE+BSC, and full CGE+BSC+CTP under these matched budgets.

2.Why are recent all-in-one restoration methods such as AdaIR, BaryIR, and HOGformer absent from the quantitative comparison? Do these methods use paired supervision, degradation labels, or different training datasets? If their original protocols are incompatible with the unpaired setting, can the authors provide matched unpaired adaptations or clearly labeled supervised upper-bound comparisons?

3.What precisely distinguishes the Table 7 “Baseline” from the original UNSB implementation? Why does the n=1n=1 baseline resample after the discriminator and energy-network updates? Were UNSB and the modified baseline trained with identical data order, update counts, learning-rate schedules, and random seeds?

4.Can the authors report the empirical mean, covariance, and descent behavior of the final update g∗g^\ast, rather than only those of mnm_n? In particular, does CTP preserve unbiasedness or reduce variance for g∗g^\ast under the actual BSC sampling procedure?

5.How are the first-moment buffers, bias correction, optimizer epsilon, and separate Adam states for GG and FF handled when applying CTP? How sensitive are the results to randomly permuting the nn views or choosing a different view as the candidate g(1)g^{(1)}?

Reviewer_JaCLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets unpaired all-in-one weather restoration, where a single generator must map a mixed pool of degraded images (fog, rain, raindrop, snow, low light) to an unaligned pool of clean images. It builds on UNSB, which trains an endpoint-predicting generator G and a trainable feature network F, and computes each joint G/F gradient from a single stochastic realization at one randomly chosen bridge position. The authors argue that this one-sample estimate is noisy and propose SSUSB, which changes only the G/F update rule and leaves the architecture, losses and inference unchanged. SSUSB has three parts. Conditional Gradient Ensembling (CGE) fixes the batch and model state after the discriminator and energy-network updates, evaluates n complete stochastic realizations, and averages their joint gradients. Bridge Stratified Coupling (BSC) samples the n bridge positions without replacement, so each view keeps a uniform marginal while the between-position variance is reduced by a finite-population factor. Consensus Tangent-space Projection (CTP) takes the first view's gradient as a candidate. In a metric built from frozen Adam second-moment buffers, it replaces the candidate's component along the consensus axis with the consensus and keeps the candidate's orthogonal deviation. A single projection coefficient is shared across the G and F blocks. Two propositions give the unbiasedness and the covariance reduction of CGE and BSC, and the appendix gives a minimal-modification characterization of CTP. On six datasets trained jointly (FoggyCityscapes, LowLightTrafficData, RainCityscapes, RainDS-syn, RSCityscapes, SnowTrafficData), SSUSB reports an average gain of about 3.5 dB PSNR over UNSB and about 1.2 dB over DDSB, together with better LPIPS and NIQE. It also reports gains on three "unseen" datasets. Ablations cover the number of views, the variance decomposition, the CTP components and metric, and the scope of the G/F projection.

Critical strengths

1. The paper isolates one clear lever, the gradient estimator for the G/F update, and keeps the backbone, losses and inference fixed. This makes the attribution of gains cleaner than in most restoration papers that change several components at once.

2. The variance analysis of CGE and BSC is correct and matches the empirical check closely. The measured BSC/IID variance ratio (0.71246) is very close to the predicted 0.7125, which confirms that the implementation follows the stated sampling law.

3. CTP has a precise geometric characterization: it is the unique direction closest to the candidate under an axial constraint in the frozen metric. The authors also state that it does not preserve norm or the Adam step size, which is a candid caveat.

4. The ablations are organized well: progressive component addition, the effect of the view count, which disagreement components are kept, Euclidean versus Adam metric, and G/F/joint projection scope. They are reported for both distortion and perceptual metrics.

Critical weaknesses

1. CGE is not compute-matched or batch-matched against a simple alternative. Training uses batch size 1, and n = 3 views of the same image at different bridge positions amount to a larger effective batch over (t, ξ). The natural controls are missing: batch size 3, three sampled bridge positions per image inside one loss (standard practice in diffusion and bridge training), or 3× more iterations at n = 1. Without them, the 0.9 dB gain from CGE cannot be distinguished from a generic batch-size or sample-count effect, and Propositions 1–2 restate textbook results (variance of a sample mean, and the finite-population correction of stratified sampling).

2. The mechanism behind CTP is unexplained and counter-intuitive. CTP adds the orthogonal deviation of a single view back onto the consensus (g* = m + A⁻¹w⊥). This reinjects exactly the single-sample noise that CGE and BSC were designed to remove. Yet Table 6 reports a 1.21 dB gain over the consensus alone. In a parameter space with millions of dimensions, w∥ is a projection onto one direction. The paper should report ‖w∥‖/‖w‖, the cosine between d and m, and how they evolve during training. Otherwise the claim that removing one radial component accounts for 2.8 dB over the candidate (Table 6, last row vs CTP) is hard to accept.

3. CTP changes the step magnitude, and this confounds the direction argument. By Eq. (62), ‖A g*‖² = ‖u‖² + ‖w⊥‖², so the CTP update has a strictly larger norm in the Adam metric than the consensus. The gain over m could come from a larger effective step or from injected noise (a Langevin-like regularization in GAN training), not from "radial filtering". Required controls: (i) m rescaled to the norm of g*; (ii) m plus a random vector orthogonal to u with the same norm as w⊥; (iii) a candidate drawn uniformly at random rather than the "first view".

4. The adversarial balance changes, and this is not discussed. G and F receive gradients averaged over n realizations, while D and E are still updated from one realization. This alters the generator–discriminator dynamics, which are known to dominate GAN-based unpaired translation. Part of the gain may come from this asymmetry. An ablation that applies the same multi-view estimate to D/E, or a fixed-D analysis, would clarify the cause.

5. The effect sizes are unusually large for an optimizer-side change, and no variance is reported. Changing only the gradient estimator yields +3.5 dB average PSNR over UNSB and +1.2 dB over the strongest bridge baseline. All results appear to come from one training run. Even the "Baseline" (UNSB with resampling after the D/E update) is 0.7 dB above UNSB, which suggests high sensitivity to update ordering. Please report several seeds with standard deviations for at least UNSB, the n = 1 baseline and SSUSB.

6. The data protocol may weaken the "unpaired" premise. Three of the six training sets (FoggyCityscapes, RainCityscapes, RSCityscapes) are synthesized from Cityscapes. The "unified clean domain" may therefore contain the exact clean counterparts of the degraded training images, and possibly of the test scenes. Please state explicitly whether clean images that correspond to any degraded image (train or test) were excluded. Otherwise the setting is closer to weakly paired data, and the PSNR comparisons to genuinely unpaired methods could be inflated.

7. The baselines and the cross-dataset claim are under-specified. It is unclear whether all eleven baselines were retrained under the same mixed-degradation, batch-size-1, 400-epoch protocol or whether numbers were taken from their papers. The "unseen" sets (RealRain-1k-L, NH-HAZE, LOL-v1) share degradation types with the training domains (rain, haze, low light), so they test dataset shift rather than generalization to unseen degradations. The wording should reflect this.

8. The variance analysis in Fig. 4 is computed at a single fixed (b, s). Showing that variance falls at one snapshot does not show that variance is the bottleneck for final quality. A plot of gradient-variance reduction against training progress, or the relationship between estimator variance and the final PSNR gain, would connect the theory to the results.

Questions for authors

1. How is R defined in Fig. 4(c), and at which training iteration was the fixed state s taken?

2. In Table 5, "G/F" (separate projection) and "GF" (joint projection) differ only in whether one or two coefficients are used. Why should coupling the two blocks through one scalar give 0.37 dB? Do the per-block coefficients typically differ in sign?

3. Coordinates without initialized Adam buffers are assigned aᵢ = 1. At the start of training this mixes Adam-scaled and unscaled coordinates in one inner product. Is CTP disabled during warm-up?

4. Does the 2.91× training-time factor at n = 3 include the D/E updates? Do the results in Table 4 hold the number of G updates or the wall-clock time fixed?

Reviewer_J8YqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SSUSB for unpaired all-in-one weather image restoration, building upon UNSB. Rather than changing the network architecture or the inference pipeline, the method modifies how the generator (GG) and feature network (FF) are updated during training. Specifically, CGE averages gradients across multiple stochastic realizations at a fixed batch and model state; BSC samples bridge timesteps without replacement; and CTP calibrates a candidate gradient using the multi-view consensus under a frozen Adam second-moment metric. Experiments across six restoration benchmarks and three unseen datasets show consistent improvements without extra inference cost.

Critical strengths
  1. Zero inference overhead: All modifications happen during training. As shown in Table 3, SSUSB keeps the exact same parameter count (14.06M) and inference latency (~81.5 ms) as the base UNSB model. Achieving clear gains across six in-domain datasets and three unseen datasets (RealRain-1k-L, NH-HAZE, LOL-v1) without target fine-tuning makes the approach practically appealing.
  2. Well-structured ablations and clean code: Tables 5–7 break down the effects of joint vs. separate G/FG/F projection, radial vs. tangential components, and the projection metric. The supplementary code is also well organized and includes configurable ablation switches.
Critical weaknesses
  1. Training compute is not matched across methods: Table 4 evaluates CGE with BSC and CTP disabled. Under the same 400-epoch training schedule, using n=3n=3 views takes 2.91x the single-view training time (4.79x for n=5n=5). These measurements show the additional cost of multi-view CGE, but do not establish the training-time ratio for full SSUSB. Please report the measured training time of the complete method. Without a matched-compute comparison, it remains unclear how much of the overall gain comes from the update strategy versus additional computation. A fair evaluation would compare against a single-view baseline trained for a similar measured GPU-time budget, with an appropriate learning-rate schedule, or against standard gradient accumulation across different mini-batches.

  2. The mechanism behind CTP's +1.21 dB gain needs a simpler baseline comparison: Since CGE (multi-sample averaging) and BSC (sampling without replacement) are relatively standard variance-reduction steps, the main technical novelty lies in CTP. However, Table 6 shows a somewhat counter-intuitive result: CGE and BSC aim to reduce gradient variance toward the consensus mean mnm_n, yet updating with mnm_n alone gives 24.65 dB, whereas adding back the tangential deviation w⊥w_\perp of a single candidate gradient g(1)g^{(1)} jumps to 25.86 dB (+1.21 dB). While Appendix A.5 proves that CTP is a minimum-perturbation projection under an axial constraint, it remains unclear from an optimization perspective why re-injecting single-sample orthogonal noise helps so much, or whether it mainly acts by changing the gradient norm / effective step size (Eq. 62). To verify that the gain truly comes from the specific tangent-space geometry rather than simply blending in single-sample variance, it would be helpful to compare CTP against a simple linear interpolation baseline: gβ=mn+β(g(1)−mn),β∈(0,1)g_\beta = m_n + \beta(g^{(1)} - m_n), \quad \beta \in (0, 1) Checking a couple of intermediate β\beta values (e.g., β=0.3,0.5\beta = 0.3, 0.5) and checking stability across a few runs on a representative setup would make the claim for CTP much more convincing.

Questions for authors
  1. Iso-compute comparison: Under a matched GPU-time or compute budget (e.g., training the single-view baseline longer or using standard cross-batch gradient accumulation), how much of SSUSB's advantage remains? Also, what is the peak GPU memory overhead of n=3n=3 compared to n=1n=1?
  2. CTP vs. simple interpolation: How does CTP compare against a simple linear interpolation gβ=mn+β(g(1)−mn)g_\beta = m_n + \beta(g^{(1)} - m_n) for intermediate β∈(0,1)\beta \in (0, 1)? How stable is the 1.21 dB gap between mnm_n and CTP across different training runs?
  3. Dataset mixing and scene-level splits: Since FoggyCityscapes, RainCityscapes, and RSCityscapes all originate from Cityscapes, how were the datasets mixed during training, and how did you ensure that the underlying clean scenes in the test sets do not appear in the unpaired training pool under another weather condition?
  4. Comparison with WeatherCycle: WeatherCycle (Fang et al., 2026) is cited in the introduction for unpaired multi-weather restoration but is not compared in Tables 1–3. Could you include a comparison, or briefly clarify the differences in experimental setup / external priors that make a direct comparison difficult?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.