acceptodds
Under review as a conference paper at ICLR 2027

Mind the Residual Gap: Probabilistic Downscaling under Real-World Bias

Abstract

Probabilistic downscaling models the conditional distribution of high-resolution fields given coarse inputs and is a core challenge in atmospheric science, climate modeling, and other multiscale physical systems. A widely used paradigm decomposes the problem into a deterministic mean predictor followed by a stochastic residual generator. While effective in idealized settings, this mean-residual approach frequently produces biased and underdispersive ensembles in real-world applications. We identify a fundamental source of this failure: residual target misspecification, where the residual distribution induced during training systematically differs from the correction distribution required at test time, a mismatch amplified by downscaling bias. We theoretically connect this misspecification to underdispersion through the spread-skill ratio and show that conditional residual distribution matching controls both residual-energy and conditional-mean mismatch. Motivated by this analysis, we introduce ReMatch (Residual Distribution Matching), which aligns training residual targets toward a held-out calibration regime via optimal transport. On a controlled synthetic benchmark with varying bias levels and a real-world HRRR-ERA5 wind-field downscaling task, ReMatch substantially reduces underdispersion, improves calibration, and outperforms strong mean-residual and super-resolution baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic downscaling, ensemble calibration, generative modeling, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L5D5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the problem of modeling the conditional distribution of high-resolution data given its low-resolution version. The starting point is the mean–residual decomposition where one first predicts the mean of the conditional distribution and then models the residual using a family of distributions (e.g. diffusion models). The idea being that it is easier to find the distribution of the residual since its variance is lower, because the conditional mean was already approximated in the first step. The paper argues that one issue with this approach is that this two-stage approach, which fixes the conditional mean before matching the residual, neglects taking into account that the conditional mean was computed using only training data rather than full data distribution. The paper provides an analysis of the spread–skill ratio to motivate this issue. The proposed solution is use calibration data (separated from training data) and then compute the optimal transport coupling between the training and calibration data (in practice, a low-dimensional PCA representation). Using this coupling the training data is readjusted to pseudo training samples which are then used for modeling the distribution of the residuals. The idea is that the calibration data adjustment helps mitigate the fact that the conditional mean was trained on only the training data. The paper compares this method on the BLASTNet dataset (synthetic benchmark) and the ERA5 dataset (real-world atmospheric downscaling task) against a number of baselines and shows that the propose dmethod is competitive.

Critical strengths

The proposed ReMatch method shows good performance in the experiments considered in the paper.

Critical weaknesses

On a high level the paper mixes a number of ideas. The first idea is to use calibration data to mitigate the residual shift. The second idea is to use optimal transport for the latter. But the motivation for optimal transport is not very clear to me. Supposedly, the analysis of the spread–skill ratio shows that W_2 can bound various quantities that appear in this analysis, but the connection seems somewhat tenuous. Thus, it is not clear whether one could take any of the probabilistic baselines methods and use calibration data to readjust them in another way. Indeed, optimal transport is computationally expensive so finding different ways to use calibration data could turn out to be much more useful.

Questions for authors

See Critical Weaknesses.

1 comment

Dear Reviewer,

We would appreciate your feedback on whether our responses and clarifications have adequately addressed your concerns. If so, we would be grateful if you could consider raising your score.

Thank you again for your time and consideration.

Reviewer_QkmSOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The topic of the paper is downscaling, specifically in the super-resolution setting, for a wind field. The investigation is around how generative models, e.g. CorrDiff, that use the projections of a determinstic model (e.g. UNET), to train a variance vector to add the missing high-frequency components, will overfit to the structure of the determinstic model projections. The paper proposes a post-training method that matches the principal components of the residual noise with an ideal one for the holdout set. The motivation is in the unreliability in the mean predictor itself, which is a valid and interesting motivation.

Critical strengths
  1. The paper is easy to follow in terms of presentation and articulates the core idea well. As an active researcher in this field, it was relatively straightforward to understand the contents.

The core idea is built on the premise that the determinstic method will underrepresent the structure of the wind field (underdispersion in the language of the paper). This is a known problem, and is valid for the broader community. The paper also includes some theoretical insights into the underlying problem in terms of energy, which is intuitive given the underdispersion, normally occuring at the high-frequency regions; adjusting the generative (noise) areas of the distribution makes sense in that regard.

Critical weaknesses
  1. Evaluation of the structural capture is under-evaluated, some experimental settings can be made clearer to support the main claims.

I dont think lower degradation in RMSE with respect to bias is always the best indicator. Another way is to compare the PSD distributions of these methods while comparing these with existing methods, which is also a useful metric given there is not a one-to-one correspodence between input and target variables.

Further, there doesnt appear to be evaluation downscaling between LR and HR pairs under a no bias setting. Under such a setting, the determinstic methods may have considerably better structural capture. Figure 2 shows poor capture, but there appears to be very correspondence between input and truth, implying a high-bias setting. My concern is that under a low bias setting, CorrDiff may be more effective, because the determinstic method is more accurate, i.e. accurate placement of high-frequency features.

A complaint i have about figure 2 is that the input and target appear to be on the same grid, at a similar (or the same) resolution. The paper states that the targets were produced from corrupting the input with a mapping D.

Questions for authors
  1. Not all downscaling occurs in a super-resolution setting. Does the method hold under a perfect prognosis setting? Here, the nosie estimator has to infer variance indirectly, making it a harder task. I think some discussion around this point might help solidify generlisablility.

  2. The paper uses a large ratio between training and holdout of 2:1 - 2 samples of training for every holdout. This would imply that ReMatch is data hungry. However, Table 3 of the Appendix appears to not display a pattern to the size of the holdout set (which reduces training set size). What is the cause of this robustness? Is it possible to run ReMatch without the finetuning aspect?

  3. One concern is that the noise adjustment process is creating large noise vectors, as can be seen by the ReMatch trajectories in Figure 3. From an experimental point of view, this could explain how structural changes are occuring, as can be seen in Figure 2 in the differences between ReMatch and the UNET and SwinIR methods. Is the main shift in structure (Fig 2) due to this, otherwise what explains it? This is related to weakness 1.

2 comments

Dear Reviewer,

We have updated our manuscript based on your feedback. Specifically, we have added an ablation study with varying levels of downscaling bias to demonstrate that ReMatch consistently improves RMSE and SSR across different bias levels (Appendix A.15). We have also included the corresponding inputs and ground truths for the ensemble sample comparisons in Figure 12 (Appendix A.12) to provide clearer context. Additionally, we have updated all figures showing low-resolution inputs, including Figures 3, 10, and 12, to display the raw low-resolution inputs without upscaling. We have also corrected the spectral analysis of the proposed methods in Appendix A.16 based on your feedback.

We would appreciate your feedback on whether our responses and additional results have adequately addressed your concerns. If so, we would be grateful if you could consider raising your score.

Thank you again for your time and consideration.

Thanks, the additional experiments have resolved my concerns. After reading these, it appears the paper has potential to be of interest to the broader downscaling/climate community; I will increase my score.

Reviewer_WprCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates underdispersion in mean–residual probabilistic downscaling (e.g., CorrDiff), where a deterministic mean predictor μϕ\mu_\phi is followed by a diffusion model trained on residuals y−μϕ(x)y-\mu_\phi(x). The authors attribute underdispersion to "residual target misspecification": residuals from the samples used to fit μϕ\mu_\phi are smaller and have a different distribution than out-of-sample residuals (ERM optimism). They further claim that downscaling bias amplifies this effect. They relate the residual mismatch to the spread–skill ratio and show that a conditional W2W_2 distance provides an upper bound on both the residual-mean mismatch and the residual-energy gap. The proposed method, ReMatch, uses a held-out calibration split. In PCA space, it solves an entropic OT problem between training and calibration residuals, using a cost function that also depends on the LR condition. Each training residual is replaced by a weighted average of its top-kk calibration matches, and the residual diffusion model is trained on these transported residuals along with the calibration residuals. Experiments include a synthetic BLASTNet benchmark with injected LR bias (comparing ReMatch with CorrDiff only) and ERA5→HRRR stratospheric wind downscaling (reported in one table using 600 test samples). A particle-advection example is also included.

Critical strengths

The paper addresses a real and practically relevant problem: mean–residual diffusion downscalers, such as CorrDiff, are known to be underdispersive. The paper also provides a concrete diagnostic of the residual-energy gap between in-sample and out-of-sample data (0.55 vs. 4.29 on ERA5–HRRR). This diagnostic is useful, but it does not, by itself, justify the proposed method.

Critical weaknesses
  1. The experiments do not isolate the contribution of the proposed method and provide only limited support for its claims.

Missing baselines. The analysis suggests a much simpler alternative: training the residual model on out-of-sample residuals, either from the calibration split alone or through K-fold cross-fitting. Neither approach is evaluated. ReMatch already includes calibration residuals in the diffusion training set, so it is unclear whether the improvements come from the OT/PCA transport or simply from incorporating held-out residuals. Other natural baselines are also missing, including post-hoc spread inflation or variance rescaling and OT-based debiasing for downscaling (Wan et al., 2023, which the paper cites). The deterministic baselines (UNet, U-FNO, and SwinIR) are relatively dated, and comparisons with recent generative downscalers are missing.

Results. In Table 1, ReMatch$_U$ achieves an SSR of 0.635, comparable to Plain Diffusion (0.642) and the Finetune baseline (0.650). Finetune also achieves a better CRPS (1.144 vs. 1.157), although its RMSE is worse. All three methods remain substantially underdispersive, making the claim that ReMatch "substantially reduces underdispersion" an overstatement. The best results come from ReMatch$_S$, but without a SwinIR-based CorrDiff control, it is impossible to distinguish the contribution of the proposed method from that of the stronger backbone. Furthermore, the results are based on a single run, with no error bars or uncertainty estimates.

Ablations. The only ablation study in the main text examines calibration-set size on synthetic data. None of the key design choices is evaluated individually, including OT versus random or nearest-neighbor matching, top-kk averaging, PCA dimensionality, the conditioning weight λcond\lambda_{cond}, or pseudo-mean conditioning. No ablation studies are conducted on the real-world task.

  1. The novelty and theoretical contributions are limited, and the presentation falls below the ICLR standard.

Theory. The central observation that in-sample residuals are optimistic under ERM is well established; the authors themselves cite Efron and Vapnik. The W2W_2 bounds follow directly from Jensen's inequality, the Cauchy–Schwarz inequality, and the reverse triangle inequality. Moreover, the theoretical analysis does not establish a connection between residual mismatch and "downscaling bias," despite bias being a central theme of both the title and the paper.

Presentation. The paper lacks a method overview figure and a dedicated related-work section in the main text. The main empirical evidence is limited to one table and two small figures. Several inconsistencies and typographical errors further affect readability, including "UQ$\mu$" in Table 1 versus "UC$\mu$" in the text, "SwinLR," "Eq. equation 5," and "Algorithm A.5." Figure 2 is also too small to read clearly.

Questions for authors
  1. How does ReMatch compare with training the residual diffusion model using only calibration residuals or K-fold out-of-fold residuals, while keeping the total amount of data the same? This is a key comparison for establishing the contribution of the OT step.

  2. How does ReMatch compare with a SwinIR-based CorrDiff baseline and a simple post-hoc spread rescaling method fitted on the calibration split?

  3. During training, the diffusion model is conditioned on the pseudo-mean μ~i=yi−r~i\tilde\mu_i = y_i - \tilde r_i, but at test time, it is conditioned on μϕ(x)\mu_\phi(x). Does this train–test mismatch in conditioning affect performance? What happens if the model is conditioned on μϕ(xi)\mu_\phi(x_i) during training instead?

  4. Averaging the top-kk calibration residuals may reduce residual variance, potentially working against the goal of increasing predictive spread. How sensitive are the results to the choice of kk?

  5. Could the authors report results from multiple random seeds, including error bars, for the experiments in Table 1?

1 comment

Dear Reviewer,

We have updated our manuscript based on your feedback. Specifically, we have revised the main experimental results in Table 1 to report the mean and standard deviation across 10 different random seeds. We have also added a discussion and additional experiments to better isolate the effect of residual distribution matching from the performance of the regression backbone, with detailed results provided in Table 7 of the appendix. Additionally, we have included an overview diagram in section 4 to help clarify our proposed method.

We would appreciate your feedback on whether our responses and additional results have adequately addressed your concerns. If so, we would be grateful if you could consider raising your score.

Thank you again for your time and consideration.

Comments on the paper

3 comments

My thanks to the authors for their response. However, some concerns remain, namely (1) Figure 12 does not show ReMatch having a higher-fidelity reconstruction of the spectral distribution; it looks like it could be the opposite case, (2a) I would still want to see experiments where no biasing operation is applied to the LR input/HR target, (2b) Figure 10 is useful, but it enforces my concerns about a degenerate mean-predictor under bias settings.

AuthorsOpenReview ↗

Response to 2a and 2b - Downscaling without additional bias

[2a] We understand your concern that the proposed method could harm downscaling performance in a no-bias setting. Since our main experiment, ERA5–HRRR downscaling, uses data from different sources, there exists inherent cross-source mismatch. We would like to clarify our controlled synthetic experiment on the BLASTNet dataset and provide additional analysis on the ERA5–HRRR experiment in the revised manuscript.

In our controlled synthetic experiment, we evaluated ReMatch under varying levels of downscaling bias (Figure 2 in updated manuscript). At Level 0, the LR input is obtained directly from the HR field through downsampling and interpolation, without introducing additional systematic bias. As shown in Figure 2, ReMatch maintains comparable RMSE while substantially improving SSR over CorrDiff, even without additional bias injection. In Level 0, there was no RMSE gain for ReMatch compared to CorrDiff but SSR improved. The proposed method becomes increasingly effective as the bias level increases. Figure 6 in the appendix provides channel-wise RMSE and SSR results for a more detailed comparison.

To further analyze how ReMatch performs in the low-bias regime on a real-world dataset, we conducted an additional analysis on ERA5–HRRR. We classified 6,000 samples (600 time steps × 10 channels) into 10 equal-sized bias intervals, where the bias level is defined as the sample-wise RMSE between the upscaled LR input and HR ground truth. As shown in Figure 15 in the appendix(please refer to the updated manuscript), ReMatch consistently improves RMSE and SSR across all bias levels compared to SwinIR and SwinIR-based CorrDiff. Even in the lowest-bias interval (0–10%), SwinIR achieves an RMSE of 1.3314, CorrDiff achieves 1.3342, and ReMatch achieves 1.2544. ReMatch also improves SSR from 0.2134 to 0.7255 compared to CorrDiff. We further illustrate qualitative samples from this lowest-bias interval in Figure 16 (Appendix A.15). These results show that ReMatch's increased ensemble spread does not come at the cost of reconstruction accuracy, even when the LR input is relatively close to the HR ground truth. Instead, ReMatch provides better corrections to the remaining errors while better quantifying the uncertainty compared to CorrDiff.

[2b] To address the reviewer's concern, we also added the corresponding LR input and HR ground truth in Figure 12 of the updated manuscript, providing additional context for the ensemble predictions and residuals shown in Figure 11. These figures help illustrate that ReMatch's larger residual variability is not simply adding arbitrary diversity, but can provide meaningful corrections to the biased mean prediction. Together with the quantitative improvements in RMSE and SSR, this supports our motivation that the residual model should correct the remaining errors from the mean predictor while representing unresolved uncertainty.

AuthorsOpenReview ↗

Response to 1 - Fidelity reconstruction of spectral distribution

We thank the reviewer for raising this concern, which prompted us to revisit our spectral evaluation procedure. Upon further investigation, we identified a spectral leakage issue in our original PSD computation. Specifically, the periodogram was computed without spatial windowing, which introduced artificial high-frequency power due to discontinuities at the field boundaries. This effect was particularly pronounced for the relatively smooth upper-atmospheric wind fields considered in our experiments. To address this issue, we recomputed the spectra using a Hann window to mitigate boundary-induced spectral leakage, applying the same procedure consistently across HRRR, ERA5, CorrDiff, and ReMatch. The corrected analysis reveals a frequency-dependent trade-off. CorrDiff achieves lower log-spectral distance (LSD) in the low-frequency (3.26 vs. 3.54 dB) and mid-frequency (3.00 vs. 5.15 dB) bands, whereas ReMatch achieves lower LSD in the high-frequency band (15.58 vs. 17.35 dB). However, both methods still overestimate high-frequency spectral power relative to HRRR. We therefore agree with the reviewer that ReMatch does not uniformly achieve higher spectral fidelity than CorrDiff. We have updated Figure 14 with the corrected spectral analysis and revised the corresponding discussion to explicitly acknowledge this limitation.

Importantly, the primary contribution of ReMatch is to address regression-induced residual distribution mismatch and improve probabilistic calibration, rather than to uniformly improve spectral reconstruction. We believe the corrected analysis provides a more accurate characterization of the spectral behavior of both methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.