acceptodds
Under review as a conference paper at ICLR 2027

Learning Clean Generative Processes from Noisy Measurements

16 views

Abstract

We study how to learn a generative model of clean images using only noisy measurements with a known noise law. Diffusion processes and their generalizations beyond Gaussian noise require posterior predictions at every noise level. We first show that it is possible to learn these predictions from independent noisy views using Noise2Noise regression, but only at or above the noise level of the training data. To learn the predictions below that level, needed to generate clean images, we impose self-consistency: a prediction should equal its expected value after continuing the model’s sampling process. We derive conditions under which self-consistency uniquely determines the continuation to the clean image distribution and verify them for Poisson, binomial, Gamma, and negative-binomial measurements. Experiments on CIFAR-10, microscopy data, and synthetic aperture radar show that the required posterior predictions can be learned from noisy views alone and that self-consistency repairs the continuation beyond the noise level of the data, which otherwise collapses.

Primary area:
generative models
Keywords:
diffusion, generative process, jump diffusion, blackout diffusion, noise2noise

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_iRAFOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper learns generative processes for clean images using only noisy measurements with a known noise law. It combines Noise2Noise targets to learn the posterior predictions needed by Gaussian, Poisson, binomial, Gamma, and negative-binomial processes at observed noise levels with self-consistency training to extend those predictions toward clean images; it also proves uniqueness of this continuation under stated conditions. Experiments on simulated CIFAR-10 measurements, photon-counting microscopy, and radar data show improved generation beyond the observed noise level and better capture of spatial uncertainty, though the implementations do not satisfy every theoretical assumption.

Critical strengths
  1. Substantive extension beyond Gaussian noise. The paper derives the posterior quantities and noisy training targets needed for several measurement laws, including Gamma and negative-binomial noise, and gives conditions under which self-consistency uniquely determines their continuation. This is a meaningful methodological and theoretical advance over applying existing Gaussian methods to mismatched data.

  2. Evidence across controlled and real measurements. Consistency training substantially repairs trajectories that collapse beyond the observed noise level on CIFAR-10 and photon-counting microscopy; the radar results also show better block-level uncertainty coverage than independent pixel sampling. These experiments support the method’s relevance beyond a purely theoretical construction.

Critical weaknesses

The evidence for reliable long-range generation is mixed. In several CIFAR-10 settings, the refined full trajectory performs worse than stopping at the observed level; the radar trajectory also underestimates uncertainty at high information levels. This limits the strength of the paper’s central practical claim.

Questions for authors

Do you observe any empirical pattern that predicts when continuing beyond the observed level improves or worsens sample quality? If so, could it guide the choice of stopping level without clean validation images?

6 comments

We thank the reviewer for their positive comment. We are happy to answer to their questions below.

W1, part 1: Refined trajectory is worse than stopping at the observed level. We have addressed these concerns in the revised version. We found that the mixed results were partly a consequence of a bug in the GPU-implementation of torch.poisson, which caused the training views for images with simulated Poisson noise to be highly correlated in a certain range of the intensity parameter. These results had motivated us to use the biased averaged estimator instead of the product estimator. We corrected the bug and switched to the product estimator. For the Poisson and negative-binomial models we also strengthened the consistency training, with a larger weight on the consistency loss (Table 9 of the revised paper), and for the negative binomial we extended the continuation. With these changes, continuation improves on or matches early stopping in all settings but one, binomial noise at 180.1180.1 counts, the highest information level for this noise model. Across settings, the gain from continuing shrinks as the information level rises. Since our original choice of estimator was based on results obtained with correlated views, we no longer use the biased averaged estimator in any experiment.

W1, part 2: The radar trajectory underestimates uncertainty at high information levels. In the radar experiment, we supervise the model up to Lobs=1/2L_{\rm obs}=1/2, half a look, and the radar data contain no reference beyond about 4.5Lobs4.5L_{\rm obs}. So this is the highest level where we can measure how well our model is calibrated. We reported that at 2Lobs2L_{\rm obs}, its 16×1616\times16 block-sum coverage is 88.688.6-89.2%89.2\% at a nominal 90%90\%, against 82.582.5-83.2%83.2\% for independent sampling (Appendix H.5). Beyond that band, the trajectory can only be compared with the network's own posterior, and it realizes 0.780.78 of the predicted per-pixel spread at 16Lobs16L_{\rm obs} and 0.620.62 at 32Lobs32L_{\rm obs}.

We traced this shortfall to the network's posterior, which does not sharpen as looks accumulate. The training targets constrain this sharpening only indirectly, through the consistency loss. We therefore fine-tuned a rerun of the model trained with both the consistency loss and the rollout regression, which before fine-tuning realizes 0.850.85-0.870.87 of the predicted spread at 16Lobs16L_{\rm obs} and 0.710.71-0.740.74 at 32Lobs32L_{\rm obs}. The fine-tune runs for 25 epochs with a four-fold consistency weight, without the rollout regression, and with a penalty that keeps the network's output on the observed data, at LobsL_{\rm obs} and 2Lobs2L_{\rm obs}, close to the starting model. It raises the fraction to 0.950.95-0.990.99 at 16Lobs16L_{\rm obs} and to 0.880.88-0.950.95 at 32Lobs32L_{\rm obs}, and the block coverage at 2Lobs2L_{\rm obs} is 90.090.0-90.3%90.3\%. A similar, coarser check against the data at 32Lobs32L_{\rm obs} rises from 0.820.82 and 0.880.88 to 0.970.97 on both seeds, against 0.640.64 and 0.730.73 for the reported model. The revised paper states the limitation and the fine-tune in the main text and reports the measurements in Appendix H.5.

Q1: About empirical pattern guiding the choice of stopping level. We observe that the noisier the training data, the more continuing beyond the observed level improves on the anchor. On CIFAR-10 the full trajectory improves on the anchor (the predicted posterior mean of the clean image at the observed level) by 9.09.0, 4.74.7 and 3.33.3 FID at 16.116.1, 178.4178.4 and 613.1613.1 Poisson counts, by 3.03.0, 3.03.0 and 0.050.05 at 8.08.0, 25.125.1 and 100.0100.0 binomial counts, and by 5.85.8, 2.62.6 and 0.10.1 at robs=5r_{\rm obs}=5, 1616 and 5050 for negative-binomial measurements. At 180.1180.1 binomial counts it is worse than the anchor by 2.22.2. The information level of the data is known without clean images, so this trend indicates where continuing helps most. It does not tell how far a learned continuation should be run, and we have no rule for choosing the stopping level without clean images. The revised paper states both in its Limitations.

Reviewer_iRAFOpenReview ↗

Thanks for the addressing

Thank you for the detailed response and the additional experiments. The revised CIFAR-10 results substantially alleviate my concern about the inconsistent benefits of continuation. I also appreciate the additional radar experiments and the clear distinction between agreement with the model’s own posterior and calibration against held-out measurements.

The observation that continuation helps more when the training measurements are noisier is useful. I acknowledge that this does not yet provide a stopping rule without clean validation data, and appreciate that this limitation is now stated explicitly.

Overall, the response strengthens the empirical support for the paper and substantially addresses my main concerns. The remaining limitations do not change my positive assessment, and I will maintain my weak-accept rating.

Dear Reviewer iRAF,

Thank you very much for acknowledging our rebuttal. Is there anything we could clarify that might improve your overall assessment (in particular: what are the weak aspects of the paper that can or perhaps simply cannot be addressed?)

We believe that both our theoretical and experimental results are strong and significant. We are the first (as far as we know) to define a generative process for clean images that works with a variety of noises and is truly useful in realistic imaging applications. Your category ratings support this. The remarks of other reviewers were nearly all resolved by mitigating the PyTorch bug which we discovered very recently (and your question is probably prompted by the same numbers). Fixing (mitigating) the PyTorch bug greatly improved all experimental results (please have a look: now our performance gains are uniform and the presentation is much simpler since we always do the same things). As you noted, the revised experiments provide a much stronger empirical support for the paper. In particular, we have expanded the revised paper to demonstrate that our method is effective on higher-resolution images, such as CelebA, and in settings where the noise is fixed before training rather than resampled at each epoch + we included comparisons with similar methods. Together with demonstrating that our method is robust to variations in hyperparameters, including the number of simulated paths used for the consistency loss, many of the initial limitations have been considerably softened.

We would be happy to address any remaining concerns.

Reviewer_84NMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learning clean-image generative processes from noisy observations under a known measurement law. It constructs the generative process directly for Gaussian, Poisson, binomial, Gamma, and negative-binomial measurements: Noise2Noise learns posterior predictions within the observed information range, while a self-consistency loss extends them into the cleaner, unsupervised range. The paper also derives conditions under which self-consistency uniquely identifies the correct continuation, and evaluates the method on CIFAR-10, photon-counting microscopy, and SAR data.

Critical strengths

1.The paper is well organized and clearly explains the connection between the measurement model, posterior prediction, and generative dynamics.

2.Learning clean generative models from noisy scientific measurements is an important and well-motivated problem, especially beyond Gaussian noise.

3.The combination of measurement-matched generative processes, Noise2Noise supervision, and self-consistent continuation is conceptually appealing. The accompanying identification analysis strengthens the contribution beyond a purely empirical loss design.

Critical weaknesses

1.The main identification results rely on exact global optimization, continuous or sufficiently rich coverage of consistency conditions, and regularity or common-density-ratio assumptions on the model outputs. In contrast, the implementation uses finite-step simulation, finitely many paths, finitely many predicted moments, and unverified regularity conditions; the Poisson and negative-binomial experiments also use the explicitly biased averaged estimator. Although the paper is transparent about these departures, the current experiments do not sufficiently establish that the theoretical mechanism remains reliable under these practical approximations.

2.The paper compares a Poisson-specific model with a rough Gaussian misspecification, but does not systematically study modest parameter errors, spatially varying or correlated noise, or mixed noise models such as Poisson-Gaussian noise. This limits the applicability of the method to real scientific-imaging settings.

3.CIFAR-10 FID is computed against the clean training set, and the consistency estimator is selected using FID. While this is useful for a controlled study, the paper should more clearly distinguish results that rely on a clean oracle for evaluation or selection from a deployable protocol with no clean reference. In the microscopy experiment, training includes frames from the test field of view and the FID reference is a 512-frame sum from that same field of view; moreover, the ordering is sensitive to the rendering/clipping protocol.

4.Although the microscopy and SAR studies provide useful real-data evidence, they cover Poisson and Gamma observations respectively and use task-specific evaluation protocols. It remains unclear whether the continuation procedure is stable on more complex, higher-resolution image distributions.

Questions for authors

1.The main theory and implementation differ in several respects, including the biased averaged estimator, finite-step rollouts, a finite number of paths, finitely many negative-binomial moment outputs, and unverified neural-network regularity conditions. Can the authors provide a more systematic empirical analysis of how each approximation affects continuation accuracy and sample quality? In particular, why does the theoretically biased averaged estimator outperform the unbiased product estimator in the Poisson setting?

2.How sensitive is the method to misspecification of the measurement law and its parameters? Please consider reporting robustness to errors in exposure/noise levels, spatially varying parameters, correlated noise, and common mixed-noise settings such as Poisson-Gaussian noise. If these settings are outside the intended scope, the paper should state this limitation more explicitly.

3.The CIFAR-10 experiments compute FID against clean training images and select the consistency estimator using FID. Please clarify whether any hyperparameter or estimator choices use clean images. Can the authors add evaluation against clean held-out test images and report a model-selection protocol that does not rely on a clean validation reference?

4.In the microscopy experiment, training includes the test field of view and the reference image is accumulated from frames of that same field. Beyond the current transductive protocol, can the authors report the consistency-refined generative process under a strictly cross-field, inductive split? Please also provide evaluation that is robust to the rendering-range and clipping protocol.

5.The controlled multi-law study is limited to 32x32 CIFAR-10. Could the authors add an experiment on a more complex and commonly used higher-resolution image benchmark, such as FFHQ or CelebA-HQ at 64x64? If compute is limited, a standard subset and one or two representative non-Gaussian measurement laws would be sufficient. Training should still use noisy views only, with clean images used solely for held-out evaluation. This would help establish whether the continuation procedure scales beyond CIFAR-10.

6.Relative to the closest methods for learning generative models from corrupted/noisy observations, such as Ambient Diffusion, DiffEM, or other known-corruption approaches, what is the advantage of the proposed method under matched observation settings and model capacity? If these methods cannot be directly adapted, please clarify the source of incomparability and include the closest feasible baselines.

5 comments

We thank the reviewer for their positive feedback. We are happy to answer to their questions below.

Q1,W1: Regarding numerical approximations. As requested by the reviewer, the revised version now includes a table showing how the numerical approximations affect performance (Table~12 of the revised paper). For the noisiest Poisson setting, with 16.116.1 counts, we vary the number of sampling steps, the number of steps per simulated trajectory, and the number of paths used to compute the consistency loss. For the negative-binomial sampler, we also vary the number of jump sizes retained. In general, we observe that our chosen parameters yield optimal or near-optimal performance within the tested ranges, though performance is stable across most choices. However, as we noted in the paper, limiting the negative-binomial jump sizes can result in large performance drops.

About your remark on the biased averaged estimator: In the original version of the paper, we used the biased averaged estimator to obtain stable estimates for the continuation of the Poisson trajectories. We subsequently discovered that the instability was caused by a bug in the GPU-implementation of torch.poisson. After correcting this bug, we switched to the unbiased product estimator for all experiments reported in the revised version. With independent views, 10,000 samples and the settings of the revised paper, the product estimator gives FID 6.316.31 at 178.4178.4 counts and 10.4410.44 at 16.116.1 counts, against 7.687.68 and 15.3315.33 for the averaged estimator with the settings of the submitted paper. At 613.1613.1 counts the two are level, 6.076.07 against 5.915.91, within the variation between repeated evaluations of one checkpoint.

In the revised paper, we also verify whether the assumptions needed for Theorem 3.1 hold or hold approximately in practice. Most assumptions hold by construction: the Poisson and binomial predictions are clipped to [0,1][0,1], the negative-binomial moments decay geometrically, and the Gamma predictions keep their shape parameter above 11. For the Gamma continuation, we evaluate the other bounds along the sampler trajectories, verifying empirically the Lipschitz bound in the state variable and the decay of the Laplace transform with jump size (Appendix H.2).

Finally, we use held-out test images to verify that consistency is enforced and that starting at aina_{\mathrm{in}} is sufficient. For the Poisson model at 16.116.1 counts, the consistency loss lowers the squared consistency error 4444- to 5353-fold (Appendix H.2). However, since the true posterior mean is unknown for real images, we cannot directly compare the model's predictions with it.

Q2: Regarding misspecifications and limitations. In the Limitations section of the revised paper, we now state that we have not tested these settings and that the method assumes noise that is independent across pixels given the clean image.

Among the settings mentioned by the reviewer, we already examined misspecification of the noise law in the original version of the paper (Table 11 in Appendix H.2 of the revised paper). There, we show that approximating the noise with a Gaussian distribution is harmful to the generation process.

Q3,W3: The revised version uses the unbiased product estimator in every experiment, so the consistency estimator is no longer selected using FID (cf. the pytorch GPU bug in the general response).

Regarding evaluation against held-out images, Table 13 of the revised paper reports FID for the same 50,00050,000 generated samples against the 10,00010,000 clean CIFAR-10 test images, which were not used to train any of the models. With this smaller reference set, all FIDs are 1.41.4 to 2.22.2 points higher, but the results show the same pattern as those obtained using the training set as the reference.

Regarding the use of a clean validation reference, in the revised version of the paper we state explicitly where and how clean images are used. For example, during experiments on CIFAR-10, the settings used to train with the consistency loss, such as its weight and the number of steps per simulated path, were selected using FID computed from 10,00010,000 generated samples against the clean training set.

Nevertheless, as reported in our answer to Question 1, our method is robust to different hyperparameter settings. We also used clean images to make a fair comparison with MAD and GAP, whose hyperparameters were selected using clean images as well.

Additionally, we have explored ways to make these choices without using clean images. For instance, in the experiments with one fixed pair of observations per image, we use the Noise2Noise loss on held-out noisy pairs to choose when the consistency stage begins (Table 14). For the radar data, no clean reference is available, and all choices are made using noisy views.

While these approaches do not yet provide a general protocol for selecting the settings used to train with the consistency loss without access to clean images (as we now state in the Limitations section of the revised paper), they nevertheless demonstrate the strength of our method.

Q4,W3: Regarding use of clean data in the microscopy experiment. We now report a strict cross-field split. Averaged over the held-out fields, the consistency-refined model is better than or level with GAP under each of three renderings, including one that doews not clip.

Our original protocol followed GAP, which also trains on frames of the test field of view. The public Conv-PC archive contains four fields of view. We now hold out each field in turn, train our model and GAP, with its original settings, only on frames of the other three, and compare 2,000 samples per run with the 512-frame sum of the held-out field.

FID needs 8-bit images, and GAP does not report how it maps photon counts to that range. We therefore divide each image by its mean and compute a rendering by using the fixed range [0,4][0,4] used in the paper, clipping it at its own 99.9th percentile, or mapping it by x/(x+4)x/(x+4). This map does not clip and is chosen such that it renders dark pixels almost as [0,4][0,4] does.

Averaged over the four fields, our FID minus GAP's is −10.5±3.1-10.5\pm3.1, −18.7±3.2-18.7\pm3.2 and +2.7±2.6+2.7\pm2.6 at 256×256256\times256 under the three renderings, and −6.8±4.9-6.8\pm4.9, +0.2±4.8+0.2\pm4.8 and −22.0±5.4-22.0\pm5.4 on 64×6464\times64 crops, ±\pm one standard error from the two runs per field. We are thus ahead by more than three standard errors in three of the six combinations of image size and rendering, and level in the other three. However, for a single held-out field the order can reverse. The revised paper reports this experiment in Appendix H.4 (Tables 16 and 17) and summarizes it in the main text.

Q5,W4: Regarding experiments on more complex image distributions. In the revised paper, we added experiments on CelebA at 64×6464\times64 with Poisson and Gamma measurements (Table 2 in the main text, Appendix H.3 and Table 15 of the revised paper). The models are trained only on noisy views of the 162,770 training images. We compute FID from 10,000 samples over two training seeds, against the 19,867 held-out validation images and against the training images. Against the validation images, stopping at aina_{\rm in} gives FID 18.9618.96 for Poisson at 178.4178.4 counts and 21.7321.73 for Gamma at Lobs=64L_{\rm obs}=64, and the full trajectory with the consistency loss gives 6.186.18 and 12.6812.68. Without the consistency loss, the Poisson trajectory gives FID 47.1547.15 and the Gamma trajectory diverges. Against the training images every FID is lower by up to 2.02.0. Continuing with the consistency loss thus improves on stopping at aina_{\rm in} at this higher resolution as well.

Q6: Regarding other methods for learning generative models from noisy observations. As far as we can tell, our method is the first to explicitly construct a generative process from the measurement law in a way that applies to a variety of noise models. This makes it applicable to real-world imaging problems well beyond the current literature. Ambient Diffusion learns from images corrupted by a linear forward operator with a nontrivial kernel, and its follow-up (Daras et al., 2024) from images with additive Gaussian noise. Our paper includes extensive references to related work on training diffusion models with noisy images, and these works assume Gaussian noise.

The methods we can compare with directly are DiffEM and GAP, which we evaluate with Poisson measurements on CIFAR-10 at 178.4178.4 and 16.116.1 counts (Table 4 of the revised paper). As a further baseline, we train a Gaussian diffusion model (EDM) directly on the measurements, without knowledge of the measurement law, so it reproduces their noise. To generate clean images using this model, manifold-attracted diffusion (MAD) samples it with a score that draws the samples toward cleaner images, again without knowledge of the measurement law.

We report FID of 50,000 samples against the clean training set. One noisy measurement of each training image achieves FID 68.9268.92 and 183.86183.86 at the two counts, and EDM trained on the measurements scores about the same, 71.0171.01 and 183.79183.79. With redrawn measurements, our full trajectory achieves FID 4.344.34 and 8.198.19, against 20.3820.38 and 77.6477.64 for EDM with MAD and 48.7148.71 and 50.9150.91 for GAP. DiffEM is formulated for a fixed dataset, so we compare it with our models trained on one fixed pair of measurements per image, which score 6.596.59 and 12.6712.67 against 20.9920.99 and 64.1764.17 for DiffEM. Our method thus has the lowest FID at both counts and in both settings. MAD's parameters and GAP's training settings were chosen on FID against clean images, which favours these two methods.

Reviewer_eYHaOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for learning clean generative models from noisy measurements with known noise model. It supports gaussian, poisson, binomial, Gamma and negative-binomial noise. The method can be presented in two steps, first, the model learns from pairs of noisy images using Noise2Noise approach, and then refined with a consistency constraint to guide generation towards cleaner images than the training data. The main theoretical contribution provides conditions under which this process recovers the correct clean image distribution. The method is evaluated on CIFAR-10, microscopy, and synthetic aperture radar data. On CIFAR-10 and microscopy, consistency improves FID compared with continuing the process without it. However, on CIFAR-10, continuing with consistency does not always outperform stopping at the noise level of the training data.

Critical strengths
  1. The paper provides a useful theoretical extension beyond Gaussian noise. It shows how noisy measurements can provide training targets for several noise types and gives conditions under which consistency can recover the clean-image distribution. This provides a useful theoretical basis for learning to generate clean images without clean training data.
Critical weaknesses
  1. The practical benefit of continuing the process is not consistently demonstrated. In 5 of 11 settings in Table 2, stopping early and returning the predicted clean image gives better FID than continuing with consistency. Since continuation toward cleaner images is central to the method, these mixed results weaken the empirical evidence for its practical advantage.
Questions for authors
  1. How does the performance change when training uses only one fixed noisy observation per CIFAR10 image?
2 comments

We thank the reviewer for their positive feedback. We are happy to answer to their questions below.

W1: The practical benefit of continuing the process is not consistently demonstrated. We have addressed these concerns in the revised version. We found that the mixed results were partly a consequence of a bug in the GPU-implementation of torch.poisson (cf. general response), which caused the training views for images with simulated Poisson noise to be highly correlated in a certain range of the intensity parameter. These (unknowingly) correlated views had prompted us to use the ``patched'' biased averaged estimator instead of the product estimator. We mitigated the bug and switched to the product estimator. For the Poisson and negative-binomial models we also strengthened the consistency training, with a larger weight on the consistency loss (Table~9 of the revised paper), and for the negative binomial we extended the continuation. With these changes, continuation improves on or matches early stopping in all settings but one, binomial noise at 180.1180.1 counts, the highest information level for this noise model. Across settings, the gain from continuing shrinks as the information level rises (see our answer to Reviewer iRAF). Since our original choice of estimator was a patch motivated by (then unknowingly) correlated samples from PyTorch, we no longer use the biased averaged estimator in any experiment.

Q1: Training with one fixed noisy observation per CIFAR-10 image. We reran the experiments with one fixed pair of noisy measurements per training image. This gives a finite set of noisy observations, which the network overfits faster. We therefore selected the network checkpoint based on a held-out set of noisy images. The results follow the same pattern as before: fixed noisy observations are enough to learn a clean generative process, and continuation with the consistency loss improves on early stopping in eight of the eleven settings, matches it for the negative binomial at robs=16r_{\rm obs}=16 and 5050, and is worse only for binomial noise at 180.1180.1 counts, as with redrawn measurements. Redrawing the noise nevertheless improves generation quality on average by 22 points in FID compared with using fixed observations, see Table~14 of the revised paper.

Thank you for the detailed rebuttal and additional experiments. Congratulations on identifying the bug! The corrected results and revised training substantially address my main concern regarding the inconsistent benefits of continuation. I also appreciate the experiments with fixed noisy observations.

The additional experiments provided in response to the other reviewers, particularly the higher-resolution CelebA results, comparisons with existing methods, and cross-field microscopy evaluation, further strengthen the empirical evidence.

Some practical limitations remain, particularly the lack of a general procedure for selecting hyperparameters and the stopping point without clean validation images. However, I think the authors have addressed most of the original concerns, and the remaining limitations do not significantly weaken the main contribution.

Overall, the rebuttal has improved my assessment of the paper, and I will increase my score to 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Comment: PyTorch Bug

We thank all reviewers for their detailed and positive comments. In this global comment, we explain a bug in the PyTorch GPU-implementation of Poisson noise that affected some of our experiments and how avoiding it addresses several concerns raised by the reviewers. We are also uploading a revised paper with updated results and all changes from the previous version highlighted. Our individual responses follow below. We hope that these explanations, together with our individual responses and the revised paper, will resolve your concerns and lead you to raise your scores.

Reviewers eYHa and iRaf noted cases in which continuation performs worse than the anchor. Indeed, we also found this puzzling, since the theory suggests otherwise, and the biased averaged estimator was an unsatisfactory workaround we introduced to mitigate it. In the meantime we have discovered a bug in the GPU-implementation of PyTorch torch.poisson function, which samples Poisson noise. Avoiding this bug resolves everything cleanly and makes the use of the averaged and biased estimator for the consistency loss unnecessary. For example, in the Poisson setting with 178.4178.4 counts, using the same training settings and 10,000 samples, training the product estimator gives an FID of 17.2117.21 with correlated views, but only 6.316.31 with independent views.

As a result, in the revised version of the paper we made sure that for all the experiments we ran torch.poisson exclusively on the CPU to avoid the bug. We also used only the product estimator for the consistency loss. This helps resolve many of the concerns raised by the reviewers, as we will explain in our individual responses.

The following script reproduces the bug:

import torch

torch.manual_seed(0)
print(torch.__version__, torch.cuda.get_device_name())

def report(label, x, y):
    samples = torch.stack((x, y)).double()
    r = torch.corrcoef(samples)[0, 1].item()
    print(f"{label:16s}: {r:+.4f}")

lam = torch.full((2**20,), 30.0, device="cuda")
report("CUDA consecutive",
       torch.poisson(lam), torch.poisson(lam))
report("CUDA single call",
       *torch.poisson(torch.stack((lam, lam))))

lam = lam.cpu()
report("CPU consecutive",
       torch.poisson(lam), torch.poisson(lam))

After the rebuttal, we will report the bug to PyTorch.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.