acceptodds
Under review as a conference paper at ICLR 2027

Enforcing probabilistic constraints with Data Consistent Inversion and Normalizing Flows

Abstract

Flow-based models such as Normalizing Flows (NFs) have become a powerful framework for learning and sampling from complex probability distributions. While highly expressive flow architectures can accurately approximate target distributions, they often introduce unnecessary computational cost due to overparameterization, which is particularly acute when only localized distributional changes must be incorporated into an existing stochastic model. In this work, we introduce a Data-Consistent Inversion (DCI)-based framework to rebase input distributions that allows for (1) systematic refinement of NFs to achieve desired distributional properties while avoiding excessive overparameterization, and (2) adapting pre-trained stochastic models. Using the iterative variant of DCI (iDCI), we construct a reweighted sample set from a NF base that is subsequently used to learn an updated base distribution whose push-forward constraints are satisfied by design. This enables the sequential refinement of coarse NF models, allowing them to approach a target distribution to a prescribed tolerance without retraining from scratch. Furthermore, the proposed approach provides a mechanism for adapting pre-trained stochastic models to distribution drift, offering a new perspective on domain adaptation through distribution-consistent updates. Numerical experiments demonstrate that the proposed methodology efficiently improves model fidelity while reducing the burden associated with full retraining, eliminating the need to retain or reuse the original target samples employed during pre-training. Although such samples can be still be incorporated when available, model improvements can be achieved using only the drifted observations and the pre-trained model itself.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Normalizing Flows, Data-Consistent Inversion (DCI), Density Estimation, Domain Adaptation, Sequential Model Updating.

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_1ygAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper integrates iterative Data-Consistent Inversion with normalizing flows to enforce observed distributional constraints through importance-weighted learning of input distributions. The framework supports progressive flow refinement and adaptation of pretrained stochastic models without modifying their forward operators.

Critical strengths
  • The framework combines iDCI with normalizing flows through a theoretically motivated weighted maximum-likelihood objective.
  • The method updates input distributions while keeping pretrained stochastic operators fixed, reducing the need for full retraining.
  • The proposed rebasing strategy progressively improves distributional consistency by learning corrective base distributions.
Critical weaknesses
  • Experiments are restricted to 2D and 10D synthetic settings, leaving performance on realistic high-dimensional distributions unverified.
  • Rebasing requires training additional normalizing flows and incurs higher runtime than both reported baselines.
  • Matching low-dimensional marginals does not guarantee recovery of the joint distribution or its dependency structure.
  • The paper lacks real-world applications demonstrating when localized distribution changes justify learning an additional normalizing flow.
  • The sensitivity of density-ratio weighting to distribution shift, support mismatch, and effective sample size remains unexplored.
Questions for authors

See Weaknesses and below:

  • How does the method perform as dimensionality increases to 100, 500, or 1000, considering accuracy, runtime, memory, and parameter overhead?
  • Can the authors demonstrate the method on a pretrained scientific simulator or image normalizing flow with localized distribution shifts?
  • How does NF rebasing compare with direct importance resampling or low-dimensional distribution updates under matched computational budgets?
  • How does effective sample size change under larger distribution shifts? How do density estimation and weighted NF training scale with sample size, particularly for datasets containing millions of observations?
2 comments

Thank you for the constructive review. We agree that the current experiments demonstrate feasibility rather than high-dimensional scalability, which is something we plan on emphasizing more clearly as we have stated for the other reviewers.

The intended advantage of the method is that the forward operator can remain fixed while the input probability measure is adapted from output-space distributional constraints via the DCI/iDCI framework. In a revision, we intend to make this scope more explicit and avoid broader scalability claims than the current evidence supports since our goal was to demonstrate the proof-of-concept of utilizing DCI as an adaptation strategy.

We also agree that a conceptual comparison with some simpler alternatives such as direct reweighting or resampling would be useful for framing the contributions more clearly. The current sequential NF baseline controls for staged architecture growth, but does not fully isolate DCI-guided rebasing from generic weighting strategies. In a revision, we intend to add a brief discussion of effective sample size and weight concentration as diagnostics for density-ratio instability and support mismatch, but the theoretical analysis is better suited for a future journal-length article in our opinion. We also intend to emphasize more clearly that enforcing low-dimensional marginal constraints does not, by itself, recover the full joint distribution as we have mentioned in response to other reviewers.

Finally, we intend to revise the computational language to clarify that one benefit is avoiding retraining or modification of the forward map, not universal reduction in wall-clock time.

Thank you for the clarification and narrowing the claims. My main concern remains unresolved. The paper still lacks comparisons with simpler but important baselines, such as direct importance reweighting or resampling. Also, the authors have acknowledged that the current work is primarily a proof-of-concept, with limited empirical validation. Overall, I don't believe the current version meets the standard for acceptance at ICLR. I will maintain my original rating.

Reviewer_jmxcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper combines Data-Consistent Inversion (DCI), its iterative version (iDCI), and normalizing flows (NF). DCI provides sample weights, which are used to train a flow through weighted maximum likelihood. Its iterative version can be used when the joint distribution is intractable, by sequential updates that satisfy the constraint of matching the marginals. The authors propose a two-stage procedure, where a first NF is trained on the samples, and then iDCI weights are used to train a second flow on samples from the base distribution of the initial flow. The authors also apply this framework to the case where they keep a trained model fixed and change its input distribution to match new output observations. The experiments cover a two-dimensional density estimation problem and a synthetic ten-dimensional adaptation problem.

Critical strengths
  • The main idea of identigying a connection between DCI weights and weighted flow training is interesting.

  • Adapting a model through its input distribution while keeping the model fixed is a potentially useful idea.

  • The density estimation experiment includes a relevant sequential-training baseline without iDCI.

Critical weaknesses
  • The connection between DCI weights and weighted flow training, even though interesting, seems quite straightforward.

  • The theoretical support for the proposed method is limited. The convergence results discussed in Section 3 are taken from previous work. The paper does not show how errors from density estimation, finite samples, and flow approximation affect the final constraints. It also provides no guarantee for repeated refinement.

  • The added value of the flow approximation is not sufficiently demonstrated. DCI already provides an updated density representation. A flow can make sampling and density evaluation more convenient, but the paper does not establish when these benefits justify the additional training and approximation error. The methodological contribution needs clearer justification.

  • The experimental evidence is narrow, they show feasibility, but provide little support for broader claims about the method. Both examples are synthetic, and the adaptation experiment only evaluates changes in three one-dimensional marginals. There is no real application, broader benchmark, or comparison with another adaptation method. Repeated runs and uncertainty estimates are not reported.

Questions for authors
  • Can the authors quantify the constraint errors introduced by fitting the flow and explain how these errors behave across refinement stages?

  • Can they demonstrate a practical advantage over simpler alternatives using the same observations and comparable computational budgets?

  • Can they clarify which settings avoid reuse of the original target samples, given the joint retraining in Section 6?

1 comment

Thank you for the review. We agree that the key methodological question is what is gained by combining DCI/iDCI with a learned density model. The DCI weights are not arbitrary importance weights: they are induced by explicit push-forward distributional constraints. The NF then approximates the resulting updated measure, providing direct sampling, density evaluation, and reuse with the fixed forward map.

It is a good point that the paper should more clearly separate the exact DCI update from errors introduced by finite samples, density estimation, and NF approximation. The current paper does not provide a complete error-propagation theorem as it was intended as a proof-of-concept of the framework utilizing non-optimized implementations. In a revision, we intend to make this limitation explicit and sharpen the discussion of these distinct approximation sources, which we are investigating for a future full-length journal article.

We also intend to clarify the repeated-refinement diagnostics. In particular, the expectation of an exact normalized density ratio is one by construction (this follows from the predictability assumption in the DCI framework), so concentration of the ratio around one is the more informative diagnostic.

Finally, the statement about not requiring original target samples applies to the adaptation setting, where the forward map is fixed. In a revision, we intend to distinguish this more clearly from the separate NF-refinement experiment.

Reviewer_qpYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates stochastic inverse problems. In particular, they work on connecting data consistent inversion with normalizing flows. The goal of DCI is to obtain a set of input samples that could have led to some observed output of a process or simulation. DCI provides weighted samples, which means it cannot be used to obtain a distribution over the input space space that led to the observed outputs. To address this, the authors note that normalizing flows can be used to map random Gaussian noise to the weighted samples obtained from DCI -- given a flow trained to do this, we would then have a tractible distribution we can sample from (Gaussian) to obtain samples from the target distribution (inputs that led to observed outputs).

Critical strengths

The intuition makes. I am not very familiar with normalizing flows / stochastic inverse problems, but the idea of connecting the two makes a lot of sense. If the goal is to obtain a distribution over the inputs for some process, then DCI on its own does not make sense. It is also interesting how this can be used to improve the outputs of the original process as a result of obtaining a better quality distribution over the inputs.

Critical weaknesses
  1. The goal of the paper seems unclear -- what is the general problem being being solved? Is it inverse problems? Is it improving normalizing flow models without retraining them? The paper would be much easier to follow if they had a preliminary section where they very briefly outlined the core problem being solved, the notation that will be used when discussing it, along with key definitions.

  2. The experiment section seems rather weak -- example 1 is 2d, and example 2 is 10d. Furthermore, both are mixture of Gaussians. Having a more diverse range of target distributions with higher dimensions would be useful. I would recommend looking into diffusion sampler / schrodinger bridge literature for experiments that might be tractible for this type of technique [1, 2].

  3. The introduction motivates the paper by describing the goal as being the recovery of the input distribution, but all the experiments are on the output distribution. In general, it was hard to map this framing to the actual experiments and evaluation.

[1] Underdamped Diffusion Bridges with Applications to Sampling. Blessing et al, 2025. [2] Denoising Diffusion Samplers. Vargas et al. 2023.

Questions for authors
  1. What is the core problem being solved? Is it matching some target distribution? For matching the target distribution, what are the constraints? Do we have access to data? As mentioned in the weakness, the introduction mentions obtaining the input distribution, but the experiments are only about matching a target output distribution.
2 comments

Thank you for the comments. We agree that the paper should state the unifying problem more clearly, which is related to our response to the emergency reviewer.

The core problem is: given an existing stochastic map and new distributional information about its outputs, update the input probability measure so that the push-forward satisfies those constraints without necessarily retraining the map. DCI/iDCI defines the measure update, and the NF provides an explicit representation of that updated measure.

The NF-refinement and pretrained-model-adaptation examples are two instances of this same formulation. In a revision, we intend to make this common problem statement explicit near the beginning of the paper and add a short notation/setup paragraph. We also intend to clarify why evaluation is performed in output space: the learned object is the updated input measure, but its defining requirement is that its push-forward satisfy the prescribed output constraints.

We agree that the numerical examples are modest in dimension because the paper is intended as a proof-of-concept of the framework and was implemented without any optimization of the methods. As we stated in response to the emergency reviewer (see our response to reviewer 2cd8), in a revision, we would narrow the empirical claims accordingly and also more prominently state that matching low-dimensional marginals does not, in general, recover the full joint dependence structure to avoid any confusion about this.

I thank the authors for their reply and discussion. Despite being a promising direction of research, the paper is stil in a proof-of-concept state and requires substantial further investigation. Therefore, in my view it does not yet meet the high standards of this venue. I will keep my score.

Reviewer_2cd8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework that combines (iterative) Data-Consistent Inversion and Normalizing Flow training. Further, the authors propose an iterative training strategy, that sequentially increases the size of the NF during training and refines the final fit.

Critical strengths

None

Critical weaknesses

The paper fails to convince me of its merit, because it is missing evidence that it is useful. Either from a theoretic side or an experimental one. The experiments serve as an illustration of how the proposed approach works (2D Gaussian mixture and 10D synthetic problem), but does not show performance of related work or state of the art methods. In Table 1 your approach takes 40% longer than the baseline, that contradicts your initial claim of lower compute needed in the Abstract (l028).

Because of the lack of evidence, it seems more like a preliminary proposal than a finished manuscript to me and thus, it does not meet the standard of ICLR.

Questions for authors

I'm sorry for this short review, but I was assigned here as an emergency reviewer.

Did you try out your approach in the benchmarks of Stimper et al. (2022); Issenhuth et al. (2022)? How do they compare?

3 comments

Thank you for taking the time to review our submission on short notice as an emergency reviewer. Below, we have responded to the points you have brought up and how we intend to revise the manuscript in the coming days that we believe addresses these points.

The current experiments are primarily proof-of-concept, and we would like to clarify that the central contribution is not simply sequentially enlarging a normalizing flow. DCI/iDCI constructs corrective weights from explicit push-forward distributional constraints; the NF then provides an explicit, reusable representation of the resulting updated input measure. Sequential NF refinement is one application, while the second is adaptation of a fixed pretrained stochastic map through its input distribution.

We can see how the Abstract can be read as overstating the computational claim. Table 1 shows that the present rebasing implementation is slower than the reported baselines. Our intended advantage is instead that, in the adaptation setting, the forward operator need not be retrained or modified and the original pretraining data need not be retained. In a revision, we intend to narrow this language accordingly to better emphasize the overarching goal, which is that this process/framework can achieve superior distributional approximation even when using a non-optimized implementation that results in a nominal computational overhead.

The suggested comparisons with Stimper et al. and Issenhuth et al. are relevant from a conceptual perspective. In a revision, we intend to clarify the distinction between their learned/reweighted bases and our DCI-derived weights, which are determined by explicit distributional consistency constraints.

Dear Authors, I have received your answer.

The fact that the experiments are primarily proof-of-concept reinforces me in my rating.

Just so I understand: When you say "In a revision, we intend to [...]" this is to say, that there is no updates to the manuscript or results that I should have a look at?

Thank you for responding to our comment. We have now uploaded the revised work for you to review.

We may have interpreted the intended ICLR standard differently when deciding to submit this work, so we would like to briefly explain our perspective. Specifically, our use of the term "proof-of-concept" is intended to characterize the scope of the numerical studies. The ICLR reviewer guidelines explicitly note that not all work needs to compete on an established leaderboard and that a lack of state-of-the-art results does not, by itself, constitute grounds for rejection. They instead emphasize whether the paper supports its claims and contributes new, relevant, and impactful knowledge. With that understanding, we intentionally scoped the present paper around establishing and isolating the proposed methodological connection rather than demonstrating broad benchmark superiority.

The revised manuscript clarifies this intended scope and makes explicit that we do not claim large-scale scalability, state-of-the-art performance, or universal computational savings. We certainly agree that broader empirical validation would strengthen the work, and it is possible that we misjudged how much such validation would be expected for this particular contribution at ICLR. However, we hope the paper can be evaluated based on whether the current experiments adequately support the claims actually made in the work, which we have sought to clarify further in response to your feedback and that of the other reviewers.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated manuscript now available

We have uploaded a revised manuscript that incorporates the clarifications motivated by the reviewer comments we have mentioned in our prior responses.

In particular, we have clarified the common measure-adaptation problem underlying the NF-refinement and stochastic-model-adaptation settings, narrowed the computational claims in the Abstract, clarified the distinction between DCI/iDCI-derived weights and generic importance weighting, emphasized the limitations of enforcing only low-dimensional marginal constraints, and added a brief discussion of effective sample size and density-ratio concentration as practical diagnostics.

We have also clarified the distinction between the NF-refinement setting, which uses the original target samples during joint retraining, and the distribution-adaptation setting, which keeps the forward operator fixed and does not require retaining the original pretraining samples.

We appreciate the reviewers' comments, which helped us improve the scope and presentation of these points. We hope the reviewers find that these revisions have also improved the manuscript when considering any possible updates to their scores.

Area_Chair_s9n9OpenReview ↗

Author-Reviewer Discussion ends in 1 day, 12 hours

Dear authors and reviewers,

Thank you to the authors for their detailed responses and for updating the manuscript yesterday. The discussion period closes in about a day and a half, so this is the last chance for a direct exchange before the reviewer-AC discussion starts on Nov 19.

Reviewers qpYZ and 1ygA: I haven’t yet seen a reply from you. Please read the authors’ responses and the revised manuscript, and let us know whether your concerns have been addressed. Even a short acknowledgment helps, and please update your recommendation if your assessment has changed.

Reviewers 2cd8 and jmxc: thank you for engaging. Please confirm whether the authors’ latest answers and the revision settle your remaining points.

Authors: please answer any open questions in the time left, focusing on the points most likely to affect the decision.

Thank you all for keeping the discussion constructive and timely.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.