acceptodds
Under review as a conference paper at ICLR 2027

PACT: Learning Couplings under Empirical Marginal Constraints

Abstract

Observations from different sources often provide rich information about individual distributions but limited evidence of how outcomes correspond. Learning reliable joint models from such partially paired data is a common challenge in conditional prediction and risk assessment. We introduce Pair-Aligned Conditional Transport (PACT), which combines relation learning with marginal calibration. We show that empirical marginal constraints during training not only shape model outputs but also systematically alter relation learning: adding independent endpoint observations can even reverse the direction of the leading learning bias. Building on this finding, we establish a theoretical link between objective bias and parameter displacement and derive a correction for first-order expected objective bias. Controlled experiments validate the predictions of the analysis, while applications demonstrate PACT's value for conditional prediction and cross-policy risk modeling. This work provides a statistical basis for understanding how dispersed observations jointly influence the learning of joint distributions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
joint distribution learning, optimal transport, marginal calibration, partially paired data, finite-sample bias, bias correction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jShjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACT which is a framework for learning joint distributions from partially paired data by combining relation learning with empirical marginal constraints. The authors analyzed how marginal estimation affects the learning objective and showed that additional unpaired observations can change the direction of the leading bias. Also, they proposed a correction to reduce first-order objective bias. The method is evaluated on controlled experiments and applications involving conditional prediction and risk assessment.

Critical strengths

I think the paper addresses an interesting problem, especially, when paired observations are limited but additional marginal information is available. The proposed framework is well motivated, and the overall structure of the paper is clear and easy to follow.

One of the main strengths is the theoretical analysis, particularly how the authors study the effect of empirical marginal constraints on the learning objective and parameter estimation. I also appreciate that the paper provides a correction method based on this analysis and investigates its effectiveness.

Another positive point is the experimental evaluation. The authors considered several settings, including synthetic experiments, Swiss roll, Colored MNIST, finance, and queueing. The experiments showed useful evidence supporting the theoretical findings. I also appreciate that the authors report cases where the proposed correction does not improve performance, which helps clarify the limitations of the method.

Critical weaknesses

My main concern is the reproducibility of the experiments. Although the paper provides detailed experimental settings and pseudocode in the appendix, I could not find the implementation code in the submitted materials. Since the method involves several numerical procedures and different experimental settings, I think providing the code would be important for independently verifying the results.

Another concern is the practical effectiveness of the proposed bias correction. While the theoretical analysis shows that the correction removes the first-order bias in the expected objective, the experimental results do not consistently demonstrate improvements in estimation performance. For example, Table 2 shows that the correction can perform worse than the uncorrected method under some settings. I think the authors should explain more clearly when the correction is beneficial in practice.

I also think the authors could further clarify the novelty of the framework compared to existing methods based on entropic optimal transport and calibrated likelihood. In addition, the theoretical guarantees are mainly established under finite discrete support assumptions, while some of the applications involve more complicated settings. It would be useful to discuss how these theoretical findings extend to the practical experiments.

Also, I had some problems with reading the math in the paper. I am not an expert in that field and not aware about the standard notations but I think that it is better to make the theory and proofs more readible and use better notations. Regarding the derivations and etc, I did not find major problems, but I did not spend too much time on it and might have some misunderstandings.

Questions for authors
  • Could the authors provide the implementation code and scripts needed to reproduce the main experiments?

  • The proposed correction theoretically removes the first-order expected objective bias, but it does not consistently improve the experimental results. Could the authors explain under which conditions the correction is expected to provide a practical benefit?

  • In Table 3, the mixed-sample likelihood approach does not outperform the paired-only alternatives. Could the authors explain the advantage of incorporating unpaired observations directly during relation learning instead of using them only for final marginal calibration?

  • Could the authors clarify the main methodological novelty of PACT compared with existing entropic transport and calibrated-likelihood approaches?

  • To what extent can the theoretical bias analysis be extended to continuous distributions and neural network models?

5 comments

Thank you for your feedback. PACT learns correspondence from limited pairs and uses marginal calibration to construct a joint distribution. With 128 pairs on Swiss roll, PACT achieved a conditional MMD of 0.108, versus 0.455 for native EBiEOT-GMM and 0.415 for a paired-only MDN, leading in all three confirmation replicates (Figure 4, Table 14). In simulated hedging, all methods shared the same empirical marginals. PACT achieved a dependence error of 0.1016, versus 0.1095 for inverse EOT and 0.1192 for projected pair diffusion. Both prespecified comparisons favored PACT after adjustment for multiple comparisons (Tables 5 and 32).

What is new. EBiEOT already combines paired and unpaired data, and our paired-only likelihood aligns with Sei and Yano’s profile likelihood under identical scores and exact calibration. Our contribution is to predict how the ratio of these two data sources changes what the model learns. When the same pairs provide supervision and help estimate the minibatch marginals, adding independent endpoints changes the weights on the variance and covariance terms in the leading bias. For example, in a binary model with uniform marginals and true correlation 0.7, the expected objective favors stronger dependence to first order with pairs alone. Adding one independent endpoint per pair on each side reverses that shift. Figure 3 tests the predicted bias directions. This analysis also explains why the split-jackknife preserves the sampling ratio in each half.

https://arxiv.org/abs/2410.02628
https://link.springer.com/article/10.1007/s10463-026-00995-8

When the correction helps. In a new synthetic Swiss-roll candidate-matching study with 4,096 training pairs per archive, the original correction reduced held-out log loss by 2.4% relative to uncorrected training at batch size 64, improving all ten independent confirmation archives under equal training-time budgets and development-based checkpoint selection. At matched 300-second checkpoints, the gain persisted in nine of ten archives, with lower mean top-1 accuracy. The primary comparison with full-archive training remained inconclusive (loss difference +0.011, paired 95% CI [−0.003, +0.025]).

Unpaired data in training. With final calibration held fixed in Table 3, mixed likelihood improved mean NLL over paired likelihood on Colored-MNIST, where paired contrastive learning performed best. Paired likelihood had the lowest mean error on Swiss roll and in finance. The framework lets practitioners choose the training objective for the task and still use unpaired data to calibrate the final coupling.

Theory and reproducibility. Theorem 1 applies to smooth neural scores on fixed finite supports under its regularity conditions. Continuously sampled supports require bounds that remain valid as the support grows. We will clarify the distinct roles of marginal and parameter curvature in the proof. The anonymous repository provides code, configurations, and run scripts for the finite-support studies and all four applications. The top-level README links to the setup and run instructions.

https://anonymous.4open.science/r/pact-review-7d2k/README.md

Thank you to the authors for the detailed rebuttal and additional clarifications. I appreciate their efforts to address my concerns, particularly by releasing the implementation code and providing further experimental evidence.


After reviewing the rebuttal, manuscript, and released code, I still have a few remaining concerns:

  1. Reproducibility: The code release substantially addresses my original concern. However, I could not locate the scripts, configurations or results corresponding to the new Swiss-roll experiment with 4,096 training pairs reported in the rebuttal. Since this experiment provides important additional evidence for the proposed correction, it would be useful to make it reproducible.

  2. Practical effectiveness: Although the new results are encouraging, the original experiments show that the bias correction does not consistently improve estimation performance. I remain somewhat unconvinced about when and how reliably the correction provides practical benefits.

  3. Theoretical scope and novelty: The clarification of the contribution relative to existing entropic OT methods is helpful. However, the theoretical analysis remains largely restricted to fixed finite supports, and its implications for the more complex applications are not fully established.


Overall, I recognize the interesting theoretical contribution and appreciate the authors' efforts during the rebuttal. Nevertheless, I believe the remaining concerns, particularly regarding the reproducibility and practical effectiveness of the correction, prevent me from increasing my score.

I therefore maintain my rating of 2 (weak rejection).

Thank you for pointing out the missing materials for the 4,096-pair Swiss-roll experiment. We have added a standalone supplement to the original anonymous repository. It includes data generation, training and evaluation scripts, frozen settings, and per-query scores for all ten confirmation archives. After unpacking the ZIP, run python3 reference/verify_reference_results.py from the supplement directory to reproduce the development selection and paired comparisons from the saved scores without a GPU.

https://anonymous.4open.science/r/pact-review-7d2k/conditional/swiss_roll_large_archive/README.md

https://anonymous.4open.science/api/repo/pact-review-7d2k/file/conditional/swiss_roll_large_archive.zip?download=1

With equal training-time budgets and batch size 64, the correction lowered matching log loss by 2.4% relative to uncorrected training, with improvements in all ten archives. The prespecified comparison with full-archive training remained inconclusive (mean loss difference +0.011, paired 95% CI [−0.003, +0.025]).

Our analysis shows that changing the ratio of paired to unpaired observations can reverse the leading bias in learned dependence even when the true relationship stays fixed. This makes the sampling ratio a training choice worth validating on held-out pairs. The theoretical guarantee applies to fixed finite supports. The Swiss-roll experiment tests the correction’s predictive effect in a synthetic neural matching task.

Reviewer_Tg3qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACT, which combines learned compatibility scores with Sinkhorn calibration to estimate couplings from partially paired data. The authors derive the first-order bias induced by empirical marginal calibration, characterize its effect on parameter learning, and propose a jackknife correction. Experiments evaluate these theoretical results and applications in conditional prediction and risk assessment.

Critical strengths
  1. Theoretical contribution: The paper provides an explicit first-order characterization of calibration-induced objective bias and shows that additional unpaired observations can reverse the direction of parameter displacement. This is an interesting statistical insight into learning under empirical marginal constraints.

  2. Theoretical validation: The controlled experiments provide convincing numerical support for the predicted bias behavior, particularly the sign reversal and asymptotic parameter displacement. The authors also report unfavorable correction results, which give a more balanced assessment of its practical limitations.

Critical weaknesses

My biggest concern is the limited novelty and unclear broader theoretical significance. The underlying formulation overlaps substantially with classical fixed-marginal dependence models and inverse entropic OT. While the bias characterization is interesting, its broader significance beyond this specific setting remains insufficiently justified.

Questions for authors
  1. Novelty and theoretical significance. While the bias characterization is interesting, I am not fully convinced of its broader significance. Could the authors clarify what they consider the key theoretical advance of this work, and why these results are important beyond the specific calibration setting studied?

  2. Practical implications. The proposed correction does not consistently improve estimation across different tables reported, while the main applications largely use different training objectives. Could the authors clarify how the theoretical findings translate into practical benefits, and which experimental results directly support these benefits?

2 comments

Thank you for recognizing the theoretical analysis and its numerical validation.

  1. Theoretical advance and broader significance

Our expansion predicts how the ratio of paired to unpaired observations affects the bias of minibatch calibrated likelihood. When pairs provide supervision and also help estimate the marginals, adding independent endpoints changes the weights on marginal sampling variance and shared-pair covariance differently. In the uniform binary model with true correlation 0.7, the leading shift of the expected objective’s minimizer favors stronger dependence with paired data alone. Adding one independent observation per pair on each side reverses that shift. Figure 3 and Table 23 test the predicted gradient directions and minimizer shifts, respectively.

The result applies to standard Sinkhorn inverse OT implementations using the same minibatch likelihood under our assumptions. For example, to align topic labels across two languages, a practitioner with paired bilingual articles and independent monolingual samples can compare two protocols on held-out pairs. One uses monolingual data in every training batch. The other uses it only for final calibration. Both can use identical final marginals. Our result explains why the number of unpaired observations per training pair is a design choice: it can change the direction of dependence bias even when the final marginals are identical.

  1. Practical implications and direct evidence

Table 1 shows lower estimation error with the correction in controlled finite-support settings. Table 2 shows that gains vary across settings. In a new synthetic Swiss-roll candidate-matching study with 4,096 training pairs per archive, the original correction lowered held-out log loss by 2.4% on average relative to uncorrected training at batch size 64, improving all ten independent confirmation archives under equal training-time budgets. The prespecified comparison with full-archive training was inconclusive (loss difference +0.011, paired 95% CI [−0.003, +0.025]).

The main applications assess the coupling framework under their respective training objectives. In simulated hedging, PACT captured cross-policy loss dependence more accurately than inverse EOT and projected pair diffusion under identical empirical marginals (Tables 5 and 32).

We would like to share a reproducibility update. Our anonymous repository now provides the implementation, configurations, and reproduction scripts for the main experiments. It also includes the new Swiss-roll experiment with 4,096 training pairs per archive, together with frozen settings, per-query results, and a CPU verifier for the reported comparisons. The README provides setup and run instructions. All materials can be browsed and downloaded without signing in.

https://anonymous.4open.science/r/pact-review-7d2k/

We welcome any further questions about our response or the released materials.

Reviewer_64ZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PACT combines relation learning from limited paired observations with calibration to empirical marginals; it studies how paired and independent endpoint samples affect expected-objective bias and local parameter displacement, including reversals in the direction of the leading bias. It derives a ratio-preserving split correction and evaluates the approach through controlled experiments and applications to conditional prediction and cross-policy risk assessment.

[Honestly, I am not an expert in this area, and not sure why I was assigned this paper]

Critical strengths

The correction is directly motivated by the theory. I also appreciate the distinction between cancelling expected-objective bias and improving finite-sample risk, which the experiments examine explicitly.

Critical weaknesses

The correction does not consistently improve finite-sample performance; but again this is far from my area.

Questions for authors

NA

3 comments

Thank you for your feedback. In our simulated hedging study, all methods used the same marginal loss distribution for each strategy. PACT captured how losses move together across strategies more accurately, with a dependence error of 0.1016 versus 0.1095 for inverse EOT and 0.1192 for projected pair diffusion. Both prespecified comparisons favored PACT after adjustment for multiple comparisons (Tables 5 and 32).

Our main finding is that adding unpaired observations can reverse the bias in how strongly the model links outcomes, even when the true relationship stays fixed. In our binary example, training on pairs alone biases the model toward stronger dependence. Adding one independent observation per pair on each side reverses the leading bias toward weaker dependence (Figure 3).

We also tested the correction in a new synthetic Swiss-roll matching experiment. The correction and uncorrected training both used batch size 64, a 300-second training budget, and checkpoints selected on development data. The correction lowered mean held-out log loss from 2.413 to 2.355 (2.4%), with improvements on all ten independent datasets, each containing 4,096 training pairs.

We would like to share a reproducibility update. Our anonymous repository now provides the implementation, configurations, and reproduction scripts for the main experiments. It also includes the new Swiss-roll experiment with 4,096 training pairs per archive, together with frozen settings, per-query results, and a CPU verifier for the reported comparisons. The README provides setup and run instructions. All materials can be browsed and downloaded without signing in.

https://anonymous.4open.science/r/pact-review-7d2k/

We welcome any further questions about our response or the released materials.

AuthorsOpenReview ↗

Summary of PACT’s contribution, experimental evidence, and public code

We thank the reviewers for their feedback and highlight four points.

  1. Core innovation

PACT quantifies how paired and unpaired data jointly affect learning under empirical marginal constraints. Even with the true joint distribution fixed, adding independent endpoints can reverse the leading learning bias. We derive the explicit bias coefficient, predict the local displacement of the expected objective’s optimum, and characterize the direction reversal. Figure 3 and Table 23 validate these predictions. The analysis also specifies a ratio-preserving split correction that cancels first-order expected-objective bias under the stated finite-support assumptions.

  1. Advantage in simulated hedging

Under identical empirical marginal loss distributions, PACT more accurately captures how losses move together across strategies. Its dependence error is 0.1016, compared with 0.1095 for inverse EOT and 0.1192 for projected pair diffusion. Both prespecified comparisons favor PACT after multiplicity adjustment, using five independent training datasets crossed with five independent simulated market datasets (Tables 5 and 32).

  1. Additional experiment directly testing the correction

We evaluated the original correction on ten independent synthetic Swiss-roll archives, each containing 4,096 training pairs. With the same model, batch size 64, equal 300-second training budgets, and development-selected checkpoints, the correction lowered mean held-out matching log loss from 2.413 to 2.355, a 2.4% reduction. All ten archives improved, with a paired 95% interval of [−0.085, −0.031] for the loss difference. The development-selected full-archive comparator achieved 2.344, with a paired interval of [−0.003, +0.025] for corrected minus full-archive loss.

  1. Reproduction materials are publicly accessible

The anonymous repository provides implementation code, configurations, and run scripts. The new Swiss-roll supplement includes frozen settings, development-selection records, all confirmation per-query scores, and a CPU verifier that reconstructs the reported comparisons. All materials can be browsed and downloaded without signing in.

https://anonymous.4open.science/r/pact-review-7d2k/

https://anonymous.4open.science/r/pact-review-7d2k/conditional/swiss_roll_large_archive/README.md

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.