acceptodds
Under review as a conference paper at ICLR 2027

Beyond Coverage: Which Observations Share an Instance Decides What a Joint Model Can Learn

Abstract

Joint models learn from partially observed instances; designs are compared by coverage, how often each variable and pair is observed, not which observations share an instance. If three variables vary widely but their sum barely does, observing all three measures the sum's variance; observing pairs assembles it from far larger entries. For a Gaussian joint we define a design's information operator and prove that coverage fixes its trace, leaving its spectrum to which observations share an instance. The spectrum sets sample cost: at identical coverage, an isolated direction's variance costs a fixed number of instances when some instance observes all its variables and, for any estimator, one growing as the inverse square of that variance when none does. Moment and likelihood learners can rank coverage-matched designs oppositely. On budget-matched echocardiogram acquisition policies, our operator predicts the likelihood learner's covariance estimation error within 3.2% with known truth, and a bound from a small pilot, before any fit, orders the policies, coverage-matched pairs included, as held-out likelihood does. A constructed domain realises the separation at four orders of magnitude. Coverage fixes how much a design observes; which observations share an instance decides what an unrestricted joint model can learn. Code: https://anonymous.4open.science/r/beyond-coverage-code-3D5D.

Primary area:
learning theory
Keywords:
observation design, planned missingness, partially observed data, co-observation, Fisher information, covariance estimation, joint models, acquisition design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AQPyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how higher-order co-observation affects learning joint distributions beyond marginal and pairwise coverage. It introduces a whitened Fisher information operator, establishes a sharp sample-complexity separation for low-variance directions, and shows that moment and likelihood learners can rank coverage-matched designs oppositely. The results are supported by theoretical constructions, Gaussian and mixture-model experiments, real-world acquisition studies, and a structured neural trajectory model.

Critical strengths

1. Novel and meaningful theoretical results.

The coverage-matched constructions convincingly isolate the effect of higher-order co-observation, and the Ω(ε−2)\Omega(\varepsilon^{-2}) lower bound provides a strong characterization of its statistical importance. The distinction between identifiability and sample efficiency is particularly insightful.

2. A systematic connection between information theory and learning behavior.

The information operator connects observation design to likelihood estimation risk, while the analysis of moment estimators explains learner-dependent ranking reversals. The experiments include careful coverage-matched comparisons, statistical uncertainty estimates, and controls supporting the theoretical mechanisms.

Critical weaknesses

1. Limited generality of the theoretical assumptions and neural validation.

The constructive O(1)O(1) estimator assumes a known low-variance direction, while finite-sample recovery of unknown directions remains less established. Neither the Ω(ε−2)\Omega(\varepsilon^{-2}) separation nor the learner-dependent ranking reversal is directly tested in neural joint generative models; the neural experiment instead uses a segment-factorized objective.

2. Practical reliability and utility of the design diagnostic remain insufficiently demonstrated.

The pilot diagnostic relies heavily on eigenvalue flooring, with the data-determined directions alone producing a reversed ranking. Leading-order risk predictions also differ substantially from observed finite-sample NLL gaps, and no end-to-end experiment demonstrates improvements from optimizing a new acquisition policy using the operator.

Questions for authors
  1. How do the theoretical guarantees extend when the low-variance direction must be estimated from incomplete data rather than assumed known, especially with multiple nearly degenerate directions?

  2. Can the authors provide direct evidence for the Ω(ε−2)\Omega(\varepsilon^{-2}) sample-complexity separation or learner-dependent ranking reversal in a neural model that explicitly learns a joint distribution, rather than a segment-factorized objective?

  3. How sensitive are pilot-based rankings to covariance regularization and model misspecification? Can the authors demonstrate that an operator-optimized acquisition policy improves upon existing policies under the same budget?

Reviewer_a7JaOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study the problem of data design to learn a joint distribution from partially masked data. They argue that coverage (how often each variable and pair of variables is seen) is insufficient to decide on the optimal data design. For a Gaussian joint, they define an information operator. They show multiple examples, including echocardiogram acquisition.

For honest disclosure, I found the paper very difficult to follow, and my assessment should be read with low confidence.

Critical strengths

Assumptions are stated explicitly and full proofs are given in the appendix.

The finding about moment and likelihood estimators is interesting and not obvious.

The figures are clear and well presented, particularly figure 1.

Critical weaknesses

Clarity

This is a big issue. I admit this is not my area of expertise (as I have reflected in my confidence score) but I found the paper extremely hard to read and understand. The writing is dense and confusing. Key terms like instance, design and coverage appear in the abstract and introduction but are not defined until section 3. The problem setting is never explained clearly, to the extent that it took me a long time to even understand the basic setup. The authors discuss echocardiograms without introducing them or explaining what the problem settings is until section 4, where the description is brief and assumes familiarity with the domain, which is not fitting for a machine learning conference. This issue is persistent throughout the manuscript, and if not addressed, I cannot recommend acceptance.

Fit for ICLR

The problem the authors are solving is a statistics problem: Optimal experiment design for estimating a covariance matrix. The authors mention things like masked diffusion in the introduction, but this connection is never made explicit neither in the theory nor the experiments, aside from a very simple neural experiment. This makes me wonder if the paper would be a better fit for a conference/journal in statistics

Examples

The practical significance in the results section is lacking. The large separations only occurs in settings that were built with that purpose in mind. On the real data examples, the coverage-matched differences are very small, and in some cases not distinguishable from zero

Questions for authors

Could you add a plain statement of the problem setting early in the introduction, and define terms such as "instance", "design" and "coverage" before they are used?

How would you apply this to practical machine learning settings such as masked diffusion?

You mention that neural learners with a joint objective over all observed variables are not tested. Do you expect the conclusions to hold in that setting? Would it be possible to test that?

Reviewer_ADrdOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the limits of "joint" models, models that learn by combining partially observed instances. In particular, the authors show that a joint model with an instance that observes the full support has constant sampling complexity, while a model with no instance that observes the full support has a complexity that scales with the inverse square of the variance of the signal.

Critical strengths

The paper seems to study an interesting and potentially important question regarding how observation patterns affect the recovery of joint distributions.

Critical weaknesses

I found the paper very difficult to follow despite careful reading. Although my research is in an adjacent area, I struggled to understand the main theoretical statements and their significance.

I believe the presentation would benefit substantially from clearer explanations, more intuition behind the mathematical results, and a more gradual introduction to the technical concepts.

I acknowledge that my limited expertise in this specific area may have contributed to these difficulties. The technical contributions may be substantial, but I was unable to assess them reliably from the current presentation.

Questions for authors

Could the authors provide a more intuitive, step-by-step explanation of the main theoretical results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.