acceptodds
Under review as a conference paper at ICLR 2027

Identifiability Precedes Generalization: A Pre-Fit Audit for Target–Domain Aliasing

Abstract

When the target label is determined by the acquisition domain, no amount of internal predictive performance establishes target-specific attribution from the observed data alone. Every cross-validated score computed from the labels equals the one computed from the domain-induced labels, so the competing explanations for the same number are indistinguishable. Distribution-shift evaluation asks how well a model trained on one distribution performs on another. This audit asks a different and prior question: what the recorded design allows a result to be attributed to. The two are complementary. The audit has a pre-fit structural part and a model-based diagnostic part. Structurally, under an additive target–domain model the classical connectivity of the bipartite support graph identifies which target main-effect contrasts are estimable, including in designs that are incompletely crossed, where some class never appears in some domain. The singular values of the residualized target basis measure proximity to aliasing, which the rank alone cannot. Diagnostically, a classifier two-sample test conditioned within a target stratum tests for domain-associated differences the target model could exploit. We report its operating characteristics and the shifts it misses. For a binary target and a score carrying no target information beyond the domain, we give an exact identity, valid at any class prevalence, between that score's AUC against the target and its AUC against the domain, given the class–domain allocation. On three datasets the audit separates different threats. On a sensor benchmark every additive target main-effect contrast is estimable despite incomplete crossing, yet random splitting reports accuracy 11.2 percentage points higher than leave-one-batch-out evaluation. In our Camelyon17 analysis the design is sound and hospital separability is nonetheless high: strong domain information and estimable target contrasts coexist. Grouping folds by slide lowers the reported AUC more than the subsequent hospital holdout does. In a clinical breath cohort disease status is inseparable from acquisition series, and two healthy series separate at AUC=1.00. The audit tells a study which contrasts its design supports and what its evaluation must block on. Applied to our own analysis it withdrew a date-prediction result, under random folds and 0.48 once folds are grouped by collection date.

Primary area:
datasets and benchmarks
Keywords:
identifiability, evaluation methodology, distribution shift, domain generalization, confounding, experimental design, classifier two-sample test, machine learning for health

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hssQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a pre-fit and diagnostic audit for target–domain aliasing, ie. the setting where target labels are partially or completely confounded with acquisition domains. The audit has four components:

  • A model-free observation that if yi=g(ci)y_i = g(c_i) deterministically, any empirical risk or cross-validated metric computed on (X,y)(X, y) is identical to that on (X,g(c))(X, g(c)), preventing target-specific attribution from observed data alone.
  • A structural estimability check under an additive linear mean model which uses the classical connectivity condition of the bipartite target–domain support graph to determine which target main-effect contrasts are estimable, supplemented by singular values to measure near-aliasing.
  • An exact identity for the target AUC of a nuisance-only score
  • A model-based diagnostic defined as the cross-validated AUC of a domain classifier trained within a single target stratum

The authors apply this audit to three datasets, UCI gas sensor drift, Camelyon17-WILDS, and an internal clinical breath mass-spectrometry cohort as well as a semi-synthetic degradation dataset constructed from it.

Critical strengths
  • The paper draws a useful practical distinction between cell-level positivity (whether all domain-by-class cells are occupied) and linear main-effect estimability under additivity (connectivity of the bipartite support graph), alongside a continuous conditioning measure via the principal angles of MZQYM_Z Q_Y. Applying this lens to the UCI gas sensor benchmark nicely illustrates how a design with missing cells can still identify all additive main-effect contrasts while requiring batch-blocked evaluation.
  • Relating the apparent target AUC(s,y)\text{AUC}(s, y) to the domain separability κ=AUC(s,c)\kappa = \text{AUC}(s, c) and class-conditional domain allocation (a−b)(a - b) under y⊥s∣cy \perp s \mid c provides a clean baseline formula for illustrating how partial domain alignment inflates cross-validated AUC in control-only reallocation experiments.
Critical weaknesses
  • A major portion of Contribution 3 (p 3, lines 108–113; Sec 5, lines 397–401) relies on characterizing the statistical operating characteristics of κ\kappa. However, the data-generating processes, MMD formulation, and experimental results for these claims are completely absent from the paper. App C is a 7-line stub that merely repeats the summary numbers from the introduction and self-referentially cites "the calibrated settings of §C". Similarly, App E promises "The reporting template below gives the checklist in two layers", but omits the template entirely. App F seems to consist almost entirely of methods copied from a different study ("Study A", "Study B" on 10-subject cross-day identification across 9 sessions, a 6-analyte "Targeted panel", serum creatinine/urea/ammonia correlations, and TabPFN v2) that never appear in the main paper, alongside broken cross-reference substitutions such as "(the leakage-cost analysis)" and ". the hyperparameter sweep".
  • Observation 1 is an immediate arithmetic identity and the bipartite support-graph rank condition Δ=L−k\Delta = L - k is textbook two-way ANOVA theory. Proposition 2 is a three-line consequence of the law of total probability applied to the Mann–Whitney statistic (whose proof is omitted in favour of "verification by simulation across 27 parameter settings", line 239). Furthermore, in a genuinely disconnected (R1) binary design where cases are collected in Domain 0 and controls in Domain 1, neither the same-target domain separability nor the score in Prop 2 can be computed across the confounded domains because no single target stratum spans both domains. The diagnostic can only be computed across surrogate domain pairs within the same class (e.g. two separate healthy control series), which measures general instrument drift and not the specific shift between the case and control acquisition series.
  • Beyond the technical flaws, the writing style of the paper makes it extremely hard to read and bears many characteristics of AI-generated text, specifically dense, chained technical statements that are hard to decipher, unearned metaphors, antitheses, "marketing copy" and so on, that are not good practice in scientific writing
Questions for authors
  • Please explain why App C omits the simulation protocol and results for Contribution 3, why App E omits the reporting template promised on line 687, and why App F describes experiments that are absent from the main paper
  • On pp. 3 and 7, you claim that the audit identifies the unit of analysis as the dominant error on Camelyon17. How does the formal audit of Section 3 detect hierarchical clustering within domains unless the analyst already knows to treat "slide" as a grouping variable and manually compares patch-level against slide-grouped cross-validation?
Reviewer_MaYJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper is very difficult to understand (for reasons I will expand on below), but this is what I think the paper is about:

When collecting data (especially in a biolab) from different domains (where a domain can be the batch, the site, the session, the instrument, etc.), then it can happen that the target variable, yy, (e.g. whether the sample is from a patient with a disease) can be perfectly predicted by the domain variable, cc. In other words, we have y=g(c)y=g(c) where gg is a deterministic function.

Among other things, the authors show that in such a situation, no statistical analysis of the trained classifier can distinguish whether the classifier is predicting yy from xx (the features) or predicting cc from xx and then transforming cc into yy.

The main contributions are:

  • a mathematical test that can be run on the data to determine whether any classifier can distinguish target and domain in principle (with some assumptions on the classifier)
  • an adaptation of the classic "classifier two-sample test" in order to check whether target and domain can be distinguished in practice
Critical strengths

None

Critical weaknesses

The paper is very hard to follow. I don't dislike LLM-written text in general, but in this case, I genuinely could not follow the paper due to the strong LLM-speak. The constant rhetorical flourishes put so much work on me to decode what it's saying, that reading this paper is a mind-draining slog.

Just some example sentences:

This is a nuisance-only reference: the performance a study would report from a score that reads acquisition and nothing else, and therefore the number a disease claim has to be argued against rather than compared favourably with.

The pipeline is therefore silent where nothing differs, and equally loud for a nuisance difference and a real one, which is exactly why the design must decide which it has found: no statistic computed from the features alone can tell drift from disease.

These sentences sound good, but I still don't really understand what they mean.

A very related problem is the constant use of synonyms. I get that when writing a novel, use of clever synonyms is encouraged, but in a scientific text, precision is what matters. We don't want to talk past one another because of the careless use of synonyms. I counted 4 synonyms for the term "domain" that are used constantly and without warning throughout the text: acquisition series, batch, nuisance, allocation. The term "target" has almost as many synonyms.

The structure of the paper is such that it essentially can't be read linearly (i.e. continuously from beginning to end) by a human. For an LLM, this is probably no problem because it can read everything at once, but to a human, it's just not possible to understand the introduction before one has read section 3, which is not how it should be!

Related to that is that many terms are just suddenly mentioned without being properly introduced or motivated. For example, there seems to be no section dedicated to how we want to define "target contrast identifiability".

The second problem of the paper is that, as far as I can tell, it mostly applies known mathematical results in order to figure out whether target and domain can be distinguished in a given dataset. There seems to be little real novelty here and the experiments are also quite simple. This paper might be better suited to a venue about applied machine learning.

Questions for authors

Please explain again, in clearer, ideally non-LLM-written words, what the contributions of the paper are.

Reviewer_nnf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pre-fit audit framework combining support graphs, estimability analysis, and same-target domain separability tests to assess whether target-related conclusions can be distinguished from acquisition-domain effects. Three real datasets and simulations illustrate its diagnostic use.

Critical strengths

1.The paper addresses a practical problem: high predictive performance does not necessarily justify target-specific interpretations. 2.The authors explicitly acknowledge the framework’s assumptions and diagnostic blind spots, helping clarify its scope.

Critical weaknesses

1.The abstract is overly long and detailed. Numerous theoretical details and experimental numbers obscure the central message. A more concise presentation would better highlight the contributions. 2.Core content is insufficiently developed. Some key derivations and experimental settings receive limited explanation. The available main-text space could be used to strengthen these aspects. 3.The introduction and framework overview lack clarity. The distinctions from distribution shift and leakage receive considerable attention, while the motivation and overall procedure remain difficult to grasp. A simple motivating example and a workflow diagram or pseudocode would help. 4.Mathematical presentation needs improvement. Key equations, including the estimability condition and AUC identity, are unnumbered. Appendix A also labels a connected but incompletely crossed design as R1, conflicting with the R2 definition in Table 1. 5.The framework’s added value is insufficiently validated. The experiments mainly provide case studies, without systematic comparisons against existing diagnostic and evaluation approaches. The calibration, power, and supplementary MMD experiments also need more complete settings and results.

Questions for authors

1.Compared with individual existing diagnostic tools, when does the complete framework change the scientific conclusion or evaluation strategy? Please clarify using the existing experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.