acceptodds
Under review as a conference paper at ICLR 2027

What Does an OOD Split Measure? Distance, Identifiability, and Completion Priors

Abstract

Out-of-distribution (OOD) benchmarks can be difficult for two different reasons: a test point may be far from the training inputs, or the training data may not contain enough information, under the chosen representation, to determine its prediction uniquely. We separate these cases on four dense six-component catalyst libraries. For each pair of elements , we hide every measured composition containing both elements and ask models to predict those held-out compositions from data in which and are observed separately but never together. A model-class-specific null-space audit shows that these test compositions can remain inside the training convex hull even though the – interaction is exactly unidentified; conversely, more distant composition-cap splits can remain fully identifiable. The resulting missing direction must therefore be filled by each estimator's inductive bias, or completion prior. An anchored difference transducer (ADT) recovers about 55–60% of the variance of the hidden interaction. Mechanism tests show that this recovery comes from a smooth separable model of property changes across composition space, rather than from low-rank structure in the pair-interaction matrix or lookup of previously observed moves. Exactly separable exponential-polynomial surfaces provide a positive control, while non-separable contamination rapidly removes the advantage. Finally, we test whether this structural match can be detected without using the true censored labels. A training-only “shadow censoring” diagnostic substantially lowers real-data model-selection regret relative to ordinary in-support holdout, although neither diagnostic reliably identifies the best method for each individual pair. The results distinguish geometric distance, design identifiability, and statistical predictability, and show that a formally unidentified interaction can still be predictable when the observed data contain transferable structure.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
out-of-distribution, transduction, completion prior, identifiability

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_DohDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors state that when trying to see how OOD your test set is, there are many things to take into account: how far the points are from the training data, if your training data allows for identification of the point under your model class (measured by the null-space fraction nu) and if the missing information is learnable. They mainly test this on 4 6-element catalyst libraries and look at 1 property with a composition grid of 0.1. When censoring all compositions that have both A and B you get points that have nu > 0. Capping compositions gets you similar distances with nu = 0. They then look at different ways to estimate the interaction that was censored (prior on completing the missing interaction). They find that the anchored difference transducer (ADT) performs best (better than a cubic) on the points that they all cover (RMSE of 25.5 vs 27.9 mV) and that the interaction it learns correlates 0.67-0.72 with the interaction learned from the entire library. They also show that using exponential ridge as a surface gives a positive control. At the end of the paper they show that you can use a training set diagnostic to select models that perform better on real data than if you randomly held out points, though this method does not outperform always selecting the ADT model.

Critical strengths
  1. I think it's important to distinguish between how OOD your test points are and if you are actually identifying those points with your model class, and if they are still predictable. The test that the authors provide using nu is helpful to see if you are doing that, which distance measures miss. Using the composition grid allows for a nice way to see this and I think it is well done. All of the numbers that they use for creating the different designs in Table 2 reproduce exactly.

  2. I think it is interesting to see that when you censor the AB interaction you can still learn some of that interaction if your model prior agrees with the structure of the data. The authors show that the ADT method learns a coefficient that is more correlated with the true coefficient (learned from the entire library) than most models (0.67-0.72 vs 0.37 for the cubic and -0.22 for the CV-selected GP). They also show with careful controls (including permuted anchors) that the model works when learning about moves that were seen and moves that were not. The authors did a good job showing the negative results as well.

Critical weaknesses
  1. I think this paper does not show that this works on other properties/materials. The authors' main results only use one property, one grid and one data source. The improvement is only a few mV and mostly comes from 2 of the 4 libraries. I think the title and last sentence of the abstract should reflect this (L31-32).

  2. There are a few claims that I think should be worded differently:

(a) When you are doing co-occurrence splits you are not removing points that only have one element. Therefore if you have all 6 single-element points measured, test points are guaranteed to be in the convex hull of your training set.

(b) In the abstract (L21-22) when they say they can predict 55-60% of the variance I think that should be worded as predicting 55-60% of the variance from 0, because if you were to explain the variance around the mean you would only be explaining up to r^2 (which would be around .45-.52).

(c) The authors mention that when they use a contamination residual the model can't learn the interaction (L743-744), even though the residual has a finite-rank move field. This means that having a finite separation rank is not enough to learn the interaction. I think this should be corrected at L421-423. The reason why the authors can learn the interaction in their positive control is because it uses some of the same parameters as the moves that are seen in the training set.

(d) The common test set (38,154 points) leaves out all points that are 4-5 away from the training set (where xA, xB >= .4). This should be mentioned in the main text. It is only hinted at when they mention the abstention rule (L612).

These can all be dealt with through wording and clarification, and I don't believe they detract from the ideas presented or the thoroughness of the analysis. As a result, I believe this paper should probably be accepted.

Questions for authors
  1. Are the points and aggregation used in Fig 1b, table of ladders (L699-701) and Tables 1/3 the same? When looking at GP-Cubic I get +9.3 mV for the isolated/ABC points from section 3.2 and fig 1b (L246-247). But from the table of ladders I get 7.2 mV. I also see that ADT-Cubic for the pair ladder is only -0.8/-1.0 mV whereas I see about a -2.4 mV difference in Tables 1/3. Is it because you are only looking at 10 out of the 15 pairs (L573)?

  2. Did you get any results for the cubic extension mentioned at L616?

  3. Since always picking the ADT model has lower regret than your shadow selection method (1.21 vs 2.46 mV), when would you recommend using shadow censoring?

  4. How does your work compare to the theoretical work that is done on learning about feature combinations that you haven't seen before? Dong & Ma ICLR 2023, Wiedemer et al. NeurIPS 2023.

2 comments
AuthorsOpenReview ↗

Response to Review

Thanks for the thorough feedback! We've fixed many of the arithmetic and numerical notes that aren't so clear in the edited the manuscript (line references to specific questions below). Replies below:

Weakness 2 (a) Convex hull. Lines changed: Submitted §3.1 L194–201 and §2 L116–123 → revised §2 L123–127, §3.1 L199–205; Appendix B.1. Agreed. Every one of the six pure-component vertices is retained in all 60 pair-censored training folds. Therefore every composition on the simplex lies in their convex hull. Wording changed.

(b) Meaning of the recovery percentage. Lines changed: Submitted abstract L21–22, §2 L149–155 and §3.3 L288–294 → revised abstract L21–24, §2 L152–160, Appendix B.5 L841–848. Agreed. The submitted 55–60% refers to the reduction in squared error of the retrospectively fitted hidden quadratic coefficient relative to zero completion, 1−SSEmodel/SSEzero1 - \mathrm{SSE}_{\mathrm{model}} / \mathrm{SSE}_{\mathrm{zero}} (sorry, can't get this to render right). We have corrected the abstract and metric description.

(c) Finite separation rank. Lines changed: Submitted Discussion L421–423, Appendix B.5 L743–744 → revised Discussion L425–429, Appendix A.3 L573–580. Agreed: finite rank alone is not sufficient. We've noted a direct counterexample of F(x)=cxAxBF(x)=c x_Ax_B, whose move field

F(a+δ)−F(a)=c(aAδB+aBδA+δAδB)F(a+\delta)-F(a)=c(a_A\delta_B+a_B\delta_A+\delta_A\delta_B)

has separation rank at most three but every training response and every observed move is independent of cc under AB pair censoring, so the coefficient cannot be recovered from training. We have revised the claim to require shared, transferable structural coupling as well as rank.

(d) Common prediction support. Lines changed: Submitted Table 1 L270–279 and Appendix B.2 L608–612 → revised §3.2 L258–260, Appendix B.3 L739–745. Agreed. The original 38,154-point common benchmark covers 38,154 of 39,413 censored fold–composition incidences. It excludes all 1,199 points at nearest-training distance four and 60 at distance five, because reach-three ADT lacks anchors there. We've fixed the note and explicitly noted that. As an additional inference-only sensitivity analysis (no retraining, Appendix B.3), extending the radius to five for excluded points gives ADT RMSE 34.92 mV on those points, versus cubic 38.86 mV and GP 51.21 mV; the mixed-radius full-region comparison is 25.81, 28.34, and 32.68 mV respectively for robustness.

Q1: Figure 1b, ladder table, and main benchmark. Lines changed: Submitted Fig. 1 caption L229–236, §3.2 L246–247, Table 1 L270–279, Appendix B.1 L573 and B.3 L699–701 → revised Fig. 1 caption L229–238, §3.2 L246–249, Appendix B.3 L689–700. These discrepancies are now exactly reproduced and explained. The submitted Figure 1b used pairwise method coverage for GP–cubic in some cells, while the appendix ladder table used all-method matched coverage. In isolation/ABC, the GP–cubic difference is +9.35 mV on 23,036 pairwise-covered point–file incidences but +7.16 mV on the 11,758 incidences covered by all methods. We've recomputed Figure 1b consistently + updated the manuscript. the qualitative result is unchanged. Added additional notes in the text.

Q2: Cubic extension of the midpoint-difference model. Lines changed: Submitted Appendix B.2 L616 → revised Appendix B.2 L654–655, Appendix B.3 L753–789, Fig. 6 L804–808. Very good suggestion with interesting results! The unregularized cubic midpoint (M5) reduces training-move RMSE from 31.15 to 27.11 mV and random within-training composition-holdout RMSE from 22.89 to 20.78 mV compared with quadratic midpoint (M4), yet worsens the censored-region RMSE from 28.39 to 56.47 mV. Great example of in-support validation favoring a poor completion prior. Ridge regularization helps (hindsight-selected raw ridge strength α=0.1\alpha=0.1 achieves 28.32 mV, approximately matching M4. But choosing its regularization using only ordinary held-out training compositions yields 41.51 mV, still much worse than M4 (28.40 mV). We report the full ridge sweep in Appendix B.

Q3: When should shadow censoring be used? Lines changed: Submitted §3.4 L363–404 and Fig. 3 L393–396; Discussion L428–438 → revised §3.4 L369–409, Discussion L435–445, Appendix B.7 L956–995. We agree that the existing benchmark does not justify using shadow censoring instead of a fixed ADT policy already known to work on these libraries. Our intended contribution is narrower. When an important combination has never been measured, its outcomes cannot be used for cross-validation (see e.g. cubic case with poor extrapolation above.

Q4: Dong and Ma (ICLR 2023), Wiedemer et al. (NeurIPS 2023). Lines changed: Submitted Introduction L60–72 and Appendix A.3 L567–571 → revised Appendix A.3 L582–587 and bibliography L486–487, L511–512. These papers are relevant and now cited.

AuthorsOpenReview ↗

New datasets

Hi, we identified three other scientific data sets that support similar censoring and have added these as a new Appendix C. As a result, we've been able to considerably clarify the claims of the paper: The main goal of the revision was to clarify the broader point: OOD performance isn't just about distance from training data, but also about what information is missing and whether a model's assumptions allow it to recover that information.

Across the datasets, we find that interaction censoring produces formally unidentified directions, but whether these interactions can be recovered—and which model does best—depends strongly on the dataset. We've therefore moved away from presenting ADT as a particularly strong extrapolation method, and instead use it as one example of how a suitable completion prior can recover missing information.

We've also expanded the shadow-censoring analysis. It shows promising agreement with dataset-level model rankings using only training data, although it's much less reliable for individual interactions and has clear limitations.

Finally, we've incorporated the additional literature suggested by the Area Chair and strengthened our analysis of ID–OOD performance relationships.

We've kept the catalyst experiments at the center of the main paper because they provide the cleanest controlled demonstration, while the new datasets and their limitations are discussed in the appendix. Overall, we think the revisions make the paper's argument considerably clearer and better supported across different experimental settings.

Reviewer_jrN9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses mathematical proofs and empirical tests to answer disentangle when OOD generalization fails because a test point is far from the training inputs and when it fails because the training data does not contain enough information, under the chosen representation, to determine its prediction uniquely. It first shows empirically that geometric distance and design identifiability can be varied independently on a dataset of "on four dense experimental six-component catalyst libraries (Rohr et al., 2020)." Then, it studies the inductive bias of how different estimators fill the cross-component interactions that do not appear in the training dataset. It proposes a specific form of cross-validation to hide cross-component interactions from training, and use that as a proxy to measure OOD generalization (under missing interactions) for model selection without true OOD data, a technique which the authors term "shadow censoring".

Critical strengths

The topic of disentangling why OOD generalization fails by analyzing cases of high input distance and unidentifiability of variable interactions in the training dataset is very interesting to me.

The paper clearly demonstrates that geometric distance (of OOD inputs) and design identifiability (of ODD targets) be varied independently, and neither can explain failures in OOD prediction alone. Having high geometric distance can still allow easy predictions if the model has seen sufficient inter-variable interactions, though the magnitude might be different from that of test-time inputs. And "formally unidentified interaction can still be predictable when the observed data contain transferable structure."

Reproducibility seems very good. Appendix B contains dataset and training details. Authors also promise "code, register and outputs will be released with the paper."

Critical weaknesses

The majority of the paper's formal analysis of how algebraic identifiability (can the parameters be identified under a particular model class in a particular training dataset) differs from good completion priors (does it nevertheless generalize to theoretically unidentifiable input/output pairs) is on the polynomial regression setting. For neural networks, such as the ADT setting included in the paper, I think in practice people use overparameterized neural networks whose outputs on test points are generally not identifiable, yet it does not prevent large neural networks from generalizing. In fact, larger and deeper neural networks tend to generalize better in many real-world applications, which is already a well-known fact. Because of this, I do not believe that algebraic identifiability is highly relevant in complex, real-world applications, where overparameterized neural networks do not generally admit any notion of algebraic identifiability.

I also feel that using only one type of materials interactions dataset feels too narrow to support the more general claims about OOD generalization that the authors are making. Yes, model selection with shadow censoring may be effective on this dataset, but what about other datasets where the best inductive biases for missing interactions are different? Do the same completion priors that win out here also apply to other data that have more complex interactions? The relevant scope of the paper seems very limited to me as it is.

The prose feels sloppy. E.g., L363-374 feels like it's just throwing out data and numbers without doing a good job interpreting the results and their significance. It makes it a bit hard to know as a reader which points are important and what points to take away from the results section.

Questions for authors

What can you say about algebraic identifiability and completion priors in more general settings (e.g. more than 6 variables) and other function classes such as MLPs and transformers? What insights can be drawn there?

4 comments
AuthorsOpenReview ↗

Response to Review

Thanks for the feedback! We've edited the manuscript (references to specific questions below). Here are some replies on the specific points raised:

Relevance to overparameterized networks. Lines changed: Submitted §1 L46–59 and §4 L407–421 → revised Appendix A.1 L531–539; Discussion L412–428. This is an interesting note. We respectfully disagree with the conclusion that identifiability becomes irrelevant because overparameterized neural networks can generalize successfully. Furthermore, in many scientific or real-world data cases, the data volume is too low to train an overparametrized network (further complicated by the complex underlying data and function).

A prediction can be formally unidentified by the training data and nevertheless be statistically predictable under a suitable inductive bias. We do not claim that identifiability is necessary for generalization, nor that identifiable models necessarily generalize better. Rather, we distinguish what information the training data determine from what a model supplies through its completion prior. Perhaps overparametrized networks learn this completion prior which could explain their generalisation.

Nonetheless, this is relevant to overparameterized neural networks. A model may produce a unique prediction after training, but that does not imply the prediction is uniquely determined by the available observations. Architecture, regularization and optimization can select among functions that agree on training data but differ on unobserved combinations.

To use a specific example: On our nonnegative composition simplex, the function (g(x)=\min(x_A,x_B)=x_A-\mathrm{ReLU}(x_A-x_B)) is exactly zero on every pair-censored training composition and positive on every held-out AB composition. Thus, a sufficiently expressive ReLU network can represent distinct continuations (f_0(x)+cg(x)) that are indistinguishable on training data but differ on the censored region. Overparameterization does not remove this ambiguity.

We aim to calculate this identifiability and its dependence on distance. Our main empirical example already uses a neural estimator: ADT successfully predicts part of a missing interaction despite its formal non-identifiability under the polynomial audit. Its success varies with the structure of the response surface and deteriorates when that structure is disrupted. This is precisely why distinguishing information supplied by the data from assumptions supplied by the estimator is useful.

We have clarified the distinction between parameter identifiability, prediction identifiability and statistical generalization in Appendix A. We agree that extending exact quantitative identifiability audits to arbitrary neural architectures is a separate and challenging problem, but it is not required for the distinction or the empirical conclusions established here.

More components and model capacity. Lines changed: Submitted §2 L116–123 and §4 L407–421 → revised Appendix A.1 L531–539 (dimension counts). On an unrestricted kk-component simplex, censoring one pair leaves the degree-dd polynomial subspace xAxBℓd−2(x)x_Ax_B\ell_{d-2}(x) unconstrained, with dimension (k+d−3d−2)\binom{k+d-3}{d-2}. In our six-component design its measured dimension is 1, 6, and 21 in degrees two, three, and four, respectively. The principle is therefore not restricted to six inputs. At sufficiently high degree, grid support restrictions also matter; at degree five the measured dimension is 52 rather than the unrestricted 56. We do not claim that these polynomial counts directly characterize MLPs or transformers.

Scope and presentation. Lines changed: Submitted abstract L31–32, Results §3.4 L363–374, Discussion L421–438 → revised abstract L31–34, §3.4 L369–409, Discussion L429–445, Appendix B.3 L747–751. We agree that four related experimental libraries and one property are not enough to establish general superiority of ADT. The effect is heterogeneous: five-seed mean pooled ADT-minus-cubic RMSE differences are −2.87, −0.27, −4.78, and −1.16 mV across the four libraries. The shadow censoring is meant to be a structural diagnostic prior to extrapolation, highlighting that extrapolation info can be learned from current data. We've added a result with a changed cubic basis in the appendix that highlights that good prediction in-distribution doesn't guarantee anything OOD. Nonetheless, we tightened these limitations in the abstract and Discussion, with additional interpretation in Appendix B. The transferable conclusion is that the data's structure determines which completion prior can work, not that ADT is universally the right model (even if it performs well on these libraries).

I thank the authors for the thorough response. Taken as a work that discusses separating the information that training data determine from what a model supplies through its completion prior, I think the empirical results in the paper are well-scoped to support this claim. I will raise my rating to 3.

Thank you, and thank you for the feedback, it helped the paper.

AuthorsOpenReview ↗

Three new datasets

Hi, we identified three other scientific data sets that support similar censoring and have added these as a new Appendix C. As a result, we've been able to considerably clarify the claims of the paper:

The main goal of the revision was to clarify the broader point: OOD performance isn't just about distance from training data, but also about what information is missing and whether a model's assumptions allow it to recover that information.

Across the datasets, we find that interaction censoring produces formally unidentified directions, but whether these interactions can be recovered—and which model does best—depends strongly on the dataset. We've therefore moved away from presenting ADT as a particularly strong extrapolation method, and instead use it as one example of how a suitable completion prior can recover missing information.

We've also expanded the shadow-censoring analysis. It shows promising agreement with dataset-level model rankings using only training data, although it's much less reliable for individual interactions and has clear limitations.

We've kept the catalyst experiments at the center of the main paper because they provide the cleanest controlled demonstration, while the new datasets and their limitations are discussed in the appendix C. Overall, we think the revisions make the paper's argument considerably clearer and better supported across different experimental settings.

Reviewer_4k4aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that OOD difficulty entangles three distinct concepts that benchmarks rarely separate: geometric distance (is the test point far from the training inputs?), identifiability (under a chosen model class, do the training data determine the prediction?), and statistical predictability (is the structure of the data exploitable by particular indictive biases?). On a chemistry dataset, the authors focus on the main task of pair-co-occurrence censoring (removing every composition A-B from the training set) and compare different estimators' performance against multiple controls (e.g., random holdout, composition caps, censoring "ladder"). The authors propose a "shadow censoring" diagnostic tool for testing structural priors by conducting censoring in the training data with a held-out validation set, though it fails to beat the baseline of always selecting a fixed estimator.

Critical strengths
  1. The conceptual framing used to separate geometric distance from design identifiability is refreshing and is practically important to consider in the context of claims about OOD benchmark difficulty.
  2. Mechanistic study of why a particular estimator may have an advantage through experimental ablations and synthetic data construction, guiding the operationalization of how to better understand completion priors.
Critical weaknesses
  1. Empirical benefits are too narrow to support practical claims, specifically small margins relative to noise. The shadow censoring underperforms compared to simply always choosing ADT and shows no practical selection advantage.
Questions for authors
  1. Do method rankings on interaction recovery hold when the reference comes from a cubic fit to the full library? If the cubic fit rises under a cubic reference, the recovery comparison party reflects the choice of basis.
3 comments
AuthorsOpenReview ↗

Response to Reviewer

Thanks for the feedback! We've edited the manuscript (references to specific questions below). Here are some replies on the specific points raised:

Cubic-reference interaction recovery.

Relevant changes to manuscript: Submitted §2 L148–157 and §3.3 L288–294 → revised Appendix B.5/Table 4 L831–865. We ran the relevant robustness study. Re-extracting the full-data AB function with a cubic Scheffé mixture model gives ADT a recovery score of 0.351 relative to zero completion, against 0.105 for the direct cubic, compared with 0.541 and 0.272 under the submitted quadratic reference. ADT's advantage under the cubic Scheffé convention is positive across all five neural seeds (+0.196 to +0.296 in score difference). Thus the direct cubic does not overtake ADT simply by changing the reference to this conventional cubic mixture basis.

We've also changed some wording in the manuscript. The six-dimensional space of cubic functions that vanish on pair-censored training data is intrinsic to the split and function class, but extracting an "AB component" of a full function requires choosing a complement. In particular, homogeneous cubic and orthogonalized variants produce different attributions. We changed the wording to explicitly note it as basis-convention-dependent recovery rather than measurement of a unique interaction. The direct held-out RMSE comparison (25.5 versus 27.9 mV) does not depend on this reference definition.

Magnitude of the gain, noise and model choice.

Relevant changes to manuscript: Submitted §3.2 L256–264 and Discussion L421–425 → revised Discussion L429–431, Appendix B.3 L747–751; original benchmark Table 1 L270–279 / revised Table 1 L270–280.

We would argue that the difference across models is significant, but that it simultaneously heterogenous and relatively modest. Across five seeds the pooled ADT-minus-cubic RMSE advantage is about −2.5 mV, concentrated in Libraries 1 and 3 (approximately −2.87 and −4.78 mV); Libraries 2 and 4 show much smaller gains (−0.27 and −1.16 mV). The single-composition measurement-noise is complex here, because 4th order method will overestimate the noise significantly. Specifically, we believe that the example of how the ADT performance with noise degrades is indicative of a much lower true noise. However, due to the non-analytic nature of the data, we don't know the true noise level. It is particularly difficult to interpret true noise in this type of data, and we believe relative model comparison is therefore the correct way to measure performance (where ADT shows consistent, if modest, performance improvements).

Shadow censoring.

Relevant changes to manuscript: Submitted §3.4 L363–404 and Discussion L428–438 → revised Discussion L435–445 and Appendix B.7 L975–995; in-support midpoint control at revised Appendix B.3 L753–789.

Agreed: it does not outperform the fixed policy of always using ADT on these libraries, and it cannot reliably select the best method for a given pair. Its narrower contribution is to make a training-only test of a model's compatibility with missing-combination extrapolation possible when the relevant test labels are unavailable. It improves over ordinary in-support holdout on these real folds, but this does not establish a general selection algorithm. Nonetheless, I think it's important to note that good in-distribution prediction does not guarantee good extrapolation, and shadow-censoring is meant to act as a measure of OOD performance estimation. The added new cubic midpoint control reinforces this: adding cubic terms improves held-out in-support composition accuracy (20.78 versus 22.89 mV for quadratic midpoint) while substantially worsening missing-combination prediction without an appropriate completion prior (56.47 versus 28.39 mV).

I thank the authors for their response and added experiments. I retain my original score of a 3 (weak accept).

The revision narrows the scope of shadow censoring to a training-only diagnostic, rather than a model selector. However, the response's claim that shadow-censoring estimates OOD performance is not supported by evidence in the manuscript. The cubic midpoint control shows that in-support accuracy can diverge from OOD performance, but it does not test whether shadow censoring would have identified this model as poor while ordinary holdout preferred it.

AuthorsOpenReview ↗

Three new datasets

Hi, we identified three other scientific data sets that support similar censoring and have added these as a new Appendix C. As a result, we've been able to considerably clarify the claims of the paper:

The main goal of the revision was to clarify the broader point: OOD performance isn't just about distance from training data, but also about what information is missing and whether a model's assumptions allow it to recover that information. The datasets show this quite cleanly, each dataset performs differently, but its performance can be measured through interactability and estimated in the training data. We've changed the claims of the paper appropriately.

Thank you for the feedback, we believe the new version of the paper is considerably improved compared to before review

Reviewer_Q3iNOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper looks at what makes an OOD split difficult, and in particular whether the difficulty comes from how far the test points are from training data or from information that's missing altogether. The authors use four catalyst datasets to show that certain element interactions can be unidentifiable even when the test points are inside the training convex hull. They also evaluate an anchored difference transducer (ADT) to predict the held out compositions and explore whether shadow censoring can help select models without needing access to the actual test labels.

Critical strengths
  • I think the distinction between geometric distance and identifiability is really useful, especially since the experiments show that they can behave quite differently. This matters because OOD benchmarks may be measuring different things under the same label.
  • I liked that the authors try to explain why ADT works rather than just reporting better results. The synthetic experiments help make the explanation more convincing. This makes the result more trustworthy.
Critical weaknesses
  • Most of the experiments use four closely related catalyst libraries, so I'm not sure how well the findings would generalize to other materials datasets (the other datasets in the appendix are only used for identifiability checks). Similarly, I'm not sure how useful these findings are for other materials science tasks
  • Shadow censoring improves over ordinary holdout, but it doesn’t beat always choosing ADT on real data, or always choosing GP on synthetic data, so its practical benefit is unclear.
Questions for authors
  • How much would the reported recovery of missing interactions change if the authors used a different basis? Like if the ground truth coefficient were defined with a cubic instead of a quadratic fit?

  • How can the findings of this paper be used to inform ML for materials science? The tested libraries are quite similar, so it is hard for a practitioner to trust how generalizable the results are. More importantly, the tested settings and models are extremely simple: Most property prediction tasks are dominated by significantly more complex models (e.g. GNNs). Many of the paper’s findings are about when ADT does or doesn’t do well, but its not clear that ADT is useful for the field or that these findings can be applied to any of the models devoloped over the last 5 years.

4 comments
AuthorsOpenReview ↗

Response to Questions

Thanks for the feedback! We've edited the manuscript (references to specific questions below). Here are some replies on the specific points raised:

Data on these libraries We're very much agreed that four related catalyst libraries aren't generalisable, but believe that the work is nonetheless valuable due to the fact that we can directly separate distance and identifiability on this data set (which is rarely the case). In addition, the experimental data present makes the task much harder (c.f. the controls from synthetic data), which we believe still make the primary contribution / central claim that different structural assumptions can succeed or fail depending on the data-generating response surface and that a particular estimator isn't always optimal. We believe that further data sets are beyond the scope of the paper as-is, but plan to do continuation work in this direction.

Cubic Reference Recovery

Changes to the manuscript: Submitted §2 L148–157, §3.3 L288–294 and Fig. 2 L340–343 → revised §2 L152–160, We've also added Appendix B.5 L831–865 and Table 4 L846–865.
We performed the requested sensitivity analysis on all 60 folds, preserving the common prediction support and comparing each method's inferred missing AB function against a cubic Scheffé mixture fit to the complete experimental library. In this conventional cubic decomposition, ADT achieves R2=0.351R^2=0.351 relative to zero completion at seed 0, compared with 0.1050.105 for direct cubic regression; their difference of +0.246 is positive across all five ADT seeds (+0.196 to +0.296). The result is not merely an artifact of the original quadratic reference (where ADT and cubic score 0.541 and 0.272), although the strength of recovery varies with the decomposition and only 35/60 individual folds favor ADT at seed 0.

To note, while the six-dimensional cubic function space invisible on AB-censored training inputs, {xAxBℓ(x)}\{x_Ax_B\ell(x)\}, is invariant, the assignment of a fitted response to an "AB interaction" and its complement is not basis-independent. A homogeneous cubic basis can attribute AB-containing monomials even to the purely linear function F(x)=xAF(x)=x_A, and correspondingly produces almost perfect reference scores for all models. More restrictive orthogonalized decompositions also change model rankings. We changed the wording to note the convention explicitly and interpret the robustness result as recovering a specified cubic mixture-model AB component rather than a uniquely defined interaction.

Shadow censoring and scope.

Changes to the manuscript: Submitted abstract L28–32, §3.4 L363–404 and Discussion L428–438 → revised abstract L29–34, Discussion L435–445, Appendix B.7 L975–995.

For the shadow censoring, we've re-framed it more as a diagnostic than a general model-selection method. We wanted to answer whether a metric that is constructed on the training data only could actually predict what method works in further extrapolation. In this sense, we believe it is useful, as it highlight that these feature can be learned from data prior to extrapolation. The manuscript now emphasizes that the data's observed-to-unobserved coupling determines which prior is useful, and that the evidence for ADT is limited to these four related libraries.

I have adjusted my rating from 3 to 2 based on the author’s response as well as reading the other reviewer’s comments. In particular, I share reviewer jrN9's original concern that the paper does not support the general claims being made.

The diversity of the tested datasets is minimal, and even the tested models are overly simplistic. I understand that simple models can be very helpful to perform controlled experiments, but given that this paper does not provide any theoretical contributions, I would expect the empirical analysis to include more recent, complex models.

AuthorsOpenReview ↗

Quick Response

Hi, we have managed to find some other data sets that support the split of distance and identifiability over the course of the last week. We will update the manuscript before the deadline to include these.

AuthorsOpenReview ↗

Additional data sets

We identified three other scientific data sets that support similar censoring and have added these as a new Appendix C. As a result, we've been able to considerably clarify the claims of the paper (see below)

The main revision is to distinguish the information that an OOD split removes from a model's ability to reconstruct that information. We now evaluate interaction censoring across four experimental systems: the original catalyst composition libraries, microbial communities, high-entropy-alloy composition spreads, and Suzuki–Miyaura reaction screens. These datasets encompass continuous composition spaces and categorical factorial designs, allowing us to examine the same formal missing-interaction problem under substantially different response structures.

Across these systems, we confirm that censoring particular combinations produces exactly unidentified functional directions under the specified representations. However, the resulting predictive performance and recoverability vary considerably. Different models successfully exploit different kinds of structure, and in some datasets no tested model reliably reconstructs the missing interaction. This strengthens our central conclusion: identifiability is determined by the training design and representation, while statistical recoverability depends on the response surface and the estimator's completion prior. Neither geometric distance nor conventional in-distribution performance alone captures this distinction.

We have correspondingly reframed the original ADT results. Our contribution is not a claim that ADT is a generally superior extrapolation method. Rather, the catalyst experiments provide a particularly clean example in which a model recovers a substantial fraction of a formally unidentified interaction. The mechanism experiments investigate how this can occur through transferable relationships between observed and unobserved response changes. The additional datasets demonstrate that such recovery is neither automatic nor specific to a universally successful architecture.

We have also extended the analysis of shadow censoring as a training-only diagnostic of model–data compatibility. Across several experimental systems, shadow-based dataset-level model rankings closely track actual OOD rankings, even though the preferred models differ between datasets. Nevertheless, the diagnostic is sensitive to noise and the limited number of available inner censoring tasks, performs poorly in some settings, and does not reliably identify the best model for individual missing interactions. We therefore present it as an approximate probe of structural compatibility, rather than a generally effective model-selection procedure.

In parallel, we have strengthened the discussion of prior work on ID–OOD generalization, accuracy-on-the-line, benchmark misspecification, and aggregation effects, including the references highlighted by the Area Chair. We added new paired ID–OOD experiments demonstrate that ordinary validation can substantially misrank completion priors, while aggregate performance relationships can conceal important differences within datasets.

We have retained the catalyst experiments as the principal main-text example, because their controlled composition grid, well-defined missing interaction and extensive mechanism tests provide the clearest demonstration of the phenomenon. The new experimental systems are analyzed in the appendix, with their respective noise, reference-definition and experimental-design limitations made explicit.

Overall, the revision shifts the paper away from the performance of any particular estimator and toward a more general characterization of OOD evaluation: what information is missing, whether that information can be statistically recovered, and what evidence the observed data provide about the suitability of different completion priors. We believe this substantially strengthens both the empirical scope and the relevance of the paper's conclusions to general machine learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.