- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors state that when trying to see how OOD your test set is, there are many things to take into account: how far the points are from the training data, if your training data allows for identification of the point under your model class (measured by the null-space fraction nu) and if the missing information is learnable. They mainly test this on 4 6-element catalyst libraries and look at 1 property with a composition grid of 0.1. When censoring all compositions that have both A and B you get points that have nu > 0. Capping compositions gets you similar distances with nu = 0. They then look at different ways to estimate the interaction that was censored (prior on completing the missing interaction). They find that the anchored difference transducer (ADT) performs best (better than a cubic) on the points that they all cover (RMSE of 25.5 vs 27.9 mV) and that the interaction it learns correlates 0.67-0.72 with the interaction learned from the entire library. They also show that using exponential ridge as a surface gives a positive control. At the end of the paper they show that you can use a training set diagnostic to select models that perform better on real data than if you randomly held out points, though this method does not outperform always selecting the ADT model.
Critical strengths
-
I think it's important to distinguish between how OOD your test points are and if you are actually identifying those points with your model class, and if they are still predictable. The test that the authors provide using nu is helpful to see if you are doing that, which distance measures miss. Using the composition grid allows for a nice way to see this and I think it is well done. All of the numbers that they use for creating the different designs in Table 2 reproduce exactly.
-
I think it is interesting to see that when you censor the AB interaction you can still learn some of that interaction if your model prior agrees with the structure of the data. The authors show that the ADT method learns a coefficient that is more correlated with the true coefficient (learned from the entire library) than most models (0.67-0.72 vs 0.37 for the cubic and -0.22 for the CV-selected GP). They also show with careful controls (including permuted anchors) that the model works when learning about moves that were seen and moves that were not. The authors did a good job showing the negative results as well.
Critical weaknesses
-
I think this paper does not show that this works on other properties/materials. The authors' main results only use one property, one grid and one data source. The improvement is only a few mV and mostly comes from 2 of the 4 libraries. I think the title and last sentence of the abstract should reflect this (L31-32).
-
There are a few claims that I think should be worded differently:
(a) When you are doing co-occurrence splits you are not removing points that only have one element. Therefore if you have all 6 single-element points measured, test points are guaranteed to be in the convex hull of your training set.
(b) In the abstract (L21-22) when they say they can predict 55-60% of the variance I think that should be worded as predicting 55-60% of the variance from 0, because if you were to explain the variance around the mean you would only be explaining up to r^2 (which would be around .45-.52).
(c) The authors mention that when they use a contamination residual the model can't learn the interaction (L743-744), even though the residual has a finite-rank move field. This means that having a finite separation rank is not enough to learn the interaction. I think this should be corrected at L421-423. The reason why the authors can learn the interaction in their positive control is because it uses some of the same parameters as the moves that are seen in the training set.
(d) The common test set (38,154 points) leaves out all points that are 4-5 away from the training set (where xA, xB >= .4). This should be mentioned in the main text. It is only hinted at when they mention the abstention rule (L612).
These can all be dealt with through wording and clarification, and I don't believe they detract from the ideas presented or the thoroughness of the analysis. As a result, I believe this paper should probably be accepted.
Questions for authors
-
Are the points and aggregation used in Fig 1b, table of ladders (L699-701) and Tables 1/3 the same? When looking at GP-Cubic I get +9.3 mV for the isolated/ABC points from section 3.2 and fig 1b (L246-247). But from the table of ladders I get 7.2 mV. I also see that ADT-Cubic for the pair ladder is only -0.8/-1.0 mV whereas I see about a -2.4 mV difference in Tables 1/3. Is it because you are only looking at 10 out of the 15 pairs (L573)?
-
Did you get any results for the cubic extension mentioned at L616?
-
Since always picking the ADT model has lower regret than your shadow selection method (1.21 vs 2.46 mV), when would you recommend using shadow censoring?
-
How does your work compare to the theoretical work that is done on learning about feature combinations that you haven't seen before? Dong & Ma ICLR 2023, Wiedemer et al. NeurIPS 2023.
2 comments
New datasets
Hi, we identified three other scientific data sets that support similar censoring and have added these as a new Appendix C. As a result, we've been able to considerably clarify the claims of the paper: The main goal of the revision was to clarify the broader point: OOD performance isn't just about distance from training data, but also about what information is missing and whether a model's assumptions allow it to recover that information.
Across the datasets, we find that interaction censoring produces formally unidentified directions, but whether these interactions can be recovered—and which model does best—depends strongly on the dataset. We've therefore moved away from presenting ADT as a particularly strong extrapolation method, and instead use it as one example of how a suitable completion prior can recover missing information.
We've also expanded the shadow-censoring analysis. It shows promising agreement with dataset-level model rankings using only training data, although it's much less reliable for individual interactions and has clear limitations.
Finally, we've incorporated the additional literature suggested by the Area Chair and strengthened our analysis of ID–OOD performance relationships.
We've kept the catalyst experiments at the center of the main paper because they provide the cleanest controlled demonstration, while the new datasets and their limitations are discussed in the appendix. Overall, we think the revisions make the paper's argument considerably clearer and better supported across different experimental settings.
Response to Review
Thanks for the thorough feedback! We've fixed many of the arithmetic and numerical notes that aren't so clear in the edited the manuscript (line references to specific questions below). Replies below:
Weakness 2 (a) Convex hull. Lines changed: Submitted §3.1 L194–201 and §2 L116–123 → revised §2 L123–127, §3.1 L199–205; Appendix B.1. Agreed. Every one of the six pure-component vertices is retained in all 60 pair-censored training folds. Therefore every composition on the simplex lies in their convex hull. Wording changed.
(b) Meaning of the recovery percentage. Lines changed: Submitted abstract L21–22, §2 L149–155 and §3.3 L288–294 → revised abstract L21–24, §2 L152–160, Appendix B.5 L841–848. Agreed. The submitted 55–60% refers to the reduction in squared error of the retrospectively fitted hidden quadratic coefficient relative to zero completion, 1−SSEmodel/SSEzero (sorry, can't get this to render right). We have corrected the abstract and metric description.
(c) Finite separation rank. Lines changed: Submitted Discussion L421–423, Appendix B.5 L743–744 → revised Discussion L425–429, Appendix A.3 L573–580. Agreed: finite rank alone is not sufficient. We've noted a direct counterexample of F(x)=cxAxB, whose move field
F(a+δ)−F(a)=c(aAδB+aBδA+δAδB)has separation rank at most three but every training response and every observed move is independent of c under AB pair censoring, so the coefficient cannot be recovered from training. We have revised the claim to require shared, transferable structural coupling as well as rank.
(d) Common prediction support. Lines changed: Submitted Table 1 L270–279 and Appendix B.2 L608–612 → revised §3.2 L258–260, Appendix B.3 L739–745. Agreed. The original 38,154-point common benchmark covers 38,154 of 39,413 censored fold–composition incidences. It excludes all 1,199 points at nearest-training distance four and 60 at distance five, because reach-three ADT lacks anchors there. We've fixed the note and explicitly noted that. As an additional inference-only sensitivity analysis (no retraining, Appendix B.3), extending the radius to five for excluded points gives ADT RMSE 34.92 mV on those points, versus cubic 38.86 mV and GP 51.21 mV; the mixed-radius full-region comparison is 25.81, 28.34, and 32.68 mV respectively for robustness.
Q1: Figure 1b, ladder table, and main benchmark. Lines changed: Submitted Fig. 1 caption L229–236, §3.2 L246–247, Table 1 L270–279, Appendix B.1 L573 and B.3 L699–701 → revised Fig. 1 caption L229–238, §3.2 L246–249, Appendix B.3 L689–700. These discrepancies are now exactly reproduced and explained. The submitted Figure 1b used pairwise method coverage for GP–cubic in some cells, while the appendix ladder table used all-method matched coverage. In isolation/ABC, the GP–cubic difference is +9.35 mV on 23,036 pairwise-covered point–file incidences but +7.16 mV on the 11,758 incidences covered by all methods. We've recomputed Figure 1b consistently + updated the manuscript. the qualitative result is unchanged. Added additional notes in the text.
Q2: Cubic extension of the midpoint-difference model. Lines changed: Submitted Appendix B.2 L616 → revised Appendix B.2 L654–655, Appendix B.3 L753–789, Fig. 6 L804–808. Very good suggestion with interesting results! The unregularized cubic midpoint (M5) reduces training-move RMSE from 31.15 to 27.11 mV and random within-training composition-holdout RMSE from 22.89 to 20.78 mV compared with quadratic midpoint (M4), yet worsens the censored-region RMSE from 28.39 to 56.47 mV. Great example of in-support validation favoring a poor completion prior. Ridge regularization helps (hindsight-selected raw ridge strength α=0.1 achieves 28.32 mV, approximately matching M4. But choosing its regularization using only ordinary held-out training compositions yields 41.51 mV, still much worse than M4 (28.40 mV). We report the full ridge sweep in Appendix B.
Q3: When should shadow censoring be used? Lines changed: Submitted §3.4 L363–404 and Fig. 3 L393–396; Discussion L428–438 → revised §3.4 L369–409, Discussion L435–445, Appendix B.7 L956–995. We agree that the existing benchmark does not justify using shadow censoring instead of a fixed ADT policy already known to work on these libraries. Our intended contribution is narrower. When an important combination has never been measured, its outcomes cannot be used for cross-validation (see e.g. cubic case with poor extrapolation above.
Q4: Dong and Ma (ICLR 2023), Wiedemer et al. (NeurIPS 2023). Lines changed: Submitted Introduction L60–72 and Appendix A.3 L567–571 → revised Appendix A.3 L582–587 and bibliography L486–487, L511–512. These papers are relevant and now cited.