- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- Summary
- Evaluated different strategies for selecting the right sample from a set of generated MOFs, but honestly very difficult to understand the way it is currently written
- I think the title invites misreading; the paper should state in the first paragraph that it benchmarks post-generation selection rather than generators
- Review is very jargon heavy/dense and difficult to follow, and this is coming from someone who has worked in both the MOF-space and ML space for >10 years
Critical strengths
- Strengths
- Hard to say, the question being answered is interesting (e.g., many people work on generation, but how do you prioritize/select the right sample for further testing). I did not think this was fully clear from the prose though, I had to work hard to figure out what the authors were doing here.
Critical weaknesses
- Weaknesses
- I found the paper very convoluted and difficult to read, what terms were was not defined and it felt like a lot of nonstandard terms were used and never defined. Inventing lots of new terminology where good terminology already exists (e.g., "success rate" instead of "oracle@K")
- For instance, what does this sentence mean? "Their value for materials design depends on converting candidate pools into reliable recommendations, a transition that candidate coverage alone cannot assess." This is the second sentence in the abstract and it only gets more confusing from there.
- There is another sentence in the abstract "MOFSelect aligns outputs from different generators under shared chemical conditions and provides candidate-level reference-recovery supervision", what even is "candidate-level reference-recovery supervision"? I have worked in both MOFs and ML for over 10 years and I don't know what this is talking about.
- Figures do not help to understand what was done at all
- Advice: terms like pool, candidate, condition, recovery, coverage, and selection should be defined in the intro with a concrete example, and a figure showing an actual pool (e.g. 10 structures with one marked as matching) would help far more than Figure 1's current schematic
- The paper is on benchmarking MOF generation/selection and not a single MOF structure was shown....
- The related work cites some irrelevant things. For instance, DiffDock is cited but the connection to MOFs or this work is not clearly made
Questions for authors
- Questions
- What are candidate "pools"? Neither what a "candidate" is (is it a building block? a topology? a full MOF structure? something else) nor a "pool" is ever defined. Does "pool" refer to a set of some kind? Why not just say "set" if so (or if it is not a set, can you explain it please)? Is it different? Just define the terms used throughout. The term pool appears 105 times in the paper yet it is never defined that I could tell.
- Hard for me to evaluate the technical soundness of this work because it took me over an hour even to understand what the authors were doing. For instance, how was data leakage controlled for across the 3 generators? were they trained the same? Then, what was the variance in the predictions? Were different seeds tried? Are the conclusions statistically significant?
5 comments
Response to Reviewer tUoF (2/2): Data leakage, training, and statistical evidence
Data leakage controls. Selectors are compared on identical frozen candidates from each generator. Chemical-condition groups are disjoint across selector training, development, calibration, and testing; the family test holds out 23 linker skeleton groups. Scorer inputs exclude reference coordinates, target cells/topologies, labels, matching runtime, and generator identity. Leave-one-generator-out runs exclude that source from fitting, development selection, and calibration. Appendix F.2 also checks cross-role reference neighbors.
Generator training and exposure. We use released checkpoints with their original training recipes, without unified retraining, and align chemical requests and sampling budgets. Cross-generator results characterize these released systems. We audited the public AtomMOF training/validation data, MOF-LLM task-pretraining, supervised fine-tuning and reinforcement-learning data, and the MOFFlow-2 inventory. A common union-exposure mask removes 73 core-test and 25 family-test conditions from every source. On the remaining 951 core and 487 family conditions, screened-pool selection gaps remain 6.38–13.35 pp under primary matching and 3.29–11.64 pp under strict matching, with all paired 95% intervals above zero. On unfiltered pools, MOF-LLM Full's family gain over Consensus remains 4.24 pp [1.63, 6.75]. Thus the selection findings persist after excluding identifiable training exposure. The audit covers released task datasets; undocumented checkpoint histories are outside its scope (§3.3, Appendix F.2).
Learned selectors use 2,048 training conditions and three seeds; neural models receive 5,120 updates and tree families search six configurations (§4.3).
Prediction variability across seeds. Table 20 reports the original three Full seeds. MOF-LLM recovery is 44.53 ± 0.78% on core and 40.17 ± 0.63% on families (sample standard deviation). Existing predictions show about 55% pairwise disagreement in selected attempt IDs, but 4.36%/4.95% in core/family recovery outcomes. Different IDs need not denote different geometries. Across generators, outcome disagreement is 11.20–22.11% on pools containing both matches and nonmatches (Appendix H.3). Thus stable aggregate performance does not imply identical individual predictions.
Statistical support for the conclusions. Section 4.4 uses 10,000 paired bootstrap replicates, averaging seeds within condition; family sensitivity resamples skeleton groups. These intervals describe condition or family variation given the three fixed seeds; Table 20 separately reports seed variability.
- Selection opportunity: after the requested structural filtering, surviving-pool coverage exceeds the best of three evaluated rankers by 6.15–13.48 pp under primary matching and 3.13–12.11 pp under strict matching. All six generator/test settings have paired 95% intervals above zero; family intervals resample skeleton groups (Table 19).
- Budget expansion: expanding Full's input from five to ten candidates loses previously correct recommendations on 1.46–5.86% of conditions across the six settings, averaging three training seeds (Table 4). Added family-group resampling gives positive 95% intervals for all three generators' losses (Appendix C.2). This is displacement, not evidence that larger pools always reduce net recovery: new successes can offset losses.
- Confidence transfer: a source-certified rule targeting 10% risk makes 174 errors among 533 accepted held-out MOF-LLM core predictions: 32.65% error, with a descriptive 95% binomial interval of [28.68%, 36.81%]. The other two seeds cannot certify the source target and accept nothing; their risk is undefined, not zero (§5.4).
Learning gains are evaluated separately. MOF-LLM Full-minus-Consensus gains are:
| Evaluation | Gain (pp) | Interval |
|---|---|---|
| Original family, primary match | 3.84 | [1.43, 5.98], 95% |
| Original family, strict match | 1.30 | [−0.88, 3.29], 95% |
| Original frozen confirmation, 2,048 conditions | 3.34 | [1.95, 4.72], nominal 97.5% |
| Post-hoc corrected-input sensitivity, same conditions | 3.11 | [1.76, 4.48], nominal 97.5% |
The strict-family advantage remains inconclusive. The corrected-input result is a post-hoc implementation sensitivity on the same confirmation conditions (Table 3). Two additional generation streams, distinct from training seeds, reuse 256 core and 256 family conditions. Averaging streams within condition gives gains of 4.30 pp [1.43, 7.23] and 5.01 pp [1.82, 8.27], using condition-level bootstrap (§5.5; Appendix D.4, Table 12). Together, these controls establish selection quality, budget use, and acceptance reliability as measurable targets for decision models.
We would appreciate your reassessment of soundness and the overall recommendation in light of these clarifications and revisions.
Thank you for the detailed response. However, I still feel substantial revisions are needed to the original submission in order to make it understandable and to make it clear why this work is significant. I maintain my score at 1 and encourage the authors to resubmit to a venue that can better assess the significance in the domain of MOF optimization, as well as to better place the work in context. Major revisions needed.
Response to Reviewer tUoF (1/2): Post-generation MOF selection: definitions and a concrete example
Thank you for reading the paper so closely and for pointing to the specific passages that were difficult to understand. Your detailed comments were particularly helpful in revising how we introduce the task and explain the terminology.
MOF-Select benchmarks post-generation selection and confidence: which generated structure should be recommended for a chemical request, and when should the method abstain? We address the possible ambiguity in the title by stating this scope directly in the revised abstract and first introduction paragraph. The evaluated methods choose among frozen generator outputs; the contribution is a dataset and protocol for those decisions.
Candidates and pools. A condition specifies building-block identities and counts. A candidate is a complete periodic MOF, with atom identities, coordinates, and a unit cell. We use “pool” to mean the ordered record of generation attempts for one condition and generator, rather than a set of unique structures. Each attempt remains a separate entry, including repeats and failures. At each budget K (1, 5, or 10), all selectors are evaluated on the first K attempts from that same record.
Recovery means matching a known structure for that condition; candidate coverage asks whether any attempt matches, whereas selection chooses one candidate without the test reference. The ordered-pool formulation appears in the original §4.1 (p. 5, Eq. 1); we now place its plain-language definition and the other five terms together in the introduction. Section 4 introduces Oracle@K as pool success rate and Selected@1 as selected success rate. Accepted coverage separately measures the fraction of conditions with an accepted recommendation. These qualifiers distinguish finding a match, choosing it, and accepting the choice.
A concrete MOF example. Figure 3(a) shows the BW hypothetical reference for condition bw_2151 alongside two MOF-LLM candidates. Candidate 1 does not match the reference, while candidate 6 meets both the primary and stricter matching criteria. Figure 3(b) gives the matching results for all ten candidates and the actual choices: Full selects 6, Consensus 7, and Energy 10. Four candidates meet the primary criterion; two also meet the stricter criterion. All three methods select a primary match here. The introduction explains how this pool distinguishes coverage from selection: a match is available, but choosing one of the six nonmatching candidates would miss it. Appendix H.3 (Figure 8) shows the reference and all ten candidate structures.
The case was chosen by a fixed, reproducible rule without considering which structures the selectors chose. Candidates 1 and 6 are the first nonmatching and first strictly matching candidates in generation order. This example illustrates the task; Section 5 compares method performance across test conditions. Figures 1 and 2 retain the decision overview and benchmark protocol.
The two abstract sentences you highlighted are now expressed directly: generating a reference match does not ensure a method can identify it or assess confidence in its choice; the dataset groups complete structures generated from the same building-block identities and counts, then labels whether each matches a known reference. These labels support training and evaluation, while test references remain hidden from scoring.
Related work and contribution. DiffDock is relevant because it trains reference-based confidence to rank generated poses and evaluates selective accuracy. Revised §2 makes this connection explicit. We cite it as a precedent for the decision problem, rather than a MOF generator or direct baseline. MOF-Select supplies reusable supervision and fixed candidate pools for developing decision models without rerunning generation. Its paired protocols separate candidate availability, selection, and acceptance reliability across MOF sources. As detailed below, decision errors persist after structural screening and exclusion of identifiable upstream training exposure. Reference recovery measures structural agreement, not physical stability or synthesizability.
We appreciate the time and care you put into this review. Your suggestions helped us explain the task more clearly and make the paper easier to follow. We would be grateful to hear whether the revised explanations and concrete MOF example address your concerns about clarity.
Our second comment answers the questions about leakage, training, variability, and significance.