acceptodds
Under review as a conference paper at ICLR 2027

CataPath: Expert-Augmented Path Search for Constraint-Aware Scientific Reasoning in Catalyst Design

Abstract

Large Language Models (LLMs) have shown promise as scientific assistants, yet their use in open-ended material design remains limited by violations of domain-specific constraints. We study rational catalyst design as a challenging testbed for scientific reasoning, where an agent must generate chemically coherent intermediate decisions. We propose CataPath, an expert-augmented LLM agent that formulates catalyst design as search over a chemically informed action space. CataPath uses Monte Carlo Tree Search to explore design trajectories and a hybrid reward mechanism that combines property-prediction signals with an Expert Preference Reward Model trained from domain feedback. To support this framework, we curate CataDesignQR, a dataset of literature-grounded design goals, verified solutions, and expert-scored reasoning trajectories. We evaluate CataPath on two catalyst-related material-design settings, CO reduction electrocatalyst generation and organic structure-directing agent optimization. Experiments show that CataPath improves generation validity, proxy effectiveness, and expert alignment over other baselines. These results suggest that process-level search with expert-informed rewards can improve the reliability of LLM agents for scientific design tasks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Models, Scientific Reasoning, LLM Agents, Material Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x4zJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CataPath, an expert-augmented LLM framework for constraint-aware scientific reasoning in catalyst design. The method formulates catalyst design as a sequential decision-making problem and employs Monte Carlo Tree Search (MCTS) over a chemically informed action space. During search, an LLM generates material candidates conditioned on predefined design actions and reasoning history, while a hybrid reward combines property predictions with an Expert Preference Reward Model (EPRM) trained on expert-annotated reasoning trajectories.

The authors also introduce CataDesignQR, a dataset containing literature-grounded design goals and expert-scored reasoning trajectories for two tasks: CO2 reduction electrocatalyst generation and organic structure-directing agent (OSDA) optimization.

Experiments show improvements over general LLM generation and existing agentic reasoning baselines in terms of reasonableness, proxy effectiveness, expert-assessed feasibility, and several molecular similarity metrics. The paper further provides component ablations and case-level external validation using DFT calculations and experimental characterization.

Critical strengths

S1. Integration of domain-specific knowledge and expert feedback into structured search.

The paper presents a practical framework that incorporates chemically informed design strategies and expert-preference feedback into LLM-guided search. The combination of property-based and expert-based rewards is particularly relevant to scientific design problems where numerical property predictors alone may not capture practical feasibility constraints. The component ablations provide empirical evidence that the structured action space and hybrid reward contribute to the observed improvements.

S2. Expert-annotated dataset and multi-level empirical validation.

The construction of CataDesignQR provides a useful resource for studying constraint-aware scientific design with intermediate reasoning trajectories. The paper evaluates the framework across two distinct material-design settings and includes expert assessments, comparisons with existing agentic methods, and case-level external validation. These efforts strengthen the practical relevance of the proposed framework beyond conventional language-generation benchmarks.

Critical weaknesses

W1. Insufficient evidence for the necessity of MCTS and its contribution to the reported improvements.

Although MCTS is a central component of CataPath, the experiments do not sufficiently isolate its contribution from constrained candidate generation and reward-guided selection. The reported improvements may primarily result from the predefined action space and hybrid reward mechanism rather than the multi-step planning capability of MCTS. While the paper includes analyses of search depth and individual components, comparisons against simpler search strategies under matched computational budgets are needed to establish the necessity of MCTS.

W2. Limited evidence that improvements in evaluation metrics translate into meaningful scientific outcomes.

The main results rely heavily on proxy-based evaluation metrics whose relationship to the underlying scientific objectives is not fully established. For CO2 reduction, Effectiveness measures embedding similarity to literature-verified designs rather than actual catalytic performance. For OSDA optimization, improvements are primarily reported in molecular similarity metrics, while direct evidence of improved binding energy is limited in the main comparison. Although the paper includes DFT calculations and experimental characterization, these provide case-level evidence rather than systematic validation against competing methods. Consequently, it remains unclear whether the reported improvements reflect better scientific designs or stronger alignment with the chosen evaluation metrics.

W3. Unclear contribution of chemical reasoning beyond search-space restriction.

The paper motivates CataPath by emphasizing the propagation of errors across intermediate reasoning steps and the difficulty of maintaining domain-specific constraints during long-horizon scientific reasoning. However, the proposed method primarily constrains candidate generation through predefined chemical actions and uses reward-guided search to select promising candidates.

The current experiments do not clearly distinguish whether the improvements arise from more accurate chemical reasoning by the LLM, the incorporation of domain knowledge through predefined actions, or simply the reduction of the effective search space. In particular, the action-space ablation compares structured chemical actions with free-form generation, which does not isolate the contribution of chemical knowledge from that of search-space restriction. This makes it difficult to determine how the proposed framework specifically addresses the reasoning failures highlighted in the motivation.

W4. Insufficiently controlled comparisons with existing LLM-based search methods.

CataPath is compared against several prompting and agentic reasoning approaches, including ToT, ReAct, GoT, and LATS. However, the reported comparisons do not clearly establish whether the methods are evaluated under equivalent search budgets and feedback conditions. Differences in the number of generated candidates, LLM calls, external reward evaluations, or computational cost could influence the observed performance gains. Without sufficiently controlled comparisons, it is difficult to attribute the improvements to the proposed search methodology rather than additional computational resources or access to domain-specific feedback.

Questions for authors

Q1. Is MCTS necessary?

The performance improvements may arise from constrained candidate generation and reward-guided selection rather than multi-step tree search. Could the authors compare CataPath against simpler search strategies, such as Best-of-N, greedy search, or beam search, under matched generation and evaluation budgets while keeping the action space and hybrid reward fixed?

Q2. Does the method improve scientific reasoning or candidate selection?

The paper attributes its gains to constraint-aware scientific reasoning, but the evaluations mainly concern final candidate quality. What evidence demonstrates that CataPath improves the correctness of intermediate reasoning steps, rather than simply selecting higher-reward candidates? Could the authors report step-level constraint violations, error correction, or trajectory consistency?

Q3. What is the specific contribution of the chemically informed action space?

The introduction highlights error propagation in long-horizon scientific reasoning as a key motivation. However, the proposed action space appears to primarily constrain the LLM's available design decisions and reduce the effective search space, rather than directly improving its chemical reasoning capability.

Could the authors clarify how the incorporated chemical knowledge enables the LLM to make more scientifically meaningful intermediate decisions beyond restricting the search space? In particular, how does this mechanism prevent or correct errors propagated across reasoning steps? Could the authors disentangle the effects of domain-specific chemical knowledge from those of search-space restriction, for example by comparing against alternative structured action spaces of similar size?

Q4. Are the search comparisons computationally matched?

The paper compares CataPath with several existing LLM-based search and reasoning methods, including ToT, GoT, and LATS. Could the authors clarify whether these methods are compared under equivalent computational budgets, including the number of LLM calls, generated candidates, search expansions, and external reward evaluations? Furthermore, how are differences in available domain-specific feedback controlled when comparing CataPath with these baselines?

Minor Concern — Incorrect reference for CatBERT Lee et al. (2020), which concerns social engineering email detection rather than catalyst property prediction.

5 comments
AuthorsOpenReview ↗

Response for W1 & Q1 & Minor Concern

We thank the reviewer for recognizing the relevance of chemical constraints, expert feedback, and the dataset, and for identifying the need for stronger attribution and scientific validation. We address the overlapping weaknesses and questions together below.

W1 & Q1: Contribution of MCTS under matched generation and evaluation budgets

We agree that the original component ablations and depth analysis do not sufficiently isolate MCTS from constrained generation, reward-based selection, and additional computation. The controlled comparison therefore keeps the LLM, chemical action space, expert-derived action priors, and frozen reward models fixed. All methods use the same node-level reward, R=0.4R^prop+0.6R^EPRMR=0.4\widehat R_{\mathrm{prop}}+0.6\widehat R_{\mathrm{EPRM}}.

We cap each method at 50 generation calls per goal, including retries, and at 50 node-level evaluations through the same scoring pipeline. The final output is the complete eligible list with the highest hybrid reward among the generated nodes. The evaluation uses the same 100 held-out goals over three runs, with maximum trajectory depth 10. Independent trajectories use history-conditioned refinement. The greedy search expands five alternatives per step and beam search uses width three and three alternatives per retained path.

Table R1. Search comparison on 100 held-out goals over three runs.

MethodGeneration-call capReasonableness ↑Proxy Effectiveness ↑Expert Feasibility ↑
Best-of-N500.850.820.65
Independent trajectories500.860.840.69
Greedy search500.900.880.73
Beam search500.910.900.77
CataPath500.910.910.80

CataPath achieves the highest reported expert feasibility (0.80) and proxy effectiveness (0.91), while matching beam search on reasonableness (0.91). Its absolute feasibility gains are 0.03 over beam search, 0.07 over greedy search, 0.11 over independent trajectories, and 0.15 over Best-of-N. The comparison with independent trajectories shows an improvement beyond history-conditioned refinement alone, while the comparison with beam search supports an additional benefit beyond maintaining multiple candidate paths. With the chemical action space, expert priors, reward models, and generation/evaluation caps held fixed, these results support the incremental value of MCTS-guided search at the evaluated budget. The equal token cost is addressed separately under Response for W4 & Q4.

Minor concern: Incorrect CatBERT citation

Thank you for identifying this error. The model used in our experiments is CatBERTa, and the correct reference is Ock, Guntuboina, and Barati Farimani (2023), “Catalyst Energy Prediction with CatBERTa: Unveiling Feature Exploration Strategies through Large Language Models,” ACS Catalysis, 13(24), 16032–16044. We will correct the model name and replace the erroneous Lee et al. (2020) citation throughout the manuscript.

AuthorsOpenReview ↗

Response for W2

W2: Scientific meaning of the evaluation metrics

CO₂RR: Calibration against measured FE, complemented by targeted experimental validation. Appendix D.2.1 defines Effectiveness as an embedding-similarity proxy relative to literature-verified designs. We agree that its relationship to measured catalytic performance requires validation. Synthesizing and testing outputs from all compared methods under a common experimental protocol would require substantial resources. We therefore added a literature-based calibration against experimentally measured Faradaic efficiency (FE), evaluated separately for CO, formate, and ethylene, with 40 records per product under the cell and electrolyte conditions listed below. This provides a tractable assessment of the metric's empirical relevance, complemented by the targeted physical validation already reported in the paper.

Table R2. Correlation between proxy Effectiveness and experimentally measured FE, reported separately for each product and condition group.

Product-specific comparable-condition cohortMeasured records, nSpearman ρ [95% CI]Pearson r [95% CI]
CO — H-cell; 0.1 M KHCO₃400.79 [0.61, 0.89]0.73 [0.54, 0.85]
Formate — H-cell; 0.5 M KHCO₃400.76 [0.56, 0.88]0.70 [0.50, 0.83]
Ethylene — H-cell; 0.1 M KHCO₃400.74 [0.53, 0.87]0.68 [0.47, 0.80]

Across the three product-specific cohorts, Spearman correlations range from 0.74 to 0.79 and Pearson correlations from 0.68 to 0.73. These positive associations provide empirical support for the proxy's relationship to measured FE on the evaluated literature samples. They support its use as a performance-associated evaluation signal within this scope.

In addition, complementary physical evidence is already provided in Appendix D.3: Table 8 reports DFT-calculated adsorption energies for selected CataPath-proposed surfaces; Figs. 9–11 characterize a synthesized Cu–Sn bimetallic nanostructure; and Fig. 12 reports its FE–potential curves alongside a Cu–Sn mixture control, polarization curves, and stability measurements. These results support the synthetic accessibility and electrochemical relevance of a selected generated design. The literature calibration and targeted experiments thus address metric relevance and case-level feasibility, respectively. However, they do not establish systematic experimental superiority over other generation methods. A common-protocol comparison of newly generated candidates remains future work.

OSDA: comparing predicted binding-energy improvement using the existing frozen predictor.

We agree that Table 3 does not establish superior binding energy. Its ED is Energy Distance, a distributional metric. Relative to OSDA-Agent, CataPath improves BLEU, Morgan, and RDK similarity, matches validity and MACCS, and does not improve ED or KL divergence. These structural metrics alone cannot demonstrate improved binding performance.

To supplement this comparison, we added predicted binding-energy results for OSDA-Agent and CataPath using the existing frozen predictor described in Sec. 4.3 and Appendix B.1. Table R3 reports 90 attempts per method for the LTA framework. We compare the raw predicted binding energy of each final molecule with that of its starting molecule:

ΔE^bind=E^bind(final molecule,LTA)−E^bind(starting molecule,LTA).\Delta\widehat E_{\mathrm{bind}}=\widehat E_{\mathrm{bind}}(\text{final molecule},\mathrm{LTA})-\widehat E_{\mathrm{bind}}(\text{starting molecule},\mathrm{LTA}).

A negative value indicates a predicted improvement. This quantity is computed before reward rescaling and does not include the EPRM score.

Table R3. Predicted binding-energy comparison using the existing frozen predictor, with 90 attempts per method.

MethodMean ΔÊ_bind, eV per OSDA ↓Predicted improvements / all attempts ↑Invalid outputs / all attempts ↓
OSDA-Agent−0.1959/90 (65.6%)0/90 (0.0%)
CataPath−0.2365/90 (72.2%)0/90 (0.0%)

Both methods report 90 scored outputs, with no invalid outputs. Improvement rates use all 90 attempts as the denominator.

CataPath achieves a mean predicted binding-energy change of −0.23 eV per OSDA, compared with −0.19 for OSDA-Agent, corresponding to an additional predicted reduction of 0.04 eV per OSDA. It improves on the starting molecule in 65 of 90 attempts (72.2%), compared with 59 of 90 (65.6%) for OSDA-Agent, yielding six additional improvements with the same number of attempts. These descriptive results support better optimization of the predictor-defined binding-energy objective in the evaluated sample, supplementing the structural metrics in Table 3. However, we admit that this is a surrogate-based comparison, not independent physical validation. The physical validation in the present study is concentrated on CO₂RR. Those case studies do not validate OSDA binding performance.

AuthorsOpenReview ↗

Response for W3 & Q2 & Q3

W3 & Q2: Reliability of intermediate decisions

We agree that final candidate quality alone does not establish the reliability of intermediate decisions. We therefore added a process-level evaluation of randomly sampled trajectories produced by independent refinement, greedy search, beam search, and CataPath. We distinguish three aspects:

  • Step-level constraint violation: whether a generated candidate introduces or retains a violation of the task chemical constraint applicable at that step.
  • Action execution consistency: whether the generated modification implements the selected operation and its explicit preservation requirements. Executing an action does not necessarily imply satisfying all other task constraints.
  • Trajectory consistency: whether the sampled trajectory avoids contradictory claims or unjustified loss of information required to be retained across steps. Explicitly permitted design revisions are not treated as contradictions.

Best-of-N is omitted from this multi-step comparison because its one-step outputs do not provide comparable trajectories. We randomly sample 500 trajectories from the same 100 held-out goals used in W1/Q1, all drawn from the paper's fixed test split. W3/Q3 uses the same trajectory-sampling and evaluation setup.

Table R4. Process-level evaluation of randomly sampled trajectories from the fixed test goals.

MethodStep violations ↓Action execution consistency ↑Trajectory consistency ↑
Independent trajectories18.6%78.8%69.4%
Greedy search14.2%86.2%78.2%
Beam search13.2%89.4%80.8%
CataPath8.0%94.2%85.2%

CataPath has the lowest step-level violation rate (8.0%) and the highest action execution consistency (94.2%) and trajectory consistency (85.2%) among the evaluated methods. These results extend the evaluation beyond final candidate quality: the sampled CataPath trajectories contain fewer constraint violations, implement the prescribed actions more consistently, and exhibit greater cross-step consistency. We interpret these findings as improved process reliability in the sampled trajectories. This does not demonstrate an improvement in the underlying LLM's intrinsic reasoning ability. Rejecting an erroneous branch is distinct from repairing an error within that branch. We will align the paper's claims with this scope.

W3 & Q3: Chemical knowledge beyond restricting the search space

We agree that Fig. 3 compares two factors simultaneously: structure and chemical content. We therefore added an action-space ablation comparing free-form actions, generic structured actions, and chemically informed actions within CataPath. The generic structured condition provides a comparison for organizing decisions without task-specific chemical instructions. The process evaluation follows the same random-trajectory sampling setup as W3/Q2: 500 trajectories from the same 100 goals in the paper's fixed test split.

Across variants, we keep the other experiment setups fixed and compare three action-proposal schemes: free-form generation, a generic structured inventory, and the chemically informed inventory. In the free-form condition, the LLM first proposes an action freely and then executes the modification. The action execution consistency is assessed against that proposed action.

The generic structured inventory contains 26 selectable actions, expressed as domain-agnostic edit templates. Examples include: (1) replace one component while keeping the remaining design unchanged; (2) add/remove one component while preserving unrelated design attributes. These templates organize how to edit a design without prescribing a chemically motivated choice.

Table R5. Action-space ablation with the same trajectory-evaluation setup as Table R4.

Action spaceStep violations ↓Action execution consistency ↑Trajectory consistency ↑Expert Feasibility ↑
Free-form22.5%76.4%58.8%0.58
Generic structured (26 actions)13.8%85.6%70.0%0.74
Chemically informed8.0%94.2%85.2%0.82

Generic structured actions improve over free-form actions. Chemically informed actions provide further gains over the generic structured condition. Thus, the chemical action specification improves all four reported measures relative to the evaluated generic structured control. The chemical actions supply concrete instructions about what to modify and preserve. For example, Appendix A.3 includes OSDA operations for ring-size correction and charge balance. The observed improvements are consistent with such domain guidance helping the system produce more constraint-preserving edits and coherent trajectories in the random sample. We therefore claim an empirical benefit over the evaluated generic structured alternative, while acknowledging that this comparison does not fully isolate chemical knowledge from action-space size.

Reviewer_q1orOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work targets the hallucination and domain‑constraint violation problem when large language models perform catalyst‑oriented material design. The authors propose CataPath, an MCTS‑based LLM agent that conducts trajectory search over a chemically predefined discrete action space. A hybrid reward combines property‑prediction signals and an EPRM trained via pairwise preference learning from human expert annotations. The authors curate CataDesignQR, a new benchmark dataset containing literature‑grounded design goals, solutions and expert‑scored reasoning trajectories for two representative tasks: CO2CO_{2}RR electrocatalyst de‑novo design and OSDA molecule optimization for zeolite synthesis. Comprehensive empirical comparisons against CoT, ToT, LATS, OSDA‑Agent and other strong baselines are provided, alongside ablation studies, backbone robustness tests, DFT calculation and limited wet‑lab experimental validation. The paper concludes that process‑level search augmented by expert‑informed rewards improves validity and expert‑alignment for scientific design agents。

Critical strengths
  1. The motivation is well-founded. When LLMs are applied to open-ended materials design, they frequently exhibit "hallucinations", generating text that is fluent yet chemically infeasible or synthetically impractical. Catalyst design is a quintessential scientific reasoning task characterized by the absence of a single, definitive "correct" answer; it demands not only a valid final outcome but also accurate, logically sound intermediate reasoning steps. This paper identifies two core bottlenecks: the lack of an "action space" aligned with domain-specific chemical knowledge, and the insufficiency of feedback mechanisms to capture the tacit knowledge possessed by human experts.

  2. The contributions of this work are multifaceted:

  • It models catalyst design as a MDP defined over an action space rooted in chemical domain knowledge, and introduces "CataPath", to facilitate scientific exploration at the trajectory level.
  • It proposes a hybrid reward mechanism that integrates numerical property predictions with a reward model derived from expert preference learning.
  • It constructs the "CataDesignQR" dataset, which contains expert-annotated catalyst design trajectories; the dataset is rigorously partitioned into training and test sets based on specific design goals, effectively preventing data leakage.
  • It provides a systematic empirical evaluation, encompassing prompting-based baselines, agent-based baselines, ablation studies, and validation through external computational simulations.
  1. The submission is clearly written and rigorously structured, with well-organized sections covering background, methodology, datasets, experiments, and an analysis of limitations. The figures and tables effectively illustrate the overall workflow, the dataset construction process, and the experimental results. Beyond presenting positive outcomes, the authors also explicitly examine potential failure modes that the agent may encounter during operation.
Critical weaknesses
  1. To what extent is there an overlap between the literature used to train the "property predictor" and the original literature used to construct CataDesignQR? Were any filtering measures implemented to remove these overlapping entries?
  2. Regarding the proxy metric for the "effectiveness" of CO2CO_{2}RR, what is the empirical correlation coefficient between the embedding similarity score and the actual catalytic performance measured on held-out candidate samples?
  3. What is the distribution of "expert feasibility scores" provided by different evaluators? What is the standard deviation of the expert scores for an individual sample?
  4. For the 12.8% of design strategies that fall outside the predefined action space, does CataPath discard them, or are they mapped to the closest existing action? How does performance degrade for these "out-of-distribution" design objectives?
  5. Were values ​​for λ\lambda other than 0.4 tested? If so, what does the performance curve look like as a function of λ\lambda? Is 0.4 the empirically optimal value?
Questions for authors
  1. To what extent is there an overlap between the literature used to train the "property predictor" and the original literature used to construct CataDesignQR? Were any filtering measures implemented to remove these overlapping entries?
  2. Regarding the proxy metric for the "effectiveness" of CO2CO_{2}RR, what is the empirical correlation coefficient between the embedding similarity score and the actual catalytic performance measured on held-out candidate samples?
  3. In the proposed "LATS + EPRM" ablation baseline, is the chemical action space identical to that of CataPath, or does it utilize the open-ended action space default to LATS?
  4. What is the distribution of "expert feasibility scores" provided by different evaluators? What is the standard deviation of the expert scores for an individual sample?
  5. For the 12.8% of design strategies that fall outside the predefined action space, does CataPath discard them, or are they mapped to the closest existing action? How does performance degrade for these "out-of-distribution" design objectives?
  6. Were values ​​for λ\lambda other than 0.4 tested? If so, what does the performance curve look like as a function of λ\lambda? Is 0.4 the empirically optimal value?
  7. When comparing against OSDA-Agent, are the input prompts, molecular validity filters, and fingerprint settings used for evaluation identical for both CataPath and OSDA-Agent?
  8. Is the EPRM "frozen" during the MCTS process? Furthermore, are the preference pairs used exclusively for training the EPRM (i.e., is this information inaccessible to the underlying LLM during the inference phase)?
5 comments
AuthorsOpenReview ↗

Response for W1/Q1 & W2/Q2 & Q3

We thank the reviewer for the detailed assessment and for recognizing the value of the domain action space, expert feedback, and dataset. We address the overlapping weaknesses and questions together and distinguish existing safeguards from additional analyses.

W1 & Q1: Overlap between predictor training data and CataDesignQR

Existing safeguards. Sec. 3.3 specifies an 80/20 design-goal split, with all trajectories of each goal assigned to the same partition and test references and expert scores withheld during generation. Appendix A.4 further describes removing duplicate query–material–target tuples across CO₂RR splits and checking catalyst–product–regulation overlaps. For OSDA, it canonicalizes SMILES and removes duplicate molecules and framework–molecule pairs across splits. The same appendix states that property predictors are fixed and are not updated with held-out trajectories, expert scores, or reference solutions.

Additional predictor-data audit. Beyond these safeguards, we cross-checked each property predictor's training data against the full CataDesignQR dataset, including both training and test partitions. We confirmed zero overlapping source publications for CO₂RR and zero identical SMILES entries for OSDA, as summarized below. Thus, the additional audit directly addresses overlap with predictor training data, complementing the dataset split and deduplication measures already documented in the paper.

Table R1. Additional overlap audit between predictor training data and the full CataDesignQR dataset.

Audit scopeOverlapping entries
CO₂RR predictor training vs. CataDesignQR source publications0
OSDA predictor training vs. CataDesignQR SMILES entries0

W2 & Q2: Correlation between Effectiveness and measured catalytic performance

As defined in Appendix D.2.1, Effectiveness measures embedding similarity to literature-verified designs. We added a literature-based calibration against experimentally measured FE using copper-based materials held out from the task-specific training data. We evaluate CO, formate, and ethylene separately, with 40 records per product under the cell and electrolyte conditions reported below.

Table R2. Correlation between proxy Effectiveness and measured FE, reported separately by product and condition group.

Product-specific copper-based cohortPotential window (V vs. RHE)Measured records, nSpearman ρ [95% CI]Pearson r [95% CI]
CO — H-cell; 0.1 M KHCO₃−0.80 ≤ E ≤ −0.60400.79 [0.61, 0.89]0.73 [0.54, 0.85]
Formate — H-cell; 0.5 M KHCO₃−1.00 ≤ E ≤ −0.80400.76 [0.56, 0.88]0.70 [0.50, 0.83]
Ethylene — H-cell; 0.1 M KHCO₃−1.10 ≤ E ≤ −0.90400.74 [0.53, 0.87]0.68 [0.47, 0.80]

Spearman correlations range from 0.74 to 0.79 and Pearson correlations from 0.68 to 0.73. These positive associations support the proxy's relationship to measured FE on the evaluated literature samples. The 40 records per product are calibration observations.

Testing all methods' generated candidates under a common physical protocol would require substantial resources. The calibration therefore assesses metric relevance, complemented by the case-level evidence already reported in Appendix D.3: Table 8 gives DFT-calculated adsorption energies, and Figs. 9–12 provide characterization and electrochemical evaluation of a CataPath-proposed Cu–Sn design. These studies support the feasibility of a selected generated design. A systematic physical comparison remains future work.

Q3: Action space and feedback in the LATS baseline

The LATS baseline reported in Table 2 uses open-ended and LLM-generated actions, rather than CataPath's predefined chemical action space. As specified in Appendix D.1, it combines property-prediction feedback with LLM-based value estimation without EPRM. This is a full-system comparison with differences in both action specification and reward feedback. We will clarify the baseline naming and configuration to avoid ambiguity.

AuthorsOpenReview ↗

Response for W3/Q4 & W4/Q5

W3 / Q4: Expert score distributions and per-sample disagreement

Appendix D.2.1 specifies five final-evaluation experts who are independent of the EPRM data-collection panel, with method identities and other annotators' scores hidden. The Fleiss' κ=0.82 in Appendix A.5 concerns training-data annotation. We now report their individual score distributions and within-output disagreement as the statistics below.

All five evaluators rated the same 120 outputs, comprising 60 matched CataPath–LATS pairs, on a raw 1–5 scale. Table R3 pools both methods: each percentage is the fraction of an evaluator's 120 ratings at that score, and the mean is that evaluator's average across the same outputs.

Table R3. Final-evaluator score distributions over 120 outputs per evaluator, pooled across CataPath and LATS. Means use the raw 1–5 scale; percentages may not sum to exactly 100% because of rounding.

EvaluatorScore 1Score 2Score 3Score 4Score 5Mean
E10.0%5.0%25.0%35.8%34.2%3.992
E20.0%5.0%35.0%33.3%26.7%3.817
E30.8%3.3%17.5%45.0%33.3%4.067
E40.0%5.0%25.8%42.5%26.7%3.908
E50.0%5.0%15.0%38.3%41.7%4.167

For each output ii, we compute the sample SD across its five expert ratings, si=∑j=15(xij−xˉi)2/4s_i=\sqrt{\sum_{j=1}^{5}(x_{ij}-\bar{x}_i)^2/4}. Across the 120 outputs, the mean SD is 0.472, the median is 0.548, and the interquartile interval [Q1, Q3] is [0.447, 0.548] on the raw 1–5 scale. These statistics quantify disagreement on the same output, separately from variation across search runs.

The evaluators' pooled mean scores range from 3.817 to 4.167, indicating differences in their overall scoring tendencies. Nevertheless, all five evaluators assign CataPath a higher mean score than LATS on the matched outputs: the per-evaluator mean gains range from 0.233 to 0.483 points, with a panel-average gain of 0.333 points. Thus, the direction of the mean advantage is consistent across evaluators in this paired sample.

W4 / Q5: Clarifying the 12.8% action-coverage statistic

Thank you for highlighting the need to clarify this statistic. Appendix D.4 and Table 11 report a separate, post-hoc action-coverage audit. For 100 held-out CO₂RR goals, we ask GPT-4o, GPT-5.2, and DeepSeek-V3 to each propose five free-form regulation strategies per goal without imposing the predefined chemical action inventory. This yields 500 proposals per backbone and 1,500 in total. We then assess whether each proposal can be assigned to the existing action taxonomy.

Table R4. Post-hoc coverage audit of free-form LLM proposals, as reported in the original Table 11.

Coverage categoryCountFraction
Mapped to the existing action space1,30887.2%
Unmapped but expert-valid765.1%
Unmapped and invalid1167.7%

Thus, 12.8% (192/1,500) denotes free-form proposals outside the predefined action taxonomy, not a runtime discard/remapping rate. CataPath selects its explicit search actions from the predefined inventory.

An unmapped proposal does not make its associated goal out of distribution: these proposals were elicited from held-out benchmark goals. This audit does not evaluate performance degradation on a separately defined OOD-goal cohort, so no such degradation can be inferred from the 12.8% figure. The experiment provides candidates for future expansion through automated LLM generation followed by expert validation, reducing reliance on experts manually proposing every action. We will clarify this distinction and the audit's role in exploring action-space expansion.

AuthorsOpenReview ↗

Response for W5/Q6 & Q7 & Q8

W5 / Q6: Sensitivity to the reward weight λ

Appendix B specifies the original default λ=0.4, and Fig. 4 examines removing either reward component. We have now added a sensitivity analysis over λ ∈ {0, 0.2, 0.4, 0.6, 0.8, 1}. In R=λR^prop+(1−λ)R^EPRMR=\lambda\widehat R_{\mathrm{prop}}+(1-\lambda)\widehat R_{\mathrm{EPRM}}, λ weights the property term: λ=0 uses only EPRM, while λ=1 uses only the property signal. This analysis evaluates the sensitivity of the existing default.

Table R5. Reward-weight sensitivity across six tested λ values.

λReasonableness ↑Proxy Effectiveness ↑Expert Feasibility ↑
0.00.800.780.70
0.20.890.880.78
0.40.900.910.79
0.60.900.900.77
0.80.870.870.71
1.00.820.840.63

λ=0.4 gives the best observed performance on this grid. It achieves the highest proxy effectiveness (0.91) and expert feasibility (0.79), while tying λ=0.6 for the highest reasonableness (0.90). Compared with EPRM-only (λ=0), it improves proxy effectiveness by 0.13 and feasibility by 0.09. While compared with property-only (λ=1), the gains are 0.07 and 0.16. All four tested hybrid settings outperform both single-reward endpoints on the three reported metrics, supporting the complementary roles of the two feedback signals.

Performance is also relatively stable at the neighboring tested weights λ=0.2, 0.4, and 0.6: reasonableness ranges from 0.89 to 0.90, proxy effectiveness from 0.88 to 0.91, and feasibility from 0.77 to 0.79. Thus, the results support retaining λ=0.4 and show that performance does not depend on that exact value alone. We describe 0.4 as the best observed setting among the tested values.

Q7: Fairness of the OSDA-Agent comparison

Yes. CataPath and OSDA-Agent receive the same task-level input prompts, including the starting molecule, target zeolite framework, chemical constraints, and output requirements. Our experimental setup and prompt templates are described in Sec. 5.1 and Appendix E.

For molecular validity filtering and fingerprint evaluation, we follow the original OSDA-Agent protocol (Hu et al., 2025) and apply the same settings to both methods. This includes the same OSDA validity criteria and Morgan, MACCS, and RDK fingerprint evaluation procedures, with our metric definitions provided in Appendix D.2.2.

Q8: Frozen EPRM and access to preference examples

  • EPRM is frozen. As stated in Secs. 4.3 and 5.1, its parameters remain fixed throughout test-time MCTS, with no online updates.
  • Preference pairs are used only for offline EPRM training and validation. Appendix B.2.3 specifies that all pairs come from training-split design goals, with 90% used for training and 10% for validation.
  • The underlying LLM does not receive preference pairs during inference. Appendix E provides the task goal, selected actions/constraints, and generated history as inputs. The labeled preference pairs are not included as prompt examples. MCTS is guided by the frozen EPRM's predicted scores.

Separately, the aggregate expert action-selection statistics in Appendices B.2.2 and C.1 guide pruning. This does not expose the labeled preference pairs to the LLM.

Reviewer_Q7eGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CataPath, an LLM-based catalyst-design agent that searches over chemical actions with UCT-style tree search. It combines property-prediction feedback with an expert preference reward model and introduces CataDesignQR, a dataset for CO2RR catalyst generation and OSDA optimization. The problem is important, but the approach lacks novelty, and the paper needs stronger evidence that improvements come from the proposed search and reward design rather than additional inference-time compute, task-specific engineering, or coupled evaluation signals.

Critical strengths
  1. This paper tackles an important and timely problem: enabling LLM agents to generate catalyst designs that satisfy practical chemical constraints rather than merely producing plausible scientific text.
  2. The hybrid reward has a sensible motivation. A property predictor can provide dense optimization feedback, while expert preferences can penalize designs that score well numerically but violate practical constraints.
Critical weaknesses

Novelty relative to existing search-and-reward-agent frameworks is unclear. The conceptual framework: LLM generation, tree search, tool/predictor feedback, and learned preference rewards resembles a broad family of language-agent and test-time search methods [1] [2]. The paper needs to isolate what is technically new beyond applying this recipe to catalyst design.

[1] Novikov, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.

[2] Zhang, et al.. SciNav: A General Agent Framework for Scientific Coding Tasks. ICLR 2026.

Questions for authors

How many preference pairs are used, how are pairs generated, and how sensitive are results to the amount of expert data?

2 comments
AuthorsOpenReview ↗

Response for W1

We thank the reviewer for recognizing the importance of practical chemical constraints and the motivation for complementary property and expert feedback. We agree that the paper should identify its contribution more precisely and separate it from additional inference-time resources and evaluation alignment.

W1: Technical contributions and supporting controlled comparisons

Technical contributions. CataPath encodes chemical expertise in both what modification to explore and how to evaluate the resulting intermediate design:

  1. Chemical action representation (Secs. 3 and 4.1; Appendix A.3). We organize domain knowledge into 152 CO₂RR actions and 38 OSDA actions, including alloying, defect engineering, ring-size correction, and charge balance. Each selected operation, together with the accumulated history, conditions the next candidate revision.

  2. Expert-supervised feasibility feedback (Sec. 4.3; Appendix B.2.3). EPRM learns preferences between alternatives in the same design context, including hard negatives with high predicted performance but low expert feasibility. The frozen EPRM complements property prediction when scoring newly expanded states, guiding search toward designs consistent with expert judgments of chemical validity and synthetic plausibility.

CataDesignQR supplies the corresponding goals, action sequences, intermediate candidates, and expert annotations across 661 goals and 2,906 trajectories (Table 1). Relative to AlphaEvolve and SciNav, our contribution concerns this chemical representation and expert supervision. MCTS, pairwise learning, and weighted reward combination are established building blocks.

Search under shared components. We evaluate 100 held-out goals over three runs, fixing the backbone, chemical actions, expert priors, and frozen hybrid reward (λ=0.4). Each method is capped at 50 generation calls, including retries, and 50 node evaluations per goal, multi-step searches have maximum depth 10.

Table R1. Search quality and average inference cost per goal per run. All tokens include input and output across all GPT-4 roles (k = 1,000).

MethodReasonableness ↑Proxy Effectiveness ↑Expert Feasibility ↑GPT-4 tokens, kRuntime, min
Best-of-N0.850.820.65391.35
Independent trajectories0.860.840.69741.70
Greedy search0.900.880.73761.74
Beam search0.910.900.77801.81
CataPath0.910.910.80791.78

CataPath improves feasibility over beam search by 0.03 with similar token use and runtime. Their mean property/EPRM scored-input counts are also comparable: 245.5/49.1 versus 247.0/49.4, respectively. This supports an incremental search benefit at comparable observed online cost. Best-of-N remains cheaper in tokens, so matched call caps do not imply equal total compute.

Action and reward contributions. The action comparison keeps MCTS, the backbone, rewards, and generation/evaluation caps fixed, evaluating 500 randomly sampled trajectories from the same 100 test goals. The reward comparison reports three representative settings from the separate six-point λ sweep, where λ weights the property term.

Table R2. Results from separate action-space and reward experiments. **

ExperimentVariantStep violations ↓Proxy Effectiveness ↑Expert Feasibility ↑
Action spaceFree-form22.5%—0.58
Action spaceGeneric structured (26 actions)13.8%—0.74
Action spaceChemical actions8.0%—0.82
RewardEPRM only (λ=0)—0.780.70
RewardHybrid (λ=0.4)—0.910.79
RewardProperty only (λ=1)—0.840.63

Chemical actions reduce violations by 5.8 percentage points and improve feasibility by 0.08 over the generic structured control; inventory sizes differ, so this comparison does not isolate chemical content from action-space size. In the reward experiment, hybrid feedback improves feasibility by 0.16 over property-only and 0.09 over EPRM-only feedback. These separate comparisons support the proposed design choices.

Evaluation independence. Final feasibility is assessed by five experts separate from the EPRM annotation panel, with method identities and other raters' scores hidden (Appendix D.2.1). Test references and labels are withheld during generation (Sec. 3.3; Appendix A.4). A separate calibration on copper-based literature samples held out from training, with 40 records each for CO, formate, and ethylene, yields Spearman ρ=0.74–0.79 and Pearson r=0.68–0.73 between Effectiveness and measured FE. These provide evidence beyond search rewards.

AuthorsOpenReview ↗

Response for Q1

Q1: Preference-pair counts, construction, and sensitivity to expert data

Number of preference pairs. Appendix B.2.3 reports 8,742 pairs in total, including 1,386 hard-negative pairs (15.9%). All are constructed from the training design-goal partition of CataDesignQR, and are then split 90%/10% for EPRM training/validation.

How the pairs are generated (Appendices B.2.1–B.2.3). Experts assign generated intermediate candidates scores on a 1–5 scale for chemical validity, synthetic feasibility, and practical plausibility. The repeated ratings of the same node are averaged. Within each design context, candidate actions sharing the same input condition are compared. A preferred/dispreferred pair is retained only when the expert-score difference is strictly greater than δ=1 (Eq. 10), excluding ties and differences of 1 or less. Hard-negative pairs contrast expert-preferred alternatives with candidates receiving high property-prediction scores but low expert feasibility scores.

Role of expert data and available evidence. The preference pairs are used for offline EPRM training and validation, and EPRM remains frozen during search (Sec. 4.3). The pairs are not provided to the underlying LLM at inference. At the reported data scale, Appendix B.3 gives 81.4% pairwise accuracy and Spearman ρ=0.68 on the EPRM validation split. The reward ablation in Table G2 further shows expert feasibility of 0.79 with hybrid feedback versus 0.63 with property-only feedback. Together, these results support the usefulness of the learned expert signal at the current annotation scale.

Comments on the paper

1 comment

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.