- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CataPath, an expert-augmented LLM framework for constraint-aware scientific reasoning in catalyst design. The method formulates catalyst design as a sequential decision-making problem and employs Monte Carlo Tree Search (MCTS) over a chemically informed action space. During search, an LLM generates material candidates conditioned on predefined design actions and reasoning history, while a hybrid reward combines property predictions with an Expert Preference Reward Model (EPRM) trained on expert-annotated reasoning trajectories.
The authors also introduce CataDesignQR, a dataset containing literature-grounded design goals and expert-scored reasoning trajectories for two tasks: CO2 reduction electrocatalyst generation and organic structure-directing agent (OSDA) optimization.
Experiments show improvements over general LLM generation and existing agentic reasoning baselines in terms of reasonableness, proxy effectiveness, expert-assessed feasibility, and several molecular similarity metrics. The paper further provides component ablations and case-level external validation using DFT calculations and experimental characterization.
Critical strengths
S1. Integration of domain-specific knowledge and expert feedback into structured search.
The paper presents a practical framework that incorporates chemically informed design strategies and expert-preference feedback into LLM-guided search. The combination of property-based and expert-based rewards is particularly relevant to scientific design problems where numerical property predictors alone may not capture practical feasibility constraints. The component ablations provide empirical evidence that the structured action space and hybrid reward contribute to the observed improvements.
S2. Expert-annotated dataset and multi-level empirical validation.
The construction of CataDesignQR provides a useful resource for studying constraint-aware scientific design with intermediate reasoning trajectories. The paper evaluates the framework across two distinct material-design settings and includes expert assessments, comparisons with existing agentic methods, and case-level external validation. These efforts strengthen the practical relevance of the proposed framework beyond conventional language-generation benchmarks.
Critical weaknesses
W1. Insufficient evidence for the necessity of MCTS and its contribution to the reported improvements.
Although MCTS is a central component of CataPath, the experiments do not sufficiently isolate its contribution from constrained candidate generation and reward-guided selection. The reported improvements may primarily result from the predefined action space and hybrid reward mechanism rather than the multi-step planning capability of MCTS. While the paper includes analyses of search depth and individual components, comparisons against simpler search strategies under matched computational budgets are needed to establish the necessity of MCTS.
W2. Limited evidence that improvements in evaluation metrics translate into meaningful scientific outcomes.
The main results rely heavily on proxy-based evaluation metrics whose relationship to the underlying scientific objectives is not fully established. For CO2 reduction, Effectiveness measures embedding similarity to literature-verified designs rather than actual catalytic performance. For OSDA optimization, improvements are primarily reported in molecular similarity metrics, while direct evidence of improved binding energy is limited in the main comparison. Although the paper includes DFT calculations and experimental characterization, these provide case-level evidence rather than systematic validation against competing methods. Consequently, it remains unclear whether the reported improvements reflect better scientific designs or stronger alignment with the chosen evaluation metrics.
W3. Unclear contribution of chemical reasoning beyond search-space restriction.
The paper motivates CataPath by emphasizing the propagation of errors across intermediate reasoning steps and the difficulty of maintaining domain-specific constraints during long-horizon scientific reasoning. However, the proposed method primarily constrains candidate generation through predefined chemical actions and uses reward-guided search to select promising candidates.
The current experiments do not clearly distinguish whether the improvements arise from more accurate chemical reasoning by the LLM, the incorporation of domain knowledge through predefined actions, or simply the reduction of the effective search space. In particular, the action-space ablation compares structured chemical actions with free-form generation, which does not isolate the contribution of chemical knowledge from that of search-space restriction. This makes it difficult to determine how the proposed framework specifically addresses the reasoning failures highlighted in the motivation.
W4. Insufficiently controlled comparisons with existing LLM-based search methods.
CataPath is compared against several prompting and agentic reasoning approaches, including ToT, ReAct, GoT, and LATS. However, the reported comparisons do not clearly establish whether the methods are evaluated under equivalent search budgets and feedback conditions. Differences in the number of generated candidates, LLM calls, external reward evaluations, or computational cost could influence the observed performance gains. Without sufficiently controlled comparisons, it is difficult to attribute the improvements to the proposed search methodology rather than additional computational resources or access to domain-specific feedback.
Questions for authors
Q1. Is MCTS necessary?
The performance improvements may arise from constrained candidate generation and reward-guided selection rather than multi-step tree search. Could the authors compare CataPath against simpler search strategies, such as Best-of-N, greedy search, or beam search, under matched generation and evaluation budgets while keeping the action space and hybrid reward fixed?
Q2. Does the method improve scientific reasoning or candidate selection?
The paper attributes its gains to constraint-aware scientific reasoning, but the evaluations mainly concern final candidate quality. What evidence demonstrates that CataPath improves the correctness of intermediate reasoning steps, rather than simply selecting higher-reward candidates? Could the authors report step-level constraint violations, error correction, or trajectory consistency?
Q3. What is the specific contribution of the chemically informed action space?
The introduction highlights error propagation in long-horizon scientific reasoning as a key motivation. However, the proposed action space appears to primarily constrain the LLM's available design decisions and reduce the effective search space, rather than directly improving its chemical reasoning capability.
Could the authors clarify how the incorporated chemical knowledge enables the LLM to make more scientifically meaningful intermediate decisions beyond restricting the search space? In particular, how does this mechanism prevent or correct errors propagated across reasoning steps? Could the authors disentangle the effects of domain-specific chemical knowledge from those of search-space restriction, for example by comparing against alternative structured action spaces of similar size?
Q4. Are the search comparisons computationally matched?
The paper compares CataPath with several existing LLM-based search and reasoning methods, including ToT, GoT, and LATS. Could the authors clarify whether these methods are compared under equivalent computational budgets, including the number of LLM calls, generated candidates, search expansions, and external reward evaluations? Furthermore, how are differences in available domain-specific feedback controlled when comparing CataPath with these baselines?
Minor Concern — Incorrect reference for CatBERT Lee et al. (2020), which concerns social engineering email detection rather than catalyst property prediction.
5 comments
Response for W2
W2: Scientific meaning of the evaluation metrics
CO₂RR: Calibration against measured FE, complemented by targeted experimental validation. Appendix D.2.1 defines Effectiveness as an embedding-similarity proxy relative to literature-verified designs. We agree that its relationship to measured catalytic performance requires validation. Synthesizing and testing outputs from all compared methods under a common experimental protocol would require substantial resources. We therefore added a literature-based calibration against experimentally measured Faradaic efficiency (FE), evaluated separately for CO, formate, and ethylene, with 40 records per product under the cell and electrolyte conditions listed below. This provides a tractable assessment of the metric's empirical relevance, complemented by the targeted physical validation already reported in the paper.
Table R2. Correlation between proxy Effectiveness and experimentally measured FE, reported separately for each product and condition group.
| Product-specific comparable-condition cohort | Measured records, n | Spearman ρ [95% CI] | Pearson r [95% CI] |
|---|---|---|---|
| CO — H-cell; 0.1 M KHCO₃ | 40 | 0.79 [0.61, 0.89] | 0.73 [0.54, 0.85] |
| Formate — H-cell; 0.5 M KHCO₃ | 40 | 0.76 [0.56, 0.88] | 0.70 [0.50, 0.83] |
| Ethylene — H-cell; 0.1 M KHCO₃ | 40 | 0.74 [0.53, 0.87] | 0.68 [0.47, 0.80] |
Across the three product-specific cohorts, Spearman correlations range from 0.74 to 0.79 and Pearson correlations from 0.68 to 0.73. These positive associations provide empirical support for the proxy's relationship to measured FE on the evaluated literature samples. They support its use as a performance-associated evaluation signal within this scope.
In addition, complementary physical evidence is already provided in Appendix D.3: Table 8 reports DFT-calculated adsorption energies for selected CataPath-proposed surfaces; Figs. 9–11 characterize a synthesized Cu–Sn bimetallic nanostructure; and Fig. 12 reports its FE–potential curves alongside a Cu–Sn mixture control, polarization curves, and stability measurements. These results support the synthetic accessibility and electrochemical relevance of a selected generated design. The literature calibration and targeted experiments thus address metric relevance and case-level feasibility, respectively. However, they do not establish systematic experimental superiority over other generation methods. A common-protocol comparison of newly generated candidates remains future work.
OSDA: comparing predicted binding-energy improvement using the existing frozen predictor.
We agree that Table 3 does not establish superior binding energy. Its ED is Energy Distance, a distributional metric. Relative to OSDA-Agent, CataPath improves BLEU, Morgan, and RDK similarity, matches validity and MACCS, and does not improve ED or KL divergence. These structural metrics alone cannot demonstrate improved binding performance.
To supplement this comparison, we added predicted binding-energy results for OSDA-Agent and CataPath using the existing frozen predictor described in Sec. 4.3 and Appendix B.1. Table R3 reports 90 attempts per method for the LTA framework. We compare the raw predicted binding energy of each final molecule with that of its starting molecule:
A negative value indicates a predicted improvement. This quantity is computed before reward rescaling and does not include the EPRM score.
Table R3. Predicted binding-energy comparison using the existing frozen predictor, with 90 attempts per method.
| Method | Mean ΔÊ_bind, eV per OSDA ↓ | Predicted improvements / all attempts ↑ | Invalid outputs / all attempts ↓ |
|---|---|---|---|
| OSDA-Agent | −0.19 | 59/90 (65.6%) | 0/90 (0.0%) |
| CataPath | −0.23 | 65/90 (72.2%) | 0/90 (0.0%) |
Both methods report 90 scored outputs, with no invalid outputs. Improvement rates use all 90 attempts as the denominator.
CataPath achieves a mean predicted binding-energy change of −0.23 eV per OSDA, compared with −0.19 for OSDA-Agent, corresponding to an additional predicted reduction of 0.04 eV per OSDA. It improves on the starting molecule in 65 of 90 attempts (72.2%), compared with 59 of 90 (65.6%) for OSDA-Agent, yielding six additional improvements with the same number of attempts. These descriptive results support better optimization of the predictor-defined binding-energy objective in the evaluated sample, supplementing the structural metrics in Table 3. However, we admit that this is a surrogate-based comparison, not independent physical validation. The physical validation in the present study is concentrated on CO₂RR. Those case studies do not validate OSDA binding performance.
Response for W3 & Q2 & Q3
W3 & Q2: Reliability of intermediate decisions
We agree that final candidate quality alone does not establish the reliability of intermediate decisions. We therefore added a process-level evaluation of randomly sampled trajectories produced by independent refinement, greedy search, beam search, and CataPath. We distinguish three aspects:
- Step-level constraint violation: whether a generated candidate introduces or retains a violation of the task chemical constraint applicable at that step.
- Action execution consistency: whether the generated modification implements the selected operation and its explicit preservation requirements. Executing an action does not necessarily imply satisfying all other task constraints.
- Trajectory consistency: whether the sampled trajectory avoids contradictory claims or unjustified loss of information required to be retained across steps. Explicitly permitted design revisions are not treated as contradictions.
Best-of-N is omitted from this multi-step comparison because its one-step outputs do not provide comparable trajectories. We randomly sample 500 trajectories from the same 100 held-out goals used in W1/Q1, all drawn from the paper's fixed test split. W3/Q3 uses the same trajectory-sampling and evaluation setup.
Table R4. Process-level evaluation of randomly sampled trajectories from the fixed test goals.
| Method | Step violations ↓ | Action execution consistency ↑ | Trajectory consistency ↑ |
|---|---|---|---|
| Independent trajectories | 18.6% | 78.8% | 69.4% |
| Greedy search | 14.2% | 86.2% | 78.2% |
| Beam search | 13.2% | 89.4% | 80.8% |
| CataPath | 8.0% | 94.2% | 85.2% |
CataPath has the lowest step-level violation rate (8.0%) and the highest action execution consistency (94.2%) and trajectory consistency (85.2%) among the evaluated methods. These results extend the evaluation beyond final candidate quality: the sampled CataPath trajectories contain fewer constraint violations, implement the prescribed actions more consistently, and exhibit greater cross-step consistency. We interpret these findings as improved process reliability in the sampled trajectories. This does not demonstrate an improvement in the underlying LLM's intrinsic reasoning ability. Rejecting an erroneous branch is distinct from repairing an error within that branch. We will align the paper's claims with this scope.
W3 & Q3: Chemical knowledge beyond restricting the search space
We agree that Fig. 3 compares two factors simultaneously: structure and chemical content. We therefore added an action-space ablation comparing free-form actions, generic structured actions, and chemically informed actions within CataPath. The generic structured condition provides a comparison for organizing decisions without task-specific chemical instructions. The process evaluation follows the same random-trajectory sampling setup as W3/Q2: 500 trajectories from the same 100 goals in the paper's fixed test split.
Across variants, we keep the other experiment setups fixed and compare three action-proposal schemes: free-form generation, a generic structured inventory, and the chemically informed inventory. In the free-form condition, the LLM first proposes an action freely and then executes the modification. The action execution consistency is assessed against that proposed action.
The generic structured inventory contains 26 selectable actions, expressed as domain-agnostic edit templates. Examples include: (1) replace one component while keeping the remaining design unchanged; (2) add/remove one component while preserving unrelated design attributes. These templates organize how to edit a design without prescribing a chemically motivated choice.
Table R5. Action-space ablation with the same trajectory-evaluation setup as Table R4.
| Action space | Step violations ↓ | Action execution consistency ↑ | Trajectory consistency ↑ | Expert Feasibility ↑ |
|---|---|---|---|---|
| Free-form | 22.5% | 76.4% | 58.8% | 0.58 |
| Generic structured (26 actions) | 13.8% | 85.6% | 70.0% | 0.74 |
| Chemically informed | 8.0% | 94.2% | 85.2% | 0.82 |
Generic structured actions improve over free-form actions. Chemically informed actions provide further gains over the generic structured condition. Thus, the chemical action specification improves all four reported measures relative to the evaluated generic structured control. The chemical actions supply concrete instructions about what to modify and preserve. For example, Appendix A.3 includes OSDA operations for ring-size correction and charge balance. The observed improvements are consistent with such domain guidance helping the system produce more constraint-preserving edits and coherent trajectories in the random sample. We therefore claim an empirical benefit over the evaluated generic structured alternative, while acknowledging that this comparison does not fully isolate chemical knowledge from action-space size.
Response for W1 & Q1 & Minor Concern
We thank the reviewer for recognizing the relevance of chemical constraints, expert feedback, and the dataset, and for identifying the need for stronger attribution and scientific validation. We address the overlapping weaknesses and questions together below.
We agree that the original component ablations and depth analysis do not sufficiently isolate MCTS from constrained generation, reward-based selection, and additional computation. The controlled comparison therefore keeps the LLM, chemical action space, expert-derived action priors, and frozen reward models fixed. All methods use the same node-level reward, R=0.4Rprop+0.6REPRM.
We cap each method at 50 generation calls per goal, including retries, and at 50 node-level evaluations through the same scoring pipeline. The final output is the complete eligible list with the highest hybrid reward among the generated nodes. The evaluation uses the same 100 held-out goals over three runs, with maximum trajectory depth 10. Independent trajectories use history-conditioned refinement. The greedy search expands five alternatives per step and beam search uses width three and three alternatives per retained path.
Table R1. Search comparison on 100 held-out goals over three runs.
CataPath achieves the highest reported expert feasibility (0.80) and proxy effectiveness (0.91), while matching beam search on reasonableness (0.91). Its absolute feasibility gains are 0.03 over beam search, 0.07 over greedy search, 0.11 over independent trajectories, and 0.15 over Best-of-N. The comparison with independent trajectories shows an improvement beyond history-conditioned refinement alone, while the comparison with beam search supports an additional benefit beyond maintaining multiple candidate paths. With the chemical action space, expert priors, reward models, and generation/evaluation caps held fixed, these results support the incremental value of MCTS-guided search at the evaluated budget. The equal token cost is addressed separately under Response for W4 & Q4.
Thank you for identifying this error. The model used in our experiments is CatBERTa, and the correct reference is Ock, Guntuboina, and Barati Farimani (2023), “Catalyst Energy Prediction with CatBERTa: Unveiling Feature Exploration Strategies through Large Language Models,” ACS Catalysis, 13(24), 16032–16044. We will correct the model name and replace the erroneous Lee et al. (2020) citation throughout the manuscript.