- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work introduces a novel scoring scheme that integrates the Graph of Thoughts (GoT) framework into molecular editing and understanding tasks, enabling more robust agentic workflows with fewer errors and improved token efficiency. The study is thorough and includes extensive ablation experiments, demonstrating that, for tasks such as organic molecule editing, the performance improvements primarily arise from the combined effect of task orchestration, scoring, and merging strategies. Each individual component, when applied in isolation, does not necessarily lead to substantial improvements.
Critical strengths
-
The conventional GoT framework is relatively abstract and does not use specific physical or chemical meanings. The operations are branching/merging between different reasoning states. This work merged molecular editing information and domain-specific knowledge into the framework, making the reasoning process on the graph more controllable and better guided for chemical tasks. This integration also enhances interpretability and reduces the generation of invalid molecules.
-
The experimental evaluation is comprehensive, with extensive ablation studies and detailed analysis. The authors objectively describe the contributions and even the limitations of individual components, providing a convincing evaluation of the proposed framework.
Critical weaknesses
-
Lack of tests in larger molecules (molecule weight's influnence on the results). In this work, the authors use relatively small molecules which can be separated into three regions for testing. I was wondering about the robustness of the current scoring functions towards editing larger systems. Since there are some similarity checks in the scoring, and a larger system could reduce the distinguishability of the scores.
-
The ablation results presented in Fig. 5a are particularly interesting. Among the evaluated components, count route appears to have the largest impact on performance. Interestingly, removing certain components, such as repair, seems to improve performance for some models, while other intuitively important components, such as merge selection, show relatively limited effects. These observations deserve a more detailed analysis to understand which aspects of the reasoning process are affected by each component and why such effects arise. Furthermore, these results may suggest that the current ablation study does not fully disentangle the contributions and interactions of different components. If feasible, the authors are encouraged to extend the ablation study to include additional components or combinations of components, which could provide a clearer understanding of their individual and synergistic effects.
Questions for authors
-
How does the proposed scoring function accommodate molecular editing tasks that require substantial modifications to the original molecule? In Eq. (1), the similarity term is defined based on the similarity to the original molecule. While this term may help prevent unreasonable deviations from the starting structure, it could potentially become counterproductive when substantial structural changes are necessary to achieve the desired outcome. In such cases, could a more accurate and desirable edited molecule actually receive a lower score due to its reduced similarity to the original molecule? It would be helpful if the authors could clarify how the scoring function handles such scenarios and whether this introduces a bias toward conservative molecular edits.
-
It would be better to add some more analysis and tests in ablation studies if possible.
6 comments
W2 / Q2. A more detailed analysis of Fig. 5(a) and an extended ablation.
We thank the reviewer for this careful reading of Fig. 5(a). We agree that this ablation does not fully disentangle the components: its variants cover only part of the graph operations, and No merge selection removes only the final selection of composed candidates, not the composition itself. Below we first explain the three observations on Fig. 5(a), and then report a new ablation that removes each graph operation completely, one at a time and in combination.
(1) A closer analysis of Fig. 5(a).
- The count route has the largest impact. The count route combines the counts of the individual regions into one answer, i.e., it aggregates regional evidence. The new ablation confirms this: without this aggregation of regional evidence, Qwen3-8B drops from 59.0% to 34.0% on the counting and understanding tasks (see (3)).
- Merge selection has a limited effect. This variant only decides whether a composed candidate may be returned as the final answer. A composed candidate is returned only if it is at least as good as the best single-branch candidate (Eq. 2), which happened in 6 of the 990 editing traces of Table 32, so the final answer seldom changes.
- Removing repair seemed to help some models. The difference from the full framework is not significant in eight of the nine models (p ≥ 0.36, exact McNemar test). The exception is R1, and it is an effect of truncation: in the original configuration, 1,033 of its 1,100 questions had at least one truncated call. With a larger output budget, removing repair lowers the accuracy of R1 by 18.3 pp (see (2)).
(2) New ablation that isolates each operation. We removed each graph operation of MolGoT completely, one at a time:
- No branching: one candidate is kept per step instead of several; for the counting tasks, the molecule is not split into regions.
- No scoring: candidates are selected at random instead of by their score; the validity check is retained.
- No aggregation: several parent states (the evidence of different regions, or compatible edits) are no longer combined into one.
- No refinement: invalid edits are no longer repaired, composed candidates are no longer revised, and disagreeing counts are no longer adjudicated.
- No repair: only the repair of invalid edits is removed.
In each condition all other operations stay active. One further condition removes aggregation and refinement together, to measure how the two interact.
The ablation uses 110 questions, 10 for each of the eleven tasks in Table 1 of the paper, drawn from the same benchmark sources as the main experiments. The models are Qwen3-8B and R1; R1 uses a larger output budget than in the paper so that its reasoning is not truncated. All numbers are accuracy in %.
Branching, aggregation, and refinement apply to all 110 questions:
| MolGoT variant | Qwen3-8B | R1 |
|---|---|---|
| Full | 38.6 | 50.9 |
| No branching | 23.2 | 43.6 |
| No aggregation | 27.7 | 42.7 |
| No refinement | 37.3 | 40.0 |
| No aggregation and no refinement | 28.2 | 34.5 |
Removing any of these operations lowers accuracy for both models. Qwen3-8B loses most from removing branching (15.4 pp) and aggregation (10.9 pp). R1 loses most from removing refinement (10.9 pp), and 16.4 pp when aggregation and refinement are both removed.
Scoring applies to 90 of the 110 questions, because the carbon- and ring-counting routes do not score candidates. On these 90 questions:
| MolGoT variant | Qwen3-8B | R1 |
|---|---|---|
| Full | 28.9 | 45.6 |
| No scoring | 20.0 | 16.7 |
Selecting candidates by score instead of at random is worth 8.9 pp for Qwen3-8B and 28.9 pp for R1.
Repair applies to the 60 editing and generation questions, because the other tasks do not execute edits. On these 60 questions:
| MolGoT variant | Qwen3-8B | R1 |
|---|---|---|
| Full | 21.7 | 48.3 |
| No repair | 19.2 | 30.0 |
Repair is worth 2.5 pp for Qwen3-8B and 18.3 pp for R1. The effect is smaller for Qwen3-8B because its repair step is rarely triggered.
W2 / Q2. A more detailed analysis of Fig. 5(a) and an extended ablation.
(3) Which part of the reasoning each operation affects, and how the operations interact.
- Aggregation takes two forms. In the counting and understanding tasks, it combines the evidence of the individual regions into one answer; in the editing and generation tasks, it combines compatible edits of several candidates into one. Without aggregation, Qwen3-8B drops from 59.0% to 34.0% on the counting and understanding tasks, and R1 drops from 63.3% to 46.7% on the editing tasks.
- Refinement and repair act on the editing and generation tasks. They revise a candidate using the feedback from executing the edit. Without refinement, R1 drops from 63.3% to 40.0% on editing and from 33.3% to 20.0% on generation, while its accuracy on the counting and understanding tasks stays at the same level (54.0% with refinement, 52.0% without).
- Branching and scoring act on all tasks that search over candidates. Without either of them, accuracy drops on the counting and understanding, editing, and generation tasks alike.
- Aggregation and refinement are complementary. For R1, removing aggregation costs 8.2 pp, removing refinement costs 10.9 pp, and removing both costs 16.4 pp, more than removing either one alone, so neither operation substitutes for the other.
We will add the new ablation and this analysis to the revised paper.
W1. Tests on larger molecules and the robustness of the scoring function.
We thank the reviewer for this suggestion. We ran the editing tasks on a new set of molecules of up to 3819 g/mol. MolGoT is more accurate than CoT at every molecular size, the region decomposition still helps on the largest molecules, and the similarity scores of candidates do not become harder to distinguish as molecules grow.
Setup. 120 distinct molecules with 9–169 carbon atoms (201–3819 g/mol), split by carbon count into four groups of 30. Each group has 10 addition, 10 deletion, and 10 substitution tasks. The model is Qwen3-8B. All numbers are accuracy in %.
(1) MolGoT improves over CoT in every size group. The table compares SMILES-CoT and MolGoT in each group. The last column is MolGoT with the whole molecule treated as one region, which we discuss in (2).
MolGoT is 11.6 to 28.3 pp above CoT in every group, and 17.9 pp above it overall (26.7% → 44.6%). Grouping the same molecules by molecular weight gives the same result:
MolGoT is higher in every weight range, and the advantage holds for the heaviest molecules: above 2000 g/mol, CoT solves 9.5% of the tasks and MolGoT solves 38.1%.
(2) The region decomposition remains useful for large molecules. Treating the whole molecule as a single region (last column of the first table) lowers accuracy in every size group, from 44.6% to 32.1% overall and from 31.7% to 23.3% in the largest group. The decomposition therefore still helps when molecules are large. The number of regions is a configurable limit, not a restriction on molecular size.
(3) The similarity term remains discriminative for large molecules. For each molecule, we take the candidates that MolGoT generated and measure the range of their similarity to the original molecule (maximum minus minimum). The average range is 0.12, 0.13, 0.17, and 0.12 in the four size groups, from the smallest to the largest. The range does not shrink as molecules grow, so the similarity term distinguishes the candidates of large molecules as well as those of small ones.
We will add these experiments to the revised paper.