- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ChemHarness, a self-evolving framework for chemical reasoning without ground-truth labels or model updates. It uses two reasoning streams: one generates natural language solutions, while the other executes programs. Their outputs are compared to provide feedback. The framework also maintains a reusable tool library and short-term memory, with asymmetric information flow to reduce correlated errors. Experiments on four SciBench chemistry datasets show consistent improvements over direct reasoning and existing chemical agents.
Critical strengths
- The dual-stream verification mechanism provides a practical way to obtain feedback without ground-truth answers. The asymmetric memory design is also reasonable for reducing correlated errors.
- The experiments cover four backbone models and show consistent accuracy improvements. The results suggest that the overall framework is effective for numerical chemical reasoning.
Critical weaknesses
- The contribution is unclear. The framework mainly combines LLM prompting, code execution, verification, and memory management. More importantly, the paper does not clearly isolate the contribution of the evolving tool library. It is difficult to determine whether the improvement comes from accumulated experience or repeated reasoning. The contribution is primarily at the system integration level, and the technical novelty is limited.
- ChemHarness uses up to eight rounds of dual-stream reasoning and additional LLM calls for verification. However, the paper lacks comparisons with simpler methods under the same inference budget. This makes it difficult to assess whether the proposed framework offers a meaningful advantage over repeated sampling or code-based reasoning.
Questions for authors
- Can the authors provide an ablation that removes the evolving tool library while keeping dual-stream verification unchanged? This would help clarify the actual contribution of self-evolution.
- How does ChemHarness compare with Self-Consistency or repeated Program-of-Thought reasoning under the same token or API-call budget?
- Since both reasoning streams and the judge use the same backbone model, how can the framework reliably handle cases where they share the same incorrect assumptions?
2 comments
Response to Reviewer 9u2f (2/2)
Table C. SciBench [1] without the tool library, three runs (Appendix C).
| Backbone | Without the tool library | With it (Table 2) |
|---|---|---|
| Qwen2.5-7B | 52.0 | 48.8 |
| Qwen2.5-14B | 65.3 | 63.3 |
| Qwen2.5-32B | 68.5 | 67.4 |
| DeepSeek-V4.1-Flash | 78.2 | 80.1 |
Question 1: “Can the authors provide an ablation that removes the evolving tool library while keeping dual-stream verification unchanged? This would help clarify the actual contribution of self-evolution.”
Reply: On UGPhysics [2], with dual-stream verification unchanged, removing all experience gives the "No experience" column of Table A. Adding the evolving tool library changes it by −0.9 to +0.6; adding the verified programs it is built from raises it by +4.5 to +6.6, positive in every fold. On SciBench, removing the tool library gives Table C. The self-evolving part that matters is the store of verified programs.
Question 2: “How does ChemHarness compare with Self-Consistency or repeated Program-of-Thought reasoning under the same token or API-call budget?”
Reply: Table B gives all seven backbones with tokens per problem. At nearly equal token cost, the clearest case is Qwen2.5-14B: with 4 samples per stream, verification matches the tokens of repeated Program-of-Thought [3] (3,940 against 3,875) and scores 2.1 points higher; with 8, it uses fewer tokens than text self-consistency [4] (6,471 against 6,815) and scores 1.9 higher.
Table D. Precision (%) of agreement on SciBench, and the judge. The first two columns cover the problems with at least two rounds (Section 4.7, Table 5), the others all problems of one run (Appendix D.4, Table 11).
| Backbone | Two streams agree | Two samples of one stream agree | Rejected by the judge | Precision, rejected / accepted |
|---|---|---|---|---|
| Qwen2.5-7B | 63.0 | 55.7 | 25.9% | 45 / 71 |
| Qwen2.5-14B | 66.7 | 46.2 | 7.5% | 53 / 74 |
| Qwen2.5-32B | 68.3 | 50.8 | 5.3% | 64 / 76 |
| DeepSeek-V4.1-Flash | 75.4 | 54.2 | 9.9% | 71 / 84 |
Question 3: “Since both reasoning streams and the judge use the same backbone model, how can the framework reliably handle cases where they share the same incorrect assumptions?”
Reply: This failure mode exists, and Section 6 states it. Agreement is a stopping signal, not a label, and every reported score counts agreed-but-wrong answers. The framework reduces the failure in three measurable ways:
- Different reasoning paradigms err jointly less often than two samples of one paradigm, on every backbone.
- The judge removes some shared errors: the agreements it rejects are less often correct on every backbone.
- Verified experience makes agreement more reliable: with the experience of Qwen3.6-27B, an agreed answer on UGPhysics is correct 3.8 to 5.1 points more often, for all four readers (Appendix C.4).
References
- [1] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
- [2] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
- [3] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR 2023.
- [4] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
Response to Reviewer 9u2f (1/2)
Thank you for your review.
Table A. What carries the gain (Section 4.6, Table 4; UGPhysics [1], 1,068 held-out problems, 5-fold cross-validation). All variants share one pipeline, the dual-stream core of ChemHarness with the symbolic-only information flow of the submission, and differ only in the experience, which is built without labels on practice problems, frozen, and evaluated on held-out folds. Gain over the same pipeline without experience, ± paired standard error:
Weakness 1: “The contribution is unclear. The framework mainly combines LLM prompting, code execution, verification, and memory management. More importantly, the paper does not clearly isolate the contribution of the evolving tool library. It is difficult to determine whether the improvement comes from accumulated experience or repeated reasoning. The contribution is primarily at the system integration level, and the technical novelty is limited.”
Reply: The gain comes from accumulated experience.
The contribution is therefore a controlled finding about the form that label-free, self-evolved experience must take to transfer: verified whole programs transfer and their abstraction into tools does not. The benefit also grows with how much more accurate the writer is than the reader (r = 0.89 over 31 writer–reader pairs; Appendix C.4).
Table B. Verification against simpler methods at a comparable number of tokens (Appendix H.1, Table 16; UGPhysics, no experience). Accuracy, tokens per problem in parentheses. Self-consistency [2] votes over 8 samples of one stream; dual-stream verification samples up to 4 or 8 times per stream and stops at the first agreement; Δ compares Dual 8+8 with Text ×8.
Weakness 2: “ChemHarness uses up to eight rounds of dual-stream reasoning and additional LLM calls for verification. However, the paper lacks comparisons with simpler methods under the same inference budget. This makes it difficult to assess whether the proposed framework offers a meaningful advantage over repeated sampling or code-based reasoning.”
Reply: We added it (Table B).