acceptodds
Under review as a conference paper at ICLR 2027

ChemHarness: A Self-Evolving System for Chemical Reasoning via Asymmetric Dual-Stream Verification

Abstract

Self-evolving harnesses improve a language model by consolidating its verified experience into reusable memory, but real-world chemical reasoning problems lack ground-truth answers for verification, and repeated samples of one model tend to repeat the same errors. We present ChemHarness, a closed-loop self-evolving harness for chemical reasoning driven by dual-stream self-verification. ChemHarness solves each problem through two streams of one language model: a neural stream derives the answer in natural language, and a symbolic stream writes a program that an interpreter executes. Over repeated rounds, cross-stream verification compares the two answers and turns them into label-free feedback in the form of cross-verified answers, intermediate lemmas, and discrepancies. ChemHarness organizes this feedback through a procedural tool library across problems, an episodic short-term memory within a problem, and an asymmetric information flow that withholds both memories from the neural stream, which therefore remains an independent verifier. On the chemistry datasets of SciBench, ChemHarness improves direct reasoning by 6.4 to 12.7 points on Qwen2.5-7B, 14B, 32B, and DeepSeek-V4.1-Flash, and exceeds ChemAgent by 23.3 and 22.9 points on Qwen2.5-7B and 14B. We further find that cross-stream verification is markedly more precise than single-stream consistency, with a precision of 63.0% to 75.4% against 46.2% to 55.7%, and that memory reaching the neural stream lowers the precision. Dual-stream self-verification thus allows a harness to evolve in chemical reasoning without ground-truth labels, parameter updates, or external supervision.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Chemical Reasoning, Self-Evolving Harness, Self-Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9u2fOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ChemHarness, a self-evolving framework for chemical reasoning without ground-truth labels or model updates. It uses two reasoning streams: one generates natural language solutions, while the other executes programs. Their outputs are compared to provide feedback. The framework also maintains a reusable tool library and short-term memory, with asymmetric information flow to reduce correlated errors. Experiments on four SciBench chemistry datasets show consistent improvements over direct reasoning and existing chemical agents.

Critical strengths
  1. The dual-stream verification mechanism provides a practical way to obtain feedback without ground-truth answers. The asymmetric memory design is also reasonable for reducing correlated errors.
  2. The experiments cover four backbone models and show consistent accuracy improvements. The results suggest that the overall framework is effective for numerical chemical reasoning.
Critical weaknesses
  1. The contribution is unclear. The framework mainly combines LLM prompting, code execution, verification, and memory management. More importantly, the paper does not clearly isolate the contribution of the evolving tool library. It is difficult to determine whether the improvement comes from accumulated experience or repeated reasoning. The contribution is primarily at the system integration level, and the technical novelty is limited.
  2. ChemHarness uses up to eight rounds of dual-stream reasoning and additional LLM calls for verification. However, the paper lacks comparisons with simpler methods under the same inference budget. This makes it difficult to assess whether the proposed framework offers a meaningful advantage over repeated sampling or code-based reasoning.
Questions for authors
  1. Can the authors provide an ablation that removes the evolving tool library while keeping dual-stream verification unchanged? This would help clarify the actual contribution of self-evolution.
  2. How does ChemHarness compare with Self-Consistency or repeated Program-of-Thought reasoning under the same token or API-call budget?
  3. Since both reasoning streams and the judge use the same backbone model, how can the framework reliably handle cases where they share the same incorrect assumptions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9u2f (1/2)

Thank you for your review.

Table A. What carries the gain (Section 4.6, Table 4; UGPhysics [1], 1,068 held-out problems, 5-fold cross-validation). All variants share one pipeline, the dual-stream core of ChemHarness with the symbolic-only information flow of the submission, and differ only in the experience, which is built without labels on practice problems, frozen, and evaluated on held-out folds. Gain over the same pipeline without experience, ± paired standard error:

ReaderNo experience+ random programs+ abstracted tools+ verified programs
Llama-3.1-8B20.4+0.3 ± 0.7−0.9 ± 0.7+6.6 ± 1.0
Qwen2.5-7B27.5+0.3 ± 0.6+0.6 ± 0.7+4.8 ± 0.9
Qwen2.5-14B34.1−0.2 ± 0.7+0.5 ± 0.7+4.7 ± 0.9
Qwen3-8B40.0−1.2 ± 0.7+0.0 ± 0.8+4.5 ± 0.9

Weakness 1: “The contribution is unclear. The framework mainly combines LLM prompting, code execution, verification, and memory management. More importantly, the paper does not clearly isolate the contribution of the evolving tool library. It is difficult to determine whether the improvement comes from accumulated experience or repeated reasoning. The contribution is primarily at the system integration level, and the technical novelty is limited.”

Reply: The gain comes from accumulated experience.

  • Repeated reasoning with more context does not explain the gain: random programs from the same pool add the same context and inference and give −1.2 to +0.3.
  • Accumulated verified programs add +4.5 to +6.6 for every reader at nearly unchanged token cost (Llama-3.1-8B: 5,183 against 5,270 tokens per problem), positive in all 75 run–fold pairs of repeated runs (Appendix C.3, Table 9).
  • The tool library, isolated as you asked, gives −0.9 to +0.6, although readers retrieve a tool for 57–96% of the problems.

The contribution is therefore a controlled finding about the form that label-free, self-evolved experience must take to transfer: verified whole programs transfer and their abstraction into tools does not. The benefit also grows with how much more accurate the writer is than the reader (r = 0.89 over 31 writer–reader pairs; Appendix C.4).

Table B. Verification against simpler methods at a comparable number of tokens (Appendix H.1, Table 16; UGPhysics, no experience). Accuracy, tokens per problem in parentheses. Self-consistency [2] votes over 8 samples of one stream; dual-stream verification samples up to 4 or 8 times per stream and stops at the first agreement; Δ compares Dual 8+8 with Text ×8.

BackboneText ×8Program ×8Dual 4+4Dual 8+8Δ
Llama-3.1-8B21.5 (8,839)17.1 (2,498)20.4 (5,217)23.2 (9,683)+1.7
Qwen2.5-7B28.0 (6,364)20.1 (2,173)26.7 (3,533)28.3 (6,206)+0.3
Qwen2.5-14B33.9 (6,815)32.3 (3,875)34.4 (3,940)35.8 (6,471)+1.9
Qwen3-8B41.4 (8,204)31.4 (2,058)40.1 (3,762)41.0 (6,131)−0.4
Qwen3.5-9B55.9 (21,618)53.8 (9,448)56.9 (11,058)58.8 (18,073)+2.9
Qwen3.6-27B60.2 (15,746)60.1 (6,816)61.0 (6,536)61.4 (9,731)+1.1
Qwen3.8-27B57.9 (17,100)59.0 (16,924)59.1 (10,212)59.8 (15,821)+1.9

Weakness 2: “ChemHarness uses up to eight rounds of dual-stream reasoning and additional LLM calls for verification. However, the paper lacks comparisons with simpler methods under the same inference budget. This makes it difficult to assess whether the proposed framework offers a meaningful advantage over repeated sampling or code-based reasoning.”

Reply: We added it (Table B).

  • Verification is ahead of text self-consistency for five of seven backbones, with fewer tokens for six, and beats repeated Program-of-Thought [3] for every backbone.
  • The two ties (Qwen2.5-7B, Qwen3-8B) are the backbones whose program streams fall furthest below their text streams (by 7.9 and 10.0 points).
  • Code alone does not explain the gain: on SciBench [4], the first-round symbolic stream scores 38.8 / 51.1 / 55.1 / 70.6, against 36.1 / 52.3 / 55.8 / 73.7 for direct reasoning.
  • Experience is a better use of budget than more samples: with it, 4 samples per stream beat 8 without it by +3.0 to +4.0, with 35–46% fewer tokens (Tables 4 and 16).

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
  • [3] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR 2023.
  • [4] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
AuthorsOpenReview ↗

Response to Reviewer 9u2f (2/2)

Table C. SciBench [1] without the tool library, three runs (Appendix C).

BackboneWithout the tool libraryWith it (Table 2)
Qwen2.5-7B52.048.8
Qwen2.5-14B65.363.3
Qwen2.5-32B68.567.4
DeepSeek-V4.1-Flash78.280.1

Question 1: “Can the authors provide an ablation that removes the evolving tool library while keeping dual-stream verification unchanged? This would help clarify the actual contribution of self-evolution.”

Reply: On UGPhysics [2], with dual-stream verification unchanged, removing all experience gives the "No experience" column of Table A. Adding the evolving tool library changes it by −0.9 to +0.6; adding the verified programs it is built from raises it by +4.5 to +6.6, positive in every fold. On SciBench, removing the tool library gives Table C. The self-evolving part that matters is the store of verified programs.

Question 2: “How does ChemHarness compare with Self-Consistency or repeated Program-of-Thought reasoning under the same token or API-call budget?”

Reply: Table B gives all seven backbones with tokens per problem. At nearly equal token cost, the clearest case is Qwen2.5-14B: with 4 samples per stream, verification matches the tokens of repeated Program-of-Thought [3] (3,940 against 3,875) and scores 2.1 points higher; with 8, it uses fewer tokens than text self-consistency [4] (6,471 against 6,815) and scores 1.9 higher.

Table D. Precision (%) of agreement on SciBench, and the judge. The first two columns cover the problems with at least two rounds (Section 4.7, Table 5), the others all problems of one run (Appendix D.4, Table 11).

BackboneTwo streams agreeTwo samples of one stream agreeRejected by the judgePrecision, rejected / accepted
Qwen2.5-7B63.055.725.9%45 / 71
Qwen2.5-14B66.746.27.5%53 / 74
Qwen2.5-32B68.350.85.3%64 / 76
DeepSeek-V4.1-Flash75.454.29.9%71 / 84

Question 3: “Since both reasoning streams and the judge use the same backbone model, how can the framework reliably handle cases where they share the same incorrect assumptions?”

Reply: This failure mode exists, and Section 6 states it. Agreement is a stopping signal, not a label, and every reported score counts agreed-but-wrong answers. The framework reduces the failure in three measurable ways:

  • Different reasoning paradigms err jointly less often than two samples of one paradigm, on every backbone.
  • The judge removes some shared errors: the agreements it rejects are less often correct on every backbone.
  • Verified experience makes agreement more reliable: with the experience of Qwen3.6-27B, an agreed answer on UGPhysics is correct 3.8 to 5.1 points more often, for all four readers (Appendix C.4).

References

  • [1] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
  • [2] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [3] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR 2023.
  • [4] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
Reviewer_E76pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ChemHarness, a test-time harness meant to improve a fixed LLM on numerical chemistry problems without labels. Each problem is solved in up to R = 8 rounds. In every round, the model writes a chain-of-thought derivation that computes in text and a Python program that an interpreter runs. The derivation's value of the first verified round is the answer. A verified solution is judged by the same model, using both streams of information as input. Problems without a verified round get one more solution attempt, which may execute code, conditioned on a short-term memory: intermediate values both streams agreed on, program variables that differ across rounds, and the tally of earlier answers. The hraness is evaluated on 229 SciBench chemistry problems released with ChemAgent.

Critical strengths
  1. The label-free signal is simple and carries real information. Verified answers are more accurate than unverified ones on every dataset and backbone, and the gain over a single CoT sample holds on all four backbones

  2. The paper is transparent. All prompts are given verbatim, and the appendices report many diagnostics.

Critical weaknesses
  1. The comparison with the baseline is not fair, and nothing shows that requiring the two streams to agree is what helps. Direct reasoning answers each problem once, in text. ChemHarness can try up to eight times, writing a text and a program answer each time, and then makes a final attempt that sees all earlier answers and may run code. The extra compute is never reported. The paper's own breakdown shows that the gain comes from these extra tries. When the two streams agree in the first round, ChemHarness returns exactly the direct-reasoning answer, so there is no gain on these problems (Fig. 7). All of the improvement comes from the remaining problems, where it keeps resampling. On problems that are never verified, the correct value already appeared in an earlier attempt in 61–81% of cases (Fig. 10), so success is largely a matter of sampling enough and picking well, as plain majority voting does. Even the variant that never uses agreed answers beats direct reasoning by 2–8 points (Table 3). Running code alone can also explain gains of this size. In SciBench's own results, switching from text to Python changes accuracy on these datasets by +16.5 points, yet the paper never reports its program-only answer. More attempts plus code execution may therefore account for the whole gain.

  2. The method is built upon teh idea when the text and the program answer agree, it is correct, and this premise does not hold necessarily. Both answers come from the same model, given the same problem and the same examples. If the model picks the wrong law, it can do so in both, and the two answers agree on the same wrong value. The judge does not solve this either, since it is the same model once more. The check also compares only numbers, not the reasoning behind them. A correct approach can still end in a wrong value through a single slip, two different flawed derivations can land on the same value, and intermediate numbers can match by coincidence. Agreement therefore does not certify the intermediate steps, and disagreement does not locate the error. The paper's picture of errors that propagate step by step through a solution, and later as "error cascades" through memory and tools, is not shown. The errors may just as well be scattered and random. The paper's own numbers show the problem. First, 25–37% of the answers on which both streams agree are wrong (Table 4), which contradicts the claim that such a value "is rarely wrong". Second, by my reading of Fig. 6 and Table 1, these agreed-but-wrong answers make up a third to three quarters of all of ChemHarness's wrong answers. Third, missed assumptions, exactly the error a second stream from the same model cannot catch, are the largest error clas. This also affects the self-evolution, because the agreed-but-wrong answers become the unit tests that admit new tools.

  3. The "self-evolving" part, which gives the paper its title, is not shown to matter. Table 3 has no variant without the tool library, and the accuracy of the runs without any memory is not reported. The available evidence suggests the library adds little. The net gain over direct reasoning is about the same on the first and second half of the problem sequence, although the library is four to five times larger in the second half. Programs call a selected tool on only very few of the problems (Fig. 11), and a library from another backbone changes accuracy by −2.0 to +2.5 points (Table 7).

  4. The reproduced ChemAgent scores 10–12 points below the bare backbone, whereas the ChemAgent paper reports +10 points on Qwen2.5-72B. This looks like a failed reproduction, yet major claims rests on it. The published rows of Table 1 use other models and other scoring rules, and DeepSeek-V4.1-Flash without the harness (73.7%) already beats all of them.

  5. The motivation of this work is that real chemistry lacks ground truth, yet all experiments use 229 textbook problems with known answers. Without a development set, the prompts and thresholds were presumably chosen on these labeled test problems. The prompts also contain worked demonstrations. The method is not specific to chemistry, yet it is tested on one small chemistry benchmark only. The backbones are Qwen2.5-Instruct models from 2024 and DeepSeek-V4.1-Flash, for which the paper does not say whether thinking was enabled. The motivation "without external feedback, self-correction fails" (Huang et al., 2024) refers to prompted self-correction of GPT-3.5/4. Current reasoning models self-verify.

  6. The novelty is overstated, because each component already exists. Accepting an answer only when a CoT and a program solution agree is the verification rule of the mixture-of-thought cascades of Yue et al. (2024). XoT (Liu et al., 2023), the CoT-vs-PAL selection of Zhao et al. (2023) and MathPrompter (Imani et al., 2023) work in the same way. Building a tool library online and without labels, by keeping functions whose executions agree, is TroVE (Wang et al., 2024). Label-free memory that evolves over a stream of test problems is Dynamic Cheatsheet (Suzgun et al., 2025). Harnesses that accumulate tools or memory for scientific reasoning, including on these same SciBench chemistry sets, exist as well: ChemAgent, RefTool, SciAgent (Ma et al., 2024), and test-time tool evolution for science (Lu et al., 2026). What remains new is the particular combination, and back to my point above the library-building part of this combination is not shown to contribute.

Questions for authors
  1. What is the accuracy of the first-round symbolic stream alone, and of self-consistency over CoT, programs, or both, at the same number of generations? How many tokens per problem does each method use?
  2. What is the accuracy without the tool library, and of the runs without any memory, over three seeds?
  3. How is Table 4 computed (which pairs, judge included or not, number of events)? What is the precision of cross-stream agreement without the judge, and how often does the judge overrule agreement despite its truncated inputs?
  4. How do all methods compare when re-scored with SciBench's original rule (5% on the full value), and why does the ChemAgent reproduction fall below the bare model?
  5. How were the prompts and thresholds chosen without a development set? Was thinking enabled for DeepSeek-V4.1-Flash?
8 comments
AuthorsOpenReview ↗

Response to Reviewer E76p (1/5)

Thank you for the very detailed review.

Table A. Verification against sampling more at a comparable number of tokens (Appendix H.1, Table 16; UGPhysics [1], no experience). Accuracy, tokens per problem in parentheses. Dual-stream verification samples up to 4 or 8 times per stream and stops at the first agreement; Δ compares Dual 8+8 with Text ×8.

BackboneText ×8Program ×8Dual 4+4Dual 8+8Δ
Llama-3.1-8B21.5 (8,839)17.1 (2,498)20.4 (5,217)23.2 (9,683)+1.7
Qwen2.5-7B28.0 (6,364)20.1 (2,173)26.7 (3,533)28.3 (6,206)+0.3
Qwen2.5-14B33.9 (6,815)32.3 (3,875)34.4 (3,940)35.8 (6,471)+1.9
Qwen3-8B41.4 (8,204)31.4 (2,058)40.1 (3,762)41.0 (6,131)−0.4
Qwen3.5-9B55.9 (21,618)53.8 (9,448)56.9 (11,058)58.8 (18,073)+2.9
Qwen3.6-27B60.2 (15,746)60.1 (6,816)61.0 (6,536)61.4 (9,731)+1.1
Qwen3.8-27B57.9 (17,100)59.0 (16,924)59.1 (10,212)59.8 (15,821)+1.9

Weakness 1: “The comparison with the baseline is not fair, and nothing shows that requiring the two streams to agree is what helps. …”

Reply: We agree that compute must be reported, and Tables 4, 16 and 17 now give tokens per problem. On SciBench [2], ChemHarness uses 3,842 (DeepSeek-V4.1-Flash) to 9,491 (Qwen2.5-7B) tokens per problem, against 438 and 649 for direct reasoning (Appendix H.1). We therefore compared at a comparable number of tokens (Table A):

  • Requiring agreement helps. Verification beats text self-consistency [3] for five of seven backbones and repeated Program-of-Thought [4] for all seven, with fewer tokens than text self-consistency for six.
  • Running code alone does not explain the gain. On SciBench, the first-round symbolic stream alone scores 38.8 / 51.1 / 55.1 / 70.6, against 36.1 / 52.3 / 55.8 / 73.7 for direct reasoning (Qwen2.5-7B / 14B / 32B / DeepSeek-V4.1-Flash).
  • Experience is cheaper than sampling. With it, 4 samples per stream beat 8 without it by +3.0 to +4.0, with 35–46% fewer tokens (Tables 4 and 16).
  • Problems verified in the first round return the answer of direct reasoning by design; on the remaining problems, resampling alone scores lower than verification (Table A).
  • Correct values in earlier rounds (Figure 10 of the submission) are not yet a selected answer: majority voting over the samples stays below verification for five of seven backbones (Table A).
  • The variant without cross-verified answers (Table 3) still uses verification, because its final solution reads the short-term memory, whose lemmas and discrepancies come from comparing the two streams. Its accuracy therefore does not measure the harness without cross-stream comparison.

Table B. Precision (%) of agreement on SciBench, and the judge. The first two columns cover the problems with at least two rounds (Section 4.7, Table 5), the others all problems of one run (Appendix D.4, Table 11).

BackboneTwo streams agreeTwo samples of one stream agreeTwo streams agree, without the judgeRejected by the judgePrecision, rejected / accepted
Qwen2.5-7B63.055.764.744 of 17045 / 71
Qwen2.5-14B66.746.272.915 of 19953 / 74
Qwen2.5-32B68.350.874.911 of 20764 / 76
DeepSeek-V4.1-Flash75.454.283.124 of 24371 / 84

Weakness 2: “The method is built upon teh idea when the text and the program answer agree, it is correct, and this premise does not hold necessarily. …”

Reply: The revision replaces "rarely wrong" and the "error cascade" description with the measured precision of Table B: cross-stream agreement is more precise than same-stream consistency on every backbone, also without the judge. Every final score includes the agreed-but-wrong answers. Of these, 32–64% are agreed in the first round, where ChemHarness returns the answer of direct reasoning: they are errors of the backbone that direct reasoning makes as well, not errors introduced by verification (Appendix D.4). Section 6 states this limitation.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
  • [3] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
  • [4] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR 2023.
AuthorsOpenReview ↗

Response to Reviewer E76p (2/5)

Table C. What carries the gain of self-evolution (Section 4.6, Table 4; UGPhysics [1], held-out folds). The pipeline and the symbolic-only information flow are fixed and only the experience varies. Accuracy without experience and gain, ± paired standard error:

ReaderNo experience+ random programs+ abstracted tools+ verified programs
Llama-3.1-8B20.4+0.3 ± 0.7−0.9 ± 0.7+6.6 ± 1.0
Qwen2.5-7B27.5+0.3 ± 0.6+0.6 ± 0.7+4.8 ± 0.9
Qwen2.5-14B34.1−0.2 ± 0.7+0.5 ± 0.7+4.7 ± 0.9
Qwen3-8B40.0−1.2 ± 0.7+0.0 ± 0.8+4.5 ± 0.9

Weakness 3: “The "self-evolving" part, which gives the paper its title, is not shown to matter. …”

Reply: The requested ablation shows that it matters and which part carries the gain.

  • Self-evolved experience matters when passed on as verified programs: +4.5 to +6.6, positive in every fold and in all 75 run–fold pairs of repeated runs (Appendix C.3), while random programs give −1.2 to +0.3.
  • Low tool usage does not explain the tool result. Readers select at least one tool for 57–96% of the problems, the weaker readers more often.
  • Library from another backbone (Table 12, Table 7 of the submission). Passed on as programs, experience from another backbone helps, and how much grows with how much more accurate the writer is than the reader: r = 0.89 over 31 writer–reader pairs (Appendix C.4, Table 10).

Table D. ChemAgent [2] reproduced with its released code and configuration, scored with ChemAgent's own released scoring function (Appendix I, Tables 18–19).

BackboneDirect reasoningChemAgent, 3 runsChemAgent, answered problems onlyReported by ChemAgent (direct / ChemAgent)
Llama-3.1-8B29.725.2 ± 1.825.917.4 / 25.8
Qwen2.5-7B41.926.1 ± 4.628.1–
Qwen2.5-14B54.140.5 ± 1.448.0–

Weakness 4: “The reproduced ChemAgent scores 10–12 points below the bare backbone, whereas the ChemAgent paper reports +10 points on Qwen2.5-72B. This looks like a failed reproduction, yet major claims rests on it. The published rows of Table 1 use other models and other scoring rules, and DeepSeek-V4.1-Flash without the harness (73.7%) already beats all of them.”

Reply: We ran the public release in its released configuration: retrieval model (text-embedding-ada-002), thresholds and prompts, with scoring, refinement and web search off as released, and its library built from the 43 released development questions. The method code is identical to the release. Appendix I lists every difference (model calls sent to the local backbone, two import fixes without which the release does not run, one process per question with the memory carried over), and the supplementary material contains the code, every output and a script that checks each file against the public release.

  • ChemAgent reproduces. On Llama-3.1-8B, the open backbone for which ChemAgent reports results, we obtain 25.2 against the published 25.8.
  • The difference is in direct reasoning. ChemAgent reports 17.4 for direct prompting; we measure 29.7 with a standard system prompt and 24.0 without one. The ordering in Table 1 therefore holds under ChemAgent's own evaluation.
  • Release errors do not explain it. On 7–8% (Qwen2.5-7B) and 15–16% (Qwen2.5-14B) of the test problems the release stops with an error and returns an error message, which counts as incorrect. On the answered problems alone it stays below direct reasoning (fourth column; 28.5 and 48.3 against 36.1 and 52.3 under the scorer of the paper, Table 18). All these failures are the same error in the released code, in the path it takes when no stored memory is similar enough, and the Qwen2.5-7B run whose library covers all four subjects scores 30.6.

Direct reasoning here is one zero-shot sample under ChemAgent's scorer; Table 2 uses the three demonstrations of SciBench [3] and the scorer of the paper. The upper block of Table 1 is copied from the original papers for context; those rows use other backbones and scoring, as Section 4.2 now states, and all our comparisons use the same backbone under the same evaluation.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] ChemAgent: Self-updating Memories in Large Language Models Improves Chemical Reasoning. ICLR 2025.
  • [3] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
AuthorsOpenReview ↗

Response to Reviewer E76p (3/5)

Table E. Reasoning models with thinking (Appendix H.2, Table 17; Qwen3.8-27B, UGPhysics [1]).

ConfigurationAccuracyTokens per problem
Thinking, text self-consistency ×455.433,096
Thinking, text 2 + program 2, stop at agreement55.428,514
Thinking, text 4 + program 4, stop at agreement56.947,345
No thinking, text self-consistency ×456.28,556
No thinking, text 2 + program 2, stop at agreement57.46,640
No thinking, text 4 + program 4, stop at agreement58.710,250

Weakness 5: “The motivation of this work is that real chemistry lacks ground truth, yet all experiments use 229 textbook problems with known answers. …”

Reply:

  • Current reasoning models self-verify. Without thinking, dual-stream verification is +2.0 to +3.3 above thinking-mode voting at one fifth to one third of the tokens (Table E). QCBench [2] (263 test problems, an additional result of this response) shows the same picture: 75.3 at 4,590 tokens without thinking, against 74.2 at 24,464 with thinking. Self-verification by thinking does not replace cross-paradigm verification. Thinking was disabled for DeepSeek-V4.1-Flash (Question 5).
  • Textbook problems with known answers. Labels are needed to measure accuracy, not to run the method. ChemHarness never reads a label, and the held-out experiments build all experience on unlabeled practice problems.
  • Beyond one small chemistry benchmark, and newer backbones. We add UGPhysics (1,068 test problems), whose practice pool also contains the 350 chemistry problems of QCBench, with backbones up to Qwen3.8-27B (Tables A, C and E; our reply to Jeq1, Question 5).
  • Development set. Nothing was tuned on test problems of the new benchmarks: a separate set of 200 UGPhysics problems served for development and is excluded from every result, the prompts are zero-shot, the 5% rule is reused unchanged, and every number comes from held-out folds.
  • Motivation. The revision rewrites the sentence citing Huang et al. [3] around cost: prompted self-correction does not improve reasoning without external feedback, and the self-verification of current reasoning models is expensive in tokens (Section 1).

Weakness 6: “The novelty is overstated, because each component already exists. …”

Reply: Each component has precedents. Section 2 now discusses mixture-of-thought cascades [4], XoT [5], CoT-vs-PAL model selection [6], MathPrompter [7], TroVE [8], LATM [9], Dynamic Cheatsheet [10], ChemAgent [11], RefTool [12], SciAgent [13] and test-time tool evolution [14], and states how ChemHarness differs from each. The contribution we claim is a finding supported by controlled experiments: label-free, self-evolved experience helps when it is passed on as verified whole programs and not when it is abstracted into tools. For the same writer and the same held-out problems, the tools give Llama-3.1-8B −0.9 and the verified programs +6.6 (Table C). How much it helps grows with how much more accurate the writer is than the reader, and a reader's own experience never lowers its accuracy (Appendix C.4).

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry. Journal of Chemical Information and Modeling 2025.
  • [3] Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024.
  • [4] Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning. ICLR 2024.
  • [5] Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts. EMNLP 2023.
  • [6] Automatic Model Selection with Large Language Models for Reasoning. Findings of EMNLP 2023.
  • [7] MathPrompter: Mathematical Reasoning using Large Language Models. ACL 2023 (Industry Track).
  • [8] TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks. ICML 2024.
  • [9] Large Language Models as Tool Makers. ICLR 2024.
  • [10] Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory. EACL 2026.
  • [11] ChemAgent: Self-updating Memories in Large Language Models Improves Chemical Reasoning. ICLR 2025.
  • [12] RefTool: Reference-Guided Tool Creation for Knowledge-Intensive Reasoning. ICLR 2026.
  • [13] SciAgent: Tool-augmented Language Models for Scientific Reasoning. EMNLP 2024.
  • [14] Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning. arXiv 2026.
Reviewer_Jeq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ChemHarness solves numerical chemistry problems using two samples of the same backbone: a written derivation and an executed Python program. Matching answers, together with a semantic judge, can stop the search within eight rounds. The harness turns verified solutions into reusable tools and keeps intermediate agreements and disagreements for a fallback answer. Memory is withheld from the written stream to reduce shared errors.

Critical strengths
  1. The paper introduces a dual-stream self-verification method that does not require ground-truth answers. It uses two independent streams, neural reasoning and executable symbolic reasoning, to cross-check both final answers and reasoning steps.

  2. The paper develops ChemHarness, a self-evolving chemical reasoning system that combines long-term tool memory with short-term experience memory. Experiments show consistent improvements across four open-source models and better performance than existing chemical agents such as ChemCrow and ChemAgent.

Critical weaknesses
  1. The paper does not clearly show whether the performance gains come from self-evolution or simply from repeated inference. The main comparison is between ChemHarness's final answer and the base model's first-round answer. However, ChemHarness can perform up to eight rounds of reasoning and verification, so the gains may come from additional computation rather than tool library evolution. Although Table 3 includes some ablation studies, it does not evaluate the effect of removing the evolving tool library while keeping the same reasoning and verification process.

  2. The reliability of self-verification and some evaluation results needs further examination. The neural stream, symbolic stream, and semantic judge all use the same base model. Even with memory isolation, they may still make similar reasoning errors. Table 4 shows that on harder problems, 24.6%–37.0% of the answers agreed upon by both streams are still incorrect. This suggests that agreement between the two streams does not necessarily mean the answer is correct.

Questions for authors

Q1. What is the final accuracy of ChemHarness if tool creation and retrieval are disabled, while keeping the same dual-stream reasoning, stopping rules, and fallback strategy?

Q2. If the tool library is built on one set of problems and then frozen, can it still improve performance on a separate set of test problems?

Q3. What proportion of the generated tools and stored lemmas are independently verified as correct, rather than simply matching the model-generated pseudo-labels?

Q4. Why is the Wien's Law answer in Table 8 marked as incorrect even though it falls within the stated error tolerance?

Q5. Why does the paper use Qwen2.5 models as backbones instead of newer models such as Qwen3 or Qwen3.5?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jeq1 (1/3)

Thank you for the precise questions. Questions 1 and 2 define the experiment that answers Weakness 1, and we ran it.

Table A. What carries the gain of self-evolution (Section 4.6, Table 4; UGPhysics [1], 1,068 held-out problems, 5-fold cross-validation). Dual-stream reasoning, the stopping rule and the fallback are fixed and only the cross-problem experience varies, so every variant runs the same repeated inference. Accuracy without experience and gain of each kind of experience, ± paired standard error:

ReaderNo experience+ random programs+ abstracted tools+ verified programs
Llama-3.1-8B20.4+0.3 ± 0.7−0.9 ± 0.7+6.6 ± 1.0
Qwen2.5-7B27.5+0.3 ± 0.6+0.6 ± 0.7+4.8 ± 0.9
Qwen2.5-14B34.1−0.2 ± 0.7+0.5 ± 0.7+4.7 ± 0.9
Qwen3-8B40.0−1.2 ± 0.7+0.0 ± 0.8+4.5 ± 0.9

Weakness 1: “The paper does not clearly show whether the performance gains come from self-evolution or simply from repeated inference. The main comparison is between ChemHarness's final answer and the base model's first-round answer. However, ChemHarness can perform up to eight rounds of reasoning and verification, so the gains may come from additional computation rather than tool library evolution. Although Table 3 includes some ablation studies, it does not evaluate the effect of removing the evolving tool library while keeping the same reasoning and verification process.”

Reply:

  • Random programs add the same context and inference and give no gain, so the gain comes from the content of the experience, not from extra computation, and it is carried by the verified programs rather than by their abstraction into tools.
  • Verified programs gain in all 75 run–fold pairs of repeated runs (Appendix C.3), at nearly unchanged tokens (Llama-3.1-8B: 5,183 against 5,270 per problem).
  • Repeated inference does not explain the gain over direct reasoning either: at a comparable number of tokens, verification beats text self-consistency [2] for five of seven backbones, by +1.1 to +2.9 (Appendix H.1, Table 16).

Table B. Precision (%) of agreement on SciBench [3]. The first two columns cover the problems with at least two rounds (Section 4.7, Table 5), the last all problems (Appendix D.4, Table 11).

BackboneTwo streams agreeTwo samples of one stream agreeTwo streams agree, without the judge
Qwen2.5-7B63.055.764.7
Qwen2.5-14B66.746.272.9
Qwen2.5-32B68.350.874.9
DeepSeek-V4.1-Flash75.454.283.1

Weakness 2: “The reliability of self-verification and some evaluation results needs further examination. The neural stream, symbolic stream, and semantic judge all use the same base model. Even with memory isolation, they may still make similar reasoning errors. Table 4 shows that on harder problems, 24.6%–37.0% of the answers agreed upon by both streams are still incorrect. This suggests that agreement between the two streams does not necessarily mean the answer is correct.”

Reply: We agree, and the revision replaces "rarely wrong" with measured precision. Agreement is a stopping signal, not a label, and every score counts agreed-but-wrong answers.

  • Cross-stream agreement is more precise than same-stream consistency on every backbone, and the judge is not the source of this precision.
  • Experience makes agreement more precise: on UGPhysics, the experience of Qwen3.6-27B raises it by 3.8 to 5.1 points for all four readers (Appendix C.4).

Section 6 states the remaining failure mode: both streams and the judge can share an incorrect assumption.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
  • [3] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
AuthorsOpenReview ↗

Response to Reviewer Jeq1 (2/3)

Table C. Frozen experience over repeated runs (Appendix C.3, Table 9; UGPhysics [1], held-out folds).

ReaderRunsNo experienceFrozen verified programsGainFolds with gain
Llama-3.1-8B421.0 ± 0.527.6 ± 0.7+6.6 ± 0.420/20
Qwen2.5-7B426.9 ± 0.431.4 ± 1.0+4.5 ± 0.620/20
Qwen2.5-14B434.7 ± 0.639.1 ± 0.3+4.4 ± 0.420/20
Qwen3-8B339.1 ± 0.944.3 ± 0.3+5.1 ± 0.615/15

Question 1: “What is the final accuracy of ChemHarness if tool creation and retrieval are disabled, while keeping the same dual-stream reasoning, stopping rules, and fallback strategy?”

Reply: On UGPhysics it is the "No experience" column of Tables A and C. On SciBench [2], over three runs (Appendix C), it is 52.0 / 65.3 / 68.5 / 78.2 (Qwen2.5-7B / 14B / 32B / DeepSeek-V4.1-Flash), against 48.8 / 63.3 / 67.4 / 80.1 with the tool library (Table 2).

Question 2: “If the tool library is built on one set of problems and then frozen, can it still improve performance on a separate set of test problems?”

Reply: Yes for the verified programs the library is built from, and no for the abstracted tools. This is the protocol of all new experiments: experience is built only from the practice problems, frozen, and evaluated on the held-out fold, with test problems that overlap the practice pool removed. Frozen verified programs gain in every fold of every run (Table C); frozen tools give −0.9 to +0.6 (Table A).

Table D. Correctness of the experience against gold answers, checked after building; gold answers are never used during building.

ExperienceCorrect against goldReference
Stored solutions, written by Qwen3.6-27B70%writer's own accuracy 61.1%
Stored solutions, written by Qwen2.5-32B54%writer's own accuracy 39.4%
Verified cases behind the 890 active tools of UGPhysics75%–
Lemmas (no gold intermediate values exist; counterfactual replay on SciBench, Appendix C.6)80 wrong answers turned correct31 correct answers turned wrong

Question 3: “What proportion of the generated tools and stored lemmas are independently verified as correct, rather than simply matching the model-generated pseudo-labels?”

Reply: Table D checks each kind of experience against gold answers only after building, so the check does not rely on pseudo-labels. Stored solutions are more accurate than their writer, which is why a more accurate writer helps more (Appendix C.4). Tools and programs of similar correctness differ by 4–7 points in effect (Table A), so what transfers is the verified solution path.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
AuthorsOpenReview ↗

Response to Reviewer Jeq1 (3/3)

Table E. Gain of ChemHarness over direct reasoning under three scoring rules (Appendix B.3, Table 6; one run per backbone).

Scoring ruleQwen2.5-7BQwen2.5-14BQwen2.5-32BDeepSeek-V4.1-Flash
Scorer of the paper+10.9+13.9+12.9+7.2
SciBench's original 5% rule+10.1+12.6+9.3+6.2
5% rule, units with a power of ten converted+12.4+16.3+13.3+8.2

Question 4: “Why is the Wien's Law answer in Table 8 marked as incorrect even though it falls within the stated error tolerance?”

Reply: Thank you for catching this; you are right. It is a scoring error in the unit's power of ten, not a wrong answer: the gold value is stored as 1.01 in units of 10⁻⁵ m, and the scorer rejected 9.993 × 10⁻⁶ m. We re-checked all 229 problems (35 carry a power of ten in the unit), converted them and re-scored with SciBench's original 5% rule; every gain grows (last row). Table 13 now shows a genuine calculation error from a re-run of the error attribution over all 1,257 wrong answers of eight runs (Appendix F), in which ChemHarness makes 24% fewer wrong answers than direct reasoning.

Table F. Newer backbones in the revision (UGPhysics [1]).

BackboneResultSource
Qwen3-8B+4.5 from verified programs, positive in every fold; +5.1 ± 0.6 over three runsTables 4, 9
Qwen3.5-9Bverification beats text self-consistency at a comparable number of tokens by +2.9Table 16
Qwen3.6-27Bthe same comparison: +1.1Table 16
Qwen3.8-27Bthe same comparison: +1.9; with and without thinkingTables 16, 17
Qwen3.8-27B as reader+2.5 ± 0.6 from the experience of Qwen3.6-27B and +2.6 ± 0.6 from its own, positive in every foldTable 10

Question 5: “Why does the paper use Qwen2.5 models as backbones instead of newer models such as Qwen3 or Qwen3.5?”

Reply: We chose Qwen2.5 to compare with ChemAgent [2] on the same backbones, and the revision adds the newer backbones of Table F.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] ChemAgent: Self-updating Memories in Large Language Models Improves Chemical Reasoning. ICLR 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

General response: does self-evolution matter, and is the gain more than extra inference?

We thank the three reviewers. All three ask whether the self-evolving part matters or the gain is extra inference (Jeq1 Questions 1–2, E76p Weakness 3 and Question 2, 9u2f Question 1). We ran the requested ablation on held-out data.

Setup (Appendix B.4). UGPhysics [1], 1,068 test problems, 5-fold cross-validation. Experience is built without labels on practice problems (the other four folds and the 350 chemistry problems of QCBench [2]), frozen, and tested on the held-out fold. Every variant runs the same dual-stream pipeline (up to 4 samples per stream, stop at agreement within 5%), and experience reaches only the symbolic stream, as in the submission. Qwen3.6-27B writes the experience and four backbones read it. A separate development set of 200 problems is excluded from every result.

Table R1 (Section 4.6, Table 4). Gain over the same pipeline without experience; ± is the paired standard error over problems.

ReaderNo experience+ abstracted tools+ verified programs
Llama-3.1-8B20.4−0.9 ± 0.7+6.6 ± 1.0
Qwen2.5-7B27.5+0.6 ± 0.7+4.8 ± 0.9
Qwen2.5-14B34.1+0.5 ± 0.7+4.7 ± 0.9
Qwen3-8B40.0+0.0 ± 0.8+4.5 ± 0.9
  1. Label-free experience transfers. Verified programs add +4.5 to +6.6 for every reader, positive in all 75 run–fold pairs of repeated runs (Appendix C.3, Table 9), at nearly unchanged token cost.
  2. The content carries the gain. Random programs from the same pool, i.e., the same extra context and inference, give −1.2 to +0.3.
  3. The ablation locates the gain. Abstracting the same solutions into tools gives −0.9 to +0.6, although readers retrieve a tool for 57–96% of the problems.
  4. At a comparable number of tokens, verification beats sampling more (Appendix H.1, Table 16). It is ahead of text self-consistency [3] for five of seven backbones, by +1.1 to +2.9. With experience, 4 samples per stream beat 8 without it by +3.0 to +4.0 at 35–46% fewer tokens.
  5. Reasoning models do not make verification redundant (Appendix H.2, Table 17). Qwen3.8-27B with dual-stream verification and no thinking scores 58.7 at 10,250 tokens per problem, against 55.4 at 33,096 for thinking-mode self-consistency.
  6. The writer matters (Appendix C.4, Table 10). Over 31 writer–reader pairs, the gain grows with how much more accurate the writer is than the reader (r = 0.89), and a reader's own experience never lowers its accuracy.

Revisions (in blue).

  • Abstract, Sections 1 and 4.6: the held-out results; the precision claim stated against same-stream consistency; the ChemAgent [4] comparison on the same backbones.
  • Section 2: SciAgent [5], Test-Time Tool Evolution [6], Dynamic Cheatsheet [7], XoT [8], automatic model selection [9] and MathPrompter [10].
  • Section 3.2: the long-term memory as two layers.
  • Section 4.2: the published rows of Table 1 use other backbones and scorers.
  • Section 6: the shared-assumption failure mode.
  • Appendices:
    • B.3 re-scoring under three rules; B.4 the held-out protocol;
    • C SciBench [11] without the tool library; C.3 repeated runs; C.4 eight writers × eight readers (Table 10, Figure 5);
    • D.4 agreement events and the judge;
    • F error attribution over 1,257 wrong answers;
    • H.1 comparison at a comparable budget, with tokens; H.2 thinking;
    • I the ChemAgent reproduction, with code in the supplementary material.
  • Figure 1 redrawn to follow one problem from the input to the submitted answer, with the criterion of the judge (Equation 1) and the paths of both memories.
  • All figures redrawn at print size.

References

  • [1] UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML 2025.
  • [2] QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry. Journal of Chemical Information and Modeling 2025.
  • [3] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
  • [4] ChemAgent: Self-updating Memories in Large Language Models Improves Chemical Reasoning. ICLR 2025.
  • [5] SciAgent: Tool-augmented Language Models for Scientific Reasoning. EMNLP 2024.
  • [6] Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning. arXiv 2026.
  • [7] Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory. EACL 2026.
  • [8] Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts. EMNLP 2023.
  • [9] Automatic Model Selection with Large Language Models for Reasoning. Findings of EMNLP 2023.
  • [10] MathPrompter: Mathematical Reasoning using Large Language Models. ACL 2023 (Industry Track).
  • [11] SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. ICML 2024.
AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chairs and Reviewers,

We thank the reviewers for their time and constructive feedback. To facilitate the AC's assessment, we summarize the strengths the reviewers recognized, how we addressed their concerns, and the one point that remains.

Strengths recognized by the reviewers

  • A label-free signal that works. “The label-free signal is simple and carries real information” (E76p); “a practical way to obtain feedback without ground-truth answers” (9u2f); “a dual-stream self-verification method that does not require ground-truth answers” (Jeq1).
  • Consistent gains. “consistent improvements across four open-source models and better performance than existing chemical agents such as ChemCrow and ChemAgent” (Jeq1).
  • Transparency and thorough diagnostics. “The paper is transparent. All prompts are given verbatim, and the appendices report many diagnostics” (E76p). During the discussion we further released the ChemAgent reproduction code, so the baseline numbers can be checked independently.

Main concerns and how we addressed them

  • Does the gain come from self-evolution or from more inference? (Jeq1, 9u2f, E76p) In held-out folds with frozen experience, verified programs raise the accuracy of every reader by 4.5 to 6.6 points, and the gain is positive in all 75 run–fold pairs; randomly drawn programs do not help (Table 4). A memory written on UGPhysics and QCBench, then frozen, also raises SciBench accuracy by 7.2 and 11.4 points.
  • Is it better than sampling more? (9u2f, E76p) At a comparable number of tokens, verification beats text self-consistency for five of seven backbones, and with experience 4 samples per stream beat 8 without it at 35–46% fewer tokens (Tables 16, 17). Against a vote over a matched mixture of text and program samples, it matches the vote at 8–42% fewer tokens and exceeds it with a cap of 8 samples per stream.
  • Does verification itself matter? (E76p) Verification raises the share of correct programs in the memory from 59.7% to 71.3% for Qwen3.6-27B, and from 15.0% to 47.6% for Llama-3.1-8B. On SciBench, the verified memory beats the unfiltered one for Qwen2.5-7B and 14B.
  • What if both streams share a wrong assumption? (9u2f) Agreement across streams is more precise than agreement within one stream on every backbone, 63.0–75.4% against 46.2–55.7%, also without the judge (Table 5); Section 6 discusses the remaining shared errors.
  • Reproducibility and scoring (Jeq1, E76p). We released the ChemAgent reproduction code and re-scored all 229 problems under three scoring rules; every gain grows (Table 6).

Reviewer 9u2f states that “the authors have addressed my main concerns” and raised the rating.

The remaining point: novelty (9u2f, E76p)

We agree that each component has precedents, and Section 2 discusses each of them. Two things are new:

  1. Agreement between two reasoning paradigms of one model serves as the label-free rule that decides what enters a self-evolving memory for scientific reasoning, and it is measurably more precise than agreement within one paradigm.
  2. A controlled finding: with the writer, the problems and the pipeline held fixed, self-evolved experience helps when it is verified and kept as whole programs, and not when it is abstracted into tools or drawn at random.

We sincerely thank all reviewers for an engaged discussion that substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.