acceptodds
Under review as a conference paper at ICLR 2027

Distillation, Self-Training, and the Fragility of Learned Chemical Reasoning

Abstract

Building specialized models for scientific reasoning presents a fundamental tradeoff: frontier models provide strong domain capabilities, but continued reliance on them for supervision limits the accessibility and adaptability of smaller open-weight models. We investigate whether frontier supervision can instead serve as a one-time bootstrap, using forward reaction prediction, a core task in computer-aided synthesis planning, as a testbed. We introduce Resonance, a post-trained model that distills frontier-model reasoning into a smaller student model and then removes the teacher while continuing to train on the student's own verified and preference-ranked generations. Resonance transforms a base model with near-zero accuracy into an effective reaction predictor. Through reasoning perturbations and counterfactual prompting analyses, we provide additional insights into the chemical reasoning acquired by trained student models. Together, our results demonstrate the effectiveness of our pipeline in instilling reaction prediction competencies while also uncovering systematic pathologies in our model's ability to reason over chemical contexts.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
chemical reasoning, forward reaction prediction, self-training, knowledge distillation, rejection-sampling fine-tuning, preference optimization, open-weight language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FVMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Resonance, a family of open-weight reasoning models (8B and 27B) for forward reaction prediction on USPTO-50k. The pipeline has three stages: (S1) supervised fine-tuning on reasoning traces generated by a frontier teacher (Kimi K3) with the product visible; (S2) rejection-sampling SFT on the student's own verified-correct traces; (S3) DPO on self-generated preference pairs (correct vs. most-similar incorrect trace), with an auxiliary NLL term on the chosen response. The main empirical findings are: (i) teacher SFT is essential for the 8B model, and self-training adds 11.8 points without teacher supervision (31.0 → 42.8); (ii) the stronger 27B model surpasses its teacher after S1 alone (79.0%) and gains nothing measurable from S2/S3; (iii) the 8B student inherits the teacher corpus's single-pass reasoning style, and prompting-based test-time interventions (think more, consider alternatives) do not help; (iv) the student is sycophantic under negative feedback, abandoning 22–32% of initially correct answers, with the loss growing after DPO; (v) self-training mainly sharpens answers the model could already reach, as shown by plurality@k/pass@k analysis.

Critical strengths

1.Clean, well-controlled pipeline and ablation. The three-stage decomposition is clearly specified, and the paper reports skip-S2 ablations, per-stage McNemar tests, per-class breakdowns, and a large appendix with training configurations, prompts, and sampling settings. This is unusually thorough for a domain-adaptation paper.

2.The "stronger student needs less of the pipeline" finding is genuinely useful. Showing that the same teacher traces take 8B from 1.2% → 31.0% but 27B from 26.3% → 79.0% (above the teacher) is a crisp, actionable result. The diagnostic—measure the student's solve rate on the training set after S1 to predict whether self-training will help—is a practical contribution.

Critical weaknesses

1.Novelty is limited at the method level. Distillation + rejection-sampling SFT + DPO is a standard recipe (STaR, RFT, iterative DPO). The paper's contribution is primarily empirical and domain-specific rather than methodological. For a top venue, the bar is that the findings justify the lack of methodological novelty—and here the findings are solid but not surprising: that self-training helps a weak student more than a strong one, and that self-training sharpens rather than expands the distribution, are both consistent with prior work (Huang et al., 2024; Yue et al., 2026). The paper cites these, which is good, but it means the headline results are largely confirmatory.

2.The central comparison with frontier models is confounded. Resonance is trained on USPTO-50k train; Claude Opus 5 and GPT-6-Astra are evaluated zero-shot. The paper acknowledges this, but it still frames the 8B model as "lagging the teacher" and the 27B as "comparable to Opus 5." Those framings are not supported by the experimental design. A fair comparison would require either fine-tuning the frontier models (infeasible) or evaluating the students zero-shot on held-out chemistry. As written, the comparison measures "what task-specific post-training adds," which is a weaker claim than the paper sometimes implies.

3.The teacher's low scores on reductions/oxidations/deprotections are a dataset artifact, not a chemistry result. The paper is aware of this (Appendix C.2), and the analysis is careful. But the main text uses these classes to argue that "students and teacher understand different chemistry" (Section 4.2), when the more accurate statement is that the teacher refuses to infer a missing reagent and the student has learned the dataset convention of guessing. This is a dataset-convention finding, not a chemistry-reasoning finding. The framing in the abstract and Section 4.2 should be tightened.

Questions for authors

1.Teacher-trace visibility. The teacher sees the recorded product when generating S1 traces. How much of the S1 gain is due to the trace content versus simply learning the output format and task distribution? The few-shot control (Appendix B.1) shows format alone is insufficient, but it does not separate "trace teaches chemistry" from "trace teaches the dataset's answer conventions." Could you train S1 on answer-only targets (no thinking) and compare?

2.DPO fragility. Is the increased feedback sensitivity at S3 specific to DPO, or would it also appear with rejection-sampling SFT alone? A comparison of S2 vs. S3 flip rates under "told wrong" (Table 16) shows 21.5% vs. 32.3% C→I, so DPO does seem to increase it. But is this because DPO sharpens the policy toward correct answers (so there are more correct answers to lose), or because it specifically increases compliance? The paper reports the loss as a share of baseline (22% at S3 vs. 11% at S2), which controls for the first explanation. So the answer seems to be "yes, DPO specifically increases fragility." It would be worth stating this more explicitly and, if possible, testing DPO with a larger λ to see if the NLL term mitigates it.

3.Generalization beyond USPTO-50k. The paper evaluates a single task and benchmark. Given that the main practical claim is "measure the S1 solve rate to decide whether to self-train," how portable is that diagnostic? Does it hold on a different reaction dataset?

2 comments

Distillation + rejection-sampling SFT + DPO is a standard recipe [...] the headline results are largely confirmatory.

We agree the self-training findings match prior work. The rebuttal experiments add findings that do not:

  • At 8B, the paper's answer-shown traces lose to plain answer supervision (31.0 vs 50.6).
  • Answer-free traces, especially the think block, beat the paper's traces and transfer best to an unseen class.
  • Only think-block students inherit the teacher's backtracking (35–90% of answers vs about 1%).
  • Answer only models abandon correct answers under pushback and, at 27B, lose general knowledge.

How much of the S1 gain is due to the trace content versus [...] the output format? [...] Could you train S1 on answer-only targets?

S1, same 32,277 reactions, USPTO-50k / MIT / ORD:

S1 target8B27B
answer-shown write-up (paper)29.3 / 21.1 / 12.877.5 / 56.5 / 47.1
answer-hidden write-up36.7 / 25.9 / 17.178.3 / 58.4 / 44.6
answer-hidden think41.4 / 30.8 / 21.975.7 / 61.0 / 49.7
answer only49.4 / 34.6 / 26.178.2 / 53.6 / 41.4*

*trained on all 40,000.

  • If traces taught only format, all trace targets would score alike. At 8B they differ by up to 12 points (p ≤ 4e-26).
  • Held-out C–C (27B, USPTO-50k): think block 69.2, paper's traces 60.4, answer only 50.4 (reply to 6v8y).
  • All 40,000 reactions: 8B answer only beats the paper's traces by about 20 points; after S3, 27B answer only is +1.0 in distribution (p = 0.08) and −3.5 on MIT and ORD.
  • Answer only abandons 65% of correct answers when told wrong (8B; S1–S3: 33–45%) and falls to 49.0 on MMLU at 27B (traces: 80.3).

The central comparison with frontier models is confounded.

Last column: 4,281 C–C training reactions held out of student training.

modelUSPTO-50kMITORDheld-out C–C
GPT-6-Astra, zero-shot63.059.072.888.2
Opus 5, zero-shot78.872.769.786.3
Kimi-K3 (teacher)57.455.755.881.2
27B Resonance S379.657.845.068.7*

*Best 27B student without C–C (answer-hidden think S1); 69.2 on the 536 C–C test reactions (reply to 6v8y).

  • Opus 5 matches 27B S3 only on USPTO-50k (p = 0.2) and leads elsewhere by 15–25 points.
  • GPT-6-Astra declines to guess missing reagents (2.2% on MIT's no-reagent rows). With reagents given it ties Opus on MIT (78.0 vs 78.1) and leads it on ORD (74.6 vs 71.1, p = 1e-5).
  • Held-out C–C reactions are from USPTO and may be in frontier pretraining data.

The teacher's low scores on reductions/oxidations/deprotections are a dataset artifact

  • Few of these prompts list a reagent (7 of 457 reductions, 0 of 55 oxidations).
  • No-reagent rows (1,439): teacher answers "no reaction" 85% (8.5%), GPT-6-Astra 96% (2.1%); trained 27B models 63–68%.
  • On reactions that list a reagent: Opus 5 86.0, 27B S3 84.3, teacher 77.2.
  • Deprotections separate chemistry from refusal: with the reagent given the teacher scores 82.8 (27B S1: 85.2); without it, when it commits, 72.7% (73.6%).

Is the increased feedback sensitivity at S3 specific to DPO [...] testing DPO with a larger λ

  • 8B "told wrong" loss vs the neutral-review turn, as a share of neutral-review accuracy (dpvj table): 23% S1, 23% S2, 35% S3.
  • Retraining S3 from the same S2 with λ = 0, 0.5, 1, 2: every S3 loses 15–17 points from its first answer; S2 loses 5.1 on that measure (9.3 vs neutral review).
  • A larger λ only raises think-block closure (81% → 86%); accuracy stays within 1 point.
  • Reruns of the published λ = 0.2 lose 15.6 and 16.1 (published 9.6): the paper underestimates S3 fragility.

..."measure the S1 solve rate to decide whether to self-train," how portable is that diagnostic?

  • Across 10 chains (8B and 27B; paper's traces, think block, answer only, LoRA recipe; with and without C–C), a higher S1 training solve rate predicts a smaller S1 → S3 gain (Spearman ρ = −0.79, p = 0.007).
  • Chains solving 17–39% gain 9–13 points on USPTO-50k.
  • 27B trace chains (85–86% solved) gain under 1 point. The one exception is 27B answer only: 96% solved, yet +2.4 (all at S2).
  • MIT and ORD gains follow the same order: 8B trace chains +4.6 to +9.8, 8B answer only +1.3 to +2.4, 27B −0.5 to +1.0.

As written, the comparison measures "what task-specific post-training adds"...

If accepted, the camera-ready version will:

  1. Drop "lagging the teacher" and "comparable to Opus 5" and limit parity to USPTO-50k.
  2. Describe the reduction/oxidation result as the dataset's missing-reagent convention, and report rows with and without a reagent separately.
  3. State that DPO, not rejection-sampling SFT, increases fragility, and correct the published S3 feedback numbers.
  4. Present the solve-rate diagnostic per training run and note it is untested on a second training dataset.

[1] Kearnes, S. M., et al. (2021). The Open Reaction Database. JACS, 143(45), 18820–18826.

[2] Jin, W., Coley, C. W., Barzilay, R., & Jaakkola, T. (2017). Predicting organic reaction outcomes with Weisfeiler-Lehman network. NeurIPS 30, 2604–2613.

Reviewer_dpvjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents Resonance, a family of models for forward reaction prediction trained through three stages: product-conditioned teacher-trace distillation, rejection-sampling fine-tuning, and preference optimization. Experiments on USPTO-50k compare two student models and examine how successive training stages affect accuracy and reasoning behavior. Self-training substantially improves the weaker 8B student but provides little additional benefit to the stronger 27B student. Analyses of the 8B model further reveal that prompting for longer reasoning does not improve accuracy, while uninformative negative feedback frequently causes correct answers to become incorrect.

Critical strengths
  1. Detailed stage and robustness analyses go beyond a final-accuracy result: skip-S2 ablations, answer transitions, repeated sampling, and uninformative negative feedback reveal how learned behavior can fail. This is the strongest part of the paper’s scientific contribution.
  2. The appendix reports scorer sensitivity, recipe differences, test-informed development, and single training runs transparently. Paired and reaction-clustered statistics make the reported diagnostic comparisons easier to assess.
Critical weaknesses
  1. The model and methodological contributions are limited, and the contribution of reasoning distillation remains insufficiently isolated. The models are trained and evaluated only on USPTO-50k, without assessing out-of-domain generalization or changes in general-task performance after training. The demonstrated model contribution is therefore largely dataset-specific adaptation. The pipeline also combines three established techniques, offering limited methodological novelty.

    More importantly, the teacher generates explanations with access to the recorded product, yet the paper lacks a matched answer-only SFT baseline on the same 40,000 training reactions. Consequently, the results do not establish how much of the improvement comes from learning reasoning traces, rather than task adaptation, product supervision, or more concise outputs. This is especially relevant to the claim in Section 4.4 that some benefits arise from the traces rather than the products alone. Missing reaction conditions further complicate interpretation: some gains may reflect learning dataset conventions rather than stronger chemical reasoning. A matched-data, comparable-budget answer-only baseline would help resolve this attribution question. I recommend positioning the analytical findings as the primary contribution and aligning the broader claims with the available evidence.

  2. The presentation underemphasizes essential analytical evidence. Most main-text figures have low information density and occupy substantial space, while results necessary to interpret the central findings remain in the appendix. Consolidating related figures would create room for these results and the negative-feedback answer-transition statistics in the main text.

  3. The Conclusion should also be shortened: it currently repeats substantial material and numerous numerical results from earlier sections, rather than synthesizing the central findings, their scope, and their implications.

  4. The claimed S2-to-S3 improvement depends strongly on answer extraction. Under the main scorer, the 8B model gains approximately 3.9 percentage points; under strict extraction of answers after the closed thinking block, accuracy increases only from 34.5% to 34.9%, with a 95% confidence interval of [−1.0, 1.8] percentage points and p = .58. This does not invalidate the main scorer, but it changes the interpretation of the claim that every stage yields a significant improvement.

  5. The conclusion about when stronger students benefit from self-training is complicated by differences in the 8B and 27B training recipes, including full-parameter versus LoRA training, epochs, loss masks, and data construction. Appendix B.3 also reports test-set evaluation before the final recipe was fixed. These factors limit attribution to the starting model’s capabilities and weaken the independence of the final evaluation. Moreover, a nonsignificant gain from a single training run does not establish equivalence. The conclusion should be restricted to the tested configurations unless supported by matched comparisons and evaluation data untouched by recipe selection.

Questions for authors
  1. How much of the S1 improvement remains under answer-only SFT using the same base model, training reactions, and a comparable training budget? Does this baseline exhibit similar sensitivity to negative feedback?
  2. What explains the difference between the main and strict answer-extraction results, particularly for S2 versus S3? Can the authors separately report format validity and product correctness to clarify what DPO improves?
2 comments

The pipeline also combines three established techniques, offering limited methodological novelty.

Our main contribution is the controlled analysis this method enables: how traces are generated (answer shown or hidden, write-up or think block) decides what the student learns, and what answer-only training costs in general knowledge.

...lacks a matched answer-only SFT baseline

We trained Qwen3-8B and Qwen3.8-27B on answer only (SMILES in, SMILES out) for the same 40,000 reactions.

  • 8B: beats Resonance S1 by about 20 points (three seeds: 50.4 ± 0.4 SD).
  • 27B, after S3: within 1 point of Resonance S3 in distribution (80.6 vs 79.6, p = 0.08). Resonance S3 leads by 3.5 on both USPTO-MIT and ORD (p ≤ 2e-5).
answer onlyUSPTO-50kUSPTO-MITORD
8B S1 → S2 → S350.6 → 56.3 → 56.835.3 → 38.7 → 37.727.8 → 29.3 → 29.1
27B S1 → S2 → S378.2 → 80.5 → 80.653.6 → 54.2 → 54.341.4 → 41.6 → 41.5

Does this baseline exhibit similar sensitivity to negative feedback?

All 5,000 test reactions; a second turn says "Your answer above is incorrect" regardless of correctness. 8B:

"told wrong"S1S2S3answer only
first-answer accuracy30.941.544.150.6
correct answers abandoned35%33%45%65%
accuracy change−7.0−9.3−15.3−19.1

Change vs the neutral-review turn (from the first answer: S2 −5.1, answer only −30.1). First answers use the feedback protocol's sampling (T 0.6, 8,192 tokens), so they differ from benchmark accuracy by ≤ 2.6.

  • Under neutral review alone, answer only abandons 24% of correct answers (S1–S3: 4–5%).
  • 27B answer only: 78.2 → 14.2 when told wrong, −31.4 even when told correct; 27B trace models lose 2.5–12.4.

...without assessing out-of-domain generalization [...] or changes in general-task performance

We added ORD [1] and USPTO-MIT [2] (deduplicated), and held C–C bond formation out of training.

test set, paper's traces8B, S1 → S2 → S3
USPTO-MIT22.0 → 29.8 → 31.8
ORD12.3 → 18.5 → 18.6
USPTO-50k C–C, seen in training34.9 → 38.4 → 40.5
USPTO-50k C–C, held out of training15.7 → 18.7 → 20.1

Think-block traces transfer better to held-out C–C (30.8 at 8B S3; reply to 6v8y).

MMLU (chat):

modelbaseS1–S3answer only
8B72.171.4–71.672.9
27B84.980.349.0
  • 8B: unchanged. 27B: traces cost about 5 points, answer only about 36.

...differences in the 8B and 27B training recipes...

We retrained 8B with the 27B LoRA recipe from the same S1.

8BS1 → S2 → S3
published (full SFT + DPO)31.0 → 38.9 → 42.8
27B LoRA recipe31.0 → 39.5 → 39.9
  • S2 gains under both recipes; S3 depends on the recipe (27B LoRA: 79.3 → 79.6).

... a nonsignificant gain from a single training run does not establish equivalence...

Two one-sided tests (TOST) on paired per-reaction differences, ±1-point margin. ✓ = 90% CI inside ±1.

A − BUSPTO-50kUSPTO-MITORD
27B S3 − S2+0.26 [−0.14, +0.66] ✓−0.46 [−0.86, −0.06] ✓+0.51 [−0.15, +1.17]
27B S3 − S1+0.60 [−0.00, +1.20]+0.88 [+0.30, +1.46]+1.02 [+0.10, +1.93]
27B answer-hidden think S3 − S2+0.06 [−0.55, +0.67] ✓−0.54 [−1.16, +0.08]+1.16 [+0.16, +2.15]
8B LoRA S3 − S2+0.42 [−0.57, +1.41]+0.78 [−0.07, +1.63]+0.32 [−0.79, +1.43]
  • 27B: S3 adds nothing beyond S2 (answer only too: +0.1 / 0.0 / −0.1); the full chain adds at most about 1 point.
  • Seed reruns of 8B S1–S3 differ by ≤ 1.0 point (p ≥ 0.15).
  • 27B answer-hidden think S1, three seeds: 75.5 ± 0.1 / 61.3 ± 0.3 / 48.9 ± 0.8 SD; its lead over the matched paper's traces is +4.8 on MIT and +1.8 on ORD.

The claimed S2-to-S3 improvement depends strongly on answer extraction.

  • DPO lowers 8B think-block closure (89.5% → 81.5%), which the strict scorer counts as wrong.
  • Where both models close the block (3,650 reactions), S3 gains +4.3 (p = 4e-8); on USPTO-MIT it gains +1.9 under the strict scorer (p = 0.0015).

I recommend positioning the analytical findings as the primary contribution...

If accepted, the camera-ready version will:

  1. Center how each target's effect depends on model size and test distribution, and the accuracy–knowledge trade-off.
  2. Merge figures and move key appendix results, including negative feedback, into the main text.
  3. Report format validity separately and limit "every stage helps" to 8B under the main scorer.
  4. Cut the Conclusion to three findings (one task, USPTO training data, 8B and 27B): trace generation decides what transfers; answer only is strong in distribution but loses stability and, at 27B, general knowledge; self-training helps in proportion to what S1 still fails.

[1] Kearnes, S. M., et al. (2021). The Open Reaction Database. JACS, 143(45), 18820–18826.

[2] Jin, W., Coley, C. W., Barzilay, R., & Jaakkola, T. (2017). Predicting organic reaction outcomes with Weisfeiler-Lehman network. NeurIPS 30, 2604–2613.

Reviewer_6v8yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses forward reaction prediction on the USPTO-50k dataset as a testbed to investigate how teacher distillation, rejection sampling fine-tuning, and DPO influence the accuracy and reasoning behavior of student models. The results indicate that subsequent self-training significantly improves an 8B-parameter student model, whereas the gains for a more capable 27B-parameter model are limited. The authors further analyze whether improvements in accuracy are accompanied by more robust reasoning by employing techniques such as reasoning-length prompting, feedback intervention, and repeated sampling.

Critical strengths

The paper provides a comprehensive, stage-by-stage breakdown of results, including comparisons that omit intermediate phases, which helps clarify the actual contributions of each training step. Rather than focusing solely on final accuracy, the study examines feedback sensitivity and reasoning behavior, revealing a potential gap between performance gains and robustness. Furthermore, the authors transparently report negative results and significant limitations, such as the marginal gains for stronger models, the absence of specific data conditions, sensitivity to scoring criteria, and the inherent uncertainty of single-run training.

Critical weaknesses
  • There is a critical confusion in the paper’s interpretation of reasoning distillation: the teacher model, when generating training explanations, has already been provided with the ground-truth product and is explicitly tasked with producing concise, structured reaction descriptions. This setup primarily requires the model to rationalize a known outcome rather than conducting real reasoning. Under this setting, it is natural to observe that student model rarely backtracks, as the training data itself rarely demands search or error-correction processes. Thus, current results are insufficient to determine what reasoning capabilities, if any, are actually distilled, or why such capabilities might be fragile.
  • A more fundamental issue is that the evaluation inputs often lack critical information required to determine the reaction outcome. The appendix reveals that no reagents are specified for 1,439 out of 5,000 test inputs. In such cases, the substrate alone is typically insufficient to uniquely determine the recorded product. The trained model might achieve higher accuracy simply by learning common transformations or dataset-specific recording conventions, whereas a zero-shot model might be penalized for refusing to make assumptions about missing reagents.
  • This paper lacks experiments to test whether the acquired knowledge generalizes beyond the training distribution. The current results are derived from the standard split of the USPTO-50k dataset; since all major reaction categories listed in Table 2 appear in both the training and test sets, the category-specific accuracy metrics cannot determine whether the models can generalize to reaction types not encountered during the adaptation phase. Given that the paper focuses on a single task, the authors should, at the very least, provide some analysis concerning the model's generalization capabilities.
Questions for authors

In the rebuttal phase, I would like to see distilled results on 'real' reasoning trajectories, and the performance of out-of-distribution reaction categories. Please refer to the weakness part for details.

4 comments

The traces rationalize a known outcome rather than conducting real reasoning.

We regenerated the teacher corpus with the product hidden.

  • Kimi-K3 solved 32,715 / 40,000 training reactions (81.8%) within four attempts.

  • We kept a trace only when the teacher's predicted product was correct.

  • From each correct run we trained on the think block (answer-hidden think, 32,277 reactions that fit 2,048 tokens) or the five-section write-up (answer-hidden write-up).

  • Baselines: the paper's traces (answer-shown write-up) and answer only. New out-of-distribution test sets:

  • ORD [1], patent-sourced reactions removed

  • USPTO-MIT [2], deduplicated against training by canonical SMILES

8B, S1 → S2 → S3USPTO-50kUSPTO-MITORD
answer-hidden think (32,277)41.4 → 47.5 → 52.630.8 → 35.6 → 39.921.9 → 23.4 → 26.5
answer-shown write-up (40,000)31.0 → 38.9 → 42.822.0 → 29.8 → 31.812.3 → 18.5 → 18.6
answer only (40,000)50.6 → 56.3 → 56.835.3 → 38.7 → 37.727.8 → 29.3 → 29.1

Bold: best S3 in each column.

  • At 8B, the think block beats the paper's traces by about 10 points everywhere. Answer only is still best in distribution and on ORD.
  • At 27B, the think block is weakest in distribution but leads on USPTO-MIT and ORD.

Paired differences, S1, matched 32,277 reactions (points, all p < 0.005):

USPTO-50kUSPTO-MITORD
8B answer-hidden think − answer-shown write-up+12.1+9.7+9.1
8B answer-hidden think − answer only−8.0−3.8−4.2
27B answer-hidden think − answer-shown write-up−1.8+4.5+2.6
27B answer-hidden think − answer only*−2.5+7.4+8.3

*27B answer only trained on all 40,000.

The student rarely backtracks because the training data rarely demands it.

  • Teacher, 163-reaction subset: 46% of think blocks backtrack with the product hidden, 17% with it shown (median 475 vs 31 tokens).
  • Students trained on either write-up backtrack in about 1% of answers.
  • Students trained on the think block backtrack in 35–90%.
  • Backtracking is more common on harder sets and in wrong answers (8B S3, USPTO-50k: 61% of correct vs 82% of wrong; 27B S3: 30% vs 54%).

No reagents are specified for 1,439 of 5,000 test inputs; zero-shot models may be penalized for refusing to assume them.

We agree zero-shot models are penalized on these rows.

no-reagent rowsaccuracymain behavior
GPT-6-Astra2.1answers "no reaction" 96%
Kimi-K38.5answers "no reaction" 85%
Opus 561.0guesses a reagent; "no reaction" 2%
27B S167.3predicts the recorded transformation
  • In 416 of these rows the substrate alone fixes the recorded product (e.g. a lone Boc group; 316 if lone nitro groups, which can stop at the hydroxylamine, are excluded). On these, GPT-6-Astra scores 2.4 and Kimi-K3 14.9, against 79.8 for Opus 5 and 84.6 for 27B S1.

Accuracy on reactions that list a reagent:

modelUSPTO-50kUSPTO-MITORD
GPT-6-Astra87.778.074.6
Opus 586.078.171.1
Kimi-K3, answer hidden77.271.857.1
27B S1 / S2 / S383.7 / 83.9 / 84.357.8 / 58.6 / 58.144.8 / 45.4 / 45.9
27B answer only84.355.342.1

Provide analysis of generalization, including out-of-distribution reaction categories.

We removed carbon–carbon bond formation (C–C) from training. It is the only class whose defining bond is C–C, so the model must transfer mechanisms learned on C–N and C–O bonds.

Unseen C–C accuracy, answer-hidden think (28,468 training reactions; others 35,719):

unseen C–C (n)8B, S1 → S2 → S327B, S1 → S2 → S3
USPTO-50k (536)23.9 → 26.5 → 30.869.2 → 68.5 → 69.2
USPTO-MIT (582)16.3 → 15.1 → 19.147.3 → 47.4 → 46.9
ORD (780)8.7 → 9.4 → 9.227.6 → 27.4 → 26.2
  • 8B S3: think block leads the paper's traces by 3–12 points and answer only by 3–11 (p ≤ 0.008).
  • 27B, USPTO-50k C–C: 69.2 think block, 60.4 paper's traces, 50.4 answer only.
  • Self-training gives answer only at most +1.7 on C–C, despite 2–7 points on seen classes.
  • At 27B, S2 and S3 leave C–C unchanged.

[1] Kearnes, S. M., et al. (2021). The Open Reaction Database. JACS, 143(45), 18820–18826.

[2] Jin, W., Coley, C. W., Barzilay, R., & Jaakkola, T. (2017). Predicting organic reaction outcomes with Weisfeiler-Lehman network. NeurIPS 30, 2604–2613.

I thank the authors for providing additional results about the answer-hidden thinking block generation. However, according to the results, answer-only achieves the highest accuracy across all benchmarks at 8B (including the non-patent OOD set ORD) and in-distribution USPTO-50k at 27B directly undermines the central premise that natural-language reasoning traces improve chemical reaction prediction. Could the authors provide more analysis about these results?

We thank the reviewer for engaging with our rebuttal.

We agree with this reading, and the revised manuscript no longer claims that reasoning traces improve in-distribution accuracy. The abstract and Section 4.1 now state that the answer-only student is the most accurate 8B student on all three test sets after S1 and is among the most accurate in distribution at 27B. The revised paper instead asks what each kind of trace teaches beyond the product. Two parts of the reviewer's summary are incomplete, and three further results address the request for more analysis.

What the summary leaves out

  • At 8B, answer only does not lead on every benchmark after self-training. After S3, the answer-hidden think student is more accurate on USPTO-MIT (39.9% vs 37.7%).
  • At 27B, the answer-hidden think student is the most accurate student on both out-of-distribution sets. It leads answer only by 7.4 points on USPTO-MIT and 8.3 on ORD, although answer only was trained on about a quarter more reactions.

1. Answer-only accuracy does not transfer to an unseen reaction class. We removed carbon–carbon bond formation from training and evaluated on the C–C reactions of all three test sets. The answer-hidden think student is the most accurate on this class at both sizes and on every test set, despite training on fewer reactions.

ModelTraining targetTraining reactionsUSPTO-50k C–C (n = 536)USPTO-MIT C–C (n = 582)ORD C–C (n = 780)
8BAnswer only35,71920.39.86.8
8BAnswer-shown write-up35,71915.77.24.0
8BAnswer-hidden write-up28,46814.78.25.4
8BAnswer-hidden think28,46823.916.38.7
27BAnswer only35,71950.425.118.7
27BAnswer-shown write-up35,71960.433.220.9
27BAnswer-hidden write-up28,46864.635.721.0
27BAnswer-hidden think28,46869.247.327.6

Top-1 accuracy (%) after S1 on held-out C–C reactions (Appendix Table 11). Bold marks the best student for each model size and test set.

At 27B, the think student leads answer only by about 19 points on USPTO-50k C–C. Self-training raises the 8B answer-only student by 5.8 points on the classes it saw in training but by only 1.3 on the held-out class (Appendix B.2), and after S3 the 8B think student still leads answer only on held-out C–C in all three test sets.

2. The answer-only student does not hold its correct answers. At 8B it abandons 65% of its correct answers when told they are wrong, compared with 14% for the answer-hidden think student, and it abandons a quarter of them even after a neutral request to review its work. An answer-only student trained on exactly the same 32,277 reactions behaves the same way (66%), so training-set size does not explain the difference (Section 4.4, Table 33).

3. At 27B, answer-only training costs general capability. MMLU accuracy falls from 84.9% to 49.0%, while trace-trained students stay within five points of the base model (Figure 4c).

Interpretation. Answer-only training learns the transformations and recording conventions present in the training data, which relates to the reviewer's second point, while traces from a teacher that had to solve each reaction teach chemistry that transfers to a new bond type and answers the student keeps under challenge. Accuracy on USPTO-50k alone ranks the answer-only student first, whereas it ranks last under feedback and, at 27B, on transfer and general capability. This gap between benchmark accuracy and what the student has learned is the central finding of the revised paper.

The revised contributions now read:

  • A controlled study of trace-based supervision in chemistry, showing that students trained on the teacher's own reasoning transfer best to an unseen reaction class and inherit its habit of revising.
  • Evidence that accuracy is an incomplete measure of post-training, since the answer-only student is accurate in distribution but the most likely to reverse correct answers under false feedback.
  • Evidence that routine post-training stages depend on student capability, since self-training helps weaker students but not capable ones, and preference optimization makes students more fragile.

Comments on the paper

1 comment

We thank the reviewers for their detailed feedback. We have uploaded a revised manuscript, and the main changes are summarized below.

  • Reframing around how teacher traces are generated. We now vary separately whether the teacher sees the product and whether the student learns from its think block or its write-up, and we compare both against an answer only baseline. All targets are trained on the same reactions, so the comparison isolates the content of the trace.
  • Out-of-distribution evaluation. We added USPTO-MIT (deduplicated), the Open Reaction Database (patent reactions removed), and a held-out carbon–carbon bond formation class.
  • Robustness and general capability. Feedback probes are now compared against a neutral review baseline, and we added MMLU. The answer only student is accurate in distribution but the most likely to reverse correct answers under false feedback, and at 27B it loses much of its general capability.
  • Self-training and preference optimization. Self-training helps mainly students that still fail much of their training set, while preference optimization adds little accuracy and increases fragility.
  • Analysis in the main text. Appendix results, including a per-reaction-class breakdown, now appear in multi-panel figures.
  • Related work and limitations. We added recent work on answer-conditioned traces (Lee et al., 2026; Peng et al., 2026) and discuss how USPTO-50k often omits the reagent that drives a reaction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.