- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Resonance, a family of open-weight reasoning models (8B and 27B) for forward reaction prediction on USPTO-50k. The pipeline has three stages: (S1) supervised fine-tuning on reasoning traces generated by a frontier teacher (Kimi K3) with the product visible; (S2) rejection-sampling SFT on the student's own verified-correct traces; (S3) DPO on self-generated preference pairs (correct vs. most-similar incorrect trace), with an auxiliary NLL term on the chosen response. The main empirical findings are: (i) teacher SFT is essential for the 8B model, and self-training adds 11.8 points without teacher supervision (31.0 → 42.8); (ii) the stronger 27B model surpasses its teacher after S1 alone (79.0%) and gains nothing measurable from S2/S3; (iii) the 8B student inherits the teacher corpus's single-pass reasoning style, and prompting-based test-time interventions (think more, consider alternatives) do not help; (iv) the student is sycophantic under negative feedback, abandoning 22–32% of initially correct answers, with the loss growing after DPO; (v) self-training mainly sharpens answers the model could already reach, as shown by plurality@k/pass@k analysis.
Critical strengths
1.Clean, well-controlled pipeline and ablation. The three-stage decomposition is clearly specified, and the paper reports skip-S2 ablations, per-stage McNemar tests, per-class breakdowns, and a large appendix with training configurations, prompts, and sampling settings. This is unusually thorough for a domain-adaptation paper.
2.The "stronger student needs less of the pipeline" finding is genuinely useful. Showing that the same teacher traces take 8B from 1.2% → 31.0% but 27B from 26.3% → 79.0% (above the teacher) is a crisp, actionable result. The diagnostic—measure the student's solve rate on the training set after S1 to predict whether self-training will help—is a practical contribution.
Critical weaknesses
1.Novelty is limited at the method level. Distillation + rejection-sampling SFT + DPO is a standard recipe (STaR, RFT, iterative DPO). The paper's contribution is primarily empirical and domain-specific rather than methodological. For a top venue, the bar is that the findings justify the lack of methodological novelty—and here the findings are solid but not surprising: that self-training helps a weak student more than a strong one, and that self-training sharpens rather than expands the distribution, are both consistent with prior work (Huang et al., 2024; Yue et al., 2026). The paper cites these, which is good, but it means the headline results are largely confirmatory.
2.The central comparison with frontier models is confounded. Resonance is trained on USPTO-50k train; Claude Opus 5 and GPT-6-Astra are evaluated zero-shot. The paper acknowledges this, but it still frames the 8B model as "lagging the teacher" and the 27B as "comparable to Opus 5." Those framings are not supported by the experimental design. A fair comparison would require either fine-tuning the frontier models (infeasible) or evaluating the students zero-shot on held-out chemistry. As written, the comparison measures "what task-specific post-training adds," which is a weaker claim than the paper sometimes implies.
3.The teacher's low scores on reductions/oxidations/deprotections are a dataset artifact, not a chemistry result. The paper is aware of this (Appendix C.2), and the analysis is careful. But the main text uses these classes to argue that "students and teacher understand different chemistry" (Section 4.2), when the more accurate statement is that the teacher refuses to infer a missing reagent and the student has learned the dataset convention of guessing. This is a dataset-convention finding, not a chemistry-reasoning finding. The framing in the abstract and Section 4.2 should be tightened.
Questions for authors
1.Teacher-trace visibility. The teacher sees the recorded product when generating S1 traces. How much of the S1 gain is due to the trace content versus simply learning the output format and task distribution? The few-shot control (Appendix B.1) shows format alone is insufficient, but it does not separate "trace teaches chemistry" from "trace teaches the dataset's answer conventions." Could you train S1 on answer-only targets (no thinking) and compare?
2.DPO fragility. Is the increased feedback sensitivity at S3 specific to DPO, or would it also appear with rejection-sampling SFT alone? A comparison of S2 vs. S3 flip rates under "told wrong" (Table 16) shows 21.5% vs. 32.3% C→I, so DPO does seem to increase it. But is this because DPO sharpens the policy toward correct answers (so there are more correct answers to lose), or because it specifically increases compliance? The paper reports the loss as a share of baseline (22% at S3 vs. 11% at S2), which controls for the first explanation. So the answer seems to be "yes, DPO specifically increases fragility." It would be worth stating this more explicitly and, if possible, testing DPO with a larger λ to see if the NLL term mitigates it.
3.Generalization beyond USPTO-50k. The paper evaluates a single task and benchmark. Given that the main practical claim is "measure the S1 solve rate to decide whether to self-train," how portable is that diagnostic? Does it hold on a different reaction dataset?
2 comments
[1] Kearnes, S. M., et al. (2021). The Open Reaction Database. JACS, 143(45), 18820–18826.
[2] Jin, W., Coley, C. W., Barzilay, R., & Jaakkola, T. (2017). Predicting organic reaction outcomes with Weisfeiler-Lehman network. NeurIPS 30, 2604–2613.
We agree the self-training findings match prior work. The rebuttal experiments add findings that do not:
S1, same 32,277 reactions, USPTO-50k / MIT / ORD:
*trained on all 40,000.
Last column: 4,281 C–C training reactions held out of student training.
*Best 27B student without C–C (answer-hidden think S1); 69.2 on the 536 C–C test reactions (reply to 6v8y).
If accepted, the camera-ready version will: