- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AutoChem automates post-training for chemistry tasks with limited training and evaluation data. Given a task description and a few labeled examples, it builds two synthesis engines, using one to generate training data and the other for evaluation. Experiments report improvements over baselines, but the claim that cross-stream evaluation reliably guides model selection remains insufficiently validated.
Critical strengths
- The reusable synthesis engines offer a practical way to adapt chemistry models from a few labeled examples, which I consider valuable given the limited training and evaluation data available for these tasks.
- The method achieves the best results in 30 of 44 benchmark settings, and the broad evaluation and component ablations give me greater confidence in its overall effectiveness, despite the unresolved questions about the proposed mechanism.
Critical weaknesses
-
The paper does not sufficiently explain what makes the two synthesis streams independent beyond being constructed separately. Both synthesizers use the same backbone and follow the same procedure (§2.2; Appendix B.5). Even without communication, they may produce similar question templates or rely on the same tools and labeling assumptions, allowing a student trained on one stream to exploit patterns or errors shared by the other. This is a possible failure mode, rather than evidence that shared biases necessarily occur. However, the paper provides neither a clear account of how such overlap is reduced nor a paired analysis of the resulting engines. This makes it difficult to assess whether the dual-stream design provides the separation claimed and what its methodological contribution is beyond repeated synthesis and cross-evaluation.
-
Table 8 lacks comparison models needed to interpret the same-stream advantage. It reports only the Qwen3-4B checkpoints trained on streams A and B, without showing how the base model or models trained by other methods perform on the same A/B evaluation sets. It also does not examine whether other synthesis methods show similar gaps between fresh data from their own generators and a common external test set. Although the bidirectional results cannot be explained solely by one engine producing easier questions, they do not distinguish learned template or labeling habits from differences in task coverage and difficulty. These missing comparisons weaken the attribution of the observed gaps to generator-specific habits, which is central to the paper’s motivation for dual-stream evaluation.
Questions for authors
In addition to the concerns about stream independence and the missing comparisons in Table 8 discussed in the Weaknesses section, I have the following questions:
-
Comparing Tables 2 and 3 shows that the answer engines substantially outperform the corresponding Qwen3-4B student models. What practical benefit does training the student provide over directly using these engines or tool-assisted agents? A comparison of inference cost, latency, or deployment requirements would help establish the value of this approach.
-
What supports using ECFP/DRFP similarity to measure task-relevant diversity rather than molecular structural diversity alone? For answers containing no molecules, the DRFP definition in Appendix F retains only question-side molecular information. Is there evidence that this measure captures differences in the skills required by the questions or offers advantages over simpler diversity measures?
-
What justifies equally combining tool-operation counts and reference-answer NLL as a difficulty score? Operation counts depend on implementation, while NLL may reflect answer format or unfamiliarity rather than reasoning difficulty. Is there evidence that the combined score better predicts student difficulty than either component alone? Removing the entire mechanism does not establish why this particular combination is appropriate.
-
Please clarify the “about 9 points” statement in §3.4. Across the eleven tasks in Table 8, the mean absolute difference between A→B and A→test is approximately 9.03 points, whereas the mean signed difference, calculated as A→B minus A→test, is −0.80 points. The reported nine-point figure therefore appears to describe an average absolute discrepancy, rather than an average decrease. Is this the intended interpretation?