acceptodds
Desk rejected at ICLR 2027

Oracle-Free Autonoumous Post-Training for Chemistry via Dual-Stream Data Synthesis

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models have shown strong potential in chemistry, while employing them for long-tail chemistry tasks still requires domain adaptation, which remains labor-intensive. Recent AI-for-AI agents have demonstrated success in automating this process across domains such as programming and mathematics. However, these methods rely on rich training data and ready-made evaluation oracles, which long-tail chemistry rarely offers. To this end, we propose AutoChem, an oracle-free agentic system that starts with a single chemistry task description, builds its own training data and evaluation oracle, and post-trains a model to meet the task's requirements. At its core is a dual-stream data synthesis stage, in which two agents independently build two data synthesis engines, so that either stream can train the model while the other grades it. AutoChem then post-trains on the samples from one stream and evaluates on the other, so that gains reflect learned chemistry rather than leakage from the training engine into the test set. Extensive experiments across three scientific discovery tasks and eleven tasks from four chemistry benchmarks demonstrate AutoChem's effectiveness, generalization, and applicability.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI4AI, Post-training, Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_iRFqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AutoChem automates post-training for chemistry tasks with limited training and evaluation data. Given a task description and a few labeled examples, it builds two synthesis engines, using one to generate training data and the other for evaluation. Experiments report improvements over baselines, but the claim that cross-stream evaluation reliably guides model selection remains insufficiently validated.

Critical strengths
  1. The reusable synthesis engines offer a practical way to adapt chemistry models from a few labeled examples, which I consider valuable given the limited training and evaluation data available for these tasks.
  2. The method achieves the best results in 30 of 44 benchmark settings, and the broad evaluation and component ablations give me greater confidence in its overall effectiveness, despite the unresolved questions about the proposed mechanism.
Critical weaknesses
  1. The paper does not sufficiently explain what makes the two synthesis streams independent beyond being constructed separately. Both synthesizers use the same backbone and follow the same procedure (§2.2; Appendix B.5). Even without communication, they may produce similar question templates or rely on the same tools and labeling assumptions, allowing a student trained on one stream to exploit patterns or errors shared by the other. This is a possible failure mode, rather than evidence that shared biases necessarily occur. However, the paper provides neither a clear account of how such overlap is reduced nor a paired analysis of the resulting engines. This makes it difficult to assess whether the dual-stream design provides the separation claimed and what its methodological contribution is beyond repeated synthesis and cross-evaluation.

  2. Table 8 lacks comparison models needed to interpret the same-stream advantage. It reports only the Qwen3-4B checkpoints trained on streams A and B, without showing how the base model or models trained by other methods perform on the same A/B evaluation sets. It also does not examine whether other synthesis methods show similar gaps between fresh data from their own generators and a common external test set. Although the bidirectional results cannot be explained solely by one engine producing easier questions, they do not distinguish learned template or labeling habits from differences in task coverage and difficulty. These missing comparisons weaken the attribution of the observed gaps to generator-specific habits, which is central to the paper’s motivation for dual-stream evaluation.

Questions for authors

In addition to the concerns about stream independence and the missing comparisons in Table 8 discussed in the Weaknesses section, I have the following questions:

  1. Comparing Tables 2 and 3 shows that the answer engines substantially outperform the corresponding Qwen3-4B student models. What practical benefit does training the student provide over directly using these engines or tool-assisted agents? A comparison of inference cost, latency, or deployment requirements would help establish the value of this approach.

  2. What supports using ECFP/DRFP similarity to measure task-relevant diversity rather than molecular structural diversity alone? For answers containing no molecules, the DRFP definition in Appendix F retains only question-side molecular information. Is there evidence that this measure captures differences in the skills required by the questions or offers advantages over simpler diversity measures?

  3. What justifies equally combining tool-operation counts and reference-answer NLL as a difficulty score? Operation counts depend on implementation, while NLL may reflect answer format or unfamiliarity rather than reasoning difficulty. Is there evidence that the combined score better predicts student difficulty than either component alone? Removing the entire mechanism does not establish why this particular combination is appropriate.

  4. Please clarify the “about 9 points” statement in §3.4. Across the eleven tasks in Table 8, the mean absolute difference between A→B and A→test is approximately 9.03 points, whereas the mean signed difference, calculated as A→B minus A→test, is −0.80 points. The reported nine-point figure therefore appears to describe an average absolute discrepancy, rather than an average decrease. Is this the intended interpretation?

Reviewer_zsCAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for constructing task-specific post-training data for long-tail chemistry tasks in which correct answers cannot be readily obtained or determined through rule-based procedures. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method and its key components.

Critical strengths
  1. The paper presents an insightful observation: in the absence of an evaluation oracle, assessing the effectiveness of synthetic training data using evaluation data generated by the same synthesis engine may introduce systematic bias, such that validation performance does not reflect the model’s actual capabilities.
  2. The paper proposes a dual-stream data synthesis method for post-training to address this issue.
  3. Extensive experiments demonstrate the effectiveness of the proposed method and the contributions of its key components, particularly the dual-stream design.
Critical weaknesses
  1. The paper overstates the results in Table 1. Different evaluation metrics should not be counted as distinct experimental settings. Both the “11 out of 18” claim and the characterization of the method as outperforming the baselines on “most tasks” are misleading and overly strong. In fact, across the six tasks, the proposed method outperforms Claude Code on three, underperforms it on one, and yields mixed results across metrics on the remaining two. Thus, the results support a claim of superior performance on only half of the tasks.
  2. I have some concerns about the results for “w/o search loop” and “w/o dual synthesis agent” in the ablation study in Figure 4. As I understand it, the data produced by the dual synthesis agents are used within the search loop. It is therefore unclear to me how each component can operate without the other and why removing them leads to such markedly different performance drops.
  3. The paper’s primary focus is on the dual synthesis agents and the associated search loop. In other words, the emphasis is on making better use of the data generated by synthesis engines, rather than on generating better synthetic data. Do the other baseline methods in Tables 1 and 2 employ comparable search procedures? If not, I believe that an additional comparison in which these baselines are augmented with a search loop would provide stronger evidence for the effectiveness of the proposed method.
Questions for authors

The differences between the two synthesis engines are not described in sufficient detail. Are they explicitly differentiated, or are they simply constructed through two parallel data construction processes, with their differences arising primarily from the model’s stochasticity and differences in the shown examples?

Reviewer_F7kgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoChem, an autonomous framework for adapting large language models to data-scarce chemistry tasks without a preexisting training corpus or evaluation oracle. Given a task description and a few solved examples, a research agent identifies required capabilities, while two independent agents construct executable engines that generate labeled chemistry questions. A verifier checks these engines against held-out examples, and coverage, diversity, and difficulty controls improve sample quality. Crucially, AutoChem trains on data from one engine and evaluates on the other, reducing the risk that performance gains merely reflect memorization of synthetic templates. An iterative search then selects data mixtures and post-training strategies. Validation covers three experimental chemistry discovery tasks and eleven tasks across four benchmarks, using four base models. AutoChem leads in 30 of 44 benchmark settings and 11 of 18 discovery comparisons. Ablations identify dual-stream synthesis as the most influential component, supporting its utility for reliable autonomous adaptation.

Critical strengths

The first strength is the practical motivation and problem formulation. AutoChem addresses the important challenge of adapting language models to data-scarce chemistry tasks without predefined training datasets or evaluation oracles. This setting is relevant to long-tail scientific discovery, where obtaining reliable experimental labels is expensive. The proposed autonomous workflow offers a potentially useful direction for reducing manual effort, although its applicability depends on accessible chemistry resources.

The second strength is the dual-stream synthesis framework, which independently constructs training and evaluation data generators to mitigate memorization and evaluation leakage. This is a thoughtful design choice that addresses a genuine limitation of existing synthetic data approaches. The authors provide supporting evidence through comparisons across multiple chemistry benchmarks, four base models, and ablation studies demonstrating the importance of dual-stream synthesis. However, these strengths do not fully establish the reliability of synthetic evaluation or sufficiently demonstrate generalization to genuinely unseen chemistry tasks.

Critical weaknesses

The first weakness is the limited validation of the proposed synthetic evaluation oracle. Although two synthesis engines are authored independently, they can still share biases from public databases, chemistry tools, and model-generated labels. Verification against only 18 solved examples cannot establish correctness across diverse chemical structures. Consequently, cross-stream evaluation may reduce template memorization without reliably measuring genuine chemical generalization, weakening the paper’s central claim.

The second weakness is the inconsistent performance on scientific discovery tasks. Despite strong aggregate benchmark results, AutoChem produces worse MAE and RMSE than the untuned models on SAMPL9 for both evaluated backbones, and does not consistently outperform Claude Code on FreeSolv. Furthermore, excluding overlapping training molecules requires knowledge of benchmark test-set molecular identities, a safeguard that may not be available in real deployment. These issues leave uncertainty about robustness and practical applicability.

Questions for authors

see weakness

Reviewer_fmN4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents AutoChem, an agentic system for chemistry tasks that builds on its own two data synthesis engines which can be used for model post training and evaluation respectively. Training is done based on one engine (and its data) and evaluation is based on the other one. Each of the two engines can be interchangeably used in either role (training or evaluation). The dual nature of the approach (i.e., using two data synthesis engines) is presented as a core feature of the approach taken. As the paper argues, this worked is aimed at cases in chemistry where training and evaluation data are scarce, such as long-tailed chemistry tasks. The paper presents technical details of the approach, as well as an empirical evaluation and a discussion of related work.

Critical strengths

The main idea of building an autonomous pipeline with LLMs, data generation, training and evaluation is sensible and it comes across as well justified/motivated. Empirical results, such as summary data presented in Tables 1 and 2 for example, show a good performance of AutoChem compared to a range of baseline algorithms.

Critical weaknesses

The details of the approach are difficult to follow in the main paper. It lacks to a good extent examples of concepts such as a complex chemistry task and its decomposition into subtasks, samples, questions generated, and answers created with function f_i.

It’s not clear from the paper what the fundamental difference between the two data synthesis engines is. Where do their bias and the potential tendency to create engine-specific patterns stem from? How is it ensured that the two engines do not suffer from the same bias? The dedicated section “2.3 Why Dual Streams?” is too short and high level. Some additional terms appear to be used without a definition, such as “training recipe” for instance. A brief reference is made about domain experts evaluating the quality of question-answer pairs. I have not seen any details about how many experts participated, how they were selected, what is their expertise, and any additional details normally provided in the case of user studies.

Questions for authors

Please see the weaknesses section for questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.