- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Training LLMs for open-ended research problems is difficult because of two challenges: (1) A single final outcome reward cannot indicate which parts of the research process are effective, and (2) intermediate rewards are difficult to define and may reduce diversity. This paper introduces PESTO (“Paper-Extracted Surrogate Task Optimization”) to improve a model’s research capabilities by learning from existing scientific papers. Specifically, they first convert papers into DAGs and assign predefined semantic roles to each node. Second, they use these graphs to generate question-answer pairs. After that, based on the reference answers, they further generate binary evaluation criteria for RL training. They train Qwen3.6-27B on the obtained data using SFT followed by RL. Finally, they evaluate the model in four settings to demonstrate the effectiveness of their method.
Critical strengths
- The idea is overall reasonable.
- The figures are easy to understand.
Critical weaknesses
- Motivation. I think the motivation and the proposed method are not well aligned. In the abstract, the authors highlight two challenges: (1) a single final outcome reward cannot indicate which research decisions contribute to the final result, and (2) intermediate rewards are difficult to define and may reduce the diversity of research directions. However, PESTO does not directly address either of these challenges. Instead of using final outcome rewards, PESTO introduces paper-derived surrogate rewards. This raises several questions. First, how are these surrogate rewards fundamentally different from directly specified intermediate rewards? Although PESTO provides rewards for individual research decisions, it still does not establish how much each decision contributes to the final research outcome. Second, in many AI/ML research tasks, there is already a clear final objective, such as improving accuracy or reducing loss. If improving this metric is the ultimate goal, why should paper-derived rewards be preferred over execution-based rewards? I understand that execution-based rewards can be expensive to obtain, but the paper does not sufficiently explain why the proposed surrogate rewards are a better alternative beyond their scalability. Finally, the paper does not clearly demonstrate how PESTO addresses the concern about overly prescriptive intermediate rewards, since its rubrics are still derived from specific reference answers in existing papers.
- Contributions. I think the methodological contribution is limited. Extracting information from scientific papers, such as research goals and questions, to construct training data is not new in this field. From my perspective, the main differences are the DAG-based decomposition and the use of multiple research decision types. However, it is unclear how these components contribute to the final performance, and the paper does not provide ablation studies to isolate their effects. In particular, if the research decision types are already predefined, why is it necessary to construct a DAG? What would be the difference if the authors directly extracted question-answer pairs from papers using these predefined categories? Without such comparisons, it is difficult to understand the necessity and effectiveness of the proposed DAG-based decomposition.
- Method. PESTO includes multiple research decision types, such as Literature → Gap and Method → Experiments. Although all these tasks are related to scientific research, they do not necessarily represent the same capability. For example, at least three different capabilities are involved: (1) idea/method proposal, (2) experiment design, and (3) result analysis. However, PESTO treats them as independent QA tasks, potentially ignoring the dependencies between different stages of the research process. For instance, the method proposed by the model in one task is not used as the context for subsequent experiment design. This raises the question of whether training on these independent tasks can effectively improve multi-stage research decision-making. Moreover, the paper does not analyze the contribution of each research-decision category to downstream performance. Is joint training on all decision types necessary, or would training only on idea/method proposal tasks achieve similar results?
- Evaluation. I do not think the current evaluations sufficiently support the paper’s main claims. First, the paper mainly compares different training strategies, such as SFT, CPT, and SFT + RL. However, SFT + RL is already a commonly used training paradigm, and showing that rubric-guided RL outperforms the evaluated SFT baselines does not necessarily demonstrate the effectiveness of PESTO’s specific task construction. I think it is more important to compare against existing paper-derived research training methods, especially those discussed in the related work. Do the downstream improvements arise from PESTO’s DAG-based task construction and research-decision decomposition, or simply from additional domain-specific rubric-guided RL? Second, although the paper includes several execution-grounded evaluations, most of them focus on generating proposals for tasks with predefined objectives and evaluation metrics. Only the final open-ended paper extension experiment evaluates multi-stage research guidance with experimental feedback, where Codex is also responsible for selecting, implementing, and executing the proposed ideas. Furthermore, this evaluation is conducted on only one research problem and reports the best trajectory out of eight runs. Therefore, it remains unclear how consistently PESTO improves open-ended research outcomes across independent trajectories and different research problems.
Questions for authors
Please see weaknesses.
8 comments
Response to Reviewer 8qxC (2/3)
3. Independent training tasks and multi-stage research decisions
Training treats the decision types as independent QA tasks. Can this improve multi-stage research, and is joint training necessary compared with idea/method-only training?
The motivation for including multiple categories is to train the same advising policy to respond at several points in a research process: proposing a direction, designing an experiment, and interpreting feedback to decide what to try next. At deployment, these contexts can come from the ongoing project. Whether the resulting capabilities compose successfully is therefore an empirical transfer question, particularly because contexts induced by the model's own decisions can differ from paper-derived training contexts.
The AI Engram evaluation tests this distinction directly. Selected model-generated decisions become context for later stages, and executed experimental results are returned to the advisor before it proposes an analysis and revision (Appendix F). The reported PESTO trajectory illustrates this behavior: after an initial method violates an exact-match (EM) constraint, the subsequent revision responds to that feedback by normalizing and clipping the correction factors.
The PaperGym comparison above also provides empirical evidence relevant to the question of training coverage: a policy trained on PESTO's broader set of research decisions outperforms the baseline trained on research-plan generation data under the evaluated settings.
4. Reference-derived rubrics, alternative solutions, and diversity
How does PESTO avoid overly prescriptive supervision if its rubrics are derived from specific reference answers? PESTO rewards criterion satisfaction. During RL, the policy does not receive the reference answer, and rewards do not use direct reference matching or embedding similarity (Appendix A). Rubric refinement aims to avoid penalizing a correct explanation solely because it uses different terminology. For example, Figure 2 revises a criterion that requires the exact term “over-squashing” so that an explanation of the underlying information bottleneck also receives credit.
We agree, however, that criteria derived from a published solution can still favor that solution over other valid approaches. Our rubrics are surrogate training signals, not an exhaustive characterization of scientifically valid answers. We add two complementary evaluations: a human assessment of response quality and a controlled audit of whether rubrics reward valid alternatives and penalize invalid answers.
Preliminary human evaluation. We sample 48 held-out questions, with eight questions from each of the six decision categories. Two ML PhD students who were not involved in this work independently compare base-model and PESTO responses based on correctness, feasibility, and completeness. Model identities, reference answers, and rubrics are hidden. Each response pair is evaluated twice, with its presentation order reversed, to account for potential order effects. We also obtain preferences from two model judges using the same blinded comparison protocol. We average valid judgments across the two presentation orders within each question, then average across questions, giving each question equal weight.
| Judge | Prefer PESTO | Prefer base |
|---|---|---|
| GPT-5.5 | 56.25% | 43.75% |
| Claude Opus-5 | 70.83% | 29.17% |
| Researcher 1 | 60.42% | 39.58% |
| Researcher 2 | 64.58% | 35.42% |
Both human evaluators prefer PESTO more often than the base model. This provides preliminary evidence that the gains are reflected in independently assessed response quality.
Response to Reviewer 8qxC (3/3)
Rubric validation. We next test whether the rubrics recognize different valid solutions while rejecting answers with substantive errors. We evaluated 100 questions, each paired with three answers: the paper-derived reference, an alternative solution, and an invalid control. To construct the alternatives, we searched the literature for different approaches applicable to the same problem, adapted them to the question’s constraints, and retained supporting citations. Two independent ML researchers verified their correctness and applicability. We constructed invalid controls by introducing a substantive error into an otherwise plausible answer, such as violating a stated constraint or reversing a necessary algorithmic step; the researchers also verified these errors. The rubric judge scored each answer individually, without source or validity labels, using comparable answer lengths and formatting. We counted a triplet as passing only when both valid answers scored at least 0.7, their scores differed by at most 0.1, and the invalid control scored at most 0.1.
The rubrics satisfied all three conditions on 71/100 questions. Across the full sample, 1/100 valid alternatives received near-zero reward (≤0.1), while 16/100 invalid controls received scores above 0.1. The rest failures are score differences greater than 0.1 between the two valid answers. These results support the rubrics’ ability to recognize valid alternatives in this sample, while identifying remaining errors in scoring invalid answers and consistently rewarding different valid solutions.
5. Scope of the execution-based evidence and the role of Codex
Most evaluations have predefined objectives. The multi-stage evaluation uses Codex for selection and execution, considers one problem, and reports the best of eight trajectories.
We agree that these evaluations answer different questions and should be interpreted separately. The local-guidance and fixed-objective settings test whether paper-derived supervision transfers to measurable execution outcomes. They do not establish general open-ended research ability. The improvements include mean outcomes, as reported in Table 3:
| Fixed-objective metric | Base | PESTO |
|---|---|---|
| Mean nanoGPT validation loss ↓ | 3.4431 | 3.2691 |
| Mean GRPO held-out accuracy ↑ | 38.23% | 45.80% |
These results support the practical usefulness of the learned research guidance beyond rubric scores. The multi-stage paper-extension experiment addresses a further question: whether the advisor can use its selected earlier decisions and experimental feedback to guide subsequent stages.
The Codex component is held fixed across the advising models. Both models use the same selection-and-execution protocol and candidate budget. In the open-ended setting, the advisor produces 16 candidates at each stage, Codex selects and implements a candidate, and the resulting context and experimental feedback are passed to the advisor. The comparison therefore measures performance when the advising model is changed within a shared research system. It does not establish that the trained model can independently select, implement, and execute an entire research project.
Additional proposal audit. We also examine the mechanisms proposed across a larger sample of advisor outputs. We sampled 16 additional proposals each from Base and the PESTO checkpoint using the same Engram paper, source code, and generation settings, retaining all 32 outputs. A qualitative assistant audit identified different emphases:
| Proposal characteristic | Base | Updated PESTO |
|---|---|---|
| Adjusts edit strength by layer | 0/16 | 8/16 |
| Requires gradient information | 9/16 | 4/16 |
| Filters reference/context covariance directions | 7/16 | 4/16 |
For example, PESTO proposals include forget-versus-retain energy scaling, while Base proposals include confidence-weighted covariance estimation. We will include all 32 proposals and the item-level analysis in the appendix, including technical errors identified in both sets.
The bootstrap confidence interval currently reported in Appendix F quantifies evaluation uncertainty conditional on the selected trajectory; These analyses can strengthen the evidence for consistency within this research problem, while the experiment remains a single-paper case study. We will keep that scope explicit.
We hope these additions and the clarified scope address your concerns and would appreciate your reconsideration of the rating.
[1] Wang, Yuhan, et al. “PaperGym: Rubric-Centered Evolution for Research-Plan Generation.” arXiv:2608.31119, 2026.
Response to Reviewer 8qxC (1/3)
Thank you for the detailed review. We now address these points below and provide new comparisons with training on PaperGym data, a rubric-refinement ablation, and an independent human assessment.
1. Motivation: learning research decisions from surrogate supervision
Our motivation is that execution-based rewards are expensive even when the final objective is clear. For example, training a model to propose improvements to LLM training may require evaluating multiple candidate ideas per training iteration. Each candidate can require implementation and several GPU-hours of training and evaluation, while different research problems require their own environments and experimental configurations. Paper-derived tasks provide a cheaper source of decision-level feedback: responses can be assessed using evidence and rubrics derived from completed research, without executing every proposed idea.
We also want to point out that Paper-derived training can complement subsequent execution-based training. Once the necessary infrastructure and compute are available, a model trained with PESTO can be further optimized using execution feedback. PESTO investigates how much useful research capability can already be learned from a cheaper, readily available source of supervision: the decisions and evidence contained in existing scientific papers.
2. Task construction: the role of the DAG and comparison with prior paper-derived training
The DAG helps expose technical decisions that direct extraction can leave unused. We compare both construction methods on the same 128 papers and six decision categories, using a shared question-rewriting stage. Across 675 pairs completed by both methods, DAG-based references contain 3.04× as much text, and explicit mathematical notation appears in 36.7% versus 19.7% of references.
For example, on an RLHF paper, direct extraction selects the overall minimax framework, while DAG construction exposes the specific enhancer procedure: sampling multiple responses, tournament selection, and an explicit KL-divergence bound. We note that these details matter for rubric generation. When reference answers omit procedural steps or constraints, the resulting criteria can be too general to distinguish technically complete proposals from vague ones. Retaining this information provides a basis for more specific, checkable criteria.
To further elaborate this point, we add The PaperGym comparison to address both the prior-training baseline and the question of proposal-focused versus broader research-decision training.** PaperGym [1] focuses on generating research plans from a paper's goals and background, with reference answers and rubrics covering the proposed methodology and experimental design. Whereas PESTO additionally constructs separate tasks for decisions throughout the research process, including identifying gaps, designing experiments given a method, and interpreting observed results. PaperGym therefore provides a relevant existing alternative centered on research-plan generation.
To compare these training approaches using a common backbone, we train Qwen3.6-27B with RL on PaperGym data under a matched compute budget. On held-out ICLR 2026 QA tasks:
The comparison between the two RL-only models is particularly relevant to the concern: using PESTO data yields a 2.22% improvement under the same backbone and matched compute, without introducing an SFT-stage difference. The full PESTO recipe achieves a 5.27% improvement over RL on PaperGym data.
We also compare the PaperGym-data baseline with PESTO on four execution-based tasks: