acceptodds
Under review as a conference paper at ICLR 2027

Training LLMs to Guide Research by Studying Scientific Papers

Abstract

Training language models to make progress on open-ended research problems is difficult because useful feedback is hard to attribute to individual research decisions, such as proposing an idea, developing a method, or designing an experiment. Directly specifying rewards for such intermediate decisions can also become overly prescriptive, narrowing what counts as a promising research direction. In AI research, even obtaining sufficient reliable feedback is quite expensive. We introduce PESTO, or "Paper-Extracted Surrogate Task Optimization", an approach for improving a model's ability to conduct open-ended AI research by deriving focused research tasks from scientific papers. We decompose papers into connected decisions such as identifying literature gaps, proposing methods, designing experiments, and interpreting results, and turn them into shorter-horizon problems with criterion-level rubric rewards. Applying this construction across many papers makes individual decisions easier to evaluate while preserving a broad range of research approaches. We train a 27B-parameter model with RL on tasks derived from ICML, ICLR, NeurIPS, and arXiv papers. Training improves performance on held-out research tasks and transfers to more open-ended settings where proposed ideas need to be implemented and evaluated. PESTO improves downstream research outcomes and shows qualitative evidence of broader exploration. In a case study, it develops an extension that improves upon the reproduced baseline of an ICML 2026 oral paper excluded from our training corpus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, AI for scientific discovery, rubric-based rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8qxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Training LLMs for open-ended research problems is difficult because of two challenges: (1) A single final outcome reward cannot indicate which parts of the research process are effective, and (2) intermediate rewards are difficult to define and may reduce diversity. This paper introduces PESTO (“Paper-Extracted Surrogate Task Optimization”) to improve a model’s research capabilities by learning from existing scientific papers. Specifically, they first convert papers into DAGs and assign predefined semantic roles to each node. Second, they use these graphs to generate question-answer pairs. After that, based on the reference answers, they further generate binary evaluation criteria for RL training. They train Qwen3.6-27B on the obtained data using SFT followed by RL. Finally, they evaluate the model in four settings to demonstrate the effectiveness of their method.

Critical strengths
  1. The idea is overall reasonable.
  2. The figures are easy to understand.
Critical weaknesses
  1. Motivation. I think the motivation and the proposed method are not well aligned. In the abstract, the authors highlight two challenges: (1) a single final outcome reward cannot indicate which research decisions contribute to the final result, and (2) intermediate rewards are difficult to define and may reduce the diversity of research directions. However, PESTO does not directly address either of these challenges. Instead of using final outcome rewards, PESTO introduces paper-derived surrogate rewards. This raises several questions. First, how are these surrogate rewards fundamentally different from directly specified intermediate rewards? Although PESTO provides rewards for individual research decisions, it still does not establish how much each decision contributes to the final research outcome. Second, in many AI/ML research tasks, there is already a clear final objective, such as improving accuracy or reducing loss. If improving this metric is the ultimate goal, why should paper-derived rewards be preferred over execution-based rewards? I understand that execution-based rewards can be expensive to obtain, but the paper does not sufficiently explain why the proposed surrogate rewards are a better alternative beyond their scalability. Finally, the paper does not clearly demonstrate how PESTO addresses the concern about overly prescriptive intermediate rewards, since its rubrics are still derived from specific reference answers in existing papers.

  1. Contributions. I think the methodological contribution is limited. Extracting information from scientific papers, such as research goals and questions, to construct training data is not new in this field. From my perspective, the main differences are the DAG-based decomposition and the use of multiple research decision types. However, it is unclear how these components contribute to the final performance, and the paper does not provide ablation studies to isolate their effects. In particular, if the research decision types are already predefined, why is it necessary to construct a DAG? What would be the difference if the authors directly extracted question-answer pairs from papers using these predefined categories? Without such comparisons, it is difficult to understand the necessity and effectiveness of the proposed DAG-based decomposition.

  1. Method. PESTO includes multiple research decision types, such as Literature → Gap and Method → Experiments. Although all these tasks are related to scientific research, they do not necessarily represent the same capability. For example, at least three different capabilities are involved: (1) idea/method proposal, (2) experiment design, and (3) result analysis. However, PESTO treats them as independent QA tasks, potentially ignoring the dependencies between different stages of the research process. For instance, the method proposed by the model in one task is not used as the context for subsequent experiment design. This raises the question of whether training on these independent tasks can effectively improve multi-stage research decision-making. Moreover, the paper does not analyze the contribution of each research-decision category to downstream performance. Is joint training on all decision types necessary, or would training only on idea/method proposal tasks achieve similar results?

  1. Evaluation. I do not think the current evaluations sufficiently support the paper’s main claims. First, the paper mainly compares different training strategies, such as SFT, CPT, and SFT + RL. However, SFT + RL is already a commonly used training paradigm, and showing that rubric-guided RL outperforms the evaluated SFT baselines does not necessarily demonstrate the effectiveness of PESTO’s specific task construction. I think it is more important to compare against existing paper-derived research training methods, especially those discussed in the related work. Do the downstream improvements arise from PESTO’s DAG-based task construction and research-decision decomposition, or simply from additional domain-specific rubric-guided RL? Second, although the paper includes several execution-grounded evaluations, most of them focus on generating proposals for tasks with predefined objectives and evaluation metrics. Only the final open-ended paper extension experiment evaluates multi-stage research guidance with experimental feedback, where Codex is also responsible for selecting, implementing, and executing the proposed ideas. Furthermore, this evaluation is conducted on only one research problem and reports the best trajectory out of eight runs. Therefore, it remains unclear how consistently PESTO improves open-ended research outcomes across independent trajectories and different research problems.
Questions for authors

Please see weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8qxC (1/3)

Thank you for the detailed review. We now address these points below and provide new comparisons with training on PaperGym data, a rubric-refinement ablation, and an independent human assessment.

1. Motivation: learning research decisions from surrogate supervision

How do paper-derived rewards address credit assignment, and why should they be preferred to execution-based rewards when the final objective is clear?

Our motivation is that execution-based rewards are expensive even when the final objective is clear. For example, training a model to propose improvements to LLM training may require evaluating multiple candidate ideas per training iteration. Each candidate can require implementation and several GPU-hours of training and evaluation, while different research problems require their own environments and experimental configurations. Paper-derived tasks provide a cheaper source of decision-level feedback: responses can be assessed using evidence and rubrics derived from completed research, without executing every proposed idea.

We also want to point out that Paper-derived training can complement subsequent execution-based training. Once the necessary infrastructure and compute are available, a model trained with PESTO can be further optimized using execution feedback. PESTO investigates how much useful research capability can already be learned from a cheaper, readily available source of supervision: the decisions and evidence contained in existing scientific papers.

2. Task construction: the role of the DAG and comparison with prior paper-derived training

If the decision categories are predefined, why construct a DAG instead of extracting question-answer pairs directly? Do the gains reflect the construction or simply additional domain-specific rubric-guided RL?

The DAG helps expose technical decisions that direct extraction can leave unused. We compare both construction methods on the same 128 papers and six decision categories, using a shared question-rewriting stage. Across 675 pairs completed by both methods, DAG-based references contain 3.04× as much text, and explicit mathematical notation appears in 36.7% versus 19.7% of references.

For example, on an RLHF paper, direct extraction selects the overall minimax framework, while DAG construction exposes the specific enhancer procedure: sampling multiple responses, tournament selection, and an explicit KL-divergence bound. We note that these details matter for rubric generation. When reference answers omit procedural steps or constraints, the resulting criteria can be too general to distinguish technically complete proposals from vague ones. Retaining this information provides a basis for more specific, checkable criteria.

To further elaborate this point, we add The PaperGym comparison to address both the prior-training baseline and the question of proposal-focused versus broader research-decision training.** PaperGym [1] focuses on generating research plans from a paper's goals and background, with reference answers and rubrics covering the proposed methodology and experimental design. Whereas PESTO additionally constructs separate tasks for decisions throughout the research process, including identifying gaps, designing experiments given a method, and interpreting observed results. PaperGym therefore provides a relevant existing alternative centered on research-plan generation.

To compare these training approaches using a common backbone, we train Qwen3.6-27B with RL on PaperGym data under a matched compute budget. On held-out ICLR 2026 QA tasks:

Training dataTraining recipeAverage rubric satisfaction
PaperGymRL-only, compute-matched48.75%
PESTORL-only, compute-matched50.97%
PESTOSFT + RL54.02%

The comparison between the two RL-only models is particularly relevant to the concern: using PESTO data yields a 2.22% improvement under the same backbone and matched compute, without introducing an SFT-stage difference. The full PESTO recipe achieves a 5.27% improvement over RL on PaperGym data.

We also compare the PaperGym-data baseline with PESTO on four execution-based tasks:

Task / metricRL on PaperGym dataPESTO
Discrete RL value estimation / mean score0.034260.15439
Pooling and aggregation / mean score0.211060.31556
Cross-modal single-cell protein data matching / mean accuracy0.807150.83345
Metal-organic framework property prediction / mean R²0.872610.90512
These results support PESTO’s overall approach across QA and execution-based evaluations.
AuthorsOpenReview ↗

Response to Reviewer 8qxC (2/3)

3. Independent training tasks and multi-stage research decisions

Training treats the decision types as independent QA tasks. Can this improve multi-stage research, and is joint training necessary compared with idea/method-only training?

The motivation for including multiple categories is to train the same advising policy to respond at several points in a research process: proposing a direction, designing an experiment, and interpreting feedback to decide what to try next. At deployment, these contexts can come from the ongoing project. Whether the resulting capabilities compose successfully is therefore an empirical transfer question, particularly because contexts induced by the model's own decisions can differ from paper-derived training contexts.

The AI Engram evaluation tests this distinction directly. Selected model-generated decisions become context for later stages, and executed experimental results are returned to the advisor before it proposes an analysis and revision (Appendix F). The reported PESTO trajectory illustrates this behavior: after an initial method violates an exact-match (EM) constraint, the subsequent revision responds to that feedback by normalizing and clipping the correction factors.

The PaperGym comparison above also provides empirical evidence relevant to the question of training coverage: a policy trained on PESTO's broader set of research decisions outperforms the baseline trained on research-plan generation data under the evaluated settings.

4. Reference-derived rubrics, alternative solutions, and diversity

How does PESTO avoid overly prescriptive supervision if its rubrics are derived from specific reference answers? PESTO rewards criterion satisfaction. During RL, the policy does not receive the reference answer, and rewards do not use direct reference matching or embedding similarity (Appendix A). Rubric refinement aims to avoid penalizing a correct explanation solely because it uses different terminology. For example, Figure 2 revises a criterion that requires the exact term “over-squashing” so that an explanation of the underlying information bottleneck also receives credit.

We agree, however, that criteria derived from a published solution can still favor that solution over other valid approaches. Our rubrics are surrogate training signals, not an exhaustive characterization of scientifically valid answers. We add two complementary evaluations: a human assessment of response quality and a controlled audit of whether rubrics reward valid alternatives and penalize invalid answers.

Preliminary human evaluation. We sample 48 held-out questions, with eight questions from each of the six decision categories. Two ML PhD students who were not involved in this work independently compare base-model and PESTO responses based on correctness, feasibility, and completeness. Model identities, reference answers, and rubrics are hidden. Each response pair is evaluated twice, with its presentation order reversed, to account for potential order effects. We also obtain preferences from two model judges using the same blinded comparison protocol. We average valid judgments across the two presentation orders within each question, then average across questions, giving each question equal weight.

JudgePrefer PESTOPrefer base
GPT-5.556.25%43.75%
Claude Opus-570.83%29.17%
Researcher 160.42%39.58%
Researcher 264.58%35.42%

Both human evaluators prefer PESTO more often than the base model. This provides preliminary evidence that the gains are reflected in independently assessed response quality.

AuthorsOpenReview ↗

Response to Reviewer 8qxC (3/3)

Rubric validation. We next test whether the rubrics recognize different valid solutions while rejecting answers with substantive errors. We evaluated 100 questions, each paired with three answers: the paper-derived reference, an alternative solution, and an invalid control. To construct the alternatives, we searched the literature for different approaches applicable to the same problem, adapted them to the question’s constraints, and retained supporting citations. Two independent ML researchers verified their correctness and applicability. We constructed invalid controls by introducing a substantive error into an otherwise plausible answer, such as violating a stated constraint or reversing a necessary algorithmic step; the researchers also verified these errors. The rubric judge scored each answer individually, without source or validity labels, using comparable answer lengths and formatting. We counted a triplet as passing only when both valid answers scored at least 0.7, their scores differed by at most 0.1, and the invalid control scored at most 0.1.

The rubrics satisfied all three conditions on 71/100 questions. Across the full sample, 1/100 valid alternatives received near-zero reward (≤0.1), while 16/100 invalid controls received scores above 0.1. The rest failures are score differences greater than 0.1 between the two valid answers. These results support the rubrics’ ability to recognize valid alternatives in this sample, while identifying remaining errors in scoring invalid answers and consistently rewarding different valid solutions.

5. Scope of the execution-based evidence and the role of Codex

Most evaluations have predefined objectives. The multi-stage evaluation uses Codex for selection and execution, considers one problem, and reports the best of eight trajectories.

We agree that these evaluations answer different questions and should be interpreted separately. The local-guidance and fixed-objective settings test whether paper-derived supervision transfers to measurable execution outcomes. They do not establish general open-ended research ability. The improvements include mean outcomes, as reported in Table 3:

Fixed-objective metricBasePESTO
Mean nanoGPT validation loss ↓3.44313.2691
Mean GRPO held-out accuracy ↑38.23%45.80%

These results support the practical usefulness of the learned research guidance beyond rubric scores. The multi-stage paper-extension experiment addresses a further question: whether the advisor can use its selected earlier decisions and experimental feedback to guide subsequent stages.

The Codex component is held fixed across the advising models. Both models use the same selection-and-execution protocol and candidate budget. In the open-ended setting, the advisor produces 16 candidates at each stage, Codex selects and implements a candidate, and the resulting context and experimental feedback are passed to the advisor. The comparison therefore measures performance when the advising model is changed within a shared research system. It does not establish that the trained model can independently select, implement, and execute an entire research project.

Additional proposal audit. We also examine the mechanisms proposed across a larger sample of advisor outputs. We sampled 16 additional proposals each from Base and the PESTO checkpoint using the same Engram paper, source code, and generation settings, retaining all 32 outputs. A qualitative assistant audit identified different emphases:

Proposal characteristicBaseUpdated PESTO
Adjusts edit strength by layer0/168/16
Requires gradient information9/164/16
Filters reference/context covariance directions7/164/16

For example, PESTO proposals include forget-versus-retain energy scaling, while Base proposals include confidence-weighted covariance estimation. We will include all 32 proposals and the item-level analysis in the appendix, including technical errors identified in both sets.

The bootstrap confidence interval currently reported in Appendix F quantifies evaluation uncertainty conditional on the selected trajectory; These analyses can strengthen the evidence for consistency within this research problem, while the experiment remains a single-paper case study. We will keep that scope explicit.

We hope these additions and the clarified scope address your concerns and would appreciate your reconsideration of the rating.

[1] Wang, Yuhan, et al. “PaperGym: Rubric-Centered Evolution for Research-Plan Generation.” arXiv:2608.31119, 2026.

Reviewer_L1fWOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns scientific papers into short-horizon research-decision tasks and trains on them with RL. Each paper is parsed into a dependency graph of sections, deterministic rules pick context and answer nodes, an LLM rewrites each pair into a self-contained question and reference answer, and ten binary rubric criteria are generated and then refined against responses from three frontier models. Training is SFT on one half of ~10k papers, then CISPO RL on the other half, with reward equal to the fraction of rubric criteria satisfied as judged by gpt-oss-20B. Data, graphs and rubrics are promised on release.

Critical strengths
  1. SFT followed by RL, with the right ablations. The paper runs SFT-only (Gemini and Qwen rewrites), CPT, RL-only and SFT + RL from the same base. SFT-only and CPT do not move the average while RL does, which separates the contribution of rubric-guided RL from simply seeing more paper text. This is what makes Result 1 credible.

  2. Fixed-objective evaluation on nanoGPT and GRPO Scoring proposals by implementing them and measuring validation loss or held-out accuracy under a fixed budget is a clean way to test whether the model’s research decisions transfer beyond rubric satisfaction. Using two very different objectives (pretraining loss, RL post-training reward) makes the check more convincing than a single downstream task.

Critical weaknesses
  1. The QA pipeline is LLM only, no human at any stage. Questions, reference answers, rubrics, refinement and the judge are all LLMs. No human calibration on the LLM judge reported either. Rubric RL is known to be gameable.

  2. Tasks are built from one paper at a time. Most real research builds on a body of work, so the literature to gap task rewards recovering the gap the authors already framed, from their own prior-work summary. Interesting as an experiment, but far from the actual scientist workflow. Worth stating as a limitation.

Questions for authors
  1. The held-out set is ICLR 2026 orals and training is 2024-2025 arXiv. Most ICLR 2026 papers were on arXiv in late 2025, often under different titles. Lines 191-192 say matching records were removed; was the match on title, content similarity, or arXiv id, and how many were removed?

  2. Research decisions usually have several valid next steps. Does the rubric penalize a valid alternative direction? The 2.18 multiplicity in Appendix B.1 is about related questions, not alternative answers to the same question. Any estimate of how often a correct but different proposal scores near zero?

2 comments
AuthorsOpenReview ↗

Response to Reviewer L1fW (1/2)

Thank you for your thoughtful review. We appreciate your recognition of the controlled training comparisons and the execution-based evaluations on nanoGPT and GRPO. In response to your concerns, we add an independent human assessment and clarify the training/evaluation filtering procedure and the scope of our single-paper tasks. We also address your question about how the rubrics treat valid alternative answers below.

1. Human assessment and the reliability of automated rewards

The QA pipeline is entirely LLM-based, with no human calibration of the judge. Rubric RL can be gameable.

We now add an independent human assessment of response quality. We sample 48 held-out questions, with eight questions from each of the six decision categories. Two ML PhD students who were not involved in this work independently compare base-model and PESTO responses based on correctness, feasibility, and completeness. Model identities, reference answers, and rubrics are hidden. Each response pair is evaluated twice, with its presentation order reversed, to account for potential order effects. We also obtain preferences from two model judges using the same blinded comparison protocol.

We average valid judgments across the two presentation orders within each question, then average across questions, giving each question equal weight.

JudgePrefer PESTOPrefer base
GPT-5.556.25%43.75%
Claude Opus-570.83%29.17%
Researcher 160.42%39.58%
Researcher 264.58%35.42%

Both human evaluators prefer PESTO more often than the base model. This provides preliminary evidence that the gains are reflected in independently assessed response quality.

2. Single-paper tasks and the scope of literature-to-gap supervision

Real research builds on a body of work, whereas these tasks recover a gap from one paper's own prior-work summary.

Although individual QA examples are extracted from single papers, we collect them across a large paper pool, yielding an average of 2.18 reference answers per question (Appendix B.1). This provides supervision from multiple answers to shared research questions. However, we agree that this exposure alone may not be sufficient for comprehensive literature synthesis and independent novelty assessment. We will clarify this limitation and discuss independent literature retrieval and joint reasoning across multiple papers as next steps in the future-work section.

AuthorsOpenReview ↗

Response to Reviewer L1fW (2/2)

3. Training/evaluation filtering and the reconstructed exclusion count

Was matching based on title, content similarity, or arXiv ID, and how many papers were removed?

Our corpus-selection filter uses normalized-title matching, with capitalization and punctuation normalized. Reapplying this rule to the archived candidate pool identifies 174 out of 10k candidate papers whose normalized titles match evaluation-paper titles; all 174 are absent from the final training dataset. We will specify the matching rule and the origin of this count in Section 3.1. This check establishes the absence of the title-matched candidates; it can miss earlier versions with substantially changed titles. We will make that limitation explicit rather than claiming that title matching rules out every form of source-paper overlap.

4. Valid alternative answers and the interpretation of answer multiplicity

Does the rubric penalize a valid alternative direction? The 2.18 multiplicity concerns related questions, not alternative answers to the same question. How often does a correct but different proposal score near zero?

As clarified above, the multiplicity of 2.18 reported in Appendix B.1 is the average number of reference answers associated with the same question. Our dataset therefore includes questions with multiple answers. We will make this definition explicit in the revision.

Can the rubric recognize different valid solutions? We evaluated 100 questions, each paired with three answers: the paper-derived reference, an alternative solution, and an invalid control. To construct the alternatives, we searched the literature for different approaches applicable to the same problem, adapted them to the question’s constraints, and retained supporting citations. Two independent ML researchers verified their correctness and applicability. We constructed invalid controls by introducing a substantive error into an otherwise plausible answer, such as violating a stated constraint or reversing a necessary algorithmic step; the researchers also verified these errors.

The rubric judge scored each answer individually, without source or validity labels, using comparable answer lengths and formatting. We counted a triplet as passing only when both valid answers scored at least 0.7, their scores differed by at most 0.1, and the invalid control scored at most 0.1.

The rubrics satisfied all three conditions on 71/100 questions. Across the full sample, 1/100 valid alternatives received near-zero reward (≤0.1), while 16/100 invalid controls received scores above 0.1. Failures also included score differences greater than 0.1 between the two valid answers. These results support the rubrics’ ability to recognize valid alternatives in this sample, while identifying remaining errors in scoring invalid answers and consistently rewarding different valid solutions.

We hope the preliminary human assessment, the targeted rubric audit, the additional proposal analysis, and the clarified filtering procedure and limitations address your concerns, and would appreciate your reconsideration of the rating.

Reviewer_J1n7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PESTO (Paper-Extracted Surrogate Task Optimization), a framework for training LLMs to guide scientific research using supervision derived from existing papers. The authors construct research dependency graphs from scientific papers, extract shorter-horizon decision-making tasks (e.g., identifying research gaps, proposing methods, and designing experiments), and generate criterion-level rubrics for reinforcement learning. They train a Qwen3.6-27B model on tasks derived from over 10,000 papers using SFT followed by rubric-guided RL.

The proposed approach improves performance on held-out research QA tasks and is evaluated in several execution-based settings, including MLS-Bench, NatureBench, nanoGPT/GRPO optimization, and an open-ended extension of an ICML 2026 paper. The authors report improvements in downstream experimental outcomes, suggesting that training on paper-derived surrogate tasks can transfer to more open-ended research scenarios.

Critical strengths
  1. Training LLMs to make research decisions without requiring expensive execution-based rewards is an important problem.

  2. I appreciate that the authors go beyond evaluating synthetic QA tasks and include execution-based benchmarks where generated ideas are implemented and tested.

  3. The results are promising. PESTO improves average held-out rubric satisfaction from 41.02% to 54.02% and achieves improvements across most evaluated execution-based tasks.

Critical weaknesses
  1. The contribution of the proposed training components is unclear due to missing ablations. The proposed framework involves several components, including research graph construction, decision-task extraction, rubric generation and refinement, and SFT followed by RL. However, the paper does not provide controlled ablations to establish which components are responsible for the improvements. In particular, Table 2 shows that RL-only achieves 53.66% average rubric satisfaction, nearly identical to PESTO's 54.02%. While PESTO may offer better training efficiency, the results do not establish the necessity of its SFT stage. More importantly, there are no comparisons against simpler task-construction strategies that do not require dependency graphs, or against unrefined rubrics. Since these are the main methodological contributions, I believe such ablations are essential.

  2. The execution-based evaluations are encouraging but insufficient to establish robust improvements in open-ended research ability. Although the paper evaluates several execution-based settings, the experiments remain relatively limited. The fixed-objective evaluations cover only two research environments, with eight proposals per model. The best-score improvements are relatively small: nanoGPT loss decreases from 3.2532 to 3.2373, while GRPO accuracy increases from 47.6% to 48.4%. The open-ended evaluation considers only one paper and reports the best of eight trajectories.

  3. The relationship between rubric optimization and genuine research quality remains insufficiently validated. The training questions and rubrics are automatically constructed from published research decisions, with criteria derived from the reference answers. However, multiple valid approaches may exist for the same research problem, and rewarding alignment with a particular published solution may discourage alternative but equally promising directions. While rubric refinement attempts to reduce brittle criteria, the paper does not provide a human validation study assessing whether the generated rewards reliably reflect the quality of research decisions.

Questions for authors
  1. Given that RL-only achieves nearly identical QA performance to PESTO, how do the two approaches compare on downstream execution-based tasks and under matched training compute?
5 comments
AuthorsOpenReview ↗

Response to Reviewer J1n7 (1/3)

Thank you for your thoughtful feedback. We appreciate your recognition of the importance of learning research decisions without requiring execution-based rewards. To address your concerns, we now provide a compute-matched comparison with RL-only training, additional downstream evaluations, a controlled rubric-refinement ablation, a comparison with training on PaperGym data, and an independent human assessment. We will also clarify the scope of the execution-based evidence and the limitations of reference-derived rubrics.

1. The role of SFT: computational efficiency and downstream performance

RL-only achieves nearly identical QA performance to PESTO. How do they compare downstream and under matched training compute?

We agree that Table 2 shows comparable QA performance for PESTO and data-matched RL-only training. PESTO uses SFT on questions derived from one half of the paper corpus, followed by rubric-guided RL on the other half. The data-matched RL-only baseline instead applies RL to the full corpus. Both substantially improve over the same Qwen3.6-27B base model: average rubric score increases from 41.02% to 53.66% with RL-only and 54.02% with PESTO. The main motivation for the SFT stage is computational efficiency. Following the FLOP-accounting convention in [1], we can approximate generation cost as 2NDrollout2ND_{\mathrm{rollout}} and training cost as 6NDtrain6ND_{\mathrm{train}}, where NN is the number of model parameters and DrolloutD_{\mathrm{rollout}} and DtrainD_{\mathrm{train}} are the generated and training token counts. Assuming equal response lengths DD and one training pass per response, RL with eight responses per question costs approximately 8(2ND+6ND)=64ND8(2ND+6ND)=64ND, compared with 6ND6ND for SFT on one demonstration. Therefore, the estimated compute reduction from pure-RL to PESTO is: 1−0.5×6ND+0.5×64ND64ND=45.3%.1-\frac{0.5\times6ND+0.5\times64ND}{64ND}=45.3\%. The compute-matched comparison provides direct empirical support for this choice. At the matched compute budget, PESTO achieves 54.02% average rubric score, compared with 50.97% for RL-only, a 3.05% improvement. The table reports absolute success rates, with gains over the same base model in parentheses.

ModelDir.→IdeaLit.→GapGap→Meth.Meth.→Exp.Res.→Anal.MixedAvg. success
RL-only, compute-matched40.48 (+11.16)58.59 (+12.75)52.87 (+9.55)44.41 (+10.81)48.63 (+4.98)52.30 (+9.21)50.97 (+9.95)
RL-only, data-matched43.02 (+13.70)61.52 (+15.68)56.45 (+13.13)46.50 (+12.90)54.11 (+10.46)53.83 (+10.74)53.66 (+12.64)
PESTO42.14 (+12.82)60.34 (+14.50)56.15 (+12.83)48.64 (+15.04)55.32 (+11.67)54.73 (+11.64)54.02 (+13.00)

We additionally evaluate the data-matched RL-only checkpoint on the downstream GRPO task:

ModelMean accuracyBest accuracy
Base38.2%47.6%
RL-only, data-matched41.1%47.2%
PESTO45.8%48.4%

In this comparison, RL-only improves only mean accuracy over the base model, while PESTO achieves the highest mean and best accuracy. These results provide initial downstream evidence supporting the combined training recipe. We are extending this comparison to additional execution-based tasks and will include them to the paper as well.

That said, we would like to mention that RL-only runs trained for much longer compute budgets can also be a valid checkpoint, but in a compute-matched setting SFT+RL seems better.

[1] Snell, Charlie, et al. “Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.” arXiv:2408.03314, 2024.

AuthorsOpenReview ↗

Response to Reviewer J1n7 (2/3)

2. Rubric refinement and alternative task construction

Missing comparisons against unrefined rubrics and simpler task construction without dependency graphs.

Rubric-refinement ablation. We first quantify the extent to which refinement changes the generated rubrics:

MeasureResult
Rubrics with at least one criterion changed97.1%
Individual criteria changed51.2%
Completely unchanged rubrics2.9%

We then compare RL training with raw and refined rubrics, starting from the same SFT checkpoint and evaluating both models at training step 50 and step 100 on held-out QA tasks:

Training conditionRL stepHeld-out QA score
Shared SFT initialization034.16%
RL with raw rubrics5036.60%
RL with raw rubrics10038.85%
RL with refined rubrics5037.68%
RL with refined rubrics10040.78%

Both conditions improve over the shared initialization, and refined rubrics yield a 1.08% advantage at step 50 and 1.93% advantage at step 100 over raw rubrics. This provides evidence that refinement benefits training.

Comparison with an alternative paper-derived task-construction pipeline. PaperGym [2] constructs research-plan generation tasks by deriving questions from a paper's goals and background, reference answers from its methods and experiments, and rubrics covering methodological innovation and experimental design. Because its reported models use backbones up to 8B parameters, we train our same Qwen3.6-27B base model with RL on PaperGym data under a matched compute budget. On held-out ICLR 2026 QA tasks, the results are:

ModelDir.→IdeaLit.→GapGap→Meth.Meth.→Exp.Res.→Anal.MixedAvg. success
RL on PaperGym data, compute-matched38.84 (+9.52)51.02 (+5.18)50.41 (+7.09)44.46 (+10.86)49.53 (+5.88)51.23 (+8.14)48.75 (+7.73)
RL on PESTO data, compute-matched40.48 (+11.16)58.59 (+12.75)52.87 (+9.55)44.41 (+10.81)48.63 (+4.98)52.30 (+9.21)50.97 (+9.95)
PESTO42.14 (+12.82)60.34 (+14.50)56.15 (+12.83)48.64 (+15.04)55.32 (+11.67)54.73 (+11.64)54.02 (+13.00)

At matched compute, RL on PESTO data exceeds RL on PaperGym data by 2.22% on average; the full PESTO recipe achieves a 5.27% advantage.

We also compare these models on four execution-based tasks. The table reports mean execution scores; higher is better for every metric:

Task / metricRL on PaperGym dataPESTO
Discrete RL value estimation / score0.034260.15439
Pooling and aggregation / score0.211060.31556
Cross-modal single-cell protein data matching / accuracy0.807150.83345
Metal-organic framework property prediction / R²0.872610.90512

These results support the effectiveness of PESTO's overall task-construction and training pipeline relative to a relevant paper-derived alternative.

[2] Wang, Yuhan, et al. “PaperGym: Rubric-Centered Evolution for Research-Plan Generation.” arXiv:2608.31119, 2026.

AuthorsOpenReview ↗

Response to Reviewer J1n7 (3/3)

3. Scope and interpretation of the execution-based results

The fixed-objective evaluations cover only two research environments, with eight proposals per model, and the best-score improvements are relatively small.

Our fixed-objective evaluation covers both research environments released by Si et al., nanoGPT and GRPO. We additionally evaluate seven local-guidance task families from MLS-Bench and NatureBench, as well as a multi-stage extension of AI Engram. We are extending the execution-based comparisons to additional tasks and will include the completed results in the revised manuscript. For AI Engram, we will also report outcomes across all eight trajectories per model, including mean performance, variability across runs, and the fraction satisfying the prespecified success criteria.

The execution gains are also visible in the mean outcomes. The original fixed-objective evaluation in Table 3 reports the following results across eight proposals per model:

MetricBasePESTO
Mean nanoGPT validation loss ↓3.44313.2691
Best nanoGPT validation loss ↓3.25323.2373
Mean GRPO held-out accuracy ↑38.23%45.80%
Best GRPO held-out accuracy ↑47.60%48.40%

The mean nanoGPT loss decreases from 3.4431 to 3.2691, and mean GRPO accuracy improves by 7.57%. These results show that the observed benefit extends beyond the best-performing proposal. As discussed in Section 4, Result 3, part of the mean improvement reflects poorly performing base-model proposals. Such proposals consume experimental budgets, making mean execution quality relevant to the usefulness of a research advisor.

We also clarify that the AI Engram experiment is a best-of-eight case study demonstrating multi-stage research guidance, rather than evidence of robust generalization across many open-ended research problems. Developing reproducible research environments from a broader set of recent papers is an important next step toward evaluating that stronger claim.

4. Rubric validity and independent human evaluation

Rubric optimization may discourage valid alternatives, and there is no human validation study.

PESTO rewards criterion satisfaction. During RL, the policy does not receive the reference answer, and rewards do not use direct reference matching or embedding similarity (Appendix A). Rubric refinement aims to avoid penalizing a correct explanation solely because it uses different terminology. For example, Figure 2 revises a criterion that requires the exact term “over-squashing” so that an explanation of the underlying information bottleneck also receives credit.

We agree, however, that criteria derived from a published solution can still favor that solution over other valid approaches. Our rubrics are surrogate training signals, not an exhaustive characterization of scientifically valid answers. To assess whether the improvements extend beyond automated rubric scores, we add an independent evaluation of response quality.

Preliminary human evaluation. We sample 48 held-out questions, with eight questions from each of the six decision categories. Two ML PhD students who were not involved in this work independently compare base-model and PESTO responses based on correctness, feasibility, and completeness. Model identities, reference answers, and rubrics are hidden. Each response pair is evaluated twice, with its presentation order reversed, to account for potential order effects. We also obtain preferences from two model judges using the same blinded comparison protocol.

We average valid judgments across the two presentation orders within each question, then average across questions, giving each question equal weight.

JudgePrefer PESTOPrefer base
GPT-5.556.25%43.75%
Claude Opus-570.83%29.17%
Researcher 160.42%39.58%
Researcher 264.58%35.42%

Both human evaluators prefer PESTO more often than the base model. This provides preliminary evidence that the gains are reflected in independently assessed response quality.

The additional experiments provide evidence for the efficiency of the SFT–RL recipe, the benefit of rubric refinement during early training, the effectiveness of PESTO relative to an alternative paper-derived training pipeline, and improvements in independently assessed response quality. We hope these results and clarifications address your concerns and would appreciate your reconsideration of the rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Gentle Reminder to Review Our Rebuttal

Hi Reviewer, thank you for your time and thoughtful feedback. We have submitted our rebuttal addressing your comments and would appreciate it if you could review our responses when you have a chance. Thank you again for your consideration.

AuthorsOpenReview ↗

Gentle Reminder to Review Our Rebuttal

With the discussion deadline fast approaching, we would like to kindly follow up on our rebuttal and additional experimental results. We would greatly appreciate it if you could take a look at our response to your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.