acceptodds
Under review as a conference paper at ICLR 2027

PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Abstract

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.

Primary area:
datasets and benchmarks
Keywords:
PaperGym, Research Planning, AI Scientists, Reinforcement Learning, Rubric as Rewards, OPSD, GRPO

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fCmGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PaperGym to turn research papers into RL training environments for research-plan generation. PaperGym firstly decomposes each paper into four stages so the question (from goal and background) and the rubric (from method and experiments) come from disjoint sections. Then, the processed data are used for OPSD and GRPO training. Experiments demonstrate that trained model not only beats SFT, but also above the far larger Kimi K2.6.

Critical strengths

The paper proposed a practical pipeline to turn papers into RL training data, with experiments demonstrating the effectiveness of the synthesized data.

Critical weaknesses
  1. Why Qwen3-235B-A22B specifically for the map-step extraction, and was it chosen by comparing against other, newer Qwen variants or simply as a convenient default? Since the entire dataset depends on this extractor, a short sensitivity check would keep the pipeline from being a black box.
  2. The evaluation is confined to the small Qwen3 series (1.7B-8B), so it is unclear whether the two-stage recipe transfers to other model families (e.g., Llama), larger models, or newer generations such as Qwen 3.5. Given that the method is presented as a general training paradigm, one cross-family or larger-scale result would go a long way toward supporting that claim.
Questions for authors

Please see weaknesses.

3 comments

Dear Reviewer fCmG,

Thank you for your time and for the constructive suggestions. Our responses are below.

1. Choice of the data-extraction model.

Qwen3-235B-A22B was chosen to balance extraction capability and cost. Two experiments support this choice:

  • Evidence in the original paper. In Table 4 of the original paper, replacing Qwen3-235B-A22B with Qwen3-8B for extraction lowers PaperGym-Innov/Design by only 0.97/0.67, indicating that the extraction stage has limited influence on the final trained model.

  • Further comparison with DeepSeek and Kimi (Appendix A.4). We build three data variants that use Qwen3-235B-A22B, Kimi-K2.6, or DeepSeek-V4-Flash-0731 for the map-step extraction, with rubric generation held fixed. Each variant trains Qwen3-1.7B for 200 OPSD steps without GRPO, and all are evaluated under identical settings. Bold marks the best result per column; Avg. is the mean over the five benchmarks.

    Model / trainingRubricHub ScienceResearchPlanGen-MLResearchQAInnovDesignAvg.
    Qwen3-1.7B32.166.8052.6110.247.2121.81
    w/ OPSD (Qwen3-235B-A22B)34.127.9258.4312.889.6824.61
    w/ OPSD (Kimi-K2.6)33.858.1556.3011.619.1323.81
    w/ OPSD (DeepSeek-V4-Flash-0731)33.067.3656.2912.428.8923.60

    All three variants improve over the base model on every benchmark. The spread across extractors is modest (Avg. 24.61, 23.81, 23.60). Qwen3-235B-A22B meets the instruction-following and long-context requirements of section-level extraction at moderate inference cost, so we retained it for the 20k corpus.

2. Transfer beyond the Qwen3 family.

We have added a cross-family experiment (Appendix A.5): Hunyuan-1.8B-Instruct trained with the same OPSD→GRPO recipe on PaperGym-20k, with Qwen3-4B supplying the reward. All variants use identical evaluation settings, with thinking disabled and DeepSeek-V4-Flash-0731 as the judge.

Model / trainingRubricHub ScienceResearchPlanGen-MLResearchQAInnovDesignAvg.
Hunyuan-1.8B-Instruct18.602.3830.922.832.4511.44
w/ OPSD17.932.2332.713.672.4011.79
w/ OPSD + GRPO18.792.3934.144.343.6812.67

The two-stage model raises the average from 11.44 to 12.67 and improves all five benchmarks over the base. This suggests that both PaperGym-20k and the two-stage training recipe remain effective beyond the Qwen3 family.

Note. These experiments use DeepSeek-V4-Flash-0731 rather than the original DeepSeek-V4-Flash (Preview, 2026.4.24), so absolute scores are not directly comparable.

Thank you again for your thorough review. We hope our rebuttal has addressed your concerns, and we would be happy to answer any further questions.

Sincerely,

PaperGym Authors

Thank you for detailed responses. I will raise my score to 3 after taking new results into consideration.

Thank you for the follow-up and for taking the time to review our responses. We appreciate your confirmation that your concerns have been addressed.

Reviewer_MQJ9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PaperGym, a framework for training language models to generate research plans. It separates research goals and background from methods and experimental design to construct questions, reference solutions, and evaluation rubrics. Training uses rubric-conditioned on-policy self-distillation (OPSD), followed by rubric-reward GRPO. The paper releases PaperGym-20k and two benchmarks, reporting improvements across three Qwen3 scales and several external benchmarks.

Critical strengths
  1. Useful resources and a relevant research problem. The dataset explicitly addresses both methodological innovation and experimental design, providing a potentially valuable resource for research-plan generation.

  2. Consistent benchmark improvements. The proposed schedule improves performance across 1.7B, 4B, and 8B models. Evaluation includes temporally held-out papers and external benchmarks, while stage-ordering ablations provide evidence for the chosen schedule.

  3. Meaningful data-quality comparisons. The matched-recipe comparison against RubricHub Science and multi-judge preference evaluation support the utility of the training corpus. Detailed construction and scoring prompts also improve reproducibility.

Critical weaknesses

Although the paper is easy to understanding, I have several concerns as follows:

  1. Unclear tool access limits interpretation of the evaluation. Research planning often requires literature retrieval and factual verification to assess novelty and feasibility. The paper does not establish whether tools were enabled or consistently configured across models. If evaluation is tool-free, the results primarily measure closed-book proposal generation, providing limited evidence of scientifically grounded planning.

  2. Rubric satisfaction is not sufficiently validated as research quality. Evaluation relies on automated judges without expert assessment or execution-based validation. Reference-grounded criteria may favor reproducing the source paper while penalizing valid alternatives. Multi-model preference evaluation is helpful, but does not establish whether the rubrics reliably distinguish scientifically sound plans from superficially compliant ones.

  3. Some examples or case comparisons are missing in the paper to further improve the clarification.

Questions for authors

No further questions.

3 comments

Dear Reviewer MQJ9,

Thank you for your time and for the helpful comments on evaluation scope and rubric validity. Our responses are below.

1. Tool access and evaluation scope.

We agree that retrieval and factual verification play an important role in research planning. In our setting, the generation process does not involve tool calls: we regard a complete automated research workflow as a multi-agent system, and this paper focuses on the research plan generation stage, leaving retrieval and document parsing to separate agents.

  • Closed-book evaluation. Consistent with this focus on research plan generation, all models are evaluated under the same closed-book setting with tools disabled, similar to RubricHub and ResearchPlanGen. Assigning the entire pipeline to a single model would demand much more capacity, and fine-tuning such a model exceeds the resources available to us.

  • Retrieval evaluation (Appendix B.5). To test the method when external evidence is available, we simulate a retrieval-filtering-generation pipeline. DeepSeek-V4-Flash-0731 generates search queries, DeepSeek's Search API returns arXiv papers, and the retrieved text is split into chunks; the five passages most relevant to the question are retained as evidence and added to the model's input context.

    Avg3 rubric scores (%), all judged by DeepSeek-V4-Flash-0731, are shown below. None denotes no retrieval.

    ModelInnov: NoneInnov: Top-5Design: NoneDesign: Top-5
    Qwen3-1.7B6.477.276.717.67
    w/ OPSD + GRPO12.2311.7510.7310.32
    Qwen3-4B10.6310.7111.0212.03
    w/ OPSD + GRPO15.7314.8115.1715.91
    Qwen3-8B14.9311.8513.9615.24
    w/ OPSD + GRPO18.3317.5817.3917.93

    Fine-tuned models score higher on both tasks across all three scales and both conditions, with Top-5 gains of 2.65–5.73 points. Retrieval itself sometimes lowers scores, for example when responses import methods from mismatched settings and reduce rubric coverage.

2. Rubric validity and expert assessment.

We agree that expert judgment is more convincing evidence that rubric scores reflect research quality than the multi-model win rates we originally used. We therefore add an expert evaluation on 50 computer-science answer pairs from Qwen3-4B and its trained variant (Appendix A.3). Three computer-science researchers form the expert panel and provide majority decisions, which we compare with a three-LLM panel (DeepSeek-V4-Flash-0731, GLM-5.1, and Kimi-K2.6) and with rubric-based decisions, where DeepSeek-V4-Flash-0731 scores both answers of a pair against the rubric three times and the answer with the higher average score (Avg3) is recorded as the winner.

Win rates and agreement across all 50 answer pairs (%):

EvaluatorTrained win / loss / tieWin rateAgreement with human panelAgreement with LLM panel
Human panel39 / 3 / 878—74
LLM panel42 / 7 / 18474—
Rubric32 / 6 / 12645860

All three setups favor the trained model, at 78% (human panel), 84% (LLM panel), and 64% (rubric), counting no-majority panel decisions and equal rubric scores as ties. Across all pairs, the expert and LLM panels agree on 74% (37/50) of cases, and rubric decisions agree with human judgments on 58% (29/50) and with the LLM panel on 60% (30/50). If we count only the pairs where both the panel and the rubric name a winner instead of a tie, agreement is 26 of 33 pairs (78.79%) with the human panel and 30 of 37 (81.08%) with the LLM panel. These results support the validity of both rubric-based evaluation and LLM win-rate judging, and indicate that the rubrics capture aspects of research quality that experts also recognize.

3. Concrete comparison.

We thank the reviewer for this practical suggestion. We have added case studies in Appendix C.3, where base and fine-tuned answers to the same question appear side by side.

Note. These experiments use DeepSeek-V4-Flash-0731 rather than the original DeepSeek-V4-Flash (Preview, 2026.4.24), so absolute scores are not directly comparable.

Thank you again for your thorough review. We hope our rebuttal has addressed your concerns, and we would be happy to answer any further questions.

Sincerely,

PaperGym Authors

Reviewer_MQJ9OpenReview ↗

Thanks for the response

Thank you for the detailed experiments and response. I have raised the rating to 3.

Thank you for the follow-up and for taking the time to review our responses. We appreciate your confirmation that your concerns have been addressed.

Reviewer_i5PeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper targets research-plan generation. Specifically, the paper turns arXiv papers into a training environment. Its diagnosis is that existing pipelines draw the question and the grading criteria from the same content, so paraphrasing earns reward. The fix decomposes each paper into four stages and synthesizes the question only from Goal + Background and the reference answer only from Method + Experimental Design. Ten atomic binary criteria per instance span methodological innovation and experimental design. Training then uses the same rubric twice: first as privileged context for an on-policy self-distillation teacher, then as the GRPO reward, graded by a frozen copy of the base model. Outputs are PaperGym-20k, two held-out benchmarks, and a two-stage recipe evaluated on Qwen3-1.7B/4B/8B.

Critical strengths
  1. The diagnosis is right and the fix attacks the cause. If question and criteria come from the same text, no reward shaping repairs it; drawing them from disjoint sections removes the shortcut at its source.

  2. Ablation coverage is broad: privileged-information choice, reward mixing, stage ordering across all three scales and five benchmarks.

Critical weaknesses
  1. Appendix C.1 contradicts the headline claim about the two-stage schedule. The main text says OPSD+GRPO "achieves the highest score on every test set at every scale". However, in Fig. 6, the two-stage method loses on 3 criteria. Your own explanation is reward hacking: "producing longer responses to match more rubric items". The two evaluations disagree because Table 1's judge rates rubric coverage per criterion, exactly what GRPO optimizes, while Figure 6's judges rate plan quality. The results suggest that either the training does not generalize or the criteria set is incomplete. Also, I find that R_gen carries 30% of the reward and is described as enforcing "completeness, specificity, soundness, efficiency, and ethical safety". Soundness is therefore nominally covered, and soundness is the single largest regression. So either 30% is too little weight, or binary verdicts from a self-grading base model do not detect soundness failures. An ablation with R_gen removed, and one with its weight raised, would separate these.

  2. The trained models beat one of nine baselines on average (Table 1): ahead of Intern-S1-Mini (28.34), behind Rebicon-Preview(30B-A3B) (40.02), S1-VL-RL (41.02) and all six proprietary models (48.11–54.53). The authors should consider using a bigger model to show the proposed method's potential to surpass Rebicon-Preview. Besides, §4.2 is written: "outperforms Intern-S1-Mini and Rebicon-Preview on Innov and Design" omits S1-VL-RL beating you on Design (23.68 vs 21.88), and "surpasses S1-VL-RL on ResearchQA" omits Rebicon-Preview beating the model there.

  3. The 3.73% leakage figure comes from DeepSeek-V4-Flash judging, per criterion, whether it "can be directly inferred from the question alone" (A.2) — and DeepSeek-V4-Flash also generated PaperGym's rubrics, while the competing datasets' rubrics came from other pipelines. There is no human annotation, expert spot-check or human agreement study anywhere in the paper, for the leakage verdicts, the rubric quality, the Method/Experiment split, or any benchmark score; Figure 2's inter-model agreement can only show the judges are correlated, not correct.

  4. The 4B and 8B models self-verify, while the 1.7B model "whose scoring ability is insufficient" is scored by the 4B model. Note also that the 1.7B row, graded by a strictly stronger model, shows the largest gain (+5.56). So why not grade the 4B model by the 8B model, only to meet the requirement of OPSD?

  5. Methods in Tables should be cited.

Questions for authors

Please refer to the weaknesses.

6 comments

Dear Reviewer i5Pe,

Thank you for your time and for this detailed analysis. Our responses are below.

1. Plan-quality gains under GRPO and the reward ratio.

  • Evaluation prompt refinement (Appendix C.1).

    We have revised this sentence in the main text. GRPO does raise plan quality, but it also tends to add examples and numerical illustrations (Appendix C.3); a holistic judge can read them as unsupported, which lowers the soundness win rate in Appendix C.1 without reflecting the plan's actual value. A research plan's scientific value lies in the relevance and novelty of its proposed approach, so we refined the evaluation prompt accordingly.

    We repeat the comparison with a concise instruction that discourages speculative figures. All models receive it with thinking enabled, and three judges vote by majority. Win rates against the base (%):

    TrainingOverallGoal alignmentNovel insightSoundnessExecutionImpact
    OPSD76.6972.2960.7073.1776.3974.93
    OPSD→GRPO78.3072.4374.0572.2978.5977.71

    With this instruction, the two-stage model's overall win rate rises from 68.2% to 78.3% and it leads on four dimensions. The gain from GRPO is therefore real but was largely masked by presentation.

  • Reward weight (Appendix B.3).

    The 7:3 specialized:general ratio is the appropriate default. From the same 200-step OPSD checkpoint, we run 100 GRPO steps and ablate the ratio on both metrics. On rubric coverage, 7:3 gives 17.17/12.35 on PaperGym-Innov/Design, above 8:2 (14.63/10.86) and 6:4 (16.40/11.54). On win rate, it leads overall at 61.91, against 58.37 (10:0, i.e., R_gen removed), 57.06 (8:2), and 61.48 (6:4).

2. Comparisons with existing models.

We compare against S1-VL-RL and Rubicon-Preview because these baselines are fine-tuned from stronger base models, Qwen3-VL-32B-Thinking and Qwen3-30B-A3B respectively, whereas our largest fine-tuned model is only 8B. Even so, our 8B model outperforms S1-VL-RL on PaperGym-Innov and ResearchQA, and scores higher than Rubicon-Preview on PaperGym-Innov and PaperGym-Design.

In response to your concern, we have revised the corresponding statement in the paper. It now makes explicit the advantages of S1-VL-RL and Rubicon-Preview on some benchmarks, and states that a fair comparison of methods requires fine-tuning the same base model under matched evaluation settings. Doing so exceeds our current computational budget.

3. Data quality and validation.

  • Leakage judge. We replaced the single DeepSeek judge with a majority vote among GLM-5.1, Kimi-K2.6, and Qwen3.5-Plus, none of which generated the rubrics. Table 3 now reports 1.84% for PaperGym-20k versus 7.00–43.41% for other automated science datasets, with expert-designed ResearchRubrics (2.95%) and ResearcherBench (4.94%) as sanity checks.
  • Case examples. Appendix A.6 presents actual criterion-leakage cases together with the reasoning behind each verdict.
  • Expert assessment. In the expert study of Appendix A.3, human, LLM, and rubric evaluations of 50 Qwen3-4B answer pairs favor the trained model at 78%, 84%, and 64%, and rubric judgments agree with expert judgments on 78.79% (26/33) of the pairs where both give a winner. The study therefore supports the validity of both rubric-based evaluation and LLM win-rate judging, and indicates that the rubrics capture aspects of research quality that experts also recognize.

4. Choice of training verifier.

Qwen3-4B scores the 1.7B policy because 1.7B is less reliable in both format and grading correctness. Appendix B.4 evaluates scoring on 200 PaperGym examples. All columns are percentages: JSON compliance is the share of outputs in the required format, audited correctness is the verdict accuracy judged by DeepSeek-V4-Flash-0731, and agreement is the match rate with its independent score.

ScorerJSON complianceDS-audited correctnessAgreement with DS
Qwen3-1.7B56.5069.4270.20
Qwen3-4B89.5092.9794.78
Qwen3-8B83.5094.2495.45

Qwen3-1.7B is weaker in both instruction following and scoring quality, whereas Qwen3-4B and 8B are both strong and differ little, so we do not introduce an additional model as the scorer. These scorers serve GRPO only; OPSD learns from the rubric-conditioned self-teacher.

5. Citations.

We have added citations for methods and datasets in the relevant tables.

Note. These experiments use DeepSeek-V4-Flash-0731 rather than the original DeepSeek-V4-Flash (Preview, 2026.4.24), so absolute scores are not directly comparable.

Thank you again for your thorough review. We hope our rebuttal has addressed your concerns, and we would be happy to answer any further questions.

Sincerely, PaperGym Authors

Dear Reviewer i5Pe,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

The Authors

Dear Reviewer i5Pe,

As the discussion period is drawing to a close, we wanted to follow up on our response to your comments. If any concerns remain, we would be happy to clarify them before the deadline. If you feel that our responses have addressed your concerns, we would be grateful if you would consider raising your rating.

Thank you again for your time and thoughtful feedback.

Sincerely, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revision

Dear Area Chairs and Reviewers,

We thank you for the constructive reviews, and we appreciate the recognition of the following strengths:

  • Data construction. The question is synthesized from the research goal and background, while the rubric is derived from the method and experimental design, so the criteria cover both methodological innovation and experimental design; criterion leakage falls to 1.84%, against 7.00%–43.41% in existing automatically generated science rubric datasets.
  • Two-stage training. Both OPSD and GRPO improve the benchmark average over the base model on their own, and applying them in sequence yields the strongest results at every scale. Under expert and LLM-judge evaluation, the trained model is also preferred over its base model on research-plan generation.
  • Data quality. Under an identical training recipe, the model trained on PaperGym-20k wins 58.1% of three-way comparisons, against 28.2% for the RubricHub-trained counterpart and 13.7% for the untrained base model.
  • Ablations. Ablations on the data-construction pipeline, the reward design, and the training schedule all support the design choices of the paper.

During the rebuttal, we provided the following clarifications and additional validations:

  1. Expert assessment [i5Pe, MQJ9]. An expert study on 50 computer-science answer pairs (Appendix A.3) shows that human (78%), LLM (84%), and rubric (64%) judgments all favor the trained model; excluding ties, rubric decisions agree with the human panel's preference on 78.79% of pairs.
  2. Case study [i5Pe, MQJ9]. Appendix C.3 adds two side-by-side comparisons of base and fine-tuned answers to the same question, and Appendix A.6 presents concrete criterion-leakage cases with the reasoning behind each verdict.
  3. Data quality [i5Pe, fCmG].
    • The leakage assessment is repeated with a majority vote among three judges that did not generate the rubrics, yielding the revised rate of 1.84%.
    • The choice of Qwen3-235B-A22B for extraction is supported by comparisons with Qwen3-8B (Table 4), Kimi-K2.6, and DeepSeek-V4-Flash-0731 (Appendix A.4): all trained variants remain above the base model, and the differences in benchmark scores are limited.
  4. Soundness judge [i5Pe]. For research-plan generation, soundness is not the primary criterion: novelty and goal alignment carry more weight, and the gap between the two-stage model and OPSD reflects presentation rather than plan quality. In a closed-book setting, the judge cannot verify the figures GRPO adds and counts them as unsupported claims (Appendix C.3). With an instruction that discourages speculative figures, the model reaches 78.30 overall (OPSD: 76.69) and leads on four of five dimensions (Appendix C.1), confirming a genuine overall-quality gain from GRPO. We also propose a revised soundness criterion and will report its re-evaluation in a future version.
  5. Tool use [MQJ9]. In the main experiments, all models are evaluated closed-book and on equal terms. Appendix B.5 simulates a retrieval-filtering-generation pipeline, where trained models improve on both tasks at all three scales.
  6. Model extension [fCmG]. Appendix A.5 applies the same recipe to Hunyuan-1.8B-Instruct, which improves on every benchmark, consistent with the gains observed across the Qwen3 models.

We believe these additions address the reviewers' concerns and strengthen PaperGym as a principled data-construction pipeline and an effective training paradigm for research-plan generation.

Thank you again for your careful review and valuable suggestions.

Sincerely,

PaperGym Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.