- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PaperGym to turn research papers into RL training environments for research-plan generation. PaperGym firstly decomposes each paper into four stages so the question (from goal and background) and the rubric (from method and experiments) come from disjoint sections. Then, the processed data are used for OPSD and GRPO training. Experiments demonstrate that trained model not only beats SFT, but also above the far larger Kimi K2.6.
Critical strengths
The paper proposed a practical pipeline to turn papers into RL training data, with experiments demonstrating the effectiveness of the synthesized data.
Critical weaknesses
- Why Qwen3-235B-A22B specifically for the map-step extraction, and was it chosen by comparing against other, newer Qwen variants or simply as a convenient default? Since the entire dataset depends on this extractor, a short sensitivity check would keep the pipeline from being a black box.
- The evaluation is confined to the small Qwen3 series (1.7B-8B), so it is unclear whether the two-stage recipe transfers to other model families (e.g., Llama), larger models, or newer generations such as Qwen 3.5. Given that the method is presented as a general training paradigm, one cross-family or larger-scale result would go a long way toward supporting that claim.
Questions for authors
Please see weaknesses.
3 comments
Thank you for detailed responses. I will raise my score to 3 after taking new results into consideration.
Thank you for the follow-up and for taking the time to review our responses. We appreciate your confirmation that your concerns have been addressed.
Dear Reviewer fCmG,
Thank you for your time and for the constructive suggestions. Our responses are below.
1. Choice of the data-extraction model.
Qwen3-235B-A22B was chosen to balance extraction capability and cost. Two experiments support this choice:
Evidence in the original paper. In Table 4 of the original paper, replacing Qwen3-235B-A22B with Qwen3-8B for extraction lowers PaperGym-Innov/Design by only 0.97/0.67, indicating that the extraction stage has limited influence on the final trained model.
Further comparison with DeepSeek and Kimi (Appendix A.4). We build three data variants that use Qwen3-235B-A22B, Kimi-K2.6, or DeepSeek-V4-Flash-0731 for the map-step extraction, with rubric generation held fixed. Each variant trains Qwen3-1.7B for 200 OPSD steps without GRPO, and all are evaluated under identical settings. Bold marks the best result per column; Avg. is the mean over the five benchmarks.
All three variants improve over the base model on every benchmark. The spread across extractors is modest (Avg. 24.61, 23.81, 23.60). Qwen3-235B-A22B meets the instruction-following and long-context requirements of section-level extraction at moderate inference cost, so we retained it for the 20k corpus.
2. Transfer beyond the Qwen3 family.
We have added a cross-family experiment (Appendix A.5): Hunyuan-1.8B-Instruct trained with the same OPSD→GRPO recipe on PaperGym-20k, with Qwen3-4B supplying the reward. All variants use identical evaluation settings, with thinking disabled and DeepSeek-V4-Flash-0731 as the judge.
The two-stage model raises the average from 11.44 to 12.67 and improves all five benchmarks over the base. This suggests that both PaperGym-20k and the two-stage training recipe remain effective beyond the Qwen3 family.
Note. These experiments use DeepSeek-V4-Flash-0731 rather than the original DeepSeek-V4-Flash (Preview, 2026.4.24), so absolute scores are not directly comparable.
Thank you again for your thorough review. We hope our rebuttal has addressed your concerns, and we would be happy to answer any further questions.
Sincerely,
PaperGym Authors