- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ReSpec, which trains a model to generate code repositories from research papers. To provide process supervision missing from paper-code pairs, the authors first use reference repositories to select papers, then have a strong teacher model synthesize approximately 430,000 process-supervision examples for SFT of the pretrained Sigma-MoE-Tiny. A judge model then scores generated plans and individual code files against automatically constructed checklists, providing rewards for GRPO training. The complete system scores 70.6 on PaperBench Code-Dev and 68.2 on Paper2CodeBench under the authors’ modified evaluation protocol.
Critical strengths
-
Training substantially improves model performance. With the inference workflow fixed, RL increases the PaperBench Code-Dev score from 60.9 to 70.6, with higher mean scores on 18 of the 20 test papers.
-
The authors construct a large process-supervision dataset. It contains approximately 430,000 training examples from 6,172 papers, covering stages from paper analysis to code writing.
Critical weaknesses
-
Given that relatively complete reference implementations have already been selected, why not train directly on these paper-code pairs rather than regenerate the trajectories and code? Is there a direct-training baseline using the original pairs that shows what the additional data-construction steps contribute?
-
Training is evaluated only on Sigma, whereas the main-table baselines use other backbone models. Could the proposed training be applied to a backbone also used by the baselines, such as Qwen, with methods compared from the same starting checkpoint? This would help separate the contribution of the method from differences in the underlying models.
Questions for authors
-
The added value of regenerating all supervision rather than using existing reference code remains unclear. The authors collect and filter relatively complete implementations, but use them only to select papers before a teacher generates new trajectories and code. Missing process traces do not make these reference implementations unusable for training.
-
Code scoring lacks validation under the actual RL setup. In the JudgeEval test reported in Table 4, the system retrieves relevant code files from a repository and gives them to Qwen3-32B to judge whether a requirement is implemented. Its judgments achieve 85.9% accuracy against human labels. During RL, however, Qwen3-32B receives only one criterion and the target file, without the other relevant files. The paper does not report scoring accuracy under this single-file setup, so the 85.9% result does not establish the reliability of the code rewards used in training.
-
How is the quality of the teacher-generated training data ensured? Section 3.1 filters reference repositories, which does not guarantee that the newly generated trajectories and code are correct.
-
In addition to the SFT + full RL result in Table 8, could the authors report three settings: (i) SFT followed by planning-only RL, (ii) SFT followed by code-only RL, and (iii) full RL directly from the pretrained Sigma checkpoint, without SFT? Keeping the RL training budget matched to the full method would help assess the contribution of each RL branch and the benefit of SFT initialization.
6 comments
Response to Reviewer zsBR (Q4)
Q4: Planning-only RL, code-only RL, and RL without SFT
We thank the reviewer for suggesting these additional ablations. We have added planning-only and code-only RL from the same SFT checkpoint. The complete comparison is shown below. A2 and A3 are the newly added experiments.
| ID | Training setting | Two-pass refinement | PaperBench Code-Dev | Evaluation protocol |
|---|---|---|---|---|
| A0 | Pretrained Sigma | Off | 0.0 | Original Sigma checkpoint for planning and code |
| A1 | SFT | On | 60.9 ± 2.8 | SFT checkpoint for planning and code |
| A2 | SFT + planning-only RL | On | 66.8 ± 1.9 | Plan-RL checkpoint for planning; SFT checkpoint for code |
| A3 | SFT + code-only RL | On | 69.1 ± 0.4 | SFT checkpoint for planning; Code-RL checkpoint for code |
| A4 | SFT + full RL | Off | 65.6 ± 0.4 | Full-RL checkpoint for planning and code |
| A5 | SFT + full RL | On | 70.6 ± 0.6 | Full-RL checkpoint for planning and code |
Because planning and code writing share a backbone, branch-specific RL can degrade the capability that is not optimized. We therefore use the planning-only RL checkpoint to generate the plan and the SFT checkpoint to generate code; conversely, for code-only RL, the SFT checkpoint generates the plan and the code-only RL checkpoint writes the code. This protocol measures the optimized capability while holding the other capability at the SFT level.
We did not run full RL directly from pretrained Sigma because the pretrained checkpoint lacks the tool-use behavior required by the reproduction scaffold and cannot produce valid agent trajectories for this RL setup. Consistent with this, pretrained Sigma scores 0.0 under DeepCode-Base. SFT establishes the tool-call behavior and task-specific initialization needed before checklist-based RL.
Manuscript revision. We have revised the corresponding sections of the manuscript; please see the highlighted changes in the revised PDF.
Thank you for the additional experiments. The single-file judge evaluation and the planning/code RL ablations address my corresponding concerns. I have two remaining clarification requests:
- Could you provide more details about the setup of the direct-SFT experiment? It is currently unclear to me how direct SFT was performed and what exactly the reported 35.3% and 71.8% scores evaluate.
- The 20-paper audit is small and does not describe the filtering of the 430K synthetic SFT examples. Does this mean that all teacher-generated examples were retained without post-generation quality filtering? If filtering was performed, please report the details.
Response to Reviewer zsBR
Critical Weakness 1 and Q1: Why not train directly on the original paper–code pairs?
We thank the reviewer for suggesting a direct-supervision comparison. We conducted a preliminary direct-supervision experiment using the original reference implementation as the target code while generating the missing intermediate reasoning traces. The prompt instructed the teacher not to reveal that it had access to the reference implementation, and a second LLM checked the resulting trajectories for information leakage.
On the nine papers available in this preliminary experiment, this approach achieved 35.3%, compared with 71.8% for regenerated supervision on the same papers. We observed that human-written repositories often contain unfinished code in nonessential components, including
TODOcomments andpassstatements. When used directly as supervision, these patterns were usually transferred to generated components that required complete implementations, reducing reproduction quality. Regenerating the trajectories and code produced supervision that was more consistent with the target agent workflow.Critical Weakness 2: Training is evaluated only on Sigma, which confounds the method with the backbone
We thank the reviewer for this suggestion and agree that training a second backbone from the same starting checkpoint would provide a valuable controlled comparison. However, repeating the 430K-sample SFT stage and RL training on Qwen requires substantial additional computation, and we were unable to complete this experiment during the rebuttal period.
Our current analysis therefore focuses on the training effect within Sigma: pretrained Sigma, SFT, planning-only RL, code-only RL, and full RL are compared from the corresponding shared checkpoints. These comparisons isolate the improvement obtained through training within the same backbone, while the resulting compact model achieves performance comparable to a Qwen model with substantially more active parameters. We regard cross-backbone training as an important direction for further validation.
Q2: Accuracy of the code judge under the actual single-file RL input
We thank the reviewer for identifying the need to validate the code judge under the exact single-file input used during RL. The file-scoped checklists used during code RL are constructed so that each requirement is assigned to one target file and can be verified from that file. Repository-level evaluation and file-scoped RL test the same core judge capability: determining whether code evidence satisfies a given criterion. The only difference is how the evidence is obtained: repository evaluation uses a separate retrieval stage to identify relevant files, whose contents are then provided to the LLM, whereas RL directly supplies the target file. Since retrieval occurs outside the LLM context, both settings ultimately present the same task to the LLM.
We have additionally evaluated Qwen3-32B under the exact single-file input format used during RL. We randomly sampled 60 files containing 1,109 distinct criterion–file judgment pairs. We first evaluated GPT-5.6-sol on JudgeEval, where it achieved 91.20% accuracy against the human labels. We then used GPT-5.6-sol as the reference annotator for the 60-file sample, providing it with only one criterion and the corresponding target file. Against these reference labels, Qwen3-32B achieved 95.87% accuracy. This experiment directly evaluates the reward judge under the input conditions used in code RL.
Q3: Quality control for teacher-generated training data
The synthetic SFT supervision contains some noise. To assess its quality, we sampled 20 papers and evaluated both their selected reference repositories and the corresponding teacher-generated repositories using the same repository-scoring procedure. The selected reference repositories achieved a mean score of 72.9%, while the teacher-generated repositories achieved 74.6%.
We retain the teacher-generated trajectories because the SFT stage primarily teaches the model the decomposed reproduction workflow and tool-use behavior; discarding all supervision for a paper because of a limited number of noisy outputs would also remove its valid process supervision.
The subsequent RL stage provides criterion-level quality control for generated plans and code files. The GRM assigns a positive reward to a criterion only when the required detail or functionality is fully satisfied. Under the same inference workflow, this stage improves PaperBench Code-Dev from 60.9 ± 2.8 for SFT to 70.6 ± 0.6 for SFT+RL.