acceptodds
Under review as a conference paper at ICLR 2027

ReSpec: Learning Paper Reproduction via Synthetic Supervision and Process Rewards

Abstract

Faithful paper-to-code reproduction requires translating a paper’s technical specifications into an implementation. Learning this capability requires intermediate supervision and fidelity feedback, which paper-repository pairs alone do not provide. We present ReSpec, a framework that automatically constructs synthetic supervision and checklist-based process rewards from papers. Its data engine selects 6,172 paper-repository pairs through implementation-fidelity assessment and synthesizes 430,428 examples for six reproduction sub-tasks. It also constructs paper-grounded checklists to define fine-grained rewards for the plan pipeline and file-scoped code writing, yielding 76,448 samples for parallel training through task-decomposed reinforcement learning. Using these paper-derived signals, we train the ReSpec model through supervised fine-tuning followed by reinforcement learning. On PaperBench Code-Dev, the ReSpec model equipped with the DeepCode-Base scaffold and two-pass plan refinement outperforms Qwen3.5-35B-A3B in the DeepCode-Base scaffold by 5.2 points (70.6 vs. 65.4) with one-sixth as many active parameters (0.5B), while approaching Claude-Sonnet-4.5 (73.5) and GLM-4.7 (75.0). These results demonstrate that automatically constructed process supervision and rewards can turn paper-to-code reproduction into a learned capability in a compact specialist.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI for Research, Paper-to-Code Reproduction, Scientific Reproducibility, Process Supervision, Data Synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zsBROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReSpec, which trains a model to generate code repositories from research papers. To provide process supervision missing from paper-code pairs, the authors first use reference repositories to select papers, then have a strong teacher model synthesize approximately 430,000 process-supervision examples for SFT of the pretrained Sigma-MoE-Tiny. A judge model then scores generated plans and individual code files against automatically constructed checklists, providing rewards for GRPO training. The complete system scores 70.6 on PaperBench Code-Dev and 68.2 on Paper2CodeBench under the authors’ modified evaluation protocol.

Critical strengths
  1. Training substantially improves model performance. With the inference workflow fixed, RL increases the PaperBench Code-Dev score from 60.9 to 70.6, with higher mean scores on 18 of the 20 test papers.

  2. The authors construct a large process-supervision dataset. It contains approximately 430,000 training examples from 6,172 papers, covering stages from paper analysis to code writing.

Critical weaknesses
  1. Given that relatively complete reference implementations have already been selected, why not train directly on these paper-code pairs rather than regenerate the trajectories and code? Is there a direct-training baseline using the original pairs that shows what the additional data-construction steps contribute?

  2. Training is evaluated only on Sigma, whereas the main-table baselines use other backbone models. Could the proposed training be applied to a backbone also used by the baselines, such as Qwen, with methods compared from the same starting checkpoint? This would help separate the contribution of the method from differences in the underlying models.

Questions for authors
  1. The added value of regenerating all supervision rather than using existing reference code remains unclear. The authors collect and filter relatively complete implementations, but use them only to select papers before a teacher generates new trajectories and code. Missing process traces do not make these reference implementations unusable for training.

  2. Code scoring lacks validation under the actual RL setup. In the JudgeEval test reported in Table 4, the system retrieves relevant code files from a repository and gives them to Qwen3-32B to judge whether a requirement is implemented. Its judgments achieve 85.9% accuracy against human labels. During RL, however, Qwen3-32B receives only one criterion and the target file, without the other relevant files. The paper does not report scoring accuracy under this single-file setup, so the 85.9% result does not establish the reliability of the code rewards used in training.

  3. How is the quality of the teacher-generated training data ensured? Section 3.1 filters reference repositories, which does not guarantee that the newly generated trajectories and code are correct.

  4. In addition to the SFT + full RL result in Table 8, could the authors report three settings: (i) SFT followed by planning-only RL, (ii) SFT followed by code-only RL, and (iii) full RL directly from the pretrained Sigma checkpoint, without SFT? Keeping the RL training budget matched to the full method would help assess the contribution of each RL branch and the benefit of SFT initialization.

6 comments
AuthorsOpenReview ↗

Response to Reviewer zsBR

Critical Weakness 1 and Q1: Why not train directly on the original paper–code pairs?

We thank the reviewer for suggesting a direct-supervision comparison. We conducted a preliminary direct-supervision experiment using the original reference implementation as the target code while generating the missing intermediate reasoning traces. The prompt instructed the teacher not to reveal that it had access to the reference implementation, and a second LLM checked the resulting trajectories for information leakage.

On the nine papers available in this preliminary experiment, this approach achieved 35.3%, compared with 71.8% for regenerated supervision on the same papers. We observed that human-written repositories often contain unfinished code in nonessential components, including TODO comments and pass statements. When used directly as supervision, these patterns were usually transferred to generated components that required complete implementations, reducing reproduction quality. Regenerating the trajectories and code produced supervision that was more consistent with the target agent workflow.

Critical Weakness 2: Training is evaluated only on Sigma, which confounds the method with the backbone

We thank the reviewer for this suggestion and agree that training a second backbone from the same starting checkpoint would provide a valuable controlled comparison. However, repeating the 430K-sample SFT stage and RL training on Qwen requires substantial additional computation, and we were unable to complete this experiment during the rebuttal period.

Our current analysis therefore focuses on the training effect within Sigma: pretrained Sigma, SFT, planning-only RL, code-only RL, and full RL are compared from the corresponding shared checkpoints. These comparisons isolate the improvement obtained through training within the same backbone, while the resulting compact model achieves performance comparable to a Qwen model with substantially more active parameters. We regard cross-backbone training as an important direction for further validation.

Q2: Accuracy of the code judge under the actual single-file RL input

We thank the reviewer for identifying the need to validate the code judge under the exact single-file input used during RL. The file-scoped checklists used during code RL are constructed so that each requirement is assigned to one target file and can be verified from that file. Repository-level evaluation and file-scoped RL test the same core judge capability: determining whether code evidence satisfies a given criterion. The only difference is how the evidence is obtained: repository evaluation uses a separate retrieval stage to identify relevant files, whose contents are then provided to the LLM, whereas RL directly supplies the target file. Since retrieval occurs outside the LLM context, both settings ultimately present the same task to the LLM.

We have additionally evaluated Qwen3-32B under the exact single-file input format used during RL. We randomly sampled 60 files containing 1,109 distinct criterion–file judgment pairs. We first evaluated GPT-5.6-sol on JudgeEval, where it achieved 91.20% accuracy against the human labels. We then used GPT-5.6-sol as the reference annotator for the 60-file sample, providing it with only one criterion and the corresponding target file. Against these reference labels, Qwen3-32B achieved 95.87% accuracy. This experiment directly evaluates the reward judge under the input conditions used in code RL.

Q3: Quality control for teacher-generated training data

The synthetic SFT supervision contains some noise. To assess its quality, we sampled 20 papers and evaluated both their selected reference repositories and the corresponding teacher-generated repositories using the same repository-scoring procedure. The selected reference repositories achieved a mean score of 72.9%, while the teacher-generated repositories achieved 74.6%.

We retain the teacher-generated trajectories because the SFT stage primarily teaches the model the decomposed reproduction workflow and tool-use behavior; discarding all supervision for a paper because of a limited number of noisy outputs would also remove its valid process supervision.

The subsequent RL stage provides criterion-level quality control for generated plans and code files. The GRM assigns a positive reward to a criterion only when the required detail or functionality is fully satisfied. Under the same inference workflow, this stage improves PaperBench Code-Dev from 60.9 ± 2.8 for SFT to 70.6 ± 0.6 for SFT+RL.

AuthorsOpenReview ↗

Response to Reviewer zsBR (Q4)

Q4: Planning-only RL, code-only RL, and RL without SFT

We thank the reviewer for suggesting these additional ablations. We have added planning-only and code-only RL from the same SFT checkpoint. The complete comparison is shown below. A2 and A3 are the newly added experiments.

IDTraining settingTwo-pass refinementPaperBench Code-DevEvaluation protocol
A0Pretrained SigmaOff0.0Original Sigma checkpoint for planning and code
A1SFTOn60.9 ± 2.8SFT checkpoint for planning and code
A2SFT + planning-only RLOn66.8 ± 1.9Plan-RL checkpoint for planning; SFT checkpoint for code
A3SFT + code-only RLOn69.1 ± 0.4SFT checkpoint for planning; Code-RL checkpoint for code
A4SFT + full RLOff65.6 ± 0.4Full-RL checkpoint for planning and code
A5SFT + full RLOn70.6 ± 0.6Full-RL checkpoint for planning and code

Because planning and code writing share a backbone, branch-specific RL can degrade the capability that is not optimized. We therefore use the planning-only RL checkpoint to generate the plan and the SFT checkpoint to generate code; conversely, for code-only RL, the SFT checkpoint generates the plan and the code-only RL checkpoint writes the code. This protocol measures the optimized capability while holding the other capability at the SFT level.

We did not run full RL directly from pretrained Sigma because the pretrained checkpoint lacks the tool-use behavior required by the reproduction scaffold and cannot produce valid agent trajectories for this RL setup. Consistent with this, pretrained Sigma scores 0.0 under DeepCode-Base. SFT establishes the tool-call behavior and task-specific initialization needed before checklist-based RL.

Manuscript revision. We have revised the corresponding sections of the manuscript; please see the highlighted changes in the revised PDF.

Thank you for the additional experiments. The single-file judge evaluation and the planning/code RL ablations address my corresponding concerns. I have two remaining clarification requests:

  1. Could you provide more details about the setup of the direct-SFT experiment? It is currently unclear to me how direct SFT was performed and what exactly the reported 35.3% and 71.8% scores evaluate.
  2. The 20-paper audit is small and does not describe the filtering of the 430K synthetic SFT examples. Does this mean that all teacher-generated examples were retained without post-generation quality filtering? If filtering was performed, please report the details.
Reviewer_R4sxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper-to-code reproduction means turning implementation requirements scattered across a paper's equations, figures and experimental protocols into an executable repository. ReSpec's bet is that this is trainable rather than something you orchestrate at inference time: the paper itself is both the supervision source and the reward source, and the paired repository is used only to pick papers. Three stages. 35,282 candidate paper-repository pairs are filtered to 6,172 by an importance-weighted criterion-satisfaction score ArepoA_{\text{repo}} with difficulty-aware thresholds. A Qwen3.5-397B-A17B teacher, run inside a DeepCode-derived scaffold extended with explicit file ordering and two-pass plan refinement, yields 430,428 SFT samples across six subtasks. Then task-decomposed GRPO trains plan generation and file-scoped code writing in parallel on offline-fixed contexts, rewarded by the satisfaction rate of LLM-constructed plan and file checklists judged by a single GRM. The backbone is a 20B-total / 0.5B-active Sigma MoE. Evaluation covers PaperBench Code-Dev (20 papers, human rubric) and all 90 Paper2CodeBench papers, 3 runs each, reaching 70.6±0.670.6 \pm 0.6 and 68.2±0.468.2 \pm 0.4.

Critical strengths
  • The data engine is disclosed better than almost anything in this area. Using the paper as the only supervision and reward source means the reward needs no reference implementation at training time, and fixing plan and file-writing contexts offline dodges both full-repository rollout cost and repository-level credit assignment. Appendices C/F/G give the full algorithms, prompts and an end-to-end case study, and the arithmetic survives recomputation: Table 9's per-paper rubric counts sum exactly to 3,671, Table 8's column averages match the 20 per-paper values, and the bolded cells are honestly the column maxima rather than the authors' own system.
  • Two-pass plan refinement is the one result here that replicates: +5.0+5.0 on ReSpec (18/20 papers) and +4.5+4.5 on an untrained Qwen3.5-35B-A3B (12/20). A second independent backbone is exactly the control the training claims lack. It's prompt-level, so other authors can use it tomorrow.
Critical weaknesses
  • Every headline gain comes from a scaffold-misaligned comparison, and the one aligned cell is in the paper's own Table 8. All baselines run DeepCode-Base, i.e. no refinement; ReSpec runs with two-pass refinement. Align them and an off-the-shelf Qwen3.5-35B-A3B gets 69.9±0.469.9 \pm 0.4 against ReSpec's 70.6±0.670.6 \pm 0.6, a 0.7-point gap at Welch t≈1.68t \approx 1.68. Not significant. The Abstract, Introduction and Conclusion quote only the misaligned 5.2-point margin, so 430K SFT samples and 300 GRPO steps buy nothing the prompt-level component does not already buy, and refinement is precisely the inference-time orchestration family the paper claims to be complementary to.
  • The Table 8 Average row mixes two definitions of ±\pm in one row, which destroys the noise baseline for the paper's only large effect. SFT reads 60.9±8.360.9 \pm 8.3 while the other four columns read ±0.4\pm 0.4 to ±0.7\pm 0.7. The 20 per-paper SDs in the SFT column sum to 166.6 and 166.6/20=8.33166.6/20 = 8.33, matching digit for digit, while the same operation on SFT+RL (Ref. off) gives 5.01 against a reported 0.4. Read against the printed ±8.3\pm 8.3, the Conclusion's "+9.7 points" for RL is 1.2 SD. No significance test, confidence interval or seed description appears anywhere, per-paper SDs reach ±21.6\pm 21.6 at n=3n = 3, and §4.2 claims one system "exceeds" another on a 0.1-point difference.
  • The second benchmark's metric is the training-time reward wearing a different hat, and nothing in the validation can rule out reward hacking. §4.1 replaces Paper2CodeBench's 5-point score with the same repository-scoring program used to filter training papers, which shares its functional form and construction pipeline with the RL reward. The pattern is diagnostic: no significant edge on the one independent metric (PaperBench's human rubric), +4.7+4.7 on the custom one. Every validation number is a recall, Table 4(a) and the 88.9% file-checklist figure both, so fabricated checklist items are invisible, and checklist length is the reward denominator.
    • The judge identity contradicts itself. §3.1's repository judge is Qwen3-32B, also the RL GRM, while §4.1 says benchmark scoring uses o3-mini-high. Never resolved.
    • The model that builds CplanC_{\text{plan}} is never named. "An LLM" in both places on p.6, absent from Algorithm 1, the G.2 prompts and the p.10 AI Use Statement, although plan samples dominate the reward at Table 6's 5:1 ratio.
    • §3.1 reports k=5k = 5 candidate files per criterion; the actual prompt (p.49) demands "exactly 3", as does the p.20 verification record. That bounds recall of xix_i in Eq. 1 and therefore every ArepoA_{\text{repo}}, the 6,172-pair selection and the 68.2.
    • The only human annotation (87.7% plan-judgement agreement) reports no annotator count, no volume and no IAA, and it checks GRM-human agreement rather than whether checklist items are grounded in the paper.
Questions for authors
  1. Please report Qwen3.5-35B-A3B, GLM-4.7 and Claude-Sonnet-4.5 under the full ReSpec scaffold, including two-pass refinement, on both PaperBench and Paper2CodeBench, with n≥5n \geq 5 runs and paired significance tests. Given that the one aligned cell you already report is 69.9±0.469.9 \pm 0.4 for an untrained Qwen3.5-35B-A3B against 70.6±0.670.6 \pm 0.6 for ReSpec, what is your estimate of the marginal contribution of the 430K-sample SFT stage and the 300 GRPO steps, as distinct from the refinement scaffold?
  2. Table 8, Average row: the SFT column reports 60.9±8.3 while the other four columns report ±0.4\pm 0.4, ±0.6\pm 0.6, ±0.7\pm 0.7 and ±0.4\pm 0.4, under a caption specifying that "scores are first averaged across the 20 papers within each run". The mean of the 20 per-paper SDs in the SFT column is 166.6/20=8.33166.6/20 = 8.33; the same computation on the SFT+RL (Ref. off) column gives 100.2/20=5.01100.2/20 = 5.01, not the reported 0.40.4. Which definition was used in each cell, and what is the between-run SD for the SFT column? Does the Conclusion's "+9.7 points" remain distinguishable from run-to-run noise once that number is supplied?
  3. Which judge produced the Paper2CodeBench scores (ReSpec 68.2, GLM-4.7 71.8, Qwen3.5-35B-A3B 63.5)? §3.1 specifies Qwen3-32B for the repository-scoring procedure, §4.1 states that benchmark scoring uses o3-mini-high, and Qwen3-32B is also the RL GRM. If the scoring judge is Qwen3-32B, what prevents the Paper2CodeBench result from being an in-sample measurement of the reward model?
3 comments
AuthorsOpenReview ↗

Response to Reviewer R4sx (weakness)

Critical Weakness 1: The headline comparison uses mismatched scaffolds

We thank the reviewer for the suggestion. The matched-scaffold comparisons are:

  • Without refinement: ReSpec model 65.6 ± 0.4 vs. Qwen3.5-35B-A3B 65.4 ± 0.7.
  • With refinement: ReSpec model 70.6 ± 0.6 vs. Qwen3.5-35B-A3B 69.9 ± 0.4.

Accordingly, we do not characterize the 0.7-point difference as a statistically advantage. Instead, we describe the trained compact model as achieving comparable performance with fewer active parameters in new version.

The within-backbone comparisons separately quantify the contributions of training and refinement. With the Sigma backbone and two-pass refinement fixed, the RL makes an improvement of 9.7 points. With model weights fixed, two-pass refinement improves the ReSpec model from 65.6 ± 0.4 to 70.6 ± 0.6.

Critical Weakness 2: The definition of the standard deviation in Table 8 is inconsistent

We thank the reviewer for pointing out the aggregation error in the SFT entry of Table 8. The printed value of 8.3 was obtained by averaging the 20 per-paper standard deviations, whereas the other entries in the Average row use the run-level aggregation specified in the caption.

Following the stated aggregation procedure, we first average the 20 paper scores within each run. The three SFT run averages are 64.14, 59.60, and 58.91, giving 60.9 ± 2.8, rather than 60.9 ± 8.3. We have corrected the table and clarified that each per-paper row summarizes repeated scores for that paper, while the Average row summarizes the three benchmark-level run averages.

The SFT+RL result with refinement remains 70.6 ± 0.6, so the mean difference remains 9.7 points. Using the benchmark-level run averages, an unpaired two-sided Welch comparison gives an unrounded difference of 9.74 points, a 95% confidence interval of [3.03, 16.44], and p = 0.024. We also replaced “exceeds” with “matching” for the 65.6 versus 65.5 comparison in Section 4.2.

Critical Weakness 3: Independence and validation of the Paper2CodeBench evaluation

We thank the reviewer for raising the important question of evaluation independence and validation. We clarify the criterion-level metric, the separation between training and evaluation, and the treatment of unsupported checklist items.

  • Paper2CodeBench assigns a single 1–5 integer score to an entire repository. Because score 1 covers failures ranging from missing core components to major implementation errors, without component-level or partial credit, repositories of different quality can collapse to the same score. We therefore retain the 90-paper benchmark while using criterion-level verification for finer-grained evaluation. We validated this choice on 30 repositories—10 implementations each for 3 papers. Across 10 experimental groups, the mean Spearman correlation with PaperBench human-rubric scores was 0.745 for our metric, compared with 0.303 and −0.058 for the reference-free and reference-based variants of the original five-point protocol, respectively.

  • Before evaluating the 90 Paper2CodeBench papers, we have filtered checklist items identified as unsupported or redundant. This filtering removed 10.63% of the initially generated items. We did not apply the same additional filtering step to the training-time checklists: in our analysis, it did not materially change the advantage allocation used by GRPO, while applying it to the full training corpus would substantially increase the construction cost. We now state this distinction explicitly.

sub-q 1: the repository and benchmark judges

The final Paper2CodeBench scores in Table 3 are produced by o3-mini-high. Qwen3-32B is used for data curation and as the code GRM during RL, but not for final benchmark scoring. We decouple the training reward signal from the final evaluator and reduce the risk of reward hacking against the benchmark judge. We choose Qwen3-32B for large-scale reward computation because it is open-weight and achieved the highest JudgeEval accuracy among the judges we tested.

sub-q 2: Model used to construct the plan checklist

The plan checklist CplanC_{plan} is generated by Qwen3.5-397B-A17B. We now state this distinction explicitly.

sub-q 3: Whether the number of retrieved files is three or five

We apologize for the typo. The implemented setting is k = 3 candidate files per criterion.

sub-q 4: Scale and agreement of the human annotation

We thank the reviewer for highlighting the need to report the annotation scale and inter-annotator agreement. The original annotation covers 20 PaperBench papers, with an average of 494 distinct checklist–plan judgment pairs per paper and 9,880 pairs in total. 4 annotators each labeled five non-overlapping papers; therefore, the original annotation did not support an inter-annotator agreement calculation. During the rebuttal period, all 4 annotators additionally labeled the same 246 checklist–plan pairs for one paper. Their inter-annotator agreement is Fleiss’ κ\kappa = 0.813.

AuthorsOpenReview ↗

Response to Reviewer R4sx (question)

Q1: Full scaffold, matched baselines, and the marginal training contribution

We thank the reviewer for suggesting this more controlled full-scaffold comparison. As an initial matched comparison, we evaluate Qwen3.5-35B-A3B with two-pass refinement on PaperBench. It scores 69.9 ± 0.4, compared with 65.4 ± 0.7 under DeepCode-Base without refinement. An unpaired two-sided Welch test gives a mean improvement of 4.51 points, a 95% confidence interval of [2.88, 6.14], t = 8.81 and p = 0.0031. For the ReSpec model, two-pass refinement improves the score from 65.6 ± 0.4 to 70.6 ± 0.6, a mean gain of 4.99 points. The corresponding run-level unpaired Welch test gives a 95% confidence interval of [3.81, 6.17], (t = 12.18), and (p < 0.001).

Because of the additional inference cost and the rebuttal timeline, we have not completed the requested five-run full-scaffold evaluation of Qwen3.5-35B-A3B, GLM-4.7, and Claude-Sonnet-4.5 on both benchmarks. We will add this broader comparison to the camera-ready version. Our current claim is limited to the demonstrated result: training turns the compact Sigma checkpoint into a paper-reproduction specialist whose matched-scaffold performance is comparable to a model with six times as many active parameters, while refinement provides a separate additional gain.

The training contribution is reported separately. SFT establishes the tool-use behavior and task-specific initialization required by the reproduction scaffold and reaches 60.9 ± 2.8 with refinement. Starting from that SFT checkpoint, full RL raises the score to 70.6 ± 0.6, a gain of 9.7 points under the same inference workflow.

Q2: Computation of the Average row in Table 8

Please see our response to Critical Weakness 2 above. The corrected SFT result is 60.9 ± 2.8, computed from the three benchmark-level run averages of 64.14, 59.60, and 58.91. All Average-row entries now use the same run-level definition. The SFT-to-SFT+RL mean difference remains 9.7 points; the run-level Welch comparison gives a 95% confidence interval of [3.03, 16.44] and p = 0.024.

Q3: Final scoring model for Paper2CodeBench

Please see our response to Critical Weakness 3, Sub-q 1. The Paper2CodeBench scores in Table 3 are produced by o3-mini-high. Qwen3-32B is used during data curation and as the RL code GRM, but it does not produce the final benchmark scores. We have made these roles explicit in Sections 3.1, 3.3, and 4.1.

Manuscript revision.

We have revised the corresponding sections of the manuscript; please see the highlighted changes in the revised PDF.

Thank you again for your careful and detailed review. We especially appreciate the attention you devoted to checking the reported results and methodological details. Your comments helped us identify an aggregation error and improve both the clarity and rigor of the manuscript.

We hope that our response and revisions—including the corrected run-level statistics, the matched-scaffold analysis, the clarification of the training and evaluation judges and checklist construction, and the additional validation and annotation details—have adequately addressed the concerns and questions you raised.

We would greatly appreciate any feedback on whether these revisions have clarified the relevant issues. If any concern remains, or if any part of our response would benefit from further explanation, we would be very happy to provide additional details and continue the discussion.

Reviewer_TmbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary
  • Proposes ReSpec, a framework that automatically constructs synthetic supervision and checklist-based process rewards from papers. Using this engine, created large scale datasets for training.
  • Using the above dataset, trained the ReSpec model by first doing SFT then RL, achieving significant gains on benchmarks such as PaperBench Code-Dev and Paper2CodeBench.
Critical strengths
  • The pipeline enables automatically constructing large scale dataset for the task under investigation. This could be quite useful for future research in this area.
  • RL training does improve the model's benchmark scores by 9.7 points, which is quite substantial and demonstrate effectiveness.
Critical weaknesses
  • The proposed data generation methodology is quite complex with many different components. While together they seem to improve model performance, the authors did not provide enough ablation and analysis on the effects of each components. The difference of finding a complex system that works and understanding why a complex system works is quite different.
  • The advantage against Qwen3.5-35B-A3B is quite limited when the settings matched exactly.
Questions for authors

none

4 comments
AuthorsOpenReview ↗

Response to Reviewer TmbQ

Critical Weakness 1: The pipeline is complex, but the contribution of each component is insufficiently analyzed

Thank you for emphasizing the importance of distinguishing overall effectiveness from component-level understanding. The original manuscript already isolates two effects: (1) SFT versus SFT+RL with two-pass refinement fixed, and (2) refinement on versus off with model weights fixed. We have now added planning-only RL and code-only RL to further separate the contributions of the two RL branches. The complete comparison is shown below. A2 and A3 are the newly added experiments.

IDTraining settingTwo-pass refinementPaperBench Code-DevEvaluation protocol
A0Pretrained SigmaOff0.0Original Sigma checkpoint for planning and code
A1SFTOn60.9 ± 2.8SFT checkpoint for planning and code
A2SFT + planning-only RLOn66.8 ± 1.9Plan-RL checkpoint for planning; SFT checkpoint for code
A3SFT + code-only RLOn69.1 ± 0.4SFT checkpoint for planning; Code-RL checkpoint for code
A4SFT + full RLOff65.6 ± 0.4Full-RL checkpoint for planning and code
A5SFT + full RLOn70.6 ± 0.6Full-RL checkpoint for planning and code

Plan-RL, Code-RL, and Full-RL all start from the same SFT checkpoint. Because planning and code generation share one backbone, optimizing only one branch can degrade the other capability; in particular, the code-only RL checkpoint could not reliably complete the planning task. We therefore evaluate the updated branch together with the SFT checkpoint for the other branch. This protocol isolates the capability optimized by each RL branch while retaining a functional end-to-end workflow.

The data engine itself contains several interdependent stages, as is common in multi-stage paper-to-code systems such as DeepCode and PaperCoder. A controlled ablation of its internal stages would require regenerating the training corpus for each variant and retraining a separate model. We were not able to complete that set of experiments during the rebuttal period, so the current analysis treats data generation as an integrated component and focuses the new ablations on the training and inference components.

Critical Weakness 2: The advantage over Qwen is limited under matched settings

We agree that the matched-setting comparison should be stated more precisely. We have therefore narrowed the main claim. Our result is that SFT and RL on data produced by the proposed data engine enable a compact specialist to achieve performance comparable to a substantially larger active backbone, while two-pass refinement provides a further improvement. Under the same DeepCode-Base scaffold without refinement, the ReSpec model scores 65.6 ± 0.4, compared with 65.4 ± 0.7 for Qwen3.5-35B-A3B, while using 0.5B rather than 3B active parameters. With two-pass refinement applied to both models, the corresponding scores are 70.6 ± 0.6 and 69.9 ± 0.4. We now present these as comparable matched-scaffold results with a smaller active computation footprint, rather than as a large performance advantage attributable to training alone.

Manuscript revision

We have revised the corresponding sections of the manuscript; please see the highlighted changes in the revised PDF.

Thank you again for your review and constructive feedback. We hope that our response, together with the newly added planning-only and code-only RL ablations and the revised characterization of the matched-setting comparison, has adequately addressed the concerns you raised.

We would greatly appreciate any feedback on whether these additions have clarified the relevant issues. If any concern remains, or if any part of our response would benefit from further clarification, we would be very happy to provide additional details and continue the discussion.

Thank you to the authors for addressing my concerns and providing additional experiments. I appreciate the effort put into the rebuttal. However, my primary concern is also with the data-generation pipeline rather than only the training pipeline. In particular, I would like to better understand the necessity and individual contributions of the following components:

  • Importance-weighted repository filtering
  • Difficulty-aware filtering thresholds
  • Six different synthetic supervision tasks
  • Two-pass plan refinement
  • Separate paper-grounded plan and file checklists
  • Checklist coverage audits and deduplication

While the additional experiments help clarify the effectiveness of the training procedure, they do not sufficiently establish whether these data-generation components are individually necessary or whether a simpler pipeline could achieve comparable results.

I believe the dataset and data-generation methodology have the potential to be particularly valuable to the community. However, the contribution of each component has not been sufficiently analyzed to the extent I had hoped for, making it difficult to assess which design choices are essential. Therefore, I will maintain my overall rating while increasing my soundness score, in recognition of the additional evidence and clarifications provided by the authors.

Thank you for clarifying the concern. We provide controlled comparisons at three levels: data construction, inference, and training.

At the data-construction level, we compared workflow-aligned regenerated supervision with Direct-SFT using the original reference code. Under the same scaffold, inference workflow, evaluation protocol, and 9-paper PaperBench Code-Dev subset, Direct-SFT achieved 35.3%, whereas regenerated supervision achieved 71.8%. The Six different synthetic supervision tasks are inherited from the original DeepCode pipeline and cover successive stages of the agent workflow. Together, they provide the intermediate process supervision required for repository construction rather than functioning as six independent architectural modules. The Separate paper-grounded plan and file checklists are generated within this pipeline from the corresponding planning and file-writing outputs: plan checklists evaluate paper-level reproduction completeness, while file checklists evaluate file-level implementation.

At the inference level, the ReSpec-added Two-pass plan refinement improved ReSpec from 65.6 ± 0.4 to 70.6 ± 0.6 and Qwen3.5-35B-A3B from 65.4 ± 0.7 to 69.9 ± 0.4. At the training level, planning-only RL and code-only RL achieved 66.8 ± 1.9 and 69.1 ± 0.4, respectively, compared with 60.9 ± 2.8 for SFT and 70.6 ± 0.6 for full RL. These results show that both the planning and file-level reward channels provide useful training signals.

The remaining components also serve specific quality-control functions. Importance-weighted repository filtering prioritizes repositories that satisfy core implementation requirements, while Difficulty-aware filtering thresholds calibrate selection across papers with different implementation difficulty. Checklist coverage audits and deduplication ensure that important paper requirements are represented while preventing repeated criteria from disproportionately affecting rewards.

Because these operations change the corpus composition or reward specification, isolating each one requires constructing a matched corpus and retraining the corresponding SFT and RL models rather than simply disabling it at inference time. This full set of corpus reconstructions and retraining runs could not be completed during the rebuttal period, so we prioritized the representative controlled comparisons above.

We sincerely thank the reviewer again for the time and care devoted to reviewing our work and for the constructive feedback. In the revised manuscript, we will clarify the role of each component and its corresponding experimental evidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair and Reviewers,

We thank the Area Chair and reviewers for their careful feedback. The initial reviews recognized several central strengths of ReSpec. TmbQ and zsBR highlighted its automatic construction of large-scale process supervision—430,428 examples from 6,172 papers across six reproduction tasks—and the fixed-workflow RL gain from 60.9 to 70.6. R4sx emphasized the detailed disclosure of the data engine, algorithms, prompts, and end-to-end case study, and noted that paper-derived rewards and offline-fixed contexts avoid reference implementations during training, full-repository rollout costs, and repository-level credit assignment. The reviewer also highlighted the effectiveness of two-pass refinement on both ReSpec and an independent Qwen backbone, indicating its broader reusability.

During the discussion, we focused on addressing the following primary concerns. The corresponding clarifications and new analyses have been incorporated into the revised manuscript, with changes highlighted in red.

  • Training, backbone capacity, and refinement (TmbQ, R4sx, zsBR). Under matched DeepCode-Base without refinement, ReSpec scores 65.6 ± 0.4, versus 65.4 ± 0.7 for Qwen3.5-35B-A3B, while using 0.5B versus 3B active parameters. With the same refinement, the scores are 70.6 ± 0.6 and 69.9 ± 0.4. We therefore claim comparable matched-scaffold performance with one-sixth as many active parameters, not superiority over Qwen. Within Sigma, with refinement fixed, RL improves SFT from 60.9 ± 2.8 to 70.6 ± 0.6, with higher means on 18/20 papers. After correcting the Table 8 SFT standard deviation to use run-level benchmark averages, the three-run Welch comparison gives a 9.74-point difference, 95% CI [3.03, 16.44], and (p=0.024). Refinement separately adds 5.0 points for ReSpec and 4.5 for Qwen (Sections 4.2–4.3; Table 8).

  • Training components and regenerated supervision (TmbQ, zsBR). From the same SFT checkpoint, planning-only RL reaches 66.8 ± 1.9, code-only RL 69.1 ± 0.4, and joint RL 70.6 ± 0.6. For each branch ablation, the updated checkpoint performs the optimized stage and SFT performs the other. Pretrained Sigma scores 0.0 because it lacks the required tool-use behavior, making SFT initialization necessary. Under the same workflow on the same nine-paper subset, direct SFT scores 35.3%, versus 71.8% for regenerated supervision. A separate 20-paper diagnostic gives 74.6% for teacher-generated repositories and 72.9% for selected references, providing preliminary support for regeneration (Section 4.3; Table 8).

  • Evaluation independence and reward reliability (R4sx, zsBR). Final Paper2CodeBench scores are produced by o3-mini-high, while Qwen3-32B is used for data curation and as the code GRM, separating the final evaluator from the training reward model. Filtering removed 10.63% of unsupported or redundant evaluation criteria. On 30 implementations, our metric has mean Spearman correlation 0.745 with PaperBench human-rubric scores, versus 0.303 and −0.058 for the original protocols. Under the exact single-file RL input, Qwen3-32B achieves 95.87% agreement with GPT-5.6-sol labels on 1,109 criterion–file pairs from 60 files; GPT-5.6-sol achieves 91.20% JudgeEval accuracy against human labels. For plan rewards, we report 9,880 judgments across 20 papers and 246 shared judgments with Fleiss’ ($ \kappa=0.813 $). We also identify Qwen3.5-397B-A17B as the plan-checklist generator, correct the candidate-file limit to (k=3), and clarify the Table 8 aggregation and judge roles (Sections 3.1, 3.3, 4.1, 4.4, and 4.5).

Reviewer zsBR then requested details of direct SFT and SFT-data filtering. We clarified that the teacher generated scaffold-aligned trajectories from each paper and reference implementation, reference code replaced the generated code targets, and a second LLM corrected leaked or inconsistent reasoning. Deterministic filters removed failed or structurally invalid runs and examples, eliminating 942 of 431,370 examples and retaining 430,428 (99.78%); no judge-based semantic threshold was applied, and the 20-paper audit was diagnostic. Reviewer zsBR stated that the added evidence addressed their main concerns and raised the rating from 2 to 3. Future work will pursue broader validation and robust semantic filtering.

We thank all reviewers again. The rebuttal provides evidence that, under a fixed Sigma workflow, both RL branches contribute to a substantial observed gain, while under matched scaffolds ReSpec performs comparably to Qwen with one-sixth as many active parameters. We hope the revised evidence, corrected reporting, and additional analyses support the acceptance of ReSpec.

Best regards,

Authors of Submission 55818

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.