- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProgramGym constructs 1,648 source-free program-reproduction environments and introduces an analysis–implementation–review agent. On ProgramBench, SFT raises Qwen3.8-27B's mean Tests Passed from 34.87% to 42.84%; Program-Agent reaches 55.71% and 56.25% with the base and SFT models.
Critical strengths
- The offline environments, reference executables, documentation, and 665,512 retained tests make ProgramGym a substantial resource for training program-reimplementation agents (Appendix B).
- Program-Agent substantially improves the same Qwen backbone (Table 2). The base-model improvement uses fewer output tokens than Claude Code (Appendix D.2).
Critical weaknesses
- I do not find sufficient evidence for reliable behavioral supervision. Reference agreement, coverage, empty-dummy rejection, and static checks do not establish whether the suites distinguish plausible faulty implementations (Eq. 3; Figure 17). The SFT gain demonstrates data usefulness, but does not validate this central property of the proposed verifier.
- The training evidence falls short of the goal of faithful reproduction. Under Claude Code, SFT improves the mean but reduces Almost Resolved from 3.5% to 1.5% and Resolved from 0.5% to 0%; with Program-Agent, SFT adds only 0.54 percentage points (Table 2). The authors acknowledge these trends, but the benefit of training for high-fidelity reproduction beyond the scaffold remains limited.
Questions for authors
- Could you validate the retained suites against representative faulty implementations to assess whether the proposed checks produce reliable behavioral supervision?
- What evidence supports improved high-fidelity reproduction from training, beyond the partial-reproduction gains already shown?
4 comments
W2/Q2: Teacher supervision and high-fidelity reproduction(2/2)
Thanks a lot for your detailed feedback. We really appreciate your recognition of our work and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper.
For W2 & Q2: Evidence of improved high-fidelity reproduction from training.
SFT outcomes depend on both the training tasks and the teacher trajectories [1]. Program-Env provides executable program-reproduction tasks and behavioral verifiers for selecting trajectories. Our original SFT used DeepSeek-V4-Flash-0731 with Claude Code as the teacher and retained any rollout with a non-zero test pass rate. This admitted many partially successful demonstrations, whose quality may limit the high-fidelity behavior learned through SFT.
The table below summarizes the teacher's score distribution on ProgramBench and among the retained Program-Env training rollouts.
| Dataset | [0.00, 0.25) | [0.25, 0.50) | [0.50, 0.75) | [0.75, 1.00] | [0.95, 1.00] | Total |
|---|---|---|---|---|---|---|
| ProgramBench | 31 (15.50%) | 37 (18.50%) | 76 (38.00%) | 56 (28.00%) | 9 (4.50%) | 200 (100.00%) |
| ProgramGym | 217 (13.17%) | 227 (13.77%) | 367 (22.27%) | 837 (50.79%) | 403 (24.45%) | 1648 (100.00%) |
The [0.95, 1.00] column is a subset of [0.75, 1.00].
Only 4.50% of the teacher's ProgramBench solutions reach a 95% test pass rate. Among the retained training rollouts, 24.45% reach this threshold, while 75.55% remain below it. This supervision profile may help explain the limited high-fidelity gains of the original SFT model.
Using the same teacher, we selected 1,204 rollouts with a test pass rate ≥50% and repeated SFT (see the Training experiments table in our W1 & Q1 response). With Claude Code fixed, Almost Resolved rises to 4.00%, above the base model's 3.50%, while Resolved returns to 0.50%, matching the base model. This provides evidence of a training gain in near-complete reproduction and supports Program-Env's use for verifier-guided trajectory selection.
We will release the constructed environments and data to support further research on post-training for program reproduction.
[1] Negin Raoof et al. OpenThoughts-Agent: Data Recipes for Agentic Models. arXiv:2606.24855, 2026. https://arxiv.org/abs/2606.24855.
Revision: We have added the stricter-filtering SFT results to Table 2 and the corresponding analysis to Section 6.2, Effectiveness of ProgramGym (RQ1).
Thank you again for your feedback. Please let us know if any concerns remain unaddressed. We would be happy to discuss them further.
Dear Reviewer 3mzt,
We sincerely appreciate your time and effort in reviewing our work and providing valuable feedback.
We have carefully addressed your concerns through detailed responses and additional experiments, and have updated our manuscript accordingly. We would be grateful if you could take another look at our rebuttal at your earliest convenience and share any further feedback or remaining concerns. We would be happy to provide additional clarification if needed.
Thank you again for your time and consideration. We look forward to hearing from you soon.
Best, The Authors of Submission #4472
Dear Reviewer 3mzt,
We have made substantial efforts to address your concerns, including additional experiments and detailed responses.
As the author–reviewer discussion is about to end, we would greatly appreciate it if you could review our responses and reconsider your rating if you find that your concerns have been adequately addressed.
Thank you for your time and consideration.
Best,
The Authors of Submission #4472
W1/Q1: Verifier correctness, discrimination, and training utility(1/2)
Thanks a lot for your detailed feedback. We really appreciate your recognition of our work and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper.
We address this concern from three angles.
1. Engineering practice. We define a reliable verifier by three properties: (1) Breadth. The suite exercises as many functional points as possible, so that it reflects the full black-box behavior of the program; (2) Faithfulness. Given broad coverage, every test passes on the golden program; (3) Discrimination. The suite fails on an empty implementation and cannot be passed unconditionally. Coverage feedback, the pass check, and the dummy check enforce these three properties respectively, while the double check and the weak-assert check further ensure stability and reduce weak oracles. Appendix B.4.1 details the design.
Retained tests must pass on golden, fail on the empty implementation, pass weak-assert checks, and match across repeated reference runs. Tests violating these conditions are discarded. Figure 13 shows these metrics improving over iterations.
2. Effectiveness analysis. We evaluate real agent implementations sampled by DeepSeek-V4-Flash-0731 in Program-Env as plausible faulty implementations. For environment ei, P^i and B^i are the agent's source and executable; Pi and Bi⋆ are their reference counterparts. The protocol is:
The verifier-effectiveness table below reports all 153,861 triples (Appendix B.5).
On these real, largely correct yet flawed agent outputs, the retained suites agree with the independent judgments on 98.80% of cases (TP + TN) and reject a correct implementation only 0.13% of the time (FN). The suites therefore provide reliable behavioral supervision that tells correct behavior from incorrect behavior, which is exactly the signal trajectory filtering relies on.
3. Training experiments
We use Program-Env's behavioral test suites to select 1,204 teacher rollouts with a test pass rate ≥50%, collected using DeepSeek-V4-Flash-0731 with Claude Code, and fine-tune Qwen3.8-27B on these rollouts to obtain Qwen3.8-27B-SFT-1204. The table below reports ProgramBench results under Claude Code. All Δ values are relative to the base Qwen3.8-27B model and are measured in percentage points.
Compared with the original SFT model, the more strictly filtered SFT model improves Tests Passed by 4.69 percentage points, Almost Resolved by 2.50 points, and Resolved by 0.50 points. Relative to the base model, it improves Tests Passed by 12.66 points and Almost Resolved by 0.50 points, while matching the base model's 0.50% Resolved rate.
Revision: We have expanded Appendix B.5 with this effectiveness analysis, including the experimental protocol, Table 15 and Table16. All additions are highlighted in blue.
Thank you again for your feedback. Please let us know if any concerns remain unaddressed. We would be happy to discuss them further.