- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CodeMidas constructs 5,545 coding RL environments by removing functionality from existing repositories, generating execution-grounded tests, and filtering candidate tasks. Training improves performance on five external benchmarks and outperforms training on public PR-derived tasks under shared settings. The contribution is the complete construction and filtering pipeline.
Critical strengths
- I particularly value the checks beyond reference execution: leakage probes and alternative-solution audits address distinct reward failures. The human audit also documents residual errors (Appendix E).
- Transfer spans issue repair, program construction, translation, and terminal work; the smaller-model results further support the tasks' training value (Figure 5; Table D7).
Critical weaknesses
- The executable environments are the central contribution, but the 500-task supplement provides statements and test patches without repository revisions, modified starting states, or build specifications. These missing components prevent reconstructing the supplied tasks as executable training environments, limiting reuse of the demonstrated method.
- Outcome filtering retains tasks with both successful and failed attempts, yet §3.4 specifies only “several” attempts from “a frontier model.” The model and attempt budget determine which tasks survive and therefore the training difficulty distribution. This part of the tested method needs a precise specification.
Questions for authors
- Can you provide or point to repository/revision mappings, starting states, and build specifications or containers for the supplied tasks?
- Which model/checkpoint, number of attempts, and rollout budget define outcome filtering? Does §3's default model also apply here?
Following Up on Our Response
Dear Reviewer sqf1,
Thank you again for your positive assessment and constructive suggestions. We would greatly appreciate your feedback on whether these additions address your concerns, or if any substantial issues remain.
Best regards,
The Authors