- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies training-input selection for code optimization. The motivation is that existing methods either incur rollout costs before selection or lack historical rewards for newly generated variants. To address this, they propose SeeThat, which combines recursive program reuse with prediction-guided selection. The core idea of SeeThat is to reuse verified program variants as training inputs and let the optimization policy predict which variants are worth expanding. Execution feedback supervises these predictions. The shared policy uses BCE for prediction tokens and GRPO for code tokens.
Critical strengths
- The motivation is clear.
- The method integrates recursive program reuse and prediction-guided selection within a shared policy.
- The reported results show consistent P90/P95 runtime improvements across both benchmarks and model backbones.
Critical weaknesses
There are many fatal mismatch between the manuscript and the code, which affect the implemented method and evaluation validity.
- The code does not implement the paper’s central prediction objective. Section 3.3 and Appendix A.3 specify BCE supervision on the two prediction-answer positions. However, in the code,
build_prediction_datumsconverts prediction correctness into group-normalized advantages, andupdate_policythrough an importance-sampling objective. This discrepancy concerns a central algorithmic contribution, rather than an incidental implementation detail. - Random/Entropy use different admission rules, depth limits, and runtime references from the proposed method. For example, a correct 2× faster variant is eligible for SeeThat but rejected by the slowdown-only baselines. The comparison therefore does not isolate prediction-guided selection.
- The reward and prediction-label definitions also differ materially from the manuscript. Equation 5 assigns -1 to failed programs and raw speedup to verified programs, without gating the reward on nontrivial edits. The implementation instead assigns 0 to nontrivial incorrect programs, -1 to trivial edits, and to correct nontrivial programs. PerfCodeBench additionally defaults to binary correctness advantages for mixed-validity groups.
Other concerns:
- Key ablations, prediction-quality analysis, and cost measurements cover only PerfCodeBench-8B. Their conclusions remain unverified in weaker-performing settings. Fewer training steps do not establish end-to-end efficiency. The reported 3.65% billing overhead excludes execution, verification, and wall-clock costs.
- Direct comparisons with closely related methods, such as AceGRPO, PCL, and GPS, are missing. The current baselines do not establish whether gains come from predictive selection generally or SeeThat’s specific designs.
Questions for authors
See Weaknesses.