acceptodds
Withdrawn from ICLR 2027

SeeThat: Learning to Optimize Code through Recursive Policy–Data Co-Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning to optimize code requires feedback on both correctness and runtime, which program execution can provide. Beyond supplying rewards, execution also identifies valid program variants that can serve as inputs for further optimization. Recursively reusing these variants allows a policy to recursively expand its own training data, but also creates more candidate inputs than a finite training budget can afford. The resulting challenge is to identify which candidates offer useful learning opportunities before spending the budget to generate and execute their refinements. We introduce SeeThat, which trains the code-optimization policy itself to predict whether an input will yield a rollout group with variation in validity or performance outcomes. These predictions are supervised by execution feedback already collected during training, with region-specific supervision for prediction and code generation. The policy then uses its predictions to rank newly admitted variants for future training, without generating or executing additional refinements or requiring a separate value network. In this way, policy learning and recursive training-data construction co-evolve. Across systems-code and competitive-programming optimization with both dense and mixture-of-experts models, SeeThat achieves the lowest P90 and P95 runtimes among the compared training methods in all four model–benchmark settings, and reaches the lowest P75 runtimes in three settings.

Primary area:
reinforcement learning
Keywords:
code optimization, code refinement, online data selection, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rXnMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies training-input selection for code optimization. The motivation is that existing methods either incur rollout costs before selection or lack historical rewards for newly generated variants. To address this, they propose SeeThat, which combines recursive program reuse with prediction-guided selection. The core idea of SeeThat is to reuse verified program variants as training inputs and let the optimization policy predict which variants are worth expanding. Execution feedback supervises these predictions. The shared policy uses BCE for prediction tokens and GRPO for code tokens.

Critical strengths
  1. The motivation is clear.
  2. The method integrates recursive program reuse and prediction-guided selection within a shared policy.
  3. The reported results show consistent P90/P95 runtime improvements across both benchmarks and model backbones.
Critical weaknesses

There are many fatal mismatch between the manuscript and the code, which affect the implemented method and evaluation validity.

  1. The code does not implement the paper’s central prediction objective. Section 3.3 and Appendix A.3 specify BCE supervision on the two prediction-answer positions. However, in the code, build_prediction_datums converts prediction correctness into group-normalized advantages, and update_policy through an importance-sampling objective. This discrepancy concerns a central algorithmic contribution, rather than an incidental implementation detail.
  2. Random/Entropy use different admission rules, depth limits, and runtime references from the proposed method. For example, a correct 2× faster variant is eligible for SeeThat but rejected by the slowdown-only baselines. The comparison therefore does not isolate prediction-guided selection.
  3. The reward and prediction-label definitions also differ materially from the manuscript. Equation 5 assigns -1 to failed programs and raw speedup to verified programs, without gating the reward on nontrivial edits. The implementation instead assigns 0 to nontrivial incorrect programs, -1 to trivial edits, and log2(1+speedup)log2(1 + speedup) to correct nontrivial programs. PerfCodeBench additionally defaults to binary correctness advantages for mixed-validity groups.

Other concerns:

  1. Key ablations, prediction-quality analysis, and cost measurements cover only PerfCodeBench-8B. Their conclusions remain unverified in weaker-performing settings. Fewer training steps do not establish end-to-end efficiency. The reported 3.65% billing overhead excludes execution, verification, and wall-clock costs.
  2. Direct comparisons with closely related methods, such as AceGRPO, PCL, and GPS, are missing. The current baselines do not establish whether gains come from predictive selection generally or SeeThat’s specific designs.
Questions for authors

See Weaknesses.

Reviewer_QqBTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SeeThat, a framework for code optimization through recursive policy-data co-evolution. The key insight is that the verified optimized code generated during RL training could be reused as new training inputs, allowing the policy to recursively expand its own training data. However, this introduces the challenge of identifying which candidates offer the most useful learning opportunities under a limited computational budget. To address this challenge, SeeThat proposes to train the code optimization policy itself to predict the training value of a given program. Specifically, the policy predicts both the validity and performance contrast, which are supervised using execution feedback on both correctness and performance through a binary cross-entropy objective. Meanwhile, the generated code is optimized using GRPO. The policy further uses its predicted training-value scores to select newly generated variants for subsequent training.

The paper evaluates SeeThat on PerfCodeBench and PIE using Qwen3-8B and Qwen3.6-35B-A3B. The results demonstrate improvements over standard GRPO and alternative input-selection strategies, particularly in P90 and P95 execution runtimes.

Critical strengths
  1. The paper identifies an important opportunity in code optimization: verified optimization results can be recursively reused as training inputs. The motivation is reasonable, and the proposed approach represents an interesting direction for improving RL training efficiency and automatically constructing useful training data.

  2. The experiments cover two code optimization benchmarks and both dense and MoE models. The experiment result show that SeeThat achieves promising results across all four settings.

Critical weaknesses

W1. Evaluation metrics validity. In L339, the authors mention they reported P50/P75/P90/P95 runtime statistics across evaluation inputs. My understanding is that these percentiles are computed over the absolute execution times of different programming tasks, rather than per-program normalized runtimes. Could the authors confirm this interpretation? If so, I find the motivation for these metrics unclear. Since different programs may have substantially different baseline execution times, the reported percentiles could be heavily influenced by intrinsically long-running tasks rather than reflecting the relative optimization improvements achieved across programs. Could the authors explain why these absolute runtime percentiles are appropriate for evaluating general code optimization effectiveness?

W2. The metrics do not sufficiently reflect code optimization capability.

Neither the original PIE nor PerfCodeBench uses these absolute runtime percentiles as its main evaluation metrics. Prior work on PIE commonly reports speedup and the percentage of optimized programs (%Opt), while PerfCodeBench focuses on metrics Faster-than-Baseline Rate (FBR), Reference-or-Better Rate (RBR), and Correctness-Gated Relative Efficiency (CGRE).

The current single-generation evaluation does not demonstrate whether SeeThat improves the best achievable optimization performance. Especially on PIE with Qwen3.6-35B-A3B, SeeThat only slightly improves P95 runtime over Random selection (4.62 vs. 4.67 ms), while correctness decreases substantially (81.33% vs. 88.56%). It is unclear whether this represents a meaningful improvement in overall code optimization capability.

W3. The proposed training-value proxy is not sufficiently justified.

The central idea of SeeThat is to predict which inputs will provide useful learning signals. However, the proposed validity and performance contrast labels do not necessarily correspond to actual learning value. For example, consider a rollout group in which all generated programs are correct and non-trivial, but their speedups range from 2× to 10×. Neither validity contrast nor performance contrast would be positive, even though the group exhibits substantial reward variance and could provide informative GRPO gradients. This raises the question of whether the proposed binary labels are sufficient to capture training value, or whether directly predicting reward variance would provide a more principled selection criterion.

W4. Mechanism underlying recursive training.

In Table 2, the Depth-1 only performs substantially worse than Original-input replay on PerfCodeBench with Qwen3-8B. This is somewhat surprising, given the motivation that verified generated variants can provide useful additional training inputs. The paper would benefit from a more detailed analysis of this phenomenon. Understanding why shallow recursive training performs poorly while deeper recursion is beneficial would provide stronger support for the proposed policy–data co-evolution mechanism.

W6. Minor clarity issue in the methodology overview.

In L163, the policy first predicts the prospective training value of an input gg. In L165, the training value for gg is determined by executing the KK generated variants. In L170, the same policy produces training-value predictions again during input selection. Although the subsequent sections clarify that these refer to different operations, the distinction amo predictions for the current training input, execution-derived supervision labels, and predictions for newly admitted candidates is not immediately clear from the overview. I suggest revising this paragraph to make the roles of these quantities more explicit.

Questions for authors

See weakness.

Q1. Definition and robustness of normalized edit distance. In Section 3.2, a refinement is considered non-trivial if its normalized edit distance exceeds 0.01. How exactly is this distance calculated (e.g., character-level, token-level, or AST-based)? Could simple semantics-preserving but performance-irrelevant changes, e.g., variable renaming or code obfuscation, satisfy this criterion? How does the method prevent such transformations from being treated as meaningful refinements, particularly under the noisy runtime measurements during training?

Q2. Reliability of performance measurements. Given that many programs in PIE have execution times on the order of milliseconds, runtime measurements may still be sensitive to system-level noise even after averaging 10 executions. Could the authors report the variance of repeated runtime measurements and analyze the sensitivity? In particular, how frequently would repeated measurements change whether a program satisfies the 10% performance threshold? Would additional measurement repetitions affect the results?

Q3. Cross-benchmark generalization. Since PerfCodeBench and PIE both involve optimizing given programs while preserving correctness, could the authors investigate whether a model trained on one benchmark generalizes to the other? Specifically, how would a policy trained on PerfCodeBench perform on PIE without additional training, and vice versa? Would jointly training a single policy on both datasets improve performance?

Reviewer_hHnQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Proposes SeeThat, a RL recipe for code optimization that "enables recursive policy–data co-evolution in code-optimization RL."
  • Co-train the policy network to also identify which problems will likely yield meaningful signal for later GRPO style RL.
  • Compared SeeThat with a few baseline methods including GRPO and random sampling, and found SeeThat to perform the best in multiple different metrics.
Critical strengths
  • Having the policy network to also identify data points to train on actually makes a lot of sense. Since whether this data point is suitable to be used for training depends a lot on the model's own ability, which the model itself likely has abundant information about.
Critical weaknesses
  • Baseline seems a bit lacking; while GRPO, entropy and random selection are nice baselines, they are a bit too fundamental. To truely highlight the proposed method's ability to select useful datapoint, the authors should add in comparisons with the mentioned alternatives such as AceGRPO, GRESO, PCL and GPS.
  • Not critical, but I would like to see the prediction quality compared against entropy and methods mentioned above. This could be an extension to appendix A.7.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.