acceptodds
Under review as a conference paper at ICLR 2027

CompassPlay: Rewarding the Proposer for Where It Moves the Solver

Abstract

In self-play, a proposer generates verifiable tasks to train a solver. Proposer rewards often depend on the solver's success rate, but equally difficult tasks can differ in their training value. We introduce CompassPlay, a self-play method that rewards the proposer through gradient alignment. The reward favors tasks whose solver loss gradients align with those of reference tasks representing the target capabilities. It draws on a first-order approximation to learning progress and scores each eligible task without additional solver training. Our experiments show gains in performance and training efficiency. In coding self-play with Qwen2.5-Coder-7B, a small external reference set guides task generation. CompassPlay improves average accuracy over AZR's difficulty reward by 1.5 percentage points on in-domain coding and 2.7 on out-of-domain mathematics. In Lean 4 theorem proving, CompassPlay matches the difficulty baseline's 150-iteration cumulative coverage with 40% fewer GPU-hours.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-play, reasoning, LLMs

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uH4dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompassPlay, a new method that uses gradient alignment to redesign the proposer reward in self-play. CompassPlay compares the loss gradients of candidate tasks and reference set to decide whether the generated tasks can help improve the solver's target capabilities. The authors use a first-order approximation and cosine similarity to compute the reward, avoiding additional solver training for each candidate task like SOAR. The authors also evaluate CompassPlay using Qwen2.5-Coder-7B for coding self-play on coding benchmarks and mathematics benchmarks. They also evaluate the method using DeepSeek-Prover-V2-7B for Lean 4 theorem proving. The experimental results show improvements in performance and training efficiency compared with difficulty-based rewards.

Critical strengths
  1. One strength of this paper is using a first-order gradient approximation to estimate the training value of generated tasks. This avoids the additional solver training required by methods like SOAR, which can save a lot of computational cost. The Lean 4 experiments also illustrate that CompassPlay can achieve the same coverage with the difficulty baseline with fewer GPU hours.

  2. The authors evaluate the method in two different domains, coding and theorem proving. They also provide ablation studies on gradient types, cosine similarity vs. dot product, and novelty weighting, which help explain how different designs affect the final performance.

Critical weaknesses
  1. One of main concern is the selection of the reference set. The coding experiments only use 32 MATH training problems as a fixed reference set, and the reward directly depends on the gradients of these problems. I am concerned that different choices of reference set may affect the stability of the training results.

  2. Although the authors treat mathematics as an out-of-domain evaluation, mathematical problems have already been used in the training process through the reference gradient. Therefore, I don't think the current results are sufficient to demonstrate that the method can generalize to domains not covered by the reference set.

Questions for authors
  1. Can the authors further evaluate the generalization ability of CompassPlay? For example, in the coding experiments, although mathematics is treated as out of domain, the reward already uses gradients from mathematical problems. If the reference set is replaced with coding problems or tasks from other domains, would we still observe similar improvements on mathematics benchmarks?

  2. More additional experiments on reference set selection is encouraged. For example, using different randomly sampled sets of 32 mathematical problems or changing the size of the Reference Set to see how much the training results vary. This would help demonstrate whether the method is sensitive to the choice of reference set.

  3. It's better to further explore whether novelty weighting actually reduces repetition at the mathematical semantic level. For example, could the generated theorems simply just the reformulations of the seed theorem rather than brand new tasks as simply Token-Jaccard cannot guarantee this.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uH4d (Part 1)

Thank you for the review! We did some new experiments and anslysis to answer your questions.

W1 / Q2: Is the method sensitive to the choice and size of the reference set?

Experimental setup. We rescored 1,296 coding tasks generated by the base model with each alternative reference set. Among tasks with the same type and solve rate, we call the top third by reward the selected tasks, and check how many of the tasks selected with the paper's reference (32 MATH problems) are still selected.

Results.

Reference setRank correlation with the paper's rewardSelected tasks kept
A different random sample of 32 MATH problems0.93584.4%
128 MATH problems0.93185.4%
8 MATH problems0.83076.7%
32 MBPP coding problems0.72262.0%

A different random sample of 32 MATH problems and a 128-problem reference keep most of the selected tasks, and almost none of them (at most 0.7%) drop to the bottom third. Switching to coding problems changes the selection much more. The reward is therefore stable within a reference domain, and the domain is what steers it.

W2 / Q1: Would a reference set without math still improve the math benchmarks?

Yes. We reran CompassPlay with 32 held-out MBPP coding problems as the reference set, disjoint from the code benchmarks, changing nothing else. At step 100, the math mean is 44.7 ± 3.9 with the coding reference, versus 44.9 ± 1.4 with the paper's math reference and 43.8 ± 3.7 for the difficulty reward. Even when the reference contains no math, the improvement on the math benchmarks is largely preserved.

Results at step 100 (mean ± sd over three seeds):

CompassPlay, code referenceCompassPlay, math referencedifficulty
LiveCodeBench45.9 ± 3.350.1 ± 2.647.6 ± 5.9
CRUXEval-I55.1 ± 1.757.1 ± 1.657.2 ± 1.5
CRUXEval-O58.2 ± 1.859.4 ± 1.359.9 ± 2.0
Code mean53.1 ± 2.055.5 ± 1.854.9 ± 2.7
MATH50058.3 ± 5.559.4 ± 4.553.5 ± 3.3
AMC2337.5 ± 9.033.3 ± 2.939.2 ± 11.5
GSM8K77.3 ± 2.376.1 ± 2.076.8 ± 0.2
Minerva23.0 ± 2.026.5 ± 1.324.0 ± 2.1
OlympiadBench27.4 ± 3.429.1 ± 2.325.6 ± 2.0
Math mean44.7 ± 3.944.9 ± 1.443.8 ± 3.7

Q3: Does novelty weighting reduce repetition at the mathematical semantic level?

Yes. We ran a blind semantic audit of (seed, conjecture) pairs. Claude Opus 5.5 labelled a stratified sample, reweighted to all eligible conjectures. One author also blind-labelled 40 pairs by hand, with κ = 0.87 agreement on copy-like vs. novel. We audited two checkpoints per method: an early and the final iteration for cos (39 and 150), and two iterations for cos × novelty after its copy rate has settled below 0.5% (95 and 105; Figure 4). Percentages are over all eligible conjectures at these iterations:

MethodRenaming or equivalent reformulationMinor variantSubstantively differentProof strategy differs from the seed's
cos, no novelty, iteration 3999.1%0.9%0.0%7%
cos, no novelty, iteration 15096.1%3.8%0.1%10%
cos × novelty, iteration 950.1%0.1%99.8%97%
cos × novelty, iteration 1050%0%100%97%

Renamings and equivalent reformulations of the seed fall from 96–99% of eligible conjectures under cos alone to 0–0.1% under cos × novelty. Under cos × novelty, 99.8–100% of conjectures are substantively different statements, and 97% of their verified proofs use a different strategy from the seed's proof.

We would welcome any feedback. In particular, does the coding-reference result above address your concern?

AuthorsOpenReview ↗

Response to Reviewer uH4d (Part 2)

W1 / Q2: Do training results vary with the choice and size of the reference set?

We trained the runs with alternative reference sets to step 100 and evaluated them with the same pipeline as the paper. We also add a different random sample of 32 MATH problems, as you suggested.

Results at step 100:

Reference setCode meanMath mean
32 MATH problems in the paper, mean ± sd over three seeds55.5 ± 1.844.9 ± 1.4
A different random sample of 32 MATH problems55.847.5
8 MATH problems53.839.6
128 MATH problems57.351.8

A different random sample of 32 problems performs on par with the paper's set. Accuracy rises with the size of the reference set, most clearly on math, so the paper's 32 problems are not the best case for CompassPlay.

This completes our response to all points. We are happy to answer any further questions.

Dear Reviewer uH4d,

Thank you again for your review. As the discussion closes soon, we would like to briefly recap our response.

  • Choice and size of the reference set. A different random sample of 32 MATH problems performs on par with the paper's set, and accuracy rises with the size of the reference set, most clearly on math.
  • Math gains without math in the reference. With a coding reference set, the math mean at step 100 is 44.7, versus 44.9 with the paper's math reference.
  • Semantic novelty. In a blind semantic audit, the novelty factor reduces renamings and equivalent reformulations of the seed from 96–99% to 0–0.1% of eligible conjectures.

Do our responses resolve your concerns? If so, we would be grateful if you would consider updating your assessment.

Reviewer_B7GQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an algorithm called CompassPlay, which uses the similarity between the gradient of the model on the evaluation set and the gradient of the model on the generated data, to reward to the proposer in self-play loop. The authors observe improvement over a difficulty reward baseline in two settings, coding and verifiable lean proofs.

Critical strengths
  1. This paper is well-written and very easy to understand
  2. The evaluation this paper presents spans two different settings, showing generalizability of this method.
Critical weaknesses
  1. I think one of the major issue is that this paper didn't compare with the closely related previous work https://arxiv.org/abs/2606.09052.
  2. The comparison with SGS in the lean setting does not match the original paper's SGS algorithm because the prompt input is from solved problems instead of unsolved problems, which invalidates SGS's premises and may lead to the underperformance of SGS. On the other hand, it is possible that the algorithm is complementary.
Questions for authors
  1. How does this work compared with the Infuser paper https://arxiv.org/abs/2606.09052?
  2. Why the novelty measurement is not necessary in the coding setting?
4 comments

Thank you for the review and for pointing us to INFUSER.

W1 / Q1: Comparison with INFUSER.

We now cite and discuss INFUSER in the related work section. Like Montessori-Instruct and OptimSyn, INFUSER rewards a data generator with an influence estimate. CompassPlay differs from it in the problem it solves, in where the reference comes from, and in the failure mode it has to handle:

  1. Verifiable self-play rather than document-grounded QA. INFUSER's generator writes questions and answer keys from textbook chunks; in its main setting the answer keys are not formally verified, and the solver is trained against them. CompassPlay is a proposer reward for verifiable self-play: every task is an executable program or a Lean statement checked by an executor or the Lean kernel, so the solver never trains on an unverified label, and the alignment gradients are computed only on verified or gold solutions.
  2. Cross-domain and local references. INFUSER aligns with the solver's policy gradient on 800 questions sampled from its target distribution (SuperGPQA Science), in the same domain as its science document pool, and notes that steering toward targets outside that domain remains open. In CompassPlay, the reference either comes from a different domain than the generated tasks (32 MATH problems steer the generation of coding tasks) or is local: in Lean, it is the verified proof of the seed the proposer is conditioned on, so no development set is needed.
  3. A failure mode that arises when the generator sees its reference, and its fix. With a local reference, the proposer is shown the seed that defines its reward and learns to copy it: the copy rate (eligible conjectures with token-Jaccard similarity above 0.8 to their seed) reaches 96% at iteration 75 (Section 4.1, Figure 4). We identify this failure mode, explain why it arises when an alignment criterion drives a generator rather than a selector, and remove it with novelty weighting: the copy rate stays below 0.5% over iterations 60–150, and over iterations 15–115 it solves on average 13.6 more targets than the difficulty reward, versus −7.9 for novelty alone (Table 3). A development set that the generator never sees, as in INFUSER, cannot expose this failure mode.
  4. A drop-in reward for existing self-play loops, including formal theorem proving. CompassPlay changes only the proposer reward of AZR (coding) and SGS (Lean), and is compared with each loop's own reward under identical settings. In Lean, the goal is coverage of a fixed set of target theorems and no development set exists, yet the reward applies unchanged.
  5. Gradient objective. INFUSER uses AdamW-preconditioned policy gradients; CompassPlay uses cross-entropy gradients on verified solutions. In our setting, replacing them with policy-gradient estimates was worse and 1.7× more expensive (Appendix C.1).

W2: The SGS comparison conditions on solved instead of unsolved problems.

Thank you for raising this. We clarify which targets each method conditions on. The SGS-reward baseline, as well as the difficulty and solve-only baselines, uses the original SGS conditioning on unsolved targets: the proposer sees only the statement of a not-yet-proved target theorem, so SGS's premise is preserved. Only the alignment-based variants (cos, cos × novelty, novelty-only) condition on solved targets, because the reference gradient is computed from the seed's verified proof. Section 4 and the "proposer conditioning" row of Table 7 now state the conditioning for each method.

We agree the two are likely complementary. For example, SGS's guide score could be multiplied with the alignment reward, or unsolved-target conjectures could be used where no verified seed proof exists. We see this as a promising direction.

Q2: Why is no novelty factor needed in coding?

In coding self-play, the reference set is an external set of MATH problems that never appears in the proposer's prompt. The proposer is conditioned only on its own past tasks. There is therefore nothing to copy that would trivially align with the reference. The copying shortcut arises in Lean because the proposer sees the very seed whose proof defines its local reference (Section 4.1).

Dear Reviewer, thank you again for your review. As the discussion period ends on November 18 (AoE), we wanted to check whether our response above addresses your concerns about INFUSER and the SGS comparison. We would be glad to clarify anything further, and if our response resolves your concerns, we would be grateful if you would consider updating your score.

I thank the authors for their rebuttal. Sorry for the late reply. Regarding SGS, I am wondering why the performance in this paper is 2% lower than what the original SGS paper reported if the algorithm is the same.

I will raise my score to 3 as I think the comparison with Infuser make sense but I think the authors should consider a more rigorous comparison with SGS.

Thank you for the response and for updating your assessment, and no worries about the timing! Our reproduction runs fewer iterations (150 rather than 200+) and differs in some implementation details, including hardware and verification timeouts, which could lead to lower coverage. All methods were run under the same experimental configuration.

Reviewer_83TFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Language model self-play generally involves a setup in which a proposer generates tasks to train a solver, and a verifier checks the answers for correctness. Self-play is not perfect, however, as the value of the system is highly elastic and dependent on the proposer’s curriculum. In the normal case, the proposer’s reward is dependent on the solver’s success rate, but this means that two tasks that are equally difficult are generally assigned the same training value. Previous work generally targets reward functions where the proposer is directed to generate problems at the edge of the solver’s current capabilities, but this kind of formulation makes an assumption that tasks with the same, or similar, success rates exercise the same skills and have the same impact on target/downstream task performance. Therefore, success rate can be an unreliable proxy. The authors propose that gradient-based signals, such as those used for data selection or synthetic data generation, could be a viable approach, and present CompassPlay as a self-play method that rewards the Proposer through gradient alignment. This works by comparing the solver’s loss gradient with the gradient of a reference loss for target capabilities, with the proposer receiving reward shaping from the cosine of the candidate and reference gradients. The authors evaluate CompassPlay on coding and theorem proving, showing improvements over AZR on coding benchmarks and efficiency gains in coverage.

Critical strengths
  • The motivation is good, and the paper has a really nice flow. It is easy to understand, and even though AI was used to assist with writing (via the use statement), it was clearly done with care. The problem the authors are addressing is explained intuitively (equal solve rate != training value, which many self-play papers assume).

  • CompassPlay is shown to improve in-domain coding performance relative to the baseline AZR across all coding benchmarks.

  • The authors do a good job of preparing and presenting the cost analysis, and the literature review covers related work in the gradient alignment domain.

  • The Lean experiments present an interesting analysis of reward hacking (e.g., cosmetic changes that pass and superficial edits that lead to proposed problems with high semantic overlap with the seed problem). The authors then propose an additional novelty reward and show renamed copies drop considerably.

Critical weaknesses
  • The performance improvements are generally small, and the scope is fairly narrow, with only 2 models used, each of the same size, which limits claims of generalizability.

  • There is a fairly extensive body of literature on gradient alignment for data value. The main contribution here is the application of said literature to the setting of language model self-play.

  • CompassPlay is framed as a “method”, but in general, it is reward shaping.

  • The paper motivates its approach by discussing first-order estimates of learning progress but provides limited analysis of whether the cosine reward predicts solver improvement or selects tasks different from those of the AZR difficulty reward.

  • For coding in particular, each step costs much more, and with the limited performance gains, this raises questions about practicality, particularly in compute-sensitive regimes.

  • Some important technical details needed for reproducibility aren’t available, such as the solver's optimizer and learning rate.

  • The paper’s focus shifts from self-play to novelty-decline, which includes a novelty multiplier in the reward that resembles diversity-weighted rewards, like the one in [1]. The authors should discuss this connection.

[1] Li, Tianjian, et al. "Jointly reinforcing diversity and quality in language model generations." arXiv preprint arXiv:2509.02534 (2025).

Questions for authors
  • Could the authors run some analysis and show the correlation between the cosine-based reward and (1-solve rate)? I think some additional analysis of the curriculum that CompassPlay uses, against a difficulty-based reward, is warranted, and whether high-reward tasks lead the solver to improve more may help test the premise of CompassPlay more directly.

  • CompassPlay builds on MATH solutions that AZR never sees, so while it does do better on math benchmarks, it’s difficult to separate whether gradient alignment is a better reward, or if it’s because the reward was aimed at math. Can you run CompassPlay again, but using a coding reference set instead?

  • The novelty fix in the lean experiments amounts to something like “similar to the seed, but not a copy,” which raises the question of whether the same could be accomplished without gradient-based methods. Could the authors provide an alternative control to better emphasize the importance of CompassPlay in this setting?

  • Could the authors report results for other model sizes and discuss how the gradient computation scales with model size? This would also be important for showing how the effects hold across various sizes.

Presentation/Typos/etc:

  • The paper ends midway through page 9. Many important technical details appear in the appendix, and some of them could have made their way to the main paper.

  • The Lean experiments highlight a phenomenon of proposer diversity collapse through reward hacking. A related line of literature is [1,2,3].

[1] Chae, Justin Yang, Md Tanvirul Alam, and Nidhi Rastogi. "Towards understanding self-play for llm reasoning." arXiv preprint arXiv:2510.27072 (2025).

[2] Dineen, J., RRV, A., Xu, Z., & Zhou, B. (2026). Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution. arXiv preprint arXiv:2604.03472.

[3] Pu, Sophia Xiao, et al. "Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL." arXiv preprint arXiv:2605.22217 (2026).

5 comments
AuthorsOpenReview ↗

Response to Reviewer 83TF (Part 1)

Thank you for the thorough review and the pointers to related work. To leave time for discussion, we are posting our response in parts. We are working on the remaining points and will post them in a follow-up comment shortly.

W4 / Q1: Correlation between the cosine reward and 1 − solve rate; does a high reward predict improvement?

  • Correlation. We used the training logs of the three paper runs, with all eligible candidates (0 < s < 1, where the solve rate s is the fraction of the solver's 8 attempts that are correct), about 18–19k per seed. The reward only ranks candidates proposed at the same training step for the same task type, so we computed the statistics within such groups.

    • Within-group Spearman ρ between the cosine reward and 1 − s is −0.04, −0.08 and −0.06 across the three seeds.
    • Among candidate pairs that both rewards can order (different solve rates), the two rewards prefer different tasks 52–53% of the time; 50% would mean the two orders are unrelated.

    So the two rewards favor different tasks.

  • Does a higher reward predict a larger improvement? Yes. In a matched test with identical solve-rate distributions, size and training budget, training on high- rather than low-alignment tasks improves the math mean by 9.4 points and the code mean by 2.3 points on average at step 100 (three seeds), and the average is higher on all eight benchmarks. Design: we took 1,296 coding tasks generated by the base model with 0 < s < 1, grouped them by task type and solve rate, and put the top and bottom thirds by alignment reward within each group into a high-alignment set H and a low-alignment set L (416 tasks each, with identical solve-rate distributions and task-type composition). Starting from the same base checkpoint, we trained the solver on each set with identical RL settings for 100 steps.

    Step 100, mean ± sd over three seeds (H − L is computed within each seed):

    H (high alignment)L (low alignment)H − L
    AMC2336.67 ± 3.8226.67 ± 10.41+10.00 ± 7.50
    MATH50059.67 ± 6.7045.00 ± 6.63+14.67 ± 1.72
    GSM8K78.87 ± 2.9764.93 ± 6.02+13.93 ± 8.64
    Minerva24.50 ± 1.8221.20 ± 5.23+3.30 ± 4.54
    OlympiadBench26.43 ± 3.4521.17 ± 3.98+5.27 ± 1.06
    Math mean45.23 ± 2.8735.79 ± 5.40+9.43 ± 2.80
    CRUXEval-I52.13 ± 2.2651.29 ± 4.89+0.83 ± 7.09
    CRUXEval-O52.92 ± 1.1348.04 ± 6.25+4.88 ± 7.26
    LiveCodeBench40.36 ± 1.9939.18 ± 6.63+1.18 ± 8.44
    Code mean48.47 ± 1.6046.17 ± 4.89+2.30 ± 6.28

Q2: Is the math gain due to alignment, or to aiming the reward at math? (code reference set)

We ran exactly this. We replaced the 32 MATH reference problems (with solutions) by 32 held-out coding problems from MBPP. Each item is a (code snippet, input, output) triple whose output comes from executing the code. The set contains no math and was checked item by item to be disjoint from the code benchmarks. All other settings follow the paper (three seeds, run to step 100).

Results at step 100 (mean ± sd over three seeds):

CompassPlay, code referenceCompassPlay, math referencedifficulty
LiveCodeBench45.9 ± 3.350.1 ± 2.647.6 ± 5.9
CRUXEval-I55.1 ± 1.757.1 ± 1.657.2 ± 1.5
CRUXEval-O58.2 ± 1.859.4 ± 1.359.9 ± 2.0
Code mean53.1 ± 2.055.5 ± 1.854.9 ± 2.7
MATH50058.3 ± 5.559.4 ± 4.553.5 ± 3.3
AMC2337.5 ± 9.033.3 ± 2.939.2 ± 11.5
GSM8K77.3 ± 2.376.1 ± 2.076.8 ± 0.2
Minerva23.0 ± 2.026.5 ± 1.324.0 ± 2.1
OlympiadBench27.4 ± 3.429.1 ± 2.325.6 ± 2.0
Math mean44.7 ± 3.944.9 ± 1.443.8 ± 3.7

Late-window averages give the same ordering:

CompassPlay, code referenceCompassPlay, math referencedifficulty
Code mean, steps 80–10053.2 ± 1.854.5 ± 1.853.9 ± 2.1
Code mean, steps 60–10052.7 ± 1.953.4 ± 1.652.9 ± 1.6
Math mean, steps 80–10043.3 ± 3.244.1 ± 2.242.6 ± 3.0
Math mean, steps 60–10042.6 ± 3.043.8 ± 1.741.3 ± 2.2
  • The math gain is largely preserved without any math in the reference. With the coding reference, the math mean is almost the same as with the math reference (44.7 vs. 44.9) and above the difficulty reward (43.8). Per benchmark, it stays above difficulty on MATH500 (58.3 vs. 53.5) and OlympiadBench (27.4 vs. 25.6). The math gain therefore does not come only from aiming the reward at math.

P.S. One code-reference seed often continues past its </answer> with a new "User: …" question, and the official extractor takes the last answer block, so we score math on the first answer block. For all other runs, this changes the math mean by at most 0.6 points.

AuthorsOpenReview ↗

Response to Reviewer 83TF (Part 2)

Q3: Could the Lean novelty fix be achieved without gradients?

The novelty-only ablation in Table 3 is exactly this control: it rewards "different from the seed" without any gradient signal. It eliminates copying as well (0% copy rate at iteration 115), but over iterations 15–115 it solves on average 7.9 fewer targets than the difficulty reward, whereas cos × novelty solves 13.6 more. The alignment term is therefore what makes the novel conjectures useful for coverage.

**Q4 / W1: The performance improvements are generally small ... do they hold at other model sizes, and how does the gradient cost scale? **

Size of the gains. CompassPlay improves over AZR's difficulty reward on all eight benchmarks in the paper (+1.5 points on the code mean and +2.7 on the math mean, three seeds), and in Lean it reaches the difficulty baseline's 150-iteration coverage with 40% fewer GPU-hours. When the reward is tested in isolation, at matched difficulty, high-alignment tasks improve the math mean by 9.4 points (Part 1, W4/Q1). The gains also hold at a second model size.

Another model size: Qwen2.5-Coder-3B (W1, Q4).

Experimental setup. We reran the coding comparison at 3B with full AZR self-play. The two runs differ only in the proposer reward: CompassPlay with the paper's reference set (32 MATH problems), or the difficulty reward. Everything else follows the paper's 7B configuration. Three seeds per method, 200 steps.

Results. Step 200, mean ± sd over three seeds:

CompassPlaydifficulty
LiveCodeBench40.3 ± 1.439.8 ± 2.7
CRUXEval-I51.5 ± 1.747.9 ± 2.2
CRUXEval-O52.9 ± 1.250.8 ± 3.4
Code mean48.2 ± 0.646.1 ± 2.0
MATH50059.5 ± 4.153.8 ± 8.7
AMC2334.2 ± 3.832.5 ± 5.0
GSM8K78.3 ± 3.974.5 ± 10.4
Minerva21.9 ± 1.521.6 ± 3.5
OlympiadBench24.4 ± 2.921.7 ± 4.9
Math mean43.7 ± 2.940.8 ± 5.7

CompassPlay is at or above the difficulty reward on every benchmark, by 2.1 points on the code mean and 2.9 on the math mean, with smaller variation across seeds. The code mean is also higher at every evaluated step (every 50 steps).

Gradient cost and model size (Q4).

The extra cost is one forward and backward pass per eligible candidate on its verified solution, plus one reference gradient per step shared by all candidates, so it grows linearly with model size, like a training step. In the paper's setting it adds 37% wall-clock per coding step (Table 4). We timed the per-candidate cost from 0.5B to 7B with the same candidates and reference set:

ModelPer candidatePer billion parameters
0.5B0.57 s1.17 s
1.5B0.89 s0.58 s
3B1.46 s0.47 s
7B3.85 s0.51 s

From 3B upward, the cost grows linearly with model size (about 0.5 s per billion parameters).

W5: Is CompassPlay practical under compute constraints? (matched GPU-hours)

In Lean, CompassPlay reaches the difficulty baseline's 150-iteration coverage with 40% fewer GPU-hours (2,787 vs. 4,652). In coding, with 69 GPU-hours, slightly less than AZR's full run (71), CompassPlay already exceeds AZR's final accuracy on both the code benchmarks (57.9 vs. 57.6) and the math benchmarks (48.8 vs. 47.9), and it keeps improving to 59.2 and 50.6 with its full budget (Table 5). It is also much cheaper than measuring learning progress directly: a 200-step CompassPlay run uses 95 GPU-hours, while 20 outer steps of our SOAR reimplementation use 109 GPU-hours and reach lower accuracy (Figure 3).

We would welcome any feedback on these results. In particular, do the new 3B results address your concern about the scope of the evaluation?

AuthorsOpenReview ↗

Response to Reviewer 83TF (Part 3)

W2 / W3: How does CompassPlay go beyond gradient alignment for data selection, and is it a method or reward shaping?

CompassPlay does change only the proposer reward of an existing self-play loop, as the paper states at the start of Section 2. Whether a new reward design is called a method or reward shaping is partly a matter of terminology; much self-play work centers on how the proposer is rewarded, such as the learnability reward of AZR, uncertainty reward of R-Zero, and the guide score of SGS.

Turning gradient alignment from a data-selection criterion (as in TracIn, LESS and GradAlign) into the reward of a generator is not a direct transfer. As you noted, the Lean experiments reveal a reward-hacking mode in which the proposer copies its seed: a generator can produce the reward's maximizer directly, which a selection method never faces, and we identify this failure mode and remove it with novelty weighting (copy rate below 0.5%, 13.6 more solved targets than the difficulty reward; Table 3). The alignment signal itself provides what neither difficulty nor novelty alone does: at matched difficulty, high-alignment tasks improve the math mean by 9.4 points (Part 1, W4/Q1), whereas novelty without gradients solves 7.9 fewer targets than the difficulty reward (Part 2, Q3).

W6: What are the solver's optimizer and learning rate?

In coding, the proposer and solver share one model, trained with AdamW at a learning rate of 1e-6, REINFORCE++ advantages and a KL loss with coefficient 0.01. In Lean, the separate solver and proposer are trained with AdamW at a learning rate of 3e-6. Table 7 (Appendix F) now lists the optimization settings for both domains.

W7: How does the novelty factor relate to diversity-weighted rewards and to proposer diversity collapse?

Li et al. reward diversity across a pool of sampled generations, whereas our novelty factor is defined relative to each conjecture's seed and prevents the proposer from copying the seed that defines its reference; the two are complementary. The novelty factor is part of the self-play reward rather than a separate focus: without it, the alignment reward drives the proposer to copy its seed, so it is what makes a local reference usable in Lean. The related work section now discusses Li et al. together with the works on proposer diversity collapse that you pointed to (Chae et al., 2025; Dineen et al., 2026; Pu et al., 2026).

Could important details move from the appendix into the main text?

Thank you for this suggestion. The revision already clarifies the per-method conditioning in Section 4 and adds the optimization settings to Table 7. In the final version, we will use the additional page to move the main new analyses from this discussion into the main text.


This completes our response to all points. We are happy to answer any further questions.

Reviewer_ibRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward design for task proposers in self-play training, arguing that the Solver’s success rate alone cannot distinguish tasks with different training value. The authors propose CompassPlay, which rewards the Proposer using cosine similarity between the Solver’s cross-entropy gradients on correct solutions to generated tasks and those on reference tasks. This avoids training additional Solver copies to estimate learning gains. For Lean theorem proving, a token-Jaccard novelty factor helps reduce copying of seed statements. Coding and theorem-proving experiments show gains in accuracy and training efficiency, respectively.

Critical strengths
  1. The paper identifies a practical limitation of difficulty-based rewards. Tasks with the same success rate may have different training value. Reference gradients offer an explicit direction for generating tasks that target desired capabilities.
  2. The experiments cover two distinct verifiable domains. Coding results span eight benchmarks, with three random seeds for the main methods. The theorem-proving experiments show efficiency gains by reaching the baseline’s cumulative coverage earlier.
Critical weaknesses
  1. The link between gradient alignment and actual RL learning gains is not directly validated. The derivation assumes a single cross-entropy update, while the Solver uses policy-gradient RL. It remains unclear whether higher alignment reliably translates into greater capability gains.
  2. The novelty metric captures lexical differences more than mathematical novelty. Token-Jaccard may miss semantically equivalent reformulations and penalize distinct statements that share common Lean syntax.
  3. Dependence on seed problems needs further discussion. Since novelty directly affects the Proposer’s reward, it would help to show whether this constraint encourages substantive diversity or mainly surface-level variations of the seeds.
Questions for authors
  1. How many tasks are generated, pass verification and filtering, and are used for Proposer and Solver updates? Including deduplicated counts and retention rates would help clarify data scale and efficiency.
  2. Would it be feasible to compare high- and low-alignment task groups matched on solve rates, task counts, and training budgets? Identical RL runs from the same Solver checkpoint, evaluated on independent test sets across multiple seeds, would provide more direct evidence.
  3. It would be helpful to include a human or LLM-assisted semantic novelty audit, with manual spot checks, distinguishing renamings, equivalent reformulations, minor variants, and substantively different statements. Do verified proofs also show changes in proof strategies?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ibRd (Part 1)

Thank you for the detailed review and the concrete suggestions. To leave time for discussion, we are posting our response in parts. We are working on the remaining points and will post them in a follow-up comment shortly.

W1 / Q2: Does higher alignment translate into larger RL learning gains? (matched high- vs. low-alignment experiment)

W1 asks whether higher alignment translates into greater capability gains, and notes that the derivation assumes a cross-entropy update while the solver is trained with policy-gradient RL. Q2 proposes a direct test of the first point. We address both below.

  1. Matched experiment (W1, Q2). We ran the experiment you proposed in Q2.

    Experimental setup. The Qwen2.5-Coder-7B base model generated coding tasks of three types: input prediction, output prediction and program induction. For each task, the base model sampled 8 solutions checked by execution, and the solve rate s is the fraction correct. We kept the 1,296 tasks with 0 < s < 1, which are the tasks the paper's reward applies to, and scored each with the paper's alignment reward at the base checkpoint. Within each cell of tasks with the same type and solve rate, the top third by reward went to a high-alignment set H and the bottom third to a low-alignment set L. After trimming to complete groups of 8, each set has 416 tasks, with identical solve-rate distributions and task-type composition. From the same base checkpoint, we trained only the solver on each set for 100 steps and 8 tasks per step.

    Results. Step 100, mean ± sd over three seeds (H − L is computed within each seed):

    H (high alignment)L (low alignment)H − L
    AMC2336.67 ± 3.8226.67 ± 10.41+10.00 ± 7.50
    MATH50059.67 ± 6.7045.00 ± 6.63+14.67 ± 1.72
    GSM8K78.87 ± 2.9764.93 ± 6.02+13.93 ± 8.64
    Minerva24.50 ± 1.8221.20 ± 5.23+3.30 ± 4.54
    OlympiadBench26.43 ± 3.4521.17 ± 3.98+5.27 ± 1.06
    Math mean45.23 ± 2.8735.79 ± 5.40+9.43 ± 2.80
    CRUXEval-I52.13 ± 2.2651.29 ± 4.89+0.83 ± 7.09
    CRUXEval-O52.92 ± 1.1348.04 ± 6.25+4.88 ± 7.26
    LiveCodeBench40.36 ± 1.9939.18 ± 6.63+1.18 ± 8.44
    Code mean48.47 ± 1.6046.17 ± 4.89+2.30 ± 6.28

    At the same difficulty, size and training budget, training on high-alignment tasks improves the math mean by 9.4 points and the code mean by 2.3 points on average, and the average is higher on all eight benchmarks.

  2. Alignment carries information beyond difficulty (W1). We used the training logs of the three paper runs, about 18–19k eligible candidates each (0 < s < 1, where s is the fraction of the solver's 8 attempts that are correct). The reward only ranks candidates proposed at the same step for the same task type, so all statistics are computed within such groups.

    • The within-group Spearman ρ between cosine and 1 − s is −0.04 / −0.08 / −0.06 across the three seeds.
    • Among candidate pairs that both rewards can order (different solve rates), 52–53% are ordered oppositely, i.e., the higher-cosine task is the easier one; 50% would mean the two orders are unrelated.
  3. CE proxy vs. RL gradients (W1). The derivation indeed assumes a single CE step, while the solver is trained with policy gradients. Appendix C.1 tests this directly by replacing the CE gradients with REINFORCE estimates of the RL objective. CE is better at the same compute and 1.7× cheaper than REINFORCE on both sides. Together with the matched experiment above, this supports the CE-based reward empirically.

We would welcome any feedback on these results while we prepare the remaining responses. In particular, does this matched experiment address the comparison you had in mind in Q2?

AuthorsOpenReview ↗

Response to Reviewer ibRd (Part 2)

W2 / Q3: Does token-Jaccard novelty reflect mathematical novelty, and do proof strategies change? (semantic audit)

In CompassPlay, the token-Jaccard novelty factor targets one failure mode, a proposer that copies its seed; it is not meant as a general measure of mathematical novelty. Lexical overlap is a standard signal for this kind of near-duplicate filtering (e.g., the ROUGE-L filter in Self-Instruct, Wang et al., 2023, and MinHash-Jaccard deduplication, Lee et al., 2022). We ran the semantic audit you proposed to check whether it does this job at the level of meaning.

Experimental setup. We audited (seed, conjecture) pairs for conjectures that receive a proposer reward: for cos at an early and the final iteration (39 and 150), and for cos × novelty, in a run of the identical configuration, at two iterations after its copy rate has settled below 0.5% (95 and 105; Figure 4). Claude Opus 5.5 labelled a stratified sample of 639 pairs as renaming, equivalent reformulation, minor variant or substantively different. One author also labelled 40 pairs by hand, with κ = 0.87 agreement on copy-like vs. novel. When both statements have verified proofs, we also checked whether the conjecture's proof uses a different strategy from the seed's.

Results. Percentages are over all eligible conjectures:

MethodRenaming or equivalent reformulationMinor variantSubstantively differentProof strategy differs from the seed's
cos, no novelty, iteration 3999.1%0.9%0.0%7%
cos, no novelty, iteration 15096.1%3.8%0.1%10%
cos × novelty, iteration 950.1%0.1%99.8%97%
cos × novelty, iteration 1050%0%100%97%

With the novelty factor, renamings and equivalent reformulations fall from 96–99% to 0–0.1% of eligible conjectures, 99.8–100% are substantively different statements, and 97% of their verified proofs use a different strategy from the seed's.

Token-Jaccard also tracks these semantic labels. Pooling the 639 labelled pairs by J, the share of renamings or equivalent reformulations rises with J:

Token-Jaccard J0–0.20.2–0.350.35–0.50.5–0.650.65–0.80.8–0.90.9–1.0
Renaming0%0%9%30%69%72%86%
Equivalent reformulation0%2%13%12%18%14%9%
Minor variant0%1%9%18%2%10%5%
Substantively different100%97%69%40%10%4%0%

Both failure modes you mention are rare. Below J = 0.35, at most 2% of pairs are renamings or equivalent reformulations, and 98–99% of cos × novelty conjectures fall in this range. At J ≥ 0.65, at most 10% of pairs are substantively different.

W3: Does the novelty constraint yield substantively different statements or surface-level variations of the seeds?

With novelty weighting, 99.8–100% of conjectures are substantively different statements rather than surface-level variations of their seeds, and 97% of their verified proofs use a different strategy from the seed's (table above).

We hope this addresses your questions on the novelty metric and are happy to provide further details.

AuthorsOpenReview ↗

Response to Reviewer ibRd (Part 3)

Q1: How many tasks are generated, verified, filtered and used for the updates?

We collected these statistics from the training logs of the paper's CompassPlay runs, averaged over all steps. In coding, we also average over the three seeds. Here s is the fraction of the solver's 8 attempts that are correct.

Coding.

StageTasks per stepShare of proposed tasks
Proposed by the proposer192100%
Valid programs that pass execution checks16786.8%
Eligible for the alignment reward9348.7%

All 192 proposals enter the proposer update, and ineligible ones receive a fixed penalty instead of the alignment reward. The solver is trained on 192 tasks per step, taken from the pool of valid tasks collected so far.

Lean.

StageConjectures per iterationShare of sampled conjectures
Sampled from the proposer1,967100%
After extraction and deduplication1,94398.8%
Proved at least once by the solver1,61882.3%
Eligible and used for the proposer update98350.0%

In each iteration, the solver is trained on its correct proofs of the statements it proved in at least one but at most half of its 8 attempts. On average, these are 593 statements per iteration.

Dear Reviewer ibRd,

Thank you again for your detailed review. As the discussion closes soon, we would like to briefly recap our response.

  • Alignment and learning gains. In the matched experiment you proposed, high-alignment tasks improve the math mean by 9.4 points and the code mean by 2.3 points over low-alignment tasks of the same difficulty, over three seeds. Appendix C.1 shows that CE gradients do better than REINFORCE estimates of the RL objective at 1.7× lower cost.
  • Semantic novelty. In the semantic audit you suggested, the novelty factor reduces renamings and equivalent reformulations of the seed from 96–99% to 0–0.1% of eligible conjectures, with κ = 0.87 agreement with an author's human labels. With the novelty factor, 97% of verified conjecture proofs use a different strategy from the seed's, and token-Jaccard tracks these labels.
  • Data scale. Per step in coding, 192 tasks are proposed, 167 pass execution checks and 93 are eligible for the alignment reward.

Do our responses resolve your concerns? If so, we would be grateful if you would consider updating your assessment.

Comments on the paper

1 comment

Dear Reviewers,

We understand it's a busy time. As the discussion period is coming to an end, we just wanted to follow up with those who haven't had a chance to respond yet. We hope our clarifications have addressed the main concerns raised in the reviews, and would appreciate it if you could take a look before the discussion closes.

Thanks for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.