- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CompassPlay, a new method that uses gradient alignment to redesign the proposer reward in self-play. CompassPlay compares the loss gradients of candidate tasks and reference set to decide whether the generated tasks can help improve the solver's target capabilities. The authors use a first-order approximation and cosine similarity to compute the reward, avoiding additional solver training for each candidate task like SOAR. The authors also evaluate CompassPlay using Qwen2.5-Coder-7B for coding self-play on coding benchmarks and mathematics benchmarks. They also evaluate the method using DeepSeek-Prover-V2-7B for Lean 4 theorem proving. The experimental results show improvements in performance and training efficiency compared with difficulty-based rewards.
Critical strengths
-
One strength of this paper is using a first-order gradient approximation to estimate the training value of generated tasks. This avoids the additional solver training required by methods like SOAR, which can save a lot of computational cost. The Lean 4 experiments also illustrate that CompassPlay can achieve the same coverage with the difficulty baseline with fewer GPU hours.
-
The authors evaluate the method in two different domains, coding and theorem proving. They also provide ablation studies on gradient types, cosine similarity vs. dot product, and novelty weighting, which help explain how different designs affect the final performance.
Critical weaknesses
-
One of main concern is the selection of the reference set. The coding experiments only use 32 MATH training problems as a fixed reference set, and the reward directly depends on the gradients of these problems. I am concerned that different choices of reference set may affect the stability of the training results.
-
Although the authors treat mathematics as an out-of-domain evaluation, mathematical problems have already been used in the training process through the reference gradient. Therefore, I don't think the current results are sufficient to demonstrate that the method can generalize to domains not covered by the reference set.
Questions for authors
-
Can the authors further evaluate the generalization ability of CompassPlay? For example, in the coding experiments, although mathematics is treated as out of domain, the reward already uses gradients from mathematical problems. If the reference set is replaced with coding problems or tasks from other domains, would we still observe similar improvements on mathematics benchmarks?
-
More additional experiments on reference set selection is encouraged. For example, using different randomly sampled sets of 32 mathematical problems or changing the size of the Reference Set to see how much the training results vary. This would help demonstrate whether the method is sensitive to the choice of reference set.
-
It's better to further explore whether novelty weighting actually reduces repetition at the mathematical semantic level. For example, could the generated theorems simply just the reformulations of the seed theorem rather than brand new tasks as simply Token-Jaccard cannot guarantee this.
3 comments
Response to Reviewer uH4d (Part 2)
W1 / Q2: Do training results vary with the choice and size of the reference set?
We trained the runs with alternative reference sets to step 100 and evaluated them with the same pipeline as the paper. We also add a different random sample of 32 MATH problems, as you suggested.
Results at step 100:
| Reference set | Code mean | Math mean |
|---|---|---|
| 32 MATH problems in the paper, mean ± sd over three seeds | 55.5 ± 1.8 | 44.9 ± 1.4 |
| A different random sample of 32 MATH problems | 55.8 | 47.5 |
| 8 MATH problems | 53.8 | 39.6 |
| 128 MATH problems | 57.3 | 51.8 |
A different random sample of 32 problems performs on par with the paper's set. Accuracy rises with the size of the reference set, most clearly on math, so the paper's 32 problems are not the best case for CompassPlay.
This completes our response to all points. We are happy to answer any further questions.
Dear Reviewer uH4d,
Thank you again for your review. As the discussion closes soon, we would like to briefly recap our response.
- Choice and size of the reference set. A different random sample of 32 MATH problems performs on par with the paper's set, and accuracy rises with the size of the reference set, most clearly on math.
- Math gains without math in the reference. With a coding reference set, the math mean at step 100 is 44.7, versus 44.9 with the paper's math reference.
- Semantic novelty. In a blind semantic audit, the novelty factor reduces renamings and equivalent reformulations of the seed from 96–99% to 0–0.1% of eligible conjectures.
Do our responses resolve your concerns? If so, we would be grateful if you would consider updating your assessment.
Response to Reviewer uH4d (Part 1)
Thank you for the review! We did some new experiments and anslysis to answer your questions.
W1 / Q2: Is the method sensitive to the choice and size of the reference set?
Experimental setup. We rescored 1,296 coding tasks generated by the base model with each alternative reference set. Among tasks with the same type and solve rate, we call the top third by reward the selected tasks, and check how many of the tasks selected with the paper's reference (32 MATH problems) are still selected.
Results.
A different random sample of 32 MATH problems and a 128-problem reference keep most of the selected tasks, and almost none of them (at most 0.7%) drop to the bottom third. Switching to coding problems changes the selection much more. The reward is therefore stable within a reference domain, and the domain is what steers it.
W2 / Q1: Would a reference set without math still improve the math benchmarks?
Yes. We reran CompassPlay with 32 held-out MBPP coding problems as the reference set, disjoint from the code benchmarks, changing nothing else. At step 100, the math mean is 44.7 ± 3.9 with the coding reference, versus 44.9 ± 1.4 with the paper's math reference and 43.8 ± 3.7 for the difficulty reward. Even when the reference contains no math, the improvement on the math benchmarks is largely preserved.
Results at step 100 (mean ± sd over three seeds):
Q3: Does novelty weighting reduce repetition at the mathematical semantic level?
Yes. We ran a blind semantic audit of (seed, conjecture) pairs. Claude Opus 5.5 labelled a stratified sample, reweighted to all eligible conjectures. One author also blind-labelled 40 pairs by hand, with κ = 0.87 agreement on copy-like vs. novel. We audited two checkpoints per method: an early and the final iteration for cos (39 and 150), and two iterations for cos × novelty after its copy rate has settled below 0.5% (95 and 105; Figure 4). Percentages are over all eligible conjectures at these iterations:
Renamings and equivalent reformulations of the seed fall from 96–99% of eligible conjectures under cos alone to 0–0.1% under cos × novelty. Under cos × novelty, 99.8–100% of conjectures are substantively different statements, and 97% of their verified proofs use a different strategy from the seed's proof.
We would welcome any feedback. In particular, does the coding-reference result above address your concern?