- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper discussed CLB is a necessary but a bottleneck to solve Lean 4 questions. Therefore, based on CLB, they propose S2B-LM, removing numerical processing and adding chain-of-thought scaffolding. This is used for adj-ZSCT scoring. Then via experiments on minif2f, it gives a claim that CLB is a necessary but insufficient condition for formalizing difficult mathematics, and there is causal evidence for this.
Critical strengths
- the method is interesting that the paper clearly states the motivation of the two changes based on CLB, and the two changes are clearly targeting the motivation.
- experiments are a lot
- the findings are interesting and method provides insights
Critical weaknesses
- High score model such as Goedel-V2、DeepSeek-V2、Kimina have been trained via CoT while S2B-LM is prompted via CoT. I am concerned that adj-ZSCT is measuring the effectiveness of CoT or the ability that the model could use CoT. - maybe just try another model is not trained extensively via CoT?
- Actually the 10 models the paper assessed are from 3 families. Between the models stemming from the same family, I am not sure why the permutation test is valid.
- 600B+ model is MoE model - so I am concerned that the claim of model size is ruled out as a confound
- casual and effects tests are too narrowed: you only have 15 AIME tests, which reduced the persuasiveness of the experiments
- I suppose another experiments are worth investigating: for the lower difficulty level questions, whether the score still holds. I understand this is not the core investigation of the paper - as you are investigating the high-difficulty questions, but it will give us a confirmation whether the decrease from 32.3% to 2.9% is really due to CLB suppresion.
- p=0.004 is from the overall mark, but try to only use 80 difficulty questions and see whether the results changed?
- The language is too complicated so I mark the presentation is difficult to follow. There are even sentences longer than 5 row. And there are places such as [y] in line 48.
I am happy to reconsider the overall evaluation if the author addressed my concerns.
Questions for authors
see above
16 comments
Rebuttal (2/4)
W5: "for the lower difficulty level questions, whether the score still holds [...] confirmation whether the decrease from 32.3% to 2.9% is really due to CLB suppresion."
We thank the reviewer for this suggestion, which we understand as a check of the construct validity of the causal result (is the drop due to CLB suppression, or to generic damage?). We ran the same experiment on 24 lower-difficulty number-theory problems from the MATH subset of miniF2F (19 problems turned out informative), with 8 samples per problem. The drop holds and is direction-specific: CLB suppression lowers the solve rate from to , while the random direction leaves it at , which confirms that the decrease is due to CLB suppression.
The drop is, however, not larger on harder problems: the relative drop (the fall in solve rate as a fraction of the unsteered rate) is 0.793 on AMC and 0.951 on MATH-NT, so our pre-registered one-sided contrast (larger drop on harder problems) is not supported (difference −0.158, p = 0.993). This is not a ceiling effect, since no difficulty gradient appears at any dose, including , where neither set saturates (relative drops of 0.405 on AMC and 0.387 on MATH-NT). It is also consistent with necessary but not sufficient: suppressing a necessary competency hurts at every difficulty level, while what sets hard problems apart lies in the competencies beyond CLB. If anything, this result extends our causal claim: the CLB direction is needed for formal proving at both difficulty levels we tested, not only at the Olympiad level.
In light of this result, we hope that you can update your appreciation of the paper and possibly your score, or point out how we could improve it further.
W2: "Actually the 10 models the paper assessed are from 3 families. Between the models stemming from the same family, I am not sure why the permutation test is valid."
We understand this point as a threat to the statistical validity of the permutation test, which assumes that the units it permutes are exchangeable. We agree that the paper should have stated its unit of analysis explicitly: it is the training recipe, not the base weights. Indeed, each model stands for the recipe (data and training procedure) that turned a base model into a formal theorem prover, and the question we test is whether any recipe that reaches the Olympiad tier of miniF2F does so without CLB competency. Under this reading, models sharing a base are not repeated measurements of one unit but distinct treatments applied to a common starting point.
In more detail, Table 2 shows that the base weights do not determine CLB competency, while the recipes do: five recipes applied to the same base, DeepSeek-Prover-V1.5-Base, yield adj-ZSCT of 10.6 (Goedel-Prover-DPO), 12.0 (DeepSeek-Prover-V1.5-SFT), 13.8 (DeepSeek-Prover-V1.5-RL), 16.3 (Goedel-Prover-SFT) and 79.6 (DeepSeek-Prover-V2-7B). A shared base would induce dependence between the scores of models sharing it, but this spread shows that such dependence is weak compared with the effect of the recipes.
We have revised the paper accordingly, with all changes highlighted in blue: Section 4.1 now states the training recipe as the unit of analysis; Appendix D states it where the exchangeability assumption of the permutation tests is introduced, with the evidence above; and the introduction and discussion now phrase the main finding per training recipe ("no training recipe reaches the Olympiad-level tier without measurable CLB competency"). In light of this clarification, we hope that you can update your appreciation of the paper and possibly your score, or point out how we could improve it further.
Rebuttal (3/4)
W3: "600B+ model is MoE model - so I am concerned that the claim of model size is ruled out as a confound"
We thank the reviewer for this point, which concerns the internal validity of our scale argument, and which is correct. Counting DeepSeek-Prover-V2-671B by its active parameters (about 37B) instead of its total, scale correlates significantly with miniF2F (exact permutation test: r = 0.61, p = 0.018; r = 0.70, p = 0.010 on a log scale), whereas the paper reported r = 0.36, p = 0.18 with total parameters. The statement that scale is "ruled out" as a confound is therefore too strong, and we have removed it.
Our claim, however, is one of necessity, and necessity separates the two variables. No training recipe reaches the Olympiad tier without CLB competency (the forbidden cell of our quadrant test remains empty, p = 0.004), and this still holds without the MoE model (N = 9, p = 0.0079). By contrast, training recipes do reach the Olympiad tier without scale: DeepSeek-Prover-V2-7B (75.6%) and Goedel-Prover-V2-8B (84.6%) are among the smallest models of the pool. Scale therefore correlates with miniF2F performance but is not necessary for it, while CLB competency is; the revised paper states it this way and reports the scale tests with both total and active parameters.
In light of this correction, we hope that you can update your appreciation of the paper and possibly your score, or point out how we could improve it further.
W6: "p=0.004 is from the overall mark, but try to only use 80 difficulty questions and see whether the results changed?"
We understand this point as a threat to the construct validity of our outcome measure: the claim concerns reaching the Olympiad tier, but the quadrant test uses the overall miniF2F Pass@32 over all 244 test problems, which also reflects the 164 easier ones. The reviewer is right that the test uses the overall score. Our tail-partition test (Appendix D), which also gives , splits models at 67.2% by design: this is the share of non-Olympiad problems in the test split (164 of 244), so a model can only score above it by solving Olympiad-level problems (AMC, AIME and IMO). In our pool, the median split of the quadrant test (67.95%) happens to give the same partition, and the models above it score between 75.6% and 88.1%.
Pass@32 restricted to the 80 Olympiad problems is, however, not available for all ten models, since published results report the overall score and we would need each model's per-problem outputs. We therefore cannot run the quadrant test on that subset now, and we report it as a limitation.
In light of this clarification, we hope that you can update your appreciation of the paper and possibly your score, or point out how we could improve it further.
Rebuttal (1/4)
We thank reviewer q4zS for a detailed review, and for noting that (S1) the two changes of S2B-LM are clearly motivated by CLBs and clearly target that motivation, (S2) the experiments are extensive, and (S3) the findings are interesting and the method provides insights. We also thank the reviewer for offering to reconsider their evaluation, and we address the weaknesses in order of their bearing on the paper's claims.
We start with the two points on the scope of the causal experiment, W4 (the narrow AIME set) and W5 (the lower-difficulty problems), which we answer with new experiments. We then turn to the three points on the correlational test: for W2 (model families), we clarify the unit of analysis; for W3 (the MoE model), we correct our scale claim; and for W6 (the 80 hard problems), we explain our threshold and state a limitation. We then address W1 (the chain-of-thought confound), and finally W7 (the readability of the paper).
W4: "casual and effects tests are too narrowed: you only have 15 AIME tests, which reduced the persuasiveness of the experiments"
This concern is shared with reviewer P3sz (Q1), and we address the underlying concerns with the same changes to the paper in Section 4.2.
We understand that the reviewer raises threats to the external validity (narrow sample) and statistical validity (power) of the causal intervention experiment of the paper (Section 4.2). We raise both validities in two ways:
We test on the problems where the unsteered model (α=0) solves at least once, because a problem the model never solves cannot show a drop under suppression. This rule is applied to the unsteered run alone, before any steered arm is generated, so the selection cannot favour our hypothesis. As a result, it keeps 10 of the 15 AIME problems (32 samples each) and 15 of the 39 AMC problems (8 samples each), giving a total of 25 informative problems. We also add an orthogonal control: a random direction orthogonal to the CLB direction, with the same norm.
Pooled over the 25 informative problems, CLB suppression lowers the solve rate from 0.629 to 0.107, while the random and orthogonal controls leave it at 0.595 and 0.586. Per problem, CLB suppression solves less often than random on 21 of 25 problems and more often on none (one-sided sign test, p = 4.8e-7), and less often than orthogonal on 20 and more often on none (p = 9.5e-7). The conclusion therefore does not depend on the 15 AIME problems alone. In the revised paper, Figure 6 now shows the per-problem and pooled solve rates on the 25 informative problems, and Figure 7 shows the pooled output coherence. The sign-test results are reported in the main text at lines 456–459. The full tables are in our answer to reviewer P3sz (Q1).
In light of this addition that extends both the external and statistical validity of the paper, we hope that you can update your appreciation of the paper and possibly update your scores, or point out how we could improve the paper further.