- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates whether mathematical fine-tuning transfers more effectively when training examples share the target's reasoning approach rather than its mathematical topic. It proposes a counterbalanced experimental design comparing same-approach/different-topic (SA) supervision against same-topic/different-approach (ST) supervision.
The authors evaluate two synthetic mathematical datasets, five pretrained language models, and three training seeds. They report average SA-ST advantages of 10.8 and 14.3 percentage points, respectively, with positive seed-pooled contrasts across all 40 model-target comparisons. Additional analyses examine full fine-tuning, training-set size, absolute transfer, and statement similarity.
The central contribution is an empirical comparison of two criteria for selecting mathematical fine-tuning data. However, the experimental design identifies a performance contrast for specific source-target pairs rather than establishing that transfer of the intended reasoning mechanism explains the observed advantage.
Critical strengths
1. Structured experimental comparison.
The rotational design is a meaningful improvement over comparing independently selected datasets. Each source appears once under each matching relationship, so additive source-quality effects cancel from the equally weighted aggregate contrast. The paper also appropriately acknowledges that source-target interactions remain.
2. Consistent reported empirical ordering.
The SA-ST advantage is positive across all reported seed-pooled model-target comparisons in both synthetic designs. The authors additionally examine transfer relative to the base models, full-parameter fine-tuning, and different training sizes. Reporting per-seed results and acknowledging that bootstrap confidence intervals exclude training-seed uncertainty are positive aspects of the analysis.
Critical weaknesses
1. Fundamental identification limitations undermine the central mathematical interpretation.
The principal theoretical justification in Section 3.3 establishes only the cancellation of additive source and target effects. Under Equation (4),
= \mu + u_\sigma + v_\tau + w_{\sigma,\tau},$$ the aggregate SA-ST contrast eliminates the additive terms but leaves differences between unrestricted source-target interactions. Consequently, the measured advantage cannot distinguish genuine transfer of the intended reasoning approach from other pair-specific interactions, including recurring solution templates, problem-generation regularities, latent structural similarities, or differences in mathematical difficulty. This limitation is especially consequential because each $2 \times 2$ design contains only two undirected source-target pairs per matching relationship, evaluated in both directions. Repeating these comparisons across models and seeds does not generate independent evidence across a broad population of mathematical topic-approach relationships. The reported positive contrasts therefore demonstrate an empirical ordering for the selected cells, not a general mathematical principle governing transfer. Moreover, the approach categories are not necessarily disjoint. Invariant arguments can overlap with counting identities, while complement and pigeonhole reasoning can coexist within the same solution. The paper does not establish that its approach labels uniquely characterize the mathematical problems or their valid solution methods. Agreement among LLM judges is not an independent mathematical validation of the classification. Since evaluation checks final-answer correctness without requiring the intended approach, the experiments also do not directly demonstrate that the models transferred or used the claimed reasoning method. The rotational cancellation argument is correct, but it does not resolve these identification problems. The interpretation of the observed accuracy difference as evidence for a particular reasoning-transfer mechanism is therefore substantially underdetermined. **2. Statistical inference, effect heterogeneity, and evaluation reliability are insufficiently established.** The bootstrap procedure in Appendix F resamples individual evaluation problems conditional on the trained checkpoints. Although the authors correctly acknowledge that the resulting intervals exclude training-seed uncertainty, the analysis also does not account for the selection of topic-approach combinations or potential dependencies among synthetically generated problems. If the generator produces related templates or problem families, independent problem-level resampling may underestimate uncertainty. The 40 reported positive model-target contrasts should not be treated as 40 independent confirmations because they reuse related evaluation sets, source checkpoints, and a common construction pipeline. The second-design replication exhibits substantial heterogeneity that is obscured by the aggregate mean. From Table 7, the average SA-ST advantage is approximately 24.7 percentage points for pigeonhole targets but only 3.7 points for complement targets. Some individual effects are very small, including +0.8 points for Mistral on number-theory/complement. No target-level confidence intervals are provided for the second design. Consequently, the replication does not establish comparably strong effects across both reasoning categories. The mathematical correctness and classification validation are also insufficient. The final synthetic dataset relies heavily on LLM-generated problems, LLM answer agreement, and LLM approach judgments, with only 50 manually reviewed problems in the primary dataset. The manuscript does not quantify residual classification errors or independently establish the reliability of its approach assignments. The precise answer-extraction and mathematical-equivalence procedures are also unspecified, making it difficult to assess whether accuracy differences reflect mathematical reasoning improvements or evaluation-format effects. The statement-similarity analysis does not eliminate alternative explanations. Average lexical and embedding similarity favoring ST does not imply that approach matching is responsible for higher accuracy at the individual-problem level. Indeed, Table 13 reports that combinatorics/invariant targets have their nearest out-of-cell embedding neighbor in the SA cell more frequently than in the ST cell (52.8% versus 47.0%). This does not establish a similarity-based explanation, but it illustrates why aggregate similarity statistics alone cannot exclude one. Finally, the training-budget ablation changes both the number of training examples and the number of optimization steps, preventing separate identification of their effects. Taken together, these concerns affect the central scientific interpretation, statistical reliability, and reproducibility of the results. Addressing them convincingly would require independent validation of the approach taxonomy, stronger controls for synthetic-data dependencies, more complete statistical analysis, and experiments capable of distinguishing the proposed mechanism from alternative source-target interactions.Questions for authors
1. Mathematical identification
Equation (4) leaves unrestricted source-target interactions after additive-effect cancellation. What evidence establishes that the remaining SA-ST contrast specifically reflects transfer of the intended reasoning approach rather than other source-target interactions? Can the authors provide an experimental control that distinguishes these explanations?
2. Approach classification and mathematical validity
How are approach labels validated independently of the generator and LLM judges? How are problems with multiple valid solution methods handled? What fraction of problems admits solutions using approaches other than their assigned labels, and how does this affect the interpretation?
3. Statistical inference and replication
How would the confidence intervals change under template- or problem-family-aware resampling, if the generated data contain dependent examples? Can the authors provide target-level confidence intervals for the second design and explain why the reported advantage is much larger for pigeonhole than for complement targets?
4. Evaluation reliability
What exact answer-extraction and mathematical-equivalence rules determine strict correctness? What independent evidence establishes the reliability of these decisions, particularly for models with near-zero base accuracy? Could grading or output-format differences account for part of the measured transfer advantage?
5. Similarity and optimization confounds
Does the SA advantage remain after controlling for individual-problem similarity and recurring solution templates? In the training-size ablation, can the authors separate the effect of increasing the number of examples from the effect of increasing optimization steps?
8 comments
Response to Weakness 2: Statistical Inference, Effect Heterogeneity, and Evaluation Reliability
1. Statistical inference and effect heterogeneity are being interpreted against an incorrect standard.
The manuscript explicitly defines the uncertainty quantified by the reported confidence intervals (Section 4.3 and Appendix F) and separately reports training-seed results. Our evidence spans five models, three seeds, four target rotations, and two distinct experimental designs; we do not treat the resulting 40 model–target comparisons as statistically independent replications.
More importantly, our hypothesis concerns whether approach-matched supervision yields greater transfer than topic-matched supervision, not whether the effect must have identical magnitude across reasoning categories. The reviewer emphasizes the smaller advantages on complement targets, but differences in effect magnitude do not contradict the primary, equally weighted SA−ST comparison defined in Section 3.2.
The second design reports positive SA−ST point estimates for every model–target combination, despite changing both mathematical topics and reasoning approaches. The reviewer identifies heterogeneity in these estimates, but heterogeneity alone does not establish that the aggregate comparative finding is unreliable. The requested dependence-aware and target-level uncertainty analyses are provided in our response to Question 3.
2. The mathematical-validation criticism does not establish that the reported results are unreliable.
Section 4.1 documents a multistage verification process. Each candidate undergoes eight unconstrained re-solutions, followed by approach classification requiring unanimous agreement among three LLM judges. Two of these models also independently solve the problems without access to the reference solutions, with answer agreement required for retention. The process also includes the rejection and replacement of problems with inconsistent answers. Separately, 50 randomly selected problems per cell underwent manual review, and all 50 were verified as correct.
The reviewer questions the sufficiency of this procedure without identifying a concrete mathematical error or demonstrating a systematic validation bias favoring SA. The use of synthetic data and LLM-assisted verification does not itself establish that the resulting empirical comparison is unsound.
Furthermore, Section 3.1 explicitly acknowledges boundary cases between mathematical approaches and defines labels according to reference solutions rather than requiring unique valid methods. The classification and alternative-method questions are examined further in our response to Question 2.
3. The evaluation-reliability criticism must be distinguished from the actual comparative estimand.
The primary outcome is the difference in final-answer accuracy between SA and ST on identical held-out target problems (Section 3.2). Both conditions are evaluated under the same protocol, and neither receives a different correctness criterion.
The reviewer raises evaluation-format differences as a possible explanation but provides no evidence of a systematic grading effect favoring SA. Moreover, the concern about near-zero base accuracy does not directly challenge the primary SA−ST estimand, which compares two fine-tuned conditions rather than relying on their absolute accuracy relative to the base model.
The exact extraction and equivalence rules, together with independent evidence of grading reliability, are provided in our response to Question 4.
4. The similarity criticism does not establish an alternative explanation for the observed ordering.
The reviewer highlights combinatorics/invariant, where the nearest out-of-cell embedding neighbor belongs to SA more frequently than ST (52.8% versus 47.0%). However, this is the only target among the four primary-design cells with that ordering. The other three favor ST, as do the aggregate lexical and embedding similarity measures reported in Section 6.
Our similarity analysis is a diagnostic showing that greater measured statement resemblance does not generally correspond to greater transfer. It is not presented as proof that every possible source–target interaction has been eliminated. The cited individual exception does not establish that similarity explains the observed advantage. We address individual-problem similarity and recurring solution templates directly in our response to Question 5.
Q1, Q2, Q3
Note on numerical results. The original run artifacts became unavailable under the university's storage-retention policy. We therefore reran the experiments using the same datasets, models, seeds, training configurations, and evaluation pipeline. The numerical results below come from these reruns; the SA−ST ordering remains positive across all 40 model–target comparisons. Additional analyses and controlled experiments are identified below.
1. Mathematical identification
We report the different-on-both (DH) control, whose training source shares neither topic nor designated approach with the target. Because each source serves once as SA, ST, and DH across the rotations, aggregate contrasts retain the cancellation of additive source effects. SA−DH compares sources differing in approach match while both differ in topic; ST−DH compares sources differing in topic match while both differ in approach.
- SA−DH: +9.2 to +15.4 percentage points (pp) in Design 1 and +12.0 to +15.2 pp in Design 2; all ten model-level 95% CIs exclude zero.
- ST−DH: −1.6 to +0.8 pp across the ten model-level comparisons.
Thus, the SA advantage is not merely a disadvantage of the particular ST source: approach-matched training also consistently outperforms a source matching neither attribute, whereas topic matching adds little relative to DH. The similarity and recurring-template alternatives raised in the question are directly tested in Q5.
2. Approach classification and mathematical validity
We added a deterministic, non-LLM approach-vocabulary check, separate from the generator and LLM judges. Using fixed, approach-specific cues in reference solutions, it finds the assigned approach's vocabulary in 95–98% of Design 1 solutions and 100% of Design 2 solutions. This checks expressed approach markers without an additional LLM judgment. Multi-method problems are retained: training uses the labeled reference solution, while grading accepts any correct method.
To examine alternative valid methods, we identify targets that at least one untuned base model answers correctly without using vocabulary associated with the assigned approach. This applies to 11–22% of Design 1 targets and 13–34% of Design 2 targets. These are measurable proxy rates, not an exhaustive enumeration of every mathematically possible solution method.
On this proxy-defined subset, SA−ST remains positive in all 40 model–target comparisons. The result therefore persists for targets where base-model behavior supplies evidence of alternative solution routes.
3. Statistical inference and replication
We recomputed confidence intervals by resampling entire similarity-defined problem families, including families spanning target cells. Families are constructed by average-linkage clustering of statement and reference-solution embeddings using Qwen3-Embedding-0.6B; we test multiple similarity thresholds.
The intervals widen as expected, yet all ten model-level SA−ST CIs remain above zero, even under the coarsest clustering (20 families in Design 1; eight in Design 2). Using the rerun outputs, Qwen's Design 1 CI widens from [15.3, 18.9] under problem-level resampling to [9.6, 21.8] under family-aware resampling; in Design 2, it widens from [13.4, 18.6] to [7.4, 18.7].
The requested Design 2 target-level SA−ST estimates and 95% family-aware CIs are below (pp; mean statement/solution cosine threshold ≥0.6, 34 families):
| Target | Qwen3-4B | Llama-3-8B | DeepSeek-Math-7B | Mistral-7B | Gemma-3-4B |
|---|---|---|---|---|---|
| Number theory / complement | +4.7 [2.1, 11.7] | +4.7 [1.2, 7.2] | +4.5 [1.2, 9.2] | +1.5 [1.0, 3.0] | +2.2 [0.1, 4.3] |
| Number theory / pigeonhole | +22.3 [11.2, 26.7] | +20.0 [15.4, 28.1] | +17.7 [10.5, 25.6] | +14.7 [6.8, 23.6] | +19.8 [14.1, 30.5] |
| Geometry / complement | +13.0 [7.7, 20.2] | +2.5 [0.6, 4.4] | +6.7 [3.4, 11.0] | +2.0 [0.2, 3.5] | +2.0 [0.5, 4.3] |
| Geometry / pigeonhole | +26.0 [13.7, 36.6] | +30.8 [9.8, 43.6] | +27.3 [7.1, 39.2] | +30.0 [8.8, 42.6] | +23.7 [2.8, 35.4] |
All 20 Design 2 point estimates favor SA; all 20 family-aware CIs exclude zero. Across both designs, all target-level family-aware CIs exclude zero at this threshold. As with the original analysis, these intervals are conditional on the trained checkpoints.
For the effect-size difference, complement performance is near floor for four models (SA accuracy only 2.5–10.0%). On pigeonhole targets, 12–34% of ST generations end without an answer at the shared token limit, versus 1–7% for SA. We also remove the 23% of pigeonhole targets for which the statement exposes a direct "bound plus one" shortcut. On the remaining 77%, SA−ST remains +5.8 to +29.7 pp, with all corresponding CIs above zero.
Response to Weakness 1: Fundamental Identification Limitations
We strongly disagree that the stated concerns undermine our central finding.
1. The four positive directional contrasts provide evidence against a purely additive explanation.
Both conditions are evaluated on identical target problems, canceling additive target-difficulty effects. Each source appears equally as SA and ST, canceling additive source effects in the aggregate.
More importantly, under an additive-only model, the contrasts must satisfy Δ(PI) = −Δ(CD) and Δ(CI) = −Δ(PD). Thus, both members of either pair cannot be positive in expectation. Yet all four observed contrasts favor SA across all five models in both designs.
This provides evidence of systematic source–target interactions aligned with approach matching, rather than differences in intrinsic source quality or target difficulty. An alternative explanation must account for why these interactions consistently favor same-approach supervision across different mathematical topics and in both transfer directions.
2. Approach overlap can favor ST rather than SA.
The criticism concerning overlapping approaches overlooks an important implication of our design. The SA source already shares the target's designated approach, whereas ST uses a different designated approach. If these approaches overlap, ST may also provide supervision relevant to the target's approach, potentially narrowing the observed SA−ST advantage. Such overlap cannot simply be assumed to inflate the SA effect.
Moreover, Section 3.1 explicitly defines approach labels according to reference solutions, not as exclusive classifications of all valid methods. Evaluation accepts any mathematically correct solution, as required by our accuracy-based transfer comparison.
3. The reviewer challenges a claim our paper does not make and overlooks the evidence from independent experimental designs.
The reviewer concludes that "the interpretation of the observed accuracy difference as evidence for a particular reasoning-transfer mechanism is therefore substantially underdetermined" and uses this to argue that our central finding is undermined.
However, our study does not claim to identify a particular internal reasoning-transfer mechanism. It directly compares the effectiveness of approach-matched versus topic-matched supervision for mathematical fine-tuning. Sections 3.3 and 7 explicitly acknowledge the limitations of mechanistic identification.
The reviewer also understates the generalizability evidence provided by our second experimental design. We do not merely repeat the same source–target relationships across models and seeds. We conduct a second factorial experiment using entirely different mathematical topics and reasoning approaches, yet observe the same positive SA−ST ordering across all model–target comparisons. This replication demonstrates that the finding is not confined to the particular topic–approach relationships of the primary design.
Identifying the internal mechanism responsible for transfer is a fundamentally different research question, requiring a different experimental design that investigates additional factors, including fine-tuning objectives and methods, training dynamics, solution representations, and model-internal computations.
Our experiment deliberately controls the fine-tuning procedure to isolate the comparison between supervision-selection criteria. Requiring identification of internal mechanisms would therefore impose an entirely different research objective rather than evaluate the validity of the experiment we conducted. The inability to uniquely identify an internal mechanism does not invalidate our empirical comparison.