- Rating
- 3
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies transfer between mathematics and code. It constructs controls intended to match the output format by adding a shared block of the fine-tuning update to the base model. It provides a method for detecting concealed transfer through parameter edits constructed without target benchmark scores. These edits test whether the update contains gains hidden in the original evaluation. The edited model is compared with both the fine-tuned model and the control to assess whether these gains become measurable. On Qwen3-4B, the analysis isolates a component carrying a tiny fraction of the update energy. Removing this component restores code openings and reveals the gain without a detectable decline in mathematics performance.
Critical strengths
1. A useful measurement approach supported by careful interventions
The paper addresses an important measurement problem: benchmark scores may not fully reflect gains learned during fine-tuning. The controls help account for changes in output format. Applying the same prefix to the control and the fine-tuned model further distinguishes the effect of the response prefix from gains carried by the update. On Qwen3-4B, the analysis compares the identified direction with random directions matched in rank and energy. It also checks that removing this direction does not measurably reduce mathematics performance. These experiments provide convincing evidence for gains that remain hidden before intervention.
2. Thorough empirical evaluation
The paper supports its claims with a thorough set of experiments. It evaluates both directions of transfer across eight base models and includes a separate test on an external Qwen3-14B checkpoint. Paired confidence intervals and checks with longer generation budgets help assess the reliability of the results. The evaluation also reports cases where concealed transfer is not observed, making the scope of the findings clearer.
Critical weaknesses
1. The conditions for control validity remain unclear
When evaluating transfer from code to mathematics, the rnj-1 control produces empty outputs on at least 470 of 500 problems under each of three decoding seeds. The base model is therefore used instead. For OLMo-3-7B, 1,071 of 1,500 control outputs are unfinished, so the paper does not interpret the score difference as transfer. These cases show that the construction does not by itself guarantee a suitable control for comparing capabilities. The handling of these failures is described case by case, without general criteria for control validity. When the base model replaces the control, format changes can again contribute to the measured difference. This limits confidence in applying the proposed measurement procedure across models.
2. The effect of the response prefix on the Qwen3-14B comparison remains unclear
The external Qwen3-14B experiment demonstrates a gain from parameter editing under the reported evaluation setup. The control also improves substantially with a code prefix, from 921 to 1,030. Equivalent prefix results are not reported for the released checkpoint or the edited model. It therefore remains unclear whether the editing advantage persists when the same prefix is applied to all models. This concerns the robustness of the result to response conditions, rather than the existence of the reported gain.
On Qwen3-14B, a random mask of matched energy also clears the family threshold (1,067 vs. 982), so discussing what this implies for the specificity of the releasing edit would be helpful.
3. The conditions under which opening genre masks gains remain open
MiMo-7B expresses code gains despite mathematics openings, and Qwen3-14B releases gains without substantially changing those openings. These cases leave open when the opening genre masks a gain. The paper does not claim that the mechanism identified on Qwen3-4B extends to other models, and it states that what recurs is the concealment rather than the direction. Even so, a brief analysis using the existing generations, such as pass rates by opening genre or whether mathematics-style openings still lead to extractable code, could help explain these differences. A general account across models would be a natural direction for future work.
Questions for authors
1. Control validity — clarification
What criteria determine whether a constructed control is suitable for comparing capabilities? When it is replaced by the base model, how should the resulting score difference be interpreted given the remaining effects of output format?
2. Qwen3-14B — limited additional validation
Does the editing advantage persist when the same code prefix is applied to the released checkpoint, the control, and the edited model? A paired comparison on the same 1,055 LiveCodeBench problems and decoding seeds would address this sensitivity without additional training.