acceptodds
Under review as a conference paper at ICLR 2027

Bidirectional Transfer, Asymmetric Expression: Math Fine-Tuning Conceals the Code Gains It Carries

Abstract

Transfer between mathematics and code is usually scored as net change on a target benchmark against the base model. Reports disagree even on its sign, yet all read that change as what the update contains. We find instead that an update carries two things, the capability it has learned and the answering mode it adopts at its first token, and the latter can mask the former across domains. We call this masking concealed transfer and measure it against a format-matched control, the base plus the update rows that carry format. Against this control, transfer on Qwen3-4B runs both ways but asymmetrically: code fine-tuning raises mathematics directly, whereas mathematics fine-tuning shows no detectable difference on code. This apparent absence is not an absence of transfer. An edit fixed before evaluation and blind to target scores lifts the code score above both the fine-tuned model and the control. The same concealment recurs at three checkpoints across two architectures, each with its own releasing edit. On Qwen3-4B the concealment traces to one direction in weight space carrying a vanishing share of the update's energy. Removing it restores code openings and releases the gain with no measurable source-domain cost. Random directions of matched rank and energy do neither, and this direction alone does not shift the opening. These findings recast cross-domain transfer as separating what a model can solve from how it begins to answer, and an undetected net difference does not establish zero transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-domain transfer, supervised fine-tuning, task vectors, weight-space editing, mathematical reasoning, code generation, evaluation methodology, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3DcFOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies transfer between mathematics and code. It constructs controls intended to match the output format by adding a shared block of the fine-tuning update to the base model. It provides a method for detecting concealed transfer through parameter edits constructed without target benchmark scores. These edits test whether the update contains gains hidden in the original evaluation. The edited model is compared with both the fine-tuned model and the control to assess whether these gains become measurable. On Qwen3-4B, the analysis isolates a component carrying a tiny fraction of the update energy. Removing this component restores code openings and reveals the gain without a detectable decline in mathematics performance.

Critical strengths

1. A useful measurement approach supported by careful interventions

The paper addresses an important measurement problem: benchmark scores may not fully reflect gains learned during fine-tuning. The controls help account for changes in output format. Applying the same prefix to the control and the fine-tuned model further distinguishes the effect of the response prefix from gains carried by the update. On Qwen3-4B, the analysis compares the identified direction with random directions matched in rank and energy. It also checks that removing this direction does not measurably reduce mathematics performance. These experiments provide convincing evidence for gains that remain hidden before intervention.

2. Thorough empirical evaluation

The paper supports its claims with a thorough set of experiments. It evaluates both directions of transfer across eight base models and includes a separate test on an external Qwen3-14B checkpoint. Paired confidence intervals and checks with longer generation budgets help assess the reliability of the results. The evaluation also reports cases where concealed transfer is not observed, making the scope of the findings clearer.

Critical weaknesses

1. The conditions for control validity remain unclear

When evaluating transfer from code to mathematics, the rnj-1 control produces empty outputs on at least 470 of 500 problems under each of three decoding seeds. The base model is therefore used instead. For OLMo-3-7B, 1,071 of 1,500 control outputs are unfinished, so the paper does not interpret the score difference as transfer. These cases show that the construction θ0+a\theta_0+a does not by itself guarantee a suitable control for comparing capabilities. The handling of these failures is described case by case, without general criteria for control validity. When the base model replaces the control, format changes can again contribute to the measured difference. This limits confidence in applying the proposed measurement procedure across models.

2. The effect of the response prefix on the Qwen3-14B comparison remains unclear

The external Qwen3-14B experiment demonstrates a gain from parameter editing under the reported evaluation setup. The control also improves substantially with a code prefix, from 921 to 1,030. Equivalent prefix results are not reported for the released checkpoint or the edited model. It therefore remains unclear whether the editing advantage persists when the same prefix is applied to all models. This concerns the robustness of the result to response conditions, rather than the existence of the reported gain.

On Qwen3-14B, a random mask of matched energy also clears the family threshold (1,067 vs. 982), so discussing what this implies for the specificity of the releasing edit would be helpful.

3. The conditions under which opening genre masks gains remain open

MiMo-7B expresses code gains despite mathematics openings, and Qwen3-14B releases gains without substantially changing those openings. These cases leave open when the opening genre masks a gain. The paper does not claim that the mechanism identified on Qwen3-4B extends to other models, and it states that what recurs is the concealment rather than the direction. Even so, a brief analysis using the existing generations, such as pass rates by opening genre or whether mathematics-style openings still lead to extractable code, could help explain these differences. A general account across models would be a natural direction for future work.

Questions for authors

1. Control validity — clarification

What criteria determine whether a constructed control is suitable for comparing capabilities? When it is replaced by the base model, how should the resulting score difference be interpreted given the remaining effects of output format?

2. Qwen3-14B — limited additional validation

Does the editing advantage persist when the same code prefix is applied to the released checkpoint, the control, and the edited model? A paired comparison on the same 1,055 LiveCodeBench problems and decoding seeds would address this sensitivity without additional training.

Reviewer_h8vsOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper studies the “concealed transfer” issue. It tends to claim such discovery as its original contribution (e.g., in the title and abstract), however, Most of these phenomena have been well studied in previous works. Its real contribution is limited to the discovery of a specific application of weight-space analysis to mathematics-to-code transfer. However such contribution is only found on a single Qwen3-4B model, which strongly questions the generalizability of the discovery.

Meanwhile, the presentation is a disaster, the authors try to mix up existing conclusions in previous works into this paper and reformalizes it as “concealed transfer”. The expression is unprofessional and both figures and tables are poorly organized to conveal the core idea of the paper. For example, Table 2 places a baseline method with the highest performance on the bottom which is extremely misleading to readers.

Critical strengths

The paper provides empirical analyses of weight-space interventions and includes comparisons across multiple model families.

Critical weaknesses
  1. Limited Novelty and Scientific Contribution.

The paper presents "concealed transfer" as a central discovery, particularly in its title and abstract. However, many of the underlying observations have already been extensively investigated in prior research, including task alignment, spurious forgetting, and the distinction between acquired capabilities and their expression. Reframing these established findings as "concealed transfer" does not constitute substantial scientific novelty. The actual contribution appears to be limited to a specific application of weight-space analysis to mathematics-to-code transfer, making the overall contribution highly incremental.

  1. Insufficient Generalizability of the Mechanistic Findings.

Even this incremental contribution is thoroughly demonstrated on only a single model, Qwen3-4B. Although the paper reports results across multiple model families, the proposed rank-one genre-direction mechanism is not consistently established across different architectures. The other experiments involve different interventions and do not provide equivalent mechanistic evidence. This substantially limits the generalizability of the central finding.

  1. Poor Presentation and Organization.

The presentation is extremely difficult to follow. The authors combine established observations with their own findings without clearly distinguishing prior knowledge from new contributions. The writing is frequently unclear, and both figures and tables are poorly organized, obscuring rather than explaining the core ideas.

For example, Table 2 mixes experimental configurations and results, placing the spectral truncation baseline, which achieves the highest reported LiveCodeBench score in that table, at the bottom. Such organization makes it unnecessarily difficult for readers to identify and compare the most relevant results. Overall, the presentation falls significantly below the standard expected of a scientific paper.

Questions for authors

What is the fundamental conceptual novelty of "concealed transfer" relative to existing studies on spurious forgetting and task alignment?

Can the authors demonstrate that the proposed weight-space mechanism generalizes beyond Qwen3-4B?

Reviewer_tV1COpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates cross-domain capability transfer between mathematical reasoning and code generation. Using a format-matched control and interventions on output behavior, including code-prefix prompting and gradient-guided weight editing, the authors find that mathematics SFT can introduce beneficial updates for code generation whose effects are obscured by shifts in response genre. These gains become measurable after modifying the model's output behavior or removing a small genre-related component of the fine-tuning update. In contrast, code SFT can yield directly observable improvements in mathematical reasoning without comparable genre-induced suppression.

Critical strengths
  1. A more insightful evaluation of cross-domain transfer. Rather than relying solely on performance differences between base and fine-tuned models, the paper introduces a format-matched control and targeted interventions to distinguish observed performance from potentially concealed transfer gains.
  2. An effective and precise weight-editing method. The proposed genre-gradient-guided rank-one projection identifies a small component of the fine-tuning update that influences response genre. Removing this component recovers substantial code performance while modifying only a negligible fraction of the update energy.
  3. Careful causal validation. The authors conduct several controlled interventions, including matched random-direction ablations, add-back experiments, and truncation controls, providing stronger evidence for the role of the identified genre-related component.
Critical weaknesses
  1. Lack of a clear definition of cross-domain capability gains. Although the paper aims to measure genuine cross-domain capability gains, it never provides a consistent operational definition of this central quantity. Different experiments use different comparisons, such as fine-tuned versus control models, or model performance after controlling the response genre, sometimes with different baselines. It remains unclear which metric actually represents the underlying capability gain. The authors should explicitly define this quantity and consistently use it throughout the paper.
  2. Limited generalizability of the proposed mechanism. The genre-gradient-based intervention is mainly validated on Qwen3-4B and does not directly generalize to other architectures. In particular, rnj-1 requires a different editing method, while Qwen3-14B recovers code performance without eliminating the genre shift. These results leave the underlying mechanism of concealed transfer across architectures unclear, indicating that the “rank-one direction along the genre gradient” is not a unified cross-architecture mechanism.
  3. The sufficiency test is the weakest and its results are incomplete: adding the direction back only partially restores concealed transfer; at λ=0.25, 1,455/3,165 sampled answers open in the mathematics genre but the score is only 805, below 908 for removing the direction; the authors acknowledge that the direction needs the remaining components of the update to take effect, which weakens the strong claim that “one direction carries the genre shift.”
  4. The reverse asymmetry phenomenon is not universal. Thorough validation of reverse code-to-mathematics transfer is mainly on Qwen3-4B; the reverse cells of other families are positive or negative, with SmolLM3-3B, Hunyuan-7B, and OLMo-3-7B showing negative transfer or a format floor, so the generality of bidirectional asymmetry remains limited.
  5. Poor presentation and organization. The paper is difficult to follow due to its limited visualizations and fragmented presentation of experimental results. The terminology (concealed transfer, genre shift, shared block, format-matched control) switches repeatedly in the main text, and readers must consult the appendix to assemble complete definitions; the presentation is clearly weaker than the experimental quality.
Questions for authors
  1. How do you define and quantify genuine cross-domain capability gains? Could you provide a unified table comparing these gains across models under comparable training and evaluation settings, including the recovery method used for each model (e.g., code-prefix prompting or genre-related weight editing)?
  2. Why is the shared-block control considered format-matched? What evidence supports retaining the updated special-token embeddings and unembeddings rather than using standardized output prefixes or decoding constraints to control response format?
  3. On Qwen3-14B, even coarse-grained random masking recovers substantial code performance and exceeds the significance threshold. How can you rule out the possibility that these gains arise from improved generation behavior or reduced truncation, rather than the recovery of concealed capabilities?
  4. In the sufficiency experiments, adding back the genre direction only partially restores the original performance suppression, while the (λ=0.25\lambda=0.25) arm still performs substantially worse than the direction-removal model. Does this suggest that the identified direction primarily controls response genre, while the remaining update (τrest\tau_{\mathrm{rest}}) carries other factors responsible for the recoverable gains?
  5. Could you provide more comprehensive robustness results across independent fine-tuning seeds, including confidence intervals for the key editing interventions? In particular, how stable are the findings for full-parameter Qwen3-4B and the externally obtained Qwen3-14B checkpoint?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.