acceptodds
Under review as a conference paper at ICLR 2027

PathBalance: Diagnosing Path-Load Imbalance in Dense Language Models

Abstract

Dense language models expose no explicit routing distribution, yet fine-tuning can concentrate their internal computation in a small subset of attention heads, MLP neurons, and residual-stream dimensions. We study this phenomenon as path-load imbalance, measuring activation strength (A), gradient pressure (G), and activation-gradient attribution (C). On Intermediate Algebra, after the best checkpoint, the top 2% of MLP neurons increase their share of attribution load from 15% to 72%, and the top 2% of residual dimensions from 15% to 36%. Higher residual-C exposure predicts poorer retention of previously correct answers, and suppressing train-hot MLP neurons reduces accuracy from 32.7% to 1.5%. Building on these findings, we introduce PathBalance, a load-aware dropout method for full fine-tuning that measures the current load before every update and masks the top-2% hot units while the remaining network is trained with cross-entropy. Across three math benchmarks and five Qwen3/Qwen3.5 models, PathBalance improves over SFT by up to 16 percentage points on GSM8K and 50 on MATH, without changing the inference architecture. Masking the same number of randomly chosen neurons yields no gain, and a matched-epoch analysis shows that PathBalance shifts attribution load from overloaded neurons to previously cold ones. The code is available at https://anonymous.4open.science/r/PathBalance/.

Primary area:
optimization
Keywords:
Large language models, path-load imbalance, mechanistic interpretability, activation-gradient attribution, load-aware dropout, supervised fine-tuning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x5HxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates "path-load imbalance" in dense Transformers during supervised fine-tuning, showing that attributional load (measured as C = E|a·g|) can concentrate on a small subset of MLP neurons, residual dimensions, and attention heads and that this concentration is causally load-bearing and predictive of forgetting on math tasks. Building on this diagnosis, the authors propose PATHBALANCE, a simple training intervention that, at every update, measures per-unit attribution load on the current batch, masks the currently hottest units (top 2% of MLP/residual coordinates; 12.5% of heads), and optimizes cross-entropy on the same data with the masked network. Across five Qwen/Qwen3.5 models and three math benchmarks, PATHBALANCE improves over matched SFT baselines by large margins (up to +50 pp on MATH subjects) without any inference-time change, with ablations indicating that load-based selection—not dropout per se—drives the gains and redistributes load from previously overloaded to cold units.

Critical strengths
  1. The paper is generally well written and clearly organized, with a coherent logical flow that makes it easy to follow.
  2. The diagnostic experiments are convincing. The authors not only observe that the load in the MLP, residual, and attention modules gradually becomes concentrated, but also conduct intervention experiments to further demonstrate that these hot units are indeed important for model performance.
Critical weaknesses
  1. All major experiments appear to use only a single training seed. Although the paper reports confidence intervals over test samples, these intervals do not capture the randomness of the training process. It would be helpful to report results from multiple runs for several representative settings.
  2. Each update requires an additional forward and backward pass to compute the load, which roughly doubles the training cost. This overhead may become substantial when scaling to larger models.
  3. The choice of a 12.5% head mask versus a 2% neuron/residual mask is motivated by the diagnostic results (e.g., top-3 out of 32 heads), but a clearer justification for these specific ratios, as well as a discussion of how they were tuned, would strengthen the paper.
Questions for authors
  1. Have you tried stronger SFT baselines, such as using a smaller learning rate or stronger regularization? Does the nearly 50-percentage-point improvement still hold under these stronger baselines?
  2. Section 3.6 suggests that stable-hot neurons are beneficial, while newly-hot neurons are more likely to be harmful. If so, why does PATH BALANCE still directly mask all currently hot neurons?
  3. Beyond mathematics, could you evaluate the method on non-mathematical domains, such as summarization or factual verification, where the reasoning structure is less pronounced? Do you observe similar load concentration patterns and performance gains in these settings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer x5Hx

Thank you for the careful review, and for finding the paper "generally well written and clearly organized" and the diagnostic experiments "convincing." We address each point with new results. Tables A–D are in the General Response, which also introduces one-pass PathBalance: no extra pass, about SFT's step time, and the same accuracy as two-pass (details under W2 below).

W1: All major experiments use a single training seed. It would be helpful to report results from multiple runs for several representative settings.

We trained all seven settings with two additional seeds on Qwen3-1.7B Intermediate Algebra, and also at two smaller learning rates (Table C). Over the three seeds:

  • SFT scores 16.69 ± 0.72; the PathBalance settings range from 54.19 ± 0.89 (M) to 56.48 ± 0.11 (M+R).
  • The gap over SFT (37.5–39.8 points) exceeds the largest seed standard deviation (1.41) by more than 25×, and the worst PathBalance seed (53.27%) beats the best SFT seed (17.39%) by 35.9 points.
  • Across all 30 PathBalance runs (seeds and learning rates), accuracy stays within 53.27–57.81%.

W2: Each update requires an additional forward and backward pass, which roughly doubles the training cost. This may become substantial for larger models.

Two-pass PathBalance costs about 2× per step, as the paper states (Section 4.1 and the cost appendix), but the extra pass is not required (first point below). Peak memory rises by only 0–4.5 GiB (0–14%) in our runs, because the measurement pass keeps no optimizer state and the two graphs are not held together. Inference is unchanged. Three points put this cost in perspective:

  • The extra pass can be removed (new). In one-pass PathBalance, each update uses the mask chosen from the load measured during the previous update's own masked pass (activations before the mask, gradients after it, so masked units still receive a fresh load; the first update is unmasked). On Qwen3-1.7B Intermediate Algebra (test %, mean ± std over three seeds), one-pass matches two-pass; SFT scores 16.69 ± 0.72. Its median step time is 0.62–0.94 s, against 0.70 s for SFT and 1.04–1.12 s for two-pass on the same shared GPUs, with similar peak memory.
MRHM+RM+HM+R+HMean
Two-pass54.19 ± 0.8955.96 ± 1.4156.11 ± 0.6556.48 ± 0.1156.33 ± 1.0056.04 ± 1.3655.85
One-pass53.86 ± 0.7555.63 ± 0.7255.96 ± 1.0057.11 ± 0.6757.33 ± 1.0555.48 ± 1.0055.90
  • More compute does not help SFT. SFT's best validation accuracy over its whole run is 25.4% on Intermediate Algebra (720 steps), 37.9% on Number Theory (900 steps) and 22.8% on APPS (3200 steps), far below every PathBalance setting (63.1–69.2%, 62.1–77.0% and 28.4–31.4%). These validation results indicate that simply extending SFT training does not necessarily recover the lost performance; however, a fully compute-matched comparison would require additional controlled experiments.
  • Prior attribution-guided masking pays this price; one-pass PathBalance does not. HeadMask's important-head variant runs a full forward and backward pass without update before every masked step, and AD-DROP needs an extra forward pass and an attribution backward pass on its AD-DROP epochs.

W3: A clearer justification for the 12.5% head mask versus the 2% neuron/residual mask, and how they were tuned.

The ratios were not tuned: one setting is used for all 5 models and the 3 paper datasets, and unchanged for the 4 new tasks and the Base model.

  • 2% for neurons and residual dimensions comes from the diagnosis: the top 2% already carry 15% of MLP and residual load at the best checkpoint (over 7× their uniform share), and this grows to 72% and 36% (Table 1a). The same top-2% share (13.5–14.5%) appears on every math task and on code (Table E in our reply to ozPY).
  • 12.5% for heads reflects their coarser granularity. A head is a whole output vector, and our models have only 8–32 heads per layer, so 2% would round to a single head and give a different fraction for each model. 12.5% is the smallest fraction that masks a whole number of heads in every architecture we use (1 of 8, 2 of 16, 4 of 32), and it covers the diagnostic hot set (the top 3 of 32 heads carry 23–29% of head load).
  • The ratio is not what drives the gain. Random masking of 10% of neurons (5× more units) gives only +1.66 points, while hot 2% gives +50.17. We will add this justification to Section 4.1.
AuthorsOpenReview ↗

Response to Reviewer x5Hx (continued)

Q1: Have you tried stronger SFT baselines, such as a smaller learning rate or stronger regularization? Does the nearly 50-point improvement still hold?

Yes. We ran SFT with learning rates 1e-5 and 5e-6 on both Qwen3-1.7B and Qwen3-4B, and with 10× weight decay on Qwen3-1.7B, each with all six PathBalance settings at the same learning rates (Tables C, D).

  • Stronger regularization does nothing: SFT with weight decay 0.1 scores 15.95%, the same as the paper's SFT.
  • On Qwen3-1.7B, SFT is substantially more sensitive to learning-rate changes than PathBalance. SFT accuracy varies by 22.92 points across seeds and learning rates, compared with at most 4.54 points for any PathBalance setting. Every SFT run (15.95–38.87%) remains below every PathBalance run (53.27–57.81%) (W1).
  • On Qwen3-4B, smaller learning rates improve SFT but do not recover initial-model performance. SFT achieves 63.01% at LR 1e-5 and 66.45% at LR 5e-6, remaining below the initial model (77.63%). In contrast, PathBalance consistently achieves 78.63–81.84% across the tested learning rates, preserving or improving upon initial-model performance (W1).

Thus, while the approximately 50-point improvement reflects the shared training recipe, the advantage of PathBalance persists after learning-rate tuning. Across three learning rates on Qwen3-4B, SFT accuracy varies by 36.00 points, compared with only 0.33–2.11 points for each PathBalance setting. Table A in the General Response shows the symptom: SFT's outputs shrink toward the length of the short gold solutions. We will report these tuned baselines in the paper.

Q2: Section 3.6 suggests that stable-hot neurons are beneficial while newly hot neurons are harmful. Why does PathBalance still mask all currently hot neurons?

The mask lasts one update and is absent at inference, so stable-hot neurons stay in the network and keep their function at test time; the mask only keeps that update from being routed through them:

  • A masked unit receives no gradient from that batch. For a masked MLP neuron, its rows of W_gate and W_up and its column of W_down get zero gradient from the batch; for a masked head, its query projection and its slice of W_O do. That batch's update is therefore carried by the rest of the network instead of loading the stable-hot neurons further, in line with Section 3.6, which finds stable-hot neurons useful.
  • Newly hot neurons are masked as soon as they become hot, which counters the concentration that Section 3.6 identifies as harmful; Table 3 shows load moving to previously cold neurons.
  • Telling the two apart requires history. Classifying a neuron as stable-hot or new-hot needs a reference checkpoint, a history window and a threshold. Masking the current top 2% is stateless and needs none of these.

We will add this explanation to Section 4.1.

Q3: Beyond mathematics, could you evaluate on non-mathematical domains, such as summarization or factual verification? Do you observe similar load concentration patterns and performance gains?

We added code generation (APPS; Table E in our reply to ozPY) and dialogue summarization (SAMSum), each with all seven settings.

  • Load concentration extends beyond mathematics. On the initial model, the top 2% of MLP neurons carry 14.0% of the load on APPS and 16.5% on SAMSum, compared with 13.5–14.5% across the three math tasks (uniform: 2%). This concentration increases to 50.7–62.4% in the final layer across these tasks.
  • Code: the same pattern as mathematics. SFT reduces APPS accuracy from 14.22% to 7.50%, whereas all six PathBalance settings achieve 14.02–15.24%, outperforming SFT by 6.52–7.74 points under an execution-based scorer.
  • Summarization: PathBalance improves upon SFT without initial-model degradation. On SAMSum (5,000 training dialogues, 819 test dialogues, ROUGE-L, one seed), SFT improves the initial model from 29.56 to 37.60. Importantly, PathBalance further improves performance: H achieves 41.80 (+4.20 over SFT), M achieves 39.85 (+2.25), and M+H achieves 38.70 (+1.10). Configurations involving residual-dimension masking remain near initial-model performance (R 29.61, M+R 29.71, M+R+H 29.33).

These results clarify PathBalance's scope. On math and code tasks, where SFT degrades initial-model performance, PathBalance preserves reasoning performance and maintains accuracy close to or above the initial model. On summarization, where SFT improves performance, masking hot MLP neurons or attention heads yields further gains, demonstrating effectiveness beyond mathematical reasoning. These findings highlight the value of targeted load redistribution across tasks. Residual-dimension masking exhibits task-dependent behavior, which we will discuss alongside masking-site selection in the revised paper.

We hope these experiments address your concerns about the baseline, robustness, scope and mechanism; if so, we would be grateful if you would consider raising your score.

Reviewer_x5HxOpenReview ↗

Response to the Authors

I appreciate the authors' thorough response. The provided explanations have resolved my initial concerns.

AuthorsOpenReview ↗

Response to the Reviewer

Dear Reviewer,

Thank you for your positive feedback. We are pleased that our response has addressed your concerns and sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_UQWzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores whether language models increasingly rely on a small set of internal units during supervised fine-tuning and introduces PathBalance. Before each update, the method selects important units using activation-gradient attribution and temporarily masks them. It then updates the model with cross-entropy loss. At inference, the full network is used. The experiments involve five Qwen3/Qwen3.5 models, along with GSM8K and two MATH subjects (§4).

Critical strengths

The paper combines diagnostic analysis with a straightforward training intervention. The authors track load concentration during fine-tuning and test the importance of high-attribution units through interventions. This provides useful evidence about how the model relies on its internal units (§3).

The training results also support the value of the method. In Table 3, PathBalance achieves 80.62% test accuracy with Qwen3-4B on Intermediate Algebra, compared with 30.45% for SFT and 28.79% for random masking at the same ratio. This indicates that standard random masking did not reproduce the gain in this setting. The paper also reports paired confidence intervals, generation failures, and training costs (Tables 10 to 12). Together, these diagnostic analyses and controlled experiments provide a valuable empirical contribution.

Critical weaknesses

W1: The initial model's performance is missing as a reference. The 4B experiments start from Qwen3-4B-Instruct-2507, but its performance under the same evaluation protocol is not reported (Appendix F.1). The current results support PathBalance's advantage over SFT with the same training recipe. However, it remains unclear whether the gain mainly comes from improvement beyond the initial model or from reducing capability loss that may occur during fine-tuning. Reporting this reference would help clarify the source of the gain. Preserving existing capabilities is also valuable in itself.

W2: The relationship with similar masking methods could be explained more clearly. The important-head masking variant of HeadMask (COLING 2022) first computes importance, masks important attention heads, and then updates the model. AD-DROP (NeurIPS 2022) randomly masks high-attribution positions in the attention matrix during language model fine-tuning. Section 2 does not discuss these two methods. Although their specific settings differ, a brief comparison of attribution computation, masking targets, and training procedures would help clarify PathBalance's specific contribution relative to prior work.

Questions for authors

[Clarification] Could you report the initial 4B model's performance under the same prompts, generation budget, and scoring rules? This would help contextualize the gain relative to SFT.

[Clarification] How does PathBalance differ from HeadMask's important-head masking variant and from AD-DROP, particularly in the attribution metric and training procedure?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UQWz

Thank you for the positive assessment and for recognizing that the diagnostic analyses and controlled experiments "provide a valuable empirical contribution." Both of your requests are now addressed. Tables A–D are in the General Response, which also introduces one-pass PathBalance: no extra pass, about SFT's step time, and the same accuracy as two-pass.

W1 / Q1: The initial model's performance is missing as a reference. Could you report the initial 4B model's performance under the same prompts, generation budget, and scoring rules?

We evaluated the initial models using the same task-specific test-evaluation pipeline as the fine-tuning runs, including identical prompts, output-length limits (4096 tokens), vLLM settings, and task-specific scorers. The initial Qwen3-4B-Instruct-2507 achieves 77.63% (701/903) on Intermediate Algebra, while PathBalance (M) improves performance to 80.62%, substantially outperforming SFT (30.45%). Table B further reports initial-model performance for Qwen3-1.7B on Intermediate Algebra (55.15%), Number Theory (69.63%), MATH-500 (70.00%), and APPS (14.22%), as well as Qwen3-1.7B-Base on Intermediate Algebra (38.32%) and Number Theory (53.33%).

The gain over SFT comes from preventing the capability loss that SFT causes, which, as you note, is valuable in itself. Three new results show that this preservation is substantial and specific to PathBalance:

  • SFT's degradation is substantial across tasks, with accuracy losses of approximately 20–47 points on math and a reduction from 14.22% to 7.50% on code. Even the Base model without post-training loses 20.05–31.85 points under SFT. In contrast, the best PathBalance setting preserves or exceeds initial-model performance on every evaluated task, achieving improvements of up to 3.33 points (Table B).
  • Lower learning rates mitigate but do not eliminate SFT degradation: on Qwen3-4B, SFT improves to 66.45% at LR 5e-6 but remains below the initial model (77.63%). In contrast, all 18 PathBalance runs across three learning rates achieve 78.63–81.84%, consistently exceeding initial-model accuracy (Table D).
  • The preservation requires the load-based mask: an empty mask (16.06%), stronger weight decay (15.95%) and random masking of the same or 5× more units (W2 below) give none of it.

Table A in the General Response shows what changes: SFT's outputs shrink toward the length of the short human-written solutions, while PathBalance's keep the initial model's length.

We will add the initial-model row to Table 2 and state in Sections 4.4 and 4.6 that the gains over SFT reflect prevented degradation.

AuthorsOpenReview ↗

Response to Reviewer UQWz (continued)

W2 / Q2: How does PathBalance differ from HeadMask's important-head masking variant and from AD-DROP, particularly in the attribution metric and training procedure?

Thank you for pointing us to these works. We checked both papers in detail and will add them to Section 2. Table H summarizes the comparison.

Table H. Comparison with HeadMask (important-head variant) and AD-DROP

HeadMask, "mask important" (COLING 2022)AD-DROP (NeurIPS 2022)PathBalance
Masked unitsattention heads onlyattention-map entries (query–key pairs)MLP neurons, residual dimensions and attention heads
Attributionhead importance E_x ∣Att_hᵀ ∂L/∂Att_h∣ (per sample)gradient of the pseudo-label logit w.r.t. attention entries (no activation term)load C = E_t ∣a·g∣ on the gold loss, per unit and token, per layer
Selectiondeterministic, top heads globally across layersstochastic: top-p positions per row are candidates, each dropped with rate q; p, q tuned per datasetdeterministic, top 2% per layer (12.5% of heads); one fixed ratio for all models and datasets
Scheduleevery batchAD-DROP on alternate epochs (cross-tuning)every update, re-measured on the current batch
Extra pass per updateyes: forward + backward without updateyes, on AD-DROP epochs: forward + attribution backwardnone in one-pass PathBalance (new; same accuracy as two-pass; General Response)
SettingNMT Transformer trained from scratch; +0.1–0.8 BLEUBERT-style fine-tuning on GLUE and similar; +0.6–2 pointsfull fine-tuning of LLMs (0.6B–4B) on math and code with long generated solutions
Motivationunequal training of headsover-reliance on high-attribution featuresmeasured load concentration that is causally load-bearing and predicts forgetting (Section 3)
  • Opposite outcome on the key question. HeadMask found that masking important heads was weaker than masking random heads. We find the opposite: masking hot units is far better than masking the same number, or 5× more, random units (80.62% vs. 28.79% and 32.12% on Qwen3-4B). Selecting units by their load on the gold loss is what makes masking work here.
  • The main gains come from units neither method masks. M alone gives +50.17 points on Qwen3-4B, and R alone gives the best Qwen3-1.7B results on Intermediate Algebra (57.59%) and full MATH (72.20%). HeadMask masks only heads and AD-DROP only attention entries.
  • Closest prior work. Our H setting is closest to HeadMask and uses a similar gradient×activation head score; we will say so directly. It differs in per-layer rather than global selection, a per-token rather than per-sample absolute value, the LLM fine-tuning setting and, in its one-pass form, the absence of an extra pass.

Thank you again for the supportive review. We hope the new reference results and the comparison make the contribution clearer; if they address your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our submission and providing constructive feedback. We have carefully prepared our rebuttal to address your concerns, supported by extensive additional experiments, analyses, and clarifications that further strengthen our findings.

We hope these additional results adequately address your concerns. If so, we would be grateful if you would consider raising your score. We would also be happy to address any further questions or concerns you may have.

Thank you again for your valuable feedback and consideration.

Best regards,
The Authors

Reviewer_ozPYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates path-load imbalance in dense language models, where fine-tuning concentrates activation–gradient attribution in a small subset of computational units. Through diagnostic experiments, the authors associate this concentration with reduced answer retention and sensitivity to targeted unit suppression. They propose PATHBALANCE, a load-aware dropout method that masks highly attributed units during fine-tuning to encourage training of the remaining network. Experiments on three mathematical reasoning benchmarks across five Qwen3/Qwen3.5 models report substantial improvements over standard SFT and random masking, without modifying the inference architecture.

Critical strengths
  1. The paper investigates whether concentrated reliance on a small subset of computational units is associated with performance degradation during fine-tuning, an interesting question for understanding and improving language model adaptation.

  2. The paper combines activation–gradient importance measurements with targeted masking, providing a practical way to study and intervene on the model’s internal computation.

  3. Experiments consider several model sizes and masking locations, including MLP units, attention heads, and residual representations, offering useful comparisons across intervention choices.

Critical weaknesses
  1. The unusually large gains require stronger validation of the SFT baseline. For Qwen3-4B, accuracy on Intermediate Algebra increases from 30.45% to 80.62%, with a substantial gap already appearing at the first validation checkpoint. Without the original checkpoint’s accuracy and a more thoroughly tuned SFT baseline, it is unclear whether PathBalance improves reasoning ability or primarily prevents degradation caused by the chosen fine-tuning configuration. Results from a single training seed further limit confidence in their robustness.

  2. Limited benchmark coverage and task diversity. Evaluation is restricted to GSM8K and two MATH subsets, all within mathematical reasoning. The two MATH subsets provide limited independent evidence of generalization. Broader mathematical benchmarks and non-mathematical tasks are needed to support the method’s wider applicability to language model fine-tuning.

  3. The proposed mechanism is not sufficiently isolated. The results do not clearly establish that improvements specifically arise from correcting path-load imbalance rather than generic regularization or changes in reasoning length. In some settings, PathBalance generates approximately twice as many tokens as SFT. Comparisons under matched inference budgets, together with a zero-masking control that retains the full two-pass training procedure, would help separate the proposed mechanism from computational and implementation-related confounds.

Questions for authors

1.Can the authors clarify the source of the unusually large gains over SFT? Does PathBalance improve upon the original model, or primarily prevent degradation during fine-tuning?

2.Does PathBalance generalize beyond mathematical reasoning?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ozPY

Thank you for the detailed review, for finding the question of concentrated reliance on a few units interesting, and for describing our combination of attribution measurement and masking as "a practical way to study and intervene on the model's internal computation." Your three weaknesses and two questions are exactly the right tests, and we answer each with new experiments. Tables A–D are in the General Response, which also introduces one-pass PathBalance: no extra pass, about SFT's step time, and the same accuracy as two-pass.

W1 / Q1: The large gains require stronger validation of the SFT baseline. Without the original checkpoint's accuracy and a more thoroughly tuned SFT baseline, it is unclear whether PathBalance improves reasoning ability or primarily prevents degradation. Results from a single seed further limit confidence.

Original checkpoint. With the paper's prompts, output-length limit and scorer, the initial Qwen3-4B scores 77.63% on Intermediate Algebra and the initial Qwen3-1.7B 55.15% (Table B). PathBalance's gain over SFT comes from preventing the degradation that SFT causes under every configuration we tried. PathBalance's best setting preserves or exceeds the initial model's accuracy across all evaluated math and code tasks, with improvements of up to 3.33 points. In contrast, SFT causes substantial degradation, with losses of approximately 20–47 points on math tasks and nearly half the initial accuracy on code. We regard this as the main result rather than a caveat, for three reasons:

  • It is consistent with the diagnosis. Section 3.5 shows that held-out accuracy falls as load concentrates (r = −0.879) and that high load exposure predicts forgetting of previously correct answers. PathBalance is designed to counter this concentration, and the accuracy loss does not occur.
  • The damage is not an artifact of a saturated starting model. On Qwen3-1.7B-Base (no post-training), SFT still more than halves accuracy (Intermediate Algebra 38.32 → 18.27; Number Theory 53.33 → 21.48), and all six PathBalance settings keep it (38.98–41.09; 51.48–56.48).
  • Preventing it is not trivial. Stronger weight decay, random masking, and an empty mask do not prevent it (Table F under W3). Smaller learning rates improve SFT performance but still fail to recover initial-model accuracy.

As Reviewer UQWz also notes, "preserving existing capabilities is also valuable in itself."

Thoroughly tuned SFT. We reran SFT with learning rates 1e-5 and 5e-6 and with 10× weight decay on Qwen3-1.7B, and with both smaller learning rates on Qwen3-4B, together with all six PathBalance settings at the same learning rates (Tables C, D):

  • On Qwen3-1.7B, SFT's score spans 22.92 points across seeds and learning rates, compared with at most 4.54 points for any PathBalance setting; every PathBalance run (53.27–57.81%) outperforms every SFT run (15.95–38.87%).
  • Smaller learning rates mitigate SFT degradation but do not eliminate it. On Qwen3-4B, SFT achieves 63.01% at LR 1e-5 and 66.45% at LR 5e-6, remaining below the initial model (77.63%). A similar pattern appears on Qwen3-1.7B, where reducing the learning rate improves SFT accuracy to 38.87%, but still leaves a substantial gap relative to the initial model (55.15%).
  • All 12 PathBalance runs at the smaller learning rates on Qwen3-4B achieve 78.63–81.84% test accuracy, demonstrating robustness across the evaluated learning rates. The best runs (M at 5e-6: 81.84%; M+R+H at 1e-5: 81.73%) exceed the initial model (77.63%) by 4.21 and 4.10 percentage points, respectively.

PathBalance therefore demonstrates substantially lower sensitivity to learning-rate changes: on Qwen3-4B, its accuracy varies by only 0.33–2.11 points per setting across three learning rates, compared with 36.00 points for SFT. This suggests greater robustness to learning-rate selection.

Single seed. We trained all seven settings with two additional seeds on Qwen3-1.7B Intermediate Algebra (Table C). Seed standard deviations are 0.11–1.41 for PathBalance and 0.72 for SFT. The gap over SFT (37.5–39.8 points) is more than 25 times the largest standard deviation, and the worst PathBalance seed (53.27%) beats the best SFT seed (17.39%) by 35.9 points.

W2 / Q2: Limited benchmark coverage and task diversity. Broader mathematical benchmarks and non-mathematical tasks are needed. Does PathBalance generalize beyond mathematical reasoning?

We added two broader math settings and one code task, each with all seven settings (Table E). Effective batch size, learning rate, weight decay, epoch limit and early stopping are the paper's. Two settings follow the dataset: the validation interval (60 steps on Number Theory, as in the paper; 300 on full MATH, which has 844 steps per epoch; 200 on APPS) and the output-length limit (4096 tokens).

AuthorsOpenReview ↗

Response to Reviewer ozPY (continued)

  • MATH Number Theory (a third, independent MATH subject): all six settings outperform SFT by 40–46 points; M+H (72.96%) and R (72.22%) exceed the initial model (69.63%) by 3.33 and 2.59 points, respectively.
  • Full MATH (all seven subjects, 6750 training problems), evaluated on MATH-500: all six settings outperform SFT by 27–35 points; R and M+R+H (72.20%) exceed the initial model (70.00%) by 2.20 points.
  • APPS code generation (5000 test problems, hidden-unit-test scorer): SFT drops from 14.22% to 7.50%, while all six PathBalance settings achieve 14.02–15.24%, preserving initial-model performance.

Across these tasks, SFT substantially degrades initial-model performance, while PathBalance consistently preserves or improves it. Similar load concentration appears in math and code (Table E), supporting the proposed mechanism. On SAMSum, where SFT improves performance, M and H outperform SFT, while settings with R remain near the initial level (x5Hx, Q3).

Table E. New tasks (Qwen3-1.7B, test %; bold = above SFT) and load concentration (share of MLP load C carried by the top 2% of neurons per layer, measured on the initial model with each task's data; uniform = 2%)

TaskInitialSFTMRHM+RM+HM+R+HTop-2% shareLast layer
Number Theory69.6327.2267.5972.2269.4469.4472.9672.2214.155.5
Full MATH → MATH-50070.0036.8066.8072.2064.0069.8071.2072.2013.551.7
APPS (code)14.227.5014.1815.2014.0215.2415.0014.6014.050.7
Intermediate Algebra (paper)55.1515.9555.0457.5956.5956.5957.4857.0314.552.4

W3: The mechanism is not sufficiently isolated (generic regularization, reasoning length). In some settings PathBalance generates about twice as many tokens. Comparisons under matched inference budgets, together with a zero-masking control that retains the full two-pass procedure, would help.

We ran both requested controls.

Zero-mask control. The full two-pass PathBalance pipeline (unmasked measurement pass followed by training on the same batch) with an empty mask scores 16.06%, near SFT (15.95%) and 39 points below M (55.04%). The two-pass procedure alone provides no benefit, while one-pass PathBalance achieves comparable accuracy without the extra pass (General Response). Together with the controls in Table F, these results support load-based selection, rather than generic regularization or masking intensity, as the key driver of the gains.

Table F. Controls (Intermediate Algebra, test %). Only masking the hot units helps.

ModelRunTest %Conclusion
Qwen3-1.7BSFT15.95
Zero-mask: two-pass procedure, empty mask (new)16.06procedure alone: no gain
SFT, weight decay 0.1 instead of 0.01 (new)15.95generic regularization: no gain
M: hot 2% masked55.04
Qwen3-4BSFT30.45
Random 2% of neurons masked (paper)28.79same amount, random units: no gain
Random 10% of neurons masked (paper)32.125× more, random units: no gain
M: hot 2% masked80.62

Matched inference budgets. With greedy decoding, we re-score saved outputs at length B: completed outputs remain unchanged, while longer ones are truncated to B tokens and re-scored using the same evaluator. At B = 4096, this reproduces the original results.

Table G. Re-scoring at matched output length B (Qwen3-1.7B, Intermediate Algebra, test %; "Truncated" = outputs reaching the 4096-token limit, of 903; initial model: median 1047 tokens, 73 truncated)

RunMedian tokensTruncatedB=1024B=2048B=3072B=4096
SFT, seed 133428323316.7216.7216.7216.72
M, seed 13349559339.9850.6152.7153.27
SFT, seed 133527222617.3917.3917.3917.39
M, seed 13359679341.9752.4954.1554.26
  • At every output length B, M leads SFT by 23–37 points. SFT already reaches its full score at B = 1024, and M still leads there by 23–25 points.
  • SFT gains nothing beyond 1024 tokens: its long outputs are mostly repetition loops that never reach an answer (226–233 truncated vs. 93 for M).

PathBalance does not lengthen outputs; SFT shortens them. On every math and code task, PathBalance's median output length is within 20% of the initial model's (at most 3% above it), while SFT's is 3–5× shorter (Table A). The length gap is SFT dropping the initial model's derivations, not PathBalance adding inference compute.

We hope these results resolve your concerns about the baseline, robustness, scope, and mechanism. If so, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our submission and providing constructive feedback. We have carefully prepared our rebuttal to address your concerns, supported by extensive additional experiments, analyses, and clarifications that further strengthen our findings.

We hope these additional results adequately address your concerns. If so, we would be grateful if you would consider raising your score. We would also be happy to address any further questions or concerns you may have.

Thank you again for your valuable feedback and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.