- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper adapts English-dominant reasoning LLMs (Qwen3-8B and 4B) to Norwegian using three rank-8 LoRA branches for knowledge, reasoning, and linguistic adaptation. A geometric teacher bypasses layers on 300 probes to build sensitivity profiles, transfers them to training samples via ridge regression, and guides a residual MLP router. With adapter capacity matched to a rank-24 LoRA (the router adds 1.05M parameters) and the same training budget, SAGR improves average scores over LoRA and MoE-LoRA baselines.
Critical strengths
-
Fair capacity matching and clean parameterization. Matching the total adapter rank, data, and training steps across baselines is good practice (L776-781), and splitting each update into branch weights, layer weights, and scalar strength (Eq. 4, L218) is intuitive.
-
Consistent gains on the accuracy tasks. SAGR improves Acc5 over rank-24 LoRA in all three seeds, by 4.07 points on average (Table 10), and the task-level results report the NoCoLA trade-off instead of hiding it in the aggregate.
Critical weaknesses
-
The routing relies heavily on task-level priors that baselines do not get, and capability is confounded with language. Training blends ridge predictions with partition means at gamma = 0.7 using source indicators (L188, L697), inference maps each evaluation benchmark to a training partition by hand (Table 6, L747), and the router's residual corrections are scaled by only tau_w = 0.2 (L766), so tasks mapped to the same partition get nearly identical weights (Table 5). In the training set, 100% of reasoning samples are English while 100% of knowledge and linguistic samples are Norwegian (Table 8, L787).
-
The complex geometric teacher adds very little, and the linguistic branch does not improve language performance. A Neutral teacher without geometric probes or teacher alignment already gets 48.59 Macro6 (Table 2, L341), almost the same as Input-only SAGR without manual partition mapping (48.80, Table 4, L372). On Qwen3-8B, NoCoLA routes 83% of its weight to the linguistic branch yet scores 16.32, no better than the frozen base model (16.37) and far below rank-24 LoRA (27.10, Table 1), and the continuation experiment mentioned at L260 and L418 is missing from the paper.
Questions for authors
- Why does NoCoLA on Qwen3-8B score 16.32 (about the same as the frozen model's 16.37, Table 1) despite routing 83.3% of its weight to the linguistic branch, while standard rank-24 LoRA reaches 27.10?
- Where is the selective continuation experiment mentioned at L260 and L418?
3 comments
W2. Contribution of the Geometric Teacher
We appreciate the reviewer's concern regarding the incremental contribution of the geometric teacher. We would like to clarify that the geometric teacher is intended to provide an initial capability-aware routing reference using only 300 probes, rather than to determine the final adaptation behavior. The main contribution of SAGR lies in the joint optimization of three capability-specific LoRA branches and a residual router, which continuously refines the teacher-guided branch weights, layer allocation, and adaptation strength throughout training.
The neutral teacher achieves 48.59 Macro6, indicating that the multi-branch architecture and learned routing mechanism already contribute substantially to SAGR's performance. Introducing geometric guidance further improves Macro6 to 49.30. Although this incremental improvement is relatively modest, it demonstrates the additional benefit of capability-specific geometric priors.
Importantly, the input-only configuration (48.80 Macro6) uses the same teacher-trained SAGR model but replaces the manually mapped inference reference with one derived from input representations and neighboring training samples. Therefore, this result evaluates the dependence on inference-time task mapping rather than the contribution of the geometric teacher itself.
These findings suggest that SAGR's effectiveness arises primarily from the interaction between capability-specific adaptation branches and dynamically learned routing, with the geometric teacher providing additional structural guidance.
W2 & Q1. Linguistic Branch Performance on NoCoLA
We agree that the high linguistic routing weight (83.3%) does not translate into a corresponding improvement on NoCoLA, where SAGR achieves 16.32 ERRANT F0.5, compared with 16.37 for the frozen base and 27.10 for conventional LoRA-r24.
An important distinction is that the linguistic supervision in the original training mixture primarily focuses on Norwegian idiomatic expressions, whereas NoCoLA evaluates grammatical error correction. Although both tasks involve linguistic competence, they require different forms of language adaptation. NoCoLA is not included in the original training mixture and serves as an unseen evaluation task to assess generalization beyond the training tasks. Under the same training data and optimization budget, SAGR achieves 8.00 accuracy on Norwegian idiom completion, compared with 2.50 for LoRA-r24, but performs worse on NoCoLA. These results suggest that different adaptation methods may exhibit different performance trade-offs across heterogeneous tasks, even under identical training conditions.
Furthermore, the linguistic routing weight reflects how adaptation signals are allocated across capability-specific branches rather than directly measuring their effectiveness on a particular task. The 83.3% linguistic routing weight indicates that SAGR primarily associates NoCoLA with the linguistic branch rather than the knowledge or reasoning branches. However, generalization to this unseen task may be influenced by differences in task requirements and the trade-offs involved in adapting multiple capabilities simultaneously.
This trade-off is further reflected in the aggregate results: SAGR achieves higher Acc5 (55.90 vs. 52.00) and Macro6 (49.30 vs. 47.85) than conventional LoRA-r24, despite its lower NoCoLA performance. We will clarify these findings in the revised manuscript.
W2 & Q2. Selective Continuation Experiments
We thank the reviewer for pointing out the missing selective continuation experiments referenced at L260 and L418. We apologize for this omission. These experiments were initially conducted as a preliminary investigation limited to the Law domain. Since we had not yet evaluated their generalizability across additional domains, we decided not to include these preliminary results in the original submission. Nevertheless, we believe these findings provide useful initial evidence regarding selective continuation. We therefore present the results below to examine whether selectively updating capability-specific branches can facilitate new-domain learning while retaining existing capabilities.
Law continuation. Starting from the calibrated Qwen3-8B SAGR model, we perform continued training on 1,381 Law examples for one epoch (87 optimizer steps). We evaluate Law token-F1 on 200 examples and assess the retention of existing capabilities using 200 examples from each of the original six benchmarks, consistent with the evaluation protocol in the original submission. The average routing weights assigned to the Law training samples by the calibrated SAGR checkpoint are shown below:
| Knowledge (K) | Reasoning (R) | Linguistic (C) |
|---|---|---|
| 18.48% | 44.49% | 37.03% |
The R/C branches account for 81.52% of the total routing weight, motivating their selection for continued adaptation. Accordingly, we update the R/C branches and router while freezing K. We compare this configuration with frozen-router and all-branch variants. We additionally introduce Branch Routing Regularization (BRR), which constrains the relative routing distribution between the dominant R branch (44.49%) and the frozen K branch against the saved teacher reference, while leaving C unconstrained for flexible adaptation. Conventional LoRA-r24 is continued on the same Law data under the same optimization budget.
Additional results on Law selective continuation
| Continuation configuration | Law token-F1 | Macro6 | ΔMacro6 |
|---|---|---|---|
| SAGR before continuation | 17.67 | 49.30 | 0.00 |
| R/C + router (K frozen) | 18.58 | 47.79 | −1.51 |
| R/C (K and router frozen) | 19.25 | 47.20 | −2.11 |
| R/C + router (K frozen) + BRR | 19.94 | 47.95 | −1.36 |
| K/R/C + router | 18.83 | 47.46 | −1.85 |
| LoRA-r24 before continuation | 17.35 | 47.85 | 0.00 |
| LoRA-r24 after Law continuation | 30.62 | 43.27 | −4.58 |
Law token-F1 measures domain-specific adaptation, while Macro6 measures performance on the original six evaluation benchmarks. ΔMacro6 represents the change relative to each method's own checkpoint before continuation.
Selective adaptation and capability preservation. The results show that different branch-update strategies lead to different adaptation–retention trade-offs. Updating the R/C branches and router achieves 47.79 Macro6 and 18.58 Law token-F1, compared with 47.20 and 19.25 when the router is frozen, and 47.46 and 18.83 when all branches are updated, respectively. Introducing BRR further improves Macro6 to 47.95 and Law token-F1 to 19.94, achieving the best performance on both metrics among the SAGR continuation variants. These findings suggest that regularizing the relative K/R routing distribution can help retain performance on existing tasks while supporting adaptation to the new domain.
Trade-off between new-domain learning and existing capabilities. With BRR, SAGR retains 47.95 Macro6, compared with 49.30 before continuation, corresponding to a decrease of approximately 1.36 points. In contrast, conventional LoRA-r24 decreases from 47.85 to 43.27, a reduction of 4.58 points. Notably, LoRA-r24 falls below the frozen base (43.27 vs. 46.15 Macro6, as reported in Table 1) after Law continuation, whereas SAGR with BRR remains above the frozen base (47.95 vs. 46.15). However, LoRA-r24 achieves higher Law token-F1 (30.62) than SAGR (19.94), indicating stronger new-domain adaptation accompanied by greater degradation on the original benchmarks.
Overall, these results suggest that selective branch updates and routing regularization can help balance new-domain adaptation with the retention of existing capabilities. Because the two methods start from different adapted checkpoints, these results illustrate their observed adaptation–retention trade-offs. The within-SAGR comparisons provide more direct evidence of the effects of branch selection and routing regularization.
We will incorporate these experimental protocols, results, and limitations into the revised manuscript to provide a more complete evaluation of SAGR's capability-specific routing mechanism.
We sincerely thank the reviewer for the constructive feedback, and appreciate the concerns regarding task-level priors, language–capability confounding, geometric teacher contributions, and linguistic adaptation performance. We address each point below and provide the previously omitted selective continuation results.
W1. Task-Level Priors and Language–Capability Confounding
In our initial experimental design, we investigated the effects of reasoning data in different languages and Norwegian local-knowledge data by performing ordinary LoRA fine-tuning on the Qwen3-8B base model. We then evaluated the performance of different training groups across various tasks. The task-group classifications follow the same definitions as those described in the paper. The results are presented in the table below. Since this initial investigation involved only a small subset of the data, the results differ somewhat from those reported in the main experiments. Nevertheless, all experimental configurations and hyperparameters were kept consistent in this investigation.
Initial investigation with different language groups
No denotes Norwegian; En denotes English; R denotes reasoning; and K denotes knowledge. Values in parentheses indicate performance differences relative to the base model.
These preliminary results reveal several interesting patterns. First, training with Norwegian knowledge data (No_K) improves performance on local-knowledge tasks but comes at the cost of reasoning performance. Second, Norwegian reasoning data (No_R) provides neither consistent improvements in reasoning nor gains in local-knowledge tasks. Third, compared with No_R, English reasoning data (En_R) better preserves the model's reasoning capabilities, although it negatively affects performance on local-knowledge tasks.
Our initial hypothesis was that the reasoning trajectories learned from Norwegian reasoning data might be misaligned with those already established in the base model. Consequently, the limited amount of training data available in this preliminary investigation might have been insufficient to effectively adapt the model's existing reasoning patterns. These observations motivated us to explore the possibility of explicitly decoupling different capabilities during adaptation.
To improve interpretability and facilitate subsequent continued fine-tuning under capability-level guidance, we therefore manually classified the coarse-grained capabilities represented in the existing training data (Table 6). These capability assignments were subsequently incorporated into both the training and evaluation procedures.
We appreciate the reviewer's insightful observation regarding this design choice. We acknowledge that relying on manually assigned capability labels introduces a potential limitation, particularly concerning the dependence on human annotation. To investigate the extent to which this manual domain information contributes to the model's performance, we additionally evaluated the model without providing domain-level annotations during testing.
As shown in Table 4, when relying solely on the input prompts, without requiring manually assigned domain-knowledge labels, the model's performance decreases by only 0.50 Macro6 points, while still outperforming all baseline methods.. This suggests that the contribution of manually provided domain information at inference time is relatively marginal.
It is also important to clarify that neither NoCoLA nor OpenScience was involved in teacher construction or model training. Both datasets were used exclusively for evaluation to assess the generalizability of the proposed approach.