acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Capabilities via Sample-Level Geometric Routing for Parameter-Efficient Low-Resource Adaptation

Abstract

Adapting reasoning-capable large language models (LLMs) to low-resource languages presents a critical bottleneck: the model must acquire language-specific knowledge and linguistic capabilities without degrading its foundational reasoning abilities. However, different tasks and individual samples can place different demands on these capabilities, making uniform adaptation particularly challenging when target-language data are scarce. Standard mixed Low-Rank Adaptation (LoRA) applies the same adaptation mechanism across samples, while existing parameter-allocation and LoRA-routing methods primarily focus on allocating parameter capacity or dynamically combining LoRA experts, rather than explicitly modeling sample-specific capability requirements. To address this challenge, we propose Sample-Level Adaptive Geometric Routing (SAGR), a capability-centric framework that dynamically allocates fixed LoRA adaptation signals across reasoning, knowledge, and linguistic directions for each sample. By constructing sparse layer-sensitivity teachers and mapping these capability priors to individual fine-tuning samples, SAGR jointly trains a lightweight residual router with three capability-oriented LoRA branches. The router learns sample-specific capability weights and adaptation strength, enabling tailored combinations of capability-specific updates without increasing adaptation capacity. Extensive experiments on two Qwen3 reasoning models demonstrate the effectiveness over four state-of-the-art baselines. Furthermore, SAGR reveals fine-grained capability contributions of individual samples, offering an interpretable view of sample-level capability allocation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large Reasoning Models, Post-Training, Parameter-Efficient Fine-Tuning, Low-Resource Language Adaptation, Geometric Routing, Capability Decoupling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e4hfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts English-dominant reasoning LLMs (Qwen3-8B and 4B) to Norwegian using three rank-8 LoRA branches for knowledge, reasoning, and linguistic adaptation. A geometric teacher bypasses layers on 300 probes to build sensitivity profiles, transfers them to training samples via ridge regression, and guides a residual MLP router. With adapter capacity matched to a rank-24 LoRA (the router adds 1.05M parameters) and the same training budget, SAGR improves average scores over LoRA and MoE-LoRA baselines.

Critical strengths
  1. Fair capacity matching and clean parameterization. Matching the total adapter rank, data, and training steps across baselines is good practice (L776-781), and splitting each update into branch weights, layer weights, and scalar strength (Eq. 4, L218) is intuitive.

  2. Consistent gains on the accuracy tasks. SAGR improves Acc5 over rank-24 LoRA in all three seeds, by 4.07 points on average (Table 10), and the task-level results report the NoCoLA trade-off instead of hiding it in the aggregate.

Critical weaknesses
  1. The routing relies heavily on task-level priors that baselines do not get, and capability is confounded with language. Training blends ridge predictions with partition means at gamma = 0.7 using source indicators (L188, L697), inference maps each evaluation benchmark to a training partition by hand (Table 6, L747), and the router's residual corrections are scaled by only tau_w = 0.2 (L766), so tasks mapped to the same partition get nearly identical weights (Table 5). In the training set, 100% of reasoning samples are English while 100% of knowledge and linguistic samples are Norwegian (Table 8, L787).

  2. The complex geometric teacher adds very little, and the linguistic branch does not improve language performance. A Neutral teacher without geometric probes or teacher alignment already gets 48.59 Macro6 (Table 2, L341), almost the same as Input-only SAGR without manual partition mapping (48.80, Table 4, L372). On Qwen3-8B, NoCoLA routes 83% of its weight to the linguistic branch yet scores 16.32, no better than the frozen base model (16.37) and far below rank-24 LoRA (27.10, Table 1), and the continuation experiment mentioned at L260 and L418 is missing from the paper.

Questions for authors
  1. Why does NoCoLA on Qwen3-8B score 16.32 (about the same as the frozen model's 16.37, Table 1) despite routing 83.3% of its weight to the linguistic branch, while standard rank-24 LoRA reaches 27.10?
  2. Where is the selective continuation experiment mentioned at L260 and L418?
3 comments

We sincerely thank the reviewer for the constructive feedback, and appreciate the concerns regarding task-level priors, language–capability confounding, geometric teacher contributions, and linguistic adaptation performance. We address each point below and provide the previously omitted selective continuation results.

W1. Task-Level Priors and Language–Capability Confounding

In our initial experimental design, we investigated the effects of reasoning data in different languages and Norwegian local-knowledge data by performing ordinary LoRA fine-tuning on the Qwen3-8B base model. We then evaluated the performance of different training groups across various tasks. The task-group classifications follow the same definitions as those described in the paper. The results are presented in the table below. Since this initial investigation involved only a small subset of the data, the results differ somewhat from those reported in the main experiments. Nevertheless, all experimental configurations and hyperparameters were kept consistent in this investigation.

Initial investigation with different language groups

GroupBaseNo_KNo_REn_R
Reasoning57.2550.50 (−6.75)49.00 (−8.25)58.75 (+1.50)
Knowledge52.7562.00 (+9.25)51.00 (−1.75)47.75 (−5.00)

No denotes Norwegian; En denotes English; R denotes reasoning; and K denotes knowledge. Values in parentheses indicate performance differences relative to the base model.

These preliminary results reveal several interesting patterns. First, training with Norwegian knowledge data (No_K) improves performance on local-knowledge tasks but comes at the cost of reasoning performance. Second, Norwegian reasoning data (No_R) provides neither consistent improvements in reasoning nor gains in local-knowledge tasks. Third, compared with No_R, English reasoning data (En_R) better preserves the model's reasoning capabilities, although it negatively affects performance on local-knowledge tasks.

Our initial hypothesis was that the reasoning trajectories learned from Norwegian reasoning data might be misaligned with those already established in the base model. Consequently, the limited amount of training data available in this preliminary investigation might have been insufficient to effectively adapt the model's existing reasoning patterns. These observations motivated us to explore the possibility of explicitly decoupling different capabilities during adaptation.

To improve interpretability and facilitate subsequent continued fine-tuning under capability-level guidance, we therefore manually classified the coarse-grained capabilities represented in the existing training data (Table 6). These capability assignments were subsequently incorporated into both the training and evaluation procedures.

We appreciate the reviewer's insightful observation regarding this design choice. We acknowledge that relying on manually assigned capability labels introduces a potential limitation, particularly concerning the dependence on human annotation. To investigate the extent to which this manual domain information contributes to the model's performance, we additionally evaluated the model without providing domain-level annotations during testing.

As shown in Table 4, when relying solely on the input prompts, without requiring manually assigned domain-knowledge labels, the model's performance decreases by only 0.50 Macro6 points, while still outperforming all baseline methods.. This suggests that the contribution of manually provided domain information at inference time is relatively marginal.

It is also important to clarify that neither NoCoLA nor OpenScience was involved in teacher construction or model training. Both datasets were used exclusively for evaluation to assess the generalizability of the proposed approach.

W2. Contribution of the Geometric Teacher

We appreciate the reviewer's concern regarding the incremental contribution of the geometric teacher. We would like to clarify that the geometric teacher is intended to provide an initial capability-aware routing reference using only 300 probes, rather than to determine the final adaptation behavior. The main contribution of SAGR lies in the joint optimization of three capability-specific LoRA branches and a residual router, which continuously refines the teacher-guided branch weights, layer allocation, and adaptation strength throughout training.

The neutral teacher achieves 48.59 Macro6, indicating that the multi-branch architecture and learned routing mechanism already contribute substantially to SAGR's performance. Introducing geometric guidance further improves Macro6 to 49.30. Although this incremental improvement is relatively modest, it demonstrates the additional benefit of capability-specific geometric priors.

Importantly, the input-only configuration (48.80 Macro6) uses the same teacher-trained SAGR model but replaces the manually mapped inference reference with one derived from input representations and neighboring training samples. Therefore, this result evaluates the dependence on inference-time task mapping rather than the contribution of the geometric teacher itself.

These findings suggest that SAGR's effectiveness arises primarily from the interaction between capability-specific adaptation branches and dynamically learned routing, with the geometric teacher providing additional structural guidance.

W2 & Q1. Linguistic Branch Performance on NoCoLA

We agree that the high linguistic routing weight (83.3%) does not translate into a corresponding improvement on NoCoLA, where SAGR achieves 16.32 ERRANT F0.5, compared with 16.37 for the frozen base and 27.10 for conventional LoRA-r24.

An important distinction is that the linguistic supervision in the original training mixture primarily focuses on Norwegian idiomatic expressions, whereas NoCoLA evaluates grammatical error correction. Although both tasks involve linguistic competence, they require different forms of language adaptation. NoCoLA is not included in the original training mixture and serves as an unseen evaluation task to assess generalization beyond the training tasks. Under the same training data and optimization budget, SAGR achieves 8.00 accuracy on Norwegian idiom completion, compared with 2.50 for LoRA-r24, but performs worse on NoCoLA. These results suggest that different adaptation methods may exhibit different performance trade-offs across heterogeneous tasks, even under identical training conditions.

Furthermore, the linguistic routing weight reflects how adaptation signals are allocated across capability-specific branches rather than directly measuring their effectiveness on a particular task. The 83.3% linguistic routing weight indicates that SAGR primarily associates NoCoLA with the linguistic branch rather than the knowledge or reasoning branches. However, generalization to this unseen task may be influenced by differences in task requirements and the trade-offs involved in adapting multiple capabilities simultaneously.

This trade-off is further reflected in the aggregate results: SAGR achieves higher Acc5 (55.90 vs. 52.00) and Macro6 (49.30 vs. 47.85) than conventional LoRA-r24, despite its lower NoCoLA performance. We will clarify these findings in the revised manuscript.

W2 & Q2. Selective Continuation Experiments

We thank the reviewer for pointing out the missing selective continuation experiments referenced at L260 and L418. We apologize for this omission. These experiments were initially conducted as a preliminary investigation limited to the Law domain. Since we had not yet evaluated their generalizability across additional domains, we decided not to include these preliminary results in the original submission. Nevertheless, we believe these findings provide useful initial evidence regarding selective continuation. We therefore present the results below to examine whether selectively updating capability-specific branches can facilitate new-domain learning while retaining existing capabilities.

Law continuation. Starting from the calibrated Qwen3-8B SAGR model, we perform continued training on 1,381 Law examples for one epoch (87 optimizer steps). We evaluate Law token-F1 on 200 examples and assess the retention of existing capabilities using 200 examples from each of the original six benchmarks, consistent with the evaluation protocol in the original submission. The average routing weights assigned to the Law training samples by the calibrated SAGR checkpoint are shown below:

Knowledge (K)Reasoning (R)Linguistic (C)
18.48%44.49%37.03%

The R/C branches account for 81.52% of the total routing weight, motivating their selection for continued adaptation. Accordingly, we update the R/C branches and router while freezing K. We compare this configuration with frozen-router and all-branch variants. We additionally introduce Branch Routing Regularization (BRR), which constrains the relative routing distribution between the dominant R branch (44.49%) and the frozen K branch against the saved teacher reference, while leaving C unconstrained for flexible adaptation. Conventional LoRA-r24 is continued on the same Law data under the same optimization budget.

Additional results on Law selective continuation

Continuation configurationLaw token-F1Macro6ΔMacro6
SAGR before continuation17.6749.300.00
R/C + router (K frozen)18.5847.79−1.51
R/C (K and router frozen)19.2547.20−2.11
R/C + router (K frozen) + BRR19.9447.95−1.36
K/R/C + router18.8347.46−1.85
LoRA-r24 before continuation17.3547.850.00
LoRA-r24 after Law continuation30.6243.27−4.58

Law token-F1 measures domain-specific adaptation, while Macro6 measures performance on the original six evaluation benchmarks. ΔMacro6 represents the change relative to each method's own checkpoint before continuation.

Selective adaptation and capability preservation. The results show that different branch-update strategies lead to different adaptation–retention trade-offs. Updating the R/C branches and router achieves 47.79 Macro6 and 18.58 Law token-F1, compared with 47.20 and 19.25 when the router is frozen, and 47.46 and 18.83 when all branches are updated, respectively. Introducing BRR further improves Macro6 to 47.95 and Law token-F1 to 19.94, achieving the best performance on both metrics among the SAGR continuation variants. These findings suggest that regularizing the relative K/R routing distribution can help retain performance on existing tasks while supporting adaptation to the new domain.

Trade-off between new-domain learning and existing capabilities. With BRR, SAGR retains 47.95 Macro6, compared with 49.30 before continuation, corresponding to a decrease of approximately 1.36 points. In contrast, conventional LoRA-r24 decreases from 47.85 to 43.27, a reduction of 4.58 points. Notably, LoRA-r24 falls below the frozen base (43.27 vs. 46.15 Macro6, as reported in Table 1) after Law continuation, whereas SAGR with BRR remains above the frozen base (47.95 vs. 46.15). However, LoRA-r24 achieves higher Law token-F1 (30.62) than SAGR (19.94), indicating stronger new-domain adaptation accompanied by greater degradation on the original benchmarks.

Overall, these results suggest that selective branch updates and routing regularization can help balance new-domain adaptation with the retention of existing capabilities. Because the two methods start from different adapted checkpoints, these results illustrate their observed adaptation–retention trade-offs. The within-SAGR comparisons provide more direct evidence of the effects of branch selection and routing regularization.

We will incorporate these experimental protocols, results, and limitations into the revised manuscript to provide a more complete evaluation of SAGR's capability-specific routing mechanism.

Reviewer_2RHwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I strongly suspect this manuscript contains a substantial amount of AI-generated content, which goes beyond the scope stated in the AI Use Statement. I provide my evidence in the following Weaknesses section.

Given the concerns, I decline to provide further feedback, and would request a desk rejection. Or alternatively, I would suggest authors to perform a major revision.

Critical strengths

Not available.

Critical weaknesses
  1. Fake citation:

Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen Zhao, Yuhao Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviate world knowledge forgetting in large language models via MoE-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 5729–5752, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.309. URL https://aclanthology.org/2024.acl-long.309/.

The attached link (https://aclanthology.org/2024.acl-long.309/) points to a different paper, Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models.

  1. Fake citation:

Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. LoraHub: Efficient cross-task generalization via dynamic LoRA composition. In Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=TrjbxzRcnf.

The attached link (https://openreview.net/forum?id=TrjbxzRcnf) is invalid and points to nothing.

  1. Fake citation:

Yuan Zhuang, Yi Shen, Yuexin Bian, Qing Su, Shihao Ji, Yuanyuan Shi, and Fei Miao. Ld-mole: Learnable dynamic routing for mixture of lora experts. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ZTjM1gH0ug.

The link (https://openreview.net/forum?id=ZTjM1gH0ug) is invalid and points to nothing.

  1. Duplicated citation:

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.

  1. Duplicated citation:

Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. Large language models are cross-lingual knowledge-free reasoners, 2024.

Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, and Shujian Huang. Large language models are cross-lingual knowledge-free reasoners. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1525–1542, 2025.

Questions for authors

See the concerns in Weaknesses section

1 comment

We sincerely thank the reviewer for carefully examining our manuscript and identifying these bibliographic inconsistencies. We take the concerns regarding AI-generated content seriously and appreciate the opportunity to address them.

We would like to clarify that our manuscript represents the authors' original research and was not AI-generated. Our use of AI tools strictly adhered to the scope disclosed in our AI Use Statement. All cited works are authentic and independently verifiable, and the identified issues concern bibliographic inaccuracies and duplicate entries rather than fabricated references.

We have carefully investigated the five issues raised by the reviewer and provide the following clarifications:

  1. LoRAMoE: Its arXiv preprint and final ACL publication contain different titles, author lists, and author ordering. Our reference retains the arXiv title ("Alleviate" rather than "Alleviating") and the ordering of certain authors, such as Zhiheng Xi and Xiao Wang. The incorrect names "Wei Shen Zhao" and "Yuhao Shen" may have resulted from the misparsing or merging of neighboring author names during reference compilation. We also acknowledge the incorrect URL and will correct the reference according to the official ACL publication record.

  2. LoraHub: The title, authors, publication venue, and year are correctly reported and can be independently verified through the official COLM 2024 proceedings. The error concerns the OpenReview identifier, which should be corrected to https://openreview.net/forum?id=TrloAXEJ2B.

  3. LD-MoLE: Similarly, the title, complete author list, publication venue, and year are consistent with the official ICLR 2026 proceedings. The incorrect OpenReview identifier will be replaced with the correct one: https://openreview.net/forum?id=4ST2YyTjI7.

  4. LoRA: The duplication arises from separately including the 2021 arXiv preprint and the 2022 ICLR publication. These versions are independently indexed in bibliographic databases such as DBLP, with separate citation records.

  5. Large Language Models Are Cross-Lingual Knowledge-Free Reasoners: Similarly, the 2024 preprint and 2025 NAACL publication are independently indexed as separate bibliographic records, resulting in duplicate entries.

These findings provide verifiable context for the identified bibliographic inconsistencies and confirm that all cited works correspond to existing scholarly publications. We fully acknowledge our responsibility for these errors and will carefully correct the affected references, remove duplicate entries, and verify the bibliography against official publication records.

However, we respectfully believe that these bibliographic inaccuracies, while requiring correction, do not by themselves constitute sufficient evidence of substantial AI-generated content beyond the scope of our AI Use Statement. Reference errors should be distinguished from the provenance and scientific validity of the research itself. This research represents more than six months of substantial work, and we are prepared to provide supporting materials, including experimental progress reports, research records, and manuscript revision histories, to demonstrate the authenticity and development of our work if necessary.

We sincerely appreciate the reviewer's careful examination and constructive observations. We remain committed to improving the manuscript and would welcome further feedback on its methodology, technical contributions, and experimental evaluation.

Reviewer_9Bs3OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

To effectively adapt LLMs in low-resource settings, the authors propose SAGR, a capability-centric parameter-efficient framework that dynamically allocates adaptation signals across three LoRA branches corresponding to their pre-defined capability categories: knowledge, reasoning, and linguistic adaptation.

They compare SAGR with baseline methods across a diverse range of Norwegian tasks, which serve as representative benchmarks of low-resource adaptation, to demonstrate SAGR’s effectiveness.

Critical strengths
  1. They clearly identify the limitations of existing approaches. Although some aspects require further explanation, SAGR offers a well-structured approach to handling heterogeneous objectives in low-resource adaptation.
  2. They conduct thorough ablation studies to examine the contribution of each component, providing empirical insights into the proposed framework.
Critical weaknesses
  1. Although using the Qwen family, developed in China, is not inherently inappropriate for evaluating SAGR, additional experiments with other foundation models, particularly those with a stronger emphasis on English-speaking countries, such as Llama, would help substantiate their claims and provide a more balanced evaluation.
  2. The evaluation focuses exclusively on Norwegian as the target low-resource language. Experiments and analyses on additional low-resource languages are needed to provide more comprehensive empirical validation of SAGR.
Questions for authors
  1. What motivates the choice of knowledge, reasoning, and linguistic adaptation as the three capability dimensions? They should justify the criteria used to define their probes, with reference to relevant prior work [1].
  2. The reference for a critical baseline contains errors. Could the authors clarify whether these are inadvertent citation errors and provide the correct reference?

[1] Lee et al., KnowProxy: Adapting Large Language Models by Knowledge-guided Proxy, ICLR 2026

6 comments

We sincerely thank the reviewer for the constructive feedback and for highlighting the importance of evaluating SAGR beyond the Qwen family, examining additional low-resource languages, and clarifying the motivation behind our capability-based design. We address each point below.

W1. Generalization Beyond the Qwen Family

We agree that evaluation exclusively on Qwen models is insufficient to establish broader applicability. To address this concern, we extended our experiments to Llama-3.1-8B-Instruct, a foundation model from a different model family.

We retained the original SAGR architecture with three rank-8 K/R/C LoRA branches and a teacher-guided residual router. We additionally examined the teacher-transfer representation dimension and learning rate. The best evaluated configuration uses SVD32, an adapter learning rate of 5 × 10⁻⁵, a router learning rate of 1 × 10⁻⁵, and linear learning-rate decay. SVD32 refers to the teacher-transfer representation dimension, not the LoRA rank.

All methods use the same 22,711-example training mixture, one training epoch, and 1,420 optimizer steps. Results are based on seed 42 and 200 evaluation examples per task.

Additional results on Llama-3.1-8B-Instruct

MethodMathOpenSciOBQANRKIdiomsNoCoLAAcc5Macro6
Frozen Base50.5047.5070.0058.008.0023.9146.8042.99
LoRA (r24)38.5042.0059.0037.503.0020.4336.0033.41
AdaLoRA (r24)40.0038.0071.0057.005.5031.1442.3040.44
MoE-LoRA (Top-2, 3E)38.5035.5069.5056.507.5026.3441.5038.97
MoE-LoRA (Top-3, 3E)34.5036.5070.0056.506.0028.1640.7038.61
SAGR43.5043.0067.5054.007.5026.2143.1040.29

Bold highlights the best results among fine-tuned methods. Acc5 averages the five accuracy tasks, while Macro6 additionally includes NoCoLA ERRANT F0.5. All scores are percentages.

Overall performance and capability preservation. SAGR achieves the highest Acc5 (43.10) among all evaluated fine-tuned methods, outperforming AdaLoRA (42.30), MoE-LoRA Top-2 (41.50), and conventional LoRA-r24 (36.00). Its Macro6 reaches 40.29, exceeding LoRA-r24 by 6.88 percentage points and MoE-LoRA Top-2 by 1.32 points, while remaining only 0.15 points below AdaLoRA.

Importantly, SAGR outperforms conventional LoRA-r24 on all six evaluation metrics. It also achieves the highest Math (43.50) and OpenScience (43.00) scores among the fine-tuned methods. Compared with the frozen base, SAGR exhibits smaller performance decreases on these two reasoning-related benchmarks than the other evaluated PEFT methods, indicating better preservation of existing reasoning-related performance under heterogeneous fine-tuning.

These results support our central motivation that low-resource adaptation should balance the acquisition of target-language capabilities with the preservation of capabilities already acquired during pretraining.

W2. Evaluation on Additional Low-Resource Languages

We agree that evaluating SAGR on additional low-resource languages, particularly those outside the Germanic language family, would provide stronger evidence of its broader applicability.

However, such an evaluation presents challenges beyond collecting general-purpose fine-tuning data. SAGR requires heterogeneous supervision covering three complementary adaptation directions: knowledge, reasoning, and linguistic adaptation. While English reasoning datasets can be reused, high-quality language-specific knowledge and linguistic resources remain scarce in many low-resource languages. In particular, culturally relevant knowledge, idiomatic expressions, and grammatical phenomena require careful curation and, ideally, validation by proficient speakers.

Applying SAGR to additional languages also requires representative capability anchors to guide adaptation and reliable benchmarks to evaluate its effectiveness.

We plan to investigate typologically diverse low-resource languages, such as Basque and Swahili, in future work and will explicitly acknowledge the current evaluation scope as a limitation.

Q1. Motivation and Selection Criteria for Capability Probes

Thank you for this question. The decomposition into knowledge, reasoning, and linguistic adaptation is motivated by the observation that low-resource language adaptation involves multiple, partially distinct learning requirements:

  • Knowledge adaptation (K) concerns the acquisition, retention, and application of factual and locally relevant information.
  • Reasoning adaptation (R) concerns the ability to perform logical inference, mathematical reasoning, and multi-step problem solving.
  • Linguistic adaptation (C) concerns language-specific understanding and generation, including lexical, grammatical, and idiomatic competence.

This design is consistent with prior findings that knowledge representations are more language-dependent than knowledge-free reasoning (Hu et al., NAACL 2025), supporting the distinction between knowledge adaptation and reasoning preservation, and that linguistic and reasoning representations are partially separable (Zhao et al., NeurIPS 2026), providing further justification for a dedicated linguistic adaptation branch.

Accordingly, we select representative probes corresponding to these three capability dimensions: 155 knowledge probes from instruction-based knowledge QA, NorOpenBookQA, and NorQuAD; 95 reasoning probes from mathematical and knowledge-free reasoning tasks; and 50 linguistic probes from Norwegian idiom completion.

Rather than relying solely on predefined task categories, SAGR measures the layer-wise sensitivity of these probes through interventions on the frozen foundation model. The resulting sensitivity profiles are used to construct model-specific capability directions that guide sample-level adaptation. Since individual samples may involve multiple capabilities simultaneously, SAGR dynamically allocates adaptation signals across the three branches rather than assigning each sample to a single fixed branch.

We also appreciate the reviewer’s reference to KnowProxy [1], which demonstrates the value of leveraging knowledge and reasoning signals from frozen LLMs to guide adaptation. While KnowProxy does not directly define our capability categories, it provides complementary motivation for using pretrained capability signals rather than relying solely on task labels.

We will clarify these motivations and the rationale for selecting our capability probes in the revised manuscript.

Q2. Correction of the Baseline Reference

We thank the reviewer for identifying the bibliographic inaccuracies in our LoRAMoE reference. Upon further examination, we found that the arXiv preprint and the final ACL publication differ in their titles, author lists, and author ordering. Our reference retained the arXiv title ("Alleviate" rather than "Alleviating") and the preprint's ordering of certain authors, including Zhiheng Xi and Xiao Wang. However, the incorrect names "Wei Shen Zhao" and "Yuhao Shen" may have resulted from the misparsing or merging of adjacent author names during reference compilation. We also acknowledge the incorrect URL. We apologize for these errors and will correct the reference according to the official ACL publication record.

The correct reference is:

Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932–1945, Bangkok, Thailand. Association for Computational Linguistics. DOI: 10.18653/v1/2024.acl-long.106. URL: https://aclanthology.org/2024.acl-long.106/

I appreciate the authors’ thoughtful responses to my review and have carefully read their rebuttal.

The authors have partially addressed my concerns. Accordingly, I have raised both my soundness score and my overall rating from 1 to 2.

I strongly recommend that the authors prioritize correcting the errors in the references and upload the revised manuscript before the rebuttal period ends.

I have one follow-up question about the evaluation on additional low-resource languages.

Q. Would it be feasible to conduct a pilot study on at least one additional low-resource language?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses to the Reviewers

We sincerely thank all reviewers for their careful evaluation, constructive feedback, and thoughtful engagement with our work. Their comments have been invaluable in strengthening the experimental validation, methodological clarity, and presentation of our manuscript.

Strengths: Reviewers highlighted SAGR's well-structured approach to handling heterogeneous adaptation objectives (Reviewer 9Bs3), its intuitive decomposition of adaptation into branch weights, layer allocation, and update strength (Reviewer e4hf), and the fair comparisons with competitive baselines under matched parameter and training budgets (Reviewer e4hf). They also recognized the comprehensive ablation studies (Reviewer 9Bs3) and the consistent improvements on accuracy-based benchmarks across multiple random seeds (Reviewer e4hf). These observations support our motivation for formulating low-resource adaptation as a sample-level capability allocation problem.

In response to the reviewers' concerns, we have substantially strengthened the manuscript. To evaluate generalization beyond Qwen and Norwegian, we conducted additional experiments on Llama-3.1-8B-Instruct, where SAGR achieves the highest Acc5 (43.10) among the evaluated fine-tuned methods, and a Basque adaptation pilot, where SAGR outperforms conventional LoRA-r24 (41.23 vs. 38.87 Macro5). We also added Law domain selective continuation experiments, examining how different branch-update strategies affect new-domain adaptation and existing capability retention. Furthermore, we strengthened the motivation for the knowledge, reasoning, and linguistic capability decomposition, clarified the construction and selection of 300 capability probes, and examined the dependence on manually assigned task-level priors and geometric teacher guidance. We also clarified the NoCoLA performance discrepancy in our responses, distinguishing routing allocation from generalization to unseen linguistic tasks. Finally, we carefully investigated and corrected the identified bibliographic inaccuracies and duplicate references. We also updated the AI Use Statement to transparently disclose the use of AI-assisted semantic evaluation in the reviewer-requested Basque pilot, where ambiguous free-response answers required reference-guided semantic assessment.

In the revised version of the paper, we added the following sections to address the reviewers' key questions and concerns:

  • Section 3.6: Selective continuation experiments, evaluating different branch-update strategies and their adaptation–retention trade-offs on the Law domain (Table 6).
  • Appendix D: Generalization beyond the Qwen family, including Llama-3.1-8B-Instruct experiments (Table 14).
  • Appendix E: Basque adaptation pilot, evaluating SAGR across five benchmarks covering knowledge QA, linguistic proficiency, reading comprehension, and reasoning (Table 15).

We also expanded the methodological explanations, clarified the language–capability confounding in our training data, discussed task-dependent adaptation–retention trade-offs, and corrected the bibliography.

Through these revisions, encompassing additional experiments, expanded analyses, methodological clarifications, and bibliographic corrections, we have substantially addressed the reviewers' major concerns while providing a more comprehensive and balanced evaluation of SAGR. We believe the revised manuscript now presents a stronger empirical foundation and clearer methodological justification for capability-aware parameter-efficient adaptation in low-resource languages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.