acceptodds
Under review as a conference paper at ICLR 2027

AMIGO: Adaptive Multilingual Instruction Tuning via Gradient Similarity and Budgeted Optimization

Abstract

Multilingual instruction tuning requires allocating a limited training budget across languages whose cross-lingual interactions evolve throughout training. Prior work typically addresses data mixing (e.g., sampling ratios) and training strategy (e.g., curricula or scheduling) separately, often relying on static or heuristic decisions that do not account for the non-stationary nature of multilingual optimization. We propose **AMIGO** (Adaptive Multilingual Instruction Tuning via Gradient Similarity and Budgeted Optimization), a unified framework that couples language grouping, training order, and budget allocation through gradient geometry re-estimated as training progresses. AMIGO operates in regimes: at each stage, it clusters languages based on validation gradient similarity, selects the next language family via a compatibility score that balances optimization potential with transition smoothness, and allocates training budget using a continuous knapsack objective grounded in a first-order approximation of the multilingual loss. The utility of a language is defined by its need-weighted gradient alignment with out-of-family languages, while its cost is derived from validation loss and gradient norms, eliminating the need for external evaluation at regime boundaries. This formulation captures the dynamic nature of cross-lingual interactions and enables adaptive reallocation of training resources as optimization progresses. Empirical evaluations on various instruction-following sets across 23 languages and multiple model families demonstrate that _AMIGO_ outperforms existing data-mixing and data-selection baselines on average under fixed training budgets, with notable gains on several benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multilingual, Instruction tuning, Budgeted optimization, Gradient geometry

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_4bb1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AMIGO, an adaptive multilingual instruction tuning framework that jointly adapts language grouping, training order, and budget allocation. The method estimates cross-lingual similarity from validation gradients, selects the language family to train in the next regime via a compatibility score, and allocates the training budget using a continuous knapsack objective. Experiments on three model families show that the method improves average benchmark performance.

Critical strengths
  1. The paper addresses an important optimization problem. Cross-lingual interactions evolve throughout training, so static data allocation may be suboptimal.

  2. The framework connects language grouping, training order, and budget allocation through shared gradient information, which makes the overall design coherent.

  3. The experimental coverage is good, including multiple baselines, component ablations, and additional generalization experiments on the Aya dataset.

Critical weaknesses
  1. The evaluation protocol appears to measure output-format compliance rather than multilingual capability, as several base models score well below random chance.

  2. Some baselines are missing. GradSim (Wang et al., 2023) also groups languages by gradient similarity and is the most directly related prior work, yet it is not included in the comparison.

  3. Some design choices contradict the paper’s own premise that gradient geometry evolves throughout training and must be continually re-estimated. For example, the transition term T uses the gradient direction stored at the previous regime boundary, which is already stale by the time it is used according to the paper’s own drift criterion.

  4. The claims about instruction following go beyond what the evaluation supports. The six benchmarks mainly test understanding, knowledge and reasoning. To support conclusions about the effectiveness of multilingual instruction tuning, the authors should add evaluations that directly measure instruction-constraint following, such as IFEval (Zhou et al., 2023), FollowBench (Jiang et al., 2024) and M-IFEval (Dussolle et al., 2025).

  5. The main text reports statistics from only one experiment.

Questions for authors
  1. Describe the evaluation protocol in detail, including the language versions of each benchmark, the prompt templates, and other settings.

  2. With 7B+ models and larger SFT budgets, all methods learn the output format quickly. Does AMIGO’s advantage still hold in this setting?

  3. How often are the gradients for drift detection recomputed? What share of the total wall-clock time goes to gradient estimation, family scoring and training, respectively?

  4. See the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4bb1

We thank the reviewer for the clear review and for finding the design coherent and the experimental coverage good.

W1 / Q1. Evaluation protocol

  • Benchmarks. All six benchmarks come from MuBench (Han et al., 2026), which provides each benchmark with the same test items in 61 languages. We use 10: ar, bn, de, en, fr, hi, ja, ru, sv, zh. We will cite MuBench in Section 5.1 and correct the dataset links in Appendix M ("MultiNLU" is MultiNLI).
  • Scoring. An answer is correct if the answer parsed from the output string-matches the gold label. Outputs with no parseable answer count as wrong. Scores are averaged over the 10 languages and 3 seeds.
  • Format vs. capability. Base checkpoints are not instruction-tuned and mostly produce no answer in the expected form, which is why most base scores fall below chance. To separate format from capability, we restrict attention to settings where every tuned method is above chance on every benchmark: Qwen3-4B on Bactrian-X, and all Aya runs. There, AMIGO beats the strongest baseline by average (GREATS) on all six benchmarks for Qwen3-4B (Table 3); RegMix and MLP-MKC+ remain ahead only on MMLU-Pro. AMIGO also has the best Aya average at both budgets (Table 10). On LLaMA-3.2-3B and Gemma-3-4B, some baselines stay near chance on MultiNLU, HellaSwag and StoryCloze, so we do not rest this point on those models.

W2. GradSim

GradSim trains a separate model per language group, so it does not map directly onto our single-model, fixed-budget setting. Its central idea, grouping languages once by gradient similarity, is tested by no_clustering (Table 6). In that variant, clusters are computed from gradients once at initialization and kept fixed, with scheduling and budget allocation unchanged. Re-estimating the clusters at every regime raises the average from 70.58 to 73.32 and wins on 4 of 6 benchmarks. We will rename the variant static_clusters, describe it as GradSim-style static grouping, and discuss GradSim explicitly in Section 2.

W3. Transition term T At each regime boundary, we recompute language-family gradients and compare them with stored gradients (Eq. 5) to capture drift and emerging language interactions, potentially forming new families as motivated in our paper. Since gradients are already recomputed, the current-parameter variant incurs no additional cost. Its impact is limited because the current-parameter transfer gain DD carries most of the weight (1−α=0.71-\alpha=0.7), while the stored direction TT still contributes meaningfully: removing it reduces the average from 0.733 to 0.714 (Table 7).

W4. Instruction-following claims

We agree that our claims go beyond what the evaluation supports. Our evaluation is generative: in each of 10 languages, the model must follow the task instruction, answer in the requested form, and be correct and more focused towards multilinguality. This captures answer-format following, but not constraint following as measured by IFEval, FollowBench or M-IFEval. We will make the scope more clear in our manuscript.

W5. Statistics in the main text

Beyond Table 1, the paper reports ablations (Table 6), sensitivity to α, β and τ_rel (Tables 7–9), Aya at two budgets (Table 10) and per-cell SDs over 3 seeds (Tables 3–5), but most of this sits in the appendix. We will move the ablation to main text and condense section 3 and 4 to make room.

Q2. Larger models and SFT budgets

We have not run 7B+ models, and agree this is an important next step. On larger SFT budgets, Aya gives direct evidence in a setting where every method is above chance. AMIGO has the best average at 100K (0.681 vs. 0.670) and at 200K (0.702 vs. 0.675). It is also the only method that gains from the extra data: +0.021, while no baseline gains more than 0.005 (Table 10). The margin over the best baseline grows from 0.011 to 0.027 as the budget doubles, so we see no sign of it shrinking once the format is learned.

Q3. Recompute frequency and wall-clock

Validation gradients for all 23 languages are recomputed every 50 steps to evaluate the drift statistic. At each regime boundary (21 in the default run), the same gradient pass also feeds clustering, family scoring and allocation. On Qwen3-4B (one H100, 50K),. All AMIGO-specific work totals 21% of wall-clock, and training takes the remaining 79%. comparison with other methods are discussed in main paper

Manuscript Revision: We have revised the manuscript to address the reviewers' suggestions. Specifically, we have updated Section 4.4 (Compatibility) in response to W3, clarified the evaluation protocol in the Evaluations paragraph of Section 5.1, moved the ablation paragraph to the main text (Table 2), renamed \texttt to \texttt and discussed it as a Grad-Sim variant in the ablation paragraph of Section 5.2, and added wall-clock time to the compute overhead paragraph in Section 5.2.

The manuscript has not been revised, and the main text still presents only one set of experimental results. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Manuscript revised.

Dear Reviewer,

We were unaware of the new feature of updating the manuscript during the rebuttal. Thank you for pointing this out, we have now updated the manuscript incorporating your suggestions. We have also added a "Manuscript Revision" point in the rebuttal above, indicating the sections we updated in the main text of our paper.

Reviewer_ZSTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMIGO (Adaptive Multilingual Instruction Tuning via Gradient Similarity and Budgeted Optimization), a unified framework that couples language grouping, training order, and budget allocation through gradient geometry re-estimated as training progresses.

Critical strengths
  1. This paper studies a well-motivated question in multilingual instruction tuning.
  2. This paper experiments with several model families and two datasets.
  3. This paper honestly reports the results in which the method loses.
  4. The paper has component ablation and good supporting materials.
Critical weaknesses
  1. Some issues in multilingual evaluation, such as English-origin or translated benchmarks, are used. The training data (BactrainX) is also machine-translated. There is a StoryCloze (maybe XStoryCloze is meant) but it doesn't cover bn, de among the evaluated languages: : ar, bn, de, en, fr, hi, ja, ru, sv, zh claimed by the paper. There are some training languages not evaluated. No instruction-following evaluation is conducted.
  2. The base model performance seems to be very low, with somewhat random guesses on HellaSwag and MMLU-Pro. But instruction-tuned models have seen significant improvement. There might be some sort of format compliance issue. In the main results table, no SD is given for the 3-seed experiment.
  3. Baseline choice and adaptation are under-specified, and key baselines are missing; for example, GradSim grouping (Wang et al., 2023) and DynamixSFT (Shin et al., 2025; cited in Section 2).
  4. Key settings are not reported for reproducibility. No code release.
  5. Appendix G also states "We plan to conduct a deeper gradient-based behavior analysis ... in the final manuscript", which signals unfinished analysis.
Questions for authors

Which multilingual versions of each benchmark are used for the 10 evaluation languages?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZSTn

We thank the reviewer for the careful review and for noting that we report the cases where AMIGO loses.

W1. Multilingual evaluation

  • Benchmark versions and coverage. All six benchmarks come from MuBench (Han et al., 2026), which translates each benchmark into 61 languages with the same test items in every language. It covers all 10 evaluation languages for every task, including StoryCloze in bn and de. This benchmark is used to keep evaluation langauges constant throughout different eval datasets
  • Machine-translated training data. This is why we include Aya, which is written natively in each language. On Aya (Qwen3-4B), AMIGO has the best average at both budgets (0.681 at 100K, 0.702 at 200K) and is best on ARC-E and ARC-C at both (Table 10). It is also the only method that improves noticeably with more data: from 100K to 200K, no baseline's average changes by more than 0.005.
  • Training languages not evaluated. The 10 evaluation languages span 6 of the 7 family groups and 7 scripts in our training set, from high-resource (en, zh) to lower-resource (bn, hi). Per language on Qwen3-4B (Bactrian-X, 50K, mean of six benchmarks), AMIGO improves on Uniform in all 10 languages and is best in 8. Its largest margins over the strongest baseline are in ar (+6.0), hi (+5.8) and bn (+5.1). We will add this per-language table and state that the remaining 13 languages are trained on but not evaluated.
  • Instruction following. We partly agree. Our evaluation is generative: in each language the model must follow the task instruction and give an answer in the requested form, and is scored on correctness. Instruction tuning is what teaches this (Qwen3-4B average: 0.154 base vs. 0.615 Uniform). This captures answer-format following but not IFEval-style constraint following, so we will narrow "instruction-following" in the abstract and conclusion accordingly.

W2. Base scores, format compliance and SDs

An answer is correct when the answer parsed from the model's output string-matches the gold label. Outputs with no parseable answer count as wrong. Base checkpoints are not instruction-tuned and mostly produce no answer in the expected form, which is why most base scores fall below chance.

This affects the base rows, not the comparison among tuned methods. All tuned methods share the protocol, budget, recipe and starting checkpoint. On Qwen3-4B and in every Aya run, every tuned method scores above chance on every benchmark. In that setting, AMIGO still leads. On Qwen3-4B it beats the strongest baseline by average (GREATS) on all six benchmarks, though RegMix and MLP-MKC+ remain ahead on MMLU-Pro. It also has the best Aya average at both budgets. Mean ± std over 3 seeds for every cell is in Tables 3–5, and we will move the SDs into Table 1.

W3. Baselines

  • GradSim. GradSim trains a separate model per language group, so it does not map directly onto a single-model, fixed-budget comparison. The closest test is no_clustering (Table 6), which is GradSim-style static grouping inside AMIGO. Clusters are computed from gradients once at initialization and kept fixed, and everything else is unchanged. Re-estimating clusters during training raises the average from 70.58 to 73.32. The caption's "removed" is imprecise; we will describe this variant as static gradient-based grouping.
  • DynamixSFT. It adapts sampling over whole datasets treated as bandit arms, using a one-step look-ahead reward, and neither groups nor orders them. With languages as arms, it becomes an adaptive mixing method of the same kind as PiKE and GREATS, which we compare against. We will discuss it, along with TAG, explicitly in Section 2.
  • Adaptation. RegMix and MLP-MKC+ each produce a fixed mixture before training, which is then held constant under the same recipe as every other method. GREATS uses its official code; PiKE is our reimplementation of the paper's algorithm, since no official code exists. A new appendix will list each baseline's adaptation and hyperparameters.

W4. Reproducibility

Table 2 (Appendix C) gives the configuration (checkpoints, data, budget, prompt format, validation split, sequence length, optimizer, schedule, precision, hardware, AMIGO hyperparameters), and Algorithms 1–3 specify each step. We intend to release code after internal discussions.

W5. Appendix G

We agree the sentence reads as unfinished, and we remove it. The gradient-behavior analyses are already in the paper: cluster migration across regimes (Figure 1), the evolution of the cost parameter across regimes (Figure 2, Appendix B), and per-language allocation and loss change on Aya (Appendix H, Table 11).

Reviewer ZSTn: has the authors addressed your concerns?

Reviewer_Bn4COpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript identifies a limitation of prior art that uses gradient geometry to steer training (that gradient similarity is non-stationary). The proposed method uses this as inspiration to define a procedure that uses a gradient-derived proxy of training utility, which determines which languages and mixture are used next during the next training segment. The results show improved language modeling capability against baselines.

Critical strengths

S1. The manuscript identifies one plausible limitation of prior art: the non-stationarity of gradient similarity has implications to utility estimates that are not considered by prior work.

S2. Experiments are conducted on a reasonable number of datasets, and there is an ablation experiment on the hyperparameters.

Critical weaknesses

W1. I would argue that the theory you present as motivation and the actual training regime are disconnected. You make the point that "estimates of language utility derived from gradient interactions [...] become outdated as training evolves." But then the method uses an estimate of training benefit over an entire regime, which has hundreds of steps (you say <500). This is counter to the intuition laid out in the manuscript, as no reason is given to expect high-dimensional non-linear landscapes to preserve gradient geometry over long horizons (or even shorter horizons, e.g., at the start of training). This is even more important because one big contribution over GradSim is the splitting of training into regimes.

W2. There is no analysis of the training that connects to the theory; so there is no evidence to attribute the performance improvements specifically to leveraging the gradient geometry. I was expecting some kind of white-box analysis to confirm the motivation and yield some insights about the dynamics of the gradient geometry.

Questions for authors

Q0. How long exactly was each regime in your experiments?

Q1. What is the reason why step-to-step changes in gradient similarity should be ignored and instead training should be split into fixed long "regimes"?

Q2. Do you have any evidence that your training benefit estimate is (1) stable over the long horizons in your experiments and (2) doesn't fluctuate too much step-to-step? This is important, because the the benefits of the method might instead come from committing to different data mixing strategies over long horizons, rather than being attributable to decisions made based on gradient interactions. E.g., if the gradient similarities fluctuate step-to-step, the decision could be very sensitive to the step in which the decision is made and to training dynamics.

Q3. What are the implications of using only the gradient of the MLP parameters of the final transformer layer? This seems like a big limitation of your experimental setup that further makes the results hard to attribute to your motivation theory. Why do you do this rather than over the entire networ or larger part of it?

Q4. What exactly is the score metric that you have in e.g., table 1? The manuscript never says how these numbers are computed (scoring method, shot count it applicable, etc), so the significance of the results cannot be fully determined.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bn4C

We thank the reviewer for the thoughtful questions. We address each point below.

W1 / Q1. Why not react to step-level changes?

AMIGO does not ignore changes within a regime. Validation gradients for clustering and allocation are recomputed at each regime boundary, and between boundaries the drift statistic ρ(t)\rho(t) is checked every 50 steps; a regime ends as soon as it exceeds τrel\tau_{rel}, so regimes are not of fixed length (Q0). What AMIGO avoids is re-deciding the mix at every step:

  1. Noise. Each similarity estimate uses at most 100 validation examples per language, so step-level changes mix real drift with sampling noise. The threshold acts only on change that has accumulated.
  2. Evidence. Re-deciding more often hurts: τrel=1.2\tau_{rel}=1.2 ends regimes 66 times and scores 0.709, against 0.732 with 21 regimes at the default (Table 9).
  3. Cost. Per-step re-estimation would need a validation backward pass for all 23 languages at every training step.

W2. Evidence tied to gradient geometry

With budget, recipe and drift rule fixed, replacing any gradient-based decision lowers the average (Qwen3-4B, Tables 6 and 7):

VariantAvg
Full AMIGO73.32
no_scheduling: next family picked at random69.74 (−3.58)
no_clustering: clusters frozen after the first estimate70.58 (−2.74)
No transition term (α=0\alpha=0)71.4 (−1.9)

Table 6's caption describes the first two variants only as "removed"; the descriptions above are what they do, and we will fix the caption.

On the dynamics themselves: the number of clusters selected by the gap statistic varies from 3 to 7 across regimes (Appendix A), Figure 1 traces how cluster membership changes, and in the default run 10 of 21 regimes ended because the similarity matrix had drifted past τrel\tau_{rel}. We will move this analysis into the main text.

Q0. Regime length. Default run (Qwen3-4B, Bactrian-X, 50K): 21 regimes of 50 to 500 steps. 10 ended early on drift; the others ended at the 500-step cap or when a language's budget ran out (Algorithm 1). Drift is checked every 50 steps.

Q2. Stability of the estimate

  1. Commitment vs. informed decisions. no_scheduling keeps the drift rule, clusters and budget allocation but picks the next family at random. The average falls from 73.32 to 69.74, and the full method wins on 5 of 6 benchmarks. Committing to a mix is therefore not enough: the gradient-informed choice adds 3.58 points on top.
  2. Step-to-step fluctuation. See W1: reacting to finer-grained changes lowers performance rather than improving it (Table 9).

Q3. Why only the final-layer MLP?

  1. Where language information sits. In decoder-only LLMs, language-specific neurons concentrate in the top and bottom layers (Tang et al., 2024); late layers map a shared latent space back to the output language (Wendler et al., 2024); MLPs act as key-value memories over the output vocabulary (Geva et al., 2021).
  2. Cost. The final MLP holds 75M of 4.0B parameters (under 2%). Parameter subsets are standard in gradient-based attribution for this reason (Koh & Liang, 2017; Xia et al., 2024).
  3. Link to the theory. Eq. (1) decomposes over parameter blocks, so the final-MLP inner product is one additive block of the full-model alignment. We use it as a proxy, and W2 shows that decisions based on it beat random ones.

We will note the restriction in the limitations, with full-model gradients as a natural extension.

Q4. Score metric. Scores are accuracy under generative evaluation on MuBench (Han et al., 2026), which provides all six benchmarks with the same test items in each of our 10 evaluation languages. The model generates an answer, which counts as correct if the answer parsed from its output string-matches the gold answer; outputs with no parseable answer count as wrong. Scores are averaged over the 10 languages and 3 seeds (per-cell mean ± std in Tables 3 to 5). This is also why base scores can fall below chance: base checkpoints often give no parseable answer. All methods share the identical protocol, budget, recipe and starting checkpoint. We will add these details to Section 5.1.

Manuscript Revision: We have revised the manuscript to address the reviewers' suggestions. Specifically, we have added a discussion on final-layer selection in Section 4.1 (Gradient Estimation) in response to Q3, included evaluation protocol in the Evaluations paragraph of Section 5.1. We have also added clarification on fixed step validation (Q1) in Section 4.2.

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided. we have now updated the manuscript incorporating your suggestions. We have also added a "Manuscript Revision" point in the rebuttal above, indicating the sections we updated in the main text of our paper.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer Bn4C: have the authors satisfactorily answered your questions?

Reviewer_AADUOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new gradient-based data mixing method for multilingual fine-tuning.

Critical strengths
  • The problem motivation is important and interesting. The paper highlights the need for better data mixing strategies in multilingual fine-tuning, since multilingual interactions would change during training.
  • The empirical results are strong. AMIGO performs well across multiple benchmarks and models, consistently outperforming existing baselines on average.
Critical weaknesses
  • There are extensive arguments for why the algorithm should be designed this way in Section 4, but much of the justification is handwavy, with a lot of math and very little empirical evidence showing how these choices actually work out. The ablation study is also insufficient. While it ablates the three main components, it doesn't test many of the specific design choices.
    • For example, the paper's central motivation is that gradient similarities change over time, but there is no comparison between dynamic clustering and clustering only once at initialization. Similarly, it is unclear whether gradient-drift-based regime switching is actually better than simply switching at fixed intervals. These seem like important experiments to support the paper's main claims.
  • The paper uses gradients from only the final layer's MLP, but provides little evidence that these gradients accurately capture cross-lingual interactions. It would be useful to show whether gradient similarity actually predicts cross-lingual transfer, since this assumption underlies the entire method.
  • Would appreciate more graphs and figures. The current experiments section is mostly prose and one table, which makes it quite difficult to understand what is actually happening during training.
    • Figure 1 is too small and convoluted and not very legible. I think the finding here is interesting, and it shows how different languages and their gradients can shift over time, but the way it is presented now undermines the finding.
Questions for authors
  • Can the authors give an ablation that shows that re-clustering actually helps?
  • Could authors give more results on languages eval? Seems like AMIGO trains on 23 languages but only evaluates on 10.
  • Seems like theorem K.1 makes a stronger claim than its proof supports. Could the authors clarify why the stated condition guarantees that gradient geometry changes?
9 comments
AuthorsOpenReview ↗

Response to Reviewer AADU

We thank the reviewer for the careful reading and for finding the results strong. We address each point below.

1. Re-clustering vs. clustering once; drift vs. fixed intervals

  • Re-clustering (Q1). This comparison is in Table 6, with the caption no_clustering. We agree that the caption needs to be improved. Here clusters are computed from gradients once at initialization and then frozen, while scheduling and budget allocation run unchanged. Re-clustering at every regime boundary raises the average from 70.58 to 73.32 and wins on 4 of 6 benchmarks. We will fix the caption and rename the variant static_clusters.
  • Drift vs. fixed intervals. AMIGO combines both: each regime lasts at most 500 steps and ends early when the drift ρ(t)\rho(t) exceeds τrel\tau_{rel}, checked every 50 steps. In the default run, 10 of 21 regimes ended early on drift. We do not have a pure fixed-interval run; the closest evidence is Table 9, which varies how readily regimes end:
τrel\tau_{rel}RegimesAvg
1.2660.709
1.4 (default)210.732
1.6130.727

Ending regimes too often costs 2.3 points.

2. Does gradient similarity capture cross-lingual transfer?

To first order, a step on language jj changes the loss of language ii by −η⟨gi,gj⟩-\eta\langle g_i, g_j\rangle (Eq. 1): aligned gradients lower ii's loss, conflicting ones raise it. Gradient alignment is thus the first-order measure of one-step transfer, the same quantity behind gradient-based influence (TracIn; Pruthi et al., 2020) and data selection (LESS; Xia et al., 2024). Prior work supports it empirically:

  • Gradient similarity along training correlates with language proximity and with model quality (GradVac; Wang et al., 2021).
  • Grouping languages by gradient similarity gives larger gains than other similarity measures and correlates better with cross-lingual performance (GradSim; Wang et al., 2023).
  • Weighting languages by gradient alignment with a dev set improves multilingual NMT (MultiDDS; Wang et al., 2020).
  • Gradient-based influence shows languages draw on each other's data during fine-tuning, increasingly so as training proceeds (Choenni et al., 2023), which also supports the non-stationarity behind AMIGO.

On the final layer: language-specific neurons concentrate in the top and bottom layers of decoder-only LLMs (Tang et al., 2024), and late layers map a shared latent space to the output language (Wendler et al., 2024). Since Eq. 1 decomposes over parameter blocks, the final-MLP inner product is one block of the full alignment. We use it as a proxy at under 2% of parameters (75M of 4.0B), and decisions based on it beat random ones (point 2).

3. Figures

We agree. In the revision we will enlarge and simplify Figure 1 and move the ablations (Table 6) and sensitivity analyses (Tables 7 to 9) into the main text.

Q2. Why evaluate 10 of 23 languages? The 10 evaluation languages cover 6 of the 7 family groups in our training set and 7 scripts, from high-resource (en, zh) to lower-resource (bn, hi), and are evaluated for every method, model and seed. On Qwen3-4B (Bactrian-X, 50K; mean of the six benchmarks), AMIGO improves on Uniform in all 10 languages and is best in 8. Its largest gains over the strongest baseline are in ar (+6.0), hi (+5.8) and bn (+5.1); it trails only in ja (−0.7) and zh (−3.3).

Q3. Theorem K.1. Thank you for catching this. The condition should use the path-averaged Hessian Hˉi=∫01Hi(θ+tΔθ) dt\bar H_i=\int_0^1 H_i(\theta+t\Delta\theta)\,dt, so that gi(θ′)=gi(θ)+HˉiΔθg_i(\theta')=g_i(\theta)+\bar H_i\Delta\theta holds exactly. Then HˉiΔθ∉span G(θ)\bar H_i\Delta\theta\notin\mathrm{span}\,G(\theta) implies gi(θ′)∉span G(θ)g_i(\theta')\notin\mathrm{span}\,G(\theta), so the span changes. Since AMIGO uses pairwise alignment, we will also state that directly. With v=∑kαkgkv=\sum_k\alpha_k g_k:

⟨gi(θ′),gj(θ′)⟩−⟨gi(θ),gj(θ)⟩=−η (gj⊤Hiv+gi⊤Hjv)+O(η2)\langle g_i(\theta'),g_j(\theta')\rangle-\langle g_i(\theta),g_j(\theta)\rangle=-\eta\,(g_j^\top H_i v+g_i^\top H_j v)+O(\eta^2)

So alignment changes at first order unless this Hessian-weighted term vanishes. We present this as motivation and will soften the wording in Section 3 accordingly; the empirical evidence is Figure 1 and the number of clusters ranging from 3 to 7 across regimes (Appendix A).

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided. we have now updated the manuscript incorporating your suggestions. Thank you for your time and consideration.

Best regards, The Authors

Reviewer AADU, please let the authors know if they have addressed your many points.

Reviewer_r3fUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate a limited multilingual instruction-tuning budget when interactions between languages change during training. It proposes AMIGO, which uses validation gradients to dynamically group languages, select the next group to train, and allocate samples within that group. The allocation step uses a continuous knapsack formulation to balance estimated utility and cost. Experiments report higher average benchmark scores across several pretrained model families, with supporting results on a second training dataset.

Critical strengths
  • The method brings language grouping, training order, and budget allocation into one adaptive framework. Using the model's current gradients to guide these decisions gives the design a clear motivation and allows it to respond as language interactions change.

  • The paper reports average improvements across several pretrained model families, with additional support from a second training dataset. The component ablations and runtime measurements also help assess the benefits and costs of the method.

Critical weaknesses
  • The clustering ablation shows that removing the component hurts the average score, but it does not isolate the value of the gradient-based grouping or of updating it during training. Comparing fixed random groups, fixed initial gradient-based groups, and dynamically updated gradient-based groups, with appropriate controls for group sizes and training budgets, would make these contributions clearer.

  • Using only the MLP in the final Transformer layer has a reasonable efficiency motivation, but the paper does not directly test whether this provides a reliable signal for AMIGO's decisions. Could the authors compare it with gradients from several layers or a larger parameter subset, reporting both downstream performance and computational cost?

  • Results are mostly reported by benchmark, so it is unclear which languages benefit and whether some get worse as the average improves. The per-language appendix reports allocations and training losses; held-out task scores for each language, compared with the baselines, would make the distribution of gains and regressions clearer.

  • The experiments cover several model families, but only at relatively small scales. Testing models in the 7B–14B range, while reporting the additional computational cost, would help establish whether the gains hold at larger scales.

  • The main text gives more space to the theoretical motivation and method description than to analyzing the experimental evidence. Although the appendix contains useful ablations and additional results, bringing the key empirical analyses into the main text and condensing some derivations would make the connection between the theory and observed behavior easier to assess.

Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r3fU

We thank the reviewer for the constructive review and for recognizing the unified design.

W1. Isolating gradient-based grouping and its updates

Table 6 (Qwen3-4B; same 50K budget and recipe for every variant) covers part of this. Its caption describes two variants imprecisely, which we will fix: in no_clustering the gradient clusters are computed once at initialization and then frozen, and in no_scheduling the next family is chosen at random.

VariantAvg
Uniform (no grouping)61.50
no_scheduling: gradient clusters, family chosen at random69.74
no_clustering: gradient clusters fixed at initialization70.58
Full AMIGO: clusters re-estimated every regime73.32

Freezing the gradient-based clusters costs 2.74 points, so updating them during training matters, and choosing the family at random costs 3.58. Clusters are formed by the same automatic procedure throughout (Appendix A). Fixed random groups matched in K are not in the paper; we agree they would complete the picture by isolating the value of the gradient signal itself.

W2. Final-layer MLP gradients

We restrict gradient computation to the final MLP for two reasons. First, language-specific neurons in decoder-only LLMs are concentrated in the top and bottom layers (Tang et al., 2024), while late layers map the shared latent space back to the output language (Wendler et al., 2024). Second, this choice reduces computational overhead, as the final MLP comprises only 75M of the 4.0B parameters (less than 2%). Following prior work, we use the final layer for gradient computation. However, the choice of which layer to use is a design decision that can be adapted based on available computational resources and is beyond the scope of this paper. Eq. (1) decomposes over parameter blocks, so the final-MLP inner product is one additive block of the full-model alignment, and decisions based on it beat random ones (W1). We have not compared layer subsets on downstream performance and cost; we will state the restriction as a limitation and name larger subsets as a natural extension.

W3. Per-language gains and regressions

Per-language scores (Qwen3-4B, Bactrian-X 50K; mean of the six benchmarks):

LangUniformPiKEGREATSRegMixMLP-MKC+AMIGO
ar57.264.866.154.252.972.1
bn45.856.958.337.638.463.4
de67.975.874.661.262.178.2
en70.279.180.462.861.482.8
fr67.074.976.259.860.578.4
hi51.261.259.846.344.867.0
ja60.668.169.555.155.868.8
ru64.472.871.460.160.973.6
sv62.470.772.156.857.573.2
zh68.478.775.661.157.775.4
Avg61.570.370.455.555.273.3

AMIGO improves on Uniform in every language and is best in 8 of 10. Its largest gains over the strongest baseline are in ar (+6.0), hi (+5.8) and bn (+5.1), and its largest gains over Uniform are in the two lower-resource languages, bn (+17.6) and hi (+15.8). It trails the best baseline in ja (−0.7 vs. GREATS) and zh (−3.3 vs. PiKE). We will add this table to the main text.

W4. Larger models

We have not run 7B to 14B models, and agree this is an important next step. On larger training budgets, the gains hold: on Aya at 100K and 200K samples, AMIGO keeps the best average (0.681 vs. 0.670; 0.702 vs. 0.675) and gains the most from the extra data (+0.021; no baseline gains more than 0.005; Table 10).

W5. Theory vs. empirical analysis in the main text

We agree. In the revision we will condense Section 3 and the derivations, and move the ablations (Table 6), sensitivity analyses (Tables 7 to 9) and per-language results into the main text, so the link between the theory and the observed behavior is easier to assess.

Manuscript Revision: We have revised the manuscript to address the reviewers' suggestions. Specifically, we have added a discussion on final-layer selection in Section 4.1 (Gradient Estimation) in response to W2, included the ``Per-language Gains and Regressions'' analysis in Appendix E.1 (W3), added additional computational cost details in Section 5.2 (Computational Overhead), and moved the ablation experiments to the main text (Table 2, Section 5.2) (W5).

Dear Reviewer,

As the rebuttal period is nearing its end, we would be happy to provide any further clarifications that may help address your remaining concerns.

If our rebuttal has resolved your concerns, we would sincerely appreciate your reconsideration of the overall rating in light of the additional information provided. we have now updated the manuscript incorporating your suggestions. We have also added a "Manuscript Revision" point in the rebuttal above, indicating the sections we updated in the main text of our paper.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer r3fU: you raised many insightful points. Has the authors addressed them?

Comments on the paper

2 comments

Reviewers: please respond to the authors, as they have spent substantial effort in replying to all five of you.

AuthorsOpenReview ↗

Review-Response Summary

We would like to sincerely thank all the reviewers for their valuable time, thoughtful feedback, and efforts in helping us improve our manuscript. We are particularly grateful to the AC for their continued dedication to facilitating a constructive discussion throughout the review process.

During the discussion period, we were fortunate to engage with Reviewer AADU, who reconsidered their evaluation and raised their score from 2 to 3 following our clarifications. Reviewer 4bb1 also highlighted that the manuscript had not been updated to reflect certain clarifications. We apologize for this oversight, which we were previously unaware of, and have now updated the manuscript accordingly.

For completeness, we summarize below the major concerns raised during the review process and the clarifications and revisions we have provided:

1. Ablation on grouping strategies: Fixed vs. Random vs. AMIGO (r3FU W1; Bn4C W2)

We addressed this concern in our responses titled "Isolating Gradient-Based Grouping and Its Updates" (Reviewer r3FU) and "Evidence Tied to Gradient Geometry" (Reviewer Bn4C). The relevant results are already presented in Tables 6 and 7 of the manuscript. We have additionally incorporated clarifications into the revised manuscript to make these findings and their implications more explicit.

2. Choice of the final layer (r3FU W2; AADU W2; Bn4C Q3)

We have addressed this question in our responses to the respective reviewers. In particular, we provided further clarification to Reviewer AADU under "Additional Clarification on Final Layer Gradients," explaining why gradients from the final layer demonstrate stronger cross-lingual transfer in our setting. We would also like to clarify that the choice of layer is an efficiency-oriented design decision rather than a central contribution of our work.

3. Per-language performance (r3FU W3)

We provided per-language accuracy results, including gains and regressions across languages, in our response titled "W3: Per-Language Gains and Regressions." These results offer a more detailed understanding of AMIGO's performance across different languages.

4. Evaluation languages and scoring metrics (AADU Q2; Bn4C Q4; ZSTn W1; 4bb1 W1)

We provided detailed clarifications regarding our selection of evaluation languages, which was intended to cover diverse language groups while including both high- and low-resource languages. We also clarified our evaluation protocol and scoring metric, namely answer string-matching accuracy, a standard metric for the datasets considered.

Regarding the remaining questions concerning additional datasets and baselines, AMIGO already includes comparisons against five baseline approaches for multilingual training and is evaluated across six tasks and ten languages to assess the effectiveness and generalizability of our approach. Furthermore, following the reviewers' suggestions, we have moved the component ablations and regime-shift threshold analysis from the supplementary ablation section into the main manuscript, making these analyses more accessible. Our manuscript also includes systematic ablations of the individual components and hyperparameters of our method.

Once again, we sincerely thank all the reviewers and the AC for their valuable time, constructive feedback, and dedication throughout the review process. Their comments have helped us improve the clarity, presentation, and overall quality of our manuscript. We are grateful for the opportunity to address these concerns and hope that our clarifications and revisions adequately resolve the remaining questions.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.