- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AMIGO, an adaptive multilingual instruction tuning framework that jointly adapts language grouping, training order, and budget allocation. The method estimates cross-lingual similarity from validation gradients, selects the language family to train in the next regime via a compatibility score, and allocates the training budget using a continuous knapsack objective. Experiments on three model families show that the method improves average benchmark performance.
Critical strengths
-
The paper addresses an important optimization problem. Cross-lingual interactions evolve throughout training, so static data allocation may be suboptimal.
-
The framework connects language grouping, training order, and budget allocation through shared gradient information, which makes the overall design coherent.
-
The experimental coverage is good, including multiple baselines, component ablations, and additional generalization experiments on the Aya dataset.
Critical weaknesses
-
The evaluation protocol appears to measure output-format compliance rather than multilingual capability, as several base models score well below random chance.
-
Some baselines are missing. GradSim (Wang et al., 2023) also groups languages by gradient similarity and is the most directly related prior work, yet it is not included in the comparison.
-
Some design choices contradict the paper’s own premise that gradient geometry evolves throughout training and must be continually re-estimated. For example, the transition term T uses the gradient direction stored at the previous regime boundary, which is already stale by the time it is used according to the paper’s own drift criterion.
-
The claims about instruction following go beyond what the evaluation supports. The six benchmarks mainly test understanding, knowledge and reasoning. To support conclusions about the effectiveness of multilingual instruction tuning, the authors should add evaluations that directly measure instruction-constraint following, such as IFEval (Zhou et al., 2023), FollowBench (Jiang et al., 2024) and M-IFEval (Dussolle et al., 2025).
-
The main text reports statistics from only one experiment.
Questions for authors
-
Describe the evaluation protocol in detail, including the language versions of each benchmark, the prompt templates, and other settings.
-
With 7B+ models and larger SFT budgets, all methods learn the output format quickly. Does AMIGO’s advantage still hold in this setting?
-
How often are the gradients for drift detection recomputed? What share of the total wall-clock time goes to gradient estimation, family scoring and training, respectively?
-
See the weaknesses above.
3 comments
The manuscript has not been revised, and the main text still presents only one set of experimental results. Therefore, I will maintain my original rating.
Manuscript revised.
Dear Reviewer,
We were unaware of the new feature of updating the manuscript during the rebuttal. Thank you for pointing this out, we have now updated the manuscript incorporating your suggestions. We have also added a "Manuscript Revision" point in the rebuttal above, indicating the sections we updated in the main text of our paper.
Response to Reviewer 4bb1
We thank the reviewer for the clear review and for finding the design coherent and the experimental coverage good.
W1 / Q1. Evaluation protocol
W2. GradSim
GradSim trains a separate model per language group, so it does not map directly onto our single-model, fixed-budget setting. Its central idea, grouping languages once by gradient similarity, is tested by
no_clustering(Table 6). In that variant, clusters are computed from gradients once at initialization and kept fixed, with scheduling and budget allocation unchanged. Re-estimating the clusters at every regime raises the average from 70.58 to 73.32 and wins on 4 of 6 benchmarks. We will rename the variantstatic_clusters, describe it as GradSim-style static grouping, and discuss GradSim explicitly in Section 2.W3. Transition term T At each regime boundary, we recompute language-family gradients and compare them with stored gradients (Eq. 5) to capture drift and emerging language interactions, potentially forming new families as motivated in our paper. Since gradients are already recomputed, the current-parameter variant incurs no additional cost. Its impact is limited because the current-parameter transfer gain D carries most of the weight (1−α=0.7), while the stored direction T still contributes meaningfully: removing it reduces the average from 0.733 to 0.714 (Table 7).
W4. Instruction-following claims
We agree that our claims go beyond what the evaluation supports. Our evaluation is generative: in each of 10 languages, the model must follow the task instruction, answer in the requested form, and be correct and more focused towards multilinguality. This captures answer-format following, but not constraint following as measured by IFEval, FollowBench or M-IFEval. We will make the scope more clear in our manuscript.
W5. Statistics in the main text
Beyond Table 1, the paper reports ablations (Table 6), sensitivity to α, β and τ_rel (Tables 7–9), Aya at two budgets (Table 10) and per-cell SDs over 3 seeds (Tables 3–5), but most of this sits in the appendix. We will move the ablation to main text and condense section 3 and 4 to make room.
Q2. Larger models and SFT budgets
We have not run 7B+ models, and agree this is an important next step. On larger SFT budgets, Aya gives direct evidence in a setting where every method is above chance. AMIGO has the best average at 100K (0.681 vs. 0.670) and at 200K (0.702 vs. 0.675). It is also the only method that gains from the extra data: +0.021, while no baseline gains more than 0.005 (Table 10). The margin over the best baseline grows from 0.011 to 0.027 as the budget doubles, so we see no sign of it shrinking once the format is learned.
Q3. Recompute frequency and wall-clock
Validation gradients for all 23 languages are recomputed every 50 steps to evaluate the drift statistic. At each regime boundary (21 in the default run), the same gradient pass also feeds clustering, family scoring and allocation. On Qwen3-4B (one H100, 50K),. All AMIGO-specific work totals 21% of wall-clock, and training takes the remaining 79%. comparison with other methods are discussed in main paper
Manuscript Revision: We have revised the manuscript to address the reviewers' suggestions. Specifically, we have updated Section 4.4 (Compatibility) in response to W3, clarified the evaluation protocol in the Evaluations paragraph of Section 5.1, moved the ablation paragraph to the main text (Table 2), renamed \texttt to \texttt and discussed it as a Grad-Sim variant in the ablation paragraph of Section 5.2, and added wall-clock time to the compute overhead paragraph in Section 5.2.