- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MACE, a gradient-based framework for multilingual instruction data selection that integrates language-specific gradient affinity with cross-lingual conflict awareness. Specifically, MACE uses the proposed MGCR to quantify cross-lingual gradient conflicts and incorporates this information into a pre-selection stage preceding KMC or LESS. Experiments with Qwen2.5-7B and Llama-3.1-8B across five multilingual benchmarks demonstrate consistent performance gains under a fixed 5% data budget. The paper further investigates the contributions of individual components, compares alternative conflict metrics, and provides a local optimization perspective to motivate MGCR. Overall, the work addresses a practically relevant problem in multilingual adaptation and offers a potentially useful strategy for improving data efficiency in instruction tuning.
Critical strengths
-
MACE is clearly motivated, and the framework is intuitive. It fits naturally into existing data selection pipelines.
-
The paper reports per-language results, which help identify the gains and degradations that matter in real deployment.
-
The experiments include useful controls: affinity-only selection, a single prototype versus multiple gradient prototypes, and comparisons across alternative conflict measures.
Critical weaknesses
-
The paper motivates multilingual instruction tuning as a way to improve LLMs’ ability to follow instructions across languages, yet all evaluation benchmarks are classification, extractive QA, or summarization tasks, and none directly measures instruction following (e.g., IFEval or AlpacaEval and their multilingual variants).
-
While MACE itself is described in detail, the baseline configurations are underspecified, including the representation used by KMC, the target data and scoring configuration of LESS, and the specific settings of LangGPS, which weakens the comparative conclusions. For example, on Qwen in Table 1, LangGPS + KMC, with a score of 60.824, even underperforms KMC alone, with a score of 61.145. Please explain the reasons for this result.
-
The hyperparameter selection procedure is unclear. The main setting, and , corresponds exactly to the highest five-benchmark average in the Qwen sweeps (Tables 8–9), and the paper does not state whether these values were chosen on an independent development set or on the reported evaluation benchmarks. Moreover, the sweeps are run only on Qwen with KMC, and the same settings are applied directly to Llama and to LESS.
-
The existing ablations do not clearly isolate the contribution of each component, and the generality of the method rests largely on a single, fairly specific selection setup.
Questions for authors
-
How were , , and chosen? Was any validation data held out from the five evaluation benchmarks?
-
LESS needs gradients from a target validation set. Which target data did you use for LESS?
-
See the weaknesses above.
4 comments
I thank the authors for their response. I will raise my score to 3.
Thank you for raising your score and for your thoughtful consideration of our responses. Your constructive feedback helped us strengthen the evaluation and clarify the experimental setup. We sincerely appreciate your time and support.
Thank you for your comments.
\textbf{1. Direct multilingual instruction-following evaluation}\\\\\text{(Weakness 1)} \end{array}$$ We evaluated M-IFEval in French, Japanese, and Spanish using the official evaluator. On Qwen2.5-7B with training seed 13, the results are: $$\begin{array}{lrrrr} \hline \textbf{Method} & \textbf{Strict prompt} & \begin{array}{l}\textbf{Strict}\\\\\textbf{instruction}\end{array} & \textbf{Loose prompt} & \begin{array}{l}\textbf{Loose}\\\\\textbf{instruction}\end{array} \\\\ \hline \text{KMC} & 17.82 & 18.23 & 19.06 & 19.25 \\\\ \text{LESS} & 18.73 & 18.41 & 19.35 & 19.18 \\\\ \text{LangGPS + KMC} & 19.94 & 19.73 & 20.99 & 20.85 \\\\ \text{MACE + KMC} & \mathbf{22.94} & \mathbf{23.03} & \mathbf{23.81} & \mathbf{24.00} \\\\ \hline \end{array}$$ Scores are percentages, macro-averaged equally over the three languages. The benchmark contains 235 French, 172 Japanese, and 115 Spanish prompts. Prompt-level accuracy requires satisfying all constraints in a prompt; instruction-level accuracy measures satisfaction of individual constraints. MACE + KMC improves strict prompt-level and instruction-level accuracy over LangGPS + KMC by 3.01 and 3.30 percentage points, respectively, with improvements on both strict metrics in each of the three languages. It also exceeds KMC by 5.12 and 4.80 percentage points. $$\begin{array}{l} \textbf{2. Baseline representations and target data}\\\\\text{(Weakness 2 \\& Question 2)} \end{array}$$ KMC follows the encoder choice in LangGPS's KMC baseline. The selection representations and target data are: $$\begin{array}{lll} \hline \textbf{Baseline} & \begin{array}{l}\textbf{Representation or selection}\\\\\textbf{signal}\end{array} & \textbf{Implementation detail} \\\\ \hline \text{KMC} & \begin{array}{l}\text{all-MiniLM-L6-v2 text}\\\\\text{embeddings}\end{array} & \text{Embedding-space clustering} \\\\ \text{LangGPS} & \begin{array}{l}\text{Backbone last-token hidden}\\\\\text{states}\end{array} & \begin{array}{l}\text{Authors' official source}\\\\\text{code}\end{array} \\\\ \text{LESS} & \text{Gradient-based target matching} & \text{mMMLU target dataset} \\\\ \hline \end{array}$$ All methods use the LoRA training and benchmark-evaluation configuration described in the manuscript. $$\begin{array}{l} \textbf{3. Why LangGPS + KMC is below KMC on Qwen}\\\\\text{(Weakness 2)} \end{array}$$ The reported difference is 60.824 - 61.145 = -0.321 average-score points. The task-level mean differences, computed before display rounding, are: $$\begin{array}{lr} \hline \textbf{Benchmark} & \textbf{LangGPS + KMC minus KMC} \\\\ \hline \text{XNLI} & -0.427 \\\\ \text{XStoryCloze} & -1.870 \\\\ \text{mMMLU} & +0.020 \\\\ \text{XQuAD} & +0.050 \\\\ \text{XLSum} & +0.620 \\\\ \hline \end{array}$$ LangGPS scores language separability in representation space. A plausible reason for the task differences is that this filtering changes the task coverage and transfer-relevant examples available to KMC. Language separability and gradient compatibility describe different properties of an example. MACE scores affinity to local gradient references and conflict with other languages. $$\begin{array}{l} \textbf{4. Component comparisons and additional evaluations}\\\\\text{(Weakness 4)} \end{array}$$ The component experiments are: $$\begin{array}{lll} \hline \textbf{Control} & \textbf{Design choice examined} & \textbf{Comparison} \\\\ \hline \text{Affinity-only} & \text{Cross-language exposure} & \begin{array}{l}\text{Affinity vs. affinity plus}\\\\\text{exposure}\end{array} \\\\ \text{Prototype count} & \text{Local reference diversity} & \text{One vs. multiple prototypes} \\\\ \begin{array}{l}\text{Alternative conflict}\\\\\text{measures}\end{array} & \text{Exposure definition} & \begin{array}{l}\text{MGCR vs. cosine, sign, negative}\\\\\text{dot, and Residual}\end{array} \\\\ \hline \end{array}$$ The conflict-measure comparison uses matched runs. Seed 1-3 label the three matched runs within each backbone. Delta denotes MGCR minus Residual; the four-task average excludes XStoryCloze.