acceptodds
Under review as a conference paper at ICLR 2027

MACE: Affinity and Conflict-Aware Data Selection for Multilingual Instruction Tuning

Abstract

Multilingual instruction tuning improves the ability of large language models to follow instructions across languages. The training mixture shapes both the supervision available to each language and interactions between languages. Data selection is therefore central to effective multilingual training. Through shared parameters, however, an example aligned with learning directions in its own language may induce updates that oppose those of other languages. Selection requires reliable references for diverse within-language directions and an explicit assessment of cross-language opposition. We introduce MACE (Multilingual Affinity and Conflict-aware Example Selection), a gradient-based data selection framework addressing both requirements. MACE constructs multiple gradient prototypes for each language and measures a candidate's in-language affinity through alignment with its closest prototype. It assesses cross-language conflict exposure using the Magnitude-weighted Gradient Conflict Ratio (MGCR), which measures the share of gradient-product magnitude contributed by opposing coordinates. Combining affinity with an exposure penalty yields a score for ranking and retaining candidates within each language. Across five multilingual benchmarks, MACE + KMC improves the average score over LangGPS + KMC by a relative 1.99% on Qwen2.5-7B and 4.59% on Llama-3.1-8B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multilingual Instruction Tuning, Data Selection, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DfJSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MACE, a gradient-based framework for multilingual instruction data selection that integrates language-specific gradient affinity with cross-lingual conflict awareness. Specifically, MACE uses the proposed MGCR to quantify cross-lingual gradient conflicts and incorporates this information into a pre-selection stage preceding KMC or LESS. Experiments with Qwen2.5-7B and Llama-3.1-8B across five multilingual benchmarks demonstrate consistent performance gains under a fixed 5% data budget. The paper further investigates the contributions of individual components, compares alternative conflict metrics, and provides a local optimization perspective to motivate MGCR. Overall, the work addresses a practically relevant problem in multilingual adaptation and offers a potentially useful strategy for improving data efficiency in instruction tuning.

Critical strengths
  1. MACE is clearly motivated, and the framework is intuitive. It fits naturally into existing data selection pipelines.

  2. The paper reports per-language results, which help identify the gains and degradations that matter in real deployment.

  3. The experiments include useful controls: affinity-only selection, a single prototype versus multiple gradient prototypes, and comparisons across alternative conflict measures.

Critical weaknesses
  1. The paper motivates multilingual instruction tuning as a way to improve LLMs’ ability to follow instructions across languages, yet all evaluation benchmarks are classification, extractive QA, or summarization tasks, and none directly measures instruction following (e.g., IFEval or AlpacaEval and their multilingual variants).

  2. While MACE itself is described in detail, the baseline configurations are underspecified, including the representation used by KMC, the target data and scoring configuration of LESS, and the specific settings of LangGPS, which weakens the comparative conclusions. For example, on Qwen in Table 1, LangGPS + KMC, with a score of 60.824, even underperforms KMC alone, with a score of 61.145. Please explain the reasons for this result.

  3. The hyperparameter selection procedure is unclear. The main setting, K=4K = 4 and λ=2.5\lambda = 2.5, corresponds exactly to the highest five-benchmark average in the Qwen sweeps (Tables 8–9), and the paper does not state whether these values were chosen on an independent development set or on the reported evaluation benchmarks. Moreover, the sweeps are run only on Qwen with KMC, and the same settings are applied directly to Llama and to LESS.

  4. The existing ablations do not clearly isolate the contribution of each component, and the generality of the method rests largely on a single, fairly specific selection setup.

Questions for authors
  1. How were λ\lambda, KK, DD and mm chosen? Was any validation data held out from the five evaluation benchmarks?

  2. LESS needs gradients from a target validation set. Which target data did you use for LESS?

  3. See the weaknesses above.

4 comments

Thank you for your comments.

\textbf{1. Direct multilingual instruction-following evaluation}\\\\\text{(Weakness 1)} \end{array}$$ We evaluated M-IFEval in French, Japanese, and Spanish using the official evaluator. On Qwen2.5-7B with training seed 13, the results are: $$\begin{array}{lrrrr} \hline \textbf{Method} & \textbf{Strict prompt} & \begin{array}{l}\textbf{Strict}\\\\\textbf{instruction}\end{array} & \textbf{Loose prompt} & \begin{array}{l}\textbf{Loose}\\\\\textbf{instruction}\end{array} \\\\ \hline \text{KMC} & 17.82 & 18.23 & 19.06 & 19.25 \\\\ \text{LESS} & 18.73 & 18.41 & 19.35 & 19.18 \\\\ \text{LangGPS + KMC} & 19.94 & 19.73 & 20.99 & 20.85 \\\\ \text{MACE + KMC} & \mathbf{22.94} & \mathbf{23.03} & \mathbf{23.81} & \mathbf{24.00} \\\\ \hline \end{array}$$ Scores are percentages, macro-averaged equally over the three languages. The benchmark contains 235 French, 172 Japanese, and 115 Spanish prompts. Prompt-level accuracy requires satisfying all constraints in a prompt; instruction-level accuracy measures satisfaction of individual constraints. MACE + KMC improves strict prompt-level and instruction-level accuracy over LangGPS + KMC by 3.01 and 3.30 percentage points, respectively, with improvements on both strict metrics in each of the three languages. It also exceeds KMC by 5.12 and 4.80 percentage points. $$\begin{array}{l} \textbf{2. Baseline representations and target data}\\\\\text{(Weakness 2 \\& Question 2)} \end{array}$$ KMC follows the encoder choice in LangGPS's KMC baseline. The selection representations and target data are: $$\begin{array}{lll} \hline \textbf{Baseline} & \begin{array}{l}\textbf{Representation or selection}\\\\\textbf{signal}\end{array} & \textbf{Implementation detail} \\\\ \hline \text{KMC} & \begin{array}{l}\text{all-MiniLM-L6-v2 text}\\\\\text{embeddings}\end{array} & \text{Embedding-space clustering} \\\\ \text{LangGPS} & \begin{array}{l}\text{Backbone last-token hidden}\\\\\text{states}\end{array} & \begin{array}{l}\text{Authors' official source}\\\\\text{code}\end{array} \\\\ \text{LESS} & \text{Gradient-based target matching} & \text{mMMLU target dataset} \\\\ \hline \end{array}$$ All methods use the LoRA training and benchmark-evaluation configuration described in the manuscript. $$\begin{array}{l} \textbf{3. Why LangGPS + KMC is below KMC on Qwen}\\\\\text{(Weakness 2)} \end{array}$$ The reported difference is 60.824 - 61.145 = -0.321 average-score points. The task-level mean differences, computed before display rounding, are: $$\begin{array}{lr} \hline \textbf{Benchmark} & \textbf{LangGPS + KMC minus KMC} \\\\ \hline \text{XNLI} & -0.427 \\\\ \text{XStoryCloze} & -1.870 \\\\ \text{mMMLU} & +0.020 \\\\ \text{XQuAD} & +0.050 \\\\ \text{XLSum} & +0.620 \\\\ \hline \end{array}$$ LangGPS scores language separability in representation space. A plausible reason for the task differences is that this filtering changes the task coverage and transfer-relevant examples available to KMC. Language separability and gradient compatibility describe different properties of an example. MACE scores affinity to local gradient references and conflict with other languages. $$\begin{array}{l} \textbf{4. Component comparisons and additional evaluations}\\\\\text{(Weakness 4)} \end{array}$$ The component experiments are: $$\begin{array}{lll} \hline \textbf{Control} & \textbf{Design choice examined} & \textbf{Comparison} \\\\ \hline \text{Affinity-only} & \text{Cross-language exposure} & \begin{array}{l}\text{Affinity vs. affinity plus}\\\\\text{exposure}\end{array} \\\\ \text{Prototype count} & \text{Local reference diversity} & \text{One vs. multiple prototypes} \\\\ \begin{array}{l}\text{Alternative conflict}\\\\\text{measures}\end{array} & \text{Exposure definition} & \begin{array}{l}\text{MGCR vs. cosine, sign, negative}\\\\\text{dot, and Residual}\end{array} \\\\ \hline \end{array}$$ The conflict-measure comparison uses matched runs. Seed 1-3 label the three matched runs within each backbone. Delta denotes MGCR minus Residual; the four-task average excludes XStoryCloze.
\textbf{4. Component comparisons and additional evaluations (continued)}\\\\\text{(Weakness 4)} \end{array}$$ Qwen2.5-7B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 61.840 & 62.266 & +0.426 & +0.0900 \\\\ \text{Seed 2} & 62.088 & 61.806 & -0.282 & -0.3550 \\\\ \text{Seed 3} & 61.604 & 62.036 & +0.432 & +0.2725 \\\\ \text{Mean} & 61.844 & 62.036 & +0.192 & +0.0025 \\\\ \hline \end{array}$$ Llama-3.1-8B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 46.868 & 47.994 & +1.126 & +0.9075 \\\\ \text{Seed 2} & 47.100 & 47.536 & +0.436 & +0.4475 \\\\ \text{Seed 3} & 46.642 & 48.044 & +1.402 & +1.4950 \\\\ \text{Mean} & 46.870 & 47.858 & +0.988 & +0.9500 \\\\ \hline \end{array}$$ The five-task mean gains are +0.192 on Qwen and +0.988 on Llama, with matched-run wins of 2/3 and 3/3. After omitting XStoryCloze, the win counts remain 2/3 and 3/3, and the mean differences are +0.0025 and +0.9500. On Qwen, MACE + KMC also obtains 61.394 $\pm$ 0.059 at a 1% budget and 61.672 $\pm$ 0.382 at a 3% budget (three-run mean $\pm$ sample SD).

I thank the authors for their response. I will raise my score to 3.

Thank you for raising your score and for your thoughtful consideration of our responses. Your constructive feedback helped us strengthen the evaluation and clarify the experimental setup. We sincerely appreciate your time and support.

Reviewer_x7cFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MACE (Multilingual Affinity and Conflict-aware Example Selection), a gradient-based data selection framework addressing both requirements. MACE constructs multiple gradient prototypes for each language and measures a candidate’s in-language affinity through alignment with its closest prototype. It assesses cross-language conflict exposure using the Magnitude-weighted Gradient Conflict Ratio (MGCR), which measures the share of gradient-product magnitude contributed by opposing coordinates.

Critical strengths
  1. Selecting multilingual SFT data with cross-language interference is an interesting research question, and the paper frames the question and approach pretty well.
  2. The proposed method is simple and flexible, with good empirical results.
  3. The experiments and reports are detailed, including three runs and useful ablation studies.
  4. This paper also tries to explain the method from an optimization perspective.
Critical weaknesses
  1. The connection between theory and the selection rules is not very clear. The terms "conflict" and "interference risk" are not well supported (or are not straightforward to understand) in empirical studies.
  2. There is no gradient-based multilingual baseline to compare. The latest baseline was published in 2024.
  3. In Table 1, on Llama 3.1, XStoryCloze's performance improved by more than 10 points, far more than on other tasks, showing the gains may concentrate in one benchmark.
  4. The experiment scale is somewhat limited. A small training set (5%) and LoRA are studied, which limits the generality. Experiments may consider an even smaller and/or larger budget or full tuning.
  5. No code or selected-subset release is mentioned in the Reproducibility Statement. Without these, it would be hard to reproduce the results.
  6. Compute efficiency (Appendix 8) is estimated but not measured.
Questions for authors
  1. How were baselines like LangGPS implemented to ensure fair comparison?
  2. Is "Full (100%)" one run or a three-run mean? Why are SDs omitted?
5 comments

Thank you for your comments.

\textbf{1. Connection between conflict exposure and actual interference}\\\\\text{(Weakness 1)} \end{array}$$ We conducted independent single-example updates for 512 candidates from English, Chinese, Spanish, and Arabic, and measured held-out loss changes in the other three languages. Each step size yields 1,536 candidate-reference pairs. The FP32 run passed numerical gradient/update/loss consistency checks. All-pair AUROC: $$\begin{array}{lrrr} \hline \textbf{Method} & 5\times 10^{-5} & 1\times 10^{-4} & 2\times 10^{-4} \\\\ \hline \text{MGCR} & \mathbf{0.9525} & \mathbf{0.9511} & \mathbf{0.9559} \\\\ \text{Residual} & 0.9407 & 0.9396 & 0.9442 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.9491 & 0.9476 & 0.9533 \\\\ \text{Signed cosine, full gradient} & 0.9313 & 0.9297 & 0.9409 \\\\ \hline \end{array}$$ AUROC on the 1,383 pairs with positive full-gradient alignment: $$\begin{array}{lrrr} \hline \textbf{Method} & 5\times 10^{-5} & 1\times 10^{-4} & 2\times 10^{-4} \\\\ \hline \text{MGCR} & \mathbf{0.9266} & \mathbf{0.9252} & \mathbf{0.9357} \\\\ \text{Residual} & 0.9217 & 0.9202 & 0.9344 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.9214 & 0.9198 & 0.9337 \\\\ \text{Signed cosine, full gradient} & 0.8711 & 0.8683 & 0.8980 \\\\ \hline \end{array}$$ At $1\times 10^{-4}$, a paired candidate-cluster bootstrap with 3,000 source-language-stratified resamples gives a full-set MGCR-minus-Residual AUROC difference of +0.0115, with a pointwise 95% interval of [+0.0070, +0.0161]. The accompanying gradient decomposition gives the following distributions, where S is selected-coordinate interaction magnitude and T is the omitted dot-product contribution: $$\begin{array}{lrrr} \hline \textbf{Quantity} & \textbf{5th percentile} & \textbf{Median} & \begin{array}{l}\textbf{95th}\\\\\textbf{percentile}\end{array} \\\\ \hline \begin{array}{l}\text{Selected-coordinate interaction}\\\\\text{magnitude S}\end{array} & 0.208 & 1.187 & 9.518 \\\\ \begin{array}{l}\text{Absolute omitted contribution}\\\\\text{relative to S, abs(T)/S}\end{array} & 0.059 & 0.535 & 0.807 \\\\ \hline \end{array}$$ $$\begin{array}{llr} \hline \textbf{Comparison} & \textbf{Statistic} & \textbf{Value} \\\\ \hline \begin{array}{l}\text{Selected-coordinate dot vs.}\\\\\text{full-gradient dot}\end{array} & \text{Spearman} & 0.9784 \\\\ \begin{array}{l}\text{Selected-coordinate vs.}\\\\\text{full-gradient dot signs}\end{array} & \text{Different signs} & \text{43/1,536 (2.80\\%)} \\\\ \hline \end{array}$$ MGCR computes opposing coordinate-product mass before net aggregation. Cosine and Residual depend on net alignment; MGCR explicitly records the opposing share alongside coexisting agreement. For continuous loss-change ranking at $1\times 10^{-4}$, the Spearman results are: $$\begin{array}{lrr} \hline \textbf{Method} & \textbf{All pairs} & \textbf{Positive full-dot subset} \\\\ \hline \text{MGCR} & 0.3834 & 0.2254 \\\\ \text{Residual} & 0.3774 & 0.2224 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.3805 & 0.2224 \\\\ \text{Signed cosine, full gradient} & 0.5411 & 0.4395 \\\\ \hline \end{array}$$ $$\begin{array}{l} \textbf{2. Baseline coverage}\\\\\text{(Weakness 2)} \end{array}$$ Our comparisons cover random selection, embedding-space diversity through KMC, lexical diversity through MTLD, instruction difficulty through IFD, gradient-based targeted selection through LESS, and LangGPS, which is specifically designed for multilingual pre-selection. MACE combines local in-language gradient references with cross-language conflict exposure before KMC or LESS refinement. $$\begin{array}{l} \textbf{3. Whether improvements depend on XStoryCloze}\\\\\text{(Weakness 3)} \end{array}$$ After excluding XStoryCloze, the equally weighted four-benchmark mean gains are: $$\begin{array}{lrr} \hline \textbf{Paired comparison} & \textbf{Qwen2.5-7B} & \textbf{Llama-3.1-8B} \\\\ \hline \text{MACE + KMC vs. KMC} & +0.676 & +0.740 \\\\ \text{MACE + LESS vs. LESS} & +0.615 & +0.538 \\\\ \text{MACE + KMC vs. LangGPS + KMC} & +0.610 & +0.775 \\\\ \hline \end{array}$$ For the MGCR-versus-Residual ablation, the matched-run results are below. Seed 1-3 label the three matched runs within each backbone. Delta denotes MGCR minus Residual; the four-task average excludes XStoryCloze.
\textbf{3. Whether improvements depend on XStoryCloze (continued)}\\\\\text{(Weakness 3)} \end{array}$$ Qwen2.5-7B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 61.840 & 62.266 & +0.426 & +0.0900 \\\\ \text{Seed 2} & 62.088 & 61.806 & -0.282 & -0.3550 \\\\ \text{Seed 3} & 61.604 & 62.036 & +0.432 & +0.2725 \\\\ \text{Mean} & 61.844 & 62.036 & +0.192 & +0.0025 \\\\ \hline \end{array}$$ Llama-3.1-8B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 46.868 & 47.994 & +1.126 & +0.9075 \\\\ \text{Seed 2} & 47.100 & 47.536 & +0.436 & +0.4475 \\\\ \text{Seed 3} & 46.642 & 48.044 & +1.402 & +1.4950 \\\\ \text{Mean} & 46.870 & 47.858 & +0.988 & +0.9500 \\\\ \hline \end{array}$$ MGCR wins 2/3 matched runs on Qwen and 3/3 on Llama under both averaging schemes. The four-task mean differences are +0.0025 and +0.9500 points, respectively. $$\begin{array}{l} \textbf{4. Data-budget coverage}\\\\\text{(Weakness 4)} \end{array}$$ We evaluated MACE + KMC on Qwen2.5-7B at 1% and 3% final data budgets, with three runs per setting: $$\begin{array}{lr} \hline \textbf{Final budget} & \begin{array}{l}\textbf{Five-benchmark average, mean +/-}\\\\\textbf{sample SD}\end{array} \\\\ \hline 1\\% & 61.394 \pm 0.059 \\\\ 3\\% & 61.672 \pm 0.382 \\\\ 5\\% & 62.036 \pm 0.230 \\\\ \hline \end{array}$$ Reducing the selected training set from 5% to 1% uses 80% fewer examples, with a 0.642-point decrease in average score. $$\begin{array}{l} \textbf{5. Code release}\\\\\text{(Weakness 5)} \end{array}$$ We will publicly release the MACE code upon acceptance. $$\begin{array}{l} \textbf{6. Measured computational cost}\\\\\text{(Weakness 6)} \end{array}$$ On two NVIDIA GeForce RTX 4090 GPUs, MACE + LESS takes 35 hours compared with 64 hours for standalone LESS, a 45.3% reduction and a 1.83x speedup. For MACE + LESS, the reported time includes both MACE pre-selection and all LESS selection stages. $$\begin{array}{llrr} \hline \textbf{Selection pipeline} & \textbf{Hardware} & \begin{array}{l}\textbf{Measured wall}\\\\\textbf{time}\end{array} & \textbf{GPU-hours} \\\\ \hline \text{LESS} & \text{2 x RTX 4090} & \text{64 h} & 128 \\\\ \text{MACE + LESS} & \text{2 x RTX 4090} & \text{35 h} & 70 \\\\ \hline \end{array}$$ The downstream LESS selection stage accounts for most of the computational cost of the combined pipeline. These measured selection times replace the earlier MACE + LESS cost estimate. $$\begin{array}{l} \textbf{7. Baseline implementation}\\\\\text{(Question 1)} \end{array}$$ We used the authors' official source code for LangGPS. For KMC, we follow the encoder choice in LangGPS's KMC baseline and use all-MiniLM-L6-v2 embeddings. LangGPS obtains its pre-selection representations from the backbone model's last-token hidden states. LESS uses mMMLU as the target dataset for gradient-based selection. All methods in the main comparison use the shared LoRA training and benchmark-evaluation configuration described in the manuscript. $$\begin{array}{l} \textbf{8. Full-data reference across three runs}\\\\\text{(Question 2)} \end{array}$$ We expanded the Full (100%) reference from one to three runs for each backbone. The resulting five-benchmark averages are: $$\begin{array}{lr} \hline \textbf{Backbone} & \begin{array}{l}\textbf{Five-benchmark average, mean +/-}\\\\\textbf{sample SD}\end{array} \\\\ \hline \text{Qwen2.5-7B} & 58.648 \pm 0.084 \\\\ \text{Llama-3.1-8B} & 51.469 \pm 0.155 \\\\ \hline \end{array}$$ SD is computed across the three run-level five-benchmark averages. We will update the Full entries with three-run means and SDs for each benchmark and Avg.

Reviewer x7cF: please respond to the authors.

Reviewer_2wmZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes MACE, a data selection framework for multilingual instruction tuning. It represents each language with multiple gradient prototypes and scores candidates by their alignment with the closest prototype. It also introduces the MGCR to measure opposing coordinate contributions between candidate and reference-language gradients. MACE combines prototype affinity with a conflict penalty to rank examples within each language. The experiments use MACE to retain 25% of the candidate data, followed by KMC or LESS selection to reach a 5% final budget. Results on two models and five multilingual benchmarks show improvements over the corresponding selectors and LangGPS with the same KMC backend.

Critical strengths
  1. This paper presents a clear approach to balancing within-language diversity and cross-language compatibility during data selection.

  2. This paper provides reasonably comprehensive empirical evidence. The experiments cover two backbones, five benchmarks, and both KMC and LESS, with three-run means and standard deviations.

Critical weaknesses
  1. The link between MGCR and actual cross-language interference needs stronger validation. Opposing coordinate contributions can coexist with positive overall gradient alignment, so they do not necessarily imply harm to another language. Proposition 1 and Appendix D also depend on interaction magnitude, omitted coordinates, and curvature, which conflict-ratio ranking does not account for. The downstream gains support MGCR’s practical usefulness, but do not establish whether it reliably ranks candidates by their actual effects on other languages.

  2. MGCR’s advantage over closely related conflict measures remains unclear. Unlike cosine and normalized negative-dot penalties, Residual distinguishes cases with nonnegative net alignment, making it an important comparator. MGCR improves over Residual by 0.192 points on Qwen2.5-7B and 0.988 points on Llama-3.1-8B (Table 3). The Llama gain is encouraging, but the Qwen gain is small and largely comes from XStoryCloze (Table 10). Although three-run standard deviations are reported, paired per-run differences would help establish whether MGCR consistently outperforms Residual.

Questions for authors
  1. Does MGCR predict observed changes in held-out reference-language loss better than Residual or cosine alignment, particularly when net alignment is positive?

  2. How much do interaction magnitudes and omitted-coordinate contributions vary across candidates, and do they change the candidate ordering suggested by MGCR alone?

  3. How consistent are the gains over Residual across individual runs and when XStoryCloze is excluded from the average?

5 comments

Thank you for the constructive comments.

\textbf{1. Does MGCR predict observed reference-language loss changes,}\\\\\textbf{particularly when net alignment is positive?}\\\\\text{(Weakness 1 \\& Question 1)} \end{array}$$ We evaluated 512 candidates from English, Chinese, Spanish, and Arabic, with 128 candidates per language. For each candidate, we computed a gradient and applied an independent single-example SGD update from the same initial model state. We measured loss changes on held-out examples in each of the other three languages. This yields 1,536 candidate-reference pairs at each of three step sizes: $5\times 10^{-5}$, $1\times 10^{-4}$, and $2\times 10^{-4}$. The run used FP32 and passed the numerical gradient/update/loss consistency checks before evaluation. AUROC measures identification of loss-increasing updates, with positive labels defined by an observed increase in held-out reference-language loss. Spearman correlation measures ranking agreement with continuous loss changes. All-pair AUROC: $$\begin{array}{lrrr} \hline \textbf{Method} & 5\times 10^{-5} & 1\times 10^{-4} & 2\times 10^{-4} \\\\ \hline \text{MGCR} & \mathbf{0.9525} & \mathbf{0.9511} & \mathbf{0.9559} \\\\ \text{Residual} & 0.9407 & 0.9396 & 0.9442 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.9491 & 0.9476 & 0.9533 \\\\ \text{Signed cosine, full gradient} & 0.9313 & 0.9297 & 0.9409 \\\\ \hline \end{array}$$ AUROC on the positive full-gradient dot-product subset: $$\begin{array}{lrrr} \hline \textbf{Method} & 5\times 10^{-5} & 1\times 10^{-4} & 2\times 10^{-4} \\\\ \hline \text{MGCR} & \mathbf{0.9266} & \mathbf{0.9252} & \mathbf{0.9357} \\\\ \text{Residual} & 0.9217 & 0.9202 & 0.9344 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.9214 & 0.9198 & 0.9337 \\\\ \text{Signed cosine, full gradient} & 0.8711 & 0.8683 & 0.8980 \\\\ \hline \end{array}$$ The positive-alignment subset contains 1,383 pairs, of which 43, 42, and 37 increase reference-language loss at the respective step sizes. MGCR achieves the highest observed AUROC among the four measures at all three step sizes in both the full set and the positive-alignment subset. Spearman correlations with continuous loss change at $1\times 10^{-4}$: $$\begin{array}{lrr} \hline \textbf{Method} & \textbf{All pairs} & \textbf{Positive full-dot subset} \\\\ \hline \text{MGCR} & 0.3834 & 0.2254 \\\\ \text{Residual} & 0.3774 & 0.2224 \\\\ \begin{array}{l}\text{Signed cosine, selected}\\\\\text{coordinates}\end{array} & 0.3805 & 0.2224 \\\\ \text{Signed cosine, full gradient} & 0.5411 & 0.4395 \\\\ \hline \end{array}$$ We compared AUROC using 3,000 paired bootstrap resamples, stratified by source language. Each resample retains all reference-language observations for a sampled candidate. At $1\times 10^{-4}$, the full-set differences are: $$\begin{array}{lrl} \hline \textbf{Comparison} & \textbf{AUROC difference} & \begin{array}{l}\textbf{Pointwise 95\\% bootstrap}\\\\\textbf{interval}\end{array} \\\\ \hline \text{MGCR minus Residual} & +0.0115 & \text{[+0.0070, +0.0161]} \\\\ \begin{array}{l}\text{MGCR minus selected-coordinate}\\\\\text{signed cosine}\end{array} & +0.0035 & \text{[+0.0015, +0.0058]} \\\\ \begin{array}{l}\text{MGCR minus full-gradient signed}\\\\\text{cosine}\end{array} & +0.0213 & \text{[+0.0114, +0.0333]} \\\\ \hline \end{array}$$ MGCR measures opposing coordinate-product magnitude relative to the total on the selected coordinates. Cosine and Residual depend on net directional alignment, with Residual equal to $1-[\mathrm{cosine}]_+^2$. MGCR retains the opposing share even when net alignment is positive. $$\begin{array}{l} \textbf{2. How much do interaction magnitudes and omitted-coordinate}\\\\\textbf{contributions vary, and do they change ordering?}\\\\\text{(Weakness 1 \\& Question 2)} \end{array}$$ Let S be the absolute gradient-product mass on the selected coordinates and T the dot-product contribution from omitted coordinates. In Proposition 1, the full first-order loss-change predictor is proportional to $S(2C-1)-T$, where C is MGCR. The distributions across 1,536 candidate-reference pairs are: $$\begin{array}{lrrr} \hline \textbf{Quantity} & \textbf{5th percentile} & \textbf{Median} & \begin{array}{l}\textbf{95th}\\\\\textbf{percentile}\end{array} \\\\ \hline \begin{array}{l}\text{Selected-coordinate interaction}\\\\\text{magnitude S}\end{array} & 0.208 & 1.187 & 9.518 \\\\ \begin{array}{l}\text{Absolute omitted contribution}\\\\\text{relative to S, abs(T)/S}\end{array} & 0.059 & 0.535 & 0.807 \\\\ \hline \end{array}$$
\textbf{2. How much do interaction magnitudes and omitted-coordinate}\\\\\textbf{contributions vary, and do they change ordering? (continued)}\\\\\text{(Weakness 1 \\& Question 2)} \end{array}$$ Ordering and sign comparisons are: $$\begin{array}{llr} \hline \textbf{Comparison} & \textbf{Statistic} & \textbf{Value} \\\\ \hline \begin{array}{l}\text{Selected-coordinate dot vs.}\\\\\text{full-gradient dot}\end{array} & \text{Spearman} & 0.9784 \\\\ \begin{array}{l}\text{MGCR vs. negative}\\\\\text{selected-coordinate dot}\end{array} & \text{Spearman} & 0.5880 \\\\ \begin{array}{l}\text{MGCR vs. negative full-gradient}\\\\\text{dot}\end{array} & \text{Spearman} & 0.5344 \\\\ \begin{array}{l}\text{Selected-coordinate vs.}\\\\\text{full-gradient dot signs}\end{array} & \text{Different signs} & \text{43/1,536 (2.80\\%)} \\\\ \hline \end{array}$$ MGCR ranks pairs differently from magnitude-sensitive dot-product scores. Selected-coordinate and full-gradient dot products have closely aligned rankings. $$\begin{array}{l} \textbf{3. Are the gains over Residual consistent across runs and without}\\\\\textbf{XStoryCloze?}\\\\\text{(Weakness 2 \\& Question 3)} \end{array}$$ We computed the equally weighted benchmark average separately for each matched run. Seed 1-3 label the three matched runs within each backbone. Delta denotes MGCR minus Residual; the four-task average excludes XStoryCloze. Qwen2.5-7B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 61.840 & 62.266 & +0.426 & +0.0900 \\\\ \text{Seed 2} & 62.088 & 61.806 & -0.282 & -0.3550 \\\\ \text{Seed 3} & 61.604 & 62.036 & +0.432 & +0.2725 \\\\ \text{Mean} & 61.844 & 62.036 & +0.192 & +0.0025 \\\\ \hline \end{array}$$ Llama-3.1-8B: $$\begin{array}{lrrrr} \hline \textbf{Matched run} & \begin{array}{l}\textbf{Residual,}\\\\\textbf{five-task Avg.}\end{array} & \begin{array}{l}\textbf{MGCR, five-task}\\\\\textbf{Avg.}\end{array} & \begin{array}{l}\textbf{Five-task}\\\\\textbf{delta}\end{array} & \begin{array}{l}\textbf{Four-task}\\\\\textbf{delta}\end{array} \\\\ \hline \text{Seed 1} & 46.868 & 47.994 & +1.126 & +0.9075 \\\\ \text{Seed 2} & 47.100 & 47.536 & +0.436 & +0.4475 \\\\ \text{Seed 3} & 46.642 & 48.044 & +1.402 & +1.4950 \\\\ \text{Mean} & 46.870 & 47.858 & +0.988 & +0.9500 \\\\ \hline \end{array}$$ MGCR improves the five-task average in 2/3 Qwen runs and 3/3 Llama runs, with mean gains of +0.192 and +0.988 points, respectively. Excluding XStoryCloze gives the same win counts, 2/3 and 3/3, with mean differences of +0.0025 and +0.9500 points.

Reviewer 2wmZ: please respond to the authors.

Comments on the paper

1 comment

Hi all,

Reviewers: please respond to the authors if you have not already. It is important to give them an opportunity to respond to any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.