acceptodds
Under review as a conference paper at ICLR 2027

Layer-Wise Causal Evidence of Script-Dependent Adaptation in Low-Resource Translation

Abstract

Parameter-efficient adaptation of translation models is judged by corpus-level metrics, while the question of where adaptation occurs is answered with a familiar set of layer-wise instruments: adapter norms, representational similarity, linear probes and layer ablations. We fine-tune IndicTrans2-1B with rank-64 LoRA under rsLoRA scaling, one adapter per translation direction shared by four language–script pairs, across eight pipelines covering English, Manipuri in two scripts, Assamese and Bodo. Adaptation is uneven and unordered by data volume: EnglishBodo, the smallest corpus, gains 8.69 points chrF++ and EnglishMeetei Mayek 4.10 points, whereas Bengali-script Manipuri is unchanged and Assamese, half of the training mixture, loses 3.34 points. Re-translating the training corpus for synthetic data changes chrF++ by less than one point in all forty comparisons and does not compound over rounds. The instruments prove less informative than they appear. Cross-lingual CKA mainly reflects whether two pipelines saw the same sentences, reaching 0.932 in the decoder for the one pair with identical inputs and at most 0.09 for the eleven pairs without; two probes reduce to one subword-fertility statistic; a part-of-speech probe reaches 0.94 accuracy at 0.03 selectivity; a script probe is perfect at every layer because its labels encode pipeline identity; and adapter norms never significantly predict ablation effects, whose load-bearing count ranges from 2 to 32 layers on differences of at most 1.05 chrF++. We release our code and resources.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Low-resource languages, Layer-wise Analysis, Parameter efficient fine tuning, Machine Translation, Centered Kernel Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EYZQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper provides a rigorous empirical investigation into how parameter-efficient fine-tuning (PEFT) handles low-resource, script-diverse languages. The authors fine-tune IndicTrans2-1B using rank-64 rsLoRA across eight pipelines covering English, Manipuri (written in two distinct scripts: Bengali and Meetei Mayek), Assamese, and Bodo. Crucially, these four highly diverse language-script pairs share a single directional adapter infrastructure. Rather than blindly accepting standard mechanistic interpretability metrics, the paper acts as a critical empirical reality check. It evaluates the joint behavior of four popular layer-wise diagnostics—adapter update norms, representational similarity (Cross-lingual CKA), linear probes, and layer ablations—and systematically demonstrates that these instruments are deeply confounded or misleading in realistic, data-constrained settings.

Critical strengths

Methodological Deconstruction:

The primary value is exposing the severe failure modes of standard interpretability tools. It proves that Cross-lingual CKA in decoders heavily reflects parallel data alignment (spiking to 0.932 purely due to identical sentence inputs across the outbound Assamese-Bodo pair, and dropping to ≤ 0.09 across the eleven pairs without) rather than deep semantic abstraction.

Probing Failure Modes:

The paper reveals that popular probing metrics are easily fooled; two successful probes collapsed into tracking a simple tokenizer artifact (subword fertility), while a part-of-speech probe achieved 0.94 accuracy but an abysmal 0.03 selectivity due to memorization dynamics.

Counter-Intuitive Optimization Regimes:

It uncovers that performance gains are completely unordered by data volume (e.g., the smallest corpus, English → Bodo, surges by +8.69 chrF++, while the largest, Assamese, suffers from negative transfer and loses 3.34 points).

High Practical Realism:

It replaces toy configurations with a massive pre-trained model and a complex socio-linguistic scenario (bi-script Manipuri and multi-family low-resource translation).

Critical weaknesses

Pre-training Priors:

The authors conclude that performance gains are fundamentally "unordered by data volume" and that metrics fail to explain why. However, this entirely overlooks the starting state of IndicTrans2-1B. IndicTrans2 is already highly optimized for major Indic languages like Assamese and Bengali-script text. The degradation in Assamese (-3.34 chrF++) likely represents catastrophic forgetting of highly mature pre-trained model. Conversely, the massive gain in Bodo (+8.69) likely reflects the model bootstrapping an under-represented language.

Interference in the Shared Adapter Infrastructure:

The decision to employ a single physical adapter per translation direction, shared across four highly diverse language-script pairs, introduces a parameter bottlenecking and gradient conflict. Under this framework, the outbound adapter must simultaneously accommodate English translating into Assamese, Bodo, and two separate scripts of Manipuri. From a linguistic perspective, this forces languages from entirely distinct families—Sino-Tibetan (Bodo, Manipuri) and Indo-Aryan (Assamese)—with fundamentally competing grammatical, morphological, and structural properties to fight over the same rank-64 LoRA subspace. The layer-wise variations reported by the diagnostics (e.g., load-bearing layer counts jumping from 2 to 32 layers) are likely a faithful representation of a broken model, rather than a failure of the metrics themselves.

Questions for authors

Gradient Conflict:

Since your architecture utilizes exactly one adapter per translation direction to service four language-script pairs, how can you prove that the "uneven and unordered" adaptation is a result of low-resource script adaptation rather than a side-effect of gradient conflict? Forcing typologically distant families (Sino-Tibetan vs. Indo-Aryan) to update the exact same low-rank parameter weights naturally induces parameter competition. Would a control baseline utilizing individual, unshared language-specific adapters exhibit this same chaotic diagnostic profile, or would the layer-wise metrics stabilize?

Pre-training Priors:

Given that IndicTrans2 already possesses massive representational capacity for Assamese, to what extent does the base model's pre-existing knowledge dictate your findings? Have you considered evaluating your pipeline using a base model with a uniform prior across these languages (e.g., a vanilla multilingual model like mT5 or ByT5)?

3 comments
AuthorsOpenReview ↗

LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION

Thank you for the careful review. All new results use the existing LoRA-FT adapters, with no retraining. [P]=Paper; T=Table; F=Fig.; [R]=readme: https://anonymous.4open.science/r/NE_Indic_Layerwise_Analysis-5BC6/rebuttal_run_file/readme.md (scripts, commands, tables).

Pre-training priors (W1, Q2)

The starting state matters. Assamese is "the pipeline the base model handles best" [P §1], with the highest zero-shot chrF++ in En→X (59.79, T2), so the En→Assamese regression is consistent with the adapter overwriting a mature prior. We will add this as an explicit hypothesis. The prior does not appear to be the whole explanation, for three reasons:

  1. Same language, same prior, different outcome. A (Meetei Mayek) and C (Bengali script) start at similar chrF++ (34.80, 34.41, T2). After adaptation A gains +4.11 (p<0.001) and C changes 0.00 (p=0.97) [T6], and the gap holds under size-matching, solo training and five seeds [P §3.2, F2, F3].
  2. Bodo is not a low-prior case. Its zero-shot chrF++ (57.12) and BLEU (24.25) are close to Assamese (59.79, 24.14) [T2], yet it gains most (+8.69 chrF++) while Assamese loses [T6].
  3. The regression is direction-specific. En→Assamese loses on both metrics (−3.33 chrF++, −2.49 BLEU), while Assamese→En gains on BLEU (+5.98) [T6]. This is suggestive only, since the directions use different checkpoints and adapters [P §2.1].

Output analysis [R §4]: En→Assamese outputs are fully in the target script, with 3-gram repetition 0.05% and no empty outputs, so degeneration and script mixing are not causes. Shorter output (length ratio 0.974) explains part of the BLEU drop (1.9 without brevity penalty vs 2.5). LaBSE similarity to the reference falls only for this pipeline (0.8809→0.8720) [R §3]. This fits forgetting but does not identify the cause. To test forgetting, we will evaluate zero-shot and adapted En→Assamese on an out-of-domain Assamese test set (checking whether IN22 provides one) and report the dev curve of the shared adapter over training [P §2.2, Eq. 4, T7].

mT5/ByT5 are not trained for translation, so fine-tuning them on 15,000–54,000 pairs [T1] would mostly measure learning translation from little data, confounded with the language and script effects. Our question is what happens when a model that already translates these languages is adapted. We will add a smaller IndicTrans2 checkpoint and list other model families as a limitation.

Interference in the shared adapter (W2, Q1)

Parameter competition across four pairs is possible, and the paper does not rule it out. Two observations argue against its simplest form:

  1. Training A and C alone instead of jointly moves chrF++ <0.2 [P §3.2, F2]. This is the one direct unshared comparison, and it covers two scripts of one language, not the four-pair mixture.
  2. The pair with the largest data share loses. Assamese supplies half the mixture (π_E=0.500), Bodo 14.1%, yet Bodo gains most [T1, Eq. 2, T6]. Smaller pairs crowding Assamese out of the rank-64 subspace would predict the opposite.

The instrument findings do not depend on adapter health. On row-aligned sentences, all six En→X pairs reach 0.953–0.991 decoder CKA vs ≤0.005 for permuted rows [R §2], while the same adapter gave ≤0.013 for five of these pairs in T3, so only the inputs changed. The two probes share one fertility label [Eq. 14, T10], and the script probe is perfect at every layer because pipeline identity is in its labels [T12]. Gradient conflict cannot produce these.

For load-bearing counts (2–32, T5), we extended ablation to 500 sentences with paired bootstrap intervals [R §1]: 234/288 (81%) include zero, and counts shift with the sample on the same adapter (C 13→28, F 14→0, E 6→12; T4 vs R §1), so sampling noise explains part of the instability. Effects are not all null: 54 intervals exclude zero, including 13 harmful removals in G. We cannot say the adapter is free of interference; only a comparison with unshared adapters can address it.

We will add:

  • Language-specific adapters for En→Assamese and En→Bodo (same rank, scaling, schedule), comparing quality with the shared adapter and repeating the ablation. Stabilised counts would be reported as evidence of interference; unstable counts as unreliable even without sharing. If the Assamese-only adapter also loses to zero-shot, sharing is not the cause.
  • Ablation across seeds 16–20 for A and C [P §3.2, F3], reporting sign agreement [R: rebuttal_ablation_ci.py --compare].

Thanks to the authors for detailed clarification regarding my concerns. The explanations are logical but not enough to raise initial evaluation score.

AuthorsOpenReview ↗

LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION

Thank you for taking the time to review our detailed response and for acknowledging the logic of our explanations. We fully understand why logical arguments alone are not enough to raise your score without seeing the completed empirical results and the revised manuscript in place.

Because your evaluation specifically highlighted the need to disentangle shared-adapter gradient conflict from script-dependent dynamics and to test pre-training forgetting, we are prioritizing the exact experiments we proposed so you can inspect the concrete numbers rather than just our reasoning. Specifically, we are finalizing:

  1. Standalone language-specific adapters. Standalone, language-specific LoRA adapters for En→Assamese and En→Bodo (under identical rank, scaling, and training schedules), alongside their layer-wise causal ablations, to directly test whether removing cross-family sharing stabilizes load-bearing layer counts or prevents the Assamese regression.

  2. Out-of-domain evaluation and forgetting. An out-of-domain Assamese evaluation (using IN22) and validation trajectories over training steps, to directly measure catastrophic forgetting of pre-training priors.

  3. Multi-seed script comparison. Multi-seed ablation sign agreement across seeds 16–20 for Manipuri in Meetei Mayek vs. Bengali script, to separate optimization variance from script effects.

  4. Smaller checkpoint. A smaller IndicTrans2 checkpoint, to verify how pre-training scale interacts with these patterns.

We are also restructuring the presentation of the paper, which you noted was difficult to follow in places, to clearly separate our methodological audit of confounded diagnostics from our empirical findings on shared vs. unshared adaptation. We will post these completed results and update the manuscript shortly, and we hope the concrete evidence will address your remaining reservations and warrant a higher score.

Reviewer_KENJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines LoRA training, synthetic data, and layer-wise analysis in low-resource machine translation. The authors fine-tune IndicTrans2-1B on 4 Indic language-script pairs in both translation directions and show that the results are uneven (improvements for some cases while losses for some others). They also investigate back-translation, iterative data augmentation, and continued training, finding little additional improvement over LoRA fine-tuning only. Finally, they conduct layer-wise analyses using representation similarity, linear probing, adapter norms, and causal ablations

Critical strengths
  • The paper presents an empirical study of low-resource Indic machine translation.
  • The paper provides an audit of layer-wise techniques, identifying potential confounds and limitations of these techniques.
Critical weaknesses
  • The novelty is limited. LoRA, synthetic data, linear probing, back-translation, etc are already well-established.
  • The experiments are restricted to a single small model and 4 Indic language-script pairs. This makes it hard to assess the generalization of the findings. The authors should cover more languages, different model scales and families to show the reliability of the results. In addition to lexical-based metrics, they should also report some semantic-based metrics like Indic-COMET or LaBSE embedding similarity, which the authors use for filtering synthetic data.
  • There is no analysis that sufficiently explains why these results are uneven. It would be interesting if the authors could show why English to Bodo improves significantly, while English to Assamese reduces the score compared to the baseline.
  • The connection between the layer-wise analyses and the paper’s main claims is unclear. The authors claim to study the joint behavior of LoRA adaptation, synthetic data augmentation, and layer-wise analysis. However, the layer-wise analyses are conducted only on the initial LoRA model, without examining the effects of synthetic data.
Questions for authors

Refer to the weaknesses

3 comments
AuthorsOpenReview ↗

LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION

Thank you for the detailed review. All new results use the existing LoRA-FT adapters, with no retraining. [P]=Paper; T=Table; F=Fig.; [R]=readme: https://anonymous.4open.science/r/NE_Indic_Layerwise_Analysis-5BC6/rebuttal_run_file/readme.md.

Novelty

We do not claim a new method. The contribution is an empirical test of how these established tools behave together where they are routinely combined but rarely checked: few script-diverse languages, tens of thousands of pairs, one shared adapter per direction [P §1, T1]. Findings:

  • Adaptation is uneven and not ordered by data volume. En→Bodo (smallest, 15,214 pairs) gains 8.69 chrF++; En→Assamese (half the mixture, 54,000) loses 3.33 [P T1, T2, T6].
  • Filtered self-generated data, including three rounds of teacher re-translation (Exp-2/3/4), changes chrF++ by <1 in all forty comparisons (8×5) without compounding; max +0.86 (C, Exp-3) [P §3.1, T2].
  • Layer-wise instruments are confounded: decoder CKA is 0.932 for the one identical-input pair and ≤0.09 for the other eleven [P T3, §4]. On row-aligned inputs, all six En→X pairs reach 0.953–0.991 vs ≤0.005 for permuted rows, so CKA tracks input identity [R §2]. The two "linguistic" probes share one fertility label [P Eq. 14, T10]. The script probe is perfect at every layer because labels encode pipeline identity [P T12, F8]. Adapter norms are never significantly positively correlated with ablation effects, and negatively for A and D [P T5]. With 500 sentences and paired bootstrap CIs, 234/288 (81%) ablation effects are indistinguishable from zero [R §1].

These matter because layer-allocation methods (Zhang et al., 2023; Zhou et al., 2025) assume such scores track layer contribution [P §4]. BLEU/chrF++ also disagree on the two Manipuri-target pipelines [P §1, T6]. The abstract and introduction will call the paper an empirical audit, not a methods paper.

Scope and metrics

Findings are specific to IndicTrans2-1B and these four language–script pairs. We will add a limitations paragraph and word claims as setting-specific; other seeds, mixtures, scales remain future work [P §5].

Since BLEU and chrF++ disagree on A, C and F [P T6], we computed mean LaBSE cosine (output vs reference), zero-shot vs LoRA-FT [R §3]. It rises for six pipelines (largest En→Bodo, G: 0.8395→0.8809), is unchanged for A (0.9557→0.9558), and falls only for En→Assamese (E: 0.8809→0.8720). Direction matches chrF++ except A (chrF++ +4.11, LaBSE flat) and C (LaBSE +0.007, chrF++ unchanged) [P T6]. We will add COMET (wmt22-comet-da) and a LoRA-FT vs zero-shot comparison on a smaller IndicTrans2 checkpoint; other model families are a limitation.

Why the results are uneven

  • Corpus size: A and C differ 1.4× [T1]. Size-matching leaves the gap unchanged (+4.05 vs +4.01); solo training moves chrF++ <0.2 [P §3.2, F2]. Over five seeds A's gain is significant in each (p<0.001; 38.41±0.15), C's in none (p>0.4; 34.52±0.04) [F3]. A and C are one language in two scripts, so the contrast points to the language–script pair.
  • Drift: En→Assamese has the largest encoder drift (CKA to base 0.228) yet loses; En→Bodo drifts less (0.360) and gains most [T9, §4].
  • Script: Meetei Mayek/Devanagari pipelines gain most (A +4.11, B +2.09, G +8.69, H +4.22), Bengali-script pairs least (C 0.00, D +0.54, E −3.33, F +0.55) [T6]. The regression is specific to En→Assamese: the reverse improves on BLEU (+5.98), chrF++ +0.55 not significant (p=0.076) [T6].
  • Hypothesis only: En→Assamese has the highest baseline chrF++ in its group (59.79), so less headroom. This is incomplete: Bodo's baseline is also high (57.12) and C starts low (34.41) without gaining [T2].
  • Output analysis [R §4]: adapted En→Assamese is 100% target script, with 3-gram repetition 0.05% and no empty outputs, so script mixing and degeneration are not causes. Output is slightly short (length ratio 0.974, BP 0.9735), explaining part of the BLEU drop (1.9 BLEU without BP, 24.14→22.24, vs 2.5, 24.14→21.65). The remainder is unexplained. We will add (i) an Assamese-only adapter vs the shared one (if it also loses, sharing is not the cause), and (ii) dev curves over training steps for the shared adapter (§2.2, Eq. 4, T7) and single-pipeline adapters, to check for early peak then decline; otherwise we will report it as open.

Layer-wise analysis and main claims

"Joint behavior" [P §1] overstates the connection and will be reworded. The paper's three questions (where a shared adapter acts, whether self-generated data helps, whether layer-wise instruments answer the first) are related, not joint. Since Exp-1–5 change chrF++ by <1 [T2], large layer-level differences were not expected, but this should be shown. We will repeat the norm, drift and ablation analyses on the Exp-2 adapter (strongest augmented [§3.1]), compare with LoRA-FT (the reference adapter [P §2.5, §2.6]), and report the outcome.

Thanks to the authors for their efforts to address my concerns. However, given that the key limitations remain unresolved and the paper still requires major revisions, I will maintain my original score.

AuthorsOpenReview ↗

LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION

Thank you for taking the time to re-evaluate our rebuttal and for your candid feedback. We fully understand your hesitation: in our previous response, several key analyses, including semantic metrics beyond LaBSE, comparisons on a different model scale, an isolated Assamese-only control to diagnose the English-to-Assamese regression, and layer-wise diagnostics on the augmented Exp-2 adapter, were framed as planned commitments rather than completed experiments. We recognize that without concrete evidence for these points, it is difficult to assess whether the core limitations can be resolved within the current revision cycle.

To address your concern directly and substantiate our claims, we are actively completing these experiments and will update the manuscript and anonymous repository prior to the end of the discussion period.

  1. Assamese-only control. To directly answer why English-to-Assamese (Pipeline E) regresses by 3.33 chrF++ despite comprising half of the training mixture, we are finalizing the single-pipeline Assamese-only LoRA control alongside step-by-step validation trajectories. Comparing the solo adapter against the shared group adapter will cleanly isolate whether the regression stems from negative cross-lingual interference within the shared adapter or from overfitting/early peaking on a language where the zero-shot baseline is already strong (59.79 chrF++).

  2. Layer-wise diagnostics on Exp-2. To bridge the gap between our layer-wise diagnostics and the synthetic data experiments, we are running the update-norm, representational drift, and causal ablation analyses on the strongest augmented checkpoint (Exp-2). Contrasting Exp-2 directly against the LoRA-FT reference adapter will provide concrete layer-level evidence of whether iterative self-translation leaves internal layer contributions invariant or shifts capacity without yielding corpus-level gains.

  3. Additional metrics and model scale. To ensure our findings are not artifacts of lexical n-gram metrics or a single model capacity, we are incorporating COMET (wmt22-comet-da) scores across all eight pipelines, alongside a LoRA-FT vs. zero-shot evaluation on a smaller IndicTrans2 checkpoint.

  4. Reframing. Finally, we are revising the abstract, introduction, and discussion to accurately frame the work as a controlled empirical audit of layer-wise adaptation diagnostics and script-dependent behavior, rather than a new modeling method, explicitly delineating the scope boundaries of our four Northeast Indian language-script pairs.

We greatly appreciate your rigorous critique, which has helped us sharpen both the causal diagnosis of the uneven adaptation results and the framing of our diagnostic audit. We hope these completed experiments and structural revisions will directly resolve your remaining concerns and warrant a reconsideration of your score.

Reviewer_RuiNOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies shared rank-64 rsLoRA adapters to IndicTrans2-1B across four Northeast Indian language/script pairs in both directions. Adaptation is uneven: English-to-Bodo gains 8.69 chrF++, English-to-Meitei gains 4.11, Bengali-script Manipuri is unchanged, and English-to-Assamese loses 3.33. Retranslating the existing parallel data changes chrF++ by less than one point across all 40 comparisons. The paper also finds that layer-wise diagnostics can be dominated by input pairing, tokenizer statistics, or pipeline identity.

Critical strengths
  1. The paper critically examines CKA, probes, and adapter norms rather than treating them as reliable by default.
  2. It studies under-served languages and reports both large gains and negative transfer, including the 3.33 chrF++ Assamese regression.
Critical weaknesses
  1. The causal conclusions use ablations over only 100 sentences, with effects no larger than 1.05 chrF++ and no confidence intervals or seed analysis. Counts of 2–32 load-bearing layers may be unstable.
  2. Several diagnostics are confounded by construction: CKA compares unmatched rows, two probes use the same label, and script identity is encoded by pipeline tags. Exp-5 is also selected using test BLEU.
Questions for authors
  1. How many layer-ablation effects remain sign-stable under paired bootstrap intervals and multiple seeds?
  2. Do the conclusions survive paired CKA inputs and non-confounded probe labels?
1 comment
AuthorsOpenReview ↗

LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION

Thank you for the careful review. Refs: [P]=Paper, [R]=readme (https://anonymous.4open.science/r/NE_Indic_Layerwise_Analysis-5BC6/rebuttal_run_file/readme.md). Several weaknesses are limitations we state ourselves [P §3.2, §4]. We ran the two points testable in the rebuttal period, W1/Q1 and W2/Q2 [R: rebuttal_ablation_ci.py, rebuttal_paired_cka.py].

W1/Q1: Ablation (100 sentences, no CIs)

You are right that the ablation had no uncertainty estimates [P §2.5, Table 4]. We extended it to 500 dev sentences (the first 100 are the paper's) [R §1: same sampler/seed as step6_layer_analysis.py]. We added paired bootstrap CIs (500 resamples, seed 16, 95% percentile) for all 36 layers of all 8 pipelines [R §1; outputs/rebuttal/ablation/r1_n500_<P>_ci.csv].

Most effects are indistinguishable from zero: 234/288 (81%) intervals include zero [R §1].

Significantly harmful removals exist only in G (13 layers), C (6), D (4) and H (4), and in none of A, B, E, F [R §1, "CI above 0"].

Positive-drop counts depend on the sample (100→500 sentences): C 13→28, F 14→0, E 6→12, H 25→18 [P Table 4 "Hurts"; R §1]. This supports our claim that load-bearing counts are not a stable importance measure [P Table 5, §3.2].

Effects are not all zero. 54 intervals exclude zero, against ~14 expected by chance (layers are not independent, so this is a rough guide). In G, 13 removals significantly lower chrF++. In A and F, 11 and 10 raise it, consistent for A with encoder 0/18 [R §1 "CI below 0"; P Table 13]. Sign and count are unreliable, which is our conclusion: single-layer ablation gives no stable account of where adaptation acts, and Table 5's counts (2–32) are not layer importance [P §3.2, Table 5; R §1].

We will add the intervals and correct the abstract's "at most 1.05 chrF++". That holds at 100 sentences (H dec.6) [P Abstract, Table 4]. At 500 sentences the largest effect is 2.31 (H), and all others are ≤0.88 [R §1 "Largest absolute drop"].

The intervals reflect sentence sampling only. We will run ablation on the five seed adapters for A and C (seeds 16–20) and report sign agreement [P §3.2, Fig. 3; R: rebuttal_ablation_ci.py --compare]. Fig. 3 covers translation quality only, and we will state this [P Fig. 3]. "Most-critical layer is a decoder layer in 7 of 8 pipelines" will be reworded as descriptive [P Table 4, Table 13/A.3.4].

W2/Q2: Confounded diagnostics

The three confounds are those the paper reports [P §4, Eq. 13, Table 12].

Paired CKA (Q2). We used the content-matched test set: 1,895 sentences present in all pipelines, aligned via the shared English side after removing 102 duplicates. We used the first 300, with a permuted-row control (5 permutations averaged) [R §2; P §2.1, §2.5].

Same inputs (En→X: A, C, E, G): all six pairs reach 0.953–0.991 decoder CKA, with 17–18/18 layers ≥0.85 [R §2]. Five of these pairs were ≤0.013 in Table 3 [P Table 3]. The permuted control is ≤0.005. E↔G is 0.953 against 0.932 in Table 3, so it is not specific to Assamese/Bodo.

Translation-equivalent inputs (X→En: B, D, F, H): encoder 0.013–0.330, decoder 0.017–0.157, no layer ≥0.85. The highest is B↔D, the same language in two scripts [R §2].

CKA therefore follows input identity [R §2; P §4]. We will revise §4: translation equivalence alone did not produce high similarity, contrary to the pairing assumption we stated [P §2.5]. We will add the control and numbers. We will also note that pooling over all positions including padding may lower the X→En values [P Eq. 9; R §2 Limitations].

Camera-ready additions for the other confounds:

  • A script probe with language tags neutralised [P §2.1, §4, Table 12, Fig. 8].
  • Since no gold morphology exists, we will merge the two probes into one fertility probe and remove the morphology wording [P Eq. 14, §4, A.3.3, Table 10].
  • The pipeline A POS probe (0.94 accuracy, 0.03 selectivity; the shuffled control does almost as well) will be described as uninformative. C, E, G show selectivity of 0.26–0.46 [P Table 11].

W2: Exp-5 selected on test BLEU

This is fair. Exp-5 starts from the best of Exp-2/3/4 by group-averaged test BLEU [P §2.6]. We will state this plainly, re-select using dev chrF++, and rerun if the round changes [R: rebuttal_misc.py pick-exp5]. The main conclusion is unaffected, for three reasons:

  1. It rests on Exp-1–4 vs LoRA-FT, with no test selection, and all forty comparisons move chrF++ by <1 (max +0.86, C Exp-3) [P Table 2, §3.1].
  2. Selection can only help Exp-5, yet it wins no pipeline on either metric [P Table 2, §3.1].
  3. Exp-5 is continued training, since subword regularisation did not activate, so it is not headline [P §2.3, §2.6, Table 7].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.