- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper provides a rigorous empirical investigation into how parameter-efficient fine-tuning (PEFT) handles low-resource, script-diverse languages. The authors fine-tune IndicTrans2-1B using rank-64 rsLoRA across eight pipelines covering English, Manipuri (written in two distinct scripts: Bengali and Meetei Mayek), Assamese, and Bodo. Crucially, these four highly diverse language-script pairs share a single directional adapter infrastructure. Rather than blindly accepting standard mechanistic interpretability metrics, the paper acts as a critical empirical reality check. It evaluates the joint behavior of four popular layer-wise diagnostics—adapter update norms, representational similarity (Cross-lingual CKA), linear probes, and layer ablations—and systematically demonstrates that these instruments are deeply confounded or misleading in realistic, data-constrained settings.
Critical strengths
Methodological Deconstruction:
The primary value is exposing the severe failure modes of standard interpretability tools. It proves that Cross-lingual CKA in decoders heavily reflects parallel data alignment (spiking to 0.932 purely due to identical sentence inputs across the outbound Assamese-Bodo pair, and dropping to ≤ 0.09 across the eleven pairs without) rather than deep semantic abstraction.
Probing Failure Modes:
The paper reveals that popular probing metrics are easily fooled; two successful probes collapsed into tracking a simple tokenizer artifact (subword fertility), while a part-of-speech probe achieved 0.94 accuracy but an abysmal 0.03 selectivity due to memorization dynamics.
Counter-Intuitive Optimization Regimes:
It uncovers that performance gains are completely unordered by data volume (e.g., the smallest corpus, English → Bodo, surges by +8.69 chrF++, while the largest, Assamese, suffers from negative transfer and loses 3.34 points).
High Practical Realism:
It replaces toy configurations with a massive pre-trained model and a complex socio-linguistic scenario (bi-script Manipuri and multi-family low-resource translation).
Critical weaknesses
Pre-training Priors:
The authors conclude that performance gains are fundamentally "unordered by data volume" and that metrics fail to explain why. However, this entirely overlooks the starting state of IndicTrans2-1B. IndicTrans2 is already highly optimized for major Indic languages like Assamese and Bengali-script text. The degradation in Assamese (-3.34 chrF++) likely represents catastrophic forgetting of highly mature pre-trained model. Conversely, the massive gain in Bodo (+8.69) likely reflects the model bootstrapping an under-represented language.
Interference in the Shared Adapter Infrastructure:
The decision to employ a single physical adapter per translation direction, shared across four highly diverse language-script pairs, introduces a parameter bottlenecking and gradient conflict. Under this framework, the outbound adapter must simultaneously accommodate English translating into Assamese, Bodo, and two separate scripts of Manipuri. From a linguistic perspective, this forces languages from entirely distinct families—Sino-Tibetan (Bodo, Manipuri) and Indo-Aryan (Assamese)—with fundamentally competing grammatical, morphological, and structural properties to fight over the same rank-64 LoRA subspace. The layer-wise variations reported by the diagnostics (e.g., load-bearing layer counts jumping from 2 to 32 layers) are likely a faithful representation of a broken model, rather than a failure of the metrics themselves.
Questions for authors
Gradient Conflict:
Since your architecture utilizes exactly one adapter per translation direction to service four language-script pairs, how can you prove that the "uneven and unordered" adaptation is a result of low-resource script adaptation rather than a side-effect of gradient conflict? Forcing typologically distant families (Sino-Tibetan vs. Indo-Aryan) to update the exact same low-rank parameter weights naturally induces parameter competition. Would a control baseline utilizing individual, unshared language-specific adapters exhibit this same chaotic diagnostic profile, or would the layer-wise metrics stabilize?
Pre-training Priors:
Given that IndicTrans2 already possesses massive representational capacity for Assamese, to what extent does the base model's pre-existing knowledge dictate your findings? Have you considered evaluating your pipeline using a base model with a uniform prior across these languages (e.g., a vanilla multilingual model like mT5 or ByT5)?
3 comments
Thanks to the authors for detailed clarification regarding my concerns. The explanations are logical but not enough to raise initial evaluation score.
LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION
Thank you for taking the time to review our detailed response and for acknowledging the logic of our explanations. We fully understand why logical arguments alone are not enough to raise your score without seeing the completed empirical results and the revised manuscript in place.
Because your evaluation specifically highlighted the need to disentangle shared-adapter gradient conflict from script-dependent dynamics and to test pre-training forgetting, we are prioritizing the exact experiments we proposed so you can inspect the concrete numbers rather than just our reasoning. Specifically, we are finalizing:
-
Standalone language-specific adapters. Standalone, language-specific LoRA adapters for En→Assamese and En→Bodo (under identical rank, scaling, and training schedules), alongside their layer-wise causal ablations, to directly test whether removing cross-family sharing stabilizes load-bearing layer counts or prevents the Assamese regression.
-
Out-of-domain evaluation and forgetting. An out-of-domain Assamese evaluation (using IN22) and validation trajectories over training steps, to directly measure catastrophic forgetting of pre-training priors.
-
Multi-seed script comparison. Multi-seed ablation sign agreement across seeds 16–20 for Manipuri in Meetei Mayek vs. Bengali script, to separate optimization variance from script effects.
-
Smaller checkpoint. A smaller IndicTrans2 checkpoint, to verify how pre-training scale interacts with these patterns.
We are also restructuring the presentation of the paper, which you noted was difficult to follow in places, to clearly separate our methodological audit of confounded diagnostics from our empirical findings on shared vs. unshared adaptation. We will post these completed results and update the manuscript shortly, and we hope the concrete evidence will address your remaining reservations and warrant a higher score.
LAYER-WISE CAUSAL EVIDENCE OF SCRIPT-DEPENDENT ADAPTATION IN LOW-RESOURCE TRANSLATION
Thank you for the careful review. All new results use the existing LoRA-FT adapters, with no retraining. [P]=Paper; T=Table; F=Fig.; [R]=readme: https://anonymous.4open.science/r/NE_Indic_Layerwise_Analysis-5BC6/rebuttal_run_file/readme.md (scripts, commands, tables).
Pre-training priors (W1, Q2)
The starting state matters. Assamese is "the pipeline the base model handles best" [P §1], with the highest zero-shot chrF++ in En→X (59.79, T2), so the En→Assamese regression is consistent with the adapter overwriting a mature prior. We will add this as an explicit hypothesis. The prior does not appear to be the whole explanation, for three reasons:
Output analysis [R §4]: En→Assamese outputs are fully in the target script, with 3-gram repetition 0.05% and no empty outputs, so degeneration and script mixing are not causes. Shorter output (length ratio 0.974) explains part of the BLEU drop (1.9 without brevity penalty vs 2.5). LaBSE similarity to the reference falls only for this pipeline (0.8809→0.8720) [R §3]. This fits forgetting but does not identify the cause. To test forgetting, we will evaluate zero-shot and adapted En→Assamese on an out-of-domain Assamese test set (checking whether IN22 provides one) and report the dev curve of the shared adapter over training [P §2.2, Eq. 4, T7].
mT5/ByT5 are not trained for translation, so fine-tuning them on 15,000–54,000 pairs [T1] would mostly measure learning translation from little data, confounded with the language and script effects. Our question is what happens when a model that already translates these languages is adapted. We will add a smaller IndicTrans2 checkpoint and list other model families as a limitation.
Interference in the shared adapter (W2, Q1)
Parameter competition across four pairs is possible, and the paper does not rule it out. Two observations argue against its simplest form:
The instrument findings do not depend on adapter health. On row-aligned sentences, all six En→X pairs reach 0.953–0.991 decoder CKA vs ≤0.005 for permuted rows [R §2], while the same adapter gave ≤0.013 for five of these pairs in T3, so only the inputs changed. The two probes share one fertility label [Eq. 14, T10], and the script probe is perfect at every layer because pipeline identity is in its labels [T12]. Gradient conflict cannot produce these.
For load-bearing counts (2–32, T5), we extended ablation to 500 sentences with paired bootstrap intervals [R §1]: 234/288 (81%) include zero, and counts shift with the sample on the same adapter (C 13→28, F 14→0, E 6→12; T4 vs R §1), so sampling noise explains part of the instability. Effects are not all null: 54 intervals exclude zero, including 13 harmful removals in G. We cannot say the adapter is free of interference; only a comparison with unshared adapters can address it.
We will add: