acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Entropic SNR Collapse in Cross-Lingual Differential Privacy

Abstract

Conventional sequence-level gradient clipping and length normalization in Differentially Private (DP) fine-tuning implicitly assume a Latin-centric, homogeneous subword distribution. However, they adversely compound under multilingual distributions because unequal vocabulary allocation inflates sequence lengths for non-Latin and morphologically rich scripts by up to . Under a stylized model of semantic signal conservation, sequence-averaged gradient normalization attenuates signal power quadratically, inducing an upper bound on vector Signal-to-Noise Ratio (SNR) against isotropic Gaussian DP noise, a degradation phenomenon we formalize as Entropic SNR Collapse. Existing solutions face severe trade-offs: sequence-level adaptive clipping fails to regulate heavy-tailed intra-sequence token spikes; per-token clipping with active length normalization collapses parameter update norms by , severely impairing generation fluency; and tokenizer retraining breaks pretrained model interfaces. To mitigate this issue, we propose ToPA (Token-Aware Privacy Alignment), a framework that decouples token sensitivity bounding from sequence aggregation. ToPA enforces localized gradient bounds via unified Token-Wise Ghost Clipping across Transformer projections without materializing per-token Jacobians, accumulates updates via a fixed-anchor aggregator to reflect token-uniform empirical risk without leaking metadata, and privately releases clipping thresholds via the Exponential Mechanism. Extensive downstream benchmarks across four model scales (1B to 9B parameters) on translation (FLORES-200), cross-lingual inference (XNLI), and question answering (TyDi QA) demonstrate that ToPA stabilizes token gradients, prevents optimization collapse on fragmented languages and substantially closes the gap to non-private baselines under strict DP budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
DP, Tokenization Fairness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rV2LOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multilingual DP fine-tuning under cross-lingual tokenization fragmentation. Sequence-level averaging disproportionately attenuates gradient signals for languages with longer token sequences, while DP noise remains length-independent, leading to what they call Entropic SNR Collapse. Under an idealized semantic-signal conservation assumption, they derive an O(NL−2)O(N_L^{-2}) SNR upper bound. To address this issue, the paper proposes ToPA, which combines token-wise ghost clipping with fixed-anchor aggregation to control token-level gradient contributions without active-length normalization. An adaptive variant privately calibrates the token clipping threshold using the Exponential Mechanism.

Critical strengths

strength1. The paper identifies a practically relevant issue in multilingual DP fine-tuning: tokenization fragmentation can lead to substantial sequence-length disparities across languages, which may interact unfavorably with standard DP-SGD. This is a meaningful problem in realistic multilingual LLM settings and has received relatively limited attention.

strength2. ToPA remains computationally efficient through its token-wise ghost clipping implementation. The reported experiments show only moderate runtime and memory overhead compared with standard DP-SGD, making the method practically applicable to parameter-efficient fine-tuning.

strength3. The proposed approach is conceptually straightforward and does not require modifying the pretrained tokenizer or model architecture. The main components can be integrated into the existing DP fine-tuning pipeline without substantial architectural changes.

Critical weaknesses

weakness1. Authors claim that tokenization fragmentation leads to longer sequences, which weakens the gradient signal under sequence averaging while the DP noise remains unchanged. However, the experiments don't directly verify this claim. Figure 1 only shows that cross-token gradient correlation decreases with token distance; it doen't show that increasing tokenization length causes worse DP fine-tuning performance. A more direct experiment in introduction would keep the semantic content and language fixed, vary only the tokenization length, and compare downstream performance under DP and non-DP fine-tuning. This would help separate the effect of tokenization length from other differences across languages.

weakness2. Proposition 1 relies on the assumption that the cumulative semantic signal S=∑tstS=\sum_t s_t remains roughly constant for equivalent content under different tokenizations. However, it is unclear how this “semantic signal” is defined or measured from actual gradients. In particular, can sts_t be directly computed or separated from the other components of the gradient in practice? This assumption is central to the claimed O(NL−2)O(N_L^{-2}) SNR degradation. A clearer mathematical definition or an empirical validation showing that the cumulative signal is approximately preserved as tokenization length changes would substantially strengthen the theoretical motivation for the proposed method.

weakness3. ToPA replaces the standard active-length normalization 1/Ni1/N_i with a fixed anchor 1/Nmax⁡1/N_{\max}. This modification does not only address gradient attenuation; it also changes the effective training objective from sequence-uniform to token-uniform, giving longer sequences proportionally larger update mass. Since the most fragmented languages are also those with the longest sequences, it is difficult to determine how much of the observed improvement comes from mitigating the proposed SNR collapse versus simply reweighting long-sequence examples. Although the paper provides a component ablation, a more controlled comparison that matches the effective objective/length weighting across methods would help isolate the contribution of the proposed privacy mechanism.

Questions for authors

See Weaknesses.

6 comments

Q1. Authors claim that tokenization fragmentation leads to longer sequences, which weakens the gradient signal under sequence averaging while the DP noise remains unchanged. However, the experiments don't directly verify this claim. Figure 1 only shows that cross-token gradient correlation decreases with token distance; it doesn't show that increasing tokenization length causes worse DP fine-tuning performance. A more direct experiment in introduction would keep the semantic content and language fixed, vary only the tokenization length, and compare downstream performance under DP and non-DP fine-tuning. This would help separate the effect of tokenization length from other differences across languages.

We thank Reviewer rV2L for engaging closely with our core thesis: how tokenization fragmentation impacts Differential Privacy in multilingual settings, and for suggesting a direct controlled experiment holding semantic content and language fixed while varying only tokenization length. To isolate tokenization fragmentation from language differences while holding semantic content fixed, we present three complementary controlled experiments (ε=4.0,δ=10−5\varepsilon=4.0, \delta=10^{-5}, Table 8 and Table 5 below; incorporated directly into Section 1 (Figure 1b–c) & Section 3.2 (Findings 1–2), Tables 1–4 & Figure 2d and detailed in Appendix Section E.6, Tables 16–17):

  1. In-Manifold Cross-Tokenizer & Parallel-Corpus Control on Identical Sentences (Zero Off-Manifold Confound): On the exact same 1,0121{,}012 held-out English →\to Thai FLORES-200 devtest sentences, Gemma-2-2B's 256k SentencePiece vocabulary tokenizes Thai into Ntha=49.51N_{\mathrm{tha}} = 49.51 tokens and suffers only a -3.7% relative DP spBLEU_th drop from Non-Private (15.38) to Vanilla DP-SGD (14.81), whereas Llama-3.2-1B's 128k BPE tokenizer fragments the exact same Thai sentences into Ntha=58.82N_{\mathrm{tha}} = 58.82 tokens and suffers a -35.4% relative DP drop (8.23 → 5.32). Similarly, on Llama-3-8B over the same parallel sentences, higher-fragmentation Thai (Ntha=58.82N_{\mathrm{tha}}=58.82) suffers a -14.2% relative DP drop (19.50 → 16.73), 8.8×8.8\times steeper than lower-fragmentation Arabic (Narb=43.67N_{\mathrm{arb}}=43.67, -1.6%, 20.50 → 20.17).
  2. Canonical Monolingual Thai Natural Fertility Stratification (Table 8, Part A): Training exclusively on canonical English →\to Thai (dev, 897897 pairs) and stratifying devtest (1,0121{,}012 pairs) by within-Thai tokenizer fertility (tok/char density: Low 0.434, Medium 0.476, High 0.513, where target token counts remain matched in 57.43–60.69), on Llama-3-8B Non-Private (1/Ni1/N_i) spBLEU_th drops by only -0.80 from Low to High fertility (20.87 → 20.07; chrF++_th: -2.77), whereas Vanilla DP-SGD (1/Ni1/N_i) suffers a 3.56x steeper spBLEU_th drop of -2.85 (18.58 → 15.73; chrF++_th: -5.39). Adaptive ToPA (1/Nmax⁡1/N_{\max}) improves High-fertility Thai quality over Vanilla DP-SGD on both 8B (16.80 / 32.73 vs. 15.73 / 30.95, +1.07 spBLEU_th / +1.78 chrF++_th) and 1B (5.45 / 19.14 vs. 4.78 / 17.39, +0.67 spBLEU_th / +1.75 chrF++_th).
  3. Controlled Intra-Sentence BPE Subword Splitting on Identical Thai Sentences (Table 8, Part B): Deterministically splitting multi-character Thai BPE tokens into valid constituent sub-tokens expands realized mean target length NtgtN_{\mathrm{tgt}} from 58.31 (1.00x) to 78.36 (1.34x), 92.55 (1.59x), and 93.41 (1.60x, saturating at the vocabulary's character/byte sub-token floor for target parameters 1.35, 1.70, 2.10). Because the base LLM embeddings are frozen, artificial sub-token sequences lie off the pretrained BPE manifold: unregularized Non-Private fine-tuning on 897897 pairs rapidly overfits (PPL: 2.61 → 16.31 on 8B), whereas DP clipping and noise regularize off-manifold drift (PPL: 2.54 → 2.64). Comparing DP methods head-to-head on identical fragmented inputs, Adaptive ToPA outperforms Vanilla DP-SGD across all four fragmentation levels on 8B (on both spBLEU_th and chrF++_th). On lower-capacity 1B, ToPA leads at canonical 1.00x (5.56 / 20.45 vs. 5.07 / 18.55, 3.06x higher Step SNR) and Split-1.70 (4.28 / 16.23 vs. 4.25 / 16.20), while off-manifold sub-token gradients push C~∈[25.43,26.50]\widetilde{C} \in [25.43, 26.50] near the Mmax⁡=30.0M_{\max}=30.0 grid ceiling—trading Split-1.35 (4.62 / 17.54 vs. 4.81 / 17.38, ToPA higher on chrF++_th) and Split-2.10 (4.28 / 16.10 vs. 4.40 / 16.26) while maintaining 1.48×1.48\times–1.52×1.52\times higher Step SNR across the artificial sub-token split levels.

Table 8: Controlled Single-Language English →\to Thai Tokenization Fragmentation Studies (ε=4.0,δ=10−5\varepsilon=4.0, \delta=10^{-5}; cells report spBLEU_th / chrF++_th; bold denotes best DP score)

Part A: Canonical Monolingual English →\to Thai Natural Fertility Tertiles (Low: 0.434, Med: 0.476, High: 0.513 tok/char) & Overall Monolingual Control

ModelMethodLow FertilityMed FertilityHigh FertilityOverall Monolingual ThaiStep SNR
Llama-3.2-1BNon-Private (1/Ni1/N_i, σ=0\sigma=0)11.09 / 30.188.71 / 27.029.19 / 26.409.64 / 27.95—
Llama-3.2-1BNon-Private (1/Nmax⁡1/N_{\max}, σ=0\sigma=0)11.32 / 30.269.39 / 27.579.52 / 26.6510.06 / 28.24—
Llama-3.2-1BVanilla DP-SGD (1/Ni1/N_i)5.42 / 19.735.00 / 18.354.78 / 17.395.07 / 18.552.31×10−52.31\times 10^{-5}
Llama-3.2-1BFixed-Anchor Only (1/Nmax⁡1/N_{\max}, No Tok. Clip)4.41 / 15.743.71 / 14.803.70 / 12.753.93 / 14.503.34×10−63.34\times 10^{-6}
Llama-3.2-1BAdaptive ToPA (1/Nmax⁡1/N_{\max}, Ours)5.77 / 21.585.47 / 20.435.45 / 19.145.56 / 20.457.06×10−5\mathbf{7.06\times 10^{-5}}
Llama-3-8BNon-Private (1/Ni1/N_i, σ=0\sigma=0)20.87 / 39.6520.59 / 37.5920.07 / 36.8820.51 / 38.11—
Llama-3-8BNon-Private (1/Nmax⁡1/N_{\max}, σ=0\sigma=0)20.97 / 39.8619.94 / 37.2820.13 / 36.8320.34 / 38.07—
Llama-3-8BVanilla DP-SGD (1/Ni1/N_i)18.58 / 36.3416.59 / 33.2315.73 / 30.9516.94 / 33.631.38×10−51.38\times 10^{-5}
Llama-3-8BFixed-Anchor Only (1/Nmax⁡1/N_{\max}, No Tok. Clip)17.49 / 35.0715.60 / 31.6115.14 / 30.2516.05 / 32.421.09×10−61.09\times 10^{-6}
Llama-3-8BAdaptive ToPA (1/Nmax⁡1/N_{\max}, Ours)19.36 / 37.2317.61 / 34.3616.80 / 32.7317.90 / 34.882.45×10−5\mathbf{2.45\times 10^{-5}}

Part B: Controlled Intra-Sentence BPE Subword Splitting on Identical English–Thai Sentences (8B Realized NtgtN_{\mathrm{tgt}} & Expansion Ratio; 1B within ±0.02 tok)

Split Setting8B Realized NtgtN_{\mathrm{tgt}}8B Non-Private (σ=0\sigma=0)8B Vanilla DP-SGD8B Adaptive ToPA1B Non-Private (σ=0\sigma=0)1B Vanilla DP-SGD1B Adaptive ToPA
Canonical (1.00×1.00\times)58.31 (1.00×1.00\times)20.51 / 38.11 (PPL 2.61)16.94 / 33.6317.90 / 34.889.64 / 27.95 (PPL 4.19)5.07 / 18.555.56 / 20.45
Split-1.3578.36 (1.34×1.34\times)7.08 / 23.02 (PPL 6.39)15.73 / 31.8715.89 / 32.042.80 / 15.19 (PPL 7.91)4.81 / 17.384.62 / 17.54
Split-1.7092.55 (1.59×1.59\times)5.41 / 20.35 (PPL 15.35)15.68 / 31.7815.71 / 31.981.89 / 13.06 (PPL 18.85)4.25 / 16.204.28 / 16.23
Split-2.10 (Ceiling)93.41 (1.60×1.60\times)5.19 / 20.30 (PPL 16.31)15.71 / 31.7815.80 / 32.061.68 / 12.71 (PPL 19.73)4.40 / 16.264.28 / 16.10

Q2. Proposition 1 relies on the assumption that the cumulative semantic signal (S=∑tstS=\sum_t s_t) remains roughly constant for equivalent content under different tokenizations. However, it is unclear how this “semantic signal” is defined or measured from actual gradients. In particular, can (sts_t) be directly computed or separated from the other components of the gradient in practice? This assumption is central to the claimed (O(NL−2)O(N_L^{-2})) SNR degradation. A clearer mathematical definition or an empirical validation showing that the cumulative signal is approximately preserved as tokenization length changes would substantially strengthen the theoretical motivation for the proposed method.

Because token positions t∈{1,…,NL(z)}t \in \lbrace 1,\dots,N_L(z)\rbrace vary with sentence realization and tokenizer, we formalize the semantic signal at the sequence level (Section 3.2 (Finding 1), Table 1; Appendix Section E.6, Table 16; formalizing Proposition 1 in Section 2.2) over a distribution of latent semantic propositions z∼PZz \sim \mathcal{P}_{\mathcal{Z}} realized as text xL(z)x_L(z) in language/tokenizer LL at a fixed LoRA state WwarmW_{\mathrm{warm}} (warmed up for 15 non-private steps so B≠0B \neq 0 and both ∇AL\nabla_A\mathcal{L} and ∇BL\nabla_B\mathcal{L} are active):

SL:=Ez∼PZ[∑t=1NL(z)gt(xL(z);Wwarm)],sˉL:=SLNˉL,gt(xL(z))=sˉL+ξt(z),Ez[∑t=1NL(z)ξt(z)]=0,S_L := \mathbb{E}_{z \sim \mathcal{P}_{\mathcal{Z}}}\left[\sum_{t=1}^{N_L(z)} g_t\bigl(x_L(z); W_{\mathrm{warm}}\bigr)\right], \qquad \bar{s}_L := \frac{S_L}{\bar{N}_L}, \qquad g_t\bigl(x_L(z)\bigr) = \bar{s}_L + \xi_t(z), \qquad \mathbb{E}_z\left[\sum_{t=1}^{N_L(z)} \xi_t(z)\right] = 0,

where NˉL=Ez[NL(z)]\bar{N}_L = \mathbb{E}_z[N_L(z)] and ξt(z):=gt(xL(z))−sˉL\xi_t(z) := g_t(x_L(z)) - \bar{s}_L captures zero-mean lexical/syntactic and example-sampling fluctuations around the mean token signal sˉL\bar{s}_L ( Ez[∑t=1NL(z)ξt(z)]=SL−Ez[NL(z)]sˉL=0\mathbb{E}_z[\sum_{t=1}^{N_L(z)} \xi_t(z)] = S_L - \mathbb{E}_z[N_L(z)]\bar{s}_L = 0 identically, distinct from DP Gaussian noise). Crucially, SLS_L requires no token-to-token alignment: its empirical estimator S^L=1n∑i=1n∑t=1NL(zi)gt(xL(zi);Wwarm)\hat{S}_L = \frac{1}{n}\sum_{i=1}^n \sum_{t=1}^{N_L(z_i)} g_t(x_L(z_i); W_{\mathrm{warm}}) is directly computable by averaging unnormalized sequence gradients across a parallel multi-way corpus at fixed WwarmW_{\mathrm{warm}}. Whenever ∥SL∥2∝NˉLαS\Vert S_L\Vert_2 \propto \bar{N}_L^{\alpha_S} with αS<1\alpha_S < 1, sequence-level 1/NL1/N_L averaging attenuates signal power as NˉL−2(1−αS)\bar{N}_L^{-2(1-\alpha_S)} against length-independent DP noise.

Evaluating ∥S^L∥2\Vert\hat{S}_L\Vert_2 across all 997997 parallel FLORES-200 dev sentences in 8 languages at WwarmW_{\mathrm{warm}} using each model's native tokenizer target lengths NL∈[26.81,66.53]N_L \in [26.81, 66.53] (Review 1, Table 5), we find α^S=0.774\hat{\alpha}_S = 0.774 on Llama-3.2-1B, −1.561-1.561 on Gemma-2-2B (reflecting high English norm 206.20 leveraged across 2B's narrow non-English length band NL∈[30.38,49.77]N_L \in [30.38, 49.77] while non-English scripts stay bounded in [10.61,36.79][10.61, 36.79]), and −0.410-0.410 on Llama-3-8B—strictly below linear growth ( αS=1\alpha_S = 1) on every model, with 6 languages (tur, ind, rus, arb, swh, tha) forming an explicit conservation plateau ∥S^L∥2∈[11.83,18.34]\Vert\hat{S}_L\Vert_2 \in [11.83, 18.34] on Llama-3-8B across NL∈[36.68,58.82]N_L \in [36.68, 58.82]. Without imposing conservation, unconditional empirical DP SNR under 1/NL1/N_L averaging decays with γ^uncond=0.452\hat{\gamma}_{\mathrm{uncond}} = 0.452 (1B), 5.1225.122 (2B), and 2.8212.821 (8B), dropping from English to Thai by 11.1x on 1B and 71.2x on 8B.

Reviewer_wVAJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper discovers a real problem in multi-lingo DP, sequence-level gradient clipping and length normalization in Differentially Private (DP) fine-tuning adversely compound under multilingual distributions, Entropic SNR Collapse. This paper has found that existing methods cannot solve this problem and propose ToPA (Token-Aware Privacy Alignment), a framework that decouples token sensitivity bounding from sequence aggregation. The proposed method TOPA, proposes Token-Wise Ghost Clipping across Transformer projections, accumulates updates via a fixed-anchor aggregator, and privately releases clipping thresholds via EM. This paper has conducted experiments on a variety of models and. datasets and the results have demonstrated its effectiveness.

Critical strengths
  1. Entropic SNR Collapse is a real problem and this paper has found an important problem in multi-lingo dp training settings, which is novel and interesting.
  2. This paper is well written and the proposed method TOPA is technically sound.
  3. The proposed method TOPA can work with LORA, and does not need to adapt the tokenizers or other related modules in the models, which can be adapted to different base models and model architectures.
  4. This paper has conducted a variety of experiments to demonstrate its effectiveness.
Critical weaknesses
  1. The proposed method is kind of incremental works. TOPA mainly combines existing works such as Ghost Clipping, DP-LoRA, norm clipping, etc. But, the identification of a new problem also makes this work novel and interesting to the research community.

2.There is some typo in this paper, such as the Flemings et al.(2024). The method proposed in this paper is not the same with its references. I hope the author to check the references one more time.

Questions for authors

Please refer to weakness.

2 comments

Q1. The proposed method is kind of incremental works. TOPA mainly combines existing works such as Ghost Clipping, DP-LoRA, norm clipping, etc. But, the identification of a new problem also makes this work novel and interesting to the research community.

Response

We sincerely thank Reviewer wVAJ for recognizing that identifying this new problem makes our work "novel and interesting to the research community." Indeed, the key contribution and novelty of our paper lie in studying and formalizing how tokenization impacts Differential Privacy in multilingual settings (Entropic SNR Collapse) rather than claiming standalone novelty in Ghost Clipping or DP-LoRA themselves as specifically how unequal subword fragmentation attenuates per-record gradient signal under sequence-level 1/Ni1/N_i averaging against length-independent DP noise. Building directly on this problem formulation, we further show theoretically and empirically (Sections 3.2 & 4.1 (Finding 5), Table 2 & Figure 2; Appendix Section E.3, Table 10) why partial combinations of existing DP-LoRA and clipping primitives fail to resolve this tokenization–DP disparity unless unified with fixed-anchor aggregation and one-shot private quantile calibration:

  1. Training-time DP-NTP (Config 3: Ghost Clipping + 1/Ni1/N_i averaging) retains data-dependent 1/Ni1/N_i length attenuation and NiN_i metadata dependence, trailing Adaptive ToPA in Step SNR across all four models in both LR regimes and in translation quality across all four models at matched η=10−3\eta=10^{-3} (7.19 / 24.23 vs. 7.38 / 24.56 on 1B; 15.57 / 34.88 vs. 16.44 / 36.01 on 2B; 20.48 / 39.18 vs. 20.54 / 39.41 on 8B; 25.19 / 43.10 vs. 25.50 / 43.51 on 9B-IT, Table 1).
  2. Fixed-anchor 1/Nmax⁡1/N_{\max} + global clipping without token clipping (Config 2a) leaves heavy-tailed intra-sequence token spikes (4.41x–7.17x median norm) unclipped prior to summation, dropping spBLEU / chrF++ from 6.58 / 22.76 to 3.97 / 14.51 on 1B and 16.37 / 32.29 to 10.75 / 25.64 on 9B-IT (and from 25.50 / 43.51 to 19.15 / 35.54 on 9B-IT at η=10−3\eta=10^{-3}).
  3. Sequence-level AdaClipping (Config 1c, Default LR) settles at an ≈9×\approx 9\times looser threshold (Cg≈2.9C_g \approx 2.9, leaving ≈50%\approx 50\% of sequences unclipped so unclipped records still experience 1/Ni1/N_i length scaling and active-length NiN_i metadata dependence), edging default-LR ToPA by 0.11–0.75 spBLEU via its looser step norm and smaller calibration deduction (ε1=0.20\varepsilon_1=0.20) while trailing default-LR ToPA in Step SNR on 1B (5.95×10−55.95\times 10^{-5} vs. 7.36×10−57.36\times 10^{-5}), 2B (2.79×10−52.79\times 10^{-5} vs. 2.84×10−52.84\times 10^{-5}), and 8B (2.25×10−52.25\times 10^{-5} vs. 2.76×10−52.76\times 10^{-5}).

Q2. There is some typo in this paper, such as the Flemings et al. (2024). The method proposed in this paper is not the same with its references. I hope the author to check the references one more time.

Flemings et al. (2024) ("Differentially Private Next-Token Prediction of Large Language Models", NAACL 2024) indeed proposes inference-time private prediction via output distribution mixing (PMixED), rather than training-time per-token gradient clipping with 1/Ni1/N_i sequence averaging. In Section 1, Section 3.1, Table 5 (table-conceptual-comparison), Appendix A, and Appendix Section E.4, we explicitly distinguish Flemings et al. (2024)'s inference-time PMixED from our training-time DP-NTP baseline (per-token gradient clipping with calibrated CglobalC_{\mathrm{global}} and active-length 1/Ni1/N_i averaging, removing any misattribution to Flemings et al., 2024), updated Table 5 PEFT attributions (Yu et al. 2022, Li et al. 2024) and inline LoRA/Ghost Clipping citations (Hu et al. 2022, Bu et al. 2023), and verified every bibliography entry against primary proceedings (Costa-jussà et al. Nature 2024; Khah et al. AISTATS 2026 PMLR v300; Yuan et al. DP-FedPUAC, Information Sciences 2026; OpenReview IDs for Li et al. 2022 and Yu et al. 2022; replacing a withdrawn preprint with Bu et al. ICML 2023; and adding Clark et al. 2020, Conneau et al. 2018, McSherry & Talwar 2007, Mironov et al. 2019, and Yousefpour et al. 2021).

AuthorsOpenReview ↗

Thank you for your assessment

Dear Reviewer wVAJ,

We sincerely thank you again for highlighting that identifying this new problem: how unequal multilingual subword fragmentation interacts with Differential Privacy (Entropic SNR Collapse), which makes our work potentially novel and interesting to the research community, and for your constructive evaluation of our ToPA framework. Following your actionable suggestions, we updated our empirical ablations, baseline positioning, and bibliography in the finalized manuscript:

  • Following your feedback to clarify what ToPA contributes beyond combining standalone primitives (Q1 — Sec. 3.2, 4.1–4.2, Table 2, Figs. 2, 3a–b, App. E.3, Table 10): We added a 2×22\times 2 factorial ablation across four models (1B–9B) showing why partial combinations of existing primitives fail to resolve Entropic SNR Collapse unless unified in ToPA: (i) Ghost Clipping + 1/Ni1/N_i averaging (DP-NTP) retains active-length signal attenuation, trailing ToPA across all four models in Step SNR and matched η\eta translation quality; (ii) 1/Nmax⁡1/N_{\max} aggregation + sequence clipping without token clipping (Config 2a) degrades sharply under 4.41×–7.17× intra-sequence token spikes (e.g., 16.37 → 10.75 spBLEU on 9B-IT); and (iii) sequence-level AdaClipping leaves ~50% of sequences unclipped under 1/Ni1/N_i while trailing ToPA in Step SNR (1B, 2B, 8B).
  • Following your suggestion to check Flemings et al. (2024) and thoroughly re-verify all references (Q2 — Sec. 1, 3.1, App. A (Table 5), E.4, iclr2027.bib): We explicitly distinguished Flemings et al. (2024)'s inference-time private prediction (PMixED) from our training-time DP-NTP baseline, updated all PEFT and Ghost-Clipping attributions (Hu et al. 2022, Yu et al. 2022, Bu et al. 2023, Li et al. 2024), and verified every one of the 33 bibliography entries against primary venue proceedings.

As the discussion window closes within 24 hours, please let us know if you have any further questions! Thank you again for your time and helpful suggestions.

Reviewer_vFLJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper indentifies and addresses a degradation problem in DP multilingual fine-tuning of language models, named as Entropic SNR Collapse. The problem arises from sequence-level averaging existing in vanilla DPSGD. The authors propose ToPA, which (i) performs unified token-wise “ghost clipping” of rank-1 linear projection contributions without per-token Jacobian materialization, (ii) aggregates token updates using a fixed anchor NmaxN_{max} to avoid 1/Ni1/N_i attenuation and length metadata leakage, and (iii) privately calibrates token clipping thresholds with the Exponential Mechanism. Theoretical results include a conditional O(NL−2)\mathcal{O}(N_L^{-2}) SNR upper-bound under semantic conservation, sensitivity guarantees, end-to-end DP, and a non-convex convergence bound.

Critical strengths
  • The paper identifies a mechanism (Entropic SNR Collapse) that connects multilingual tokenization fragmentation with DP gradient averaging and noise addition, yielding a simple conditional SNR upper-bound under a semantic conservation assumption.

  • Fixed-anchor aggregation is a simple but effective design that decouples token sensitivity control from sequence-level normalization, shifting optimization toward a token-uniform objective.

  • Experimental results show the method’s effectiveness. Importantly, the work presents failure cases on highly fragmented scripts (e.g., Thai, Arabic), where the proposed approach yields the largest benefits.

Critical weaknesses
  • The writing of the paper is hard to follow in some parts (e.g. step 1 of section 2.1) and needs to be improved. I read some parts multiple times to fully understand them.

  • The $\mathcal{O}(N_L^{-2}) $ SNR result rests on a strong semantic conservation assumption. While stated as a stylized bound, its practical tightness and applicability are unclear.

  • The ghost clipping guarantee is stated in terms of W′=W+αBA/rW' = W + \alpha BA/r, but the mapping to LoRA parameter gradients (AA and BB) involves additional linear transformations. The paper states that boundedness carries over but does not fully quantify operator-norm effects or give a tight cross-parameter bound.

  • Some reported translation scores (SacreBLEU in the 80–96 range with only 1k training pairs) appear unusually high for FLORES-200, raising questions about evaluation details. These very strong absolute numbers may mask relative differences.

  • The DP-NTP baseline collapsing to 0.00 BLEU on multiple settings suggests either a methodological mismatch or potential misconfiguration. While the paper justifies the collapse via 1/Ni1/N_i aggregation effects, more careful hyperparameter tuning (e.g., learning rates, optimizer states, loss reduction details) are needed for fairness.

  • The utility function and sensitivity bound of the Exponential Mechanism are briefly discussed. The paper should formalize the utility function (e.g., negative absolute deviation from target quantile) and detail its global sensitivity with respect to adding/removing one record, justifying the stated NmaxN_{max} bound. Also, clarify the candidate grid design and its effect on DP utility loss.

Questions for authors
  • As stated in lemma 1, the token gradient clipping guarantees a bounded sensitivity for the sequence gradient. Why is the global clipping needed? (line 156 and line 11 of Algorithm 1).

  • Having the global clipping for bounded sequence sensitivity, why is the token gradient clipping needed? In order to avoid the SNR collapse, why not to use the fixed-anchor aggregation followed by a global sequence gradient clipping? Intuitively, I am yet not convinced that the token clipping is also needed for utility improvements.

  • Can you provide a formal inequality showing that clipping per-token W’-gradients to CtokenC_{token} ensures bounded per-token updates for AA and BB (including α/r and operator norms)? Are there conditions on AA or BB (e.g., norm constraints) required to keep the bound tight?

  • DP-NTP baseline: can you share results showing nontrivial performance under reasonable configurations? If collapse persists, can you provide a diagnostic isolating the exact failure mode beyond the 1/Ni1/N_i argument?

  • Fixed-anchor sensitivity and fairness: How sensitive are results to NmaxN_{max} (e.g., 256, 512, 1024)? Does token-uniform risk systematically over-emphasize long records, and have you evaluated potential adverse effects on short-sequence languages or tasks?

  • I have mentioned my other concerns in the "Weaknesses".

11 comments
AuthorsOpenReview ↗

Thanks for actionable suggestions

Q1. As stated in lemma 1, the token gradient clipping guarantees a bounded sensitivity for the sequence gradient. Why is the global clipping needed? (line 156 and line 11 of Algorithm 1).

Under the baseline setting Cglobal=CˉtokenC_{\mathrm{global}}=\bar{C}_{\mathrm{token}} (written C~≡Cˉtoken\widetilde{C} \equiv \bar{C}_{\mathrm{token}} in Appendix Sections B & E), sequence clipping in Algorithm 1 (line 9) is indeed inactive (0.0% clipped across all four models, Config 4a in Table 1) since Lemma 1 guarantees ∥Gi∥2=∥1Nmax⁡∑t=1Nigˉi,t∥2≤NiNmax⁡C~≤C~\lVert G_i \rVert_2 = \lVert \frac{1}{N_{\max}}\sum_{t=1}^{N_i}\bar{g}_{i,t} \rVert_2 \leq \frac{N_i}{N_{\max}}\widetilde{C} \leq \widetilde{C}; line 9 is retained solely to certify exact per-record sensitivity Δ2=Cglobal\Delta_2 = C_{\mathrm{global}} whenever a tighter post-processed Cglobal<C~C_{\mathrm{global}} < \widetilde{C} is chosen (Theorem 1, Section 4.1; Appendix Sections B.5 & E).

Empirically, rapid cross-token decorrelation causes summed token gradients to cancel: across 1B–8B (Nmax⁡=256N_{\max}=256), ∥Gi∥2/C~\|G_i\|_2/\widetilde{C} averages only 0.0246–0.0301 (max <0.05). Setting Cglobal=C~/Nmax⁡C_{\mathrm{global}}=\widetilde{C}/\sqrt{N_{\max}} (4b) cuts noise variance by 256×256\times while clipping ≤0.1%\le 0.1\% of records, and our default Cglobal=0.30 C~/Nmax⁡C_{\mathrm{global}}=0.30\,\widetilde{C}/\sqrt{N_{\max}} (4c, using fixed public c0=0.30c_0=0.30 at zero extra privacy cost, certified by line 9 with 57.7%–89.7%57.7\%\text{--}89.7\% clipping) cuts Gaussian noise variance by 2,844×2,844\times relative to C~\widetilde{C} (Sections 3.2 & 4.1, Table 2, Figures 2c & 3a; Appendix Table 10), lifting Step SNR from 3.83×10−83.83\times 10^{-8} to 7.36×10−57.36\times 10^{-5} (1B) and spBLEU / chrF++ from 2.14 / 8.03 to 6.58 / 22.76 (1B) and 7.16 / 21.00 to 16.37 / 32.29 (9B-IT, Table 1).

Q2: Having the global clipping for bounded sequence sensitivity, why is the token gradient clipping needed? In order to avoid the SNR collapse, why not to use the fixed-anchor aggregation followed by a global sequence gradient clipping? Intuitively, I am yet not convinced that the token clipping is also needed for utility improvements.

Global sequence clipping rescales GiG_i after summation without changing its direction, whereas token clipping suppresses heavy-tailed intra-sequence outlier spikes before summation and concentrates ∥Gi∥2\|G_i\|_2 for one-shot analytical calibration (Sections 3.2 & 4.1–4.2 (Findings 5–6), Table 2, Figures 2a–b & 3a–b; Appendix Sections B.2 & E.3, Table 10):

  1. Intra-sequence spikes (2a vs. 4c): Unclipped intra-sequence token norms exhibit 4.41x–7.17x mean max-to-median spikes (max 22.23x). Removing token clipping under identical 1/Nmax⁡1/N_{\max} aggregation (2a) drops spBLEU / chrF++ from 6.58 / 22.76 to 3.97 / 14.51 (1B), 15.21 / 34.10 to 9.99 / 25.19 (2B), 19.10 / 37.63 to 17.33 / 34.86 (8B), and 16.37 / 32.29 to 10.75 / 25.64 (9B-IT)—and remains consistently inferior at η=10−3\eta=10^{-3} parity (Table 1).
  2. One-shot calibration vs. oracles (1b, 2b) and AdaClipping (1c): Tight sequence-only bounds (1b, 2b) saturate clipping (81%–100%81\%\text{--}100\%), where ClipCg(1Zi∑tgi,t)=Cg∑tgi,t∥∑tgi,t∥2\mathrm{Clip}_{C_g}(\frac{1}{Z_i}\sum_t g_{i,t}) = C_g \frac{\sum_t g_{i,t}}{\lVert \sum_t g_{i,t} \rVert_2} cancels Zi∈{Ni,Nmax⁡}Z_i \in \lbrace N_i, N_{\max} \rbrace without isolating outlier tokens and required non-private oracle tuning (ε1=0\varepsilon_1=0), whereas token clipping concentrates ∥Gi∥2\lVert G_i \rVert_2 so ToPA sets Cglobal=0.30 C~/Nmax⁡C_{\mathrm{global}} = 0.30\,\widetilde{C}/\sqrt{N_{\max}} in one private shot (ε1=0.40,σ=0.9668\varepsilon_1=0.40, \sigma=0.9668). At default LR, AdaClipping (1c, ε1=0.20,σ=0.9448\varepsilon_1=0.20, \sigma=0.9448) edges 4c by 0.11–0.75 spBLEU via an ≈9×\approx 9\times looser threshold (Cg≈2.9C_g \approx 2.9, leaving ≈50%\approx 50\% of sequences unclipped under 1/Ni1/N_i) and smaller ε1\varepsilon_1, while 4c achieves higher Step SNR on 1B, 2B, and 8B. At matched η=10−3\eta=10^{-3} parity across fixed-schedule methods (1a, 2a, 3, 4c), 4c achieves top Step SNR across all four scales, beats 2a and DP-NTP (3) on all four models, beats Vanilla DP-SGD (1a) on 1B, 8B, and 9B-IT, and matches within 0.04 spBLEU on Gemma-2-2B (16.44 vs. 16.48, ROUGE-L 0.4510 vs. 0.4499).

(See Table 1 below)

Table 1: 2×22\times 2 Factorial Ablation (Aggregation ×\times Token Clipping) and CglobalC_{\mathrm{global}} Sensitivity on Disjoint FLORES-200 devtest (ε=4.0,δ=10−5\varepsilon=4.0, \delta=10^{-5}; bold: best private per LR regime; †^\daggeroracle Cg,ε1=0C_g, \varepsilon_1=0)

ConfigurationAggregationToken ClipCglobalC_{\mathrm{global}}1B spBLEU / chrF++2B spBLEU / chrF++1B Step SNR8B spBLEU / chrF++9B-IT spBLEU / chrF++8B Step SNR
1a. Vanilla DP-SGD1/Ni1/N_iOff5.05.06.05 / 21.2815.35 / 34.414.13×10−54.13\times 10^{-5}18.45 / 36.6415.34 / 31.061.48×10−51.48\times 10^{-5}
1b. Vanilla DP-SGD (Oracle CgC_g, ε1=0\varepsilon_1=0)†^\dagger1/Ni1/N_iOff1.01.06.81 / 23.4115.71 / 34.907.11×10−57.11\times 10^{-5}19.74 / 38.6419.00 / 35.282.59×10−52.59\times 10^{-5}
1c. AdaClipping (ε1=0.20\varepsilon_1=0.20)1/Ni1/N_iOffOnline (≈2.9\approx 2.9)6.69 / 23.0515.51 / 34.775.95×10−55.95\times 10^{-5}19.24 / 38.1817.12 / 33.112.25×10−52.25\times 10^{-5}
2a. Fixed-Anchor Only (Loose CgC_g)1/Nmax⁡1/N_{\max}Off2.02.03.97 / 14.519.99 / 25.194.57×10−64.57\times 10^{-6}17.33 / 34.8610.75 / 25.641.09×10−61.09\times 10^{-6}
2b. Fixed-Anchor Only (Oracle CgC_g, ε1=0\varepsilon_1=0)†^\dagger1/Nmax⁡1/N_{\max}Off0.300.306.72 / 23.3215.86 / 35.177.09×10−57.09\times 10^{-5}19.72 / 38.5918.91 / 35.182.51×10−52.51\times 10^{-5}
3. DP-NTP (Default LR)1/Ni1/N_iOn (C~\widetilde{C})0.15 C~0.15\,\widetilde{C}6.44 / 22.4914.95 / 33.666.92×10−56.92\times 10^{-5}19.10 / 37.8616.30 / 32.182.50×10−52.50\times 10^{-5}
4a. ToPA (Theorem 1: Cg=C~C_g = \widetilde{C})1/Nmax⁡1/N_{\max}On (C~\widetilde{C})C~\widetilde{C}2.14 / 8.030.59 / 5.683.83×10−83.83\times 10^{-8}17.13 / 34.457.16 / 21.001.34×10−81.34\times 10^{-8}
4b. ToPA (Mean-Field: Cg=C~/Nmax⁡C_g = \widetilde{C}/\sqrt{N_{\max}})1/Nmax⁡1/N_{\max}On (C~\widetilde{C})C~/Nmax⁡\widetilde{C}/\sqrt{N_{\max}}4.44 / 15.889.11 / 23.579.51×10−69.51\times 10^{-6}17.71 / 35.6510.86 / 25.745.38×10−65.38\times 10^{-6}
4c. Adaptive ToPA (Ours, Default LR, ε1=0.40\varepsilon_1=0.40)1/Nmax⁡1/N_{\max}On (C~\widetilde{C})0.30 C~/Nmax⁡0.30\,\widetilde{C}/\sqrt{N_{\max}}6.58 / 22.7615.21 / 34.107.36×10−57.36\times 10^{-5}19.10 / 37.6316.37 / 32.292.76×10−52.76\times 10^{-5}
1a. Vanilla DP-SGD (η=10−3\eta=10^{-3} Parity)1/Ni1/N_iOff5.05.07.30 / 24.1816.48 / 36.082.77×10−52.77\times 10^{-5}20.26 / 39.0825.35 / 43.186.21×10−66.21\times 10^{-6}
2a. Fixed-Anchor Only (η=10−3\eta=10^{-3} Parity)1/Nmax⁡1/N_{\max}Off2.02.05.34 / 19.1015.20 / 34.425.66×10−65.66\times 10^{-6}19.19 / 37.8819.15 / 35.542.12×10−62.12\times 10^{-6}
3. DP-NTP (η=10−3\eta=10^{-3} Parity)1/Ni1/N_iOn (C~\widetilde{C})0.15 C~0.15\,\widetilde{C}7.19 / 24.2315.57 / 34.883.73×10−53.73\times 10^{-5}20.48 / 39.1825.19 / 43.101.07×10−51.07\times 10^{-5}
4c. Adaptive ToPA (Ours, η=10−3\eta=10^{-3} Parity)1/Nmax⁡1/N_{\max}On (C~\widetilde{C})0.30 C~/Nmax⁡0.30\,\widetilde{C}/\sqrt{N_{\max}}7.38 / 24.5616.44 / 36.014.22×10−54.22\times 10^{-5}20.54 / 39.4125.50 / 43.511.14×10−51.14\times 10^{-5}

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and reviewers (Reviewer vFLJ, Reviewer wVAJ, and Reviewer rV2L) for their constructive feedback and for acknowledging the novelty and research value of our core contribution: studying how tokenization impacts Differential Privacy in multilingual settings. As highlighted in the reviews, identifying this previously overlooked problem makes this work novel and valuable to the community as how unequal subword fragmentation across languages interacts with sequence-level 1/Ni1/N_i gradient averaging and length-independent DP noise (Entropic SNR Collapse) to create structural cross-lingual utility disparities. Uncovering, theoretically formalizing, and empirically isolating this multilingual tokenization-DP interaction is the central contribution and novelty of our paper, and ToPA is purpose-built around resolving this mechanism.

To address every methodological and empirical question raised by the reviewers, we executed a unified, additional benchmark suite across Llama-3.2-1B, Gemma-2-2B, Llama-3-8B, and Gemma-2-9B-IT adapting the four self-attention projections (Wq,Wk,Wv,WoW_q, W_k, W_v, W_o; r=8,α=16r=8, \alpha=16) via exact per-token LoRA Ghost-Clipping DP-SGD and Opacus Rényi DP accounting (ε=4.0,δ=10−5\varepsilon=4.0, \delta=10^{-5}; B=32B=32, 55 epochs) across strictly disjoint FLORES-200 translation (897897 4-language dev[0:897] training records, 100100 disjoint dev[897:997] calibration records at frozen W0W_0 with ε1=0.40,ε2=3.60\varepsilon_1=0.40, \varepsilon_2=3.60, and 1,0121,012 held-out devtest pairs in each of English$\to$Thai and English$\to$Arabic), Official TyDi QA GoldP extractive QA (1,2001,200 train / 150150 disjoint cal / 1,1131,113 held-out validation items across 6 languages: en, id, sw, ar, bn, te), and Multilingual XNLI (1,0001,000 train / 120120 disjoint cal / 2,0002,000 held-out test items across en, ar, sw, th under both Ni=1N_i=1 single-token and Ni∈[2,10]N_i \in [2,10] native-script verbalizers). We have revised Sections 1–5 (including Figure 1 in Section 1, direct (A,B)(A,B) LoRA Ghost Clipping in Section 2.1 Step 1, Tables 1–4 & Figures 2–3 across Sections 3–4, and Findings 1–8 across Sections 3–4) and updated Appendix Sections A–E (Tables 5–17) accordingly:

  1. Sequence-Level Definition and Empirical Validation of Semantic Signal Scaling (Reviewer vFLJ Q6.2; Reviewer rV2L Q1, Q2 — Section 1 (Figure 1b–c), Section 3.2 (Findings 1–2), Tables 1 & 3A, Figure 2d; Appendix Section E.6, Tables 16–17):
  • Over parallel realizations xL(z)x_L(z) of latent semantic propositions z∼PZz \sim \mathcal{P}_{\mathcal{Z}} at a fixed LoRA state WwarmW_{\mathrm{warm}} (warmed up for 15 non-private steps so B≠0B \neq 0 and both ∇AL\nabla_A\mathcal{L} and ∇BL\nabla_B\mathcal{L} are active, avoiding degenerate ∇AL(W0)=0\nabla_A\mathcal{L}(W_0)=0 at B0=0B_0=0), we formalize the cumulative semantic signal SL:=Ez∼PZ[∑t=1NL(z)gt(xL(z);Wwarm)]S_L := \mathbb{E}_{z \sim \mathcal{P}_{\mathcal{Z}}}[\sum_{t=1}^{N_L(z)} g_t(x_L(z); W_{\mathrm{warm}})] and mean token signal sˉL:=SL/NˉL\bar{s}_L := S_L / \bar{N}_L (gt(xL(z))=sˉL+ξt(z)g_t(x_L(z)) = \bar{s}_L + \xi_t(z), where Ez[∑t=1NL(z)ξt(z)]=SL−Ez[NL(z)]sˉL=0\mathbb{E}_z[\sum_{t=1}^{N_L(z)} \xi_t(z)] = S_L - \mathbb{E}_z[N_L(z)]\bar{s}_L = 0 identically), avoiding any cross-token index alignment.
  • Across 997997 parallel FLORES-200 dev sentences in 8 languages spanning native Llama-3 128k BPE and Gemma-2 256k SentencePiece target lengths NL∈[26.81,66.53]N_L \in [26.81, 66.53] (Table 5 below; Main Table 1 & Appendix Table 16), sample-averaged ∥SL∥2\|S_L\|_2 never grows linearly with token length (α^S=0.774\hat\alpha_S = 0.774 on 1B, −1.561-1.561 on 2B—reflecting high English norm 206.20206.20 leveraged across 2B's narrow non-English length band NL∈[30.38,49.77]N_L \in [30.38, 49.77] while non-English scripts stay bounded in [10.61,36.79][10.61, 36.79]—and −0.410-0.410 on 8B, with a 6-language conservation plateau ∥SL∥2∈[11.83,18.34]\|S_L\|_2 \in [11.83, 18.34] on 8B). Consequently, without imposing conservation, unconditional population DP SNR under 1/NL1/N_L averaging drops from English to Thai by 11.1×11.1\times on 1B (3.52×10−53.52\times 10^{-5} →\to 3.16×10−63.16\times 10^{-6}) and 71.2×71.2\times on 8B (4.36×10−54.36\times 10^{-5} →\to 6.13×10−76.13\times 10^{-7}). Three controlled single-language and cross-tokenizer studies (Table 8 below; Figure 1c, Main Table 3A & Appendix Table 17) further confirm this mechanism with semantics held fixed.
  1. 2×22\times 2 Factorial Ablation and Objective-Matched Controls Separating 1/Nmax⁡1/N_{\max} Reweighting from ToPA's Privacy Mechanism (Reviewer vFLJ Q1, Q2; Reviewer wVAJ Q1; Reviewer rV2L Q3 — Sections 3.2 & 4.1–4.2 (Findings 5–6), Table 2 & Figures 2, 3a–b; Appendix Sections B.2 & E.3, Tables 10–11):

    • In non-private fine-tuning (σ=0\sigma=0), sequence-uniform (1/Ni1/N_i) and token-uniform (1/Nmax⁡1/N_{\max}) objectives yield nearly identical disjoint FLORES-200 translation quality (8.72 / 27.80 vs. 8.89 / 27.94 spBLEU/chrF++ on 1B; 20.00 / 39.31 vs. 19.60 / 39.01 on 8B, Table 9 below; Main Table 2 & Appendix Table 11), confirming that length reweighting alone does not drive the gain.
    • Under DP (ε=4.0\varepsilon=4.0), holding the 1/Nmax⁡1/N_{\max} objective fixed, removing token clipping (Config 2a: Fixed-Anchor Only, Table 1 below; Main Table 2 & Appendix Table 10) drops disjoint spBLEU / chrF++ from 6.58 / 22.76 to 3.97 / 14.51 (1B), 15.21 / 34.10 to 9.99 / 25.19 (2B), 19.10 / 37.63 to 17.33 / 34.86 (8B), and 16.37 / 32.29 to 10.75 / 25.64 (9B-IT) because heavy-tailed outlier tokens (4.41×–7.17×4.41\times\text{--}7.17\times mean max-to-median norm) skew the summed gradient direction.
    • While Lemma 1 bounds ∥Gi∥2≤C~\lVert G_i \rVert_2 \le \widetilde{C} (0.0% global clipping at Cglobal=C~C_{\mathrm{global}}=\widetilde{C}), cross-token cancellation leaves E[∥Gi∥2]≈0.025–0.030 C~\mathbb{E}[\lVert G_i \rVert_2] \approx 0.025\text{--}0.030\,\widetilde{C}; calibrating Cglobal=0.30 C~/Nmax⁡C_{\mathrm{global}} = 0.30\,\widetilde{C}/\sqrt{N_{\max}} (with fixed model-agnostic public constant c0=0.30c_0=0.30) reduces Gaussian noise variance by 2,844×2,844\times relative to Cglobal=C~C_{\mathrm{global}}=\widetilde{C} (Main Table 2, Figures 2c & 3a, & Appendix Table 10), achieves the highest Step SNR among fixed-schedule methods across all four models (and higher than AdaClipping on 1B, 2B, and 8B), and at matched η=10−3\eta=10^{-3} parity (7.38 / 24.56 on 1B, 16.44 / 36.01 on 2B, 20.54 / 39.41 on 8B, 25.50 / 43.51 on 9B-IT) achieves the highest Step SNR across all four scales, outperforms matched-η\eta Fixed-Anchor Only (2a) and DP-NTP (3) across all four models, outperforms Vanilla DP-SGD (1a) on 1B, 8B, and 9B-IT, and matches within 0.04 spBLEU on Gemma-2-2B (16.44 / 36.01 vs. 16.48 / 36.08).
  2. Disjoint FLORES-200 (dev →\to devtest) Evaluation with Official spBLEU & chrF++, and Diagnostic/Tuning of DP-NTP (Reviewer vFLJ Q4, Q6.3; Reviewer wVAJ Q2 — Sections 3.1–3.3 & 4.1, Tables 2 & 4, Figures 2 & 3a; Appendix Sections A, E.2 & E.4, Tables 5, 9 & 12):

    • Re-evaluating strictly on disjoint FLORES-200 dev (897897 train ++ 100100 disjoint calibration =997= 997) →\to devtest (1,0121,012 held-out test pair per direction) with official 250k XLM-R SentencePiece spBLEU and chrF++ (Table 6 below; Main Tables 2 & 4A, Appendix Table 9) replaces inflated overlapping Char-BLEU scores (85.39 on 8B, 83.29 on 9B-IT) with standard low-resource MT ranges (6.05–24.21 spBLEU across Table 6; 0.51–25.50 across all diagnostic and η=10−3\eta=10^{-3} sweeps).
    • Fixing two implementation bugs in legacy DP-NTP (duplicate 1/Ni21/N_i^2 division and uncalibrated Cglobal=C~C_{\mathrm{global}}=\widetilde{C}), analyzing AdamW second-moment noise locking, and sweeping η∈[10−4,10−3]\eta \in [10^{-4}, 10^{-3}] (Table 3 below; Sections 1 & 4.1 (Finding 5), Table 4B, Figure 3a, & Appendix Section E.4, Table 12) restores strong nontrivial DP-NTP utility across all four models. We also verified that Flemings et al. (2024) is explicitly distinguished as inference-time PMixED rather than our training-time DP-NTP baseline (Section 1, Section 3.1, Appendix A (Table 5), & Appendix E.4) and updated our bibliography.
  1. Fixed-Anchor Nmax⁡N_{\max} Sensitivity, Short-Sequence Fairness, and Cross-Task Generalization to Official TyDi QA GoldP & XNLI (Reviewer vFLJ Q5 — Section 3.3 (Findings 3–4), Tables 3B & 4C; Appendix Sections C & E.5, Tables 6 & 13):

    • Across Nmax⁡∈{128,256,512,1024}N_{\max} \in \{128, 256, 512, 1024\} (Table 4 below; Main Table 3B & Appendix Table 13), any Nmax⁡∈[256,512]N_{\max} \in [256, 512] eliminates truncation (0.00%) and maintains stable quality (6.58 vs. 6.39 spBLEU on 1B; 19.10 vs. 19.07 on 8B), with lower Short-tertile perplexity than Vanilla DP-SGD across all four non-English FLORES languages (Appendix Section E.5, Table 13).
    • Generalizing sequence calibration to prompt-masked tasks via Task-Anchor ToPA (Cglobal=max⁡(0.30Nout,max,0.50) C~/Nmax⁡C_{\mathrm{global}} = \max(0.30\sqrt{N_{\mathrm{out,max}}}, 0.50)\,\widetilde{C}/N_{\max}, which reduces identically to 0.30 C~/Nmax⁡0.30\,\widetilde{C}/\sqrt{N_{\max}} when Nout,max=Nmax⁡N_{\mathrm{out,max}}=N_{\max} on FLORES-200), ToPA outperforms Vanilla DP-SGD and AdaClipping across all four models on Official TyDi QA GoldP (+2.23 Macro F1 and +4.15 Bengali F1 on 1B; +0.97 F1 / +4.38 chrF++ on 2B; +0.41 F1 on 8B; 80.04 F1 / 67.02 EM vs. 79.70 / 66.45 on 9B-IT) while matching DP-NTP Step SNR identically on single-token (Ni=1N_i=1) XNLI across all four models and lifting native-label Thai accuracy on 1B (+0.80%), 8B (+2.40%), and 9B-IT (+0.20%; Main Table 4C & Appendix Section C, Table 6).
  2. LoRA Operator-Norm Inequality vs. Direct (A,B)(A,B) Clipping, Exponential Mechanism Public Grid Analysis, and Clarity Revisions (Reviewer vFLJ Q3, Q6.1, Q6.4 — Section 2.1 Step 1, Sections 4.2–4.4 (Findings 6–8), Table 4B & Figure 3b–d; Appendix Sections B.4–B.5, D, E.1 & E.6, Tables 7–8, 14–15):

    • We show evidence ∥gˉA,t∥F2+∥gˉB,t∥F2≤κlCtoken\sqrt{\lVert \bar{g}_{A,t} \rVert_F^2 + \lVert \bar{g}_{B,t} \rVert_F^2} \le \kappa_l C_{\mathrm{token}} alongside zero-slack direct (A,B)(A,B) clipping qtq_t at 1.00x–1.02x Default-LR runtime overhead (Table 2 below; Section 2.1 Step 1, Sections 4.2 & 4.4 (Findings 6 & 8), Main Table 4B, Figure 3b & 3d, Appendix Sections E.1 & E.6, Tables 8 & 14), formalize the Exponential Mechanism count-deviation utility u(c,D)=−∣R(c,D)−τM(D)∣u(c,D) = -|R(c,D) - \tau M(D)| at strict W0W_0, decompose sensitivity from full-length max⁡(τ,1−τ)Nmax⁡=204.8≤256\max(\tau,1-\tau)N_{\max} = 204.8 \le 256 to Δu=max⁡(τ,1−τ)Ncal=51.2\Delta u = \max(\tau, 1-\tau)N_{\mathrm{cal}} = 51.2 (Ncal=64N_{\mathrm{cal}}=64, Table 7 below; Section 4.3 (Finding 7), Figure 3c, & Appendix Sections B.5, E.1, Table 15), bridge main-text notation (C~≡Cˉtoken\widetilde{C} \equiv \bar{C}_{\text{token}}, Gi≡g~iG_i \equiv \tilde{g}_i) in Appendix Sections B, B.3 & E, and integrate a clear three-stage exposition directly into Section 2.1 Step 1.
AuthorsOpenReview ↗

The rebuttal deadline is approaching.

Dear all reviewers,

Thank you for your time and effort in assessing our paper. The author response window closes within 48 hours. We would like to hear your follow-up questions and actionable suggestions.

Open questions are aslo welcome.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.