- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of DP-noise amplification in privacy-preserving split fine-tuning of LLMs. Building on the problem setting explored by prior work such as HiddenEcho, the paper proposes NIFT, a two-stage framework that aims to mitigate this issue at the backbone level rather than correcting noisy representations only after amplification has occurred. The method is evaluated on multiple text classification datasets and two LLM backbones, showing improved utility over HiddenEcho in most reported settings.
Critical strengths
strength1. The paper studies a practically relevant and meaningful problem. Continuing the line of work represented by HiddenEcho, this is an important problem for practical client–server LLM adaptation.
strength2. The motivation of making the backbone itself less sensitive to DP noise is intuitive. Conceptually, this is a sensible alternative to post-hoc denoising or hidden state correction, since a more noise-invariant backbone could reduce the burden on downstream client-side modules.
strength3. The method is relatively lightweight and compatible with the split-learning setting.
Critical weaknesses
weakness1. The main problem is that the author claims that Stage 1 achieves "robust representations", but what it actually optimizes is noise invariance, not "noise invariance while preserving clean semantic utility". Stage 2 can at most compensate for the information remaining after Stage 1, and it cannot put back the task-relevant information that Stage 1 has erased into the backbone. The current paper indirectly proves the utility is not damaged through downstream experiments, rather than ensuring it through the objective function itself. At the method level, this assumption is currently mainly supported by empirical results rather than being explicitly guaranteed by the objective.
weakness2. While Table 1 provides a relatively comprehensive comparison against GAN-DP, LDP, Split-and-Denoise, and HiddenEcho on Qwen2-1.5B, Table 2 reports only HiddenEcho and NIFT on Llama-3.2-1B. This substantially weakens the evidence for the claimed cross-backbone generality of NIFT. In particular, since the paper itself notes that the embedding scale, clipping radius, and resulting privacy behavior differ across backbone families, the relative ranking of LDP/SnD/HiddenEcho observed on Qwen cannot simply be assumed to hold on Llama. At minimum, I would expect the strongest relevant non-HiddenEcho baselines, especially LDP and Split-and-Denoise, to be evaluated on the second backbone as well.
weakness3. There is a lack of a representation-level experiment that directly proves that "hidden states truly become noise-insensitive". This is the aspect that this paper should present the most.
weakness4. As discussed in weakness 1, the Stage-1 objective only encourages consistency across noisy views and prevents degenerate collapse; it does not explicitly preserve task-relevant semantic information. This concern is reflected empirically in Table 3. On Financial Phrasebank, adding Stage 1 consistently reduces AUC across all reported privacy budgets compared with the Stage-2-only variant (0.890 -> 0.886 at \eta=100, 0.901 -> 0.885 at \eta=1000, 0.892 -> 0.881 at \eta=5000, and 0.896 -> 0.886 at \eta=6000).
Questions for authors
question 1: Is the additional server-side perturbation in Stage 1 applied to the \tilde e_0 that has already been client-side DP-perturbed, rather than generating multiple DP views independently from the clean embedding?
question 2: HiddenEcho is also based on the filtering of key transformer layers and the compression of hidden states, and does the proposed split bottleneck adapter have an essential difference from it?