- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Existing defenses face a dilemma: they cannot protect private tokens while retaining training utility. To address this, the authors propose ALIASGUARD, a surrogate-based defense. The client first identifies the private tokens in each training sequence and removes them, leaving the unprotected context C. An external language model then generates surrogate replacements for the private tokens using only C. The client runs each selected surrogate sequence through the head and averages the resulting smashed data; it likewise averages the surrogates’ one-hot labels to compute the gradients. Empirically, BF recovery of private tokens drops from about 83–100% for the baselines to 0.35–30.7% (Table 1a), while task utility is largely retained.
Critical strengths
The problem is real. Passive inference in split-learning fine-tuning is a real threat. The paper clearly explains why existing perturbation defenses fail: each one leaves either the forward or the backward channel exploitable.
Good empirical privacy results on the evaluated datasets. Across three datasets and three models, BF recovery of private tokens drops from about 83–100% for the baselines to 0.35–30.7% (Table 1a).
Formal privacy guarantees. The authors provide theoretical and empirical proofs of their method.
Critical weaknesses
My main concerns are novelty, experiments, and scope.
Specifically, the work lacks novelty. Replacing the critical token is a well-known method, even in the unlearning domain.
Second, the experiments are small-scale. The largest model is Qwen2.5-3B, and each run trains on only 169 sequences.
Finally, the scope is limited. The method requires accurate private-token detection beforehand and cannot be extended to coding, math, or other domains.
Questions for authors
Can you report utility with larger models, larger training sets, and more seeds? Can you justify your novelty?
3 comments
Adding experiments with larger models, longer training, and more seeds
We have completed GSM8K experiments with larger models, longer training, and more seeds.
First, we extend the evaluation from Qwen2.5-3B to Qwen3.5-9B. Each run performs 2,048 updates on 2,048 distinct training sequences, 12.1 times the original 169 sequences. We compare AliasGuard and NoDef across five paired random seeds, giving ten completed runs.
Second, we test whether attackers can recover private tokens. F uses smashed data (the Head’s output) and the Trunk’s output; B uses gradients from the backward pass; BF uses both. Recovery is the percentage of private-token positions guessed correctly. We evaluate 546 positions in 128 training sequences. Averaged across five seeds, recovery falls from 43.63% with NoDef to 1.87% with AliasGuard for F, from 18.94% to 1.03% for B, and from 47.95% to 1.87% for BF.
For task utility, we measure the percentage of correct answers on all 1,319 GSM8K test questions. Averaged across five seeds, AliasGuard answers 23.11% correctly, versus 22.71% for NoDef.
The above results demonstrate that the conclusion of original manuscript still holds under larger models, longger training and more seeds.
The scope
The reviewer first raises the limitation of accurate private-token detection. However, many data owners can identify sensitive information before training. For example, hospitals, banks, and enterprises can mark patient names, account numbers, and confidential identifiers. Clients in split learning and federated learning can likewise identify private tokens in their own sequences. Thus, accurate private-token identification is a practical requirement that can be met across multiple application domains. Investigating how to use this information for privacy protection is valuable, and this work provides an example under SL.
We then address the reviewer's concern that the method cannot be extended to coding, math, or other domains. In these domains, models need to learn general knowledge and task-specific rules. During SL fine-tuning, our goal is to allow models to learn these rules while protecting private information in the training sequences. For example, companies' source code may contain confidential identifiers that are not necessary for learning how functions work. The reviewer's unlearning and other domains also need the trained models to forget the confidential identifiers without forgetting the general knowledge. SL fine-tuning does this at the training duration.
Coding: We tested AliasGuard on CodeAlpaca-H4. Our goal is to protect private identifiers without preventing models from learning the logic of functions in code. We evaluate generated code using lexical F1 (word overlap), together with Python grammar and compilation checks (Appendix G.3). The results in Table 1 demonstrate AliasGuard's ability to support code generation while protecting designated private identifiers.
Math: GSM8K evaluates mathematical reasoning while protecting person names and preserving numerical quantities. In our new 9B experiment, attackers using smashed data and gradients correctly recovered 47.95% of private-token positions with NoDef, compared with 1.87% with AliasGuard, averaged across five paired seeds. Together with Table 1, these results demonstrate that AliasGuard can protect person names during mathematical fine-tuning.
Other domains: We also tested dialogue summarization on DialogSum. Our goal is to train the model to summarize conversations while protecting person names. AliasGuard replaces these names and preserves the other words. With Qwen2.5-1.5B, AliasGuard achieves a ROUGE-L score of 41.57, compared with 41.19 for NoDef (Table 1). ROUGE-L measures word-sequence overlap between generated and reference summaries. These results demonstrate that AliasGuard can also support dialogue summarization.
In summary, our objective is to protect designated private tokens, not to prevent models from learning general mathematical, scientific, or programming knowledge.
The methodological novelty of AliasGuard
We thank the reviewer for raising this important question. Although surrogate replacement (SR) has been explored in machine unlearning and text anonymization, AliasGuard uses it differently in split learning (SL). We explain four crucial differences: the purpose of SR, the computation path, the training objective, and the resulting task utility and computational cost. The four differences give the benefits: AliasGuard provides an efficient training path, while maintaining the training utility and data privacy.
The purpose of SR
Computation path
In unlearning, it needs to generate N surrogate versions of S. For example, the three surrogate sequences are “Bob was born in the USA”, “Carol was born in the USA”, and “David was born in the USA”. With the original model fixed, it performs three forward passes using the three sequences, obtaining the probabilities of “USA” as 10%, 20%, and 30%, respectively. It then averages the three probability distributions and uses S as input to a trainable copy of the model, which initially predicts “USA” with a probability of 95%. It performs one backward pass to move this probability 95% toward (10+20+30)%/3=20%, while matching the other next-word probabilities. Thus, it needs N+1 forward passes.
In text anonymization, it uses a privacy evaluator, a utility evaluator, and an optimizer to repeatedly change S into S′. Thus, it computational path is not related to model training.
In AliasGuard, users execute N forward passes through the Head, obtaining N Head outputs, i.e., N smashed data. They average the N outputs into one Head output, together with the corresponding training labels. The server uses this output to compute one forward pass through the Trunk, obtaining a Trunk output, whereas the user uses the Trunk output to compute one forward pass through the Tail.
Objectives
-
-
-
\mathrm{CE}(h(g(f(X_k))),Y_k).$$ - AliasGuard instead averages the $N$ Head outputs and their labels before computing one loss: $$L_{\mathrm{AG}}= \mathrm{CE}\left( h\left(g\left(\frac1N\sum_{k=1}^{N}f(X_k)\right)\right), \frac1N\sum_{k=1}^{N}Y_k \right).$$ Here, $f$, $g$, and $h$ are the Head, Trunk, and Tail; $X_k$ and $Y_k$ are each surrogate sequence and its training labels. Averaging before the Trunk and loss calculation is generally different from averaging $N$ complete losses. **Therefore, AliasGuard changes the training objective, raising the question of whether it can preserve task utility while reducing computation.** **Task utility and computational cost** On DialogSum, we compared AliasGuard with standard surrogate replacement using identical sets of three surrogates and three paired random seeds. - AliasGuard achieved higher ROUGE-L in all three seeds, averaging 39.931 versus 39.358 on standard surrogate replacement. Thus, changing the training objective did not reduce AliasGuard’s task utility in these experiments. - Meanwhile, update time decreased from 796.12 to 341.88 ms (2.33× faster), and communication decreased from 7.310 to 2.437 MiB (66.7% less). These results demonstrate that AliasGuard can reduce SL training costs while maintaining comparable task utility, despite using an more efficient training objective.In unlearning, the original model predicts “USA” with probabilities of 10%, 20%, and 30% after “Bob/Carol/David was born in”, respectively. Their average is 20%. Another model, initially predicting “USA” with 95% probability after “Alice was born in”, is trained to move its prediction toward 20% while matching the other averaged word probabilities.
In text anonymization, the objective is to change S into S′ so that Alice is difficult to identify.
In standard multi-surrogate SL that is directly deduced from the unlearning paradigm, the model computes a loss for each of the N surrogate sequences and averages these losses: