- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RouteTrace, a white-box membership inference attack against Mixture-of-Experts models. RouteTrace uses the alignment between router gradients and parameter updates observed between base and post-trained checkpoints. The method combines global alignment, layerwise information, and route-conditioned features to infer whether a candidate record was included in the post-training dataset.
The authors evaluate RouteTrace on three MoE model families and two datasets. The method shows better performance than some output-based MIA baselines, although LiRA and RMIA achieve higher AUC in the reported Qwen experiments when sufficient reference models are available.
Critical strengths
1. Transparent reporting and analysis of negative results. The authors transparently report the limitations of their proposed approach. RouteTrace has lower performance compared to LiRA and RMIA in the evaluated reference-based setting.
2. Investigation of membership leakage through MoE router updates. The paper investigates an important privacy concern in Mixture of Experts models by examining whether router parameter updates reveal membership information.
Critical weaknesses
1. Limited training scale and insufficient analysis of its impact on membership inference The experimental evaluation is conducted under small-scale fine-tuning settings. Specifically, Natural Questions uses 4,096 training samples and 256 optimizer steps, while Dolly uses only 1,024 samples and 64 optimizer steps. Both settings involve only a single epoch of fine-tuning. Since RouteTrace directly exploits the alignment between candidate gradients and router parameter updates, it is unclear whether the reported attack performance would remain effective under larger-scale fine-tuning settings.
Although Appendix H.4 includes a limited exposure comparison, it does not systematically show the effects of dataset size and training duration under the primary fine-tuning configuration.
2. Restricted fine-tuning setup and potential dependence on router updates. The primary experimental setup is updating the router matrices and rank-8 LoRA adapters on the attention Q/V projections while freezing the expert parameters. This setting is relatively restrictive and may favor RouteTrace, since the proposed attack directly exploits router parameter updates. It is unclear whether the reported effectiveness generalizes to other fine-tuning configurations, particularly those that also update expert parameters.
3. Incomplete comparison with strong reference-based MIA baselines.
The evaluation of LiRA and RMIA is restricted to two Qwen targets on Natural Questions. However, these methods achieve higher AUC than RouteTrace when sufficient reference checkpoints are available.
4. Insufficient discussion of shared experts. The evaluated MoE architectures differ in their use of shared experts. In particular, Qwen and DeepSeek-MoE incorporate shared experts alongside routed experts.
However, the paper does not clearly discuss the implications of shared experts for the proposed membership inference mechanism. Since shared experts are activated independently of the top-k routing decision, they may influence the representations and gradient signals used by RouteTrace.
5. Missing sensitivity analysis of LoRA rank. The primary experiments use a fixed LoRA rank of 8 for the attention Q/V adapters. However, the paper does not investigate how different LoRA ranks affect membership inference performance.
Since LoRA rank controls the adaptation capacity of the attention modules, varying this parameter may change the learned representations and the corresponding router gradients used by RouteTrace.
Questions for authors
- How does RouteTrace's performance change as the fine-tuning dataset size increases while keeping the model architecture and training configuration fixed?
- How sensitive is the attack to the number of fine-tuning steps or epochs?
- How does RouteTrace's performance vary across different LoRA ranks?
- Why are LiRA and RMIA evaluated only on two Qwen Natural Questions targets? Extending these evaluations to the remaining model families and datasets would provide a more comprehensive assessment of the results.
- What motivates the choice of updating router matrices and Q/V adapters while freezing all expert parameters?
- How does RouteTrace perform under alternative fine-tuning configurations, such as expert fine-tuning or full-model fine-tuning?
- How are shared experts and their associated components treated during fine-tuning and RouteTrace feature extraction?
2 comments
Thank you for your rebuttal. My questions about the fine-tuning setup and LiRA/RMIA were requests for further evaluation. Testing a setting where both routers and experts are updated would help assess whether the findings depend on freezing the experts. Extending the LiRA/RMIA comparison beyond the two Qwen NQ targets would also help assess RouteTrace’s relative performance, taking the different reference and shadow-model budgets into account.
The explanation of shared experts is helpful. Since they can still affect router gradients indirectly, a controlled analysis of their impact on attack performance would strengthen the explanation of the mechanism.
My main concerns remain unresolved. I am therefore maintaining my original rating.
Response on experimental scope
Thank you for identifying these scope limitations.
Training scale. The main experiments use one epoch: 4,096 training records for NQ and 1,024 for Dolly. The existing exposure contrast is not a systematic dataset-size or training-duration study. We did not test larger datasets or more epochs.
Fine-tuning configuration. The experiments update routers and rank-8 Q/V LoRA adapters while freezing experts. We did not evaluate expert fine-tuning, full-model fine-tuning, or other LoRA ranks. The current evidence is therefore limited to router-changing post-training.
LiRA/RMIA. The strict reference-based comparison covers two Qwen NQ targets. With 16 references, LiRA and RMIA have higher AUC than RouteTrace, which uses three shadows. We did not build equivalent reference banks for the remaining models and datasets.
Shared experts. Shared experts are not selected by the top-k router and therefore do not enter the selected/boundary row sets. They can still affect hidden representations, the candidate loss, and thus router gradients. The Qwen and DeepSeek-MoE results include this indirect effect, but we did not isolate shared experts in a separate ablation.