- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies membership inference against fine-tuned Mixture-of-Experts LLMs using changes in expert routing. The proposed attack, RouteMIA, compares the experts selected by a public base model and its fine-tuned counterpart, using their Jaccard distance as a membership signal. It then calibrates this score against token identity and base-model routing margin using non-member reference data. The attack assumes that the target’s per-token expert selections can be recovered through a hardware side channel. Experiments cover three MoE models and four datasets under expert-only LoRA fine-tuning.
Critical strengths
S1. Studying membership leakage through expert routing is an interesting direction. The reported results suggest that changes in internal computation paths can carry some membership information without observing the target’s token probabilities. This observation can be potentially useful independently of whether the proposed attack is practical. (See W2)
S2. The analysis of token identity and routing margin provides useful empirical observations. The paper recognizes that large routing changes are not necessarily caused by membership and attempts to account for these alternative explanations rather than relying only on raw Jaccard distance, which aligns with the domain adaptation signal model introduced in works like Chen et al., 2026a (cited in the paper).
Critical weaknesses
W1. I am not convinced that the proposed method is well-motivated. The paper argues that training members experience larger hidden-state drift, which leads to more routing changes. However, the theoretical analysis only establishes a relationship between drift magnitude and routing changes under a fixed direction, not that these changes reliably distinguish members from non-members. The two-stage calibration also feels somewhat heuristic, without sufficient justification for why this specific design is preferable to simpler alternatives.
W2. The practical feasibility of the attack is questionable. The method seems to requier recovering per-token expert identities through a hardware side channel, but the paper does not implement or evaluate this process. Appendix C assumes that prefix caching allows individual token routing to be isolated through sequential queries. This is a strong assumption that needs experimental validation. Without an end-to-end evaluation, the paper essentially assumes access to the most challenging component of the attack.
W3. The evaluation relies too heavily on AUROC, which is insufficient for assessing practical MIA effectiveness. Recent MIA literature emphasizes TPR at low FPR (Carlini et al., 2022, cited in the paper), where high-confidence membership predictions matter most. Table 9 reports TPR@1%FPR shown in the appendix, and the results are often disappointing. For example, Qwen–Reddit achieves 0.679 AUROC but only 0.9% TPR@1%FPR, which suggests the method is not effective. I encourage the authors to report TPR at 1% and 0.1% FPR for all baselines and include log-scale ROC curves in the main text.
W4. I also find the calibration evaluation protocol problematic. If I understand it correctly, Appendix F seems to state that non-members are scored using a single held-out calibration fold, whereas members are scored by averaging results from five calibration tables. This means members and non-members are evaluated using different scoring procedures, which will affect the score distributions. A cleaner evaluation would use an independent reference set and apply the same calibration procedure to both classes.
Questions for authors
See W1-W4.