acceptodds
Under review as a conference paper at ICLR 2027

When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

Abstract

Mixture-of-Experts (MoE) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety auditing. Unlike ordinary model outputs, this telemetry reveals a view of the model's internal computation, raising a privacy question: can it reveal whether an example was used to fine-tune the deployed model? We introduce a router-augmented membership inference attack that combines conventional output-side signals with aggregated routing features and learns transferable membership patterns from independently fine-tuned shadow models. Across three MoE architectures and three data domains, router telemetry consistently improves membership inference over a strong output-signal ensemble, increasing TPR at 1% FPR by 2.7–9.4 percentage points across all nine settings. The leakage persists across full fine-tuning, frozen-router training, LoRA, and instruction tuning, and remains observable with only discrete expert selections, restricted telemetry, or a single shadow model. Mechanistic analysis further shows that the leakage does not require router-specific memorization: fine-tuning introduces membership information into hidden representations, while the router exposes a projection of this signal even when its parameters are frozen. Perturbing the telemetry reduces this additional leakage only as its fidelity degrades. Our results show that router telemetry can turn an operational signal into an additional privacy surface for fine-tuned MoE models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Mixture-of-Experts, Membership Inference, Router Telemetry, Model Privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XNycOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies membership inference risks arising from router telemetry exposed during inference in fine-tuned MoE models. The authors concatenate conventional output-side MIA features with aggregated routing features, and use same-domain auxiliary data to train shadow models and a membership classifier.

Across 3 MoE architectures × 3 data domains, adding router telemetry improves TPR@1% FPR by approximately 2.7--9.4 percentage points over the output-only ensemble. The authors further evaluate frozen-router training, LoRA, instruction tuning, discrete expert selection, partial telemetry, and reduced shadow-model access. The mechanistic analysis suggests that the leakage primarily comes from membership signals encoded in hidden representations after fine-tuning, while the router exposes a projection of these signals. The paper also evaluates simple defenses including noise, dropout, and quantization, and observes a privacy--utility trade-off.

Critical strengths
  1. The paper shows that router telemetry should not be treated as inherently benign operational metadata, as it may create an additional membership leakage surface. This is a practically meaningful problem setting.

  2. The experimental controls are reasonably comprehensive. The authors observe additional leakage across all 3×3 model--domain settings, and use frozen-router, LoRA, discrete-telemetry, and limited-shadow experiments to rule out several simple alternative explanations. In particular, the frozen-router result is convincing.

Critical weaknesses
  1. The attack's practical realizability is not established. External API users receive neither full logits nor router telemetry. Operators who hold the telemetry can access the training data, so the attack adds nothing for them. The headline gains rely on full continuous expert loads, which no real service exposes.

  2. Shadow and target models match in base checkpoint, domain, and fine-tuning recipe, while the threat model assumes no knowledge of the victim's optimizer, learning rate, training duration, data scale, or adaptation strategy. The results show that telemetry contains membership signal, they do not establish a practical attack.

  3. The output-side ensemble reaches AUROC above 0.99 in eight of nine settings, so membership is nearly trivial from outputs alone.

  4. TPR@1% FPR pools predictions from four held-out targets with no per-target results, confidence intervals, or seed repeats. The four targets share shadow models under leave-one-out, so the nine settings are not independent. The smallest gain, 2.7 points, is indistinguishable from noise without variance estimates.

  5. The defense evaluation applies noise, dropout, and quantization and finds that leakage falls as fidelity falls. This outcome follows from the design, since the attack consumes the telemetry itself. Utility is measured as near-perfect correlation with the original loads, a bar stricter than any operational need, and alternatives such as batch aggregation or calibrated differential privacy are never compared.

  6. Prior work already shows that routing signals expose input properties, and GateBleed performs membership inference through a routing side channel. The delta here is aggregating expert loads into an existing MIA ensemble, which is a feature-engineering contribution.

Questions for authors
  1. Does the router-augmented MIA retain consistent gains when the shadow and victim models differ in learning rate, training duration, data scale, or adaptation strategy? This would directly affect my assessment of practical exploitability.

  2. Could you report per-target TPR@1% FPR / AUROC together with bootstrap confidence intervals, to verify that the smaller gains are not driven by target-level variance hidden by pooling?

  3. Do you have results on substantially larger MoE models, or at least an argument for why the current findings should be expected to generalize to much larger MoE deployments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XNyc (Part I)

We thank the reviewer for recognizing the importance of router telemetry as a privacy surface and the breadth of our experimental controls. We provide further evidence addressing the deployment assumptions, transferability, statistical robustness, and scope of our findings.

W1. Practical Realizability and Telemetry Access

Our threat model considers an adversary with access to request-associated inference telemetry and token-level prediction scores. This includes observers with scoped monitoring or tracing permissions, infrastructure insiders, and actors who obtain access to internal observability interfaces. Serving logs, model parameters, and private fine-tuning datasets can be protected by separate access controls. Accordingly, access to routing traces can create a meaningful membership inference risk even when the underlying training records remain inaccessible.

Router information is available at concrete internal execution interfaces. Hugging Face Transformers supports output_router_logits=True for supported MoE models (https://huggingface.co/docs/transformers/v4.47.1/model_doc/qwen2_moe). NVIDIA TensorRT provides MoE tensors such as selectedExpertsForTokens and scoresForSelectedExperts (https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-moe.html). Our evaluated setting requires traces linked to individual requests and access to token-level prediction scores through a scoring or instrumentation interface. We will specify these permissions explicitly in the threat model.

Importantly, the additional leakage persists with discrete expert selections. Table 3 reports positive TPR@1%FPR gains in all nine model and domain settings using expert-selection frequencies, ranging from +0.0283 to +0.1043. This substantially reduces the precision required of exposed routing information. The results establish a concrete privacy risk for systems that expose the specified observations.

W2 & Q1. Robustness to Shadow and Target Training Mismatch

We performed additional experiments on OLMoE with arXiv using 49 shadow and target configuration pairs. The seven configurations cover full fine-tuning, frozen-router training, LoRA, and five learning rates between 5e-6 and 1.2e-5. Both attack variants are trained on the shadow configuration and evaluated on held-out targets. Adding routing telemetry improves TPR@1%FPR in all 49 pairs.

The following results fix the target to the original full fine-tuning setting (LR=8e-6) and vary the shadow training configuration.

Shadow configurationA_out+RouterΔTPR@1%FPR
Full FT, LR=8e-6 (matched)0.8760.953+0.076
Frozen router0.8770.952+0.074
LoRA0.8710.924+0.053
Full FT, LR=5e-60.8540.963+0.109
Full FT, LR=6e-60.8680.960+0.092
Full FT, LR=1e-50.8840.930+0.046
Full FT, LR=1.2e-50.8630.905+0.043

The positive gains under LoRA, frozen-router training, and learning-rate mismatch demonstrate that the attack retains its added value across substantially different shadow training configurations. The present matrix controls the architecture, data domain, and one-epoch training duration, allowing the effects of adaptation choice and learning rate to be evaluated directly.

W3. Additional Leakage in More Challenging MIA Settings

Our new experiments also cover target configurations where output-based inference is considerably more difficult. Using shadow models trained with the original LR=8e-6, we evaluated full fine-tuning targets trained at lower learning rates.

Target LRA_out AUROC+Router AUROCTPR@1%FPR (A_out → +Router)
5e-60.9050.9490.411 → 0.528
6e-60.9550.9800.602 → 0.736
8e-60.9920.9970.876 → 0.953

At LR=5e-6, adding router telemetry improves TPR@1%FPR by 11.7 percentage points even though output-only AUROC is 0.905. At LR=6e-6, the gain is 13.4 percentage points. These results extend the main empirical finding to settings with substantially greater headroom for membership inference.

AuthorsOpenReview ↗

Response to Reviewer XNyc (Part II)

W4 & Q2. Per-Target Results and Statistical Robustness

We conducted additional per-target analyses across all 36 held-out targets from the nine model and domain settings. For each target, we report TPR@1%FPR, AUROC, and paired bootstrap 95% confidence intervals for the improvements obtained by incorporating router telemetry, using 1,000 bootstrap replicates.

The TPR@1%FPR improvement is positive for all 36 targets, with 28/36 target-level confidence intervals excluding zero. AUROC improvements have confidence intervals excluding zero for 35/36 targets. When predictions are pooled within each model and domain setting, both improvements have confidence intervals excluding zero in all nine settings.

The following table presents the complete per-target results for OLMoE across all three data domains.

DomainTargetTPR@1%FPR (A_out → +Router)ΔTPR [95% CI]AUROC (A_out → +Router)ΔAUROC [95% CI]
arXivM10.8696 → 0.9676+0.0980 [0.0689, 0.1301]0.9914 → 0.9979+0.0065 [0.0051, 0.0082]
M20.8764 → 0.9644+0.0880 [0.0653, 0.1095]0.9920 → 0.9980+0.0060 [0.0047, 0.0075]
M30.8604 → 0.9344+0.0740 [0.0575, 0.0983]0.9887 → 0.9971+0.0083 [0.0069, 0.0099]
M40.8680 → 0.9528+0.0848 [0.0594, 0.1042]0.9931 → 0.9980+0.0049 [0.0037, 0.0060]
Pooled0.8685 → 0.9527+0.0842 [0.0745, 0.0995]0.9913 → 0.9974+0.0061 [0.0054, 0.0068]
GitHubM10.9233 → 0.9360+0.0127 [-0.0176, 0.0423]0.9951 → 0.9970+0.0019 [0.0007, 0.0033]
M20.8420 → 0.9027+0.0607 [0.0020, 0.1563]0.9919 → 0.9955+0.0036 [0.0019, 0.0058]
M30.7840 → 0.9153+0.1313 [0.0345, 0.2253]0.9928 → 0.9960+0.0033 [0.0018, 0.0050]
M40.8520 → 0.9000+0.0480 [-0.0150, 0.0995]0.9940 → 0.9958+0.0019 [0.0004, 0.0035]
Pooled0.8585 → 0.9133+0.0548 [0.0201, 0.0843]0.9933 → 0.9960+0.0027 [0.0018, 0.0036]
WikipediaM10.9420 → 0.9653+0.0233 [0.0053, 0.0511]0.9981 → 0.9987+0.0007 [0.0002, 0.0011]
M20.9353 → 0.9533+0.0180 [-0.0066, 0.0447]0.9974 → 0.9984+0.0010 [0.0005, 0.0016]
M30.9373 → 0.9573+0.0200 [-0.0096, 0.0485]0.9976 → 0.9982+0.0006 [0.0000, 0.0011]
M40.9407 → 0.9813+0.0407 [0.0121, 0.0744]0.9977 → 0.9992+0.0015 [0.0006, 0.0026]
Pooled0.9373 → 0.9655+0.0282 [0.0146, 0.0399]0.9977 → 0.9986+0.0009 [0.0006, 0.0012]

Across all three domains, all 12 OLMoE targets exhibit positive TPR@1%FPR improvements, with eight target-level confidence intervals excluding zero. Notably, the pooled improvements remain positive with confidence intervals above zero in all three domains.

The wider confidence intervals for some GitHub and Wikipedia targets reflect the limited number of non-member examples at the 1% FPR operating point. Each target contains 1,500 non-members, corresponding to approximately 15 false positives at this threshold. The paired bootstrap results quantify evaluation-sample uncertainty conditional on the trained models, while the per-target results reveal variation across independently fine-tuned targets.

Together, these analyses provide additional evidence that the observed improvements are consistent across held-out targets and are not solely driven by a small number of favorable target models.

W5. Interpretation of the Telemetry Defense Experiments

Section 5.6 quantitatively evaluates the privacy and telemetry-fidelity trade-off under Gaussian noise, dropout, and quantization. We use two complementary fidelity metrics: Pearson correlation measures preservation of expert loads, while top-1 expert agreement measures preservation of the most heavily loaded expert (Appendix F).

For Gaussian noise at σ=4\sigma=4, the additional TPR@1%FPR gain decreases from 0.0841 to 0.0159, while correlation and top-1 agreement drop to 0.588 and 0.407. At dropout rate p=0.99p=0.99, the gain falls to 0.0040, with correlation of 0.047 and top-1 agreement of 0.050. Quantization at B=4B=4 yields a gain of 0.0079, with correlation of 0.045 and top-1 agreement of 0.076.

These results quantify the substantial loss of routing fidelity associated with suppressing membership leakage across three perturbation mechanisms. They provide concrete privacy and fidelity operating points, while task-specific monitoring utility and alternative mechanisms such as batch aggregation and differential privacy remain valuable directions for further evaluation.

AuthorsOpenReview ↗

Response to Reviewer XNyc (Part III)

W6. Distinction from Prior Routing Privacy Work

Our contribution addresses a distinct privacy question: whether routing information reveals membership in the fine-tuning dataset of a deployed MoE model. Earlier routing-related privacy work primarily investigates inference-time input properties, including prompt information and expert choices. GateBleed demonstrates a hardware timing side channel, including membership inference in a separate model setting and MoE expert-choice recovery.

Our study directly evaluates router telemetry as a fine-tuning membership signal, measures its incremental value against a strong ensemble of output-based MIA features, and characterizes how the signal changes across fine-tuning regimes, telemetry restrictions, and shadow-model configurations. The frozen-router and matched-rank projection experiments further identify a mechanism through which updated hidden representations can expose membership-related information through routing. Together, these findings establish a distinct privacy risk and provide a systematic empirical and mechanistic account of its behavior.

Q3. Generalization to Larger MoE Models

Our evaluation covers three MoE architectures with different expert configurations and routing structures, consistently demonstrating additional membership leakage from router telemetry. The frozen-router analysis further reveals a general mechanism: fine-tuning changes hidden representations, allowing even unchanged router parameters to expose membership-related information through expert routing decisions.

This mechanism is rooted in the fundamental routing operation shared by sparse MoE architectures, including larger models that use learned expert selection. The consistency across architectures and fine-tuning regimes provides evidence that the observed leakage is associated with general MoE routing behavior. While its magnitude may vary with model scale and configuration, our findings establish a mechanistic basis for understanding this privacy risk beyond the specific models evaluated.

Reviewer_FNkQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether router telemetry from fine-tuned Mixture-of-Experts (MoE) language models can reveal membership information beyond conventional model outputs, thereby introducing additional privacy risks when such telemetry is exposed during inference. The authors propose a router-augmented membership inference attack that combines output-based membership signals with aggregated expert-routing statistics. A membership classifier is trained using independently fine-tuned shadow models and subsequently transferred to a held-out target model. Experiments across three MoE architectures and three data domains demonstrate consistent improvements over output-only membership inference baselines, particularly in terms of TPR at 1% FPR. The paper further examines the leakage under different fine-tuning regimes, restricted telemetry exposure, and several telemetry perturbation mechanisms.

Critical strengths
  1. The experiments provide convincing support for the paper's central claim by comparing router-augmented MIA against an output-only attack while keeping the shadow models, membership labels, classifier family, and held-out targets fixed. The consistent gains across all nine model–domain settings make the incremental value of routing information credible. Importantly, the frozen-router experiment and router-subspace projection analysis go beyond attack performance to investigate why the signal exists: they suggest that the router can expose membership-related information in hidden representations without directly memorizing examples through updates to its own parameters (Tables 1–2; Section 5.5).

  2. The paper describes its experimental procedure in sufficient detail to make the comparisons transparent. It specifies how the shadow models are fine-tuned and rotated against held-out targets, and how both output-side and router-telemetry features are constructed. These details help readers assess possible data overlap, understand the source of the attacker's training labels, and evaluate whether the performance gains can reasonably be attributed to routing features (Section 4).

  3. The motivation and threat model are clearly articulated. The authors explain why intermediate representations might contain membership signals that are not fully reflected in model outputs, and why routing telemetry potentially retained for monitoring, debugging, or expert-load analysis.

Critical weaknesses
  1. The attack largely follows an established shadow-model-based MIA paradigm: train a membership classifier using labeled examples from auxiliary fine-tuned models and apply it to a held-out target [1]. Adding aggregated routing statistics is an interesting choice of signal, but it does not substantially change the attack's learning or transfer mechanism. As a result, the methodological novelty is limited, and the contribution depends primarily on the empirical discovery and characterization of an additional leakage surface.

  2. The controlled evaluation uses relatively favorable conditions for transferring a membership classifier: shadow and target models are trained on disjoint samples from the same source corpus, and follow closely matched fine-tuning configurations. While this improves experimental control, it leaves unclear whether the learned routing–membership relationship would transfer when the attacker has only approximate knowledge of the victim's data distribution or training procedure. Moreover, the output-only baseline already obtains AUROC values of 0.9832–0.9977, so the results provide limited evidence about the value of telemetry in genuinely difficult membership inference settings (Table 1).

[1] Membership Inference Attacks Against Machine Learning Models, Shokri et al., IEEE Symposium on Security and Privacy (SP), 2017.

Questions for authors
  1. The main experiments show that router telemetry complements a strong output-based MIA ensemble, but they do not establish how much membership information telemetry contains independently. Moreover, the projection analysis operates on hidden-state subspaces rather than the telemetry available to the assumed attacker. Could the authors report a router-only MIA under the same shadow-to-target protocol, ideally including a comparison between routing features from the pretrained and fine-tuned models? This would help clarify the router's independent contribution and better connect the proposed mechanism to the attack results.

  2. The threat model explicitly assumes that per-request router telemetry may be available through monitoring, debugging, or expert-load analysis, and that the attacker can also obtain token-level probabilities/logits. However, this is a stronger access model than a typical text-generation API. Could the authors identify concrete deployment interfaces or attacker roles for which such telemetry is realistically accessible, and clarify whether the proposed attack requires privileged observability access rather than ordinary black-box inference access?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FNkQ (Part I)

We thank the reviewer for the constructive feedback and for recognizing the value of our controlled comparisons, experimental coverage, and mechanistic analysis. We provide additional evidence and clarification addressing each concern below.

W1. Methodological Novelty

Our primary contribution is the identification and systematic characterization of membership leakage through MoE router telemetry, revealing an additional privacy surface associated with model inference. Previous studies have investigated how routing signals expose properties of inference-time inputs. Our work extends the understanding of routing-related privacy risks to the membership of private fine-tuning data.

Specifically, our contributions are threefold. First, we establish that router telemetry provides additional membership information beyond strong output-based MIA methods across three MoE architectures and three data domains. Second, we systematically characterize this leakage across different fine-tuning regimes, telemetry granularities, and shadow-model access conditions. Third, our mechanistic analysis reveals that membership leakage persists even when router parameters remain frozen, indicating that routing can expose membership information encoded in hidden representations during fine-tuning.

Together, these findings establish a previously underexplored membership leakage surface and provide systematic empirical and mechanistic evidence for understanding the privacy implications of exposing MoE router telemetry.

W2. Shadow-Target Mismatch and Difficult MIA Settings

We conducted additional experiments on OLMoE with arXiv to evaluate the robustness of router-augmented MIA under different shadow and target training configurations. We considered 49 shadow and target configuration pairs, covering full fine-tuning, frozen-router training, LoRA, and five learning rates ranging from 5e-6 to 1.2e-5.

The following table fixes the target to full fine-tuning with LR=8e-6 and varies the shadow configuration.

Shadow ConfigurationA_out+RouterΔTPR@1%FPR
Full FT, LR=8e-6 (matched)0.8760.953+0.076
Frozen router0.8770.952+0.074
LoRA0.8710.924+0.053
Full FT, LR=5e-60.8540.963+0.109
Full FT, LR=6e-60.8680.960+0.092
Full FT, LR=1e-50.8840.930+0.046
Full FT, LR=1.2e-50.8630.905+0.043

Router augmentation consistently improves TPR@1%FPR across all 49 configuration pairs. For the main target, mismatched shadow configurations yield improvements ranging from +0.043 to +0.109, compared with +0.076 under the matched configuration. These findings demonstrate that the additional membership signal remains exploitable across different training configurations, reducing the need for precise knowledge of the victim's fine-tuning procedure.

We further evaluated more challenging MIA settings by reducing the target's learning rate while keeping the shadow models at the original LR=8e-6. With target LR=5e-6, the output-only AUROC is 0.905 and TPR@1%FPR is 0.411. Incorporating router telemetry increases these values to 0.949 and 0.528, respectively. Similarly, with target LR=6e-6, TPR@1%FPR increases from 0.602 to 0.736.

These results extend our main findings to settings where output-based membership inference is substantially more difficult. They further demonstrate the consistent value of router telemetry under training configuration mismatch within the evaluated architecture and data domain.

AuthorsOpenReview ↗

Response to Reviewer FNkQ (Part II)

Q1. Independent Membership Information in Router Telemetry

We conducted additional router-only MIA experiments on OLMoE with arXiv using the same shadow-to-target evaluation protocol and classifier family as our main attack. These experiments directly measure the membership information contained in router telemetry without incorporating output-side features.

We consider three router representations. Pretrained routing probabilities provide a control before fine-tuning on the evaluated membership dataset. Fine-tuned routing probabilities measure the membership information in the exposed telemetry. Routing changes capture the difference between fine-tuned and pretrained routing probabilities for the same input, providing a reference-based measure of membership-related changes.

Router FeaturesAUROCTPR@1%FPR
Pretrained routing probabilities0.48890.0086
Fine-tuned routing probabilities, 1 epoch0.52080.0118
Routing changes, 1 epoch vs. pretrained0.71180.0694
Fine-tuned routing probabilities, 2 epochs0.65640.0467
Routing changes, 2 epochs vs. pretrained0.87170.2355

The one-epoch setting corresponds to our main experiments. Raw routing probabilities yield an AUROC of 0.5208, while routing changes relative to the pretrained model achieve 0.7118. This substantial improvement indicates that fine-tuning-induced changes in routing behavior contain detectable membership information that is difficult to isolate from raw routing probabilities alone.

After two epochs, the membership signal becomes considerably stronger. Raw routing probabilities achieve an AUROC of 0.6564, while routing changes reach 0.8717. This trend is consistent with our mechanistic analysis in Section 5.5, where membership separability in the router subspace increases with fine-tuning.

These findings provide further evidence that fine-tuning introduces membership-related information into routing behavior. They also clarify the relationship between the standalone membership signal in router telemetry and its complementary contribution to output-based MIA, which is demonstrated in our main experiments.

Q2. Practical Telemetry Access and Attacker Privileges

Our threat model considers an adversary with access to request-associated router telemetry and token-level prediction scores. Relevant scenarios include personnel with scoped monitoring or logging permissions, infrastructure insiders, and actors who obtain access to internal observability interfaces. Inference telemetry, model parameters, and private fine-tuning datasets may be governed by separate access controls. Consequently, access to routing information can expose membership-related information even when the underlying training data remain inaccessible.

Concrete routing interfaces exist in current MoE implementations. Hugging Face Transformers supports output_router_logits=True for supported MoE models (https://huggingface.co/docs/transformers/v4.47.1/model_doc/qwen2_moe). NVIDIA TensorRT provides token-level selectedExpertsForTokens and scoresForSelectedExperts tensors (https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-moe.html). These interfaces demonstrate that routing information is accessible to suitably privileged model execution and instrumentation components.

The evaluated attack requires routing traces associated with individual requests and token-level output scores, which may be obtained through privileged tracing or scoring interfaces. These requirements define a specific observability access setting within MoE deployment environments. Furthermore, Table 3 demonstrates that discrete expert selections alone provide additional membership leakage, with TPR@1%FPR improvements ranging from +0.0283 to +0.1043 across all nine model and domain settings.

These findings highlight the privacy implications of exposing routing information through inference monitoring and observability systems. We will further clarify the access requirements and relevant deployment scenarios in the revised paper.

Reviewer_FNkQOpenReview ↗

Response to Author Comment

I acknowledge the additional experiments evaluating mismatched fine-tuning configurations and more challenging MIA settings. The results provide useful evidence that router telemetry remains effective under different training procedures and when the output-only baseline becomes less reliable (by modifying the LR).

However, my concern regarding the similarity between the shadow and target training distributions remains unaddressed. The additional experiments are still conducted within the same OLMoE/arXiv setting, where shadow and target training data originate from the same source corpus. While the experiments demonstrate robustness to training-configuration mismatch, they do not establish whether the learned routing–membership relationship transfers when the attacker's auxiliary data come from a different source or exhibit a distribution shift relative to the victim's private fine-tuning data. I would appreciate further clarification or discussion addressing this distinction and please point out if I had any misunderstanding.

Reviewer_7LAoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies membership inference against fine-tuned MoE models when an attacker can observe model outputs and router telemetry. It aggregates per-expert routing probabilities across tokens and layers, combines them with output-side signals, and trains an attack classifier using independently fine-tuned shadow models. Across three architectures and three data domains, telemetry improves TPR at 1% FPR by 2.7-9.4 percentage points. The paper also studies several fine-tuning regimes, reduced telemetry, limited shadow-model access, and telemetry perturbations. Its mechanistic experiments argue that routing exposes membership information already present in hidden representations rather than requiring router-specific memorization.

Critical strengths
  1. The paper tests the additional value of router telemetry against a strong ensemble of output-side signals rather than relying on a weak output-only baseline. The consistent gains across the experiment settings make the empirical comparison more informative.
  2. The analysis spans multiple telemetry granularities, fine-tuning regimes, and shadow-model budgets. The finding that frozen routers can still expose a signal is especially relevant to deployment choices and argues against treating router-parameter updates as the sole source of risk.
Critical weaknesses

The paper describes telemetry as an operational signal, but that label does not establish that an external or ordinary API user can obtain it. The attack also uses output probabilities/logits, which many user-facing services do not return. Please distinguish realistic attacker classes (API user, service administrator, infrastructure insider), identify which telemetry each can observe, and state the access-control assumptions required by the experiments. Without this, the paper demonstrates a conditional risk but does not establish the prevalence of the exposure it frames as a deployment concern.

Questions for authors
  1. In a real-world setup, what specific interface actually exposes the per-token routing probabilities or expert assignments you used? Also, what privilege level does an attacker need to see them, and would they get the output logits at the same time?
  2. I am quite curious about how the attack's performance might change if the shadow models used a slightly different setup. What happens to the attack's performance if the shadow models use another MoE architecture?
  3. I noticed that the true positive rate (TPR) improvement is somewhat smaller with LoRA (+0.0305) compared to frozen-router training (+0.0904). Could you share some insights on why this might happen?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7LAo (Part I)

We thank the reviewer for the constructive feedback and for recognizing the value of our experimental comparisons and frozen-router analysis. We provide further evidence and clarification addressing the three questions below.

Weakness & Q1. Practical Accessibility of Router Telemetry and Output Logits

Our threat model considers an adversary with access to request-associated router telemetry and token-level prediction scores. Relevant scenarios include personnel with scoped monitoring or logging permissions, infrastructure insiders, and actors who obtain access to internal observability interfaces. Importantly, inference telemetry, model parameters, and private fine-tuning datasets may be governed by separate access controls. Consequently, access to routing information can create a meaningful membership inference risk even when the underlying training data remain inaccessible.

Concrete routing interfaces exist in current MoE implementations. Hugging Face Transformers supports output_router_logits=True for supported MoE models (https://huggingface.co/docs/transformers/v4.47.1/model_doc/qwen2_moe). NVIDIA TensorRT provides token-level selectedExpertsForTokens and scoresForSelectedExperts tensors (https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/transformers-moe.html). These interfaces demonstrate that routing information is accessible to suitably privileged model execution and instrumentation components.

The evaluated attack requires routing traces associated with individual requests and token-level output scores, which may be obtained through privileged tracing or scoring interfaces. These access requirements define the scope of our threat model. Furthermore, Table 3 demonstrates that discrete expert selections alone provide additional membership leakage, with TPR@1%FPR improvements ranging from +0.0283 to +0.1043 across all nine model and domain settings. This finding establishes that the leakage persists even when precise continuous routing probabilities are unavailable. We will make the access requirements explicit and further clarify the deployment scenarios addressed by our work.

Q2. Generalization Across Shadow Model Configurations

Our threat model assumes knowledge of the publicly available pretrained checkpoint underlying the victim model. This enables the attacker to construct shadow models with the same architecture while independently selecting their fine-tuning configurations.

To evaluate robustness to imperfect knowledge of the victim's training procedure, we conducted additional experiments covering 49 shadow and target configuration pairs on OLMoE with arXiv. These configurations include full fine-tuning, frozen-router training, LoRA, and five learning rates ranging from 5e-6 to 1.2e-5.

The following table fixes the target to full fine-tuning with a learning rate of 8e-6 and varies the shadow configuration.

Shadow ConfigurationA_out+RouterΔTPR@1%FPR
Full FT (matched)0.8760.953+0.076
Frozen Router0.8770.952+0.074
LoRA0.8710.924+0.053
Full FT (LR=5e-6)0.8540.963+0.109
Full FT (LR=6e-6)0.8680.960+0.092
Full FT (LR=1e-5)0.8840.930+0.046
Full FT (LR=1.2e-5)0.8630.905+0.043

Router augmentation consistently improves TPR@1%FPR across all 49 configuration pairs. For the main target, mismatched shadow configurations yield improvements ranging from +0.043 to +0.109, compared with +0.076 under the matched configuration. These results demonstrate that the additional membership signal remains exploitable across different training configurations, reducing the need for precise knowledge of the victim's fine-tuning procedure.

Cross-architecture shadow transfer presents an additional generalization setting because different MoE architectures may have different expert identities, layer structures, and router feature dimensions. Our current evaluation establishes robustness to training configuration mismatch within a known pretrained architecture. Extending the approach through architecture-independent routing representations is a promising direction for further investigation.

AuthorsOpenReview ↗

Response to Reviewer 7LAo (Part II)

Q3. Smaller Improvement Under LoRA

The difference in incremental improvement is closely related to the strength of the output-only baseline. As shown in Table 2, LoRA achieves an output-only TPR@1%FPR of 0.9226, substantially higher than the 0.8612 obtained under frozen-router fine-tuning. This stronger baseline leaves less headroom for additional improvement.

Importantly, the final router-augmented performance is nearly identical across the two regimes, reaching 0.9531 under LoRA and 0.9516 under frozen-router fine-tuning. Our additional experiments further confirm this pattern, with LoRA improving from 0.925 to 0.953 and frozen-router training improving from 0.878 to 0.952.

These findings highlight the consistent utility of router telemetry across adaptation strategies. The smaller incremental gain under LoRA is consistent with its stronger output-only baseline, while the final attack performance remains comparable to that under frozen-router fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.