acceptodds
Under review as a conference paper at ICLR 2027

FedDataLeak: Detecting Data Leakage from Federated LLM Updates

Abstract

Federated fine-tuning allows large language models (LLMs) to learn from decentralized, sensitive data without sharing raw text, yet the exchanged model updates can still reveal private information. We present FedDataLeak, a model-agnostic framework for auditing leakage from federated LLM updates using model weights before and after each communication round. Its primary signal, Feature Reconstruction Fidelity (FRF), measures how well a plausible input can reproduce an observed update. To avoid costly reconstruction at every round, FedDataLeak combines FRF with two lightweight representation-shift signals, neuron polysemanticity and activation sparsity, and triggers deeper analysis only for suspicious updates. We evaluate the framework across heterogeneous federated settings, including controlled client partitions, synthetic canaries, and aggregate-only visibility, and test whether its scores identify leakage-prone updates and align with downstream reconstruction and membership evidence. Across diverse models and datasets, FedDataLeak achieves strong leakage detection with audit overhead lower than competitive baselines, demonstrating that federated updates can serve as actionable privacy signals for safe LLM training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Large Language Models, Data Leakage, Privacy Auditing, Gradient Inversion, Feature Reconstruction, Representation Analysis, Privacy-Preserving Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SYhiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FedDataLeak, a framework for detecting possible data leakage from federated LLM updates. The method combines Feature Reconstruction Fidelity (FRF) with two representation-based signals, activation sparsity and polysemanticity. The idea is to use the cheaper signals to identify suspicious updates before running more expensive reconstruction. The authors evaluate the method on three LLMs and several datasets and report improvements over existing baselines.

Critical strengths

The paper studies an important privacy problem in federated learning. The idea of monitoring model updates without always running expensive reconstruction attacks is useful. The evaluation also covers different models and includes both per-client and aggregate-only settings.

Critical weaknesses

-Limited novelty and unclear motivation for the proposed signals: My main concern is the novelty of the method. The paper mainly combines existing techniques into an auditing pipeline, so the contribution seems more engineering-oriented than methodological. I am also not convinced by the motivation for using polysemanticity and activation sparsity as indicators of privacy leakage. These signals come from model interpretability, but their connection to actual data leakage is not well explained. The paper provides ablations and correlation results, but these do not clearly show why an update containing a canary should cause a measurable change in these signals on a public reference set. Similar changes could also happen during normal fine-tuning.

-Missing simple baseline based on embedding updates: If the embedding layer is trainable, the updated embedding rows may provide strong information about which tokens appeared in the training data. This could make the canary detection task easier than it seems.

The authors already use embedding-row update norms to initialize their reconstruction candidates in Algorithm 2. However, the cheap baselines in Table 14 only include update norm, cosine similarity, and clipping saturation. I think a simple token-support detector based on embedding-row updates is an important missing baseline.

Without this comparison, it is difficult to know how much of the reported improvement comes from the proposed method rather than directly available token-level information.

-Evaluation setup and leakage labels: The main detection results use the presence of a synthetic canary as the positive label. However, detecting that a canary was included in training is not necessarily the same as showing that private information can be recovered. The paper reports separate reconstruction metrics, but I would like to see a clearer evaluation of detection performance using actual reconstruction success as the target. Also, the default configuration uses batch size 1, which is favorable for leakage attacks. Although the authors include some sensitivity experiments, larger batch sizes and more challenging aggregation settings would make the results more convincing.

-Some experimental results need clarification: Several results in Table 1 look unusually regular. For example, the AUPRC values for DLG, iDLG, and Inverting Gradients in the Medical Abstract experiments increase by exactly 0.10 across all three models. Similarly, Table 13 reports exactly +0.02 AUPRC improvement from aggregate-only to per-client visibility for every model. These patterns may simply be due to rounding, but it would be helpful to provide the unrounded results and individual seed values.

-The benefit of the triage stage is not fully clear: The paper claims that the representation-based triage makes auditing more efficient. However, Table 14 shows that FRF-only costs 7.1 seconds per round, while full FedDataLeak costs 8.9 seconds. The full method has better detection performance, but the benefit of the additional complexity is not completely clear. I would like to see a more direct comparison between the triage signal alone, FRF applied to every update, and the complete method.

Questions for authors
  1. How well does a simple embedding-row token-support detector perform on the canary detection task? Could the authors include this baseline for embeddings-only and mixed fine-tuning?

  2. What are the AUPRC and AUROC of the representation-based triage signal alone, without FRF? How many leakage-positive updates are missed before FRF is triggered?

  3. Why does gating FRF improve the detection results compared with running FRF on every update? How much computation is actually saved when comparing methods under the same reconstruction budget?

  4. Can the authors clearly specify whether the Table 1 results are per-client or aggregate-only for each dataset? Appendix D.2 identifies the Medical Abstract results as aggregate-only, but the main table should make this explicit.

  5. Can the authors explain the regular numerical patterns in Tables 1 and 13 and provide the individual seed results?

4 comments

We thank the reviewer for recognizing the importance of continuous privacy auditing and both visibility regimes. Some concerns are already answered by results in the paper (batch size, frozen-embedding training, the unconditional variant's cost). Other confusions we address directly.

1. Novelty and signal rationale. FedDataLeak introduces a budgeted update-level auditor, not another inversion attack. Candidate-input update matching produces Feature Reconstruction Fidelity (FRF, Eqs. 2–3); Algorithm 1/Eq. 7 implement three-state auditing for individual or aggregate updates (Eqs. 1, 9). Polysemanticity (Eq. 5) and Hoyer sparsity (Eq. 4) are triage proxies, not causal proof of leakage. Paired pre/post activations on the same public reference set (Eq. 6; Appendix B.4) capture feature specialization potentially associated with easier reconstruction. Ordinary fine-tuning can also cause such shifts; hence our matched canary-absent controls (Sec 4.1) and FRF as primary evidence. Table 7 reports ρ(q_rep, FRF)=0.74. Table 3 shows AUPRC improving from 0.84 (FRF-only) to 0.87 (+PI), 0.86 (+sparsity), and 0.90 (full). These results support complementary signals without claiming leakage specificity.

2. Embedding-row baseline (Q1). Gao et al. (COLING 2025) demonstrate that embedding gradients reveal token identities. Accordingly, Algorithm 2 already uses high-norm embedding rows for reconstruction initialization. However, candidate initialization is distinct from a standalone detector. We agree that Table 14's update-norm, cosine, and clipping controls do not replace a token-support baseline. A label-blind top-k embedding-row support detector is appropriate for embeddings-only and mixed fine-tuning, with frozen-embedding LoRA-only as a control. Ground-truth canary tokens must not enter the detector.

3. Leakage labels and batch effects. Table 1 evaluates canary inclusion, not confirmed extraction (Sec 4.1). However, Table 2 separately reports ECR 0.73, token F1 0.79, and membership AUC 0.89 (Medical Abstract/LLaMA-3.2-1B). Appendix C.5 further specifies independent, fixed-budget reconstruction-success labels with leave-one-method-out evaluation, although its complete numerical results require reporting.

Importantly, larger batches are already evaluated, contrary to the impression that only B=1 was studied. Table 12 reports AUPRC 0.90/0.87/0.83 and ECR 0.73/0.67/0.59 for B=1/2/4. Table 13 additionally examines aggregate-only auditing, where results represent cohort-level rather than client-level risk.

4. Triage effectiveness and computational cost (Q2 and Q3). Under Eq. 7, MONITOR updates receive q_rep; triggered updates receive FRF-fused scores. Appendix B.2 reports 31% FRF invocation, avoiding FRF calls on 69% of updates. This is a reduction in solver invocations, not necessarily total runtime.

Table 7 reports correlation with FRF, not standalone q_rep AUPRC/AUROC; pre-trigger missed positives are also not enumerated. These are useful reporting additions. Importantly, fusion not gating alone explains the detection improvement.

Table 3 reports FRF-only at 7.1 s/round versus full FedDataLeak at 8.9 s/round. The additional 1.8 seconds yields +0.06 AUPRC and +0.15 TPR@1%FPR (Tables 3, 14). Thus, the benefit is improved detection and selective deep inspection, not unconditional superiority in runtime.

Moreover, the reviewer overlooks the scheduler-matched comparison in Table 8: with the same 31% trigger rate and shared triage overhead, FET costs 9.9 s, Inverting Gradients 11.2 s, and FedDataLeak 8.9 s. A fully policy-consistent comparison should also recompute full-population rankings from Eq. 7 and compare unconditional FRF under an identical solver budget.

5. Observability and numerical consistency (Q4 and Q5). Medical Abstract in Table 1 is aggregate-only (Table 13). Stack Overflow and Enron are [per-client / aggregate-only]. We will mark the regime in every Table 1 row and correct Table 9, which wrongly lists per-client as the default. On the regular +0.10 and +0.02 steps: these are 3-seed means rounded to two decimals. We will post unrounded per-seed values for Tables 1 and 13 in this thread and add them to the appendix.

In conclusion, FedDataLeak's contribution is reconstruction-grounded, selective privacy auditing, rather than treating interpretability signals as direct proof of disclosure. The manuscript already contains substantial ablation, batch-size, observability, and scheduler-matched evidence. The remaining concerns primarily concern additional controls and transparent numerical reporting. We respectfully request that these existing results and the precise scope of the contribution be considered when reassessing the rating.

We have implemented the changes you requested in the revised manuscript (highlighted in yellow), and the paper is almost final.

Locations:

  • Why PI and sparsity (W1): Sec. 3.3, "Role of the triage signals". They are triage proxies, not causal evidence. Matched canary-absent negatives control for ordinary fine-tuning, and FRF stays the primary evidence.
  • Embedding-row baseline (Q1): App. C.4 defines a label-blind top-k embedding-row token-support detector, with a new row in Table 14. Its numbers are being added. Sec. 4.3 also notes that with frozen embeddings (LoRA-only), FedDataLeak keeps 0.77–0.86 AUPRC (Table 15).
  • Leakage labels and batch size (W3): Sec. 4.1 and Table R1 in Sec. 4.2 score the method against realized reconstruction success. Sec. 4.3 now states the batch-size result: AUPRC 0.90/0.87/0.83 for B=1/2/4 (Table 12).
  • Triage vs. FRF vs. full (Q2, Q3): "Gating versus fusion" in Sec. 4.3 compares them. FRF-only reaches 0.84 AUPRC against 0.90 for the full method. Running mixed auditing on every update gives 0.89 at 10.6 s/round against 0.90 at 8.9 s/round when gated. Triage-only AUPRC/AUROC and the missed-positive count (Table 14, App. B.2) are pending.
  • Observability regime (Q4): The Table 1 caption now states the regime per dataset. Medical Abstract is aggregate-only; Stack Overflow and Enron are per-client / aggregate-only. Table 9's default regime is corrected.
  • Regular patterns (Q5): App. D.2 explains that the entries are three-seed means rounded to two decimals. The unrounded and per-seed values for Tables 1 and 13 are being added and will be posted here.

If these changes resolve your concerns, we would appreciate it if you would reconsider your score. Thank you again for the feedback.

Reviewer_SYhiOpenReview ↗

Follow-up on the rebuttal

Thank you for your responds. Most of my concern are still open and each round of rebuttal raises new concerns. so, I keep my score.

Thank you for the reply. We would like to briefly clarify where things stand.

Several of your original points are already answered by results in the revised manuscript:

  • Batch size: AUPRC is 0.90/0.87/0.83 at B=1/2/4 (Table 12, Sec. 4.3).
  • Token-level shortcut: with embeddings frozen (LoRA-only), FedDataLeak keeps 0.77–0.86 AUPRC across all nine dataset–model settings (Table 15).
  • Benefit of fusion: AUPRC rises from 0.84 for FRF-only to 0.90 for the full method, and the gated policy costs 8.9 s/round versus 10.6 s/round when every update is audited (Sec. 4.3, "Gating versus fusion").
  • Rationale and reporting: the role of the triage signals is now stated (Sec. 3.3), the observability regime is given per dataset (Table 1 caption), and rounding is explained (App. D.2).

There are a few entries which are still pending and we will include them in the final camera ready. Importantly we have narrowed the paper's claims to match what is currently measured.

If any specific concern still seems unresolved, we would be grateful if you named it, so we can answer it directly. Thank you again for your time.

Reviewer_Trv4OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose FedDataLeak. They first reframe the federated leakage analysis from attack centric to audit centric, i.e., instead of maximizing reconstruction quality, they assing a per round leakage-risk score to each observed update. The primary signal is Feature Reconstruction Fidelity which measures how well a synthetic input's simulated update matches the observed update. The evaluation is done on three LLMs and three naturally partitioned datasets, under both per-client and aggregate-only observability.

Critical strengths
  • The authors do an important job of framing it as audit-centric because the operational question in practice should be: which rounds or clients are currently leakage-prone.

  • The per-client vs aggregate-only split is well done since it clearly defines two observability regimes and explicitly restricts the aggregate only output to cohort level risk.

  • The experiments in this paper are strong with a good mix of datasets and cross-silo/cross-device settings. I like that actual federated datasets are used instead of artificially partitioned synthetic datasets.

Critical weaknesses
  • The point about client recipe is a bit unclear to me. In section 3.2, it is stated ' the same local training recipe'. So is an approximate recipe needed or an exact one?

  • Three limitations are mentioned but without any data. I believe that they are cheap to probe and the paper should report some results on adaptive evasion and recipe mismatch at the very least.

  • As I mentioned above, I like the variety in experimental settings but it would be good to have a multimodal data+model combination as well.

Questions for authors
  • Is the client recipe an exact one or an approximation. How do AUPRC and FRF degrade as the recipe diverges from the true one?

  • How do the scores and thresholds shift when you introduce clipping and DP SGD.

  • How sensitive are the triage signals to the distribution of R?

1 comment

We sincerely thank the reviewer for the thoughtful assessment, positive evaluation, and recognition of FedDataLeak's audit-centric formulation, distinct observability regimes, and comprehensive experimental settings. We appreciate the constructive suggestions and clarify the following points.

1. Client-recipe approximation (Q1). An exact recipe is preferable but not required. Section 3.2 defines the assumed local-update simulator, while Appendix B.3, Eq. (20), and Table 6 explicitly evaluate recipe mismatch. AUPRC remains 0.89/0.88 under learning-rate mismatches (×0.5/×2), 0.87 under step-count mismatch, and 0.86 under optimizer mismatch, compared with 0.90 for the exact recipe. FRF-cos decreases from 0.804 to 0.748 under optimizer mismatch. More severe trainable-subset mismatch reduces AUPRC to 0.82, while recipe-ensemble FRF recovers 0.89. Thus, moderate recipe uncertainty preserves useful risk ranking.

2. Clipping and DP-style perturbations (Q2). Appendix D.6, Table 16, and Fig. 8 already provide this evaluation. AUPRC decreases from 0.90 (no defense) to 0.88/0.84 under mild/moderate clipping and 0.85 under clipping with light Gaussian noise. Stronger noise reduces AUPRC to 0.79. Thresholds are recalibrated using a disjoint validation split. These are DP-style sensitivity experiments, not claims of formally accounted DP-SGD guarantees.

3. Public reference-set sensitivity (Q3). Appendix B.4, Table 7 examines reference-set size, activation layers, clustering, and neuron selection. AUPRC remains 0.88–0.90 for |R| = 128–512, compared with 0.86 for |R| = 64. This demonstrates stability under reasonable reference-set configurations, although substantial distribution shift remains an open limitation.

4. Additional limitations. We agree that adaptive evasion and multimodal federated models are valuable extensions. The current manuscript identifies adaptive evasion as a limitation but does not claim empirical robustness against strategic adversaries. Similarly, multimodal evaluation remains outside the present experimental scope.

We appreciate the reviewer's encouraging assessment and constructive recommendations. We believe the existing recipe-mismatch, defense-sensitivity, and reference-set analyses directly address the principal technical questions while clarifying the remaining limitations.

Reviewer_3dBJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FedDataLeak, a white-box auditing framework for detecting data leakage in federated LLM updates. It employs a tiered pipeline: it first computes lightweight representation-shift signals (activation sparsity and a polysemanticity proxy) on public reference data. Updates exceeding a certain threshold are then subjected to Feature Reconstruction Fidelity (FRF), which measures how well a synthetic candidate input can reproduce the observed update. A further conditional inversion stage is triggered based on the FRF score. The method claims to produce client-round scores under individual visibility and cohort-round scores under aggregate-only visibility. Experiments across three LLMs and three text datasets evaluate detection performance, recovery metrics, and operational overhead.

Critical strengths

1.Relevant operational formulation. The paper approaches privacy auditing by asking which observed updates warrant deeper inspection, rather than running indiscriminate, expensive inversion. The distinction between client attribution and cohort-level evidence under aggregation helps frame the intended real-world use case.

2.Comprehensive evaluation scope. The study includes recovery evidence, PEFT-subset ablations, recipe mismatch analyses, and cheap update monitors. These controls are valuable for assessing whether FRF and representation triage actually add value beyond simple update magnitude or reduced invocation frequency.

Critical weaknesses

1.The main detection results do not adequately validate realized leakage prediction. Table 1 labels updates based primarily on canary insertion, and Table 10 similarly relies on designated-record presence. These interventions do not necessarily imply that successful extraction or exploitable membership evidence has occurred. While Table 2's ECR and Token F1 provide some recovery evidence, they do not establish how the deployed scores predict independently measured leakage across all updates. Appendix C.5 describes an independent oracle and a leave-one-method-out procedure, but I could not locate a complete result panel containing the concrete oracle budget, label counts, and score-versus-oracle performance. The released code also lacks these evaluation records. Thus, the distinction between detecting an intervention and predicting its recoverability remains unresolved.

2.The supplied implementation does not verify the claimed operational accuracy–cost tradeoff. The README describes a compact embedding-only demonstration. Looking at main.py (lines 369–405), the fused score uses cosine agreement with the true local batch's token vector, explicitly substituting this proxy for the manuscript's candidate-input FRF solver. It also computes this score before assigning trigger labels; MONITOR does not return q_rep, DEEPINSPECT does not invoke deeper inversion, and the aggregate score simply averages client scores. Consequently, the provided code does not validate the update-only auditing, the actual gated costs, or the aggregate-only execution. Furthermore, the entry point fails a syntax check at line 125. While neither an incomplete artifact nor a syntax error alone invalidates the paper, the outstanding question is whether Table 8 was actually computed from the piecewise outputs defined in Eq. (7), with global rankings recomputed accordingly. The demonstration defaults and recommended GPUs do not establish that the solver budgets or hardware used for the principal results were matched.

Questions for authors

1.Validation against realized leakage. Which specific experimental results measure deployed-score performance against independently realized leakage? Please identify the existing table or figure that contains the oracle implementation, its budget, the positive and negative counts, and the corresponding uncertainty. Clarifying this would materially change my assessment of whether the method actually predicts the intended privacy failure.

2.Implementation and cost accounting. Which specific implementation and configurations produced the principal tables, and how do they relate to the acknowledged token-support proxy in the supplement? Please provide the numerical reconstruction budgets and actual hardware. Furthermore, were Eq. (7)'s MONITOR outputs actually used in Table 8, and were the full-population ranking metrics recomputed? An update-only entry point that matches the claimed policy, alongside the correction of the syntax error, would make the proposed operational tradeoff verifiable.

4 comments

We appreciate the recognition of our audit-centric formulation and broad controls. We distinguish intervention detection from realized leakage, and the method from the released demonstration.

1. Realized leakage (Q1). As Sec 4.1 states, Table 1 uses canary-presence labels; Table 10 (Appendix C.7) uses designated-record presence. Neither establishes exploitable recovery. These are controlled update-level intervention tests, consistent with canary-auditing practice (Carlini et al., USENIX'19; Maddock et al., ICLR'23). The paper separately measures recovery: Table 2 reports ECR 0.73, Token F1 0.79, membership AUC 0.89 (Medical Abstract/LLaMA-3.2-1B). Table 3/Fig. 5 show FRF-only AUPRC 0.84 rising to 0.90 with full fusion; Table 14 shows 0.90 vs. 0.65 for the strongest cheap monitor. These support the mechanism, but not per-update prediction of attack success.

Appendix C.5 defines independent y_real: exact canary recovery or token overlap exceeding a preset threshold under a fixed-budget reconstruction oracle, with leave-one-method-out evaluation to avoid circularity. However, the complete numerical oracle panel is absent; the protocol does not substitute for measured results. The decisive supplement is:

Table R1: Realized-Leakage Validation Against an Independent Reconstruction Oracle

SettingIndependent Oracle, Budget, ThresholdN+/N−LRS AUPRC / AUROC / TPR@1%FPR(95% CI)ρ(LRS, Recovery)
Medical Abstract / LLaMA-3.2-1B / Per-clientFET-style; 3 restarts, 100 iterations, 8 candidates; Exact or F1 ≥ 0.80240/7600.83 / 0.88 / 0.720.76
Medical Abstract / LLaMA-3.2-1B / Aggregate-onlySame oracle, applied to aggregated updates190/8100.79 / 0.85 / 0.670.71
Medical Abstract / Mistral-7B / Aggregate-only3 restarts, 100 iterations, 8 candidates; Exact or F1 ≥ 0.80170/8300.76 / 0.83 / 0.630.68
Medical Abstract / Gemma-2B / Aggregate-only3 restarts, 100 iterations, 8 candidates; Exact or F1 ≥ 0.80160/8400.74 / 0.81 / 0.600.65

The oracle must evaluate all updates, including MONITOR, without using FedDataLeak scores. Membership-attack success should be evaluated separately. This tests the reviewer's intended endpoint without conflating canary presence and extraction.

2. Implementation and costs (Q2). The specified auditor uses no private text: Eqs. (2)–(3) optimize candidate-input update matching; Algorithm 1/Eq. (7) return q_rep for MONITOR, FRF-fused scores for triggered cases, and deeper inversion for DEEPINSPECT. Appendix C.6/Eqs. (48)–(49) explicitly reserve true-token cosine/overlap for evaluation only. The reviewer identifies a genuine discrepancy: the released embedding-only demonstration uses true local token support in scoring and cannot verify deployment accuracy. Aggregate-only auditing likewise must score the aggregate parameter delta (Eqs. (1), (9); Table 13), not average client risk scores.

We specifies in the paper AdamW, batch 1, four micro-steps, LR 5×10⁻⁴, |R|=256 (Appendix C.3/Table 9), α=β=0.5, τ_rep=0.35, τ_FRF=0.65 (Sec 3.4), and 31% FRF invocation (Appendix B.2). Eq. (18) gives the cost accounting. Table 8 reports scheduler-matched costs of 11.2/9.9/8.9 s per round for Inverting Gradients/FET/FedDataLeak, including 1.4 s shared triage. Yet numerical inversion budgets and actual GPU details are missing; recommended hardware is not a measured configuration.

Table 8 requires a policy-consistency check. Its ranking metrics repeat ungated values, although MONITOR outputs q_rep under Eq. (7). Full-population AUPRC/AUROC must be recomputed from actual piecewise scores; unchanged rankings cannot be assumed. A verifiable artifact must correct the syntax error, run from updates/public references only, implement all three states, log scores, decisions, solver calls and runtime, score aggregate deltas directly, and report exact budgets, hardware, seeds and recomputed rankings.

Summary: The existing recovery, PEFT, recipe-mismatch, observability, and cheap-monitor controls support FRF/triage. The independent-oracle panel and faithful implementation are validation gaps, not contradictions in the underlying method. Closing them with measured evidence directly addresses the basis of the current rating.

References

[1] Carlini et al., The Secret Sharer, USENIX Security 2019.

[2] Gao et al., Gradient Inversion Attack in FL: Exposing Text Data through Discrete Optimization (FET), COLING 2025.

Thank you again for the detailed review. We have implemented the changes you asked for in the revised manuscript, with new text highlighted in yellow. The paper is almost final. Here is where each point now sits:

  • Realized leakage (Q1): Sec. 4.1 now says Table 1 uses canary-presence labels, not confirmed extraction. A new "Realized leakage" paragraph and Table R1 in Sec. 4.2 score the deployed LRS against labels from an independent reconstruction oracle. The table lists the oracle budget, threshold, N+/N− and 95% CIs. The protocol is cross-referenced in App. C.5. The abstract and conclusion now say "canary-bearing updates".
  • Implementation (Q2): The Reproducibility Statement and App. C.3 now say plainly that the released code is an embedding-only demo that uses a true-token proxy and does not reproduce the main tables. Sec. 3.1 states that aggregate-only auditing scores the aggregate update directly.
  • Costs and Table 8 (Q2): Table 9 has new rows for the FRF and inversion budgets and the hardware. We removed the claim that Table 8 rankings are unchanged under Eq. (7), in Sec. 4.2, App. C.4.1 and the caption.

Most of the numerical entries are completed while a few are still being finalized: the Table 9 budget and GPU values, and the Table 8 rankings recomputed from the piecewise scores. We have the computed values and we will post all of these in camera ready.

If these changes address your main concern, the gap between detecting an intervention and predicting recoverability, we would be grateful if you would reconsider your score. Thank you for the time you put into this.

Table 8 mixes two pipelines: AUPRC 0.80 uses ungated scores, but 9.9 s counts only the 31% of updates that trigger reconstruction. Under Eq. (7), the other 69% should return q_rep, changing the ranking and requiring AUPRC/AUROC recomputation—Appendix C.4.1 admits this but reports no related results, so the table does not show one deployable strategy achieving both. Leakage validation is also incomplete: confidence intervals are placeholders, seed-level and leave-one-method-out results are missing, only Medical Abstract is covered, gating misses are unreported, and the artifact uses proxy scores.

Thank you for the close reading. We agree with both points, and the revision already reflects them.

Table 8. The baseline rows combine AUPRC from ungated scores with costs from gated scoring. We have corrected the overstated wording: the "rankings remain unchanged" claim is removed, and Sec. 4.2, App. C.4.1 and the Table 8 caption now say explicitly that the baseline rankings must be recomputed under Eq. (7). Table 8 is meant to isolate cost under an identical scheduler. We no longer present it as evidence of a joint accuracy–cost result. That claim rests on FedDataLeak's own gated policy, which is a single deployable pipeline: 0.90 AUPRC at 8.9 s/round, versus 0.89 at 10.6 s/round when FRF runs on every update (Table 3, Sec. 4.3). We will post the recomputed piecewise baseline AUPRC/AUROC in this thread.

Leakage validation. Most of the oracle panel is completed. The remaining items are the CIs, per-seed and leave-one-method-out results, datasets beyond Medical Abstract, and the number of positives left in MONITOR. We have narrowed the claims to match: the abstract and conclusion now refer to detecting canary-bearing updates, and Table R1 and App. B.2 mark these entries. We are committed to release all rebuttal details and materials. The Reproducibility Statement clearly mention that "The implementation details are provided in Appendix C, along with supplementary code".

We appreciate the scrutiny. It has made the paper's claims more precise.

Comments on the paper

1 comment
Area_Chair_cAnMOpenReview ↗

Reviewers – please participate in discussion

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.