- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents AUDITRACE, a method for attributing a specified model behavior to clients in federated PEFT. Each client records the coordinate indices and signs of its largest updates, while dense numerical updates can remain on a separate secure-aggregation path. The auditor compares these sparse records with signed gradients of a behavior-specific objective to rank supporting and opposing clients. The paper evaluates agreement with dense references and exact leave-one-client-out retraining, client-removal interventions, audit costs, and information leakage.
Critical strengths
- The paper addresses a clear and relevant problem and offers a simple way to obtain behavior-specific client rankings without revealing dense numerical updates to the auditor.
- The evaluation is broad, covering dense references, exact leave-one-out retraining, client removal, ablations, multiple models and PEFT settings, system costs, and leakage tests.
- The paper clearly states its limitations, including hidden-tail ambiguity, unavailable quantities in its sufficient conditions, and the absence of a formal privacy guarantee.
Critical weaknesses
- The methodological novelty over existing federated attribution and auditing methods (e.g., TraceFL (ICSE 2025), ProToken (MLSys 2026), INDEXGUARD (ICML 2026)) is insufficiently established. (Q1)
- Audit records may not describe the actual training update. Section 2.4 leaves this binding outside scope. Although the zero-knowledge proof in Appendix B verifies that an audit signature matches a committed hidden update, it does not establish that this committed update is the same update actually submitted to secure aggregation. A malicious client could therefore submit one update for training and a different one for auditing, unless an additional verifiable-input mechanism is integrated. (Q2)
- Appendix B tests a small proof circuit, not the millions of LoRA coordinates used in the experiments. Its timings therefore do not establish that clients can afford the proof during training. (Q3)
- The practical reliability of identifying the most influential individual client remains limited. In Table 2's ten-client AG News experiment, AUDITRACE ranks the client with the largest leave-one-out effect first for only 39% of queries, despite strong overall ranking correlation. (Q4)
- Attribution under client dropout is untested. Appendix D uses full participation. It is unclear how rankings change when clients participate in different subsets of rounds. (Q5-6)
- The manuscript needs more proofreading before submission. Several grammatical and formatting errors include but are not limited to:
- on page 1, "Against exact leave-one-client-out retraining it attains" -> ... retraining, it achieves
- on page 1, "within 0.020 Spearman of dense cosine", do you mean "within 0.020 in Spearman correlation of the dense-cosine baseline"?
- on page 1, "A 5% signature retains Spearman 0.860" -> A 5% signature retains a Spearman correlation of 0.860
- on page 6, what is the "IndexTrace" in fig 3?
- on page 15, "the final condition in Equation equation 13"
- on page 15, "with the circuit written in Circom (?)", what does ? mean, maybe a missing citation?
- on page 23, "increasing the number of monotonically improves", number of what?
Questions for authors
- What are the essential methodological differences between AUDITRACE and prior approaches such as TraceFL, ProToken, and INDEXGUARD? Beyond applying signed Top-K overlap to behavior-conditioned gradients, what fundamentally new capability or attribution principle does AUDITRACE introduce? Why are they not compared in evaluation? Could the authors provide more direct comparisons with them under matched information-disclosure budgets?
- How can the auditor verify that the update described by the audit record is the same update submitted to secure aggregation?
- What are the expected proof-generation time, verification cost, memory consumption, and communication overhead when the circuit scales from 32 private inputs to millions of PEFT parameters?
- Why does AUDITRACE achieve only 39% Top-1 recovery against exact LOO effects despite relatively high Spearman correlation and AUROC? Under what conditions can auditors reliably identify the most influential individual client rather than merely obtain an approximate overall ranking? Could the authors report Top-1 confidence or ranking margins, evaluate performance across different client populations, and extend counterfactual validation beyond the ten-client AG News setting?
- How should client dropouts be handled? When clients participate in only some training rounds, with different clients participating in different rounds, how does attribution accuracy change?
- Under partial client participation, does Eq. (4) rank average alignment per active round or cumulative influence? Could the authors test how unequal participation affects the resulting rankings?
5 comments
Thank you for the detailed response. Some concerns still remain:
-
Appd B addresses the update-binding concern at the protocol level. Could the authors clarify whether Table 7 tests the integrated protocol in the case where a valid audit proof corresponds to an update different from the one submitted for aggregation?
-
The new circuit benchmarks are useful, but it remains unclear whether they include commitment generation, EIFFeL input validation, and the associated communication. Will the additional commitment binding significantly affect the original AUDITRACE pipeline, e.g., PEFT update size, audit frequency, update encoding?
-
Do the proof-cost benchmarks in Table 8 include the overhead of the new audit-to-aggregation binding mechanism?
Response to Reviewer nYzp: input-binding test and full binding costs
Thank you, Reviewer nYzp, for the careful follow-up and for finding the new circuit benchmarks useful. We revised Appendix B accordingly (changes in red): Table 7 now includes the requested substitution test with EIFFeL validation, and Table 8 has two new panels with the binding costs. No retraining was needed: the added rows of Table 7 and panels B and C of Table 8 are computed on client updates already saved from the paper's existing training runs, using the Appendix B circuit templates; only the proofs and validation checks were run on these saved updates.
1. Does Table 7 test a valid audit proof paired with a different aggregation input?
Yes, the revised Table 7 now includes this case. The client logs a commitment to u with an accepting audit proof for u, then submits a different encoded vector v ≠ u as its aggregation input; EIFFeL validates the actual secure-aggregation input against the logged commitment.
| Outcome (20 client–round attempts) | Result |
|---|---|
| Audit proofs for u that verify | 20/20 |
| Substituted inputs v ≠ u admitted | 0/20 |
| Honest control inputs admitted | 20/20 |
Setup: n = 1,024, K = 103, real layer-0 q-projection LoRA-A updates from the AG News/Llama-3.2-1B run, 10 clients at rounds 5 and 10, signed 32-bit encoding with one scale per round. In every attempt the proof for u verifies, yet input validation rejects v, while all honest inputs are admitted.
2.1 Do the benchmarks include commitment generation, EIFFeL input validation and communication?
Yes. The new Table 8C reports both Groth16 relations for all 128 LoRA factors of one real client update (M = 3,407,872 coordinates; 16 cores, factors processed sequentially):
| Full adapter | Commitment | Witness + proving | Verification | Proof + public inputs |
|---|---|---|---|---|
| Audit relation | 144.2 s | 131.85 min | 147.7 s | 135,871 B |
| Input opening, Eq. (14) | 142.7 s | 70.21 min | 126.8 s | 124,161 B |
All 128 proofs verify for each relation. These are single-node times with the factors proved one after another. The 128 factor proofs are independent, so running them in parallel across nodes reduces the wall-clock time roughly in proportion to the number of nodes; the total compute is unchanged.
- Commitment generation. Inside the audit proof, the commitment opening is 380,829 of 5,120,753 constraints (7.44%) at n = 65,536. Native computation takes 2.4 s per 65,536 coordinates (Table 8B).
- EIFFeL input validation. The input-opening proof for Eq. (14) has 2,477,981 constraints at n = 65,536; proving takes 71.2 s and verification 0.95 s (Table 8B), and 70.21 min / 126.8 s for the full adapter.
- Communication. Each proof is about 805 B plus about 0.26 KB of public inputs; the ~1 KB in our first reply is this per-proof size. A full adapter uses one proof per factor, so 136 KB for audit proofs and 124 KB for input-opening proofs, with the commitment included as a public input.
2.2 Does the binding significantly affect the pipeline (PEFT update size, audit frequency, update encoding)?
No. All three are unchanged:
- PEFT update size. The adapter keeps its M = 3,407,872 trainable coordinates; the binding adds only the proof and public-input files listed in 2.1.
- Audit frequency. The audited rounds are unchanged. Audit proofs can be generated after the round; input validation precedes acceptance of the aggregation input.
- Update encoding. The signed 32-bit encoding keeps every selected index and sign in all 120 tested records (10 clients × 2 rounds × 3 budgets × 2 selection rules), so the replayed attribution scores are unchanged.
Because proofs are generated per LoRA factor, certified records use per-factor Top-K; at a 10% budget, Spearman correlation with the dense reference is 0.787 [0.751, 0.822], versus 0.808 [0.776, 0.840] for global Top-K.
3. Does Table 8 include the overhead of the binding mechanism?
Yes, the revised Table 8 now includes it. The original Table 8 measured the audit relation, which already contains the commitment opening (7.44% of constraints at n = 65,536); reusing that commitment for binding adds no constraints to it. New panels B and C add the remaining client-side binding costs: panel B gives native commitment computation and the Eq. (14) input-opening relation per 65,536 coordinates, and panel C gives both relations for the full adapter. We also renamed the "Complete" column to "Audit total" to make its scope explicit.
We hope the direct substitution test and the full cost breakdown answer the remaining questions. Your follow-up made Appendix B considerably more complete, and we would be grateful if you would reconsider your assessment in light of these additions.
Response to Reviewer nYzp on methodological distinctions and additional verification evidence
e sincerely thank you, Reviewer nYzp, for recognizing the value of behavior-specific client attribution, the simplicity of the method, and the breadth of its validation. Your suggestions helped us clarify what AUDITRACE adds to prior work and strengthen the evaluation with a sign ablation, proof benchmarks up to 65,536 coordinates, and Top‑2 and Top‑3 shortlist results. (colored in the revision).
W1/Q1: Novelty over TraceFL, ProToken and IndexGuard
AUDITRACE is, to our knowledge, the first method combining three properties: (i) the behavior is specified post hoc as any differentiable objective; (ii) the output is a signed ranking separating supporting from opposing clients; (iii) no individual dense update is observed, reconstructed or estimated (Sec. 2.3).
TraceFL and ProToken trace neuron provenance through individual client models, which secure aggregation withholds, so they cannot run under a matched disclosure budget. IndexGuard is a training-time, query-free screening defense. It produces no behavior-conditioned ranking.
The new principle is that signs of the sparse support, matched to a query gradient, carry the attribution signal. We clarified these differences in method and access to client information in Section 2.3 (Related Work), particularly in the Positioning paragraph.
W2/Q2: Is the audited update the one submitted to secure aggregation?
Appx. B now specifies this binding. The client commits once to its update; an adapted EIFFeL input validation accepts an aggregation input only if it opens that commitment, and the Groth16 proof shows the record is the signed Top-K of the vector opening the same commitment. Commitment binding then makes a "train one update, audit another" attack fail. In our implementation tests, 13 of 13 tampered-record cases are rejected and none of 2,310 fabricated candidates is accepted (Table 7).
W3/Q3: Proof cost at scale
We replaced the 32-input prototype with a dimension-parameterized circuit, measured across four vector sizes spanning a 64x range up to a full LoRA projection (2,000x larger than before), (Appx. B, Table 8). Verification is constant: about one second and 1 GiB at every size. Communication is constant and negligible: the proof and public inputs total about 1 KB per client and audited round, about 0.1% of the audit record they certify. Proving time grows sublinearly (22x for a 64x larger vector) and is paid off the training path.
W4/Q4: Top-1 recovery of 39%
39% does not mean the top contributor goes undetected. We added Top-k recovery and margins (Table 3):
The leading clients are near-ties (median first-second gap: 6.6% of the effect range; AUDITRACE's median top-1 regret: 3.4%), so adjacent clients swap even with full dense updates. With better-separated clients, Top-1 reaches 0.60-0.91 across five PEFT variants (Table 21), and the audit-score gap tells the auditor when one client can be named (AUROC 0.71). Fidelity holds from 50 to 300 clients (Spearman 0.895-0.961; Appx. D.3.2), and the BeaverTails client-removal intervention validates counterfactual effects beyond AG News (Table 14). We also extended exact LOO to 1% and 5% budgets: a 5% record keeps a Spearman correlation of 0.784 (Appx. D.2.2).
W5/Q5-6: Client dropout and unequal participation
New experiments (Appx. E.7):
Accuracy stays within 0.03 of full participation. Eq. (4) ranks average alignment per active audited round, not cumulative influence; replacing it with the cumulative sum changes Spearman only from 0.885 to 0.896. Under the phase-skewed schedule the ranking changes by less than 0.01.
W6: Proofreading
Thank you for catching these. All seven items are fixed (abstract wording, the Figure 3 label, "Equation equation", the Circom citation, the missing noun), and we proofread the whole paper for the same patterns (consistent "a Spearman correlation of", headings, captions and references).
We are grateful for this thorough review, which made the paper clearer and stronger.