- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes IBIS, an efficient canary crafting framework for one-run privacy auditing. The paper addresses the interference among multiple canaries, which can reduce membership inference attack (MIA) performance when canaries are randomly included in a single training run. IBIS combines influence-based greedy selection, which favors high self-influence and low cross-influence, with bilevel optimization that improves canary detectability while encouraging orthogonality in the embedding space. To reduce computational costs, the authors employ approximate implicit differentiation to incrementally update the model and canaries without retraining from scratch at every iteration.
Experiments on CIFAR-10, CIFAR-100, and MNIST demonstrate improved MIA performance over standard canary constructions. Compared with MetaGradient, IBIS achieves better attack performance while reducing canary crafting time by up to two orders of magnitude.
Critical strengths
S1. Meaningful problem formulation and methodological contribution. The paper addresses an important limitation of one-run privacy auditing: interference among canaries. Jointly optimizing canary detectability and non-interference is a meaningful contribution beyond methods that primarily optimize individual canaries. The combination of influence-based initialization and bilevel refinement is well motivated.
S2. Substantial computational improvements with encouraging empirical results. The incremental bilevel optimization avoids repeated model retraining, significantly reducing canary crafting costs. The experiments demonstrate improved MIA performance over existing approaches, and the ablation studies provide evidence that the proposed regularization reduces interference and improves audit performance.
Critical weaknesses
W1. Insufficient theoretical justification for the interference suppression mechanisms. Proposition 4.1 establishes a connection between interference and orthogonality under the metric induced by in least squares regression. However, the proposed regularizer instead enforces ordinary embedding-space orthogonality, which does not necessarily imply low cross-influence in deep neural networks. Furthermore, Equation (1) uses signed cross-influence in the denominator, although negative cross-influence can also cause interference. The objective may also become unstable when the denominator approaches zero. These issues weaken the theoretical justification for the proposed design.
W2. Limited experimental evidence for the broader effectiveness of privacy auditing. The experiments primarily consider relatively small-scale models and datasets. Moreover, the improvement over mislabeled canaries on CIFAR-100 is marginal (0.35 vs. 0.33 [email protected]), while the reported DP auditing results are comparable to MetaGradient rather than substantially better. Since the main motivation is improving privacy auditing, stronger evidence in terms of empirical DP lower bounds and more diverse model architectures would better substantiate the claims. Additionally, part of the runtime comparison relies on previously reported results, making the computational comparison less controlled.
Questions for authors
Q1. Why does Equation (1) use signed cross-influence rather than its absolute value? How are zero or negative denominators handled?
Q2. Why is ordinary embedding orthogonality an appropriate proxy for the -weighted orthogonality derived in Proposition 4.1?