- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1(−1)
Read review
Paper summary
This paper proposes an empirical framework, HERA, for assessing different methods aiming to ensure guarantees on "unlearnable data". I found this paper extremely difficult to read -- I have done substantial prior work, e.g., on ensemble classification methods, but I found it very difficult to parse this paper, primarily because the authors do not do a good job of making the paper reader-accessible. There is a lot of jargon that is not defined and a lot of motivation that is missing.
Critical strengths
Again, I had great difficulty following this paper.. the main strength of the paper is in defining a sound test-bed/testing framework for evaluating "unlearnable example" approaches. The authors have also conducted extensive experiments.
Critical weaknesses
There is much terminology and motivation that is not well-defined in this paper: shard is nowhere defined.; CA(q) is used throughout the paper but is only defined in sec. 5.3. In Figure 1, what is the purpose of the clean routing set ? This is not explained.
Both Theorem 1 and Theorem 2 appear very straightforward.
Some experimental results are not given convincing explanation by the authors (gain of 23.49 points)
Questions for authors
Line 42: "For an authorized curator, however, recovery answers..." -- What data does an authorized curator have access to ? Original data, or only corrupted data ? Line 43: "After training, a few records may need to be corrected or deleted" -- please give scenarios where correction/deletion may be needed. Line 49: "correct certificate coverage" -- please explain what certification "coverage" means here. "savings when deletions touch many shards" -- shards are never truly defined in this paper -- and "savings" with respect to what ? Line 74: CA(1) -- CA(q) is used throughout the paper, but isn't defined until sec. 5.3 -- is this what is meant by "certified coverage" ? Line 96: "error-minimizing noise" -- unclear what this is. Line 106: What is the main contribution/distinction of Hera, compared to DPA ? P. 131: what is the "frozen variant" here ? Line 178: For each shard, are you selecting one of the possible views (based on "routing accuracy" -- that is, the routing set is treated as a validation set for selecting the view ?) Line 266: "how much correct accuracy is certifiable" -- please define what certifiable means here.
Gain of 23.49 points -- "locates the gain with identifying its cause" and "HERA's bagging advantage remains unresolved" -- what are the causes for these findings ? -- Can you explain the behavior in Figure 2 (as mean-loss penalty lambda_j is varied ?)