- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the machine unlearning evaluation method and propose ATR (Artifact-Trace Reduction). Authors highlight the seed sensitivity of another evaluation methods. With ATR, which summarizes per-example unlearning trajectories into descriptors and measures the MMD between the forget set and reference descriptors. The descriptors combine loss, logit margin, gradient norm, and activation norm through six summaries. They report the results on the image classification and language model settings, demonstrating improved cross-seed stability and agreement with membership-based rankings in the evaluated settings.
Critical strengths
- The motivation is important and clearly presented. Cross-seed evaluation stability must be emphasized.
- ATR is intuitive and relatively simple.
- In particular, the advantage of ATR in terms of computational cost appears to be very significant.
- Experiments are well organized, these type of controlled analyses help explain the roles of the different components rather than only focusing on an overall improvement.
Critical weaknesses
**Major **
- The presentation of the paper should be improved. The placement of several figures and tables interrupts the reading flow.
- Authors argue that the direction of this indicator varies depending on the experimental setup. Doesn't this raise a fundamental concern about its generalizability as an evaluation metric? Although authors provide explanations for this behavior, but it is questionable why they do not address problems that could be solved with just minor improvements in metric design. Could authors further clarify this limitation?
- Is it really "retrain-free"? Although the ATR score itself can be computed without a retrained model, determining its appropriate direction in a novel unlearning setting appears to require additional calibration against a gold-standard (retrain) reference. Without such calibration, it is unclear how ATR can be reliably applied to new settings. This somewhat limits its generalizability.
Minor
- For the statements on 7B models, authors should more clearly distinguish experimentally validated results from claims based on indirect evidence. Especially, a valid membership-based ranking is not directly established at 7B scale.
- Setup section can be moved into the Experiments section. Wouldn't it be better to cover them together in one section rather than as separate ones?
- Calling the evaluation protocol "ATR-OOD" is somewhat misleading, given that the benchmark primarily relies on held-out in-distribution samples rather than genuinely OOD data. How about "ATR-HO" instead?
Questions for authors
See the weaknesses.
General comment
To all reviewers and the AC: what changed since our first responses (8 Nov).
Reviewer rpzJ's follow-up made us re-check our cost numbers against the job logs and test the low-cost setting the paper recommends. This gave new results. It also showed that some numbers in the paper and in our first responses are wrong. This comment lists both, once. No ranking averaged over seeds changes.
What the paper claims now
Before and after
Withdrawn or corrected
Where the detail is. The orientation rule and the cost breakdown are in our reply to Reviewer rpzJ, with the 7B account in its last comment.