- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper performs an in-depth empirical analysis of the EZ-MIA method, and shows that adding line-breaks to the text inflates AUC because of the design of the score function. The authors propose to report the scoring rule, edge cases, zero rates, and per-view scores so that MIA scores can be interpreted correctly. Overall, the manuscript is poorly written, the paper is an empirical observation that might be a better fit for other venues such as online forums, and the research project as a whole can benefit from more human oversight.
Critical strengths
The paper raises a good point that edge cases should be more thoroughly reported in MIA methods.
Critical weaknesses
The writing is unclear and hard to follow.
The main point of the paper is that MIA methods should report their edge cases and provide per-transformation scores. The point can be argued without a paper. A case study on a single MIA method does not show generalization to other methods or reporting conventions.
Questions for authors
None
2 comments
Remaining questions on the revision
Thank you for your review. The revised manuscript and our updated responses are now available. Are there any remaining concerns that you would like us to clarify or address? We would be happy to provide further details.
Response to TCdj: the revision makes the empirical contribution directly assessable
Thank you for asking for a clearer contribution and stronger evidence beyond a reporting recommendation. The revised introduction now separates scoring behavior, calibration, and the incremental value of aggregation. Added and substantively revised passages are blue. Locations below refer to the revision.
1. The scoring example identifies a mechanism with a measured boundary
The same passage can score 0 as a paragraph and 2.67 with line breaks because formatting moves the released EZ-MIA score out of its zero-denominator fallback. In the short-passage intermediate-update study, the convention changes AUROC by .091–.227 and reverses its ranking among six detectors across 48 fits. The upstream 128-token GPT-2/WikiText replication finds one member zero among 30,000 scores, showing where this mechanism is negligible. Appendix G reports the original-setting and error-count controls.
2. The revised comparison distinguishes calibration from aggregation
Suppose training used a rewrite of the auditor's sentence. Detecting exposure to that source differs from detecting the exact string. The revised manuscript states this membership target before comparing the original, a selected available rewrite, and their aggregates.
The expanded paraphrase-training study contains 96 fits. Its stronger controls change the interpretation of the earlier positive result.
Appendix F specifies held-out selection, adjustment families, shared-shadow dependence, and query cost. The shadow constructor knows the designated training paraphrase, which is excluded from scored audit views. The resulting claim is conditional on this access. It does not establish that averaging generally improves text membership inference.
3. The revision states what the study contributes and where it stops
The evidence shows how the same apparent transformed-query advantage can arise from a scoring fallback, calibration, or additional views. The calibrated single-view control determines which explanation the data support. This comparison changes the paper's own earlier positive interpretation, as the abstract and Section 5 now state.
Main Table 5 reports a validated LoRA recipe. Appendix F adds a limited 3B diagnostic and upstream neighbourhood/SPV-MIA integrations, with material deviations retained. These additions broaden the evaluation without turning one detector's fallback into a universal defect or claiming complete coverage of attack pipelines. The introduction presents the examples before the protocol terminology, and the conclusion states each finding with its comparator and scope.