- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes JUMP for membership inference on fine-tuned diffusion language models. Its reference-assisted method selects low-confidence positions, jointly masks them, and aggregates clipped target–reference reconstruction gains. Experiments on two model families and six domains demonstrate improved detection with substantially fewer model evaluations than SAMA. The paper also introduces a target-only extension based on grouped masking and entropy-centered scores.
Critical strengths
A compelling accuracy–efficiency improvement. Substantial experimental support for the main design.
Critical weaknesses
The robustness of the selected configuration and stringent operating points remains incompletely established. The default token budget, clipping threshold, and selector-training budget are justified through sweeps on the reported evaluation domains, without a clearly identified independent development set. Furthermore, each domain contains only 1,000 non-members, making 0.1% FPR estimates depend on approximately one false positive.
The target-only scoring rule has inconsistent benefits. Under identical grouped measurements and filtering, entropy centering improves LLaDA but underperforms the simpler MaskedScore baseline on Dream in both mean AUC (0.638 versus 0.645) and TPR at 10% FPR (0.200 versus 0.232). Comparisons with Random and CleanScore additionally change the probing budget and masking protocol. Consequently, the extension’s independent contribution is less convincingly established than that of the reference-assisted method.
Questions for authors
What explains the target-only entropy correction’s degradation on Dream, particularly GitHub? Is there evidence identifying when it improves upon matched masked log-probability scoring?
What exactly does entropy correction remove in the target-only setting? What source of variation unrelated to membership is the predictive entropy intended to correct for? Why does this correction help on LLaDA but reduce AUC on Dream GitHub from 0.6372 to 0.5810?
Why does clipping yield such a large improvement? In Table 19, adding clipping increases AUC from 0.8166 to 0.9024. Does this improvement primarily come from suppressing anomalously large positive gains among non-members, or from changing the ranking of member samples?
2 comments
Thank you again for the careful review. We have revised the paper and added the requested controls and analyses. We wanted to check whether the revision addresses your main concerns. Please let us know if anything remains unclear or if you have any additional questions.
W1: Tuning and low-FPR confidence. We interpret the low-FPR result at the resolution supported by the evaluation size. The submitted K=32/64/128 sweep gives 0.8880/0.9024/0.8650; the revision also reports sensitivity to the clipping threshold and selector budget. No disjoint development set was used. With 1,000 nonmembers, [email protected]% FPR corresponds to approximately one false positive. JUMP exceeds SAMA in all 12 cells at AUC, TPR@10% FPR, and TPR@1% FPR, and in 10 of 12 cells at [email protected]% FPR.
W2 / Q1 / Q2: Target-only controls and the Dream/GitHub exception. H compares the true-token likelihood with uncertainty over alternative tokens, reducing ambiguity from context-dependent token difficulty. With the 16 grouped outputs and support size matched, Random, Clean, and Target-Only JUMP give 0.5780/0.6008, 0.6023/0.6356, and 0.6095/0.6375 on LLaDA/Dream. On Dream, q+H/q gives macro AUC 0.6375/0.6454 and TPR@10% FPR 0.1997/0.2317. On Dream/GitHub, 67.2% of positions are nearly deterministic. Filtering raises q+H from 0.5401 to 0.5810, but it remains below q=0.6372, showing that saturation explains only part of the gap. We therefore report explicitly that the entropy correction is helpful but not universal.
Q3: Clipping mechanism. Holding learned PRISM fixed, clipping raises macro AUC from 0.8152 to 0.9024. It corrects 10.98% of member-nonmember pairs while reversing 2.26%. The top-10% nonmember tail accounts for 30.81% of the corrected pairs. Thus, broader reranking and tail suppression both contribute; the upper tail is enriched but not dominant.
The revision states the low-FPR resolution limit and provides the full target-only and clipping analyses in Appendices D and F.