acceptodds
Under review as a conference paper at ICLR 2027

JUMP: Efficient Membership Inference on Fine-Tuned Diffusion Language Models

Abstract

Membership inference attacks (MIAs) test whether a candidate example was used to train a language model. Existing attacks on fine-tuned discrete diffusion language models (dLLMs) often aggregate reconstruction signals across many mask configurations, requiring repeated model evaluations. We propose Joint Uncertainty Guided Mask Probing (JUMP), an efficient MIA that exploits the ability of dLLMs to predict masked tokens in parallel. Using the pre-fine-tuning checkpoint as a reference, JUMP selects low-confidence positions, masks them jointly, and aggregates clipped target–reference reconstruction gaps. This focuses the attack on positions that reveal stronger membership signals from fine-tuning. After mask selection, all selected tokens are evaluated with one scoring query per model. Across six MIMIR domains, JUMP improves mean ROC-AUC over a prior multi-mask attack from 0.819 to 0.902 on LLaDA and from 0.851 to 0.942 on Dream. Including mask selection, it requires only three forward passes per example, compared with 32 for the baseline. We further extend JUMP to the target-only setting by replacing target–reference scoring with relative token preference, which compares the observed token with alternative predictions at the same masked position. Target-Only JUMP achieves mean ROC-AUCs of 0.609 and 0.638 on LLaDA and Dream.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Language Models, Membership Inference Attack

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DPPcOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JUMP for membership inference on fine-tuned diffusion language models. Its reference-assisted method selects low-confidence positions, jointly masks them, and aggregates clipped target–reference reconstruction gains. Experiments on two model families and six domains demonstrate improved detection with substantially fewer model evaluations than SAMA. The paper also introduces a target-only extension based on grouped masking and entropy-centered scores.

Critical strengths

A compelling accuracy–efficiency improvement. Substantial experimental support for the main design.

Critical weaknesses

The robustness of the selected configuration and stringent operating points remains incompletely established. The default token budget, clipping threshold, and selector-training budget are justified through sweeps on the reported evaluation domains, without a clearly identified independent development set. Furthermore, each domain contains only 1,000 non-members, making 0.1% FPR estimates depend on approximately one false positive.

The target-only scoring rule has inconsistent benefits. Under identical grouped measurements and filtering, entropy centering improves LLaDA but underperforms the simpler MaskedScore baseline on Dream in both mean AUC (0.638 versus 0.645) and TPR at 10% FPR (0.200 versus 0.232). Comparisons with Random and CleanScore additionally change the probing budget and masking protocol. Consequently, the extension’s independent contribution is less convincingly established than that of the reference-assisted method.

Questions for authors

What explains the target-only entropy correction’s degradation on Dream, particularly GitHub? Is there evidence identifying when it improves upon matched masked log-probability scoring?

What exactly does entropy correction remove in the target-only setting? What source of variation unrelated to membership is the predictive entropy HH intended to correct for? Why does this correction help on LLaDA but reduce AUC on Dream GitHub from 0.6372 to 0.5810?

Why does clipping yield such a large improvement? In Table 19, adding clipping increases AUC from 0.8166 to 0.9024. Does this improvement primarily come from suppressing anomalously large positive gains among non-members, or from changing the ranking of member samples?

2 comments

W1: Tuning and low-FPR confidence. We interpret the low-FPR result at the resolution supported by the evaluation size. The submitted K=32/64/128K=32/64/128 sweep gives 0.8880/0.9024/0.86500.8880/0.9024/0.8650; the revision also reports sensitivity to the clipping threshold and selector budget. No disjoint development set was used. With 1,000 nonmembers, [email protected]% FPR corresponds to approximately one false positive. JUMP exceeds SAMA in all 12 cells at AUC, TPR@10% FPR, and TPR@1% FPR, and in 10 of 12 cells at [email protected]% FPR.

W2 / Q1 / Q2: Target-only controls and the Dream/GitHub exception. HH compares the true-token likelihood with uncertainty over alternative tokens, reducing ambiguity from context-dependent token difficulty. With the 16 grouped outputs and support size matched, Random, Clean, and Target-Only JUMP give 0.5780/0.60080.5780/0.6008, 0.6023/0.63560.6023/0.6356, and 0.6095/0.63750.6095/0.6375 on LLaDA/Dream. On Dream, q+H/qq+H/q gives macro AUC 0.6375/0.64540.6375/0.6454 and TPR@10% FPR 0.1997/0.23170.1997/0.2317. On Dream/GitHub, 67.2%67.2\% of positions are nearly deterministic. Filtering raises q+Hq+H from 0.54010.5401 to 0.58100.5810, but it remains below q=0.6372q=0.6372, showing that saturation explains only part of the gap. We therefore report explicitly that the entropy correction is helpful but not universal.

Q3: Clipping mechanism. Holding learned PRISM fixed, clipping raises macro AUC from 0.81520.8152 to 0.90240.9024. It corrects 10.98%10.98\% of member-nonmember pairs while reversing 2.26%2.26\%. The top-10% nonmember tail accounts for 30.81%30.81\% of the corrected pairs. Thus, broader reranking and tail suppression both contribute; the upper tail is enriched but not dominant.

The revision states the low-FPR resolution limit and provides the full target-only and clipping analyses in Appendices D and F.

Thank you again for the careful review. We have revised the paper and added the requested controls and analyses. We wanted to check whether the revision addresses your main concerns. Please let us know if anything remains unclear or if you have any additional questions.

Reviewer_18U9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies membership inference attacks (MIAs) against fine-tuned discrete diffusion language models (dLLMs). For settings where the pre-fine-tuning checkpoint is available, the authors propose Joint Uncertainty-Guided Mask Probing (JUMP), this method uses PRISM to estimate how confidently the reference model can reconstruct tokens across sequence positions, picks the K positions with the lowest confidence estimates, and masks them simultaneously, and it then scores examples by clipping and averaging the target-to-reference differences in true-token reconstruction log-probability. Because dLLMs predict all masked tokens in parallel, scoring requires a single forward pass per model. JUMP uses three forward passes per example including position selection.

The authors also present Target-Only JUMP for scenarios without a pre-fine-tuning reference, which reconstructs tokens using a balanced partition across 16 mask groups, filters out tokens where true-token reconstruction probability is near 1, and averages the relative token preference score q+H, where q is the true-token log-probability and H is predictive entropy.

In experiments section, the authors evaluate LLaDA-8B-Base and Dream-v0-Base-7B across six MIMIR domains. Compared to SAMA, reference-assisted JUMP increases mean ROC-AUC from 0.8193 to 0.9024 on LLaDA and from 0.8513 to 0.9415 on Dream, while cutting forward passes per example from 32 to 3. Also, target-Only JUMP reaches mean ROC-AUCs of 0.6095 and 0.6375, respectively. Additional experiments are impressive, which examine position selection strategies, score clipping, fine-tuning duration, reference checkpoint mismatch, and parameter-efficient tuning (LoRA and DP-LoRA).

Critical strengths
  1. The efficiency and accuracy gains are solid within the tested setup. In Table 1, JUMP outperforms SAMA across all six domains on both target models, reaching average ROC-AUCs of 0.9024 on LLaDA and 0.9415 on Dream while dropping required model forward passes from 32 to 3. Also, the reported online inference time per example drops from 1.863 seconds to 0.184 seconds, and the appendix separately accounts for the one-time cost of training the selector. Although JUMP trails at specific strict operating points (such as TPR at 0.1% FPR on LLaDA/Pile-CC and Dream/GitHub), its overall macro-averaged performance shows a substantial improvement.

  2. Impressively, the ablation and control experiments clearly outline where the approach holds up and where it falls off. The authors evaluate alternative selectors, token budgets, clipping thresholds, selector training sets, and fine-tuning epochs. It is reported that performance drops noticeably when using a mismatched reference checkpoint or when evaluating checkpoints saved after fewer fine-tuning epochs. Also, the PRISM-Free, target-only, and LoRA/DP-LoRA results further clarify the dependence on reference information and the fine-tuning setup. Reporting these weaker results just makes the paper much more grounded.

Critical weaknesses
  1. The evaluation split and fine-tuning setup limit how broadly these attack results can generalize. The main experiments rely on the original MIMIR train/test split. Although the authors verified that member and non-member sets do not share exact content hashes, this does not guarantee that the two groups have comparable prediction difficulty before fine-tuning. While the target-to-reference gap should reduce some of the confounding from static difficulty, a matched evaluation would provide more direct evidence. In addition, Appendix A.3 mentions that checkpoint selection across all six LLaDA domains relied on the MIMIR test split, which was subsequently reused as the non-member evaluation set. For experiments on GitHub and HackerNews, the fixed fourth-epoch controls yield even stronger JUMP results. This is reassuring about the direction of the effect, but it does not resolve the lack of independence between model selection and final evaluation.

    • The LoRA experiments also show sensitivity to the fine-tuning setup. In Appendix G.1, reference-assisted JUMP gets an AUC of 0.512 under rank-16 LoRA and 0.675 under rank-1024 LoRA, compared to 0.938 under full fine-tuning. This capacity test covers only one domain, so it does not settle every PEFT setting, but it hints that the high attack AUCs under full fine-tuning may not carry over to parameter-efficient tuning. I would like the paper to state these limits more explicitly.
  2. The individual gains from position selection versus score aggregation could be separated more cleanly. The experiments test separate selector variations and clipping checks, but lacks a full cross-comparison of selector choice and clipped versus unclipped aggregation on both model families. On Dream, PRISM-Free reaches a mean AUC of 0.9404, very close to JUMP's 0.9415 (though the learned PRISM selector retains an edge at low FPRs). On LLaDA, fixing the PRISM positions and adding clipping moves mean AUC from 0.8166 to 0.9024, indicating that clipping makes a substantial difference under this configuration. I see that the target-only extension also shows mixed results: on Dream, q+H averages 0.6375 AUC, which falls behind the Masked Score baseline (0.6454) under the same grouped masks and filtering rule. Therefore, I think a clearer attribution would help readers understand which components are responsible for the improvements.

Questions for authors
  1. Could you provide a matched evaluation, either by matching member and non-member examples on their pre-fine-tuning reference loss or by using a split balanced for difficulty? I'm interested in whether JUMP's margin over SAMA remains consistent under balanced difficulty. Could you also run checkpoint selection on a dedicated validation split separate from the final non-member test set? The existing fixed-epoch controls are helpful, but clarifying how much this evaluation-set reuse affects the magnitude and interpretation of the results.

  2. With the target/reference checkpoints, K, and evaluation examples held fixed, could you compare Random Joint, Entropy Top, PRISM-Free, and PRISM under both clipped and unclipped aggregation? Reporting both mean AUC and low-FPR numbers would be particularly helpful. Could you also confirm whether K=64, τ=log 1.5, and the 200k PRISM training budget were chosen in advance using independent development data or were informed by results on the six evaluation domains? In addition, extending the existing SAMA T=2 (4-forward) comparison to the six main domains would make the accuracy-cost comparison more complete. Also, a brief note comparing against the concurrent work Q-Skew (Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry) would also be welcome, though I do not view that comparison as a requirement for acceptance.

  3. Appendix A.4 lists 189,919 preprocessed PRISM training sequences, 4 GPUs, a per-GPU batch size of 4, gradient accumulation of 2, and 1 training epoch, but cites 2,968 total optimizer updates. With an effective batch size of 32 (4 x 4 x 2), a full epoch should take roughly 5,935 updates. Could you clarify the exact number of sequences seen, the batch definition, and how the reported update count relates to the stated training time? Please also clarify how the Dream PRISM selector was obtained (separately trained or transferred/adapted), including its training setup and offline compute footprint. Because default PRISM relies on internal representations from the reference model during offline training, please distinguish its access requirements clearly from PRISM-Free, which only uses output scores.

  4. While the expected log-ratio identity for q+H is valid, the score does not necessarily increase monotonically with true-token probability across different positions or varying output distributions. On Dream/GitHub, adding the entropy term causes a sharp drop in AUC. What is the intuition behind this drop, and under what conditions is the entropy adjustment intended to help? I see Appendix F.6 notes that the exact one-hole diagnostic only validates the approximation for q and the p_true cutoff, not the aggregated q+H score across a document. I think noting this scope limit in Section 6.3 would be helpful. If OK, could you compare the grouped and exact one-hole q+H document scores on a representative subset?

3 comments

W1 / Q1: Checkpoint independence, exposure, and difficulty. We separate checkpoint reuse, training exposure, and sample difficulty with three end-to-end seeds, completed-warmup final checkpoints, and outcome-blind matching. Completed-warmup runs use 8 warmup steps and checkpoint 80 with no checkpoint selection; JUMP is higher in all 12 cells (0.9709/0.97190.9709/0.9719 vs. 0.9128/0.91440.9128/0.9144 on LLaDA/Dream), and Loss AUC increases in all 12 cells. On LLaDA/GitHub, the most imbalanced cell identified without attack outcomes, matching reduces SMD from −0.2569-0.2569 to +0.0062+0.0062, while the JUMP-SAMA margin changes from 0.08570.0857 to 0.09220.0922. Submitted checkpoints were selected by reconstruction loss, not attack performance. The settings were selected from LLaDA sweeps and applied to Dream unchanged; no disjoint development set was used, which we state as a limitation.

W2: LoRA sensitivity. In the tested LLaDA/ArXiv cell, JUMP AUC falls from 0.9380.938 under full fine-tuning to 0.6750.675 with rank-1024 LoRA and 0.5120.512 with rank-16 LoRA. This control shows that JUMP weakens under LoRA; we do not generalize beyond this cell and limit the main claim to full fine-tuning.

W3: Selection versus clipping. Holding each selector and its positions fixed, changing only clipping improves Random Joint, Entropy Top, PRISM-Free, and learned PRISM at every reported operating point. Learned PRISM rises from 0.81520.8152 to 0.90240.9024 on LLaDA and from 0.87210.8721 to 0.94150.9415 on Dream.

Q2: Hyperparameters, SAMA budget, and related estimators. The revision reports KK, clipping-threshold τ\tau, and selector-training-size sweeps; settings were selected from LLaDA analyses and applied to Dream unchanged. The submitted K=32/64/128K=32/64/128 sweep gives 0.8880/0.9024/0.86500.8880/0.9024/0.8650. At 4/32/3 NFE, SAMA T=2T=2, standard SAMA T=16T=16, and JUMP give 0.7568/0.8193/0.90240.7568/0.8193/0.9024 on LLaDA and 0.7930/0.8513/0.94150.7930/0.8513/0.9415 on Dream. Q-Skew aggregates a concurrent statistic, whereas JUMP selects positions once and evaluates them jointly.

Q3: Selector exposure, access, and cost. We correct the earlier one-epoch description: 2,9682{,}968 are microbatch iterations, corresponding to approximately 1,4841{,}484 optimizer updates and 47,48847{,}488 sequence presentations; the 200k label denotes the data-pool budget. Default PRISM uses reference representations in the selector pass, whereas PRISM-Free uses output scores only. Training the separate Dream selector takes about 33 minutes on four L40S GPUs.

Q4: Grouping and entropy. HH compares the observed-token likelihood with the model's uncertainty over alternative tokens, reducing ambiguity from intrinsically difficult positions. Exact one-hole and grouped q+Hq+H give AUC 0.62220.6222 and 0.61690.6169 on LLaDA/ArXiv, with support Jaccard 0.92220.9222; q+Hq+H improves AUC in all six LLaDA domains. At nearly deterministic positions, both qq and HH saturate and q+Hq+H approaches zero. On Dream/GitHub, filtering raises q+Hq+H from 0.54010.5401 to 0.58100.5810, but it remains below q=0.6372q=0.6372; we report this exception and the scope limitation in Sec. 6.3.

The corresponding protocols and additional tables are provided in Appendices A-D and F.

Reviewer_18U9OpenReview ↗

Thanks for the Thoughtful Rebuttal

Thank you for the response, I really appreciate the effort you put into addressing my concerns. The additional experiments on fixed checkpoints, difficulty matching, selector and clipping comparisons, and the lower-budget SAMA baseline are especially helpful, they make the main findings more convincing. I also appreciate your honest discussion of the limitations and the clarifications on the PRISM training setup.
I think most of my concerns have been addressed or substantially alleviated. A few limitations are still there, but actually, no paper is perfect, and I don't think every open question needs to be fully resolved for a paper to make a valuable contribution. The new results have strengthened my confidence in this work, and I hope they will be included in the final version. ;)

So, I'm glad to raise my rating from Weak Accept to Accept. Kudos to the authors for the careful experiments and all the solid work behind this paper. Wish you all the best.

Thank you for taking the time to read our response and update your assessment. We appreciate your feedback throughout the discussion. Please let us know if you have any additional questions about the revision.

Reviewer_7gZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose JUMP, a MIA on fine-tuned discrete diffusion language models (dLLMs). In the reference-assisted setting, a PRISM selector trained on the pre-fine-tuning checkpoint ranks positions by predicted reference confidence. The lowest-confidence positions are masked jointly, and the clipped target-reference token-level reconstruction advantages are averaged, using three NFE per example. A variant, PRISM-Free, replaces the learned selector with the reference model’s clean-input true-token score. In the target-only setting, Target-Only JUMP uses balanced 16-way grouped masking, excludes near-deterministic positions, and averages the relative token preference. Experiments cover LLaDA-8B-Base and Dream-v0-Base-7B on six MIMIR domains, with Loss, ZLIB, and SAMA as baselines. The appendix adds controls on checkpoint selection, training exposure, selector design, probing budget, clipping, non-exact references, target-only design choices, and LoRA/DP-LoRA.

Critical strengths
  1. JUMP improves ROC-AUC over SAMA in every domain on both model families (Table 1), with about one tenth of SAMA’s forward passes and per-sample runtime (Section 5.1). When selector training is amortized over the 12,000-example evaluation suite, the total per-sample cost also remains well below SAMA’s (Table 20). This is the main reason the paper merits attention: if the gains survive the controls requested below, a cheap and accurate attack would make membership auditing of fine-tuned dLLMs practical.
  2. PRISM-Free needs no selector training and no hidden-state access, yet it still outperforms SAMA on both families (Table 2). This suggests the gain does not rely solely on the learned selector, and it makes the attack easier to deploy.
  3. The appendix reports checkpoint-selection robustness, exposure and mixed-domain controls, selector transfer, and LoRA/DP-LoRA experiments. It also discloses results that work against the method, such as the reuse of the non-member set for checkpoint selection and the sharp drop with non-exact references. This makes the method’s scope and limits assessable.
Critical weaknesses
  1. The baseline set is thin, and the comparison with the main baseline is underspecified. Table 1 compares against only three methods. SAMA is the only dLLM-specific baseline, and Loss and ZLIB stay close to chance in most domains. None of the stronger and more recent attacks discussed in Section 2 is adapted as a baseline, suchas reference-calibrated attacks, Min-K% and Min-K%++, ReCaLL, neighborhood-based and context-aware attacks, and information-theoretic membership inference. Without them, the claimed advantage rests on a single prior method. That comparison is itself underspecified: Eq. (2) defines SAMA as an average of target-reference reconstruction gaps, whereas Appendix A.5 describes subset-level comparison votes, and the two scoring rules are not obviously equivalent. The paper should state which rule was run and whether it follows the official implementation.
  2. The reference-assisted threat model is narrower than presented. Section 3 describes score access to the reference. However, PRISM is trained with LoRA adapters on the reference backbone, and at attack time it computes its scores from reference-model representations. The default attack thus requires hidden-state access to the reference both offline and online, while Appendix A.1 mentions representation access only for offline training. In addition, performance drops sharply when the exact pre-fine-tuning checkpoint is replaced by a related one (Appendix D.2). The strong reference-assisted results therefore do not establish comparable effectiveness in unknown-lineage or approximate-reference settings.
  3. JUMP clearly outperforms Random Joint, so position selection does contribute. However, removing clipping lowers JUMP’s mean ROC-AUC to roughly SAMA’s LLaDA level. Random Joint with the same clipped score already roughly matches SAMA in ROC-AUC on LLaDA and exceeds it on Dream (Table 14). Without evaluating SAMA under the same token-wise clipping, it is unclear how much of the improvement over SAMA comes from uncertainty-guided selection and joint probing rather than from clipping.
  4. The LLaDA and Dream targets of the main evaluation end training within the warmup phase and never reach the configured peak learning rate. Attack performance also depends strongly on exposure, with all attacks near chance after one epoch on LLaDA (Table 9). The paper does not show that its results hold under a standard schedule that completes warmup. Targets and selectors are trained with a single seed, and no standard deviations are reported for any method. The stability of the reported gains across independent runs is therefore unknown.
  5. The target-only contribution is insufficiently isolated. Random, Clean Score, and Ref. Guided score a fixed number of positions under one joint mask. Target-Only JUMP instead uses 16-way grouped masking over all valid positions, with filtering (Appendix A.7). The Table 3 controls therefore conflate the scoring rule with position coverage and query budget. Table 3 also contains no external baseline, and even Loss and ZLIB from Table 1 are absent. The only matched comparison, Masked Score, supports the relative token preference on LLaDA but not on Dream, where Masked Score has the higher mean ROC-AUC.
Questions for authors
  1. Which SAMA scoring rule was used in the experiments, the averaged gap of Eq. (2) or the subset-level comparison votes of Appendix A.5? Does it follow the official implementation?
  2. How was the Dream PRISM selector trained? Are the LoRA adapters active during the attack-time reference pass?
  3. Were the probing budget and the near-deterministic filtering threshold also chosen from sweeps over the evaluation domains? Was any tuning data disjoint from the evaluation sets used?
  4. Require a substantially broader baseline comparison in both settings, adapting the stronger and more recent attacks discussed in Section 2 to masked reconstruction outputs. These include the reference-calibrated attacks, Min-K% and Min-K%++, ReCaLL, neighborhood-based and context-aware attacks, and the information-theoretic approach.
  5. SAMA with the same token-wise clipping, and SAMA ROC-AUC versus NFE on all six domains and both families.
  6. Mean and standard deviation over multiple independent seeds for every method in Tables 1 to 3, including the baselines. The seeds should vary target fine-tuning, selector training, and the random masking used by SAMA and Random.
  7. Targets retrained with a schedule that completes warmup and a validation split disjoint from the non-members.
  8. Loss and ZLIB in Table 3, and target-only comparisons that vary only the position-selection rule while holding the masking budget and scoring rule fixed. Separately, on the same 16-way grouped outputs and filtering support, a Min-K%++ variant that also applies its selection of low-scoring tokens.
2 comments

W1 / Q1 / Q4: SAMA specification and broader baselines. Eq. (2) now matches the binary subset-vote aggregation used by the official implementation; the standard T=16T=16 results are unchanged. We evaluated the requested baselines on both families and all six domains. The revised tables include Ratio, InfoRMIA, SAMA, PRISM-Free, Ref. Guided, Loss, ZLIB, Min-K, Min-K++, ReCaLL, Neighbor, CAMIA, and Masked Score; their macro AUCs are summarized below.

SettingMethodNFELLaDADream
Reference-assistedRatio320.82360.8462
Reference-assistedInfoRMIA320.80580.8507
Reference-assistedSAMA (T=16T=16)320.81930.8513
Reference-assistedPRISM-Free30.86770.9404
Reference-assistedJUMP30.90240.9415
Target-onlyLoss160.52730.5461
Target-onlyZLIB160.53110.5508
Target-onlyMin-K160.56180.6085
Target-onlyMin-K++160.57560.6194
Target-onlyReCaLL320.53870.5547
Target-onlyNeighbornominal 970.50930.5072
Target-onlyCAMIA160.57370.6221
Target-onlyMasked Score160.58540.6454
Target-onlyTarget-Only JUMP160.60950.6375

W2 / Q2: Reference access and the Dream selector. Default JUMP uses reference-model representations in its single attack-time selector pass; we make this access explicit in Sec. 3 and Appendix A.1. PRISM-Free removes representation access and uses reference output scores only, while reaching 0.8677/0.94040.8677/0.9404, above SAMA's 0.8193/0.85130.8193/0.8513. Our reference-assisted claim assumes the exact pre-fine-tuning checkpoint and does not cover unknown-lineage or approximate references. Dream uses a separately trained selector with the same C4 recipe, and membership scoring uses the raw target and reference backbones.

W3 / Q5: Selection, clipping, and NFE. We ran the requested component and SAMA controls on both families and all six domains. For SAMA, clipped vote means clipping each cached token gap before the subset sum and sign vote. The paired PRISM control uses the full-sample estimate 0.81520.8152; the corresponding submitted Table 19 entry is 0.81660.8166.

Requested control (macro AUC)LLaDADream
Random Joint, unclipped →\rightarrow clipped0.7647→0.81720.7647\rightarrow0.81720.7854→0.87000.7854\rightarrow0.8700
Learned PRISM, unclipped →\rightarrow clipped0.8152→0.90240.8152\rightarrow0.90240.8721→0.94150.8721\rightarrow0.9415
SAMA paired control, unclipped →\rightarrow clipped vote0.8193→0.82380.8193\rightarrow0.82380.8513→0.85360.8513\rightarrow0.8536
SAMA T=2T=2 / standard T=16T=16 / JUMP0.7568/0.8193/0.90240.7568/0.8193/0.90240.7930/0.8513/0.94150.7930/0.8513/0.9415

Clipping changes SAMA only slightly (+0.0044/+0.0023+0.0044/+0.0023). On LLaDA, neither clipped random selection (0.81720.8172) nor unclipped learned selection (0.81520.8152) exceeds standard SAMA (0.81930.8193); combining learned selection and clipping yields 0.90240.9024. JUMP therefore remains more accurate while using 3 rather than 32 NFE.

W4 / Q6 / Q7: Seeds and warmup. We jointly varied target, selector, and attack/masking seeds; the revised appendix reports three-seed macro AUC for the submitted methods plus Min-K, Min-K++, ReCaLL, and InfoRMIA. Across three seeds, JUMP/SAMA give 0.9006±0.0031/0.8178±0.00460.9006\pm0.0031/0.8178\pm0.0046 on LLaDA and 0.9380±0.0057/0.8492±0.00190.9380\pm0.0057/0.8492\pm0.0019 on Dream. Completed-warmup controls use 8 warmup steps and checkpoint 80 with no checkpoint selection. JUMP is higher in all 12 cells (0.9709/0.97190.9709/0.9719 vs. 0.9128/0.91440.9128/0.9144), and Loss AUC also rises in all 12 cells, indicating a more strongly memorized setting.

Q3: Hyperparameter selection. We agree that an independent development set would provide stronger validation. The submitted K=32/64/128K=32/64/128 sweep gives 0.8880/0.9024/0.86500.8880/0.9024/0.8650, and varying the probability threshold changes macro AUC by less than 0.0050.005. These settings were selected from the LLaDA sweep and transferred to Dream unchanged. We did not use a disjoint development set and state this limitation explicitly.

W5 / Q8: Target-only controls. Table 3 reports all target-only methods above. For the requested selection controls, we matched the grouped-output budget and support size. Ref. Guided remains in Table 1 because its selector uses the reference model.

MethodLLaDADream
Random0.57800.6008
Clean0.60230.6356
Matched Min-K++0.59310.6302
Target-Only JUMP0.60950.6375

Target-Only JUMP has the highest macro AUC among these matched controls, although the margins are small. On Dream, the masked score qq alone is higher (0.64540.6454), which we report directly. These changes now appear in Eq. (2), Tables 1 and 3, and Appendices A-D and F; the response retains the headline results so they can be assessed without consulting the appendices.

Thank you again for the detailed review. We have now completed the multi-seed evaluation requested in W4/Q6 and added the full results to Appendix B.3. The appendix reports per-domain ROC-AUCs over 3 seeds for all methods in Tables 1 and 3, on both LLaDA and Dream. JUMP remains stable, with macro AUCs of 0.9006±0.00310.9006 \pm 0.0031 and 0.9380±0.00570.9380 \pm 0.0057, respectively. We hope these results address the remaining concern about seed robustness.

Comments on the paper

1 comment

Thank you for the careful reviews and constructive suggestions. We uploaded a revised manuscript with updates marked in blue. We completed the following additional experiments and revisions in response to the reviewers' requests:

  • Broader baselines (Reviewer 7gZn). We added seven external baselines across both model families and all six domains.
  • Training robustness (Reviewer 7gZn). We expanded the evaluation to three end-to-end seeds for every method in the original main tables and retrained all 12 model–domain targets with completed warmup.
  • Evaluation controls (Reviewers 7gZn and 18U9). We added predetermined final-checkpoint evaluations to address checkpoint-selection concerns, and reference-loss matching on the most imbalanced setting to address sample-difficulty differences.
  • Selection, clipping, and query cost (Reviewers 7gZn, 18U9, and DPPc). We added SAMA budget and clipping comparisons, controlled selector-by-clipping ablations, and an analysis of how clipping changed member–nonmember rankings.
  • Target-only controls and failure analysis (Reviewers 7gZn, 18U9, and DPPc). We added selection controls using the same grouped reconstruction outputs and investigated the entropy correction's degradation on Dream/GitHub.
  • Exact one-hole validation (Reviewer 18U9). We compared exact and grouped q+Hq+H scoring on all 2,000 LLaDA/ArXiv candidates.
  • Specification and access clarifications (Reviewers 7gZn and 18U9). We corrected the SAMA formula and selector-training accounting, and clarified reference-access requirements and Dream selector training.

Across three seeds, JUMP retained its accuracy–cost advantage over SAMA: mean macro AUCs were 0.9006/0.9380 versus 0.8178/0.8492 on LLaDA/Dream, using 3 rather than 32 model evaluations per example.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.