acceptodds
Under review as a conference paper at ICLR 2027

D-Miner: Extract Alignment Data from Masked Diffusion Language Models

17 views

Abstract

Masked diffusion language models (DLMs) have emerged as promising alternatives to autoregressive models for language generation. However, their iterative denoising and bidirectional generation may also facilitate the extraction of alignment data used in post-training. Existing extraction methods often rely on target prefixes or costly sampling and filtering, limiting their applicability and efficiency in this setting. In this work, we propose D-Miner, a novel framework that steers the denoising process of DLMs to efficiently generate alignment records from masks alone. Specifically, a lightweight corrector shifts intermediate hidden representations within the frozen DLM. We use membership signals to select the model’s own generations for hidden-state and token-level supervision. We theoretically establish that an ideal corrector can transform the model’s output distribution into one conditioned on successful recovery. Extensive experiments on 12 target DLMs demonstrate improved recovery coverage under semantic reference matching. We further show that the extracted data provides effective supervision for improving downstream coding performance. Code is available at https://anonymous.4open.science/r/D-miner.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
masked diffusion language models, alignment data extraction, memorization, membership inference, activation steering

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8rXZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a data extraction framework named D-Miner targeting Masked Diffusion Language Models (DLMs). The framework extracts post-training alignment data by inserting a lightweight hidden-layer corrector and guiding the generation process using a self-generated MScore signal.

Critical strengths

Novel Problem and Approach: Exploring the data extraction risks of DLMs starting from a pure mask state is an interesting and relatively new problem. The idea of guiding generation through lightweight feature shifts in the hidden layers is also somewhat inspiring.

Critical weaknesses
  1. I am not an expert in the security domain, but I have concerns regarding the realistic plausibility of the paper's settings. The paper assumes that an attacker can arbitrarily tamper with the activation values of intermediate layers (e.g., layer 16) during decoding. However, in a black-box API scenario, the attacker cannot access the hidden states, rendering this method completely ineffective. In an open-source white-box scenario, since the attacker already possesses the complete model weights, they can typically employ more efficient white-box attack methods (such as gradient analysis, weight inversion, or fine-tuning) to extract information. The authors need to provide a more compelling argument for why this specific threat scenario is indispensable in the real world.
  2. This paper uses "semantic matching" to evaluate the success rate of extraction. If the model generates high-quality responses that are semantically consistent with the training set but differ in exact wording, this actually proves that the model is well-aligned and generalizes well. Equating this with "training data leakage" appears to be a conflation of concepts.
Questions for authors

Same as Critical Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8rXZ (1/2)

We appreciate the reviewer's recognition of the problem and of the hidden-state guidance idea.

To W1: Threat-Model Practicality

“they can typically employ more efficient white-box attack methods (such as gradient analysis, weight inversion, or fine-tuning) to extract information” “The authors need to provide a more compelling argument for why this specific threat scenario is indispensable in the real world.”

We would like to clarify that our access assumption reflects a common deployment setting. Leading models such as DeepSeek and GLM release their weights for local deployment, while their post-training data remain unavailable. The DLMs we study (Dream, LLaDA, ESPO) are released in the same way. The attacker therefore holds the released weights but not the post-training data, nor any artifact of the training process such as gradients. Black-box APIs are outside our scope.

The white-box methods mentioned by the reviewer assume access beyond the released weights:

  • Gradient inversion [1–3] requires training gradients. It reconstructs a training batch from the gradient computed on that batch, which is observable only during training (e.g., in federated learning).
  • Weight inversion [4] requires a small model trained to convergence. It has not been shown for 7–8B language models.
  • Fine-tuning-based extraction [5] requires known training samples. It fine-tunes the target on known samples to elicit further ones.

D-Miner requires none of these: it uses only the released weights and starts from a fully masked input.

Although these assumptions do not hold in our setting, we transfer the gradient-based and fine-tuning-based methods to the released weights to test whether they recover more training records than D-Miner. Both start from the same candidates selected by MScore as D-Miner, so that the comparison differs only in how the candidates are used, and each generates 1,024 outputs per seed over three seeds:

  • Gradient-based optimization (adapted from [6]). Since training gradients are unavailable, we use the gradients of the released model, as in soft-prompt extraction [6]: we optimize 20 soft-prompt embeddings so that the model reproduces the candidates, keep the model frozen, and generate with the prompt prepended.
  • Fine-tuning (adapted from [5]). We fine-tune the target with LoRA on the candidates and generate from a fully masked input.

As Table 1 shows, D-Miner recovers more training records than both transferred methods on all four targets, even though they are given the same candidates and may optimize the input or update the model weights. Gradient-based optimization stays below Simple on all four targets, and fine-tuning exceeds Simple only on the two LLaDA targets. Thus, the white-box methods that the reviewer mentions either require information that released models do not provide or recover fewer training records than D-Miner.

Table 1. UMS of the transferred white-box methods (mean over three seeds; Simple and D-Miner from Table 2 in the paper).

TargetSimpleGradient-based optimizationFine-tuningD-Miner
Dream-PubMed389.7154.095.0471.3
Dream-GitHub145.044.359.7195.3
LLaDA-PubMed269.0146.0273.7407.0
LLaDA-GitHub57.010.397.3219.0

[1] Zhu et al. Deep Leakage from Gradients. NeurIPS 2019.

[2] Balunović et al. LAMP: Extracting Text from Gradients with Language Model Priors. NeurIPS 2022.

[3] Petrov et al. DAGER: Exact Gradient Inversion for Large Language Models. NeurIPS 2024.

[4] Haim et al. Reconstructing Training Data from Trained Neural Networks. NeurIPS 2022.

[5] Chen et al. The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks. CCS 2024.

[6] Zhang et al. Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation. ACL 2023.

AuthorsOpenReview ↗

Response to Reviewer 8rXZ (2/2)

To W2: Generalization vs. Data Extraction

“If the model generates high-quality responses that are semantically consistent with the training set but differ in exact wording, this actually proves that the model is well-aligned and generalizes well.”

We believe that generalization in capability and leakage of private data do not conflict: a model can generalize well and still expose its post-training data. To verify this, we conduct an experiment on ESPO-MATH that compares outputs reflecting generalization with the outputs of D-Miner.

Setting. The high-quality outputs that reflect generalization are the 197 correct responses of ESPO-MATH to the MATH-500 test problems (held out from its training data). These responses and the D-Miner outputs are matched against the post-training data of ESPO-MATH with the same encoder and threshold as Table 2 in the paper, and each response is matched both with and without its problem to separate the contribution of the given problem. The metric is the fraction of outputs whose nearest post-training record has similarity ≥ 0.90, so a higher fraction means that more outputs are close to the post-training data.

If semantic matching only rewarded high-quality responses that generalize, as the reviewer suggests, the correct responses would be matched to the post-training data at least as often as the outputs of D-Miner. Table 2 shows the opposite:

  1. The correct responses are the generalization case that the reviewer describes. They are semantically consistent with the reference solutions but differ in wording: none of them is identical to its reference solution, while their mean embedding similarity to it is 0.89. The model produces them only as answers to the problems we provide.
  2. Generalization yields only a limited match rate. Even with the problem included, 23.9% of the correct responses are matched, and the rate drops to 18.3% when only the response is matched. Part of their matches thus comes from the problem we provide rather than from the generated solution.
  3. D-Miner is matched more often, without any input. D-Miner receives no problem, yet 31.8% of its outputs are matched. Generalization alone therefore does not explain the matches of D-Miner.

Overall, capability and leakage coexist in the same model: generalization appears only when a problem is given, whereas D-Miner exposes the post-training data without any input.

Table 2. Fraction of outputs matched to the post-training data of ESPO-MATH (similarity ≥ 0.90).

OutputSim ≥ 0.90
Correct responses on MATH-500 (problem + response)23.9%
Correct responses on MATH-500 (response only)18.3%
D-Miner31.8%

Dear Reviewer 8rXZ,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_jB5KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I read this paper as a study of how to extract post-training alignment data from masked diffusion language models (DLMs) without supplying target prefixes or partial records. D-Miner uses a lightweight corrector to modify hidden states in a frozen DLM, guiding generation from fully masked inputs toward alignment records. The corrector learns from the model's own outputs selected using membership signals, with supervision at both the hidden-state and token levels. The authors show theoretically that an ideal corrector would produce the model's output distribution conditioned on successful recovery. Their experiments on 12 target DLMs suggest that D-Miner covers more distinct records than direct generation under semantic reference matching and, with a larger return budget, can surpass filtering from a fixed candidate pool. They also report that training another model on the extracted data improves its performance on four coding benchmarks.

Critical strengths

S1. What I find useful here is that D-Miner trains its corrector from the model's own generations, without requiring target records or prefixes. This fits the threat model in Section 2.2, where generation starts from fully masked content positions. Section 3.3 explains how selected candidates supply hidden-state-level and token-level supervision while the DLM remains frozen.

S2. The improvement over Simple on all 12 targets is a strength in my view. With 1,024 returned outputs, D-Miner has higher UMS than Simple on all 12 targets and than both filtering baselines on 10, despite the filters drawing from a candidate pool three times larger. On LLaDA-HackerNews, for example, UMS is 429.3 for D-Miner, 100.7 for Simple, and 190 for the better filter. I take this as evidence that guiding generation can reach more distinct reference records than generating and then filtering under these settings. I interpret the higher UMS as better coverage under the semantic matching criterion in Section 4.1, rather than as evidence of exact training-record recovery.

Critical weaknesses

W1. I am unsure how much of the UMS improvement reflects recovering training records rather than generating similar material from the same domain. Section 4.1 determines matches using BGE-M3 similarity to the nearest reference record, with thresholds of 0.80 for SFT targets and 0.90 for the released models. In discussing the Simple outputs in Figure 3(c), Section 4.3 describes SFT outputs as being from the right domain while rarely reproducing a specific record. I could not find a control in the main text showing how often the generated outputs would also match records from the same domain that were not used to train the target model. I accept the choice to evaluate semantic rather than verbatim recovery, as stated in Section 5. My concern is whether the semantic matches are specific enough to support the interpretation that training data has been recovered.

W2. The coverage gains per generated sample do not tell me whether D-Miner would still outperform the baselines at the same total computational cost. Section 3.3 requires candidate generation, repeated masking trials for MScore, and two stages of corrector training before guided generation. Section 4.1 explicitly states that the main comparison fixes the return budget while generation and computational budgets differ. I could not find a comparison in the main text that includes these preparation costs and guided inference in a common computational budget. The fixed 3,072-candidate pool result in Section 4.2 does not answer this question either. I would need that comparison before concluding that D-Miner reduces the total computation needed to reach a given UMS.

Questions for authors

Q1. In Equation (2), could the authors clarify whether D^\widehat{\mathcal D} is a set of distinct outputs or a collection that retains duplicates? Section 4.1 counts duplicate outputs toward the return budget. If D^\widehat{\mathcal D} is intended to retain duplicates, would an indexed sequence or multiset be more appropriate?

Q2. Could the authors state alongside Equation (13) how cases with It=∅I_t=\varnothing are handled? Since the loss is divided by ∣It∣|I_t|, explicitly stating whether such masks are resampled or excluded would make the definition clearer.

Q3. Figure 1 illustrates recovery after generating an incorrect token, whereas Section 3.2 states that previously committed tokens remain unchanged. Could the authors clarify whether Figure 1 assumes remasking or revision of committed tokens, and how this relates to the decoding procedure used in the experiments?

Q4. Section 4.2 states that D-Miner “equals KMeans-Filter's coverage” on ESPO-Code, but Table 2 reports UMS values of 114.3 and 113, respectively. Would “approximately matches” be more accurate here?

Q5. Could the authors clarify which outputs are included in the Similarity column of Table 1 and what the ±\pm values represent? The reported means are below the 0.90 matching threshold, so stating whether they cover all 1,024 outputs or a selected subset would help interpret the table.

6 comments
AuthorsOpenReview ↗

Response to Reviewer jB5K (1/2)

We thank the reviewer for the careful reading and the concrete questions.

To W1: Specificity of Semantic Matches

To further test whether the semantic matches are specific to the training records rather than to similar material from the same domain, we match the extracted outputs against both the training records and same-domain records that were not used for training, as in our response to Reviewer vsWq.

Setting. For each of the eight SFT targets, (i) we construct a reference pool containing 900 SFT training records and 900 held-out records sampled from the corresponding domain in the MIMIR test split [1]; (ii) for each successfully extracted record, we identify its nearest match in the combined pool using the same BGE-M3 semantic similarity measure; and (iii) we report, as the mean ± SD over three generation seeds:

number of nearest matches in training recordsnumber of nearest matches in training records+number of nearest matches in held-out records\frac{\text{number of nearest matches in training records}}{\text{number of nearest matches in training records}+\text{number of nearest matches in held-out records}}

Higher values indicate a stronger preference for training records over same-domain held-out records. If the matches reflected only similar material from the same domain, both sets would be matched equally often, giving 0.5.

As Table 1 shows, training records account for 64% to 80% of nearest matches on all eight targets: the generated outputs match same-domain records that were not used to train the target model in only 20–36% of cases, although such records make up half of the pool. Thus, the semantic matches favor the training records over other records of the same domain, which is consistent with recovering training data at the level of content. We add these results as Table 17 in Appendix E.5 of the revised paper.

Table 1. Fraction of nearest matches belonging to the training set, using a reference pool of 900 training and 900 same-domain held-out records per target (mean ± SD over three generation seeds).

DomainDreamLLaDA
GitHub0.797 ± 0.0720.775 ± 0.014
PubMed0.684 ± 0.0220.794 ± 0.015
HackerNews0.720 ± 0.0140.699 ± 0.027
ArXiv0.674 ± 0.0470.640 ± 0.015

To W2: Comparison at Equal Total Compute

We agree that an equal-compute comparison complements Table 2 in the paper. We therefore compare D-Miner with the baselines under a common total budget that includes its one-time preparation. Specifically, on the four SFT domains of Table 2 in the paper, we measure the GPU time of each stage on an RTX A6000 (Table 2 below), so that the total cost of returning RR outputs is

CSimple(R)=R⋅cS,CD-Miner(R)=Cprep+R⋅cD,C_{\text{Simple}}(R) = R \cdot c_{\mathrm{S}}, \qquad C_{\text{D-Miner}}(R) = C_{\mathrm{prep}} + R \cdot c_{\mathrm{D}},

and at each total budget, each method returns as many outputs as the budget allows. As Table 2 shows, the one-time preparation of D-Miner is small: it takes about 50 min (≈ 0.84 GPU-h) in total and is paid only once. We compare with Simple, since the Sample + Filter baselines need three times the generation budget of Simple to return the same number of outputs.

Table 2. GPU time of each stage (mean over Dream and LLaDA on the four domains).

StageGPU timePaid
Candidate generation and MScore~27 minonce
Corrector training (hidden-state and token-level supervision)~24 minonce
Preparation in total (CprepC_{\mathrm{prep}})~50 min (≈ 0.84 GPU-h)once
Generation of one output, Simple (cSc_{\mathrm{S}})~15.6 sper output
Generation of one output, D-Miner (cDc_{\mathrm{D}}, with the corrector)~16.2 sper output

As Table 3 shows, at the cost of one extraction run (5.44 GPU-h for 1,024 outputs), D-Miner covers more distinct records than Simple on all four domains, and the difference between D-Miner and Simple increases with the budget on every domain. Since extraction aims to recover as many distinct records as possible, it operates at the scale of Table 2 in the paper, where the one-time preparation is amortized. Thus, at this scale D-Miner reaches a given UMS with less total computation than Simple. We add these results as Tables 15 and 16 in Appendix E.4 of the revised paper.

Table 3. UMS at equal total GPU time (Simple / D-Miner), averaged over Dream and LLaDA. 5.44 GPU-h is the cost of D-Miner at R = 1,024.

Domain2 GPU-h3 GPU-h4 GPU-h5.44 GPU-h
GitHub60.2 / 82.275.8 / 129.893.7 / 167.0116.2 / 207.8
ArXiv137.8 / 145.2185.3 / 243.0220.8 / 323.0268.2 / 409.8
HackerNews95.2 / 127.5134.0 / 212.8167.0 / 279.0205.7 / 366.8
PubMed188.8 / 162.8257.3 / 265.5309.3 / 344.5381.2 / 438.8

[1] Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM 2024.

AuthorsOpenReview ↗

Response to Reviewer jB5K (2/2)

To Q1:

“In Equation (2), could the authors clarify whether D^\widehat{\mathcal D} is a set of distinct outputs or a collection that retains duplicates?”

D^\widehat{\mathcal D} retains duplicates. The duplicates in Sec. 4.1 (L337–338) mainly refer to different outputs that match the same record, which are counted by AMS but not by UMS (Eq. (14), L348). Identical generated texts are rare (at most about 1% of outputs per target), so the set notation in Eq. (2) (L139) closely reflects the returned outputs. Following the reviewer's suggestion, we define D^\widehat{\mathcal D} as a multiset in the revised paper.

To Q2:

“Could the authors state alongside Equation (13) how cases with It=∅I_t=\emptyset are handled?”

Draws with no masked token are excluded rather than counted as zero (Appendix F.11, L1780–1781), and the population form of Eq. (13) likewise conditions on It≠∅I_t\neq\emptyset (Appendix F.9, L1635). We state this after Eq. (13) in the revised paper.

To Q3:

“Could the authors clarify whether Figure 1 assumes remasking or revision of committed tokens, and how this relates to the decoding procedure used in the experiments?”

Figure 1 does not assume remasking or revision of committed tokens, and our experiments likewise keep committed tokens unchanged (Sec. 3.2). The two are consistent: in Figure 1(b), the deviating token had stays committed, and the remaining masked positions, which see context on both sides, still lead to an output close to x⋆x^\star (★), i.e., semantically similar to the record, which is the recovery we evaluate. Figure 1 thus illustrates that a DLM can still reach a record after a deviation without revising it. Intuitively, remasking samplers [1] may further improve extraction, since they could also revise such deviations.

To Q4:

“Section 4.2 states that D-Miner ‘equals KMeans-Filter's coverage’ on ESPO-Code … Would ‘approximately matches’ be more accurate here?”

D-Miner's 114.3 is the mean over three seeds with a standard deviation of 8.0, while KMeans-Filter's 113 comes from a single filtered set, so the difference of 1.3 is far smaller than the variation across seeds. We change the wording in Sec. 4.2 (L361–362) to "approximately matches".

To Q5:

“Could the authors clarify which outputs are included in the Similarity column of Table 1 and what the ± values represent?”

The column covers all 1,024 outputs per length (L116–117) and reports the mean ± standard deviation of each output's nearest-neighbor similarity, as in the full sweep of Table 10 (Appendix C.1, L1087–1090). Because unmatched outputs are included, the means fall below the 0.90 threshold. We state this in the caption of Table 1.

[1] Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling. NeurIPS 2025.

Thank you for the detailed rebuttal and additional experiments. I appreciate the effort to address my concerns.

The equal-compute comparison addresses my W2 concern, and the clarifications to Q1–Q5 are helpful.

Regarding W1, the new member/non-member analysis is helpful, but one concern remains. Appendix E.5 of the revised manuscript indicates that Table 17 only considers outputs already selected based on their similarity to the training records. This selection favors the training set, so 0.5 is not necessarily an appropriate baseline for the comparison. The reported results therefore do not fully resolve my concern about training-data-specific recovery.

I would consider W1 largely addressed if the authors repeated this analysis on all generated outputs, matching each against the combined training and held-out pool without prior training-set-based selection. A clear preference for training records under this evaluation, ideally with Simple evaluated in the same way, would be convincing.

At this stage, while the rebuttal substantially addresses my other concerns, I consider W1 unresolved. I am therefore keeping my rating unchanged.

Reviewer_UFgMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents D-Miner, a new framework for extracting alignment data used in the post-training of masked diffusion language models (MDLMs). The method consists of three main components, generating candidate samples from fully masked inputs and selecting potential alignment records using Mscore, training a lightweight corrector through activation-level flow matching and token-level supervision with KL regularization, and applying the learned corrector to guide the denoising process toward alignment records. The author provides both theoretical analysis and actual STF and RL results on various dlm models, showing D-Miner could effectively extract sementically similar alignment data than baselines.

Critical strengths
  1. I think the idea of extracting alignment data from dlms without using any target-specific prefix is interesting and useful.
  2. The experimental results are good overall. The authors evaluate 12 target dlms with both SFT and RL settings, demonstrating improvements than sampling-and-filtering baselines in most settings.
  3. The design of the proposed framework is valid. The authors combine membership-based candidate selection with a learned activation corrector to guide the denoising process. Each part has some motivations to support.
Critical weaknesses
  1. The theoretical analyses of the ideal corrector and its approximation are board and clear. Though the actual connection between the theorem and the practical algorithm is somehow limited.
  2. The controlled SFT experiments use relatively small training sets of 900 records per target. Whether D-Miner would maintain its effectiveness on models trained with substantially larger and more diverse alignment datasets. The corrector is trained using only 32 selected candidates, I wonder whether such a small candidate set would provide sufficient guidance in larger-scale SFT settings.
Questions for authors
  1. Theorem 3.1 establishes the existence of an ideal recovery corrector, which provides a useful theoretical perspective. Could the authors further elaborate on how this result motivates the practical design of D-Miner, particularly the use of MScore for candidate selection and dual-level supervision for training the corrector?
  2. I am interested in the broader implications of this work for future DLMs extraction research. When no target-specific prefixes or records are available, what are the key design considerations for developing an extraction algorithm? Could future methods better leverage the unique properties of DLMs, such as bidirectional attention, to guide the extraction process? It would be helpful if the authors could discuss some general insights from D-Miner which may inform some future research directions.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UFgM (1/2)

We appreciate the reviewer's recognition of the prefix-free setting, the evaluation on both SFT and RL targets, and the design of the framework. Below we address the questions.

To Q1 & W1: How Theorem 3.1 Motivates the Practical Design

Theorem 3.1 motivates the practical design by fixing the target that D-Miner approximates. Below we elaborate on how approximating this target with only the released weights leads to each component, including MScore for candidate selection and dual-level supervision for training the corrector.

Theorem 3.1 (L198–206) specifies the target of extraction: an ideal corrector C⋆C^\star under which sampling yields pθ(x0∣c=1)p_\theta(\mathbf x_0\mid c=1), i.e., the model's own outputs conditioned on successful recovery. The attacker cannot compute C⋆C^\star, since it is a ratio of recovery probabilities (L208–211). Approximating C⋆C^\star with only the released weights leads to three design choices:

  • Where to apply the correction: hidden states. Since the weights are frozen, D-Miner approximates the effect of C⋆C^\star by a shift of intermediate hidden states (Eq. (7), L223), and Theorem 3.2 shows that such a shift realizes the correction up to a second-order error. The required correction depends on the current denoising state (Appendix F.4), so the shift is produced by a learned corrector that reads the hidden state, mask ratio, mask indicator and position.
  • What to learn from: MScore. Training this corrector requires samples from the target pθ(x0∣c=1)p_\theta(\mathbf x_0\mid c=1), but cc is not observed. Since the target is a distribution over the model's own generations, MScore selects the generations with the strongest membership signal to stand in for c=1c=1. Appendix F.12 (Eq. (36)) shows that favoring high-scoring generations raises the recovery probability when the score is positively correlated with cc.
  • How to learn: dual-level supervision. Eq. (7) asks for a hidden-state shift whose effect appears in the token predictions, whereas the selected candidates give token targets but no hidden-state shifts (L293–294). Supervision is therefore applied at both levels: flow matching fits the corrector to the hidden states of the candidates (Eq. (12), L297–305), and masked-token cross-entropy through the frozen model fits the corrected token predictions to the candidates, with a KL term limiting the change given the small candidate set (Appendices F.9 and F.10).

Thus, Theorem 3.1 fixes what D-Miner must approximate, and the components supply what the theorem leaves unspecified: MScore supplies samples that stand in for the unobserved success event, and dual-level supervision fits the hidden-state corrector to them. Theorem 3.1 is an existence result: how closely the trained corrector approaches C⋆C^\star depends on the selected candidates (as noted in Appendix F.5, L1532–1533).

To W2: Larger Training Sets and 32 Candidates

Beyond the 900-record SFT targets, the four released RL targets in Table 2 in the paper show whether D-Miner maintains its effectiveness on substantially larger and more diverse alignment data, and whether 32 candidates still provide sufficient guidance at that scale. Although we do not run SFT experiments at a larger scale, their post-training sets are much larger than the 900-record SFT sets and span code, mathematics, and general-domain distillation data (Appendix A.2):

  • Dream-Coder: RL-17k (17,030 rows);
  • ESPO-Code: AceCode-87K, from which the official recipe uses the hard subset (17,430 records);
  • ESPO-MATH: the MATH training split (7,500 problems);
  • Dream-Bridge: the General-Distillation subset of KIMI-K2.5-1000000x (75,000 examples).

All of them use the same number of candidates (32) as the SFT targets, and D-Miner improves coverage over Simple on all four targets (Table 2 in the paper). These results indicate that D-Miner remains effective on larger and more diverse post-training sets, and that 32 candidates can guide extraction at this scale. The candidates are not meant to cover the post-training set: they supervise the correction of the denoising process, and coverage comes from the 1,024 returned outputs. A systematic study of larger and more diverse SFT mixtures and of the candidate count remains valuable, and we discuss it as a limitation in Appendix H of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer UFgM (2/2)

To Q2: Design Considerations and Future Directions

We thank the reviewer for this broader question. Below we summarize the design considerations that shaped D-Miner and the directions it suggests for future extraction research on DLMs.

Design considerations without prefixes or records. Two observations shaped D-Miner:

  • Denoising reveals memorization. How confidently a DLM reconstructs a span indicates whether it was trained on it [1, 2]. Without target records, the model's own denoising is the available guidance signal, which D-Miner uses through MScore.
  • Sampling and filtering repeatedly recover the same records. From masks alone, the 1,024 outputs of Dream-Coder match only 85 distinct records, so filtering spends most of the budget on unmatched or repeated outputs. D-Miner therefore steers the denoising process itself rather than filtering its outputs.

Leveraging the properties of DLMs.

  1. Bidirectional context already strengthens privacy attacks. Related work [2] finds that infilling around revealed context extracts up to three times more training sequences than prefix-based extraction. D-Miner shows that extraction remains possible even when nothing is revealed.
  2. Revocable decoding can strengthen extraction further. Recent decoders remask committed tokens [3]. Applied to extraction, this would let guidance revise committed deviations from a memorized record.

We add this discussion as future work in Appendix H of the revised paper.

[1] Chen et al. Membership Inference Attacks Against Fine-tuned Diffusion Language Models. ICLR 2026.

[2] Wang and Asokan. Extracting Training Data from Diffusion Language Models via Infilling. arXiv:2605.24173, 2026.

[3] Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling. NeurIPS 2025.

I thank the authors for their response. Most of my concerns have been addressed, and I decide to maintain my current score. The study of data extraction on dlms is interesting.

We appreciate your positive feedback and are glad that most of your concerns have been addressed. We will incorporate the clarifications into the final version.

Reviewer_vsWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies extracting alignment data from diffusion language models using only fully masked inputs. D-Miner ranks the model's own generations by a membership score and trains an activation corrector on the top-ranked ones to steer decoding, with theory showing that an ideal corrector exists. Experiments on 12 SFT and RL targets measure matched reference records, and a code fine-tuning study evaluates the recovered data.

Critical strengths
  1. At an equal budget, D-Miner recovers more distinct matched records than direct sampling on all 12 targets in the three-seed mean, with gains ranging from modest to more than fourfold. It also outperforms filtering baselines that use three times the budget on 10 of 12 targets.
  2. The problem is timely. Steering decoding with a corrector trained only on the model's own membership-ranked samples is a new, data-free mechanism.
Critical weaknesses
  1. The evaluation does not separate recovered training records from ordinary in-domain text. A match is simply a similarity above a threshold, the held-out MIMIR test split is never used as a non-member reference for the extraction matches, and the utility study lacks a control. This undermines the central extraction claim.
  2. No ablation trains the corrector on random instead of top-ranked samples, so the gains may reflect generic self-training rather than membership guidance.
Questions for authors

For the eight SFT targets, please match the existing outputs against a size-matched sample of the held-out test split, and report results for a corrector trained on random samples. A clear member-over-non-member margin would raise my assessment.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vsWq (1/2)

We thank the reviewer for the precise and actionable questions.

To W1 & Q: Member-over-Non-Member Margin with Same-Domain Held-Out Records

“please match the existing outputs against a size-matched sample of the held-out test split”

Following your suggestion, we compare the extracted outputs against both training and same-domain held-out records to assess whether their semantic matches are specific to the target's SFT data.

Setting. For each of the eight SFT targets, (i) we construct a reference pool containing 900 SFT training records and 900 held-out records sampled from the corresponding domain in the MIMIR test split [1], where the held-out records follow the same data distribution as the training records but were not used for SFT; (ii) for each successfully extracted record, we identify its nearest match in the combined pool using the same BGE-M3 semantic similarity measure as in Sec. 4.1; and (iii) we report, as the mean ± SD over three generation seeds:

number of nearest matches in training recordsnumber of nearest matches in training records+number of nearest matches in held-out records\frac{\text{number of nearest matches in training records}}{\text{number of nearest matches in training records}+\text{number of nearest matches in held-out records}}

Higher values indicate a stronger preference for training records over same-domain held-out records.

As Table 1 shows, we observe a clear member-over-non-member margin across all eight targets: training records account for at least 64% of nearest matches, with this proportion approaching 80% in several cases. Thus, most extracted outputs remain closer to training records even when an equally sized set of same-domain held-out records is available as alternative references. This comparison provides a more specific assessment of the reported training-record matches than matching against the training set alone. We add this analysis as Table 17 in Appendix E.5 of the revised paper.

Table 1. Fraction of nearest matches belonging to the training set, using a reference pool of 900 training and 900 same-domain held-out records per target (mean ± SD over three generation seeds).

DomainDreamLLaDA
GitHub0.797 ± 0.0720.775 ± 0.014
PubMed0.684 ± 0.0220.794 ± 0.015
HackerNews0.720 ± 0.0140.699 ± 0.027
ArXiv0.674 ± 0.0470.640 ± 0.015

To W1: Utility of D-Miner Data Compared with Unguided Generations

“the utility study lacks a control.”

We add a control to test whether the utility gains obtained with D-Miner data can also be achieved using ordinary, unguided model outputs. This comparison evaluates the additional training value of D-Miner data beyond simple generations.

Specifically, we construct a control dataset containing 109 outputs generated by Dream-Coder without D-Miner guidance (Simple). We fine-tune Dream-v0-Instruct-7B on either this control dataset or 109 examples extracted by D-Miner, using the same data size, filtering procedure, and fine-tuning recipe. We then evaluate both models on four coding benchmarks and include the original model without fine-tuning as a reference.

As shown in Table 2, D-Miner data consistently outperform the control across all four benchmarks, with additional Pass@1 gains reaching nearly 3 percentage points. In contrast, the unguided control brings only small changes over the base model: its gains stay below 2 percentage points, and it even falls slightly below the base model on MBPP (67.99 vs. 68.30). Thus, under the same data budget and fine-tuning settings, D-Miner data provide greater downstream utility than unguided generations. We add these results to Table 3 of the revised paper as a new Control column.

Table 2. Pass@1 (%) before fine-tuning and after fine-tuning on 109 examples from each data source.

BenchmarkBase (no fine-tuning)Control (unguided data)D-Miner data
HumanEval60.9862.2064.02
MBPP68.3067.9969.05
BigCodeBench23.0023.2526.23
LiveCodeBench9.7011.3713.55

[1] Duan et al. Do Membership Inference Attacks Work on Large Language Models? COLM 2024.

AuthorsOpenReview ↗

Response to Reviewer vsWq (2/2)

To W2 & Q: Ablation on Guidance Samples

“No ablation trains the corrector on random instead of top-ranked samples, so the gains may reflect generic self-training rather than membership guidance.”

To test whether the gains reflect generic self-training rather than membership guidance, we repeat Table 2 in the paper on the eight SFT targets with a single change: the corrector is trained on samples drawn at random from the same candidate pool (Random), instead of the same number of top-ranked ones. We report UMS as in Table 2 in the paper (higher means more training records recovered), together with Δ, the gap between D-Miner and the stronger of Simple and Random.

As Table 3 shows, Random does not reproduce the gains: D-Miner outperforms it on all eight targets. Therefore, the gains come from the MScore-based selection rather than from generic self-training. We add these results as Table 18 in Appendix E.6 of the revised paper.

Table 3. UMS of the corrector trained on random samples (Δ denotes the gap between D-Miner and the stronger of Simple and Random).

TargetSimpleRandomD-MinerΔ
Dream-GitHub145.0122.3195.3+50.3
Dream-ArXiv246.7142.0388.7+142.0
Dream-HackerNews261.382.0304.3+43.0
Dream-PubMed389.7377.0471.3+81.6
LLaDA-GitHub57.0119.3219.0+99.7
LLaDA-ArXiv219.7329.7430.7+101.0
LLaDA-HackerNews100.7147.7429.3+281.6
LLaDA-PubMed269.0314.3407.0+92.7

Dear Reviewer vsWq,

Thank you again for your time and valuable feedback. We kindly ask whether our rebuttal has addressed your concerns, and we would be happy to discuss any remaining issues.

AuthorsOpenReview ↗

Additional Result on W1

We provide an additional analysis for W1. Table 1 is computed over the successful extractions so that it stays consistent with the main extraction results in Table 2 of the paper. Following a related question from Reviewer jB5K, we extend this analysis to all generated outputs to further test whether D-Miner recovers training records rather than ordinary in-domain text.

Setting. We keep the reference pool (900 training and 900 held-out records) and the matching protocol of Table 1. Both D-Miner and Simple are evaluated on all generated outputs without any selection or filtering.

Table 4. Fraction of all generated outputs matched to training records (mean ± SD across three seeds).

DomainDream SimpleDream D-MinerLLaDA SimpleLLaDA D-Miner
GitHub0.498 ± 0.0070.653 ± 0.0130.518 ± 0.0190.608 ± 0.007
PubMed0.535 ± 0.0020.613 ± 0.0180.590 ± 0.0090.646 ± 0.020
HackerNews0.546 ± 0.0220.656 ± 0.0150.511 ± 0.0130.606 ± 0.015
ArXiv0.581 ± 0.0130.638 ± 0.0130.549 ± 0.0030.609 ± 0.017

Without any selection or filtering, over 60% of D-Miner's outputs match training records on every target, although the pool contains equal numbers of training and held-out records. D-Miner also exceeds the Simple baseline on all eight targets by 0.056 to 0.155. Thus, D-Miner's outputs remain closer to the training records than to same domain held-out text, which shows a clear member-over-non-member margin of all outputs.

We hope this additional analysis addresses your concern regarding W1.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Brief Introduction to the Revised PDF

We thank all reviewers for their time and constructive feedback. All changes are marked in blue. The line numbers below refer to the revised PDF, while those in our responses refer to the original submission.

We revise the text as summarized below:

  • Revised statements: we revise several statements to make them more precise. The caption of Table 1 states how the Similarity column is computed (L112–114), the extracted collection is defined as a multiset (L142), draws with an empty mask set are excluded after Eq. (13) (L315–316), and "equals" is changed to "approximately matches" (L362).
  • Limitations and future work: Appendix H (L2134–2154) adds a limitation on the size of the SFT data and the number of candidates, and a paragraph on future work.

Furthermore, we add four tables and revise one existing table to address the reviewers' questions:

  • Table 3 adds a Control column, fine-tuned on unguided generations with the same filtering and recipe.
  • Tables 15 and 16 report the GPU time of each stage of D-Miner and compare D-Miner with Simple under a common computational budget.
  • Table 17 shows that successful extractions are closer to training records than even to unseen records of the same domain.
  • Table 18 reports the extraction results when the corrector is trained on randomly drawn candidates instead of those selected by MScore.
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

We sincerely thank Reviewers vsWq, UFgM, jB5K, and 8rXZ and the Area Chair for their careful evaluation and constructive feedback.

Significance, Novelty & Strengths Recognized by Reviewers

Masked DLMs have emerged as promising alternatives to autoregressive models in instruction following, reasoning, and code generation. Their bidirectional, iterative denoising may also facilitate training-data extraction, yet extracting alignment data without target-specific prefixes or partial records remains largely unexplored. We are encouraged that the reviewers recognized the contributions of our work:

  • Timely and interesting problem of prefix-free extraction (Section 2). Unlike existing attacks that rely on known prefixes or partial records, D-Miner extracts alignment data from fully masked inputs (highlighted by all Reviewers: vsWq, UFgM, jB5K, 8rXZ).
  • Novel alignment-data extraction mechanism guided by membership signals (Section 3). Instead of sampling and filtering, a lightweight corrector shifts the frozen DLM’s hidden states and is trained solely on the model’s own membership-ranked generations, which Reviewer vsWq describes as “a new, data-free mechanism” (Reviewers vsWq, UFgM, jB5K, 8rXZ).
  • Efficient extraction across models and training regimes, surpassing filters with 3× the generation budget (Section 4). Under an identical return budget of 1,024 outputs, D-Miner recovers more distinct records than direct sampling across all 12 targets under SFT and RL, and outperforms filtering baselines using three times the generation budget on 10 of 12 targets (Reviewers vsWq, UFgM, jB5K).

Summary of Experiments

Guided by the reviewers’ suggestions, we conducted the following experiments:

  • Non-member control with same domain held-out records (Reviewers vsWq, jB5K). Against 900 training and 900 held-out records, training records account for 64%–80% of nearest matches among successful extractions (Table 17, Appendix E.5) and over 60% among all unselected outputs, exceeding Simple on every target (Table 4 in the discussion). We will add the all output analysis to Appendix E.5 in the final version.
  • Ablation on candidate selection (Reviewer vsWq; Table 18, Appendix E.6). Replacing MScore-selected candidates with random ones lowers UMS across all eight targets, and even falls below direct sampling on all four Dream targets. The gains thus come from membership-based selection rather than generic self-training.
  • Control baseline for data utility (Reviewer vsWq; Table 3). D-Miner data outperform unguided generations on all four coding benchmarks. These results support the additional training value of D-Miner data over unguided generations under the same fine-tuning recipe.
  • Computational resource analysis (Reviewer jB5K; Tables 15 and 16, Appendix E.4). Including one-time preparation, D-Miner covers more records than Simple across all four domains at the GPU time of a single extraction run.
  • White-box baselines and generalization analysis (Reviewer 8rXZ). Transferred to the released weights, gradient-based optimization and fine-tuning recover fewer records than D-Miner on all four tested targets. Without any input, 31.8% of D-Miner outputs match the ESPO-MATH post-training data, above 23.9% for correct responses to held-out problems.

Summary of Discussion Status

  • Reviewer jB5K (Score: 2 → 3). After our initial response, Reviewer jB5K confirmed that W2 and Q1–Q5 were addressed and requested an all-output analysis for W1. After this follow-up, Reviewer jB5K considered the main concerns adequately addressed and raised their score, noting that it “provides stronger evidence that D-Miner’s improvements are not simply due to ordinary in-domain generation.”
  • Reviewer UFgM (Score: 3, most concerns resolved). Reviewer UFgM stated that most concerns had been addressed, maintained the rating of 3, and noted that “the study of data extraction on dlms is interesting.“
  • Reviewer vsWq (Score: 2, no follow-up yet). Reviewer vsWq’s main concern (W1) is whether matches reflect training records rather than ordinary in-domain text, which is the same question as Reviewer jB5K’s W1. Our non-member controls on successful extractions and on all unselected outputs resolved this concern for Reviewer jB5K. We believe they also address the concern of Reviewer vsWq, as they show the “clear member-over-non-member margin” that Reviewer vsWq named as an explicit signal for raising the assessment. The requested ablation and utility control are also provided.
  • Reviewer 8rXZ (Score: 2, no follow-up yet). We responded to both concerns with the white-box baseline comparisons and the generalization analysis above.

We thank the reviewers and the Area Chair again for their time and consideration, and we would appreciate it if Reviewers vsWq and 8rXZ could be encouraged to review our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.