acceptodds
Under review as a conference paper at ICLR 2027

Ambient Discrete Diffusion: Using the Wrong Data at the Right Time for Data Efficient Learning

22 views

Abstract

We introduce RefineMix, a framework for training discrete diffusion models under severe data scarcity, a common constraint in scientific applications. RefineMix uses out-of-distribution data at selected diffusion times to improve generalization without biasing the sampling distribution. Although this strategy has been explored in continuous diffusion, discrete diffusion presents a distinct challenge: unlike Gaussian noise, masking preserves domain information in surviving tokens, limiting the use of related data at high noise levels. At low noise levels, however, the domains’ effectively disjoint supports become an advantage, allowing the model to learn from both in-domain and out-of-distribution data without biasing the sampler. We formalize these intuitions and provide a theoretical analysis for the proposed method. Experimentally, across five domain-shift settings, RefineMix matches or outperforms in-domain finetuning and data mixing. For protein sequence generation, finetuning with just 197 in-domain examples nearly doubles the fraction of generated proteins that are simultaneously novel, foldable, and in-family compared to standard finetuning.

Primary area:
generative models
Keywords:
ambient, discrete diffusion, masked diffusion language models, data mixing, domain adaptation, transfer learning, out-of-distribution data, data-scarce training, language modeling, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EidBOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper aims to improve the performance of discrete diffusion models under severe data scarcity by leveraging out-of-distribution data. The authors analyze why an existing method developed for image generation (continuous diffusion models) fails when applied to text generation (discrete diffusion models). Based on this analysis, they propose a simple, widely applicable framework and validate it empirically.

Critical strengths
  • The paper provides an insightful analysis of why the existing method fails in the text generation.
  • The proposed method is well-motivated by the findings from this failure analysis.
  • RefineMix is easy to implement and widely applicable.
  • Extensive experiments including protein sequence generation supports the effectiveness of RefineMix.
Critical weaknesses
  • Validity of Assumption 3: When the shared-factor estimator has sufficient capacity and the sequence-content pairs (xt,v)(x_{t}, v) are disjoint between the source and target domains, the shared factor could potentially learn the two domains independently. Consequently, it is unconvincing to assume that the training error of the shared factor on the target domain scales inversely with the total number of training samples, rather than specifically with the number of samples in the target domain.
  • Justification for claims regarding discreteness: Discreteness does not appear to play a critical role in the theoretical analysis, despite RefineMix being presented specifically as a framework for discrete diffusion models. While the difference in C(t)C(t) between image and text generation is illustrated in Figure 1(b), C(t)C(t) could depend on several confounding factors, such as data modality, underlying training distributions, or the specific formulation of the diffusion process. It is not sufficiently justified that C(t)C(t) is predominantly governed by whether the variables are continuous or discrete.
Questions for authors
  • If a low-noise region exists where the source and target domain distributions are disjoint, would you expect RefineMix to be effective for continuous diffusion models as well?
4 comments

We thank the reviewer for the careful reading. Both concerns are fair.

W1 (Assumption 3). The reviewer is right. Assumption 2 only states that the true realization is shared across domains. It says nothing about whether a model actually learns it jointly. A model that fits the two domains with separate parameters would learn the shared factor from the NTN_T target samples alone. Assumption 3 is therefore an assumption on the estimator rather than on the data, and we will state this explicitly in the revision.

A weaker version suffices. We can decompose the error of the shared factor into a component informed by both domains and a component informed only by the target, eN(t)=esh(t)/N+epr(t)/NTe_N(t) = e_{sh}(t)/N + e_{pr}(t)/N_T. Theorem 4 then holds with eshe_{sh} in place of ee, since the private component is identical for both models and cancels in Δ\Delta. The reviewer's scenario corresponds to esh≡0e_{sh} \equiv 0, in which case the theorem gives Δ=−B(tmax⁡)≤0\Delta = -B(t_{\max}) \le 0, i.e., mixing never helps.

Empirically, eshe_{sh} is not zero. When we train from scratch, RefineMix lowers the held-out target perplexity, and it helps more when the target is smaller. With 1% of the target data, it lowers the perplexity by more than half (Figure 2(c)); both models train for the same 30k steps. When we raise λ\lambda from 0.1 to 0.95 below the cutoff, the perplexity falls from 46.7 to 22.1 (Table 6, seed 1). This is not just less overfitting. In Table 1, fine-tuning is stopped at its best checkpoint and uses its best learning rate, and RefineMix still lowers the target perplexity by 9–10% on the two arXiv pairs. We cannot explain this if nothing the model learns from the source carries over to the target.

On disjoint inputs. Held-out target sequences are themselves disjoint from the training sequences, so any 1/N1/N rate on held-out data, including 1/NT1/N_T, already relies on the network pooling information across disjoint inputs. How far this pooling extends across domains is an empirical question, but it is the same weight sharing that lets masked language models transfer between languages with no shared vocabulary (Conneau et al., ACL 2020).

W2 (Role of discreteness). We agree on both points. Our theorems do not use discreteness, and Figure 1(b) changes the data and the noise process at the same time.

Theory. The theorems only require that both domains share a Markov forward process that ends in a data-independent prior, and they reduce the question of where the source is safe to use to one quantity, the overlap C(t)C(t). The noise process enters only through this curve. Masking can only hide tokens, never change them, so the domains stay distinguishable as long as a few visible tokens identify the domain. Gaussian noise degrades every entry at once, which contracts the two noised distributions towards each other (Daras et al., 2025, Thm. 4.2).

Control experiment. To separate the noise process from the data, we corrupted the same images once by masking pixels and once with Gaussian noise (variance-preserving, αˉt=cos⁡2(πt/2)\bar\alpha_t=\cos^2(\pi t/2)), and trained the same domain classifier in each case (three seeds). We compare at the same fraction of surviving signal, 1−t1-t under masking and αˉt\bar\alpha_t under Gaussian noise. On CIFAR-10 vs. SVHN, balanced accuracy stays above 0.9 until only 2% of the signal survives under masking, whereas under Gaussian noise it falls below 0.9 already at 21%. On the closer pair CIFAR-10 vs. CIFAR-100 (clean accuracy 0.78 for masking, 0.81 for Gaussian), accuracy stays above 0.75 down to 20% under masking, compared with 65% under Gaussian noise.

Scope of our claim. The gap in Figure 1(b) is therefore not merely an artifact of comparing text with images: on the same data, masking keeps the domains separable considerably longer than Gaussian noise. We do not claim that the variable type itself is the cause, and we will state the claim consistently as one about the noise process and add the control experiment to the appendix.

Q (Continuous diffusion). Yes, and we tested this directly. We trained Gaussian diffusion models with RefineMix on CIFAR-10 (target) and SVHN (source, λ=0.9\lambda = 0.9) at six cutoffs and measured the fraction of generated samples that belong to the target domain (one seed). At every cutoff, this fraction matches the prediction of Theorem 1, computed from the domain classifier, to within about 0.03: it is 0.94 at tmax⁡=0.5t_{\max} = 0.5, 0.61 at tmax⁡=0.8t_{\max} = 0.8, and 0.07 with full mixing.

This experiment tests where the source is safe to use, not whether it also improves the target there, which we leave to future work. The results suggest that Theorem 1 also describes Gaussian diffusion. The main difference is that the safe cutoff is lower, since Gaussian noise makes the domains indistinguishable earlier. We will add this experiment to the appendix.

Thank you for the detailed response.

Regarding W1.

I agree that knowledge transfer can occur even when the source and target content supports are disjoint, and the empirical results provide evidence. My main concern was not that knowledge transfer is impossible, but that Assumption 2 alone does not justify the equal contribution of source and target samples assumed in Assumption 3. I would find the original Assumption 3 reasonable if the authors more explicitly positioned it as an idealized setting in which knowledge transfer is particularly effective, and clarified the scope of the resulting theoretical analysis. This might also avoid introducing an additional decomposition of the estimation error, which can require further justification.

Regarding W2.

While the additional experiments provide useful evidence for the importance of the noise process, I am not sure whether these results alone justify replacing discreteness with the noise process as the central explanation. At the same time, I do not think it is necessary to identify which factors primarily determine separability. Theorem 1 already characterizes how separability governs the risk of mixing source data, independently of what causes it. In my view, it would be more appropriate to position separability as the central principle, while retaining discrete diffusion as the primary application setting.

Thank you for the follow-up. We agree with both suggestions.

W1. We keep the original Assumption 3 and present it as an idealization. We state that in practice, what the model learns from the source may carry over only in part, and that the gain is then smaller. We do not introduce the additional decomposition.

W2. We agree, and now position separability as the central principle, while retaining discrete diffusion as the primary application setting. Theorem 1 only needs that the domains are separable, independently of what causes it. We keep the comparison between masking and Gaussian noise as motivation, framed in terms of separability, and we explain why discrete data is the natural application: a related corpus is off topic but correct in its fine structure, so it helps at low noise. Figure 1(b) now points to a new appendix section, where we apply both noise processes to the same images (the results from our previous comment) and test Theorem 1 on a Gaussian diffusion model.

Thank you for the clarification and for addressing my concerns. I updated my overall rating.

Since the revisions are expected to involve substantial changes, I would like to make my final assessment based on the revised manuscript.

Reviewer_mtpWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training discrete diffusion language models when target data is scarce but a larger related corpus exists. Its starting observation is that, unlike Gaussian noise on images, masking or uniform corruption does not erase domain identity: surviving domain-specific tokens reveal where a sequence came from, so the continuous-diffusion practice of borrowing off-domain data at high noise is unsafe here. The authors propose RefineMix, a data-loader change that admits source data only below a noise cutoff and trains on target data alone above it, leaving loss, schedule, network and sampler unchanged. The analysis separates cost from gain. Assuming the domains share no sequence, the probability of generating source text equals a classifier overlap at the cutoff. Assuming further that both domains turn content into tokens the same way, the improvement over target-only training is the shared structure learned below the cutoff minus a bias paid at the cutoff, and the best cutoff grows as the target shrinks.

Critical strengths
  1. A real discrete-specific obstacle, clearly identified. A time-conditioned domain classifier stays far above chance almost until every token is corrupted, for every pair tested and under both kernels, so the continuous-diffusion recipe cannot simply be imported. The method that follows is a data-loader change leaving loss, schedule, network and sampler untouched, so it applies at any architecture or training stage.

  2. Analysis is simple, falsifiable, and actually tested. The first theorem turns the risk of off-domain generation into a classifier overlap at the cutoff, and an appendix checks the underlying identity on a trained, deliberately worst-case model, including why the estimated curve should be an upper bound rather than a match. The second theorem's prediction that the best cutoff rises as the target corpus shrinks is borne out by the target-size sweep under both kernels.

Critical weaknesses

Cutoff selected on the reported metric:

baselines get no matching search. It is found per setting by a grid sweep on target validation perplexity, the quantity the tables report; no held-out test split is described. Fine-tuning keeps one fixed learning rate and the source rate of data mixing is left untuned by choice, so part of the margin may be search budget. Test perplexity at the selected cutoff, plus a comparable search for fine-tuning, would settle this.

Questions for authors

Do the reported gains survive on a held-out test split, with the cutoff fixed beforehand by the validation sweep? If validation is the only split, how large is the sweep's selection effect?

1 comment

We thank the reviewer for the careful reading and agree with both points.

Baseline search. We now tune fine-tuning separately, sweeping six learning rates per setting (Appendix A.3, Table 7). Both methods then use the learning rate that is best for fine-tuning; RefineMix receives no learning-rate tuning of its own. We also sweep the source rate for data mixing (Appendix A.2, Table 6). With these tuned baselines, RefineMix still outperforms fine-tuning on all five pairs in Table 1.

Selection of the cutoff. We agree that this is a limitation. For the text experiments, validation is our only held-out split, so the reported perplexity is also the one used to select the cutoff. We do not believe this explains the gain, for two reasons. First, fine-tuning is now selected on the same split, among six learning rates and with early stopping, so it benefits from selection in the same way. Second, the on-target rate, the generation frontier (Appendix A.4), and all protein metrics (Table 3) are computed on generated samples, so none of them is the criterion used for selection.

Reviewer_QuzUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RefineMix, a framework for training discrete diffusion models under severe data scarcity. Particularly, the authors propose to mix in out-of-distribution data at selected diffusion times to facilitate the training of discrete diffusion models when the in-domain data is scarce. The authors also provide a framework for theoretically analyzing RefineMix. In the task of text source to target transformation, when applied to diffusion language models of different sizes, RefineMix outperforms baselines like finetuning and data mixing. RefineMix especially stands out in the domain of protein generation.

Critical strengths
  • The proposed algorithm is extremely simple in concept and easy to generalize across different diffusion language model types.
  • The authors provide error bars for all of their experiments.
Critical weaknesses
  • The writing is chaotic. First, a related works section is missing in the main paper. Second, the structure of this paper is extremely unbalanced. Section 3 is too verbose, and the experiment section only shows up at the end of Page 7. In Table 1 and Table 2, it is confusing what "Math", "Hep-Ex", and "Rare disease" etc. refer to since they are never mentioned in the main paper. In Table 2, the column LoRA is also confusing, because it is not comparable with other data mixing baselines, but rather in an orthogonal dimension. The conclusion part should not be placed within the section of experimental results, either. Overall, this paper needs a major edit pass, and the current writing clearly does not meet the standard of an ICLR paper.
  • Baselines are not properly tuned. In the abstract and introduction sections, the authors repeatedly state that finetuning needs careful tuning of the hyperparameters. However, in the authors' own experiments, they only use one set of fixed hyperparameters for this baseline. In Figure 8, the fact that finetuning demonstrates significant domain knowledge forgetting highly resembles an overly large learning rate. Given the marginal improvement of RefineMix, I seriously doubt this performance gain is based on poorly tuned baselines rather than real improvement.
  • The setting of the transformation experiments is dubious. Most checkpoints are trained on a large data corpus, and it is unclear if model backbones in Table 2 and the protein backbone in Table 3 were already trained on either the source data or the target data. If they have already been trained on domain data before, then the soundness of these sets of experiments very probably does not hold.
Questions for authors
  • Can the authors provide experimental results where the hyperparameters of the finetuning baselines are carefully tuned?
  • Can the authors provide experimental results for data mixing and RefineMix if lambda is tuned rather than fixed at an ad hoc value?
  • Can the authors elaborate if the source and target data have already been seen by the pretrained checkpoints, and how this fact will impact the soundness of the corresponding experiments?
  • Can the authors explain why they chose to write the paper in such a non-standard way?
1 comment

We thank the reviewer for the detailed and constructive review. We agree with the main concerns and have revised the paper accordingly.

Q1: Tuned fine-tuning baselines. We agree. We now sweep the learning rate for fine-tuning (six values per setting, 10−510^{-5} to 3×10−33\times10^{-3}; new Appendix A.3, Table 7). Both RefineMix and fine-tuning in Tables 1 and 2 now use the learning rate that is best for fine-tuning. RefineMix gets no tuning of its own. With these tuned baselines, RefineMix still beats fine-tuning on all five pairs of Table 1 (by 4–10% on four, about 1% on Patents-G →\to AG News). In Table 2, RefineMix improves by 2–4% on Rare disease and ChemRxiv for all four models; on Lean/mathlib the two methods match (−1.7%-1.7\% to +0.4%+0.4\%). The reviewer's suspicion was justified for SMDM, whose best rate (about 10−510^{-5}) is far below our original 3×10−43\times10^{-4}: tuning improved both methods by 3–7%, and the gap remained.

Q2: Tuning λ\lambda. We now sweep λ∈{0.1,0.5,0.7,0.9,0.95,0.99}\lambda \in \{0.1, 0.5, 0.7, 0.9, 0.95, 0.99\} for both RefineMix and data mixing (CS →\to Math, 1% target, trained from scratch; Appendix A.2, Table 6). RefineMix stays on target at every λ\lambda and is best at λ∈{0.9,0.95}\lambda \in \{0.9, 0.95\}, so our default of 0.90.9 is near-optimal. Data mixing reaches lower perplexity in this setting, but its on-target rate falls roughly as 1−λ1-\lambda (0.13 at λ=0.9\lambda = 0.9): most of its samples are source text, which perplexity on target text does not measure. We therefore consider this a failure mode. At the only λ\lambda where data mixing stays on target (0.10.1), its perplexity is 45.5, about twice that of RefineMix at λ=0.9\lambda = 0.9 (23.3). Data mixing therefore has no λ\lambda that is both on target and competitive.

Q3: Contamination. For the text experiments (Table 2), the source is a sample of each model's pretraining corpus by design, and the targets may have been seen during the original pretraining. All methods start from the same checkpoint, so such exposure affects them equally and cannot explain the differences between them. For proteins (Table 3), DPLM-650M is pretrained on UniRef50 and has likely seen peroxiredoxins; again, all methods share this checkpoint.

Q4: Writing. We thank the reviewer and have restructured the paper. Related Work is now a section in the main text, and the Conclusion is its own section, no longer part of the experiments. All datasets of Tables 1 and 2 (Math, Hep-Ex, Rare disease, etc.) are now named and cited in the main text (Section 5). We agree that the method section (now Section 4) was too long, and we have shortened it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.