- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper aims to improve the performance of discrete diffusion models under severe data scarcity by leveraging out-of-distribution data. The authors analyze why an existing method developed for image generation (continuous diffusion models) fails when applied to text generation (discrete diffusion models). Based on this analysis, they propose a simple, widely applicable framework and validate it empirically.
Critical strengths
- The paper provides an insightful analysis of why the existing method fails in the text generation.
- The proposed method is well-motivated by the findings from this failure analysis.
- RefineMix is easy to implement and widely applicable.
- Extensive experiments including protein sequence generation supports the effectiveness of RefineMix.
Critical weaknesses
- Validity of Assumption 3: When the shared-factor estimator has sufficient capacity and the sequence-content pairs are disjoint between the source and target domains, the shared factor could potentially learn the two domains independently. Consequently, it is unconvincing to assume that the training error of the shared factor on the target domain scales inversely with the total number of training samples, rather than specifically with the number of samples in the target domain.
- Justification for claims regarding discreteness: Discreteness does not appear to play a critical role in the theoretical analysis, despite RefineMix being presented specifically as a framework for discrete diffusion models. While the difference in between image and text generation is illustrated in Figure 1(b), could depend on several confounding factors, such as data modality, underlying training distributions, or the specific formulation of the diffusion process. It is not sufficiently justified that is predominantly governed by whether the variables are continuous or discrete.
Questions for authors
- If a low-noise region exists where the source and target domain distributions are disjoint, would you expect RefineMix to be effective for continuous diffusion models as well?
4 comments
Thank you for the detailed response.
Regarding W1.
I agree that knowledge transfer can occur even when the source and target content supports are disjoint, and the empirical results provide evidence. My main concern was not that knowledge transfer is impossible, but that Assumption 2 alone does not justify the equal contribution of source and target samples assumed in Assumption 3. I would find the original Assumption 3 reasonable if the authors more explicitly positioned it as an idealized setting in which knowledge transfer is particularly effective, and clarified the scope of the resulting theoretical analysis. This might also avoid introducing an additional decomposition of the estimation error, which can require further justification.
Regarding W2.
While the additional experiments provide useful evidence for the importance of the noise process, I am not sure whether these results alone justify replacing discreteness with the noise process as the central explanation. At the same time, I do not think it is necessary to identify which factors primarily determine separability. Theorem 1 already characterizes how separability governs the risk of mixing source data, independently of what causes it. In my view, it would be more appropriate to position separability as the central principle, while retaining discrete diffusion as the primary application setting.
Thank you for the follow-up. We agree with both suggestions.
W1. We keep the original Assumption 3 and present it as an idealization. We state that in practice, what the model learns from the source may carry over only in part, and that the gain is then smaller. We do not introduce the additional decomposition.
W2. We agree, and now position separability as the central principle, while retaining discrete diffusion as the primary application setting. Theorem 1 only needs that the domains are separable, independently of what causes it. We keep the comparison between masking and Gaussian noise as motivation, framed in terms of separability, and we explain why discrete data is the natural application: a related corpus is off topic but correct in its fine structure, so it helps at low noise. Figure 1(b) now points to a new appendix section, where we apply both noise processes to the same images (the results from our previous comment) and test Theorem 1 on a Gaussian diffusion model.
Thank you for the clarification and for addressing my concerns. I updated my overall rating.
Since the revisions are expected to involve substantial changes, I would like to make my final assessment based on the revised manuscript.
We thank the reviewer for the careful reading. Both concerns are fair.
W1 (Assumption 3). The reviewer is right. Assumption 2 only states that the true realization is shared across domains. It says nothing about whether a model actually learns it jointly. A model that fits the two domains with separate parameters would learn the shared factor from the NT target samples alone. Assumption 3 is therefore an assumption on the estimator rather than on the data, and we will state this explicitly in the revision.
A weaker version suffices. We can decompose the error of the shared factor into a component informed by both domains and a component informed only by the target, eN(t)=esh(t)/N+epr(t)/NT. Theorem 4 then holds with esh in place of e, since the private component is identical for both models and cancels in Δ. The reviewer's scenario corresponds to esh≡0, in which case the theorem gives Δ=−B(tmax)≤0, i.e., mixing never helps.
Empirically, esh is not zero. When we train from scratch, RefineMix lowers the held-out target perplexity, and it helps more when the target is smaller. With 1% of the target data, it lowers the perplexity by more than half (Figure 2(c)); both models train for the same 30k steps. When we raise λ from 0.1 to 0.95 below the cutoff, the perplexity falls from 46.7 to 22.1 (Table 6, seed 1). This is not just less overfitting. In Table 1, fine-tuning is stopped at its best checkpoint and uses its best learning rate, and RefineMix still lowers the target perplexity by 9–10% on the two arXiv pairs. We cannot explain this if nothing the model learns from the source carries over to the target.
On disjoint inputs. Held-out target sequences are themselves disjoint from the training sequences, so any 1/N rate on held-out data, including 1/NT, already relies on the network pooling information across disjoint inputs. How far this pooling extends across domains is an empirical question, but it is the same weight sharing that lets masked language models transfer between languages with no shared vocabulary (Conneau et al., ACL 2020).
W2 (Role of discreteness). We agree on both points. Our theorems do not use discreteness, and Figure 1(b) changes the data and the noise process at the same time.
Theory. The theorems only require that both domains share a Markov forward process that ends in a data-independent prior, and they reduce the question of where the source is safe to use to one quantity, the overlap C(t). The noise process enters only through this curve. Masking can only hide tokens, never change them, so the domains stay distinguishable as long as a few visible tokens identify the domain. Gaussian noise degrades every entry at once, which contracts the two noised distributions towards each other (Daras et al., 2025, Thm. 4.2).
Control experiment. To separate the noise process from the data, we corrupted the same images once by masking pixels and once with Gaussian noise (variance-preserving, αˉt=cos2(πt/2)), and trained the same domain classifier in each case (three seeds). We compare at the same fraction of surviving signal, 1−t under masking and αˉt under Gaussian noise. On CIFAR-10 vs. SVHN, balanced accuracy stays above 0.9 until only 2% of the signal survives under masking, whereas under Gaussian noise it falls below 0.9 already at 21%. On the closer pair CIFAR-10 vs. CIFAR-100 (clean accuracy 0.78 for masking, 0.81 for Gaussian), accuracy stays above 0.75 down to 20% under masking, compared with 65% under Gaussian noise.
Scope of our claim. The gap in Figure 1(b) is therefore not merely an artifact of comparing text with images: on the same data, masking keeps the domains separable considerably longer than Gaussian noise. We do not claim that the variable type itself is the cause, and we will state the claim consistently as one about the noise process and add the control experiment to the appendix.
Q (Continuous diffusion). Yes, and we tested this directly. We trained Gaussian diffusion models with RefineMix on CIFAR-10 (target) and SVHN (source, λ=0.9) at six cutoffs and measured the fraction of generated samples that belong to the target domain (one seed). At every cutoff, this fraction matches the prediction of Theorem 1, computed from the domain classifier, to within about 0.03: it is 0.94 at tmax=0.5, 0.61 at tmax=0.8, and 0.07 with full mixing.
This experiment tests where the source is safe to use, not whether it also improves the target there, which we leave to future work. The results suggest that Theorem 1 also describes Gaussian diffusion. The main difference is that the safe cutoff is lower, since Gaussian noise makes the domains indistinguishable earlier. We will add this experiment to the appendix.