acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Parallel Tempering for Test-Time Steering in Masked Diffusion Models

Abstract

Test-time steering aims to sample from reward-tilted distributions defined by a pre-trained model and an external reward function, without updating the model's parameters. However, samplers may struggle to move between high-reward regions separated by low-probability barriers, leaving important modes unexplored. Sequential Monte Carlo may concentrate particles around early-discovered modes, while replica exchange across diffusion noise levels may mix slowly when reward-agnostic exchange proposals are frequently rejected. We propose Temperature-Enhanced Masked Proposal and Exchange for Reward Steering (TEMPERS), a reward-centered parallel tempering framework for masked diffusion models that operates on complete sequences. Remask-and-denoise proposals enable reward-based acceptance at every local update, allowing each replica to refine candidates at its reward temperature before exchange. In an analytic example with a known reward-tilted distribution, TEMPERS closely approximates the target region probabilities. Experiments on protein and DNA sequence design and sentiment-controlled text generation further show higher rewards than the evaluated test-time steering baselines, alongside competitive output quality.

Primary area:
generative models
Keywords:
test-time guidance, masked diffusion models, parallel tempering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h7pzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temperature-Enhanced Masked Proposal and Exchange for Reward Steering (TEMPERS), which is a test-time steering method for diffusion models that uses replica exchange to avoid some of the drawbacks of other steering methods, such as reduced diversity. In contrast to prior work, which uses replicates for different diffusion noise levels, they maintain replicates at different reward temperatures. They generate candidates by remasking sampled complete sequences and pair larger remasking fractions with higher reward temperatures and smaller remasking fractions with lower temperatures, with the intuition that the former encourages broader exploration while the latter encourages local refinement. They demonstrate the effectiveness of this approach first in a synthetic setting, where they show that TEMPERS achieves lower KL to a known reward-tilted target than other methods. They then show that it achieves higher rewards on DNA generation, protein generation, and sentiment-controlled text generation. They further show increased diversity in protein generation as well as the ability to maintain instruction following capability during sentiment-controlled text generation.

Critical strengths

This paper has a few strengths.

First, the algorithm design is clear and fairly intuitive. Additionally, ensuring that each state is a complete sequence is useful for reward evaluation.

Second, the synthetic experiment is clean and effective at demonstrating the method's purpose and performance.

Third, the authors are able to show improvements on the reward for which they optimize as well as some external metrics in all settings. The results on DNA in particular are strong and it is notable that the authors take care to separate training and evaluation rewards using models trained on disjoint subsets.

Critical weaknesses

There are a few different weaknesses that the authors should address.

First, the generation of complete sequences seems to be computationally expensive. The authors provide some experiments at matched NFE, which is good, but I would like to see comparisons against baselines at fixed computational budget for the different settings, as well as performance against wall-clock time.

Second, there are a few issues with the evaluation measures. For the sentiment analysis experiments, the authors should additionally report scores from an external classifier not used as the reward model. Additionally, Hamming distance seems like an important but potentially weak measure for diversity. Additional measures, such as those considered in [1] should also be reported, e.g. motif-based diversity, cluster-based diversity, and sequence PPL. It may also be interesting to report Fréchet Biological Distance [2].

Third, there is a slight mismatch with the theory section when sampling all tokens at once (to the author's credit, this is mentioned directly). I think the authors should state directly that the exactness guarantees do not transfer directly to the approximation they use. If possible, they could provide some theoretical characterization of how the stationary distribution under the approximation differs from that of the true conditional, but absent that an empirical analysis of the gap could be sufficient. In particular, it would be interesting to see if the approach they use in practice preserves the prior when you set β\beta to 0.

Finally, it seems concerning that while the model is able to optimize the exact reward function, it seems to suffer on other measures. For example, in proteins we see worse results on scRMSD and sentiment seems to suffer on perplexity.

[1] Lal, A., Gunsalus, L., Gupta, A. et al. Polygraph: a software framework for the systematic assessment of synthetic regulatory DNA elements. Genome Biol 26, 114 (2025). https://doi.org/10.1186/s13059-025-03584-9

[2] Stark, Hannes, et al. "Dirichlet flow matching with applications to dna sequence design." arXiv preprint arXiv:2402.05841 (2024).

Questions for authors

Can I see the within-region distributions for the synthetic experiment? It is possible to match the region histogram but be distorted within each region.

Can you correct the link for DDG? It gives a 404 error when clicked.

For the sentiment experiments, can you also report results from the baseline model using temperature sampling, not only greedy decoding?

Can you provide more details on hyperparameter tuning? At times you mention an "untuned uniform configuration" and there is not too much information on the tuning procedure for the method and baselines.

6 comments
AuthorsOpenReview ↗

W1. Add more evaluation metrics for the computational efficiency

W1. First, the generation of complete sequences seems to be computationally expensive. The authors provide some experiments at matched NFE, which is good, but I would like to see comparisons against baselines at fixed computational budget for the different settings, as well as performance against wall-clock time.

We report both budgets on the biological tasks. Under matched denoising NFE, TEMPERS is faster than RERD and CREPE on DNA and attains the highest Success Rate on protein at a sampling time below CREPE's. Under shared wall-clock limits it has the highest mean Success Rate at every limit on DNA, and on protein it is ahead at 25 and 40 seconds and level at 50 seconds. All denoising calls are counted, including initialization and every replica, and the measured times include the guidance oracle.

Matched denoising NFE. 4,610 denoising NFE per output on DNA and 1,000 on protein, as in the updated Tables 1 and 2 of our response to Reviewer x4X1 W1/Q1.

DNA sampling time at 4,610 denoising NFE:

MethodSampling time (s/output) ↓
Best-of-K (K=10)0.402
FK-Steering (K=10)0.950
CREPE0.802
RERD (β=5)1.220
TEMPERS0.592

Protein sampling time at 1,000 denoising NFE:

MethodSampling time (s/output) ↓
Best-of-K (K=10)0.663
FK-Steering (K=10)1.317
CREPE1.022
RERD (β=5)1.276
TEMPERS1.016

TEMPERS is faster than RERD, CREPE, and FK-Steering. While Best-of-K is the fastest method, its Success Rate is worse than existing test-time steering methods in Table 1 and Table 2.

Shared wall-clock limits. Every method was also run under a common per-batch time limit on the same Ada6000 GPU, with three seeds throughout: 60, 90 and 120 seconds per batch of 64 on DNA, and 25, 40 and 50 seconds per batch of 16 on protein.

DNA Success Rate (Activity ≥ 5 and a 3-mer Pearson Corr ≥ 0.3) at each limit:

Method60 s90 s120 s
Pretrained0.00 ± 0.000.00 ± 0.000.00 ± 0.00
Best-of-K1.04 ± 0.371.04 ± 0.371.04 ± 0.37
FK-Steering47.66 ± 1.1047.66 ± 1.1047.66 ± 1.10
CREPE45.05 ± 2.2451.56 ± 6.4753.39 ± 4.05
RERD (β=3)6.25 ± 4.6017.19 ± 14.036.25 ± 6.28
RERD (β=5)17.45 ± 24.685.21 ± 7.370.52 ± 0.74
TEMPERS57.03 ± 8.9356.51 ± 7.9859.64 ± 8.08

TEMPERS attains the highest mean Success Rate, exceeding CREPE by 11.98, 4.95 and 6.25 points. RERD reaches higher raw Activity, as in Table 1, but fewer of its outputs clear both thresholds.

Protein Success Rate at each limit:

Method25 s40 s50 s
Pretrained33.5 ± 1.733.5 ± 1.733.5 ± 1.7
Best-of-K (K=10)54.2 ± 1.754.2 ± 1.754.2 ± 1.7
FK-Steering (K=10)—†60.42%60.42%
CREPE (K=10)64.7 ± 1.567.4 ± 1.170.9 ± 1.4
RERD (β=5)63.1 ± 4.762.8 ± 2.966.8 ± 1.6
TEMPERS (K=10)66.1 ± 2.071.6 ± 2.270.8 ± 1.0

† FK-Steering did not complete all prescribed outputs within the time limit for 3 of the 12 backbones at 25 and 40 seconds, and 2 of the 12 at 50 seconds, where "—" indicates incomplete output coverage within the budget.

Among methods with complete results for all 12 backbones, TEMPERS has the highest mean Success Rate at 25 and 40 seconds. At 50 seconds, CREPE and TEMPERS are close, with CREPE slightly higher (70.9% vs. 70.8%).

AuthorsOpenReview ↗

W2. Add more evaluation metrics for the generalization and the diversity

W2. Second, there are a few issues with the evaluation measures. For the sentiment analysis experiments, the authors should additionally report scores from an external classifier not used as the reward model. Additionally, Hamming distance seems like an important but potentially weak measure for diversity. Additional measures, such as those considered in [1] should also be reported, e.g., motif-based diversity, cluster-based diversity, and sequence PPL. It may also be interesting to report Fréchet Biological Distance [2].

We report an external sentiment classifier for text generation, cluster-based diversity for DNA and protein design, and motif-based diversity and FBD for DNA design.

External sentiment classifier. We rescored the existing outputs with SiEBERT based on DistilBERT's steering outputs.

MethodSiEBERT ↑
Pretrained (t=0.7)0.533 ± 0.065
Best-of-K (K=10)0.633 ± 0.063
SMC0.583 ± 0.064
CREPE0.650 ± 0.062
TEMPERS0.667 ± 0.061

Cluster-based diversity. Reported in our response to Reviewer x4X1 (W2/Q2), where we cluster the valid high-reward outputs on both tasks and report the effective clusters exp(H).

Motif-based diversity. On DNA, we construct binary motif-presence vectors using JASPAR/FIMO at p ≤ 10⁻³, over the same qualified outputs as the clustering analysis.

MethodEffective Motif Combinations ↑Mean Jaccard Distance ↑
Best-of-K (K=10)4.33 ± 1.250.571 ± 0.047
FK-Steering (K=10)32.00 ± 0.000.618 ± 0.002
RERD (β=3)20.58 ± 2.290.496 ± 0.046
RERD (β=5)16.96 ± 0.280.418 ± 0.043
CREPE32.00 ± 0.000.641 ± 0.011
TEMPERS32.00 ± 0.000.641 ± 0.014

Fréchet Biological Distance. On DNA, we compute FBD using Gosai/oracle_eval features against a high-HepG2-activity reference distribution and the same 736 reference sequences and 6,144-dimensional features for every method.

MethodFBD (HepG2 Reference) ↓
RERD (β = 3)11049.89 ± 3348.00
RERD (β = 5)11568.47 ± 2047.25
CREPE2160.37 ± 230.16
TEMPERS1541.35 ± 99.60

These results support its ability to generate valid high-reward candidates across multiple sequence clusters.

AuthorsOpenReview ↗

W3. Theoretical analysis and empirical implementation

W3. Third, there is a slight mismatch with the theory section when sampling all tokens at once (to the author's credit, this is mentioned directly). I think the authors should state directly that the exactness guarantees do not transfer directly to the approximation they use. If possible, they could provide some theoretical characterization of how the stationary distribution under the approximation differs from that of the true conditional, but absent that an empirical analysis of the gap could be sufficient.

The exactness in Proposition 1 does not transfer to the one-step parallel refill used in our experiments, and the remark following Proposition 1 now states this directly. We keep the one-step configuration because its lower proposal cost permits more MCMC sweeps under a fixed denoising budget.

Specifically, Proposition 1 assumes a mask of fixed size chosen uniformly and refilling from the exact joint conditional under the same prior pθp_\theta used in the target. This can be implemented by unmasking the tokens sequentially, each sampled from its exact conditional given the unmasked context and the tokens already regenerated. Our configuration uses uniform remasking, so the first assumption holds. The second is not guaranteed by our practical configuration: the main experiments unmask all masked tokens in a single denoising pass, sampling tokens independently from the model-predicted distributions and ignoring the dependence among the remasked positions. When a single token is remasked, parallel and sequential refilling have no factorization discrepancy. Exactness still requires the refill distribution to match the conditional of the same prior pθp_\theta used in the target, including any effect of the sampling temperature.

We cannot carry this through to a closed form for the stationary distribution, so we take the empirical route the reviewer suggests, on the analytic example where the exact target is available.

In particular, it would be interesting to see if the approach they use in practice preserves the prior when you set β\beta to 0.

Preserving the prior at β=0\beta = 0. At β=0\beta = 0 every proposal is accepted, so any movement of the output distribution is attributable to the proposal alone. We initialize from the pretrained model's generative distribution and measure region TV against independent reference samples from the same model.

Proposal Denoising StepsRegion TV Before Updates ↓Region TV After 60 Updates ↓
10.0160.061 ± 0.015
20.0160.061 ± 0.011
40.0160.047 ± 0.009
80.0160.024 ± 0.010

The prior is not preserved exactly. The 0.016 before any sweep is the resolution of this measurement, being the distance between two independent draws from the same model, and it is identical across the four rows because the initialization does not depend on the proposal. After 60 sweeps the one-step sampler sits at about four times that resolution, and one and two steps are indistinguishable at this seed spread. Eight steps brings the deviation to 0.024, within one and a half times the resolution, so it is largely removed by moving the proposal toward the sequential procedure that Proposition 1 assumes.

Agreement with the reward-tilted target. We also measure region TV after 60 sweeps against the exact regional probabilities under p~β(x)∝p0(x)exp⁡(βR(x))\tilde{p}_\beta(x)\propto p_0(x)\exp(\beta R(x)).

β1 step2 steps4 steps8 steps
00.059 ± 0.0150.053 ± 0.0100.040 ± 0.0080.021 ± 0.006
0.50.083 ± 0.0060.077 ± 0.0090.054 ± 0.0070.031 ± 0.012
10.143 ± 0.0080.112 ± 0.0040.082 ± 0.0130.063 ± 0.006
20.304 ± 0.0160.269 ± 0.0050.221 ± 0.0130.190 ± 0.012
50.367 ± 0.0140.347 ± 0.0060.299 ± 0.0140.258 ± 0.009
Denoising NFE4007001,3002,500

At β=0\beta = 0 the target coincides with the prior and the first row agrees with the measurement above, which also shows that the trained model and the analytic prior differ by less than the resolution of either measurement. Four and eight steps reduce the deviation at every β\beta, at a denoising cost that the last row makes explicit.

Reviewer_nJCWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Temperature-Enhanced Masked Proposal and Exchange for Reward Steering (TEMPERS). In a nutshell, the paper runs PT for targets at different reward strength (instead of CREPE which runs at different reward strength + different masking propotions). Experiments show that TEMPERS yields a more accurate and effective control on protein and DNA sequence design, as well as text sentiment control.

Critical strengths
  1. The paper proposes an intereting and potentially promising axis for test time control: instead of controling along the diffuson time, the paper control across a new axis for purely reward strength.

  2. The paper experiments are comprehensive, it includes analytical experiments, protein, language, DNA, showing the effective of the proposed method.

Critical weaknesses
  1. The novelty appears limited. [1] proposed a similar algorithm in the continuous-time setting, whereas this paper focuses primarily on discrete masked diffusion. However, none of the proposed algorithmic components appears specific to masking. This raises concerns: (i) it is unclear why the paper restricts its scope to masked diffusion; (ii) the paper does not clearly identify the conceptual or technical challenges overcome when extending the existing framework to this setting, making the contribution appear primarily to be a change in application domain. Also, this might lead to risks of “salami slicing, even I believe this is not authors’ intention. Therefore i really encourage a clear discussion on the scope.

  2. Another key issue is that the paper does not adequately explain or investigate why its approach should be more effective or accurate than CREPE or related SMC methods. These methods guide sampling across diffusion steps, whereas the proposed approach constructs an annealing path by varying the reward strength. Moreover, the local exploration proposals do not appear to use reward guidance. The central question is therefore: why might annealing along the reward-strength axis be preferable to annealing along the diffusion trajectory? The paper would benefit from a systematic investigation of when and why this choice helps, what trade-offs it introduces, and how guidance in the local proposals affects sampling efficiency. Such an investigation could yield more substantive insights than introducing another method and reporting performance improvements alone.

[1] Test-Time Guidance for Flow-Based Generative Models via Parallel Tempering on Source Distributions. ICML 2026.

Questions for authors

See my weakness:

  1. Why this work only consider mask diffusion?

2 a. The control did not follow any guidance, why the local exploration is effective?

2 b. Why this new annealing path is more effective than path along diffusion path?

4 comments
AuthorsOpenReview ↗

W1/Q1. Clarify the novelty and contribution of TEMPERS for Masked Diffusion Models

W1. The novelty appears limited. [1] proposed a similar algorithm in the continuous-time setting, whereas this paper focuses primarily on discrete masked diffusion. However, none of the proposed algorithmic components appears specific to masking This raises concerns: (i) it is unclear why the paper restricts its scope to masked diffusion; (ii) the paper does not clearly identify the conceptual or technical challenges overcome when extending the existing framework to this setting, making the contribution appear primarily to be a change in application domain. [1] Test-Time Guidance for Flow-Based Generative Models via Parallel Tempering on Source Distributions. ICML 2026. Q1. Why this work only consider mask diffusion?

Masked Diffusion Models (MDMs) have been widely used for discrete sequence generation on several domains based on an absorbing corruption process in which tokens are iteratively replaced by the mask token. However, we recognize that we cannot directly apply Parallel Tempering (PT) to MDMs, following ideas from SPT [1] or PATHS [2]. Specifically, PT requires carefully designing the proposal method to obtain the samples satisfying Metropolis–Hastings (MH) criteria. In SPT/PATHS, they rely on the Gaussian distribution N(0,I)N(0, I) as the source distribution to utilize pCN/pCNL proposals, which is not suitable for standard absorbing MDMs with a fully masked sequence. SPT also notes that moving beyond a Gaussian source requires designing new proposal kernels to maintain reasonable acceptance rates, and leaves this as the future work.

Since the fully masked endpoint offers no source randomness to perturb, we design a new PT algorithm for MDMs. We build the proposal on complete sequences using remask-and-denoise with the pretrained model (Eq. 11). Every state is then a complete sequence whose reward can be evaluated directly. Without a continuous step-size parameter such as pCN's/pCNL's ρ\rho, we use the masking fraction as the proposal scale and bind it to a replica's temperature (Eq. 14). To target the reward-tilted distribution, we adopt a reward-only acceptance criterion (Eq. 16) together with replica exchange across temperatures (Eq. 20), and Proposition 1 gives the condition under which this acceptance is exact.

Redesigning PT for MDMs is therefore not a change of application domain. The proposal mechanism, the step-size control, and the acceptance rule all had to be reconstructed, and the construction relies on conditionals over arbitrary masked subsets that masked diffusion provides natively. We disagree with the characterization of this work just change in application domain and firmly stand our ground that this redesign is sufficiently novel, advancing parallel tempering as a sampler for test-time steering of diffusion models.

[2] Parallel Tempering Initial Sampling in Inference-Time Reward Alignment, ICML 2026 SPIGM Workshop

AuthorsOpenReview ↗

W2/Q2. Clarify the differences from existing works and our investigation the effectiveness of TEMPERS

In a nutshell, the paper runs PT for targets at different reward strength (instead of CREPE which runs at different reward strength + different masking propotions).

We want to clarify that both methods target the same reward-tilted distribution in Eq. (1) but construct different annealing paths toward it. TEMPERS anneals along the reward-strength axis: replicas differ in the reward inverse temperature. Every replica state is a complete sequence, and the masking fraction is the proposal scale tied to that temperature (Eq. 14). CREPE anneals along the diffusion trajectory: replicas sit at different noise levels (masking proportions), and proposals are generated by forward noising and backward denoising. TEMPERS therefore also varies masking fractions across replicas. The difference is that we design the masking fraction is a property of the proposal, while every replica state remains complete.

W2. Another key issue is that the paper does not adequately explain or investigate why its approach should be more effective or accurate than CREPE or related SMC methods. These methods guide sampling across diffusion steps, whereas the proposed approach constructs an annealing path by varying the reward strength. Moreover, the local exploration proposals do not appear to use reward guidance. The central question is therefore: why might annealing along the reward-strength axis be preferable to annealing along the diffusion trajectory? The paper would benefit from a systematic investigation of when and why this choice helps, what trade-offs it introduces, and how guidance in the local proposals affects sampling efficiency. Such an investigation could yield more substantive insights than introducing another method and reporting performance improvements alone.

Figure 1 compares each method's output region masses with the exact reward-tilted target. FK/SMC falls into the local-mode trap: 67.7% of its mass sits in B4B_4 against the target's 45.0%, and it recovers only 17.9% of the 45.0% the target places in B1B_1, the low-prior high-reward region. CREPE, annealing along the diffusion trajectory, holds 22.5% in B3B_3 against the target's 8.0% and reaches 19.5% in B1B_1, where TEMPERS reaches 28.3% and is closest to the target overall (region KL 0.07 against 0.20 for both).

Q2 b. Why this new annealing path is more effective than path along diffusion path?

Figure 2 shows our investigation comparing the intermediate updates of TEMPERS and CREPE, and shows that CREPE's exchange proposals carry lower reward because they are generated without directly accounting for the rewards of the intermediate noisy level (incomplete sequences). In contrast, TEMPERS iteratively revises complete sequences and maintains the high-reward samples with a high acceptance rate, so it reaches high-reward samples in fewer MCMC sweeps.

Q2 a. The control did not follow any guidance, why the local exploration is effective?

The local updates do use the reward. As noted in our response to W1, every replica state is a complete sequence, so the reward is evaluated for every proposal and enters the acceptance test in Eq. (16). For βk>0\beta_k>0, reward-decreasing proposals are accepted with probability exp⁡(βkΔRk)<1\exp(\beta_k\Delta R_k)<1, while reward-increasing proposals are always accepted. Thus, reward guidance enters every within-chain acceptance step, favoring higher-reward proposals.

I appreciate the clarification, but my main concern remains: none of the proposed algorithmic components appears specific to masked diffusion. PT consists of two components: local exploration and replica exchange. Both can be directly implemented for other diffusion models, including Gaussian diffusion:

  1. Local exploration. The remask-and-denoise proposal in Eq. (11) can be directly replaced by Gaussian noising followed by denoising. The reward-only acceptance in Eq. (16) applies equally to Gaussian diffusion with the same approx made in this paper. The masking fraction can simply be replaced by the Gaussian noise level.

  2. Replica exchange. The acceptance rule in Eq. (20) depends only on reward differences and inverse temperatures. It is the standard PT swap rule and requires no modification for other diffusion models.

Therefore, I remain unconvinced that the proposed method requires any algorithmic design specific to masked diffusion. The rebuttal still does not explain why the proposed framework is specific to MDMs or what new conceptual challenge has been overcome.

My suggestion and point is not that this was exactly some previous method. Instead, my point is the entire method can be seamlessly applied to any diffusion. Hyperparameters might vary but the overall method, alg, approx being taken are all generic.

AuthorsOpenReview ↗

TEMPERS for Masked Diffusion and Its Possible Extension to Gaussian Diffusion

We agree that TEMPERS follows classical parallel tempering (PT), rooted in the replica Monte Carlo work of Swendsen and Wang [1]. PT combines local exploration within each chain with replica exchange across temperatures [2]. For test-time reward steering, this framework leaves open where in the diffusion process to construct an effective local proposal and how to set its scale. Our proposal for masked diffusion models (MDMs) applies the remask-and-denoise operation to a complete sequence (Eq. 11) and ties the remasking fraction to each chain’s reward temperature (Eq. 14). As the reviewer comments, a continuous diffusion model with a Gaussian source prior could make an analogous local move by adding noise to the chain's current sample and denoising it. However, we disagree that this possibility diminishes the value of the proposal we develop and validate for MDMs, which are widely used for text and biological sequence generation.

Diffusion modelPrior sourceForward & backward processTermination (complete) state
Continuous diffusion modelsSPT, PATHSCREPEPossible TEMPERS extension via Gaussian perturbation and denoising of a complete sample
Discrete masked diffusion models—CREPETEMPERS (this work)

SPT and PATHS perturb a Gaussian draw at the source. A fixed-length absorbing MDM starts from a single fully masked sequence, so it has no analogous source variable to perturb. CREPE applies to both model families by using forward and backward diffusion transitions to propose exchanges between noise levels. Our concern for CREPE was that the proposal states with a noisier level would deliver a low-reward candidate to the output chain. In both our analytic and protein experiments, the complete candidates offered by CREPE had low median reward measured before exchange acceptance (Fig. 2, panel C). To improve the candidates available for exchange, we designed TEMPERS so that each chain refines a complete sequence at its own reward temperature.

TEMPERS's single unmasking step per proposal worked well in our MDM benchmarks, as illustrated by the inverse protein folding ablation (Appendix C.5, Table 7). For continuous diffusion, we expect an effective one-step proposal to require more careful denoiser design, especially after a large Gaussian perturbation, which is a well-known challenge solved by existing works Consistency Trajectory Models and PaGoDA (Kim, Lai et al., ICLR 2024, Kim, Lai et al., NeurIPS 2024). How well such a denoiser would work as a PT proposal remains an open empirical question.

Classical PT specifies how replicas exchange states [2] and leaves the choice of within-chain proposal open. Our methodological contribution is to design and validate a proposal for absorbing MDMs that ties the fraction of tokens remasked in a complete sequence to each chain's reward temperature. Each chain can then use the change in reward to accept or reject the completed edit before exchange.

[1] Robert H. Swendsen and Jian-Sheng Wang (1986). Replica Monte Carlo Simulation of Spin-Glasses.

[2] Charles J. Geyer (1991). Markov Chain Monte Carlo Maximum Likelihood, Eq. 7.

Reviewer_x4X1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMPERS, a test-time steering method for masked diffusion models based on parallel tempering over complete sequences. Each chain remasks part of the current sequence, refills it with the pretrained model, and accepts the proposal using only the reward difference. Chains at different reward temperatures use different remasking fractions and exchange states periodically. Experiments on an analytic example, protein and DNA sequence design, and sentiment-controlled text generation show higher rewards than FK steering, CREPE, and Best-of-K.

Critical strengths
  • The method is simple and cheap to run. Both the within-chain acceptance and the exchange step depend only on reward differences, and Proposition 1 gives a clear condition under which the reward-only acceptance is exact.
  • The analytic example compares the output distribution with an exact target. The protein and DNA experiments use held-out oracles for evaluation, and the gains are consistent across three domains.
Critical weaknesses
  • Baseline missing. TEMPERS repeatedly remasks part of a complete sequence, refills it, and keeps or rejects the result based on reward. Reward-guided iterative refinement [1] follows the same overall loop of partial noising and denoising on complete sequences, and it is evaluated on the same DRAKES protein and DNA tasks. The paper mentions [1] only briefly in Appendix B. Could the authors add it as a baseline under the same number of NFE?

  • Evidence for mode coverage on real tasks. The paper’s main motivation is to sample from the reward-tilted distribution and cover multiple high-reward modes. This is shown directly only in the analytic example. On the real tasks, the evidence is limited to pairwise Hamming diversity. In addition, masked tokens are refilled in one parallel step, refills use a sampling temperature (0.3 or 0.7), and β reaches 3333 in protein design, where the target is close to an argmax. Appendix C.5 also shows that refills closer to the assumption do not improve reward. On the protein task, TEMPERS has higher diversity than the pretrained model (29.5 vs. 24.1), which seems unexpected under such a sharp target. Could the authors provide more direct evidence of mode coverage on the real tasks, e.g., by clustering the high-reward outputs?

[1] https://arxiv.org/abs/2502.14944

Questions for authors
  • How does TEMPERS compare with reward-guided iterative refinement (Uehara. et al.) on the protein and DNA tasks under the same number of NFE?
  • What distribution does the practical sampler target, given one-step refills, sampling temperature, and large β\beta? Can the authors provide more direct evidence of mode coverage on the real tasks?
4 comments
AuthorsOpenReview ↗

W1/Q1. Add RERD[1] and update the main Table 1 & Table 2

W1. Baseline missing. TEMPERS repeatedly remasks part of a complete sequence, refills it, and keeps or rejects the result based on reward. Reward-guided iterative refinement [1] follows the same overall loop of partial noising and denoising on complete sequences, and it is evaluated on the same DRAKES protein and DNA tasks. The paper mentions [1] only briefly in Appendix B. Could the authors add it as a baseline under the same number of NFE? Q1. How does TEMPERS compare with reward-guided iterative refinement (Uehara. et al.) on the protein and DNA tasks under the same number of NFE?

We evaluate RERD [1] using its public implementation, with the same pretrained and reward models as TEMPERS and CREPE. These three methods use 1,000 denoising NFE per output for protein and 4,610 for DNA, including initialization and all replicas. At matched denoising NFE, TEMPERS achieves a higher mean success rate than RERD on both protein and DNA design under the joint reward and validity criteria defined in W2/Q2.

Updated Table 1: DNA sequence design. Success Rate requires Activity ≥ 5 and per-sequence 3-mer Pearson correlation ≥ 0.3.

MethodActivity ↑ATAC-Acc (%) ↑3-mer Corr ↑Success Rate (%) ↑Diversity (%) ↑Unique Sequences (%) ↑
Pretrained0.181 ± 0.0111.8 ± 0.4−0.058 ± 0.0260.05 ± 0.0774.65 ± 0.06100.00 ± 0.00
Best-of-K (K=10)1.787 ± 0.0777.6 ± 1.00.530 ± 0.0330.68 ± 0.1974.82 ± 0.01100.00 ± 0.00
FK-Steering (K=10)5.047 ± 0.01235.7 ± 1.60.838 ± 0.00548.59 ± 0.8973.52 ± 0.04100.00 ± 0.00
CREPE (K=10)5.060 ± 0.07945.57 ± 1.330.833 ± 0.02840.89 ± 4.1074.31 ± 0.12100.00 ± 0.00
RERD (β=3)9.305 ± 0.53498.70 ± 1.850.202 ± 0.41847.92 ± 40.9348.32 ± 2.1148.44 ± 0.64
RERD (β=5)9.470 ± 0.08398.44 ± 1.690.393 ± 0.19648.44 ± 1.2846.61 ± 3.1941.67 ± 0.97
TEMPERS (K=10)5.506 ± 0.02763.02 ± 1.850.915 ± 0.02457.03 ± 5.4574.01 ± 0.16100.00 ± 0.00

Updated Table 2: Protein sequence design. Success Rate requires DDG >0> 0 and scRMSD <2< 2 Å.

MethodDDG ↑DDG > 0 (%) ↑scRMSD ↓scRMSD < 2 (%) ↑Success Rate (%) ↑Diversity (%) ↑Unique Sequences (%) ↑
Pretrained−0.537 ± 0.05835.5 ± 0.70.837 ± 0.00591.0 ± 0.533.29 ± 0.4424.1 ± 0.199.76 ± 0.11
Best-of-K (K=10)0.348 ± 0.01159.0 ± 0.40.869 ± 0.00691.7 ± 0.454.1 ± 0.423.8 ± 0.199.80 ± 0.05
FK-Steering (K=10)0.704 ± 0.02169.6 ± 0.50.857 ± 0.01193.9 ± 0.464.5 ± 0.421.4 ± 0.198.74 ± 0.19
CREPE (K=10)0.811 ± 0.02671.8 ± 0.40.871 ± 0.02492.3 ± 2.065.28 ± 1.3031.1 ± 0.3100.00 ± 0.00
RERD (β=5)0.897 ± 0.12972.0 ± 2.40.854 ± 0.02188.9 ± 3.465.02 ± 5.9924.4 ± 0.661.11 ± 1.57
TEMPERS (K=10)1.033 ± 0.04277.3 ± 0.70.861 ± 0.02592.2 ± 0.270.92 ± 0.6129.3 ± 0.6100.00 ± 0.00

RERD reaches higher raw Activity and ATAC-Acc on DNA, but fewer of its outputs clear both thresholds and fewer distinct sequences.

Together with the cluster analysis in W2/Q2, these results support TEMPERS's ability to generate diverse, high-reward candidates within a fixed denoising budget.

AuthorsOpenReview ↗

W2/Q2. Evaluate mode coverage on real tasks with more metrics

(1/3) Mode coverage on the real tasks

W2. Evidence for mode coverage on real tasks. The paper's main motivation is to sample from the reward-tilted distribution and cover multiple high-reward modes. This is shown directly only in the analytic example. On the real tasks, the evidence is limited to pairwise Hamming diversity. Could the authors provide more direct evidence of mode coverage on the real tasks, e.g., by clustering the high-reward outputs? Q2. Can the authors provide more direct evidence of mode coverage on the real tasks?

We clustered the valid high-reward outputs on both tasks. TEMPERS's valid high-reward outputs span multiple clusters on both biological tasks. Effective clusters exp⁡(H)\exp(H) uses the entropy HH of cluster proportions to account for uneven cluster sizes pjp_j. Cluster statistics use only successful outputs.

In DNA design, success requires Activity ≥ 5 and per-sequence 3-mer Pearson correlation ≥ 0.3. We cluster successful sequences using motif presence, Jaccard distance, and complete linkage with a distance cutoff of 0.5.

MethodSuccess Rate (%) ↑Unique Sequences (%) ↑Clusters ↑Effective Clusters exp(H) ↑
Pretrained†0.05 ± 0.07100.00 ± 0.00——
Best-of-K (K=10)†0.68 ± 0.19100.00 ± 0.00——
FK-Steering (K=10)48.59 ± 0.89100.00 ± 0.0028.16 ± 0.3026.59 ± 0.46
CREPE (K=10)40.89 ± 4.10100.00 ± 0.0029.38 ± 1.2728.42 ± 1.73
RERD (β=5)48.44 ± 1.2841.67 ± 0.974.42 ± 1.303.13 ± 0.81
TEMPERS (K=10)57.03 ± 5.45100.00 ± 0.0029.21 ± 1.0728.00 ± 1.63

Note. Success Rate and Unique Sequences use all outputs † Pretrained and Best-of-K have fewer than 32 successful outputs in each seed.

In protein design, success requires DDG > 0 and scRMSD < 2 Å. We build shared complete-linkage clusters at 80% sequence identity from successful outputs, separately within each backbone.

MethodSuccess Rate (%) ↑Unique Sequences (%) ↑Clusters ↑Effective Clusters exp(H) ↑
Pretrained33.29 ± 0.4499.76 ± 0.117.40 ± 0.027.25 ± 0.02
Best-of-K (K=10)54.14 ± 0.3899.80 ± 0.057.53 ± 0.037.42 ± 0.04
FK-Steering (K=10)64.52 ± 0.4298.74 ± 0.197.48 ± 0.027.36 ± 0.02
CREPE (K=10)65.28 ± 1.30100.00 ± 0.007.80 ± 0.017.74 ± 0.01
RERD (β=5)65.02 ± 5.9961.11 ± 1.574.30 ± 0.233.83 ± 0.20
TEMPERS (K=10)70.92 ± 0.61100.00 ± 0.007.55 ± 0.077.43 ± 0.09

Success Rate and Unique Sequences use all outputs across all 12 backbones.

TEMPERS combines the highest Success Rate on both tasks with effective cluster counts close to those of CREPE and FK-Steering, and substantially higher than RERD (β=5). These results support its ability to generate valid high-reward candidates across multiple sequence clusters.

(2/3) Diversity under a sharp target

W2. ... β reaches 3333 in protein design, where the target is close to an argmax. … On the protein task, TEMPERS has higher diversity than the pretrained model (29.5 vs. 24.1), which seems unexpected under such a sharp target.

Increasing β\beta favors higher-reward sequences, but these sequences can lie in several separated regions, so the outcome depends on the prior p(x)p(x) and the landscape of the reward model R(x)R(x) rather than on β\beta alone. Reward reweighting can therefore preserve multiple modes and can even raise diversity by promoting high-reward regions that were less probable under the pretrained model.

For protein design, we use β≈3333\beta\approx3333, the reciprocal of the KL regularization coefficient 0.0003 used by DRAKES. TEMPERS, CREPE, and FK-Steering all achieve high predicted DDG rewards under this setting in our experiments.

For a concrete example, take

\qquad R(x)=\mathbf 1\{x\in\{-3,0,3\}\}.$$ The prior has a single mode at $0$. Under $\tilde p_\beta(x)\propto p(x)\exp(\beta R(x))$, $$\frac{\tilde p_\beta(3)}{\tilde p_\beta(2)}=e^{\beta-5/18}, \qquad \frac{\tilde p_\beta(3)}{\tilde p_\beta(4)}=e^{\beta+7/18}, \qquad \frac{\tilde p_\beta(\pm3)}{\tilde p_\beta(0)}=e^{-1/2}\approx0.61,$$ so for every $\beta>5/18$ there are three strict local modes at $-3,0,3$, and the two added modes keep relative probability $0.61$ for all $\beta$. As $\beta\to\infty$, $\tilde p_\beta$ converges to $p$ restricted to these three points. A sharp target therefore need not concentrate on a single sequence.

(3/3) What the practical sampler targets

W2. ... masked tokens are refilled in one parallel step, refills use a sampling temperature (0.3 or 0.7) … Appendix C.5 also shows that refills closer to the assumption do not improve reward.

Our theoretical preservation result holds under the proposal assumptions stated in Proposition 1. The one-step refill is a computational approximation to those assumptions, so we assess its distributional accuracy empirically rather than claiming exactness, as discussed in the Remark. Appendix C.5 evaluates protein DDG and Success Rate, which measure reward and validity. Improving agreement with the reward-tilted distribution need not increase DDG.

Q2. What distribution does the practical sampler target, given one-step refills, sampling temperature, and large β\beta?

We vary β\beta, the number of proposal denoising steps, and sampling temperature separately, comparing within-region Hamming-distance distributions with the exact target. The plots appear in Appendix C.1.2, Figures 6–8, of the revised manuscript.

Effect of β\beta (Figure 6). With one-step proposals, discrepancies are already present at β=0\beta=0. As β\beta increases, the exact target becomes more concentrated near the centers of B1B_1, B3B_3, and B4B_4, and TEMPERS reproduces this concentration with smaller conditional TV distances in these regions.

Effect of proposal denoising steps (Figure 7). At β=1\beta=1, increasing the number of steps from 1 to 8 reduces the conditional TV distances in B1B_1–B4B_4 from (0.023,0.163,0.077,0.012)(0.023, 0.163, 0.077, 0.012) to (0.009,0.029,0.025,0.002)(0.009, 0.029, 0.025, 0.002). Additional proposal denoising therefore improves agreement with the target's within-region Hamming-distance distributions in this setting.

Effect of sampling temperature (Figure 8). The refill temperature τ\tau controls how sharp the proposal is: lowering it concentrates the refill on the tokens the pretrained model is most confident in, while raising it above 1 makes the refill more random than the model itself.

Thank you for the response and the new experiments. The RERD comparison at matched NFE addresses my main concern, and the unique-sequence and cluster results are informative. One suggestion for the revision: the effective cluster counts of TEMPERS, CREPE, and FK-Steering are close on both tasks, so the analysis mainly shows that TEMPERS does not collapse rather than that it covers more modes than the SMC baselines. I would suggest adjusting the motivating claim accordingly. I will raise my score to Weak Accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

TEMPERS promotes Parallel Tempering Sampling for Test-Time Steering and Masked Diffusion Models

Dear Reviewers and Chairs,

Thank you for your time and feedback. Reviewer x4X1 described TEMPERS as "simple and cheap to run." Reviewer h7pz highlighted the value of scoring complete sequences directly and of using separate models for DNA steering and evaluation. Both x4X1 and h7pz acknowledge TEMPERS and raise their ratings.

Sampling accuracy (x4X1, h7pz). In the analytic example, TEMPERS matched the known target's regional probabilities more closely than CREPE or FK-Steering (Figure 1). Proposition 1 gives sufficient conditions for exact reward-only acceptance. For the one-step approximation used in practice, we quantified deviations from the target both across and within regions.

Design success and diversity (x4X1, h7pz). At matched denoising budgets, a larger share of DNA and protein designs met both reward and quality thresholds with TEMPERS than with CREPE or RERD (Tables 1–2). TEMPERS also had the highest mean DNA success rate at every wall-clock budget tested. Its successful designs spanned multiple clusters, with effective cluster counts comparable to those of CREPE and FK-Steering. Reviewer x4X1 confirmed that the RERD comparison addressed their main concern.

Scope of the contribution (nJCW). We agree with Reviewer nJCW that an analogous within-chain update could be developed for Gaussian diffusion and respectfully disagree with their overall assessment of this work. An absorbing masked diffusion model starts from a single fully masked sequence, so there is no Gaussian source draw to perturb as in SPT and PATHS. Each TEMPERS chain uses its reward temperature to set how much of a complete sequence to remask and refill, then scores the revised sequence before exchange. CREPE uses forward and reverse diffusion to propose exchanges across noise levels. In both the analytic example and the protein design task, the median reward of sequences proposed for exchange, measured before acceptance, was higher for TEMPERS than for CREPE (Figure 2, panel C).

Closing Remarks

We believe the additional experiments address most reviewer concerns and clarify the limits of the one-step approximation. TEMPERS offers a practical way to use parallel tempering for test-time steering in masked diffusion models, and future work can test how well analogous updates perform in Gaussian diffusion under a fixed denoising budget.

Sincerely, Author Submissions 46594

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.