- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Temperature-Enhanced Masked Proposal and Exchange for Reward Steering (TEMPERS), which is a test-time steering method for diffusion models that uses replica exchange to avoid some of the drawbacks of other steering methods, such as reduced diversity. In contrast to prior work, which uses replicates for different diffusion noise levels, they maintain replicates at different reward temperatures. They generate candidates by remasking sampled complete sequences and pair larger remasking fractions with higher reward temperatures and smaller remasking fractions with lower temperatures, with the intuition that the former encourages broader exploration while the latter encourages local refinement. They demonstrate the effectiveness of this approach first in a synthetic setting, where they show that TEMPERS achieves lower KL to a known reward-tilted target than other methods. They then show that it achieves higher rewards on DNA generation, protein generation, and sentiment-controlled text generation. They further show increased diversity in protein generation as well as the ability to maintain instruction following capability during sentiment-controlled text generation.
Critical strengths
This paper has a few strengths.
First, the algorithm design is clear and fairly intuitive. Additionally, ensuring that each state is a complete sequence is useful for reward evaluation.
Second, the synthetic experiment is clean and effective at demonstrating the method's purpose and performance.
Third, the authors are able to show improvements on the reward for which they optimize as well as some external metrics in all settings. The results on DNA in particular are strong and it is notable that the authors take care to separate training and evaluation rewards using models trained on disjoint subsets.
Critical weaknesses
There are a few different weaknesses that the authors should address.
First, the generation of complete sequences seems to be computationally expensive. The authors provide some experiments at matched NFE, which is good, but I would like to see comparisons against baselines at fixed computational budget for the different settings, as well as performance against wall-clock time.
Second, there are a few issues with the evaluation measures. For the sentiment analysis experiments, the authors should additionally report scores from an external classifier not used as the reward model. Additionally, Hamming distance seems like an important but potentially weak measure for diversity. Additional measures, such as those considered in [1] should also be reported, e.g. motif-based diversity, cluster-based diversity, and sequence PPL. It may also be interesting to report Fréchet Biological Distance [2].
Third, there is a slight mismatch with the theory section when sampling all tokens at once (to the author's credit, this is mentioned directly). I think the authors should state directly that the exactness guarantees do not transfer directly to the approximation they use. If possible, they could provide some theoretical characterization of how the stationary distribution under the approximation differs from that of the true conditional, but absent that an empirical analysis of the gap could be sufficient. In particular, it would be interesting to see if the approach they use in practice preserves the prior when you set to 0.
Finally, it seems concerning that while the model is able to optimize the exact reward function, it seems to suffer on other measures. For example, in proteins we see worse results on scRMSD and sentiment seems to suffer on perplexity.
[1] Lal, A., Gunsalus, L., Gupta, A. et al. Polygraph: a software framework for the systematic assessment of synthetic regulatory DNA elements. Genome Biol 26, 114 (2025). https://doi.org/10.1186/s13059-025-03584-9
[2] Stark, Hannes, et al. "Dirichlet flow matching with applications to dna sequence design." arXiv preprint arXiv:2402.05841 (2024).
Questions for authors
Can I see the within-region distributions for the synthetic experiment? It is possible to match the region histogram but be distorted within each region.
Can you correct the link for DDG? It gives a 404 error when clicked.
For the sentiment experiments, can you also report results from the baseline model using temperature sampling, not only greedy decoding?
Can you provide more details on hyperparameter tuning? At times you mention an "untuned uniform configuration" and there is not too much information on the tuning procedure for the method and baselines.
6 comments
W2. Add more evaluation metrics for the generalization and the diversity
W2. Second, there are a few issues with the evaluation measures. For the sentiment analysis experiments, the authors should additionally report scores from an external classifier not used as the reward model. Additionally, Hamming distance seems like an important but potentially weak measure for diversity. Additional measures, such as those considered in [1] should also be reported, e.g., motif-based diversity, cluster-based diversity, and sequence PPL. It may also be interesting to report Fréchet Biological Distance [2].
We report an external sentiment classifier for text generation, cluster-based diversity for DNA and protein design, and motif-based diversity and FBD for DNA design.
External sentiment classifier. We rescored the existing outputs with SiEBERT based on DistilBERT's steering outputs.
| Method | SiEBERT ↑ |
|---|---|
| Pretrained (t=0.7) | 0.533 ± 0.065 |
| Best-of-K (K=10) | 0.633 ± 0.063 |
| SMC | 0.583 ± 0.064 |
| CREPE | 0.650 ± 0.062 |
| TEMPERS | 0.667 ± 0.061 |
Cluster-based diversity. Reported in our response to Reviewer x4X1 (W2/Q2), where we cluster the valid high-reward outputs on both tasks and report the effective clusters exp(H).
Motif-based diversity. On DNA, we construct binary motif-presence vectors using JASPAR/FIMO at p ≤ 10⁻³, over the same qualified outputs as the clustering analysis.
| Method | Effective Motif Combinations ↑ | Mean Jaccard Distance ↑ |
|---|---|---|
| Best-of-K (K=10) | 4.33 ± 1.25 | 0.571 ± 0.047 |
| FK-Steering (K=10) | 32.00 ± 0.00 | 0.618 ± 0.002 |
| RERD (β=3) | 20.58 ± 2.29 | 0.496 ± 0.046 |
| RERD (β=5) | 16.96 ± 0.28 | 0.418 ± 0.043 |
| CREPE | 32.00 ± 0.00 | 0.641 ± 0.011 |
| TEMPERS | 32.00 ± 0.00 | 0.641 ± 0.014 |
Fréchet Biological Distance. On DNA, we compute FBD using Gosai/oracle_eval features against a high-HepG2-activity reference distribution and the same 736 reference sequences and 6,144-dimensional features for every method.
| Method | FBD (HepG2 Reference) ↓ |
|---|---|
| RERD (β = 3) | 11049.89 ± 3348.00 |
| RERD (β = 5) | 11568.47 ± 2047.25 |
| CREPE | 2160.37 ± 230.16 |
| TEMPERS | 1541.35 ± 99.60 |
These results support its ability to generate valid high-reward candidates across multiple sequence clusters.
W3. Theoretical analysis and empirical implementation
W3. Third, there is a slight mismatch with the theory section when sampling all tokens at once (to the author's credit, this is mentioned directly). I think the authors should state directly that the exactness guarantees do not transfer directly to the approximation they use. If possible, they could provide some theoretical characterization of how the stationary distribution under the approximation differs from that of the true conditional, but absent that an empirical analysis of the gap could be sufficient.
The exactness in Proposition 1 does not transfer to the one-step parallel refill used in our experiments, and the remark following Proposition 1 now states this directly. We keep the one-step configuration because its lower proposal cost permits more MCMC sweeps under a fixed denoising budget.
Specifically, Proposition 1 assumes a mask of fixed size chosen uniformly and refilling from the exact joint conditional under the same prior used in the target. This can be implemented by unmasking the tokens sequentially, each sampled from its exact conditional given the unmasked context and the tokens already regenerated. Our configuration uses uniform remasking, so the first assumption holds. The second is not guaranteed by our practical configuration: the main experiments unmask all masked tokens in a single denoising pass, sampling tokens independently from the model-predicted distributions and ignoring the dependence among the remasked positions. When a single token is remasked, parallel and sequential refilling have no factorization discrepancy. Exactness still requires the refill distribution to match the conditional of the same prior used in the target, including any effect of the sampling temperature.
We cannot carry this through to a closed form for the stationary distribution, so we take the empirical route the reviewer suggests, on the analytic example where the exact target is available.
In particular, it would be interesting to see if the approach they use in practice preserves the prior when you set to 0.
Preserving the prior at . At every proposal is accepted, so any movement of the output distribution is attributable to the proposal alone. We initialize from the pretrained model's generative distribution and measure region TV against independent reference samples from the same model.
| Proposal Denoising Steps | Region TV Before Updates ↓ | Region TV After 60 Updates ↓ |
|---|---|---|
| 1 | 0.016 | 0.061 ± 0.015 |
| 2 | 0.016 | 0.061 ± 0.011 |
| 4 | 0.016 | 0.047 ± 0.009 |
| 8 | 0.016 | 0.024 ± 0.010 |
The prior is not preserved exactly. The 0.016 before any sweep is the resolution of this measurement, being the distance between two independent draws from the same model, and it is identical across the four rows because the initialization does not depend on the proposal. After 60 sweeps the one-step sampler sits at about four times that resolution, and one and two steps are indistinguishable at this seed spread. Eight steps brings the deviation to 0.024, within one and a half times the resolution, so it is largely removed by moving the proposal toward the sequential procedure that Proposition 1 assumes.
Agreement with the reward-tilted target. We also measure region TV after 60 sweeps against the exact regional probabilities under .
| β | 1 step | 2 steps | 4 steps | 8 steps |
|---|---|---|---|---|
| 0 | 0.059 ± 0.015 | 0.053 ± 0.010 | 0.040 ± 0.008 | 0.021 ± 0.006 |
| 0.5 | 0.083 ± 0.006 | 0.077 ± 0.009 | 0.054 ± 0.007 | 0.031 ± 0.012 |
| 1 | 0.143 ± 0.008 | 0.112 ± 0.004 | 0.082 ± 0.013 | 0.063 ± 0.006 |
| 2 | 0.304 ± 0.016 | 0.269 ± 0.005 | 0.221 ± 0.013 | 0.190 ± 0.012 |
| 5 | 0.367 ± 0.014 | 0.347 ± 0.006 | 0.299 ± 0.014 | 0.258 ± 0.009 |
| Denoising NFE | 400 | 700 | 1,300 | 2,500 |
At the target coincides with the prior and the first row agrees with the measurement above, which also shows that the trained model and the analytic prior differ by less than the resolution of either measurement. Four and eight steps reduce the deviation at every , at a denoising cost that the last row makes explicit.
W1. Add more evaluation metrics for the computational efficiency
We report both budgets on the biological tasks. Under matched denoising NFE, TEMPERS is faster than RERD and CREPE on DNA and attains the highest Success Rate on protein at a sampling time below CREPE's. Under shared wall-clock limits it has the highest mean Success Rate at every limit on DNA, and on protein it is ahead at 25 and 40 seconds and level at 50 seconds. All denoising calls are counted, including initialization and every replica, and the measured times include the guidance oracle.
Matched denoising NFE. 4,610 denoising NFE per output on DNA and 1,000 on protein, as in the updated Tables 1 and 2 of our response to Reviewer x4X1 W1/Q1.
DNA sampling time at 4,610 denoising NFE:
Protein sampling time at 1,000 denoising NFE:
TEMPERS is faster than RERD, CREPE, and FK-Steering. While Best-of-K is the fastest method, its Success Rate is worse than existing test-time steering methods in Table 1 and Table 2.
Shared wall-clock limits. Every method was also run under a common per-batch time limit on the same Ada6000 GPU, with three seeds throughout: 60, 90 and 120 seconds per batch of 64 on DNA, and 25, 40 and 50 seconds per batch of 16 on protein.
DNA Success Rate (Activity ≥ 5 and a 3-mer Pearson Corr ≥ 0.3) at each limit:
TEMPERS attains the highest mean Success Rate, exceeding CREPE by 11.98, 4.95 and 6.25 points. RERD reaches higher raw Activity, as in Table 1, but fewer of its outputs clear both thresholds.
Protein Success Rate at each limit:
† FK-Steering did not complete all prescribed outputs within the time limit for 3 of the 12 backbones at 25 and 40 seconds, and 2 of the 12 at 50 seconds, where "—" indicates incomplete output coverage within the budget.
Among methods with complete results for all 12 backbones, TEMPERS has the highest mean Success Rate at 25 and 40 seconds. At 50 seconds, CREPE and TEMPERS are close, with CREPE slightly higher (70.9% vs. 70.8%).