acceptodds
Under review as a conference paper at ICLR 2027

A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decoding

Abstract

Discrete diffusion models offer a simple and stable likelihood-based framework for sequence generation, recently extended to any-length settings via token insertion. Principled reward-guided fine-tuning for any-length discrete diffusion, however, remains largely unexplored. We introduce Fine-Tuning **A**ny-Length **D**iscrete **D**iffusion for **A**daptive Decoding (**A2D2**), a unified framework for reward-guided fine-tuning of any-length discrete diffusion models via joint optimization of the insertion and unmasking policies together with a quality-based inference schedule. We derive the Radon-Nikodym derivative for the joint insertion-unmasking path measures, enabling theoretically guaranteed convergence to the intractable reward-tilted sequence distribution without requiring target samples. Building on this, we establish unmasking and insertion quality as tractable approaches for minimizing decoding error and introduce the **Adaptive Joint Decoding (AJD)** loss, which provably yields the optimal path measure that generates the reward-tilted distribution. Empirically, A2D2 improves reward optimization while enhancing generation flexibility and accuracy over prior fixed-length fine-tuning and inference-time guidance methods.

Primary area:
generative models
Keywords:
iscrete diffusion, fine-tuning, reinforcement learning, reward optimization, reward alignment, adaptive decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nb2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes A2D2, a framework for finetuning discrete diffusion models with insertion and unmasking towards the reward-tilted distribution. The contribution mainly includes the adaptive joint decoding objective that uses importance-weighted denoising and insertion loss and the quality predictors that guide adaptive inference. Experiments on small molecule generation, peptide design, and language modeling show that A2D2 can improve over the baselines.

Critical strengths
  1. The paper tackles the important problem of reward-tilted finetuning of adaptive length discrete diffusion models. Compared with fixed length discrete diffusion, optimizing adaptive length models yields more flexibility during inference.

  2. Empirically, the reward-tilted finetuning seems to yield decent improvement over the pretrained adaptive length model, across several tasks and benchmarks.

Critical weaknesses
  1. The paper claims theoretical optimality of the AJD objective. However, it may need more careful reconsiderations. See Q1 and Q2.

  2. The contribution of the quality-based adaptive decoding is not fully supported by the experiment. Some parts of the experiments also raise concerns on the effectiveness of the proposed approach. See Q3 and Q4.

Questions for authors

Q1. The insertion rates are parameter-dependent. However, the exit-rate terms in general CTMC likelihood are omitted when deriving Eq. 125. Why is this allowed? If they cannot be omitted, then is the ADJ objective still provably optimal for minimizing the KL towards the reward-tilted target measure?

Q2. Algorithm 2 additionally includes quality-based remasking and deletion, while Proposition 4.1 does not consider these operations. How do these operations affect the importance weights? Does the derivation of the importance weights still hold?

Q3. Could the authors provide more explanations/analyses on why jointly using both predictors sometimes performs worse than using only one of them (e.g. see Table 1)?

Q4. Table 3 only reports the pretrained instruction-tuned baseline but not an any-length model funetuned with the AJD objective without quality predictors. It may be helpful to include this ablation.

Q5. It seems that Eq. 2 is minimizing the negative of the defined scalar Bregman divergence \phi, which in turn maximizes the divergence. Is this the intended behavior or a typo?

5 comments
AuthorsOpenReview ↗

Author Response To Reviewer nb2p (1/3)

Dear Reviewer nb2p,

Thank you for the careful and technically precise review. We are glad you recognized the importance of reward-based fine-tuning for adaptive-length discrete diffusion, the inference flexibility it provides, and the empirical improvements over the pretrained model across tasks.

We hope to clear up your concerns with the additional experiments and responses below:

[Q1a] The insertion rates are parameter-dependent. However, the exit-rate terms in general CTMC likelihood are omitted when deriving Eq. 125. Why is this allowed?

You are correct that the exit-rate term cannot be omitted. This was an error in our writing rather than an implementation error. For generator AtθA_t^\theta with exit rate λtθ(x)=∑y≠xAtθ(x,y)\lambda_t^\theta(x) = \sum_{y\ne x} A_t^\theta(x,y), the path negative log density is

−log⁡P0θ(X0)−∑jlog⁡Atjθ(Xtj−,Xtj)+∫01λtθ(Xt) dt.-\log P_0^\theta(X_0) - \sum_j \log A_{t_j}^\theta(X_{t_j^-}, X_{t_j}) + \int_0^1 \lambda_t^\theta(X_t)\,dt.

The normalized unmasking posterior has total exit rate independent of θ\theta, so that contribution drops out. The insertion exit rate depends on gθg_\theta and must be retained, and it supplies exactly the positive predicted-rate contribution that accompanies the insertion jump term. Concretely, the insertion loss in our implementation computes DPois(x∥g)=g−x+xlog⁡(x/g)D_{\mathrm{Pois}}(x\Vert g) = g - x + x\log(x/g), where xx is the realized gap count and g=gθ(xt,t)[ℓ]g = g_\theta(\boldsymbol{x}_t,t)[\ell] the predicted rate, and adds it with a positive weight into the total training loss. The leading gg is exactly the compensator, and the orientation is minimizing. The omission was therefore a small error in our write-up, which we have now fixed: the correction aligns the written derivation with the objective the models were actually trained under, and leaves the trained models themselves unchanged.

[Q1b] If they cannot be omitted, then is the ADJ objective still provably optimal for minimizing the KL towards the reward-tilted target measure?

Yes, it is optimal. We now show a complementary conditional endpoint-consistency result. Write ptarget(x)=Z−1ppre(x)er(x)/αp_{\mathrm{target}}(\boldsymbol{x})=Z^{-1}p_{\mathrm{pre}}(\boldsymbol{x})e^{r(\boldsymbol{x})/\alpha} and let Q\mathbb{Q} be the actual proposal on a common supported path space. With exact weight w=dP⋆/dQw=\mathrm{d}\mathbb{P}^\star/\mathrm{d}\mathbb{Q} and fresh corruption randomness included in ℓθ\ell_\theta, EQ[wℓθ(X1)]=Ex∼ptarget[ℓθ(x)].\mathbb{E}_{\mathbb{Q}}[w\ell_\theta(\boldsymbol{X}_1)]=\mathbb{E}_{\boldsymbol{x}\sim p_{\mathrm{target}}}[\ell_\theta(\boldsymbol{x})]. At an unrestricted population optimum, cross-entropy fits the target conditional token probabilities and the positive Poisson divergence fits the conditional expected gap counts. If these statistics define the chosen interpolant's generator and are realizable, matched initialization and uniqueness of the forward equation give endpoint law ptargetp_{\mathrm{target}}.

The endpoint law is the unique optimum of the KL-regularized reward objective because Ex∼p[r(x)]−αDKL(p∥ppre)=αlog⁡Z−αDKL(p∥ptarget).\mathbb{E}_{\boldsymbol{x}\sim p}[r(\boldsymbol{x})]-\alpha D_{\mathrm{KL}}(p\Vert p_{\mathrm{pre}})=\alpha\log Z-\alpha D_{\mathrm{KL}}(p\Vert p_{\mathrm{target}}). The appendix separately derives the optimal path law and generator. With ht(x)=EPpre[er(X1)/α∣Xt=x]h_t(\boldsymbol{x})=\mathbb{E}_{\mathbb{P}^{\mathrm{pre}}}[e^{r(\boldsymbol{X}_1)/\alpha}\mid\boldsymbol{X}_t=\boldsymbol{x}], the transformed rates are At⋆(x,y)=Atpre(x,y)ht(y)/ht(x)\boldsymbol{A}_t^\star(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{A}_t^{\mathrm{pre}}(\boldsymbol{x},\boldsymbol{y})h_t(\boldsymbol{y})/h_t(\boldsymbol{x}). The derivation includes the reward scaling, initial-law condition, and waiting-time compensator. For the deterministic empty initialization, h0(x0)=Zh_0(\boldsymbol{x}_0)=Z and the initial law is unchanged. The quality predictors learn auxiliary reconstruction statistics for the adaptive decoder. We state their role separately from the base-process path guarantee and evaluate the resulting quality-guided procedure empirically.

AuthorsOpenReview ↗

Author Response To Reviewer nb2p (2/3)

[Q2] How do remasking and deletion affect the importance weights?

The revised proposition specifies exact importance accounting for the base insertion–unmasking CTMC. For matched reference and proposal initial laws, common schedules, and the stated support conditions, its log weight contains the terminal reward, the reference-to-proposal jump-rate ratios evaluated at pre-jump states, and the exit-rate compensator. The manuscript defines the unnormalized score WvW^v without −log⁡Z-\log Z, so eWv/Ze^{W^v}/Z includes the normalizer once.

The implemented adaptive decoder uses the quality heads to guide remasking and deletion after policy warmup. The replay score evaluates the retained unmasking outcomes and post-deletion insertion counts using the policy and reference model probabilities. Subsequent policy evaluations condition on the resulting modified sequence states. The heads therefore influence both the generated trajectories and the outcomes entering the replay score. We accordingly describe this quality-guided replay weighting as approximate and distinguish it from the exact importance-accounting result for the specified base insertion–unmasking CTMC.

The appendix now presents these statements as follows: the base-process path density, the optimal reference tilt, the insertion–unmasking likelihood ratio, and the conditional AJD endpoint result. This makes clear which proposal the exact identity concerns and which procedure is evaluated in the experiments. The auxiliary quality losses train reconstruction scores, with the trained configurations and the frozen-checkpoint diagnostic providing complementary empirical views of the adaptive decoder.

[Q3] Why can both heads perform worse than either head alone?

The heads supply distinct controls over the same evolving sequence. Remasking revisits the identities of decoded tokens, while deletion rejects newly inserted mask positions. Removing positions changes the sequence context, length, and mask count used by subsequent remasking decisions. This interaction between remasking and deletion motivates analyzing the joint decoder as a coupled procedure and evaluating its configurations empirically.

The submitted training ablations describe task-dependent reward–coverage operating points. On molecules, insertion quality alone attains the highest A2D2 quality at 82.37% and QED at 0.818. Both heads attain the highest diversity among the A2D2 quality variants at 0.843, together with uniqueness of 93.695%. On peptides, the full quality-guided procedure raises TfR validity from 41.27% to 48.60% relative to the head-free model.

The quality configurations can therefore be chosen according to the application's objectives. The revised Results section describes this reward-coverage trade-off. Notably, every configuration with at least one quality head improves uniqueness and diversity over the head-free model in the submitted molecule training ablations, supporting quality prediction as a way to preserve coverage during reward fine-tuning. These trained configurations provide different operating points for reward and diversity.

AuthorsOpenReview ↗

Author Response To Reviewer nb2p (3/3)

[Q4] Can you include AJD without quality predictors in the language comparison?

Yes. Reproducing the necessary checkpoints for our adapted any-length model and its instruction-fine-tuned version requires costly retraining due to lost checkpoints, so we are unable to provide this ablation and the requested functional evaluations at this time. We will add the full language ablations and functional results in a later revision. The current language comparison evaluates the full A2D2 procedure against instruction fine-tuning on the same any-length backbone under matched sampling-step budgets.

[Q5] Does the negative sign in Eq. 2 maximize the insertion divergence?

Yes, thank you for catching this typo! The insertion loss is now written with a positive coefficient, +α˙t/(1−αt)+\dot\alpha_t/(1-\alpha_t), multiplying DPois(c∥g)=g−c+clog⁡(c/g)D_{\mathrm{Pois}}(c\Vert g)=g-c+c\log(c/g). This is the positively weighted divergence used in the minimized training objective. We use 0log⁡(0/g)=00\log(0/g)=0 and the extended-value convention when g=0g=0. The sign and divergence notation are consistent across the main text, background, and training algorithm.

For a fixed target count cc, g−clog⁡gg-c\log g differs from DPois(c∥g)D_{\mathrm{Pois}}(c\Vert g) by a target-dependent constant. The gradients and minimizer are therefore the same. For a positive predicted count, the derivative is 1−c/g1-c/g, which raises an underpredicted positive count and lowers a positive prediction when the target count is zero. The written correction thus agrees with the trained objective and preserves the trained models.

[W1] The AJD optimality claim needs reconsideration.

The revised theory explicitly separates exact base-process importance accounting, AJD population endpoint optimality, and the quality-guided decoding implementation. The optimal-generator and path-measure lemmas give the full reward-tilt derivation with waiting times and initialization stated. The AJD proposition then establishes its endpoint result through exact weighting and conditional-statistic recovery under realizability and forward-equation assumptions. This retains the optimality result while specifying the additional generator and bridge identification needed for full path recovery. Our replies to Q1a, Q1b, Q2, and Q5 give the corresponding details.

[W2] The quality-based decoder needs stronger empirical support.

The revised discussion connects the insertion-quality bound to a reconstruction surrogate, explains how deletion and remasking act on different objects, and interprets the quality configurations through the submitted training ablations. It also separates those comparisons from the frozen-checkpoint inference diagnostic. Our reply to Q3 describes the reward–coverage operating points, and our reply to CshR W2 explains the deletion mechanism and operation-order study. The head-free language training ablation will be included as soon as possible.


If we have addressed your concerns, we hope you will consider raising your score in support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

Reviewer_CshROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces A2D2, a novel framework for reward-based fine-tuning of any-length discrete diffusion models. A2D2 trains insertion and unmasking policies alongside two auxiliary quality predictors that guide token remasking and deletion of newly inserted masks. Its Adaptive Joint Decoding (AJD) objective uses importance-weighted MLE training to amortize sampling from a reward-tilted distribution. Experiments demonstrate effective fine-tuning for small-molecule and peptide generation.

Critical strengths
  1. Novel and well-written A2D2 is the first framework for fine-tuning any-length discrete diffusion models with user-specified rewards. The paper is well written, and the method is presented convincingly.

  2. Generality. A2D2 demonstrates its effectiveness across diverse domains, including small-molecule generation using SAFE strings, peptide generation, and mathematical reasoning. These results suggest that its underlying approach is broadly applicable beyond a single domain.

  3. Effective AJD loss can bypass the integration of likelihood through the measure, unlike VIDD [1], bringing computational advantages like DiffusionNFT [2] over DDPO [3]

[1] Su, Xingyu, et al. "Iterative distillation for reward-guided fine-tuning of diffusion models in biomolecular design." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Black, Kevin, et al. "Training diffusion models with reinforcement learning." International Conference on Learning Representations. Vol. 2024. 2024.

Critical weaknesses
  1. While the framework is well motivated and appears broadly applicable, the core AJD objective follows established probabilistic inference-based approaches like importance sampling. Could the authors more concretely distinguish their approach from probabilistic inference-based methods, such as VIDD [4] and RTB [5, 6]? While A2D2 avoids recomputing the inner RND at each parameter update, a more detailed comparison with existing reward-weighted fine-tuning methods would help clarify its methodological and practical advantages.

  2. The paper addresses reward-based alignment of any-length discrete diffusion models. However, the complementary benefit of insertion-quality-guided deletion and reward fine-tuning seems unclear. Compared with the underlying FlexMDM, quality-guided insertion deletion appears to be an additional contribution, but variable-length generation and fine-tuning are already possible without this mechanism. What benefit does deletion provide specifically to reward alignment, beyond serving as a validity correction?

  • Minor comment: Please use distinct symbols for the Bregman divergence ϕ\phi in Eq. (2) and the quality predictors’ parameters ϕ\phi for clarity.

[4] Su, Xingyu, et al. "Iterative distillation for reward-guided fine-tuning of diffusion models in biomolecular design." International Conference on Learning Representations. Vol. 2026. 2026.

[5] Venkatraman, Siddarth, et al. "Amortizing intractable inference in diffusion models for vision, language, and control." Advances in neural information processing systems 37 (2024): 76080-76114.

[6] Deleu, Tristan, et al. "Relative trajectory balance is equivalent to Trust-PCL." arXiv preprint arXiv:2509.01632 (2025).

Questions for authors
  1. Off-policy formulation of A2D2 suggests a potential benefit from combining inference-time search with distillation. Because AJD relies on importance sampling, a substantial mismatch between the sampling and reward-tilted target distributions may reduce ESS and increase weight variance. Could the authors investigate whether generating samples with reward-guided inference, such as [7], and subsequently distilling them through AJD improves sample efficiency? Evaluating this under matched reward-evaluation budgets, with appropriate importance-weight corrections for the guided proposal, would help assess this possibility.

[7] Nisonoff, Hunter, et al. "Unlocking guidance for discrete state-space diffusion and flow models." International Conference on Learning Representations. Vol. 2025. 2025.

3 comments
AuthorsOpenReview ↗

Author Response To Reviewer CshR (1/2)

Dear Reviewer CshR,

Thank you for the encouraging review and for recognizing the framework's novelty, cross-domain applicability, and computational appeal. We clarify the methodological distinction, the role of deletion, and the guided-proposal experiment below.

[W1] How does AJD differ from VIDD and RTB beyond established importance sampling?

We agree that importance weighting and reward-tilted inference are established. Our contribution is their use over a joint insertion and unmasking path measure, together with insertion-quality prediction and an adaptive decoder. Our algorithm also differs from prior methods in sample efficiency and is, to our knowledge, the first to apply importance weighting techniques to any-length discrete diffusion fine-tuning.

RTB [3] fits a squared log trajectory-balance residual that involves the current trajectory likelihood and a learned normalizer. VIDD [4] uses iterative roll-in, rollout, and value-weighted likelihood updates to distill reward-guided transitions. AJD stores weighted endpoints and trains on fresh corruptions, with the stored weights held fixed during the replay updates. This distinction determines the computation performed when a stored sample is reused. After collecting a replay buffer and computing its scores, AJD draws new corruption times and intermediate states from each clean endpoint and updates the insertion and unmasking predictors with their denoising losses. It avoids reevaluating the current full trajectory likelihood at every replay update. In the main molecule recipe, the buffer supports 25 policy updates, allowing multiple training updates per reward evaluation. The revised theory gives the exact change-of-measure identity for the specified base process, and the implementation discussion identifies the quality-guided replay score as approximate.

The Additional Baselines section complements this methodological comparison by varying the training objective at a matched budget of 36,400 oracle calls with both quality heads disabled. It includes REINFORCE with KL regularization, reward-weighted maximum likelihood, and head-free A2D2. Those rows evaluate our implementations of the stated objectives. The RTB and VIDD discussion in the Related Works in the revision explains their learning mechanisms and the distinct computation used by AJD. The new proposal-guidance study in Table R2 additionally tests how reward-informed replay generation can be combined with the endpoint-distillation update.

[W2] What does learned deletion contribute beyond validity correction?

Deletion acts on newly inserted mask positions before their token identities are generated. It can therefore reject a proposed position and change length. Remasking revisits an existing decoded token identity while retaining its position. These operations support different decisions during generation: whether a position is needed and whether an existing token should be reconsidered.

The insertion-quality predictor makes this decision context-dependent. A mask proposed early in the trajectory can be reevaluated after neighboring tokens have been decoded. Under the insertion-quality proposition's stated assumptions, the product of insertion qualities bounds reconstruction success and motivates a tractable reconstruction surrogate for retaining proposed positions. The revised text explains this relationship and evaluates the resulting decoder through the training ablations and the separate inference diagnostic.

The appendix also studies operation order on one released molecule checkpoint with both learned heads, using three generation seeds with 256 attempts each and a matched padded model-call budget. The comparison examines when insertion, unmasking, remasking, and deletion act on the current state. Inserting before unmasking makes new positions available for decoding in the same step, while subsequent correction decisions use the updated context.

AuthorsOpenReview ↗

Author Response To Reviewer CshR (2/2)

Minor: use distinct symbols for the divergence and the quality-predictor parameters.

We now use DPois(c∥g)=g−c+clog⁡(c/g)D_{\mathrm{Pois}}(c\Vert g)=g-c+c\log(c/g) for the insertion divergence and reserve ϕ\phi for the quality-predictor parameters. The insertion divergence enters the minimized loss with the positive coefficient α˙t/(1−αt)\dot\alpha_t/(1-\alpha_t). This notation is used consistently in the main equations, background, and training algorithm, with 0log⁡(0/g)=00\log(0/g)=0 and the extended-value convention at g=0g=0.

[Q1] Can reward-guided proposals improve sample efficiency with corrected weights?

Yes, this is a well-motivated extension, and we have now added early evidence that supports it! Reward-guided proposals are compatible with the base-process change-of-measure construction. Following discrete-state guidance [2], a positive value estimate h^t\hat h_t defines rates Atg(x,y)=Atpre(x,y)h^t(y)/h^t(x)\boldsymbol{A}_t^g(\boldsymbol{x},\boldsymbol{y})=\boldsymbol{A}_t^{\mathrm{pre}}(\boldsymbol{x},\boldsymbol{y})\hat h_t(\boldsymbol{y})/\hat h_t(\boldsymbol{x}). For matched initial laws, their correction is log⁡dPgdPpre(X0:1)=∑jlog⁡h^tj(Xtj)h^tj(Xtj−)−∫01(λtg−λtpre)(Xt) dt.\log\frac{\mathrm{d}\mathbb{P}^g}{\mathrm{d}\mathbb{P}^{\mathrm{pre}}}(\boldsymbol{X}_{0:1})=\sum_j\log\frac{\hat h_{t_j}(\boldsymbol{X}_{t_j})}{\hat h_{t_j}(\boldsymbol{X}_{t_j^-})}-\int_0^1(\lambda_t^g-\lambda_t^{\mathrm{pre}})(\boldsymbol{X}_t)\,\mathrm{d}t. Replay then uses dP⋆/dPg\mathrm{d}\mathbb{P}^\star/\mathrm{d}\mathbb{P}^g. Under the stated support and integrability assumptions, this correction accommodates any positive estimate h^t\hat h_t, with estimate quality affecting the distribution of importance weights.

Table R2 in the General Response evaluates finite-step branch-and-select guidance on molecules. At two intermediate stages, the sampler considers four candidate continuations, scores independent lookahead completions, and selects a partial state from which to continue. This uses nine reward calls per episode. Each selection contributes −log⁡(Ksselected)-\log(Ks_{\mathrm{selected}}) on the augmented proposal space. Guided, uniform-branching, and all-candidate replay use equal reward-call budgets, identical distillation settings, and globally normalized untempered weights at α=0.01\alpha=0.01. We average five independent replay-bank and training seeds, with three evaluation seeds and 300 attempts per evaluation seed. Evaluation disables guidance.

The guided arm has the highest observed mean at every reward budget, reaching 50.44%, 50.91%, and 51.22% quality at 1,152, 2,304, and 4,608 calls. At the largest budget, guided minus all-candidate replay is +6.44 percentage points, with a paired 95% t-confidence interval of [−5.24, 18.13] across the five independent bank and training seeds. Before distillation, the corresponding bank qualities are 44.49% and 42.23%. These observed improvements make guided proposals a promising direction for reward-efficient fine-tuning. Since evaluation uses guidance-free decoding, the result concerns the trained policy rather than continued reward search during evaluation.


If we have addressed your concerns, we hope you consider raising your score in stronger support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

AuthorsOpenReview ↗

Thank you for your positive assessment and new language ablation

Dear Reviewer CshR,

As the discussion period comes to a close, we would like to thank you again for your positive assessment and for recognizing the novelty, breadth, and computational appeal of A2D2. Your comments directly shaped the revision. The Related Work now clarifies how AJD differs from RTB and VIDD, the method section explains the distinct roles of deletion and remasking, and Table R2 evaluates reward-guided proposals at matched reward-call budgets.

Since our initial response, we have also completed the head-free language ablation with functional evaluation on the full GSM8K and HumanEval infilling benchmarks. Quality-guided decoding improves GSM8K from 36.6% to 45.6% and code-infilling pass@1 from 69.3% to 72.2% (Table R3 in the global comment).

In light of these new experiments and clarifications, we would be grateful if you would consider raising your score in stronger support of acceptance.

Sincerely,

The Authors

Reviewer_zMjxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A2D2 proposes reward-based fine-tuning for discrete diffusion models that can change sequence length through insertion. It combines insertion and unmasking policies with two quality predictors that guide deletion of newly inserted positions and remasking of generated tokens. The training procedure uses trajectory importance weighting, and the paper supplies extensive theoretical analysis alongside experiments on molecules, peptides, GSM8K, and code infilling. The approach improves several of its own pretrained or instruction-tuned baselines. However, I find the empirical evidence insufficient to establish the framework's practical advantage over previous insertion-based or variable-length approaches, particularly for language generation.

Critical strengths
  1. The problem is relevant: a variable-length diffusion model must decide both what to generate and how many positions to create. Combining reward alignment with explicit quality control for these two decisions is a reasonable and potentially useful design. The insertion-quality component addresses a requirement that does not arise in the same form for fixed-length models.
  2. The experiments show improvements over the chosen starting points and include molecules and peptides as well as language tasks. The training-objective comparisons, single-quality-head ablations, and best-of-N controls are useful efforts to examine simpler alternatives. These provide a credible starting point for assessing the method, even though the language-specific evidence remains limited.
Critical weaknesses
  1. The language evaluation is too limited to establish the broader benefit of adaptive-length generation. GSM8K and single-line HumanEval infilling are useful initial tests, but they cover a narrow range of reasoning and completion behavior. In particular, the code metric is reference exact match rather than execution-based functional correctness, and the task does not test substantial variable-length program generation or editing. More demanding tasks with varied output lengths and meaningful opportunities for insertion or revision would make a stronger case for the proposed framework. The large gains on the current benchmarks are encouraging, but do not by themselves show where adaptive length is essential.
  2. The empirical distinction from earlier insertion-based and variable-length methods is not sufficiently clear. The paper builds on an existing any-length diffusion framework, and the central language comparison is between its instruction-tuned starting point and the complete A2D2 system. This establishes an improvement from the overall fine-tuning recipe, but leaves unclear how much advantage comes specifically from the new framework relative to prior variable-length decoding and training strategies. The fixed-length molecule/peptide baselines and training-objective comparisons are useful, but do not fully answer this question. A focused comparison using common backbones, rewards, data, and budgets would better establish the incremental contribution. The contribution of the adaptive quality mechanisms needs stronger empirical isolation in the language setting. Table 3 does not separate reward fine-tuning from insertion-quality and unmasking-quality control, and the fixed-length language model mentioned in the appendix is not included in the reported comparison. The molecule ablations already show that the complete two-head system does not uniformly outperform its single-head variants, while peptide gains involve trade-offs across properties and diversity. These findings make it particularly important to identify when the full method is beneficial, rather than infer its component-level advantages from the overall improvement.
Questions for authors
  1. Clarification: Which reported comparison most directly separates the contribution of reward fine-tuning from length adaptation and the two quality mechanisms, especially in language tasks, and how does it compare with earlier variable-length approaches under matched resources?
  2. Clarification: What evidence demonstrates that adaptive length is important to the reported gains, rather than the fine-tuning recipe alone? Results on varied output lengths or more substantial code editing with functional evaluation would materially strengthen the empirical case; these are possible ways to resolve this concern rather than a request for an unrelated benchmark suite.
5 comments
AuthorsOpenReview ↗

Author Response To Reviewer zMjx (1/2)

Dear Reviewer zMjx,

Thank you for the thoughtful review. We appreciate your recognition of reward alignment with insertion and unmasking quality control and the breadth of the molecule, peptide, and language experiments. We hope to clear up your concerns with the additional experiments and responses below:

[W1] The language evaluation is too limited. Exact match is not functional correctness, and the tasks do not show where adaptive length is essential.

The language experiment compares A2D2 with instruction fine-tuning on the same any-length backbone under matched sampling-step budgets. The revised paper keeps the code metric explicitly labeled as exact match. This measures whether the generated span reproduces the canonical completion, while the GSM8K evaluation measures final-answer correctness. Together, these results evaluate the full A2D2 procedure on the adapted language model. The code-infilling setup supplies the surrounding prefix and suffix without supplying the reference span length.

Reproducing the necessary checkpoints for our adapted any-length model and its instruction-fine-tuned version requires retraining due to lost checkpoints. We are therefore unable to provide the requested head-free language ablation, fixed-length comparison, and functional evaluations at this time. These comparisons will be included in a later revision, together with the full language ablations and results. We will report executable-test evaluation alongside exact match so that both functional correctness and reference agreement can be assessed.

For a canonical reference completion that passes the executable tests, exact match is a stricter acceptance criterion than functional pass@1: it requires the reference text itself, whereas test-based evaluation can accept alternative correct implementations. The current language comparison supports reward alignment on the any-length backbone. The paired peptide study in Table R1 provides the additional application-level motivation for treating length as an optimization variable.

[W2] The empirical distinction from earlier insertion-based and variable-length methods is not sufficiently clear. A focused comparison using common backbones, rewards, data, and budgets would better establish the incremental contribution.

The revised introduction makes the contribution explicit. We evaluate A2D2 within FlexMDM [1], which supplies the insertion-based any-length generative framework. A2D2 adds reward fine-tuning of the joint insertion and unmasking policies together with quality-based adaptive decoding. Prior variable-length approaches such as FlexMDM and Edit Flows [5] establish generative mechanisms with flexible sequence structure. Our question is how to align such generation with a user-specified terminal reward while training from replayed endpoints.

The submitted comparisons examine the added components at different levels. On language, A2D2 is compared with the instruction-tuned model on the same any-length backbone. On molecules, the Additional Baselines section compares policy-training objectives at a matched budget of 36,400 oracle calls with both heads disabled. On peptides, the comparison with fixed-length reward fine-tuning evaluates the complete systems under the same five therapeutic objectives. The revised peptide Results section states the scope of each comparison so that improvements from reward fine-tuning, the quality-guided training-and-decoding procedure, and the fixed-length system comparison can be interpreted separately.

We emphasize that our approach applies more generally to insertion-based generative processes where the likelihood of the generative trajectory can be estimated, to optimize any terminal reward. These realizability and regularity assumptions specify how to transfer the approach to another insertion-based model.

AuthorsOpenReview ↗

Author Response To Reviewer zMjx (2/3)

[W3] Table 3 does not separate reward fine-tuning from the quality mechanisms, and both heads do not uniformly outperform either head alone.

We agree, and the revision states this directly. A2D2 builds on FlexMDM's any-length backbone [1], with reward fine-tuning and quality-guided decoding as two further components layered on top.

Our controls show task-dependent trade-offs rather than uniform dominance. For instance, the molecule ablations demonstrate useful operating points for different reward and coverage preferences. The insertion-quality-only configuration reaches the highest A2D2 quality at 82.37% and QED at 0.818. Enabling both heads gives the highest diversity among the A2D2 quality variants at 0.843, with uniqueness of 93.695%. The quality configurations can therefore be chosen according to the application's objectives. The revised Results section describes this reward-coverage trade-off and attributes the reported values to the complete trained configurations. Notably, every configuration with at least one quality head improves uniqueness and diversity over the head-free model in the submitted molecule training ablations, supporting quality prediction as a way to preserve coverage during reward fine-tuning. These trained configurations provide different operating points for reward and diversity.

The mechanisms act on different objects within the sequence but are coupled through the evolving state. Remasking revisits the identities of decoded tokens, while deletion rejects newly inserted mask positions. Removing positions changes the sequence context, length, and mask count used by subsequent remasking decisions. This interaction motivates analyzing the joint decoder as a coupled procedure and evaluating its configurations empirically.

AuthorsOpenReview ↗

Author Response To Reviewer zMjx (3/3)

[Q1] Which comparison separates reward fine-tuning, length adaptation, and the quality mechanisms under matched resources?

The revised peptide Results section now organizes the evidence into three comparisons. First, pretrained any-length versus A2D2 without quality evaluates reward fine-tuning on the same backbone. On TfR, validity increases from 10.06% to 41.27%, predicted affinity from 7.756 to 10.241, and predicted solubility from 0.689 to 0.792. This comparison keeps the inherited length-modeling framework fixed while adding reward fine-tuning.

Second, A2D2 without quality versus A2D2 with both heads evaluates the combined effect of quality-predictor training and quality-guided decoding. On TfR, validity increases from 41.27% to 48.60%. We interpret this result together with the predicted therapeutic properties as a task-dependent operating point of the full quality-guided procedure.

Third, head-free A2D2 versus fixed-length off-policy RL compares two complete reward-trained systems. On TfR, head-free A2D2 reaches solubility 0.792 and permeability −6.102, compared with 0.655 and −7.109 for fixed-length RL. This comparison changes both length modeling and the training objective, so its scope is system-level performance. The three comparisons jointly describe the contribution of reward fine-tuning, the quality-guided procedure, and the behavior of the full any-length system.

[Q2] What evidence shows that adaptive length contributes beyond the fine-tuning recipe?

The new paired length study in Table R1 examines synthetic peptides at verified residue counts of 4, 8, 12, and 16. Composition fractions are held fixed within each of 16 held-out motif families. Counts are verified from the amino-acid backbone, and validation families are separated from the test families. The reported uncertainty is across families, which is the appropriate unit for these paired comparisons. This provides a controlled view of how predicted properties vary with residue count.

The preferred length depends on the objective. Mean predicted affinity to both TfR and GLP-1R is highest at 16 residues, while solubility, non-hemolysis, and non-fouling are highest at 8, and permeability is highest at 4. These different optima motivate making length part of the optimization space for multi-objective design. The study reports property predictions and serves as motivation for adaptive length alongside the trained-model comparisons.

The peptide system comparisons complement that motivation. Head-free A2D2 improves TfR solubility and permeability relative to fixed-length off-policy RL. On GLP-1R, it improves solubility from 0.684 to 0.830 and non-fouling from 0.247 to 0.618. We distinguish this system-level evidence from a matched intervention on length alone. The language ablations and functional evaluations planned for the later revision will provide additional evidence on the behavior of variable-length language generation.


If we have addressed your concerns, we hope you will consider raising your score in support of acceptance, and if any remain, we are happy to continue the discussion!

Thank you,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to Reviewers (1/2)

We sincerely thank all three reviewers for their careful and constructive comments. We appreciate that the reviewers recognized that the problem of reward alignment for any-length discrete diffusion is relevant (zMjx, nb2p), the novelty and computational advantage of the method (CshR), and the generality demonstrated across modalities from the molecule, peptide, and language experiments (zMjx, CshR, nb2p).

The main questions concern the theoretical basis of AJD, the contribution of the quality mechanisms, the role of variable length, and the scope of the language evaluation. We have revised the manuscript to address these questions. We summarize the updates below and address the individual questions in the reviewer replies. Revised passages are highlighted in red\textcolor{red}{red} in the updated manuscript. The two tables below use the same R numbers as the revised manuscript. The original manuscript table numbers are unchanged.

Overview of Key Updates

  • AJD theory and optimality. The derivation now includes the parameter-dependent insertion exit rate and a positively weighted Poisson divergence. The appendix presents the optimal generator, reward-tilted path measure, and AJD population optimality result with explicit support, realizability, and initial-law assumptions.
  • Peptide length study (Table R1). A paired study across 16 held-out motif families examines verified lengths of 4, 8, 12, and 16 residues. Different properties favor different lengths, motivating length as an optimization variable.
  • Reward-guided proposals (Table R2). At matched reward-call budgets, guided proposals give the highest observed mean distilled-policy quality at all three budgets. Evaluation disables guidance, so the comparison measures the resulting trained policies.
  • Quality mechanisms and comparison scope. The revised text explains the distinct roles of deletion and remasking and distinguishes training-configuration ablations from inference diagnostics. The peptide results separately discuss reward fine-tuning, the quality-guided training-and-decoding combination, and comparison with fixed-length RL.
  • Language ablation with functional evaluation (Table R3). On the full GSM8K (1319 questions) and HumanEval single-line infilling (1033 examples) benchmarks at 256 sampling steps, quality-guided decoding improves GSM8K from 36.6% to 45.6% and code infilling from 69.3% to 72.2% pass@1, with code scored by executing the HumanEval unit tests.

Shared Tables

Table R1: Predicted properties of synthetic peptides at verified residue counts. Mean ± standard deviation across 16 held-out motif families, with composition fractions fixed within each family. Families 0–15 are reserved for validation and families 16–31 for testing. Residue counts are verified from the amino-acid backbone. Higher predictor values are preferred for all properties. Only affinity is target-conditioned.

Property4 residues8 residues12 residues16 residues
TfR affinity5.566 ± 1.5045.617 ± 1.0916.204 ± 1.4026.750 ± 1.329
GLP-1R affinity6.505 ± 1.6626.506 ± 1.0876.975 ± 1.1507.387 ± 1.031
Solubility0.822 ± 0.1080.870 ± 0.0730.844 ± 0.0940.793 ± 0.164
Non-hemolysis0.856 ± 0.0640.932 ± 0.0740.921 ± 0.0720.918 ± 0.059
Non-fouling0.359 ± 0.1750.465 ± 0.2220.378 ± 0.2140.241 ± 0.190
Permeability−6.724 ± 0.184−7.249 ± 0.289−7.407 ± 0.314−7.464 ± 0.265
AuthorsOpenReview ↗

Global Response to Reviewers (2/2)

Table R2: Distilled-policy quality (%) at matched reward-call budgets. Mean across five independent replay-bank and training seeds. Guided, uniform-branching, and all-candidate replay use identical distillation settings and equal reward budgets. Evaluation disables guidance and averages three evaluation seeds with 300 attempts per seed within each independent training run. Here quality counts attempts that are valid with QED ≥ 0.6 and SA ≤ 4.

Arm1,152 reward calls2,304 reward calls4,608 reward calls
Guided50.4450.9151.22
Uniform branching49.3846.6746.16
All-candidate replay46.3345.4744.78
Frozen pretrained, no training42.2342.2342.23

Citations

[1] Kim et al. “Any-Order Flexible Length Masked Diffusion.” ICLR 2026. (FlexMDM)

[2] Nisonoff et al. “Unlocking Guidance for Discrete State-Space Diffusion and Flow Models.” ICLR 2025.

[3] Venkatraman et al. “Amortizing Intractable Inference in Diffusion Models for Vision, Language, and Control.” NeurIPS 2024. (RTB)

[4] Su et al. “Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design.” arXiv:2507.00445, 2025. (VIDD)

[5] Havasi et al. “Edit Flows: Flow Matching with Edit Operations.” 2025.

AuthorsOpenReview ↗

Global Response to Reviewers: added quality-head ablation

Quality head ablation with functional evaluation. We retrained the quality fine-tuned A2D2 GSM8K and code-infilling models. Because the original checkpoints were lost, these models are not step-matched to those in Table 3, so absolute numbers differ from the paper. Both rows of the table below use the same backbone and decoder and differ only in whether the quality heads are active. All runs use the full test sets (1319 GSM8K / 1033 HumanEval single-line infilling), 256 sampling steps, and greedy decoding. As requested, code is scored by functional pass@1, executing the HumanEval unit tests, instead of exact match.

Quality-sampling improves GSM8K by 8.9 points and code infilling by 2.9 points. We will include the insertion-head and combined ablations in the final revision.

Table R3: Retrained quality-head ablation, 256 sampling steps. GSM8K: Pass@1 (%) on 1319 questions. HumanEval infilling: Pass@1 (%) on 1033 examples. (required retraining, so not step-matched to Table 3)

MethodGSM8KHumanEval Infilling
A2D2, no quality heads36.669.3
A2D245.672.2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.