- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes A2D2, a framework for finetuning discrete diffusion models with insertion and unmasking towards the reward-tilted distribution. The contribution mainly includes the adaptive joint decoding objective that uses importance-weighted denoising and insertion loss and the quality predictors that guide adaptive inference. Experiments on small molecule generation, peptide design, and language modeling show that A2D2 can improve over the baselines.
Critical strengths
-
The paper tackles the important problem of reward-tilted finetuning of adaptive length discrete diffusion models. Compared with fixed length discrete diffusion, optimizing adaptive length models yields more flexibility during inference.
-
Empirically, the reward-tilted finetuning seems to yield decent improvement over the pretrained adaptive length model, across several tasks and benchmarks.
Critical weaknesses
-
The paper claims theoretical optimality of the AJD objective. However, it may need more careful reconsiderations. See Q1 and Q2.
-
The contribution of the quality-based adaptive decoding is not fully supported by the experiment. Some parts of the experiments also raise concerns on the effectiveness of the proposed approach. See Q3 and Q4.
Questions for authors
Q1. The insertion rates are parameter-dependent. However, the exit-rate terms in general CTMC likelihood are omitted when deriving Eq. 125. Why is this allowed? If they cannot be omitted, then is the ADJ objective still provably optimal for minimizing the KL towards the reward-tilted target measure?
Q2. Algorithm 2 additionally includes quality-based remasking and deletion, while Proposition 4.1 does not consider these operations. How do these operations affect the importance weights? Does the derivation of the importance weights still hold?
Q3. Could the authors provide more explanations/analyses on why jointly using both predictors sometimes performs worse than using only one of them (e.g. see Table 1)?
Q4. Table 3 only reports the pretrained instruction-tuned baseline but not an any-length model funetuned with the AJD objective without quality predictors. It may be helpful to include this ablation.
Q5. It seems that Eq. 2 is minimizing the negative of the defined scalar Bregman divergence \phi, which in turn maximizes the divergence. Is this the intended behavior or a typo?
5 comments
Author Response To Reviewer nb2p (2/3)
[Q2] How do remasking and deletion affect the importance weights?
The revised proposition specifies exact importance accounting for the base insertion–unmasking CTMC. For matched reference and proposal initial laws, common schedules, and the stated support conditions, its log weight contains the terminal reward, the reference-to-proposal jump-rate ratios evaluated at pre-jump states, and the exit-rate compensator. The manuscript defines the unnormalized score without , so includes the normalizer once.
The implemented adaptive decoder uses the quality heads to guide remasking and deletion after policy warmup. The replay score evaluates the retained unmasking outcomes and post-deletion insertion counts using the policy and reference model probabilities. Subsequent policy evaluations condition on the resulting modified sequence states. The heads therefore influence both the generated trajectories and the outcomes entering the replay score. We accordingly describe this quality-guided replay weighting as approximate and distinguish it from the exact importance-accounting result for the specified base insertion–unmasking CTMC.
The appendix now presents these statements as follows: the base-process path density, the optimal reference tilt, the insertion–unmasking likelihood ratio, and the conditional AJD endpoint result. This makes clear which proposal the exact identity concerns and which procedure is evaluated in the experiments. The auxiliary quality losses train reconstruction scores, with the trained configurations and the frozen-checkpoint diagnostic providing complementary empirical views of the adaptive decoder.
[Q3] Why can both heads perform worse than either head alone?
The heads supply distinct controls over the same evolving sequence. Remasking revisits the identities of decoded tokens, while deletion rejects newly inserted mask positions. Removing positions changes the sequence context, length, and mask count used by subsequent remasking decisions. This interaction between remasking and deletion motivates analyzing the joint decoder as a coupled procedure and evaluating its configurations empirically.
The submitted training ablations describe task-dependent reward–coverage operating points. On molecules, insertion quality alone attains the highest A2D2 quality at 82.37% and QED at 0.818. Both heads attain the highest diversity among the A2D2 quality variants at 0.843, together with uniqueness of 93.695%. On peptides, the full quality-guided procedure raises TfR validity from 41.27% to 48.60% relative to the head-free model.
The quality configurations can therefore be chosen according to the application's objectives. The revised Results section describes this reward-coverage trade-off. Notably, every configuration with at least one quality head improves uniqueness and diversity over the head-free model in the submitted molecule training ablations, supporting quality prediction as a way to preserve coverage during reward fine-tuning. These trained configurations provide different operating points for reward and diversity.
Author Response To Reviewer nb2p (3/3)
[Q4] Can you include AJD without quality predictors in the language comparison?
Yes. Reproducing the necessary checkpoints for our adapted any-length model and its instruction-fine-tuned version requires costly retraining due to lost checkpoints, so we are unable to provide this ablation and the requested functional evaluations at this time. We will add the full language ablations and functional results in a later revision. The current language comparison evaluates the full A2D2 procedure against instruction fine-tuning on the same any-length backbone under matched sampling-step budgets.
[Q5] Does the negative sign in Eq. 2 maximize the insertion divergence?
Yes, thank you for catching this typo! The insertion loss is now written with a positive coefficient, , multiplying . This is the positively weighted divergence used in the minimized training objective. We use and the extended-value convention when . The sign and divergence notation are consistent across the main text, background, and training algorithm.
For a fixed target count , differs from by a target-dependent constant. The gradients and minimizer are therefore the same. For a positive predicted count, the derivative is , which raises an underpredicted positive count and lowers a positive prediction when the target count is zero. The written correction thus agrees with the trained objective and preserves the trained models.
[W1] The AJD optimality claim needs reconsideration.
The revised theory explicitly separates exact base-process importance accounting, AJD population endpoint optimality, and the quality-guided decoding implementation. The optimal-generator and path-measure lemmas give the full reward-tilt derivation with waiting times and initialization stated. The AJD proposition then establishes its endpoint result through exact weighting and conditional-statistic recovery under realizability and forward-equation assumptions. This retains the optimality result while specifying the additional generator and bridge identification needed for full path recovery. Our replies to Q1a, Q1b, Q2, and Q5 give the corresponding details.
[W2] The quality-based decoder needs stronger empirical support.
The revised discussion connects the insertion-quality bound to a reconstruction surrogate, explains how deletion and remasking act on different objects, and interprets the quality configurations through the submitted training ablations. It also separates those comparisons from the frozen-checkpoint inference diagnostic. Our reply to Q3 describes the reward–coverage operating points, and our reply to CshR W2 explains the deletion mechanism and operation-order study. The head-free language training ablation will be included as soon as possible.
If we have addressed your concerns, we hope you will consider raising your score in support of acceptance, and if any remain, we are happy to continue the discussion!
Thank you,
The Authors
Author Response To Reviewer nb2p (1/3)
Dear Reviewer nb2p,
Thank you for the careful and technically precise review. We are glad you recognized the importance of reward-based fine-tuning for adaptive-length discrete diffusion, the inference flexibility it provides, and the empirical improvements over the pretrained model across tasks.
We hope to clear up your concerns with the additional experiments and responses below:
You are correct that the exit-rate term cannot be omitted. This was an error in our writing rather than an implementation error. For generator Atθ with exit rate λtθ(x)=∑y=xAtθ(x,y), the path negative log density is
−logP0θ(X0)−∑jlogAtjθ(Xtj−,Xtj)+∫01λtθ(Xt)dt.
The normalized unmasking posterior has total exit rate independent of θ, so that contribution drops out. The insertion exit rate depends on gθ and must be retained, and it supplies exactly the positive predicted-rate contribution that accompanies the insertion jump term. Concretely, the insertion loss in our implementation computes DPois(x∥g)=g−x+xlog(x/g), where x is the realized gap count and g=gθ(xt,t)[ℓ] the predicted rate, and adds it with a positive weight into the total training loss. The leading g is exactly the compensator, and the orientation is minimizing. The omission was therefore a small error in our write-up, which we have now fixed: the correction aligns the written derivation with the objective the models were actually trained under, and leaves the trained models themselves unchanged.
Yes, it is optimal. We now show a complementary conditional endpoint-consistency result. Write ptarget(x)=Z−1ppre(x)er(x)/α and let Q be the actual proposal on a common supported path space. With exact weight w=dP⋆/dQ and fresh corruption randomness included in ℓθ, EQ[wℓθ(X1)]=Ex∼ptarget[ℓθ(x)]. At an unrestricted population optimum, cross-entropy fits the target conditional token probabilities and the positive Poisson divergence fits the conditional expected gap counts. If these statistics define the chosen interpolant's generator and are realizable, matched initialization and uniqueness of the forward equation give endpoint law ptarget.
The endpoint law is the unique optimum of the KL-regularized reward objective because Ex∼p[r(x)]−αDKL(p∥ppre)=αlogZ−αDKL(p∥ptarget). The appendix separately derives the optimal path law and generator. With ht(x)=EPpre[er(X1)/α∣Xt=x], the transformed rates are At⋆(x,y)=Atpre(x,y)ht(y)/ht(x). The derivation includes the reward scaling, initial-law condition, and waiting-time compensator. For the deterministic empty initialization, h0(x0)=Z and the initial law is unchanged. The quality predictors learn auxiliary reconstruction statistics for the adaptive decoder. We state their role separately from the base-process path guarantee and evaluate the resulting quality-guided procedure empirically.