acceptodds
Under review as a conference paper at ICLR 2027

Backpropagating Verifier Rewards Through Continuous Flow Language Models

Abstract

Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose CFRL, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, CFRL reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.

Primary area:
generative models
Keywords:
NLP, flow matching, credit assignment, verifiable reasoning, diffusion language models, post-training, self-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oqagOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning for continuous-flow language models and proposes CFRL, which backpropagates verifier rewards through a differentiable flow transition. The paper develops a score-function baseline and two stochastic value-gradient variants: a sequence-level critic and a token-wise critic. The critic reuses a frozen denoising backbone and is refit online, while the policy receives a pathwise gradient through one flow step. On Countdown, Sudoku, Knights-and-Knaves, and Game24, the token-wise variant improves over supervised continuous-flow baselines and the other CFRL variants. The paper further analyzes gradient variance, critic architecture and refresh frequency, sampler choices, and a self-training loop that amortizes the improved samples into cheaper inference.

Critical strengths
  • The pathwise-gradient formulation is a meaningful and well-reasoned contribution for continuous-flow language models. The paper identifies a concrete property that distinguishes continuous flows from discrete autoregressive generation and uses it to replace a high-variance score-function update with backpropagation through an intermediate state and a learned value function.
  • Several experiments directly test the proposed reasoning rather than only reporting endpoint accuracy. The half-split analysis measures gradient reproducibility, the one-step intervention tests whether the estimated direction improves reward, the frozen-critic experiment demonstrates critic exploitation, and swapping the 1D and 2D critics during training shows that the critic architecture causally changes the subsequent optimization trajectory.
Critical weaknesses
  • The evidence does not establish that the 2D critic performs token-level credit assignment in the semantic sense claimed by the paper. Every position is trained against the same sequence-level binary label. The critic swap convincingly shows that the 2D head changes joint training, but it does not identify why. In fact, the fixed-policy probes show similar reward discrimination and highly aligned update directions for the 1D and 2D critics. The gain could therefore come from gradient scaling, conditioning, or optimization dynamics rather than identifying which token caused the failure.
  • The empirical evidence is limited to one 105M continuous-flow model that is separately fine-tuned for four synthetic tasks with exact verifiers. This is sufficient to demonstrate that CFRL can work, but not that the approach remains stable or computationally worthwhile at language-model scale. The autoregressive rows are also not matched in pretraining, model size, or task-specific adaptation, so they should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL.
Questions for authors
  1. Can the authors evaluate the 2D readings on examples with known corrupted positions and test whether changing the highest-scored positions has a larger counterfactual effect than changing other positions? This would distinguish token-level credit assignment from a position-wise parameterization of a sequence-level value.
  2. What is the total training cost of CFRL, including critic pretraining, fresh rollouts, and repeated critic refitting, relative to CFRL-GRR? A compute-matched comparison would clarify whether the improvement comes from the estimator or from substantially more learned supervision.
6 comments
AuthorsOpenReview ↗

Response to Reviewer oqag

We thank the reviewer for the positive assessment and thoughtful feedback. We particularly appreciate the recognition of our pathwise-gradient formulation as a meaningful contribution to continuous-flow language models, as well as their acknowledgment that our gradient reproducibility, single-step intervention, frozen-critic, and critic-swap experiments provide direct evidence supporting our methodology. We also appreciate the reviewer's constructive suggestions regarding token-level credit assignment and computational efficiency. We address these concerns below and provide additional experiments and analyses to further strengthen our empirical findings.

The evidence does not establish that the 2D critic performs token-level credit assignment in the semantic sense claimed by the paper. Every position is trained against the same sequence-level binary label. The gain could come from gradient scaling, conditioning, or optimization dynamics rather than identifying which token caused the failure. Can the authors evaluate the 2D readings on examples with known corrupted positions and test whether changing the highest-scored positions has a larger counterfactual effect than changing other positions? This would distinguish token-level credit assignment from a position-wise parameterization of a sequence-level value.

We thank the reviewer for this insightful suggestion. We agree that supervising each token-wise critic output with the same sequence-level reward does not guarantee semantic token-level credit assignment. To investigate this, we conducted two additional experiments examining whether the 2D critic identifies incorrect token positions and whether its token-wise readings respond to targeted perturbations.

(1) Correlation between token-wise readings and incorrect positions. We first evaluate whether the 2D critic assigns lower readings to incorrect token positions in failed Sudoku solutions. We focus on Sudoku because each output position corresponds directly to a specific answer cell, enabling unambiguous identification of incorrect tokens. For each failed response, we take the single token with the lowest reading and record 1 if that token is actually wrong and 0 otherwise; Precision is the average over all failed responses. The baseline is the fraction of wrong tokens in the response, i.e. the probability that a randomly chosen position is a wrong token. This table can be found in Table 14 in the revised manuscript.

Metricrange over K steps
Precision0.62–0.65
Random baseline precision0.42

The precision exceeds random performance. This suggests that the 2D critic's readings contain information about which token positions are incorrect, despite being trained with sequence-level supervision.

(2) Counterfactual token corruption. Following the reviewer's suggestion, we conduct a controlled perturbation experiment to evaluate whether the 2D critic responds to token-level errors. Starting from a correct solution, we corrupt a single token and measure the change in each position's critic reading. We report the percentage of cases in which the corrupted position exhibits the largest decrease in its critic reading among all positions, compared with the random-position baseline.

TaskCorrupted position with largest reading decreaseRandom baseline
Sudoku11–22%1.2%
K&K23–30%6.5%
Countdown8-61%3.7%
Game2413–26%5.6%

Across all four tasks, the corrupted position is substantially more likely to exhibit the largest decrease in critic reading than expected by chance. An incorrect answer token can be sufficient to establish sequence-level failure, whereas a correct token alone cannot establish success. This asymmetry motivates position-sensitive critic readings even under sequence-level supervision. These results demonstrate that the 2D critic is sensitive to localized token perturbations, providing evidence of position-specific reward attribution despite being trained with sequence-level supervision. This table can be found in Table 15 in the revised manuscript.

Our experiments show that the critic captures position-specific information and responds to localized corruption, while they do not establish that semantic token-level credit assignment is the sole source of its advantage and we do not rule out the possibility that the 2D critic's performance gains may also arise from gradient scaling, conditioning, or optimization dynamics. We have therefore added this clarification in the revised manuscript (E.3).

AuthorsOpenReview ↗

Response to Reviewer oqag

The empirical evidence is limited to one 105M continuous-flow model that is separately fine-tuned for four synthetic tasks with exact verifiers. This is sufficient to demonstrate that CFRL can work, but not that the approach remains stable or computationally worthwhile at language-model scale. The autoregressive rows are also not matched in pretraining, model size, or task-specific adaptation, so they should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL.

We thank the reviewer for highlighting the importance of scalability and fair comparisons. We agree that our autoregressive (AR) baselines, which differ in pretraining, model size, and task-specific adaptation, should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL. The motivation of this work is to explore the potential of continuous language models via post-training instead of competing with existing AR models.

To investigate whether CFRL's improvements extend beyond the 105M-parameter ELF-B model, we conducted additional experiments on ELF-L (652M parameters) (Table 2 in the revised manuscript), approximately 6.2× larger than ELF-B. We applied the same critic pre-training and RL hyperparameters without additional tuning, using the same validation-based checkpoint selection and averaging test accuracy over five sampling seeds.

Test accuracy (%) on ELF-L (652M). Improvements over task-SFT are shown in parentheses.

TaskSFT BaseCFRL-GRRCFRL-SVG-1DCFRL-SVG-2D
Countdown60.563.3 (+2.8)72.6 (+12.1)74.4 (+13.9)
Sudoku71.070.3 (-0.7)79.3 (+8.3)77.9 (+6.9)
Game2458.666.9 (+8.3)70.5 (+11.9)71.0 (+12.4)
K&K100.0–––

CFRL-SVG consistently improves over the SFT baseline and CFRL-GRR across all three non-saturated tasks. In particular, SVG-2D achieves improvements of +13.9, +6.9, and +12.4 percentage points on Countdown, Sudoku, and Game24, respectively. These gains are comparable to or greater than those observed on ELF-B, demonstrating that CFRL's effectiveness extends to a substantially larger continuous-flow model.

While these experiments provide encouraging evidence of scalability from 105M to 652M parameters, they do not establish stability or computational efficiency at multi-billion-parameter scale. We leave the further scaling studies as future work.

AuthorsOpenReview ↗

Response to Reviewer oqag

What is the total training cost of CFRL, including critic pretraining, fresh rollouts, and repeated critic refitting, relative to CFRL-GRR? A compute-matched comparison would clarify whether the improvement comes from the estimator or from substantially more learned supervision.

We thank the reviewer for raising this question about compute matching. We added total training FLOPs across all network evaluations in the revised manuscript, including rollouts, actor updates, critic pre-training, and online critic updates. Following the standard parameter-based FLOPs approximation ([1]), we count a forward pass as one unit and a full backward pass as two units. We additionally approximate an input-gradient-only backward pass as one unit, since the critic backbone is frozen during RL.

As shown in the new Table below (table 10 in the revised manuscript), CFRL-SVG-2D consistently outperforms CFRL-GRR at matched total FLOPs across all four tasks, even after accounting for the full critic cost. For fair comparison, we additionally trained CFRL-GRR using the same deployment sampler as CFRL-SVG (64 steps, or 32 for Sudoku), rather than its original 16-step sampler. This approximately doubles GRR's computational cost while keeping all other training settings unchanged. And we also added figure 3 in the revised manuscript to show Acc vs Flops across four datasets on three variants of CFRL. It also shows that with the comparable compute, CFRL-SVG outperforms CFRL-GRR consistently. For example, at approximately 200 PFLOPs on Countdown, SVG-2D achieves 49.6% accuracy versus 43.0% for GRR; at approximately 100 PFLOPs on Game24, SVG-2D achieves 56.4% versus 46.3% for GRR with the deployment sampler. Similar improvements hold on Sudoku and K&K.

Test accuracy (%) and total training compute (PFLOPs). The GRR 64-sampler row is newly evaluated; all other accuracies are from Table 1. Critic-stage costs are included in the SVG totals.

MethodCountdown Acc.PFLOPsSudoku Acc.PFLOPsK&K Acc.PFLOPsGame24 Acc.PFLOPs
Task-SFT base42.8064.0068.2044.00
CFRL-GRR (γ4@16)42.69767.922171.413047.497
CFRL-GRR (γ56@64; γ28@32 Sudoku), new42.820864.230677.027847.3208
CFRL-SVG-1D47.426364.126976.527751.2232
CFRL-SVG-2D49.726368.726977.527758.2232
Critic stage (included in SVG)–67–113–103–36

The new compute-matched experiments show that this advantage persists after accounting for the critic's cost and matching the rollout sampler. We have incorporated the detailed FLOPs accounting into Appendix D.

We sincerely thank the reviewer for their time and thoughtful feedback, which have helped us strengthen the paper and improve its clarity and rigor. We have incorporated these additional experiments and clarifications into the revised manuscript, with all changes highlighted in blue.

[1] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

Reviewer_6mKEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

By refining continuous token embeddings, continuous flow language models make generation differentiable in principle, but the final discrete decoding and verification steps block gradient flow. To bridge this gap, the paper proposes CFRL, which learns a critic that predicts the final verifier outcome from intermediate states and uses its gradient to update the policy through one denoising transition. The paper compares pooled and per-position critic heads, with the latter improving task accuracy by 4.7–14.2 percentage points over SFT on ELF-B. It also studies critic adaptation and self-training to transfer these gains to cheaper samplers.

Critical strengths
  1. The critic enables single-transition policy updates without backpropagating through the full denoising trajectory.

  2. The single-step tests, critic swaps, and frozen-critic experiments provide useful insights into optimization. The experimental section is clearly presented.

Critical weaknesses
  1. Eq. (7) maximizes VϕV_\phi, while Table 3 uses logit⁡(Vϕ)\operatorname{logit}(V_\phi). Eq. (10) sums position logits, whereas Appendix B.4 assumes mean pooling. These discrepancies affect the method's specification and theoretical justification.

  2. The pathwise update includes a self-conditioning gradient absent from the REINFORCE baseline (Appendix B.3). Together with possible critic bias, this prevents attributing the gains to variance reduction alone.

Questions for authors
  1. Do Tables 1–3 use VϕV_\phi or logit⁡(Vϕ)\operatorname{logit}(V_\phi) as the actor objective, and a sum or mean of position logits for the critic? Which analysis applies to the implemented version?

  2. How much of the pathwise improvement remains when the self-conditioning gradient is detached?

  3. Does the Game24 verifier require every input number to be used exactly once? For example, would it reject 3*8=24 for inputs [1,2,3,8]? Appendix A only describes at-most-once use for the reused Countdown verifier.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6mKE

We thank the reviewer for their thoughtful and constructive feedback and for recognizing the strengths of CFRL, particularly its ability to enable single-transition policy updates without backpropagating through the full denoising trajectory. We also appreciate the reviewer's positive assessment of our experimental analysis, including the single-step tests, critic swaps, and frozen-critic experiments. We acknowledge the concerns regarding the consistency of our method specification, gradient estimation, and verifier implementation, and address each point below.

Eq. (7) maximizes VϕV_{\phi}, while Table 3 uses logit(Vϕ)\mathrm{logit}(V_{\phi}). Eq. (10) sums position logits, whereas Appendix B.4 assumes mean pooling. These discrepancies affect the method's specification and theoretical justification.

We thank the reviewer for raising this question. In the actor's training objective, Eq. (7) maximizes the expected reward E[Vϕ]\mathbb{E}[V_\phi]. Given that Vϕ(sk)≈P(R=1∣sk)V_\phi(s_k)\approx P(R=1\mid s_k), the actor is trained to maximize the expected reward. In our SVG implementation, we use E[logit(Vϕ)]\mathbb{E}[\mathrm{logit}(V_{\phi})] instead. We have added this clarification in the revised manuscript after Eq. (7). This design encourages the actor to stay away from wrong answers, even when its output is correct. It shares the same spirit as MaxRL [1].

Maximizing the logit of VϕV_\phi encourages the generator to move towards states leading to successful trajectories. Furthermore, the two objectives give the same per-sample gradient direction, since ∇θlogit(Vϕ)=∇θVϕ/(Vϕ(1−Vϕ))\nabla_\theta\mathrm{logit}(V_\phi)=\nabla_\theta V_\phi/\big(V_\phi(1-V_\phi)\big). However, directly maximizing VϕV_\phi introduces a multiplicative factor Vϕ(1−Vϕ)V_\phi(1-V_\phi) in the pathwise gradient, which attenuates the learning signal when the critic confidently predicts success or failure. The logit transformation removes this sigmoid-induced attenuation while preserving the per-sample gradient direction, providing a stronger optimization signal, particularly for samples predicted to fail with high confidence. Table 5 (original Table 3) confirms the effect: ascending VϕV_\phi instead lowers accuracy. We have added this in Appendix B.4 in the revised manuscript.

For the pooling method over positions, we used mean pooling in the implementation. The sum in Eq. (10) was a typo and we have corrected Eq. (10) in the revised manuscript. We thank the reviewer for the careful reading.

Does the Game24 verifier require every input number to be used exactly once? For example, would it reject 3*8=24 for inputs [1,2,3,8]? Appendix A only describes at-most-once use for the reused Countdown verifier.

Yes, the shared verifier for both Game24 and Countdown requires every input number to be used exactly once and the resulting expression to evaluate to the target. Thus, for inputs [1, 2, 3, 8], it rejects 3*8=24 because 1 and 2 are unused. The “at-most-once” description is incomplete. We apologize for the ambiguity and have revised Appendix A to state the full acceptance criteria explicitly.

AuthorsOpenReview ↗

Response to Reviewer 6mKE

The pathwise update includes a self-conditioning gradient absent from the REINFORCE baseline (Appendix B.3). Together with possible critic bias, this prevents attributing the gains to variance reduction alone. How much of the pathwise improvement remains when the self-conditioning gradient is detached?

We thank the reviewer for this insightful observation. We acknowledge that GRR retains the REINFORCE gradient computed through one denoising step but does not account the gradient from the computation path of deterministic self-conditioning.

In our new version, we clarify the action definition to be ak=ηkxtk+ψkϵa_k = \eta_k x_{t_{k}} + \psi_k \epsilon and then update our derivation in the submission as follows. The optimization of the actor's objective JGRR(θ)\mathcal J_{\mathrm{GRR}}(\theta) (Eq. 5) requires the computation of the gradient ∇θlog⁡πθ(akg∣skg)\nabla_\theta \log\pi_\theta\bigl(a^g_k \mid s^g_{k}\bigr) for each group gg. Below we omit gg for notation simplicity. Note that aka_k is from a Gaussian distribution N(ηkxtk,ψk2cσ2I)\mathcal{N}(\eta_k x_{t_{k}}, \psi_k^2 c_\sigma^2 I), and we have:

∇θlog⁡πθ(ak∣sk)=ηk(ak−ηkxtk)⊤ψk2cσ2(∂xtk∂θ+∂xtk∂xtk−1⋅Dθxtk−1)≈ηk(ak−ηkxtk)⊤ψk2cσ2∂xtk∂θ.\nabla_\theta \log\pi_\theta\bigl(a_k \mid s_{k}\bigr) = \frac{\eta_k(a_k - \eta_k x_{t_{k}})^\top }{\psi_{k}^2 c_\sigma^2} \left( \frac{\partial x_{t_{k}}}{\partial \theta} + \quad \frac{\partial x_{t_{k}}}{\partial x_{t_{k-1}}} \cdot D_{\theta} x_{t_{k-1}} \right) \approx \frac{\eta_k(a_k - \eta_k x_{t_{k}})^\top }{\psi_{k}^2 c_\sigma^2}\frac{\partial x_{t_{k}}}{\partial \theta}.

There is a deterministic computation path from xt0x_{t_0} to xtkx_{t_{k}} through the self-conditioning mechanism in the denoiser. The term Dθxtk−1D_\theta x_{t_{k-1}} is the total derivative of xtk−1x_{t_{k-1}} with respect to θ\theta through the path. The exact gradient calculation requires backpropagation through denoising steps, which has the same issue as backpropagation through time. In our implementation, we truncate the backpropagation through steps and only use the local gradient ∂xtk∂θ\frac{\partial x_{t_{k}}}{\partial \theta} as the approximate gradient.

From this derivation, we agree that the gradient of GRR is biased, and computing the unbiased estimation is computationally hard. Similar gradient approximations have been adopted in diffusion-model optimization: DPOK ([2], Sec. 4.1 and App. A.3) omits gradients through earlier denoising transitions for efficiency, while DRaFT-K ([3]) truncates backpropagation to the final K sampling steps.

Therefore, bias in gradient estimation is a weakness of GRR. In contrast, SVG estimates the additional self-conditioning gradient using a differentiable critic, avoiding the need to backpropagate through the entire sampling trajectory. Consequently, SVG's improvement over GRR may reflect its ability to reduce both bias and variance in gradient estimation. To isolate the contribution of gradient estimation to performance, we conduct two additional experiments. First, we remove the second gradient term in Eq. (28) from SVG to match the gradient formulation of GRR. Second, we remove self-conditioning entirely for both GRR and SVG, under which the GRR gradient estimator becomes unbiased.

The table below shows the results from the experiment that removes the second gradient term in Eq. (28) from SVG.

TaskBaseGRRSVG-2D (Detached x^\hat{x})SVG-2D (Full)
Countdown41.842.642.749.2
Game2444.546.752.758.4

Detaching the self-conditioning gradient reduces SVG-2D accuracy from 49.2% to 42.7% on Countdown and from 58.4% to 52.7% on Game24. The detached variant performs comparably to GRR on Countdown but retains a substantial advantage on Game24 (+6.0 percentage points).

The table below shows the experiment omitting the self-conditional at all for both GRR and SVG for a fair comparison. For this experiment, we retrain the task-SFT base of ELF-B with self-conditioning disabled, with the same supervised budget as Table 1, and rerun the whole pipeline without self-conditioning: rollout collection, critic pre-training, RL and evaluation. The base drops by about 30 points, so self-conditioning matters greatly for the model, yet SVG-2D still leads GRR by +13.5+13.5 and +7.8+7.8.

TaskBaseGRRSVG-2D
Countdown12.713.426.9
Game2413.615.323.1

These results suggest that self-conditioning is important for overall model performance, while the SVG retains its effectiveness even in its absence. The SVG gain thus comes from both reduced variance and reduced bias. The tables can be found in Table 3 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 6mKE

We sincerely thank the reviewer for their time, thoughtful feedback, and constructive suggestions. We have carefully addressed the concerns regarding methodological consistency, gradient estimation, and verifier implementation, and revised the manuscript accordingly, with all changes highlighted in blue. We believe these revisions have improved the clarity and rigor of our manuscript.

[1] Tajwar, F., Zeng, G., Zhou, Y., Song, Y., Arora, D., Jiang, Y., Schneider, J., Salakhutdinov, R., Feng, H., & Zanette, A. ICML 2026.

[2] Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. NeurIPS 2023.

[3] Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Directly Fine-Tuning Diffusion Models on Differentiable Rewards. ICLR 2024.

Dear Reviewer 6mKE,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_oAZUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CFRL for reinforcement learning post-training of continuous flow language models. A learned critic predicts terminal verifier outcomes from intermediate denoising states, allowing policy updates through a single differentiable transition. The proposed 2D critic applies a shared prediction head to individual token representations and supervises these predictions using the sequence-level reward. Experiments on four verifiable reasoning tasks show improvements over task-specific SFT and the evaluated score-function baselines.

Critical strengths
  • A useful application of pathwise policy optimization to continuous language generation. The framework exploits differentiable denoising transitions while accommodating a non-differentiable decoder and verifier. The gradient agreement and single-update experiments provide useful evidence that the learned critic supplies actionable optimization directions.
Critical weaknesses
  • The within-ELF comparison lacks compute matching and clear attribution. Appendix D reports 4–6M rollout denoiser calls for CFRL-SVG versus 1.2M for CFRL-GRR (about 2M for SVG on Sudoku), with additional critic pre-training and online computation. The same-rollout gradient experiments support update quality but do not establish performance under equal total compute. A budget-matched comparison including critic costs is needed.
Questions for authors
  • The unusually weak LLaDA baseline. In Table 1, after task-specific SFT, LLaDA-8B achieves only 12.1/0.5/61.4/15.9 on Countdown/Sudoku/K&K/Game24, compared with 42.8/64.0/68.2/44.0 for ELF-B with a 105M-parameter denoiser or GPT-2 performance. Is the LLaDA baseline fairly trained and evaluated? I think if the SFT/RL is well-tuned, LLaDA should have better performance and can be viewed as somewhat a upper bound given the training data.
5 comments
AuthorsOpenReview ↗

Response to Reviewer oAZU

We thank the reviewer for the positive assessment of our work and for recognizing the strengths of CFRL, particularly its application of pathwise policy optimization to continuous language generation and its ability to accommodate non-differentiable decoding and verification. We also appreciate the reviewer's acknowledgment that our gradient agreement and single-update experiments provide evidence of the critic's ability to guide effective policy optimization. We address the two concerns below.

The within-ELF comparison lacks compute matching and clear attribution. Appendix D reports 4–6M rollout denoiser calls for CFRL-SVG versus 1.2M for CFRL-GRR (about 2M for SVG on Sudoku), with additional critic pre-training and online computation. The same-rollout gradient experiments support update quality but do not establish performance under equal total compute. A budget-matched comparison including critic costs is needed.

Compute accounting. Appendix D previously reported only rollout denoiser calls, which did not fully capture the computational costs. We have now added a comprehensive FLOPs accounting and compute-matched comparisons that include critic pre-training and online computation. We measure total training FLOPs across all network evaluations, including rollouts, actor updates, critic pre-training, and online critic updates. Following the standard parameter-based FLOPs approximation ([1]), we count a forward pass as one unit and a full backward pass as two units. We additionally approximate an input-gradient-only backward pass as one unit, since the critic backbone is frozen during RL. The full-run Table1 compute of CFRL-SVG is 1.2–2.7× that of the original CFRL-GRR, rather than the 3–5× suggested by rollout counts alone. The critic stage accounts for 14–42% of the total SVG compute.

Compute-matched comparison. We additionally trained CFRL-GRR using the same deployment sampler as CFRL-SVG (64 steps, or 32 for Sudoku), rather than its original 16-step sampler. This approximately doubles GRR's computational cost to match the compute as SVG. As shown in the Table below (table 10 in the revised manuscript), CFRL-SVG-2D consistently outperforms CFRL-GRR at matched total FLOPs across all four tasks, even after accounting for the full critic cost. We also added figure 3 in the paper to show Acc vs Flops across four datasets on three variants of CFRL. It also shows that with the comparable compute, CFRL-SVG outperforms CFRL-GRR consistently. For example, at approximately 200 PFLOPs on Countdown, SVG-2D achieves 49.6% accuracy versus 43.0% for GRR; at approximately 100 PFLOPs on Game24, SVG-2D achieves 56.4% versus 46.3% for GRR with the deployment sampler. Similar improvements hold on Sudoku and K&K.

Test accuracy (%) and total training compute (PFLOPs). The GRR 64-sampler row is newly evaluated; all other accuracies are from Table 1. Critic-stage costs are included in the SVG totals.

MethodCountdown Acc.PFLOPsSudoku Acc.PFLOPsK&K Acc.PFLOPsGame24 Acc.PFLOPs
Task-SFT base42.8064.0068.2044.00
CFRL-GRR (γ4@16)42.69767.922171.413047.497
CFRL-GRR (γ56@64; γ28@32 Sudoku), new42.820864.230677.027847.3208
CFRL-SVG-1D47.426364.126976.527751.2232
CFRL-SVG-2D49.726368.726977.527758.2232
Critic stage (included in SVG)–67–113–103–36

Our existing same-rollout experiments of Table 1 (right panels) demonstrate that the pathwise estimator provides more effective updates under identical rollout conditions. The new compute-matched experiments further show that this advantage persists after accounting for the critic's cost and matching the rollout sampler. Together, these results support that the performance gains arise from the improved optimization signal rather than simply greater computational expenditure. We have incorporated the detailed FLOPs accounting into Appendix D.

AuthorsOpenReview ↗

Response to Reviewer oAZU

The unusually weak LLaDA baseline. In Table 1, after task-specific SFT, LLaDA-8B achieves only 12.1/0.5/61.4/15.9 on Countdown/Sudoku/K&K/Game24, compared with 42.8/64.0/68.2/44.0 for ELF-B with a 105M-parameter denoiser or GPT-2 performance. Is the LLaDA baseline fairly trained and evaluated? I think if the SFT/RL is well-tuned, LLaDA should have better performance and can be viewed as somewhat a upper bound given the training data.

We thank the reviewer for raising this concern. The relatively low LLaDA accuracy reflects the greater difficulty of our benchmarks compared with d1's[2] original tasks: Countdown uses 4–5 numbers instead of 3, Sudoku uses 9×9 grids instead of 4×4, and solutions require programmatically verified derivations. For reference, d1 [2] reports 42.2% on 3-number Countdown and 22.1% on 4×4 Sudoku with LLaDA-8B after SFT and RL in their original paper. We do not claim that these results represent the performance ceiling of masked diffusion models, nor do we claim continuous flow models perform better than masked diffusion models. To further address the reviewer's concern, we evaluated CFRL on d1's original Countdown-3 and 4×4 Sudoku test sets (256 problems each), using d1's own reported LLaDA results as references. We applied the same critic pre-training and RL procedures as in Table 1. Checkpoints were selected using separate validation data, and accuracies were averaged over 5 sampler seeds.

Test accuracy (%) on d1's original benchmarks.

Method4×4 SudokuCountdown (3 numbers)
LLaDA-8B-Instruct (d1)11.720.7
d1-LLaDA: SFT + diffu-GRPO (d1)22.142.2
ELF-B task-SFT base (105M, data-limited)49.241.6
+ CFRL-GRR58.943.7
+ CFRL-SVG-1D95.961.4
+ CFRL-SVG-2D96.162.8

On 4×4 Sudoku, CFRL-SVG and CFRL-GRR improves the ELF-B baseline from 49.2% to approximately 96% and 59% respectively, compared to d1-LLaDA that improves the baseline from 11.7% to 22.1%. On Countdown-3, CFRL-SVG and CFRL-GRR improves the ELF-B baseline from 41.6% to approximately 63% and 43.7% respectively, compared to d1-LLaDA that improves the baseline from 20.7% to 42.2%.

Our LLaDA baseline in Table 1 follows d1's released code and hyperparameters. All models use the same exact-success metric and validation-based checkpoint selection.We have added these experiments and clarifications to the revised manuscript (Table 9).

Thanks for your valuable comments and suggestions, which helped in improving the manuscript. We have revised the manuscript with changes highlighted in blue. We thank the reviewer again for your constructive feedback. The additional compute accounting and external benchmark evaluation strengthen our empirical analysis and clarify the scope of our conclusions.

[1] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.

[2] Zhao, S., Devaansh G., Qinqing Z., and Aditya G., d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning. NeurIPS 2025

Dear Reviewer oAZU,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response for Manuscript Revision

We sincerely thank all reviewers for their thoughtful feedback, constructive suggestions, and recognition of our contributions. We have substantially strengthened the manuscript through additional experiments, theoretical clarifications, and more comprehensive evaluations. All revisions are highlighted in blue and summarized below:

  • Compute-Matched Comparisons: We introduced comprehensive FLOPs accounting covering rollout collection, actor updates, critic pre-training, and online critic refitting. We additionally trained CFRL-GRR using the same deployment sampler as CFRL-SVG and evaluated accuracy against cumulative training FLOPs, demonstrating that SVG-2D maintains its advantage at matched computational budgets.

  • Self-Conditioning Gradient Analysis: We expanded the theoretical analysis of GRR gradients, showing that accounting for self-conditioning requires backpropagation through the denoising trajectory, which is computationally expensive and therefore omitted in GRR, introducing bias. In contrast, SVG efficiently incorporates the self-conditioning gradient through a single transition. To isolate its contribution, we conducted two additional ablations: detaching the self-conditioning gradient in SVG and removing self-conditioning entirely from both methods. The results help disentangle the contributions of reduced bias and variance in SVG relative to GRR and demonstrate that SVG retains substantial advantages over GRR even without self-conditioning.

  • Actor Objective Clarification: We clarified the actor objective. Eq. (7) maximizes the expected reward E[Vϕ]\mathbb{E}[V_\phi] and our implementation uses logit(Vϕ)\mathrm{logit}(V_{\phi}) instead. We added this clarification in the main paper and theoretical justification in the appendix.

  • Token-Level Credit Assignment Analysis: We conducted two additional experiments evaluating whether the 2D critic captures position-specific information. We measured the correlation between token-wise readings and incorrect Sudoku positions and introduced controlled token-corruption tests across all four tasks. The results demonstrate sensitivity to localized errors while clarifying the distinction between position-sensitive reward estimation and semantic token-level credit assignment.

  • Scalability to Larger Models: We extended our experiments from ELF-B (105M) to ELF-L (652M). Scaling from ELF-B (105M) to ELF-L (652M) raises CFRL-SVG accuracy on all three non-saturated tasks (Table 2). Despite stronger SFT baselines, ELF-L retains comparable or larger RL gains and outperforms CFRL-GRR, suggesting that increased model capacity complements CFRL and supports its scaling potential.

We sincerely appreciate the reviewers' feedback, which has helped us improve the clarity, rigor, and empirical support of our work. We believe these revisions substantially strengthen the manuscript and more clearly establish the contributions and limitations of CFRL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.