- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reinforcement learning for continuous-flow language models and proposes CFRL, which backpropagates verifier rewards through a differentiable flow transition. The paper develops a score-function baseline and two stochastic value-gradient variants: a sequence-level critic and a token-wise critic. The critic reuses a frozen denoising backbone and is refit online, while the policy receives a pathwise gradient through one flow step. On Countdown, Sudoku, Knights-and-Knaves, and Game24, the token-wise variant improves over supervised continuous-flow baselines and the other CFRL variants. The paper further analyzes gradient variance, critic architecture and refresh frequency, sampler choices, and a self-training loop that amortizes the improved samples into cheaper inference.
Critical strengths
- The pathwise-gradient formulation is a meaningful and well-reasoned contribution for continuous-flow language models. The paper identifies a concrete property that distinguishes continuous flows from discrete autoregressive generation and uses it to replace a high-variance score-function update with backpropagation through an intermediate state and a learned value function.
- Several experiments directly test the proposed reasoning rather than only reporting endpoint accuracy. The half-split analysis measures gradient reproducibility, the one-step intervention tests whether the estimated direction improves reward, the frozen-critic experiment demonstrates critic exploitation, and swapping the 1D and 2D critics during training shows that the critic architecture causally changes the subsequent optimization trajectory.
Critical weaknesses
- The evidence does not establish that the 2D critic performs token-level credit assignment in the semantic sense claimed by the paper. Every position is trained against the same sequence-level binary label. The critic swap convincingly shows that the 2D head changes joint training, but it does not identify why. In fact, the fixed-policy probes show similar reward discrimination and highly aligned update directions for the 1D and 2D critics. The gain could therefore come from gradient scaling, conditioning, or optimization dynamics rather than identifying which token caused the failure.
- The empirical evidence is limited to one 105M continuous-flow model that is separately fine-tuned for four synthetic tasks with exact verifiers. This is sufficient to demonstrate that CFRL can work, but not that the approach remains stable or computationally worthwhile at language-model scale. The autoregressive rows are also not matched in pretraining, model size, or task-specific adaptation, so they should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL.
Questions for authors
- Can the authors evaluate the 2D readings on examples with known corrupted positions and test whether changing the highest-scored positions has a larger counterfactual effect than changing other positions? This would distinguish token-level credit assignment from a position-wise parameterization of a sequence-level value.
- What is the total training cost of CFRL, including critic pretraining, fresh rollouts, and repeated critic refitting, relative to CFRL-GRR? A compute-matched comparison would clarify whether the improvement comes from the estimator or from substantially more learned supervision.
6 comments
Response to Reviewer oqag
The empirical evidence is limited to one 105M continuous-flow model that is separately fine-tuned for four synthetic tasks with exact verifiers. This is sufficient to demonstrate that CFRL can work, but not that the approach remains stable or computationally worthwhile at language-model scale. The autoregressive rows are also not matched in pretraining, model size, or task-specific adaptation, so they should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL.
We thank the reviewer for highlighting the importance of scalability and fair comparisons. We agree that our autoregressive (AR) baselines, which differ in pretraining, model size, and task-specific adaptation, should not be interpreted as evidence that continuous-flow RL is competitive with autoregressive RL. The motivation of this work is to explore the potential of continuous language models via post-training instead of competing with existing AR models.
To investigate whether CFRL's improvements extend beyond the 105M-parameter ELF-B model, we conducted additional experiments on ELF-L (652M parameters) (Table 2 in the revised manuscript), approximately 6.2× larger than ELF-B. We applied the same critic pre-training and RL hyperparameters without additional tuning, using the same validation-based checkpoint selection and averaging test accuracy over five sampling seeds.
Test accuracy (%) on ELF-L (652M). Improvements over task-SFT are shown in parentheses.
| Task | SFT Base | CFRL-GRR | CFRL-SVG-1D | CFRL-SVG-2D |
|---|---|---|---|---|
| Countdown | 60.5 | 63.3 (+2.8) | 72.6 (+12.1) | 74.4 (+13.9) |
| Sudoku | 71.0 | 70.3 (-0.7) | 79.3 (+8.3) | 77.9 (+6.9) |
| Game24 | 58.6 | 66.9 (+8.3) | 70.5 (+11.9) | 71.0 (+12.4) |
| K&K | 100.0 | – | – | – |
CFRL-SVG consistently improves over the SFT baseline and CFRL-GRR across all three non-saturated tasks. In particular, SVG-2D achieves improvements of +13.9, +6.9, and +12.4 percentage points on Countdown, Sudoku, and Game24, respectively. These gains are comparable to or greater than those observed on ELF-B, demonstrating that CFRL's effectiveness extends to a substantially larger continuous-flow model.
While these experiments provide encouraging evidence of scalability from 105M to 652M parameters, they do not establish stability or computational efficiency at multi-billion-parameter scale. We leave the further scaling studies as future work.
Response to Reviewer oqag
What is the total training cost of CFRL, including critic pretraining, fresh rollouts, and repeated critic refitting, relative to CFRL-GRR? A compute-matched comparison would clarify whether the improvement comes from the estimator or from substantially more learned supervision.
We thank the reviewer for raising this question about compute matching. We added total training FLOPs across all network evaluations in the revised manuscript, including rollouts, actor updates, critic pre-training, and online critic updates. Following the standard parameter-based FLOPs approximation ([1]), we count a forward pass as one unit and a full backward pass as two units. We additionally approximate an input-gradient-only backward pass as one unit, since the critic backbone is frozen during RL.
As shown in the new Table below (table 10 in the revised manuscript), CFRL-SVG-2D consistently outperforms CFRL-GRR at matched total FLOPs across all four tasks, even after accounting for the full critic cost. For fair comparison, we additionally trained CFRL-GRR using the same deployment sampler as CFRL-SVG (64 steps, or 32 for Sudoku), rather than its original 16-step sampler. This approximately doubles GRR's computational cost while keeping all other training settings unchanged. And we also added figure 3 in the revised manuscript to show Acc vs Flops across four datasets on three variants of CFRL. It also shows that with the comparable compute, CFRL-SVG outperforms CFRL-GRR consistently. For example, at approximately 200 PFLOPs on Countdown, SVG-2D achieves 49.6% accuracy versus 43.0% for GRR; at approximately 100 PFLOPs on Game24, SVG-2D achieves 56.4% versus 46.3% for GRR with the deployment sampler. Similar improvements hold on Sudoku and K&K.
Test accuracy (%) and total training compute (PFLOPs). The GRR 64-sampler row is newly evaluated; all other accuracies are from Table 1. Critic-stage costs are included in the SVG totals.
| Method | Countdown Acc. | PFLOPs | Sudoku Acc. | PFLOPs | K&K Acc. | PFLOPs | Game24 Acc. | PFLOPs |
|---|---|---|---|---|---|---|---|---|
| Task-SFT base | 42.8 | 0 | 64.0 | 0 | 68.2 | 0 | 44.0 | 0 |
| CFRL-GRR (γ4@16) | 42.6 | 97 | 67.9 | 221 | 71.4 | 130 | 47.4 | 97 |
| CFRL-GRR (γ56@64; γ28@32 Sudoku), new | 42.8 | 208 | 64.2 | 306 | 77.0 | 278 | 47.3 | 208 |
| CFRL-SVG-1D | 47.4 | 263 | 64.1 | 269 | 76.5 | 277 | 51.2 | 232 |
| CFRL-SVG-2D | 49.7 | 263 | 68.7 | 269 | 77.5 | 277 | 58.2 | 232 |
| Critic stage (included in SVG) | – | 67 | – | 113 | – | 103 | – | 36 |
The new compute-matched experiments show that this advantage persists after accounting for the critic's cost and matching the rollout sampler. We have incorporated the detailed FLOPs accounting into Appendix D.
We sincerely thank the reviewer for their time and thoughtful feedback, which have helped us strengthen the paper and improve its clarity and rigor. We have incorporated these additional experiments and clarifications into the revised manuscript, with all changes highlighted in blue.
[1] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.
Response to Reviewer oqag
We thank the reviewer for the positive assessment and thoughtful feedback. We particularly appreciate the recognition of our pathwise-gradient formulation as a meaningful contribution to continuous-flow language models, as well as their acknowledgment that our gradient reproducibility, single-step intervention, frozen-critic, and critic-swap experiments provide direct evidence supporting our methodology. We also appreciate the reviewer's constructive suggestions regarding token-level credit assignment and computational efficiency. We address these concerns below and provide additional experiments and analyses to further strengthen our empirical findings.
We thank the reviewer for this insightful suggestion. We agree that supervising each token-wise critic output with the same sequence-level reward does not guarantee semantic token-level credit assignment. To investigate this, we conducted two additional experiments examining whether the 2D critic identifies incorrect token positions and whether its token-wise readings respond to targeted perturbations.
(1) Correlation between token-wise readings and incorrect positions. We first evaluate whether the 2D critic assigns lower readings to incorrect token positions in failed Sudoku solutions. We focus on Sudoku because each output position corresponds directly to a specific answer cell, enabling unambiguous identification of incorrect tokens. For each failed response, we take the single token with the lowest reading and record 1 if that token is actually wrong and 0 otherwise; Precision is the average over all failed responses. The baseline is the fraction of wrong tokens in the response, i.e. the probability that a randomly chosen position is a wrong token. This table can be found in Table 14 in the revised manuscript.
The precision exceeds random performance. This suggests that the 2D critic's readings contain information about which token positions are incorrect, despite being trained with sequence-level supervision.
(2) Counterfactual token corruption. Following the reviewer's suggestion, we conduct a controlled perturbation experiment to evaluate whether the 2D critic responds to token-level errors. Starting from a correct solution, we corrupt a single token and measure the change in each position's critic reading. We report the percentage of cases in which the corrupted position exhibits the largest decrease in its critic reading among all positions, compared with the random-position baseline.
Across all four tasks, the corrupted position is substantially more likely to exhibit the largest decrease in critic reading than expected by chance. An incorrect answer token can be sufficient to establish sequence-level failure, whereas a correct token alone cannot establish success. This asymmetry motivates position-sensitive critic readings even under sequence-level supervision. These results demonstrate that the 2D critic is sensitive to localized token perturbations, providing evidence of position-specific reward attribution despite being trained with sequence-level supervision. This table can be found in Table 15 in the revised manuscript.
Our experiments show that the critic captures position-specific information and responds to localized corruption, while they do not establish that semantic token-level credit assignment is the sole source of its advantage and we do not rule out the possibility that the 2D critic's performance gains may also arise from gradient scaling, conditioning, or optimization dynamics. We have therefore added this clarification in the revised manuscript (E.3).