acceptodds
Under review as a conference paper at ICLR 2027

Future-Aligned Reinforcement Learning for Asynchronous Language Model Post-Training

Abstract

Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, at the cost of staleness: a rollout is generated by a behavior policy that is one optimizer update behind the target policy by the time it is consumed. Most existing methods address this staleness reactively, for example through importance-sampling and clipping schemes or distribution-matched replay frameworks, repairing stale data after it has been produced. We introduce Future-Aligned Reinforcement Learning (FARL), a proactive method that, at rollout-dispatch time, forecasts the consumption-time policy (the learner one update ahead) and generates the entire trajectory from this single forecasted, frozen worker policy. The forecast extrapolates the learner's next parameter displacement from the most recent updates using a consistency-constrained multi-lag predictor. We show that the consistency constraint cancels the first-order constant-drift component of policy lag: under fixed-step smooth learner dynamics this yields an asymptotic upper bound on the forecast residual against for stale generation, where is the learner's step size, and correspondingly tightens the local bound on the variance of the exact importance weights from to . For finite-step stochastic training we give a separate coefficient-aware data-dependent bound and measure the realized prequential forecast error directly. Because FARL acts on the behavior policy before generation rather than repairing data afterward, it is a drop-in plug-in module orthogonal to existing reactive correctors, and across a thorough set of experiments it improves the average accuracy of every one of them. Code is available at https://anonymous.4open.science/r/farl-anonymous-0442.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from verifiable rewards, large language model post-training, asynchronous reinforcement learning, policy staleness, off-policy correction, GRPO, importance sampling variance, policy forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PAjxOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FARL addresses policy staleness in asynchronous RL by forecasting future learner parameters from recent updates and using the forecasted policy for rollout generation. The reported results support its effectiveness in the evaluated short-delay settings. However, its memory–performance tradeoff and the relationship between parameter prediction and policy alignment require further evaluation.

Critical strengths

S1. The experiments include multiple baselines, ablations, and direct measurements of forecast quality and policy mismatch.

Critical weaknesses

W1. FARL stores five full-sized update vectors, requiring approximately 17 GB of additional GPU memory for the 1.7B model and 40 GB for the 4B model. This is a substantial cost that limits its practicality and scalability.

W2. A small parameter prediction error does not guarantee a small policy distribution error. Further paired token-level KL evaluations are needed to verify that better parameter prediction consistently improves policy alignment. The current Table 8 does not fully establish this point.

Questions for authors

Q1. Could the authors evaluate FARL on larger models and report final benchmark performance, peak GPU memory usage, and parameter prediction error? These results would clarify whether the current predictor remains effective and practical at larger scales.

Q2. Please correct the formatting issues on page 13.

11 comments
AuthorsOpenReview ↗

Response to Reviewer PAjx (1/3)

We thank the reviewer for the careful assessment and constructive feedback. We particularly appreciate the concerns regarding FARL's scalability and the relationship between parameter prediction and policy alignment. These are important aspects of our work, and we address them below through implementation clarifications and additional empirical analysis.

Weaknesses:

W1: FARL stores five full-sized update vectors, requiring approximately 17 GB of additional GPU memory for the 1.7B model and 40 GB for the 4B model. This is a substantial cost that limits its practicality and scalability."

W1: Memory cost of the update buffer.

Thank you for raising this concern. On re-checking our implementation, we found that Table 6 of the submission incorrectly described the buffer as GPU-resident and sharded in bf16. In our implementation, the update history is stored in host (CPU) memory, not GPU memory. We have corrected this in the revision (Table 7 and App. H), and we verified it from our training logs.

  • GPU memory: no increase (measured). Peak trainer GPU memory is the same with and without FARL (mean over 3 seeds, from our training logs):
Qwen3-1.7B Async-GRPOQwen3-1.7B FARLQwen3-4B Async-GRPOQwen3-4B FARL
Peak GPU memory allocated28.2 GB28.2 GB38.2 GB38.2 GB
Peak GPU memory reserved75.7 GB75.9 GB76.0 GB75.9 GB

At each weight synchronization, the rank-0 trainer copies the parameters to host memory, computes the forecast on the CPU, and broadcasts it to the rollout workers. All methods use identical batch sizes and GPU memory settings (Table 4).

  • Host memory. The buffer holds five fp32 parameter-sized vectors (the previous parameters and the m=4m=4 most recent updates). The measured increase in host memory is about 41 GB for Qwen3-1.7B and 80 GB for Qwen3-4B.

Scalability. The host-memory footprint scales linearly with model size and history length, and can be reduced in three ways:

  1. Sharding: each trainer rank stores only its own shard of the update history, so the per-node cost falls with the number of ranks and nodes.
  2. Reduced precision: storing the updates in bf16, while keeping the previous parameters in fp32 so the update itself is computed exactly, reduces the buffer to about 24 GB and 48 GB, respectively.
  3. Shorter history: the constant-velocity variant (m=1m=1) stores a single extra vector and still improves average accuracy from 43.3% to 44.2%, outperforming Async-GRPO on all nine benchmarks (App. L, Table 10).

The revision states the actual memory placement, the measured GPU and host memory, and these memory–accuracy trade-offs explicitly (App. H, Table 7).

AuthorsOpenReview ↗

Response to Reviewer PAjx (2/3)

W2: A small parameter prediction error does not guarantee a small policy distribution error. Further paired token-level KL evaluations are needed to verify that better parameter prediction consistently improves policy alignment. The current Table 8 does not fully establish this point.

We agree that parameter error is not a policy metric. Our theory states its policy-level results in terms of the Fisher-weighted error e⊤Fee^\top F e, not ∥e∥2\Vert e\Vert^2 (Thm. 2). We tested this directly with a paired, token-level evaluation (math workload, 3 seeds, steps 5–24: 60 step–seed pairs per candidate), now in App. T (Tables 18–19).

  • At every step ss, all candidate behavior policies are scored against the consumption-time policy θs+1\theta_{s+1} on the same prefixes (256 prompts × 4 rollouts from θs+1\theta_{s+1}).
  • Token-level KL and χ2\chi^2 are computed exactly over the full vocabulary with the trainer (no sampling noise, no engine mismatch). Summed over ~650–700 response tokens, the O(10−5)\mathcal{O}(10^{-5}) per-token KL matches the trajectory KL of Table 9 (Table 8 in the submission).
  • Matched-norm controls and α\alpha-scaled forecasts test whether alignment comes from prediction rather than from merely moving the sampler.

Table A: Paired token-level alignment at d=1d=1 (median over 60 pairs; win rate = fraction of pairs with lower token KL than stale on the same prefixes).

ModelBehavior policy qqRparamR_{\text{param}}Token KL$(p\Vert q)$Token χ2(p∥q)\chi^2(p\Vert q)KL ratio vs. staleWin rate vs. stale
1.7BStale θs\theta_s1.005.2×10−55.2\times10^{-5}1.1×10−41.1\times10^{-4}1.00—
1.7BConstant velocity0.833.7×10−53.7\times10^{-5}7.6×10−57.6\times10^{-5}0.710.750 (45/60)
1.7BFARL0.538.0×10−6\mathbf{8.0\times10^{-6}}1.6×10−5\mathbf{1.6\times10^{-5}}0.150.967 (58/60)
1.7BFARL, α=0.5\alpha=0.50.542.1×10−52.1\times10^{-5}4.3×10−54.3\times10^{-5}0.400.883 (53/60)
1.7BFARL, α=1.5\alpha=1.50.991.5×10−51.5\times10^{-5}3.1×10−53.1\times10^{-5}0.290.833 (50/60)
1.7BRandom direction (matched norm)1.947.1×10−57.1\times10^{-5}1.5×10−41.5\times10^{-4}1.370.217 (13/60)
1.7BAnti-forecast (matched norm)3.361.7×10−41.7\times10^{-4}3.6×10−43.6\times10^{-4}3.270.033 (2/60)
4BStale θs\theta_s1.002.3×10−52.3\times10^{-5}4.7×10−54.7\times10^{-5}1.00—
4BConstant velocity0.529.2×10−69.2\times10^{-6}1.9×10−51.9\times10^{-5}0.400.917 (55/60)
4BFARL0.514.0×10−6\mathbf{4.0\times10^{-6}}8.1×10−6\mathbf{8.1\times10^{-6}}0.171.000 (60/60)
4BFARL, α=0.5\alpha=0.50.539.0×10−69.0\times10^{-6}1.8×10−51.8\times10^{-5}0.390.900 (54/60)
4BFARL, α=1.5\alpha=1.50.989.0×10−69.0\times10^{-6}1.9×10−51.9\times10^{-5}0.390.867 (52/60)
4BRandom direction (matched norm)1.853.2×10−53.2\times10^{-5}6.5×10−56.5\times10^{-5}1.390.183 (11/60)
4BAnti-forecast (matched norm)3.107.9×10−57.9\times10^{-5}1.6×10−41.6\times10^{-4}3.430.033 (2/60)

Table B: Does parameter error predict policy error, step by step? Pooled over 360 non-stale (step–seed, candidate) pairs per scale; F^s\hat F_s is a diagonal empirical Fisher from the previous step's trainer gradients. Concordance =Pr⁡^(KL(p∥q)<KL(p∥θs) ∣ Rparam<1)=\widehat{\Pr}\big(\mathrm{KL}(p\Vert q)<\mathrm{KL}(p\Vert\theta_s)\,\vert\,R_{\text{param}}<1\big).

ModelSpearman(RparamR_{\text{param}}, token KL)ConcordanceSpearman(e⊤F^see^\top\hat F_s e, token KL)
1.7B0.5892.6% (176/190)0.94
4B0.5495.1% (193/203)0.96

Takeaways.

  • Better parameter prediction does give better policy alignment. FARL lowers token KL vs. stale on 58/60 pairs at 1.7B and 60/60 at 4B, cutting the median by 83–85%. Moving the sampler without prediction hurts: matched-norm random and anti-forecast shifts raise token KL (~1.4× and 3.3–3.4×).
  • Euclidean error is a reliable coarse test, but not a fine-grained metric, as you suspected. It predicts whether a candidate beats stale (concordance 93–95%) but misses rankings: α=0.5\alpha=0.5 and α=1\alpha=1 have nearly identical RparamR_{\text{param}} (0.54 vs. 0.53) yet differ by >2× in token KL.
  • The Fisher-weighted error explains the rest. e⊤F^see^\top\hat F_s e tracks token KL closely (Spearman 0.94–0.96), as Thms. 2 and 4 predict.
  • The full forecast is the right magnitude in policy space. A quadratic fit of token KL over α\alpha is minimized at α⋆≈1.08\alpha^\star\approx1.08 (1.7B) and 1.001.00 (4B), whereas Euclidean error is minimized near α≈0.75\alpha\approx0.75 (App. M). The consistency-preserving α=1\alpha=1 is thus near-optimal in policy space (Prop. 1).
AuthorsOpenReview ↗

Response to Reviewer PAjx (3/3)

Questions:

Q1: Could the authors evaluate FARL on larger models and report final benchmark performance, peak GPU memory usage, and parameter prediction error? These results would clarify whether the current predictor remains effective and practical at larger scales.

Following your suggestion, we trained Qwen3-8B with the same configuration on the math workload (d=1d=1, 3 seeds). We report the 5-benchmark math average (MATH-500, AIME24, AIME25, AMC23, AMC24; mean ±\pm per-seed std):

ModelMethodMath-5 AVG (%)Peak GPU allocated / reserved (GB)Median RparamR_{\text{param}}Pr⁡(Rparam<1)\Pr(R_{\text{param}}<1)Token KL ratio vs. stale
Qwen3-8BAsync-GRPO58.4±0.558.4\pm0.552.4 / 76.2——1.00
Qwen3-8BFARL59.4±0.4\mathbf{59.4\pm0.4} (+1.0)52.4 / 76.30.491.000.18
  • The predictor remains effective at 8B. The forecast beats the stale checkpoint on 100% of matured steps (median Rparam=0.49R_{\text{param}}=0.49) and cuts token-level KL by 82%. FARL improves the Math-5 average by +1.0, more than twice the per-seed standard deviation (±0.4\pm0.4).
  • It remains practical at 8B. Peak GPU memory is unchanged relative to Async-GRPO, because the update history is kept in host memory (see W1).
  • Host memory scales linearly, to about 164 GB at 8B with our unsharded fp32 implementation. As noted in W1, storing the updates in bf16 reduces this to about 98 GB, and sharding the history across the 4 trainer ranks brings it to about 41 GB per rank (fp32) or 25 GB per rank (bf16).

Q2. Please correct the formatting issues on page 13.

Thank you for catching this. The issue is resolved now in the revised version.


If any concerns remain, please let us know and we will gladly address them during the discussion period. If our responses have resolved your concerns, we would be grateful if you would consider raising your score.

Reviewer_wwSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Asynchronous reinforcement learning (RL) improves training efficiency by overlapping rollout generation and policy optimization, but this introduces policy staleness because the policy used to generate a trajectory may differ from the policy that consumes it. This paper proposes Future-Aligned Reinforcement Learning (FARL), which predicts the learner's future parameter update and uses the predicted policy to generate trajectories that better match the policy at consumption time. FARL combines recent parameter updates through a consistency-constrained linear predictor and learns its coefficients using delayed online ridge regression based only on historical observations. The paper provides theoretical analysis connecting prediction errors to policy mismatch and importance-weight variance, and evaluates FARL on mathematical reasoning and code-generation benchmarks. Results show consistent improvements over several asynchronous RL baselines, and FARL can be combined with existing methods to further improve performance.

Critical strengths
  1. A clear and useful idea for reducing policy staleness. Instead of correcting policy mismatch only after trajectories have been generated, FARL predicts the future learner update and adjusts the behavior policy in advance. The consistency constraint gives the predictor a clear mathematical basis, while delayed online regression allows it to adapt to observed training dynamics without using future information. This is a meaningful and complementary approach to existing asynchronous RL methods.
  2. Strong theoretical analysis and convincing empirical validation. The paper connects parameter prediction errors to policy mismatch and importance-weight variance, providing a clear explanation of why the proposed method can help asynchronous RL. The experiments cover multiple reasoning and coding benchmarks and include useful ablations and counterfactual controls. In particular, the results show that FARL can improve existing asynchronous RL methods, suggesting that its benefits are not limited to replacing a particular baseline.
Critical weaknesses

Limited empirical validation under variable policy lags. The main experiments use a fixed one-step policy lag, and Appendix O further evaluates several larger fixed lags. The paper also analyzes the additional error caused by an incorrectly predicted lag and proposes a scheduler that estimates the consumption lag from queue states and actor–learner throughput. However, the effectiveness of this scheduling mechanism has not been directly demonstrated in a setting where the consumption lag varies dynamically. The current results therefore provide strong evidence for FARL under fixed-lag settings, but leave its practical effectiveness under variable-lag conditions less clear.

Questions for authors
  1. Evaluation under variable policy lags: Section 3.4 proposes predicting the consumption lag from queue states and actor–learner throughput. Has this scheduling mechanism been implemented and evaluated end to end? If not, could the authors provide a controlled experiment with dynamically varying consumption lags, reporting both lag prediction errors and the resulting accuracy of FARL compared with a fixed-horizon predictor?

  2. Practical impact of lag prediction errors: The theoretical analysis bounds the additional error caused by a mismatch between the predicted and actual consumption lags. Could the authors use training logs to analyze how lag prediction errors affect the realized parameter-space forecast residual, and clarify how accurate the lag prediction needs to be for FARL to retain its empirical benefits?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wwSH (1/2)

We thank the reviewer for the thoughtful feedback and for recognizing FARL’s theoretical and empirical contributions. We agree that evaluating FARL under dynamically varying policy lags is important for establishing its practical applicability. We appreciate these insightful suggestions and address both questions below.

Weaknesses:

W1: Limited empirical validation under variable policy lags. The main experiments use a fixed one-step policy lag... .

Thank you for this point. We agree the scheduler needed a direct test, so we ran one and added it to the paper (new App. S).

Setup. Qwen3-1.7B with the configuration of App. O, 3 seeds. We vary rollout latency and queue congestion so that the consumption lag changes dynamically, with lag distribution 50% d=1d=1, 30% d=2d=2, 15% d=3d=3, 5% d=4d=4. We compare:

  • Async-GRPO;
  • FARL with a fixed lag prediction d^=1\hat d=1;
  • FARL with the adaptive scheduler (queue state and actor–learner throughput, Sec. 3.1);
  • oracle-lag FARL, which knows the actual consumption lag but not future learner parameters.
MethodAVG (%)Lag prediction accuracy (%)Lag MAE
Async-GRPO42.7 ± 0.5——
FARL (fixed d^=1\hat d=1)43.9 ± 0.450.00.75
FARL (adaptive scheduler)44.1± 0.480.00.20
FARL (oracle lag)** 44.5 ± 0.5 **100.00.00

Takeaways.

  • FARL remains effective under variable lag. The adaptive scheduler improves over Async-GRPO by +1.4.
  • FARL is robust to lag misprediction. Even with a fixed d^=1\hat d=1, which is wrong for half the rollouts, FARL retains +1.2. This matches Prop. 3: a wrong lag adds an error term proportional to ∣d−d^∣|d-\hat d|, and 80% of rollouts here have ∣d−d^∣≤1|d-\hat d|\le1.
  • The scheduler helps, but modestly. It raises lag-prediction accuracy from 50% to 80% and cuts MAE from 0.75 to 0.20, closing about a third of the gap between fixed and oracle lag (+0.2 of +0.6). Most of the benefit comes from forecasting the learner itself; better lag prediction is a refinement on top.
  • Variable lag lowers the ceiling. Even oracle-lag FARL gains less here (+1.8) than at fixed d=1d=1 (+2.5), because longer lags are harder to forecast (Sec. 5.3).

Further evidence already in the paper. FARL + D-ARL is also a variable-lag setting. D-ARL's replay buffer consumes samples 1–4 steps after generation, while FARL forecasts only one step ahead (App. G). FARL still improves D-ARL by +2.1 (1.7B) and +1.1 (4B), so its gains hold when the actual consumption lag varies and differs from the predicted one.

Revisions (in blue): new App. S (Table 16) with this experiment; a summary in Sec. 5.3; and Sec. 3.1 now points to App. S for the scheduler evaluation.

Questions:

Q1: Evaluation under variable policy lags: Section 3.4 proposes predicting the consumption lag .....

Q1: Evaluation under variable policy lags.

Yes. The scheduler is now implemented and evaluated end to end in a controlled variable-lag experiment; please see our response to W1 above and the new App. S (Table 16). In short, under a dynamically varying lag (50% d=1d=1, 30% d=2d=2, 15% d=3d=3, 5% d=4d=4):

  • Lag prediction: the scheduler raises lag-prediction accuracy from 50% to 80% and lowers the MAE from 0.75 to 0.20, compared with the fixed-horizon predictor (d^=1\hat d=1).
  • Accuracy: FARL with the scheduler reaches 44.1, vs. 43.9 with the fixed-horizon predictor, 44.5 with the oracle lag, and 42.7 for Async-GRPO.

So FARL helps under variable lag even with a fixed horizon (+1.2), and the scheduler closes about a third of the remaining gap to the oracle.

AuthorsOpenReview ↗

Response to Reviewer wwSH (2/2)

Q2: Practical impact of lag prediction errors: The theoretical analysis bounds the additional error caused by a mismatch between... .

Thank you for this question. We analyzed the training logs of the variable-lag experiment (see our W1 response); the analysis is now in App. S (Table 17).

Realized residual vs. lag error. For each matured step we compute Rparam=∥Ss,d−S^s,d^∥2∥Ss,d∥2,R_{\text{param}}=\frac{\|S_{s,d}-\widehat S_{s,\hat d}\|^2}{\|S_{s,d}\|^2}, where dd is the actual consumption lag and d^\hat d the predicted one. Rparam<1R_{\text{param}}<1 means the forecast is still closer to the consumption-time parameters than the stale checkpoint. Median over matured steps, grouped by lag pair:

Actual ddPredicted d^\hat dLag error d−d^d-\hat dRparamR_{\text{param}}
1100.526
21+1 (under)0.515
31+2 (under)0.720
12−1 (over)2.011
13−2 (over)5.319

The key finding, which the symmetric bound of Prop. 3 does not resolve, is that the effect of a lag error depends on its direction. Under constant learner velocity, S^s,d^≈d^ Δs\widehat S_{s,\hat d}\approx \hat d\,\Delta_s and Ss,d≈d ΔsS_{s,d}\approx d\,\Delta_s, so Rparam≈((d−d^)/d)2R_{\text{param}}\approx((d-\hat d)/d)^2 plus a forecast-noise floor:

  • Under-prediction is benign. The forecast moves only part of the way, but in the right direction, so RparamR_{\text{param}} stays below 1 (idealized 0.25 and 0.44; measured 0.515 and 0.720).
  • Over-prediction is harmful. The forecast overshoots, and once d^≥2d\hat d\ge 2d it is no better than the stale checkpoint (idealized 1 and 4; measured 2.011 and 5.319).

This mirrors our forecast-strength ablation (App. M, Table 11): under-scaling (α=0.5\alpha=0.5) keeps most of the gain (+1.6), while over-scaling (α=1.5\alpha=1.5) loses most of it (+0.6).

One subtlety: under-predicting at d=2d=2 gives a slightly lower Euclidean residual (0.515) than the matched d=2d=2 forecast (0.71, Table 2), yet oracle-lag FARL is more accurate than fixed d^=1\hat d=1. As in App. M, Euclidean RparamR_{\text{param}} is a reliable test of whether the forecast beats stale generation, but not a fine-grained ranking of configurations; what matters downstream is the Fisher-weighted error (Thm. 2).

How accurate must lag prediction be? Not very, as long as it avoids over-prediction. In the variable-lag run (App. S, Table 16), a fixed d^=1\hat d=1, which is right only 50% of the time but can only under-predict, keeps +1.2 of the +1.8 oracle-lag gain (67%). The adaptive scheduler (80% accuracy, MAE 0.20) keeps +1.4 (78%). A scheduler should therefore break ties toward the shorter lag, as we now note in Sec. 5.3.

Thank you for the detailed rebuttal and additional experiments.

The new variable-lag evaluation in Appendix S directly addresses my main concern regarding FARL's effectiveness under dynamically varying consumption lags. The comparison between the fixed-horizon predictor, adaptive scheduler, and oracle-lag setting provides useful empirical evidence.

I also appreciate the analysis of lag prediction errors and their impact on realized forecast residuals. In particular, the observed asymmetry between under-prediction and over-prediction offers valuable insight into the robustness of the proposed method.

Overall, I believe my concerns have been adequately addressed. I will maintain my original rating of 3 (Weak acceptance).

AuthorsOpenReview ↗

Thank you for the follow-up (reviewer wwSH)

Thank you for reading our responses carefully and for confirming that your concerns are addressed. Your questions on variable and unknown lag improved the paper.

We appreciate your time and constructive feedback.

Reviewer_6AHrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a compelling and novel idea: address the inference-train mismatch during async RL by setting the inference policy to a prediction of the train policy – Future Aligned RL (FARL). If this works, then a litany of fixes that have been explored for addressing this mismatch, including importance sampling, could become more robust. Across 9 benchmark datasets and four approaches to async RL, experiments consistently find that the proposed approach boosts performance, consistent with the provided theory that suggests that the inference-train mismatch will be lower when using FARL.

Critical strengths

The paper is addressing a major problem affecting a substantial amount of post-training efforts, it will likely have a very broad audience.

The approach to dealing with async instability is truly novel while being simple and intuitive.

Critical weaknesses

The problem being explored is highly related to the problem addressed in the recent "Score Centering" paper (Marek & Ryabinin, 2026). I understand that this is concurrent work, but I think that discussion -- and ideally new experiments -- that address score centering would greatly improve the relevance of this work to the async RL community. In particular, this submission and score centering both address train-inference mismatch, and it would be good to know if FARL provides benefits on top of score centering, and to see this submission's authors relate the instability driver outlined in the score centering paper -- distillation toward the sampler -- to FARL.

On the other hand, rather than applying “reactive” corrections, some RL approaches are formulated to be robust to off-policy data. A foundational example in the async RL for LLM literature is “Trajectory Balance with Asynchrony” (TBA), which does not require importance sampling or selection over rollouts. A recent special case of TBA, called CORO (which is TBA with the reference policy set to be the inference policy), was recently used to post-train the humans& Ampersand model. I think the related work discussion should be augmented to include approaches such as TBA, and I don't think that “reactive” is a helpful label for such approaches, “off-policy robust” might be better.

Questions for authors

Can you test FARL combined with Score Centering and/or TBA (using the inference policy as the reference policy, for simplicity)? It is important to see if FARL can help Score Centering and/or TBA, as these methods are robust to data that is off-policy for other reasons (like lower-precision inference) and thus their combination with FARL could be especially powerful given FARL’s ability to address off-policyness from an older inference policy.

  • TBA is implemented simply by subtracting the (beta-weighted) K1 estimate of reverse KL from the reward before advantage computation. Score centering has a similarly simple implementation.

Can you move results with higher staleness levels into the main text and augment them? I think it's important to show these to give readers an understanding of the regimes in which FARL is most effective. In particular, results for 5 and 10 steps off policy (at a minimum) would be helpful/informative.

Could you please clarify the meaning of line 470? (“vanishing at precisely the horizon where the parameter-space forecast stops beating the stale checkpoint…”)

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6AHr (1/3)

We sincerely thank the reviewer for the constructive feedback, and for recognizing the novelty, simplicity, and broad relevance of FARL. We particularly appreciate the insightful connections to concurrent work on Score Centering and off-policy-robust learning objectives. These suggestions have helped us better contextualize our contribution and identify important directions for additional empirical validation. We address each concern below and discuss the implications for FARL's applicability to asynchronous RL.

Weaknesses:

W1: The problem being explored is highly related to the problem addressed in the recent "Score Centering" paper (Marek & Ryabinin, 2026)...

Thank you for pointing us to Score Centering (SC). It is concurrent work, and we agree it gives a complementary view of FARL. We worked out the connection, ran the comparison you suggested, and revised the paper accordingly.

How FARL relates to SC's drift. SC writes the drift at a prefix hh as Eq[R] sˉ(h)\mathbb{E}_q[R]\,\bar s(h), with sˉ(h)=∑vqv∇log⁡pv\bar s(h)=\sum_v q_v\nabla\log p_v. In our notation (App. B.5), p=πθ⋆p=\pi_{\theta^\star} and q=πθ⋆+eq=\pi_{\theta^\star+e}, where ee is the behavior–target parameter error. Under the local regularity of Thm. 4, a Taylor expansion gives

sˉ(h)=Fh(θ⋆) e+O(∥e∥2),\bar s(h)=F_h(\theta^\star)\,e+O(\|e\|^2),

where FhF_h is the token-level Fisher information at hh.

The two notions are different: FARL's constant drift describes the learner's parameter motion, while SC's drift is a bias in the gradient estimator. This expansion links them. With bounded FhF_h and the smooth dynamics of Cor. 1, the upper bound on the staleness-induced expected score falls from O(η)O(\eta) for stale generation (eS=−Δse_S=-\Delta_s) to O(η2)O(\eta^2) for FARL.

We also checked this link empirically. In a new paired token-level evaluation (details in our response to another reviewer, W2), the Fisher-weighted error e⊤F^ee^\top \hat F e tracks token-level KL closely (Spearman 0.94–0.96 at 1.7B and 4B), and FARL lowers token KL relative to stale generation on 58/60 and 60/60 paired steps, cutting it by 83–85%.

As SC notes, the sign of the drift at each prefix follows the expected advantage. The learner is pulled toward the sampler after prefixes with positive expected advantage and pushed away after negative ones. Because the drift is linear in ee, and stale errors are strongly autocorrelated across steps (lag-1 ≈ 0.71 for Δs\Delta_s, App. N), the stale drift tends to be systematic rather than averaging out. FARL's consistency constraint removes its constant-velocity component.

Even exact SC leaves one first-order term. Its remaining update is Covq(R,s)\mathrm{Cov}_q(R,s), measured under the sampler rather than the learner (SC, Eq. 5). Importance weighting corrects this partly. FARL shrinks it at the source by generating from a policy closer to the learner.

Does FARL help on top of SC? Setup: Qwen3-1.7B, d=1d=1, nine-benchmark average. Our shared surrogate is unchanged (GRPO clip [0.8,1.2][0.8,1.2], dual clip 3.0, TIS at 2.0), so every arm, including "+ SC", already contains TIS. On top of it we add SC's weighted top-kk centering (SC App. A.2, k=128k=128).

MethodAVG (%) ± STD
Async-GRPO43.3 ± 0.5
Async-GRPO + SC44.4 ± 0.4
FARL45.8 ± 0.4
FARL + SC46.0 ± 0.3
  • FARL still helps after SC: +1.6 over Async-GRPO + SC (44.4 → 46.0).
  • In our one-step setting, FARL helps more than SC: +2.5 vs. +1.1 over Async-GRPO. This is consistent with SC leaving the sampler mismatch in place, so the covariance gap above remains.
  • SC adds only +0.2 once FARL is on. This is consistent with FARL having already removed most of the mismatch that produces drift; FARL lowers the behavior–target KL by ~7× (Table 9).

FARL and SC are complementary, with partly overlapping gains. FARL removes predictable version lag before generation. SC removes residual drift at the learner, and also covers what FARL cannot: numerical training–inference mismatch, and long lags where forecasting stops working (App. O, d=4d=4).

Revisions made (in blue in the updated PDF):

  1. Sec. 2 and App. D: a short paragraph in Sec. 2, with the full discussion of SC and training–inference mismatch in App. D.
  2. App. B.5: Proposition 4, the drift–lag expansion above with the O(η)O(\eta) vs. O(η2)O(\eta^2) bounds, plus remarks on drift direction, persistence, and the covariance term SC leaves. The full proof is in App. C.
  3. New App. R (Table 15): the FARL/SC experiment above, with setup and implementation details.
  4. Sec. 5.2 and Conclusion: short summaries of the SC result and the complementarity.
AuthorsOpenReview ↗

Response to Reviewer 6AHr (2/3)

W2: On the other hand, rather than applying “reactive” corrections, some RL approaches are formulated to be robust to off-policy data... .

Thank you for this pointer. We agree that "reactive" is the wrong label for objectives such as TBA, and we have revised the paper accordingly.

Taxonomy. We now separate reactive correctors, which repair stale samples after generation (IS/clipping, replay selection), from off-policy-robust objectives, which learn from stale data directly without requiring importance sampling. The introduction now states explicitly that the latter are not reactive. FARL is a third, proactive option that changes the sampler itself.

Related work (Sec. 2, with the full discussion in App. D).

  • TBA (Bartoldson et al., 2026) uses the trajectory balance objective, which is valid for samples from any full-support behavior policy, to enable asynchronous learning without requiring importance sampling.
  • OAPL (Ritter et al., 2026) instead regresses the trainer–inference log-ratio onto an estimated optimal advantage. Its KL reference is the lagged inference policy, and it is robust to lags of hundreds of gradient steps.
  • CORO, used to post-train Ampersand, drops OAPL's value-calibration term. Up to a constant factor, it matches the VarGrad form of TBA's loss with the reference policy set to the inference policy.

How FARL relates to these methods.

  • Orthogonal mechanism. FARL changes only which policy generates the data, so it can supply rollouts to any of these objectives. For OAPL and CORO, whose KL reference is the sampler itself, FARL would also anchor the regularizer at the forecast rather than at the stale checkpoint.
  • Different regimes. Off-policy-robust objectives are designed for large or highly variable lags. In our own measurements, that is exactly where forecasting stops helping: the FARL gain vanishes at d=4d=4 (App. O). FARL targets the short, predictable lag of one-step-off pipelines, where IS-based GRPO variants are the standard setup.

FARL with TBA. To test the orthogonality claim, we trained a TBA-style objective with and without FARL. FARL adds +1.0 on top of it and gives our best result overall; please see our answer to your question below for the setup and results. We have not tested FARL with OAPL or CORO; the paper states this.


Questions

Q1: Can FARL be combined with Score Centering and/or TBA?

Yes. We tested both.

Score Centering. Please see our response to the first weakness above. On Qwen3-1.7B, FARL adds +1.6 on top of SC (44.4 → 46.0), while SC adds only +0.2 on top of FARL. The setup, implementation, and analysis are in App. R (Table 15).

TBA (reference = inference policy). Following TBA′ and your suggestion, we subtract the β\beta-weighted K1 estimate of the reverse KL to the sampler from each reward before computing advantages:

r~i=ri−β∑tlog⁡πθ(yi,t∣hi,t)μ(yi,t∣hi,t),\tilde r_i = r_i - \beta\sum_t \log\frac{\pi_\theta(y_{i,t}\mid h_{i,t})}{\mu(y_{i,t}\mid h_{i,t})},

where μ\mu is the policy that generated the rollout: the stale checkpoint θs\theta_s for "Async-TBA", and the forecast θ^s,1\widehat\theta_{s,1} for "FARL + TBA". This is the update of TBA′ (Eq. 7 of Bartoldson et al.), with the reference reset to the sampler at every step instead of every 50 steps, i.e., the CORO-style variant. We use β=0.005\beta=0.005 (TBA′'s default) and otherwise leave the shared surrogate unchanged, as in the SC experiment. Setup: Qwen3-1.7B, d=1d=1, nine-benchmark average.

MethodAVG (%) ± STD
Async-GRPO43.3 ± 0.5
Async-TBA45.4 ± 0.6
FARL + GRPO45.8 ± 0.4
FARL + TBA46.4 ± 0.3
  • TBA is a strong baseline (+2.1 over Async-GRPO), close to FARL alone (+2.5). This confirms your point that off-policy-robust objectives handle one-step lag well.
  • FARL helps on top of TBA: +1.0 (45.4 → 46.4), and FARL + TBA is our best result overall.
  • TBA also helps on top of FARL (+0.6), more than SC does (+0.2).

We read the combination's gain as two complementary effects. TBA makes the learner robust to whatever mismatch remains, while FARL reduces the mismatch itself. Because TBA's reference here is the sampler, FARL also moves TBA's KL anchor. Under stale generation the regularizer pulls the trainer back toward θs\theta_s; with FARL it anchors at the forecast of θs+1\theta_{s+1}, i.e., toward where the learner is heading rather than where it was.

Scope. We agree these combinations are most attractive when mismatch also comes from low-precision inference. Our setup has only bf16 engine-level mismatch, so we have not tested quantized samplers. This is a limitation of our current study and a promising direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 6AHr (3/3)

Q2: Can you move results with higher staleness levels into the main text and augment them?

Thank you, we agree this matters for readers. We moved the lag study into the main text as a new Sec. 5.3 ("When does FARL help?").


Q3: Could you please clarify the meaning of line 470? (“vanishing at precisely the horizon where the parameter-space forecast stops beating the stale checkpoint…”)

Q3: Meaning of line 470.

Thank you, the sentence was too terse. It compares two measurements taken at each consumption lag dd:

  1. Forecast quality in parameter space. We measure, leakage-free on matured steps, the error ratio Rparam(d)=∥Ss,d−S^s,d∥2∥Ss,d∥2,R^{(d)}_{\text{param}}=\frac{\|S_{s,d}-\widehat S_{s,d}\|^2}{\|S_{s,d}\|^2}, where Ss,d=θs+d−θsS_{s,d}=\theta_{s+d}-\theta_s is the true displacement to the consumption-time policy and S^s,d\widehat S_{s,d} is FARL's forecast. The denominator is the error of stale generation (forecast =0=0). So Rparam(d)<1R^{(d)}_{\text{param}}<1 means the forecasted worker is closer to the consumption-time parameters than the stale checkpoint is, and Rparam(d)>1R^{(d)}_{\text{param}}>1 means it is farther.
  2. Downstream accuracy gain of FARL over Async-GRPO at the same dd.

"Stops beating the stale checkpoint" means Rparam(d)R^{(d)}_{\text{param}} crosses 1. The sentence says that the accuracy gain reaches zero at the same lag where this happens: Rparam(d)R^{(d)}_{\text{param}} rises 0.52→0.71→0.90→1.080.52 \to 0.71 \to 0.90 \to 1.08 for d=1,…,4d=1,\dots,4, while the gain falls +2.5→+1.5→+0.5→+0.0+2.5 \to +1.5 \to +0.5 \to +0.0 (table in our new Sec. 5.3).

Two independent measurements, one in parameter space and one in benchmark accuracy, agree on where forecastability runs out. We see this as evidence that the gain comes from prediction rather than from perturbing the sampler. The matched-norm controls in App. N (Table 12) support this directly: moving the sampler by the same distance in a random direction (42.5) or against the forecast (40.9) does not help, whereas the forecast does (45.8).

"Precisely" was too strong: we only measure integer lags, so the crossing lies between d=3d=3 and d=4d=4, and the gain first reaches zero at d=4d=4. We have reworded this in Sec. 5.2 and App. O accordingly, and the new Sec. 5.3 now presents this relationship with a table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.