- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FARL addresses policy staleness in asynchronous RL by forecasting future learner parameters from recent updates and using the forecasted policy for rollout generation. The reported results support its effectiveness in the evaluated short-delay settings. However, its memory–performance tradeoff and the relationship between parameter prediction and policy alignment require further evaluation.
Critical strengths
S1. The experiments include multiple baselines, ablations, and direct measurements of forecast quality and policy mismatch.
Critical weaknesses
W1. FARL stores five full-sized update vectors, requiring approximately 17 GB of additional GPU memory for the 1.7B model and 40 GB for the 4B model. This is a substantial cost that limits its practicality and scalability.
W2. A small parameter prediction error does not guarantee a small policy distribution error. Further paired token-level KL evaluations are needed to verify that better parameter prediction consistently improves policy alignment. The current Table 8 does not fully establish this point.
Questions for authors
Q1. Could the authors evaluate FARL on larger models and report final benchmark performance, peak GPU memory usage, and parameter prediction error? These results would clarify whether the current predictor remains effective and practical at larger scales.
Q2. Please correct the formatting issues on page 13.
11 comments
Response to Reviewer PAjx (2/3)
W2: A small parameter prediction error does not guarantee a small policy distribution error. Further paired token-level KL evaluations are needed to verify that better parameter prediction consistently improves policy alignment. The current Table 8 does not fully establish this point.
We agree that parameter error is not a policy metric. Our theory states its policy-level results in terms of the Fisher-weighted error , not (Thm. 2). We tested this directly with a paired, token-level evaluation (math workload, 3 seeds, steps 5–24: 60 step–seed pairs per candidate), now in App. T (Tables 18–19).
- At every step , all candidate behavior policies are scored against the consumption-time policy on the same prefixes (256 prompts × 4 rollouts from ).
- Token-level KL and are computed exactly over the full vocabulary with the trainer (no sampling noise, no engine mismatch). Summed over ~650–700 response tokens, the per-token KL matches the trajectory KL of Table 9 (Table 8 in the submission).
- Matched-norm controls and -scaled forecasts test whether alignment comes from prediction rather than from merely moving the sampler.
Table A: Paired token-level alignment at (median over 60 pairs; win rate = fraction of pairs with lower token KL than stale on the same prefixes).
| Model | Behavior policy | Token KL$(p\Vert q)$ | Token | KL ratio vs. stale | Win rate vs. stale | |
|---|---|---|---|---|---|---|
| 1.7B | Stale | 1.00 | 1.00 | — | ||
| 1.7B | Constant velocity | 0.83 | 0.71 | 0.750 (45/60) | ||
| 1.7B | FARL | 0.53 | 0.15 | 0.967 (58/60) | ||
| 1.7B | FARL, | 0.54 | 0.40 | 0.883 (53/60) | ||
| 1.7B | FARL, | 0.99 | 0.29 | 0.833 (50/60) | ||
| 1.7B | Random direction (matched norm) | 1.94 | 1.37 | 0.217 (13/60) | ||
| 1.7B | Anti-forecast (matched norm) | 3.36 | 3.27 | 0.033 (2/60) | ||
| 4B | Stale | 1.00 | 1.00 | — | ||
| 4B | Constant velocity | 0.52 | 0.40 | 0.917 (55/60) | ||
| 4B | FARL | 0.51 | 0.17 | 1.000 (60/60) | ||
| 4B | FARL, | 0.53 | 0.39 | 0.900 (54/60) | ||
| 4B | FARL, | 0.98 | 0.39 | 0.867 (52/60) | ||
| 4B | Random direction (matched norm) | 1.85 | 1.39 | 0.183 (11/60) | ||
| 4B | Anti-forecast (matched norm) | 3.10 | 3.43 | 0.033 (2/60) |
Table B: Does parameter error predict policy error, step by step? Pooled over 360 non-stale (step–seed, candidate) pairs per scale; is a diagonal empirical Fisher from the previous step's trainer gradients. Concordance .
| Model | Spearman(, token KL) | Concordance | Spearman(, token KL) |
|---|---|---|---|
| 1.7B | 0.58 | 92.6% (176/190) | 0.94 |
| 4B | 0.54 | 95.1% (193/203) | 0.96 |
Takeaways.
- Better parameter prediction does give better policy alignment. FARL lowers token KL vs. stale on 58/60 pairs at 1.7B and 60/60 at 4B, cutting the median by 83–85%. Moving the sampler without prediction hurts: matched-norm random and anti-forecast shifts raise token KL (~1.4× and 3.3–3.4×).
- Euclidean error is a reliable coarse test, but not a fine-grained metric, as you suspected. It predicts whether a candidate beats stale (concordance 93–95%) but misses rankings: and have nearly identical (0.54 vs. 0.53) yet differ by >2× in token KL.
- The Fisher-weighted error explains the rest. tracks token KL closely (Spearman 0.94–0.96), as Thms. 2 and 4 predict.
- The full forecast is the right magnitude in policy space. A quadratic fit of token KL over is minimized at (1.7B) and (4B), whereas Euclidean error is minimized near (App. M). The consistency-preserving is thus near-optimal in policy space (Prop. 1).
Response to Reviewer PAjx (3/3)
Questions:
Q1: Could the authors evaluate FARL on larger models and report final benchmark performance, peak GPU memory usage, and parameter prediction error? These results would clarify whether the current predictor remains effective and practical at larger scales.
Following your suggestion, we trained Qwen3-8B with the same configuration on the math workload (, 3 seeds). We report the 5-benchmark math average (MATH-500, AIME24, AIME25, AMC23, AMC24; mean per-seed std):
| Model | Method | Math-5 AVG (%) | Peak GPU allocated / reserved (GB) | Median | Token KL ratio vs. stale | |
|---|---|---|---|---|---|---|
| Qwen3-8B | Async-GRPO | 52.4 / 76.2 | — | — | 1.00 | |
| Qwen3-8B | FARL | (+1.0) | 52.4 / 76.3 | 0.49 | 1.00 | 0.18 |
- The predictor remains effective at 8B. The forecast beats the stale checkpoint on 100% of matured steps (median ) and cuts token-level KL by 82%. FARL improves the Math-5 average by +1.0, more than twice the per-seed standard deviation ().
- It remains practical at 8B. Peak GPU memory is unchanged relative to Async-GRPO, because the update history is kept in host memory (see W1).
- Host memory scales linearly, to about 164 GB at 8B with our unsharded fp32 implementation. As noted in W1, storing the updates in bf16 reduces this to about 98 GB, and sharding the history across the 4 trainer ranks brings it to about 41 GB per rank (fp32) or 25 GB per rank (bf16).
Q2. Please correct the formatting issues on page 13.
Thank you for catching this. The issue is resolved now in the revised version.
If any concerns remain, please let us know and we will gladly address them during the discussion period. If our responses have resolved your concerns, we would be grateful if you would consider raising your score.
Response to Reviewer PAjx (1/3)
We thank the reviewer for the careful assessment and constructive feedback. We particularly appreciate the concerns regarding FARL's scalability and the relationship between parameter prediction and policy alignment. These are important aspects of our work, and we address them below through implementation clarifications and additional empirical analysis.
Weaknesses:
W1: Memory cost of the update buffer.
Thank you for raising this concern. On re-checking our implementation, we found that Table 6 of the submission incorrectly described the buffer as GPU-resident and sharded in bf16. In our implementation, the update history is stored in host (CPU) memory, not GPU memory. We have corrected this in the revision (Table 7 and App. H), and we verified it from our training logs.
At each weight synchronization, the rank-0 trainer copies the parameters to host memory, computes the forecast on the CPU, and broadcasts it to the rollout workers. All methods use identical batch sizes and GPU memory settings (Table 4).
Scalability. The host-memory footprint scales linearly with model size and history length, and can be reduced in three ways:
The revision states the actual memory placement, the measured GPU and host memory, and these memory–accuracy trade-offs explicitly (App. H, Table 7).