- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CFA, a content-free anchoring method for mitigating capability forgetting during RL post-training.It uses random token sequences and a Fisher-weighted embedding-space MMD penalty to constrain the current model toward a frozen reference model, and evaluates forgetting with the proposed worst-k curve.
Critical strengths
-
The paper identifies that suite-wide averages can hide severe capability-specific forgetting during single-domain RL post-training and proposes the worst-k curve as a useful tail-risk diagnostic.
-
CFA consistently improves retention across three model scales and multiple domains without requiring real, licensed, or curated anchor data.
Critical weaknesses
-
The theoretical claim about anchor content may be overstated: Finite-sample concentration around an anchor-distribution-specific quantity does not by itself show that random anchors preserve arbitrary real-world capabilities. The connection between random-token anchors and deployment-task behavior remains insufficiently established.
-
Limited generality and evaluation scope: The experiments focus on one model family, three domains, and a single multimodal evaluation suite. It remains unclear whether CFA generalizes to text-only models, other architectures, different post-training algorithms, or longer sequential training.
Questions for authors
-
How sensitive are the results to the choice of benchmark suite? Does CFA still outperform the baselines when forgetting is evaluated on different or expanded OOD benchmarks?
-
Can the authors evaluate it on other model families to demonstrate the robustness and generality of the method?
5 comments
Response to Reviewer V9L2 (Part 2/4)
What the theory does not claim. Theorem 2 states the limit: content-free anchors remove collateral drift, but not interference with capabilities that directly conflict with the target task. Whether is small, not merely finite, for a given real capability remains an empirical question. Every assumption is stated explicitly (App. D.5), and Table 25 lists which result uses which. The abstract and conclusion now claim exactly this, rather than that anchor content never matters.
Connecting to real behavior. cannot be computed exactly at multi-billion-parameter scale, so we validate the chain in a fully observable testbed (App. D.8):
- , evaluated at the pretrained parameters on the subspace spanned by the training updates, rank-predicts each capability's forgetting (Spearman ρ = 0.93).
- With measured remainders, the bound of Theorem 1(vi) holds at every checkpoint of all 1000 RL steps, for every capability, anchor set, and seed (8,000 pairs).
- On every capability outside the curated pool, random anchors reach a lower floor than curated anchors (2.7 vs. 7.9 and 9.4 vs. 88 in the two settings), and they lose only where Theorems 2–3 predict.
On the real models, the link to deployment behavior is measured directly. Retention is evaluated on benchmarks the anchors never see: three VLM families, four text-only models, and sixteen VLM benchmarks in total (below). The extended evaluation also shows the pattern Theorem 3 predicts. Curated anchors retain the capability their pool represents (OCRBench) but lose one it does not (AI2D), while random anchors stay close to base on both.
Revision in the paper: abstract, conclusion, Sec. 3.5, Remark 2, App. D.4–D.8, Table 25.
Q2 / W2: Can CFA be evaluated on other model families, architectures, and post-training algorithms?
1. Other VLM families (new Sec. 5.4, Fig. 8, App. C.5). We applied CFA unchanged (, ) to InternVL3.5-1B and Gemma 4 E2B, which differ from Qwen3-VL in both vision encoder and language backbone. Every method is trained on MedBookVQA for 600 steps and evaluated on the common ten-benchmark suite. Without the compute for per-method sweeps, we use one learning rate per training type, so in-domain accuracy could likely be improved slightly by tuning.
Table R4. Other VLM families (MedBookVQA, 600 steps). Avg., worst-1, and worst-3 are relative % change vs. base over the ten benchmarks; target is in-domain accuracy.
| Model | Method | Target | Avg. | Worst-1 | Worst-3 |
|---|---|---|---|---|---|
| InternVL3.5-1B | Base | 45.7 | — | — | — |
| GRPO | 75.0 | −28.2 | −56.6 | −45.8 | |
| GRPO+KL | 72.5 | −6.8 | −25.6 | −19.3 | |
| GSPO | 77.9 | −10.1 | −43.1 | −29.6 | |
| SFT | 71.8 | −30.6 | −78.4 | −55.5 | |
| LoRA | 58.6 | −12.9 | −29.1 | −27.0 | |
| GSPO + CFA | 78.0 | +1.3 | −5.1 | −2.3 | |
| Gemma 4 E2B | Base | 52.8 | — | — | — |
| GRPO | 79.1 | +2.5 | −13.4 | −9.0 | |
| GRPO+KL | 73.5 | −26.5 | −52.7 | −42.6 | |
| GSPO | 68.7 | −3.8 | −48.0 | −25.4 | |
| SFT | 68.6 | −32.5 | −52.4 | −49.3 | |
| LoRA | 65.6 | −4.6 | −12.0 | −9.0 | |
| GSPO + CFA | 77.0 | +7.7 | −3.6 | −1.1 |
On both families, CFA has the smallest worst-k forgetting at every k, and it is the only method whose largest drop stays within about 5% of base.
- Relative to GSPO, the algorithm it regularizes, CFA cuts worst-1 forgetting from −43.1% to −5.1% and from −48.0% to −3.6%, while matching or exceeding its in-domain accuracy. On InternVL3.5-1B it reaches the highest in-domain accuracy of any method.
- Specific capabilities are preserved: on InternVL3.5-1B, CFA keeps CharXiv near base, where every other method loses 25–43%; on Gemma 4 E2B, it keeps CountBenchQA at base, where GSPO loses 48%.
As on Qwen3-VL, the strongest baseline changes from model to model. GRPO+KL is runner-up on InternVL3.5-1B but loses 52.7% on Gemma 4 E2B, while GRPO forgets little on Gemma 4 E2B but loses 56.6% on InternVL3.5-1B. CFA is the only method that is safe on both.
2. Text-only models (App. C.6, Tables 19–21). CFA needs no changes for text-only models, since its anchors are already random text tokens. We ran RL on four text-only models from two families, including a mixture-of-experts model, with and without CFA. Text-only RL forgets much less than VLM RL (at most −7.2% worst-1, against up to −67.4% for Qwen3-VL), and CFA reduces even this, at unchanged target accuracy:
Response to Reviewer V9L2 (Part 3/4)
Table R3. Text-only RL with and without CFA. Worst-1 is the largest relative drop vs. base over held-out benchmarks (HellaSwag, MMLU, GSM8K, Winogrande, ARC-C, and others).
| Model | RL task | Target (RL) | Target (RL + CFA) | Worst-1 (RL) | Worst-1 (RL + CFA) |
|---|---|---|---|---|---|
| Qwen2.5-1.5B-Instruct | Arithmetic | 80.0 | 79.7 | −0.9% | −0.2% |
| Qwen2.5-0.5B-Instruct | Medical ontology | 80.0 | 80.4 | −6.2% | −1.5% |
| Qwen2.5-3B-Instruct | Medical ontology | 52.8 | 52.7 | −7.2% | −1.7% |
| Qwen2.5-3B-Instruct | Search query extraction | 85.0 | 85.4 | −4.2% | −2.7% |
| Phi-tiny-MoE-Instruct | Search query extraction | 42.7 | 42.5 | −0.1% | −0.3% |
Where RL alone forgets almost nothing (Phi-tiny-MoE), CFA leaves retention unchanged within noise. This matches the theory: CFA removes collateral drift, and there is little of it to remove in these runs.
3. Other post-training algorithms (Table 13). The main experiments already use two RL algorithms (GRPO and GSPO). Because the anchor term depends only on the model's outputs on anchors, it can be added unchanged to any training loss. On MedBookVQA with Qwen3-VL-2B:
- SFT, the most destructive method in our study, has a worst-1 change of −78.8%. Adding CFA reduces it to −9.1% while keeping most of the target gain (92.5 vs. 95.1).
- LoRA learns little of the task in this setting and so forgets little; adding CFA leaves target accuracy unchanged (51.3 vs. 50.8).
4. Longer sequential training. We agree this is not covered, and the Limitations (App. J) now say so. This limit is shared by every regularizer that keeps the model close to a reference, whether KL, EWC, replay, or anchoring: each gives a weaker guarantee as training moves further from the reference. Within our single-domain runs of up to 600 steps, CFA's guarantee stays tight: the anchor-side factor of the any-step bound stays at most 1.08 (Theorem 1(vi); Table 24). Over many sequential tasks this factor could grow, but because CFA evaluates the exact anchor loss rather than a frozen approximation, the growth is visible during training rather than silent.
Revision in the paper: Sec. 5.4, Fig. 8, App. C.5 (Tables 17–18); App. C.6 (Tables 19–21); Table 13; App. J.
Q1: How sensitive are the results to the choice of benchmark suite? Does CFA still outperform the baselines on different or expanded OOD benchmarks?
The anchors are random tokens drawn without reference to any benchmark, and every method's hyperparameters, including CFA's, are selected by in-domain accuracy only (Sec. 5.1). The evaluation suite plays no role in training or selection, so CFA's advantage is not an artifact of fitting it. The worst-k curve reports the k worst benchmarks for every k, rather than a fixed benchmark or the average.
We evaluate the Navigation, Qwen3-VL-2B checkpoints of every qualifying method on six benchmarks outside the common suite, covering capabilities it does not test (new App. C.8, Table 23):
- hallucination (HallusionBench, aAcc);
- text recognition (OCRBench, out of 1000);
- diagram understanding (AI2D);
- high-resolution perception (HRBench-4K);
- GUI grounding (ScreenSpot);
- embodied spatial reasoning (EmbSpatialBench).
No training or selection is repeated, and all checkpoints use the same decoding settings.
Table R5. Expanded OOD evaluation (Navigation, Qwen3-VL-2B). Accuracy ± half the 95% bootstrap CI. W-1 and W-3 are worst-1 and worst-3 relative change vs. base (%), over the six new benchmarks and over all sixteen (common suite plus the six).
| Method | Hallusion | OCRBench | AI2D | HRBench-4K | ScreenSpot | EmbSpatial | W-1 (6 new) | W-3 (6 new) | W-1 (all 16) | W-3 (all 16) |
|---|---|---|---|---|---|---|---|---|---|---|
| Base | 70.35 ± 3.05 | 691.0 ± 28.5 | 72.31 ± 1.64 | 52.25 ± 3.44 | 76.26 ± 2.32 | 69.15 ± 1.48 | — | — | — | — |
| GRPO (wd=0.1) | 71.19 ± 2.89 | 615.0 ± 30.5 | 71.53 ± 1.64 | 57.88 ± 3.50 | 79.25 ± 2.24 | 73.76 ± 1.40 | −11.0 | −3.6 | −40.5 | −21.6 |
| GRPO+KL | 67.93 ± 3.00 | 570.0 ± 31.0 | 71.34 ± 1.67 | 47.00 ± 3.38 | 48.03 ± 2.71* | 72.77 ± 1.46 | −37.0 | −21.5 | −37.0 | −29.3 |
| GSPO | 69.82 ± 2.95 | 611.0 ± 30.0 | 71.99 ± 1.60 | 51.62 ± 3.56 | 54.09 ± 2.67* | 69.95 ± 1.47 | −29.1 | −14.0 | −29.1 | −23.5 |
| CPO | 68.14 ± 2.95 | 441.0 ± 31.5 | 72.34 ± 1.68 | 53.87 ± 3.44 | 67.22 ± 2.59* | 70.14 ± 1.47 | −36.2 | −17.1 | −36.2 | −26.2 |
| RDA | 70.14 ± 3.05 | 685.0 ± 29.0 | 67.62 ± 1.65 | 51.75 ± 3.44 | 53.46 ± 2.71* | 74.07 ± 1.41 | −29.9 | −12.4 | −29.9 | −17.5 |
| CFA | 69.72 ± 2.94 | 640.0 ± 30.0 | 72.93 ± 1.65 | 52.62 ± 3.38 | 81.13 ± 2.16 | 70.49 ± 1.48 | −7.4 | −2.5 | −7.4 | −4.4 |
* These checkpoints answer ScreenSpot's direct grounding prompt with step-by-step reasoning, and reach the 256-token cap on 5–22% of items (≤ 0.5% for every other method), so their ScreenSpot drop is partly a truncation effect.
Response to Reviewer V9L2 (Part 1/4)
We thank Reviewer V9L2 for the positive assessment. We are glad they found that the worst-k curve exposes severe capability-specific forgetting hidden by suite-wide averages, and that CFA consistently improves retention across scales and domains without real, licensed, or curated anchor data. The review raised two concerns: whether the theory supports the claim about anchor content, and how far the results generalize. We address both with a sharpened theory and new experiments on other model families, text-only models, other post-training algorithms, and an expanded benchmark suite.
In brief.
Summary
We agree, and the revised theory supplies exactly the missing link, in three steps.
Estimation (Props. 5–7). A finite anchor set converges to its own population curvature Gν at an N−1/2 rate. As the reviewer notes, this alone says nothing about real tasks, and the paper now presents it only as estimation.
Transfer (Theorem 1). Drift Fτ on a real capability τ, measured on its own real inputs, is bounded by the anchor loss Lν through a transfer coefficient: Fτ≤κS(τ∥ν)Lν,κS(τ∥ν)=supu∈Su⊤Gνuu⊤Gτu, where S is the subspace in which training moves. This holds to second order, and at any training step once measured remainder factors are included (Thm. 1(vi)). Anchor content matters only through κS.
How much content can matter (Theorem 3, new). For any curated anchor set ν and random anchors μ, every capability satisfies κS(τ∥μ)≤cS(ν∥μ)κS(τ∥ν). The factor cS depends only on the two anchor sets and the update subspace, and uses no held-out data.
Content can therefore help, but only by a bounded factor, while missing coverage can cost an unbounded amount. Isotropic anchors minimize the worst case when the at-risk capability is unknown, and uniform random tokens attain this up to an explicit factor computable from the embedding table (Prop. 9).