acceptodds
Under review as a conference paper at ICLR 2027

CFA: Mitigating Forgetting in RL Post-Training with Content-Free Anchors

Abstract

RL post-training is deployed under the assumption that it forgets little of what it was not trained on. We show this is incomplete: across Qwen3-VL at 2B, 4B, and 8B scale, single-domain RL post-training silently collapses held-out capability, with worst-decile drops up to −67% hidden behind suite-wide averages that stay flat only because unrelated benchmarks improve on the same run. We introduce the worst-k curve to expose this tail. Every existing defense, parameter-importance penalties, replay, KL toward a reference, assumes the anchor set used to constrain the model must represent the real capabilities being protected. Protecting a handful of anticipated domains this way is expensive; protecting general capability this way is infeasible, since it requires anchoring on essentially the model’s entire pretraining distribution. We find this assumption is not load-bearing: anchors drawn from pure random noise, with no connection to any capability, match and at modest budgets outperform a curated real coreset spanning five capability dimensions. We instantiate this as CFA, a Fisher-weighted Maximum Mean Discrepancy penalty between current and reference next-token distributions, which we derive as a Gauss-Newton generalization of Elastic Weight Consolidation whose finite-sample guarantees hold regardless of anchor content. Across all nine domain-scale combinations tested, CFA holds worst-decile forgetting to −0.3% to −9.1%, against −8.1% to −67.4% for the best baseline, while matching or exceeding in-domain gain, using no real, licensed, or curated data at all.

Primary area:
reinforcement learning
Keywords:
RL post-training, catastrophic forgetting, vision language models, content-free anchors

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_V9L2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CFA, a content-free anchoring method for mitigating capability forgetting during RL post-training.It uses random token sequences and a Fisher-weighted embedding-space MMD penalty to constrain the current model toward a frozen reference model, and evaluates forgetting with the proposed worst-k curve.

Critical strengths
  1. The paper identifies that suite-wide averages can hide severe capability-specific forgetting during single-domain RL post-training and proposes the worst-k curve as a useful tail-risk diagnostic.

  2. CFA consistently improves retention across three model scales and multiple domains without requiring real, licensed, or curated anchor data.

Critical weaknesses
  1. The theoretical claim about anchor content may be overstated: Finite-sample concentration around an anchor-distribution-specific quantity does not by itself show that random anchors preserve arbitrary real-world capabilities. The connection between random-token anchors and deployment-task behavior remains insufficiently established.

  2. Limited generality and evaluation scope: The experiments focus on one model family, three domains, and a single multimodal evaluation suite. It remains unclear whether CFA generalizes to text-only models, other architectures, different post-training algorithms, or longer sequential training.

Questions for authors
  1. How sensitive are the results to the choice of benchmark suite? Does CFA still outperform the baselines when forgetting is evaluated on different or expanded OOD benchmarks?

  2. Can the authors evaluate it on other model families to demonstrate the robustness and generality of the method?

5 comments
AuthorsOpenReview ↗

Response to Reviewer V9L2 (Part 1/4)

We thank Reviewer V9L2 for the positive assessment. We are glad they found that the worst-k curve exposes severe capability-specific forgetting hidden by suite-wide averages, and that CFA consistently improves retention across scales and domains without real, licensed, or curated anchor data. The review raised two concerns: whether the theory supports the claim about anchor content, and how far the results generalize. We address both with a sharpened theory and new experiments on other model families, text-only models, other post-training algorithms, and an expanded benchmark suite.

In brief.

  1. Theory. We agree that concentration alone does not imply retention. The revision adds the missing transfer step (Theorem 1), a new exchange bound showing that anchor content can improve on random anchors by at most a factor computable without held-out data (Theorem 3), and an explicit list of assumptions. The abstract and conclusion now claim only what is proven.
  2. Other model families. On InternVL3.5-1B and Gemma 4 E2B, two families with different vision encoders and language backbones, CFA again has the smallest worst-case forgetting of every method (worst-1 −5.1% and −3.6%, against −43.1% and −48.0% for GSPO alone), at equal or higher target accuracy than GSPO.
  3. Other models and algorithms. CFA transfers unchanged to four text-only models (two families, including a mixture-of-experts model) and to SFT, where it cuts worst-1 forgetting from −78.8% to −9.1%.
  4. Other benchmarks. CFA is never tuned on the evaluation suite. On six additional VLM benchmarks outside the common suite, it has the smallest worst-1 (−7.4%) and worst-3 (−2.5%) change of every method, against −11.0% to −37.0% worst-1 for the RL baselines and −29.9% for curated anchors (RDA). It also reduces forgetting on an entirely different text-only suite.

Summary

Reviewer concernHow we addressed itWhere in the revised PDF
W1. Concentration does not show that random anchors preserve real capabilitiesTransfer theorem; exchange bound on the value of content; explicit assumptions; testbed validation; claims narrowedSec. 3.5, Remark 2, App. D.5–D.8, Table 25
Q2 / W2. Other model families and architecturesCFA on InternVL3.5-1B and Gemma 4 E2BSec. 5.4, Fig. 8, App. C.5 (Tables 17–18)
W2. Text-only models; other post-training algorithmsFour text-only models; SFT and LoRAApp. C.6 (Tables 19–21), Table 13
Q1 / W2. Sensitivity to the benchmark suiteNo suite-dependent tuning; six additional VLM benchmarks; a different text-only suiteSec. 5.1, App. C.8 (Table 23), App. C.6
W2. Longer sequential trainingStated as a limitation, with the scope of our evidenceApp. J

W1: Finite-sample concentration around an anchor-specific quantity does not show that random anchors preserve real-world capabilities.

We agree, and the revised theory supplies exactly the missing link, in three steps.

  1. Estimation (Props. 5–7). A finite anchor set converges to its own population curvature GνG_\nu at an N−1/2N^{-1/2} rate. As the reviewer notes, this alone says nothing about real tasks, and the paper now presents it only as estimation.

  2. Transfer (Theorem 1). Drift FτF_\tau on a real capability τ\tau, measured on its own real inputs, is bounded by the anchor loss LνL_\nu through a transfer coefficient: Fτ≤κS(τ∥ν) Lν,κS(τ∥ν)=sup⁡u∈Su⊤Gτuu⊤Gνu,F_\tau \le \kappa_S(\tau\|\nu)\, L_\nu, \qquad \kappa_S(\tau\|\nu)=\sup_{u\in S} \frac{u^\top G_\tau u}{u^\top G_\nu u}, where SS is the subspace in which training moves. This holds to second order, and at any training step once measured remainder factors are included (Thm. 1(vi)). Anchor content matters only through κS\kappa_S.

  3. How much content can matter (Theorem 3, new). For any curated anchor set ν\nu and random anchors μ\mu, every capability satisfies κS(τ∥μ)≤cS(ν∥μ) κS(τ∥ν)\kappa_S(\tau\|\mu)\le c_S(\nu\|\mu)\,\kappa_S(\tau\|\nu). The factor cSc_S depends only on the two anchor sets and the update subspace, and uses no held-out data.

    • The bound is tight: it is attained on the capability the curated set itself represents.
    • The reverse gap is unbounded: a curated set has κS=∞\kappa_S=\infty on any capability that depends on a direction outside its pool.

    Content can therefore help, but only by a bounded factor, while missing coverage can cost an unbounded amount. Isotropic anchors minimize the worst case when the at-risk capability is unknown, and uniform random tokens attain this up to an explicit factor computable from the embedding table (Prop. 9).

AuthorsOpenReview ↗

Response to Reviewer V9L2 (Part 2/4)

What the theory does not claim. Theorem 2 states the limit: content-free anchors remove collateral drift, but not interference with capabilities that directly conflict with the target task. Whether κS\kappa_S is small, not merely finite, for a given real capability remains an empirical question. Every assumption is stated explicitly (App. D.5), and Table 25 lists which result uses which. The abstract and conclusion now claim exactly this, rather than that anchor content never matters.

Connecting to real behavior. κS\kappa_S cannot be computed exactly at multi-billion-parameter scale, so we validate the chain in a fully observable testbed (App. D.8):

  • κS\kappa_S, evaluated at the pretrained parameters on the subspace spanned by the training updates, rank-predicts each capability's forgetting (Spearman ρ = 0.93).
  • With measured remainders, the bound of Theorem 1(vi) holds at every checkpoint of all 1000 RL steps, for every capability, anchor set, and seed (8,000 pairs).
  • On every capability outside the curated pool, random anchors reach a lower floor than curated anchors (2.7 vs. 7.9 and 9.4 vs. 88 in the two settings), and they lose only where Theorems 2–3 predict.

On the real models, the link to deployment behavior is measured directly. Retention is evaluated on benchmarks the anchors never see: three VLM families, four text-only models, and sixteen VLM benchmarks in total (below). The extended evaluation also shows the pattern Theorem 3 predicts. Curated anchors retain the capability their pool represents (OCRBench) but lose one it does not (AI2D), while random anchors stay close to base on both.

Revision in the paper: abstract, conclusion, Sec. 3.5, Remark 2, App. D.4–D.8, Table 25.


Q2 / W2: Can CFA be evaluated on other model families, architectures, and post-training algorithms?

1. Other VLM families (new Sec. 5.4, Fig. 8, App. C.5). We applied CFA unchanged (N=1280N=1280, T=2048T=2048) to InternVL3.5-1B and Gemma 4 E2B, which differ from Qwen3-VL in both vision encoder and language backbone. Every method is trained on MedBookVQA for 600 steps and evaluated on the common ten-benchmark suite. Without the compute for per-method sweeps, we use one learning rate per training type, so in-domain accuracy could likely be improved slightly by tuning.

Table R4. Other VLM families (MedBookVQA, 600 steps). Avg., worst-1, and worst-3 are relative % change vs. base over the ten benchmarks; target is in-domain accuracy.

ModelMethodTargetAvg.Worst-1Worst-3
InternVL3.5-1BBase45.7———
GRPO75.0−28.2−56.6−45.8
GRPO+KL72.5−6.8−25.6−19.3
GSPO77.9−10.1−43.1−29.6
SFT71.8−30.6−78.4−55.5
LoRA58.6−12.9−29.1−27.0
GSPO + CFA78.0+1.3−5.1−2.3
Gemma 4 E2BBase52.8———
GRPO79.1+2.5−13.4−9.0
GRPO+KL73.5−26.5−52.7−42.6
GSPO68.7−3.8−48.0−25.4
SFT68.6−32.5−52.4−49.3
LoRA65.6−4.6−12.0−9.0
GSPO + CFA77.0+7.7−3.6−1.1

On both families, CFA has the smallest worst-k forgetting at every k, and it is the only method whose largest drop stays within about 5% of base.

  • Relative to GSPO, the algorithm it regularizes, CFA cuts worst-1 forgetting from −43.1% to −5.1% and from −48.0% to −3.6%, while matching or exceeding its in-domain accuracy. On InternVL3.5-1B it reaches the highest in-domain accuracy of any method.
  • Specific capabilities are preserved: on InternVL3.5-1B, CFA keeps CharXiv near base, where every other method loses 25–43%; on Gemma 4 E2B, it keeps CountBenchQA at base, where GSPO loses 48%.

As on Qwen3-VL, the strongest baseline changes from model to model. GRPO+KL is runner-up on InternVL3.5-1B but loses 52.7% on Gemma 4 E2B, while GRPO forgets little on Gemma 4 E2B but loses 56.6% on InternVL3.5-1B. CFA is the only method that is safe on both.

2. Text-only models (App. C.6, Tables 19–21). CFA needs no changes for text-only models, since its anchors are already random text tokens. We ran RL on four text-only models from two families, including a mixture-of-experts model, with and without CFA. Text-only RL forgets much less than VLM RL (at most −7.2% worst-1, against up to −67.4% for Qwen3-VL), and CFA reduces even this, at unchanged target accuracy:

AuthorsOpenReview ↗

Response to Reviewer V9L2 (Part 3/4)

Table R3. Text-only RL with and without CFA. Worst-1 is the largest relative drop vs. base over held-out benchmarks (HellaSwag, MMLU, GSM8K, Winogrande, ARC-C, and others).

ModelRL taskTarget (RL)Target (RL + CFA)Worst-1 (RL)Worst-1 (RL + CFA)
Qwen2.5-1.5B-InstructArithmetic80.079.7−0.9%−0.2%
Qwen2.5-0.5B-InstructMedical ontology80.080.4−6.2%−1.5%
Qwen2.5-3B-InstructMedical ontology52.852.7−7.2%−1.7%
Qwen2.5-3B-InstructSearch query extraction85.085.4−4.2%−2.7%
Phi-tiny-MoE-InstructSearch query extraction42.742.5−0.1%−0.3%

Where RL alone forgets almost nothing (Phi-tiny-MoE), CFA leaves retention unchanged within noise. This matches the theory: CFA removes collateral drift, and there is little of it to remove in these runs.

3. Other post-training algorithms (Table 13). The main experiments already use two RL algorithms (GRPO and GSPO). Because the anchor term depends only on the model's outputs on anchors, it can be added unchanged to any training loss. On MedBookVQA with Qwen3-VL-2B:

  • SFT, the most destructive method in our study, has a worst-1 change of −78.8%. Adding CFA reduces it to −9.1% while keeping most of the target gain (92.5 vs. 95.1).
  • LoRA learns little of the task in this setting and so forgets little; adding CFA leaves target accuracy unchanged (51.3 vs. 50.8).

4. Longer sequential training. We agree this is not covered, and the Limitations (App. J) now say so. This limit is shared by every regularizer that keeps the model close to a reference, whether KL, EWC, replay, or anchoring: each gives a weaker guarantee as training moves further from the reference. Within our single-domain runs of up to 600 steps, CFA's guarantee stays tight: the anchor-side factor of the any-step bound stays at most 1.08 (Theorem 1(vi); Table 24). Over many sequential tasks this factor could grow, but because CFA evaluates the exact anchor loss rather than a frozen approximation, the growth is visible during training rather than silent.

Revision in the paper: Sec. 5.4, Fig. 8, App. C.5 (Tables 17–18); App. C.6 (Tables 19–21); Table 13; App. J.


Q1: How sensitive are the results to the choice of benchmark suite? Does CFA still outperform the baselines on different or expanded OOD benchmarks?

The anchors are random tokens drawn without reference to any benchmark, and every method's hyperparameters, including CFA's, are selected by in-domain accuracy only (Sec. 5.1). The evaluation suite plays no role in training or selection, so CFA's advantage is not an artifact of fitting it. The worst-k curve reports the k worst benchmarks for every k, rather than a fixed benchmark or the average.

We evaluate the Navigation, Qwen3-VL-2B checkpoints of every qualifying method on six benchmarks outside the common suite, covering capabilities it does not test (new App. C.8, Table 23):

  • hallucination (HallusionBench, aAcc);
  • text recognition (OCRBench, out of 1000);
  • diagram understanding (AI2D);
  • high-resolution perception (HRBench-4K);
  • GUI grounding (ScreenSpot);
  • embodied spatial reasoning (EmbSpatialBench).

No training or selection is repeated, and all checkpoints use the same decoding settings.

Table R5. Expanded OOD evaluation (Navigation, Qwen3-VL-2B). Accuracy ± half the 95% bootstrap CI. W-1 and W-3 are worst-1 and worst-3 relative change vs. base (%), over the six new benchmarks and over all sixteen (common suite plus the six).

MethodHallusionOCRBenchAI2DHRBench-4KScreenSpotEmbSpatialW-1 (6 new)W-3 (6 new)W-1 (all 16)W-3 (all 16)
Base70.35 ± 3.05691.0 ± 28.572.31 ± 1.6452.25 ± 3.4476.26 ± 2.3269.15 ± 1.48————
GRPO (wd=0.1)71.19 ± 2.89615.0 ± 30.571.53 ± 1.6457.88 ± 3.5079.25 ± 2.2473.76 ± 1.40−11.0−3.6−40.5−21.6
GRPO+KL67.93 ± 3.00570.0 ± 31.071.34 ± 1.6747.00 ± 3.3848.03 ± 2.71*72.77 ± 1.46−37.0−21.5−37.0−29.3
GSPO69.82 ± 2.95611.0 ± 30.071.99 ± 1.6051.62 ± 3.5654.09 ± 2.67*69.95 ± 1.47−29.1−14.0−29.1−23.5
CPO68.14 ± 2.95441.0 ± 31.572.34 ± 1.6853.87 ± 3.4467.22 ± 2.59*70.14 ± 1.47−36.2−17.1−36.2−26.2
RDA70.14 ± 3.05685.0 ± 29.067.62 ± 1.6551.75 ± 3.4453.46 ± 2.71*74.07 ± 1.41−29.9−12.4−29.9−17.5
CFA69.72 ± 2.94640.0 ± 30.072.93 ± 1.6552.62 ± 3.3881.13 ± 2.1670.49 ± 1.48−7.4−2.5−7.4−4.4

* These checkpoints answer ScreenSpot's direct grounding prompt with step-by-step reasoning, and reach the 256-token cap on 5–22% of items (≤ 0.5% for every other method), so their ScreenSpot drop is partly a truncation effect.

Reviewer_bL9NOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that single-domain RL post-training of VLMs (Qwen3-VL 2B/4B/8B on three MRCL domains) causes severe forgetting on individual held-out benchmarks, such as CountBenchQA and DocVQA. Suite-wide averages hide this, so the authors introduce a “worst-k” curve to expose the tail. Their mitigation, CFA, penalizes a varentropy-weighted, linear-kernel MMD between the policy’s and a frozen reference’s expected token embeddings on a fixed anchor set. The central claim is that the anchors can be content-free noise rather than curated data. A Gauss-Newton analysis frames CFA as a generalization of EWC, supported by concentration bounds and a predicted budget “crossover” at which noise anchors overtake curated ones. Validation includes a toy MLP testbed, a 3×3 domain-scale grid against GRPO/GSPO, KL, EWC, CPO, SFT, LoRA and WiSE-FT baselines, and anchor-budget and divergence ablations. The paper reports that CFA keeps worst-benchmark degradation within −0.3% to −9.1% while matching in-domain gains.

Critical strengths
  1. A practically important, well-motivated problem with a useful reporting practice. Tail forgetting hidden by suite averages is a real deployment risk for RL post-training. The full-curve presentation is simple, and Appendix B argues convincingly for it: if one method’s curve lies above another’s at every k, the dominance holds for every risk-averse aggregate, not just at one cherry-picked k. The full per-benchmark tables are also a strength; they made it possible to verify the claims independently.

  2. A striking empirical observation, if it holds. On Navigation-2B, uninformative anchors retain CountBenchQA better than a curated coreset that explicitly includes a counting/grounding dimension (79.6 vs 70.9). Random anchors also beat real anchors under every divergence in Table 6. If robust, “anchor content is not load-bearing” would be a useful and cheap result for practitioners. This finding is the main reason the paper is interesting.

  3. A cheap, data-free regularizer that is easy to adopt. CFA needs no curated, licensed or stored prior-task data. The frozen reference is evaluated once on a fixed anchor set, and only d-dimensional expected-embedding vectors are cached, so the reference model does not need to stay on the GPU during training (Section 3.4, Appendix G). It is implemented as a small trainer subclass on top of an existing GRPO/GSPO codebase (Algorithm 2). This makes it immediately usable by practitioners who cannot assemble representative replay data for the many capabilities a broad pretrained model has. This matters to my assessment, because the barrier to adoption is unusually low compared with replay, EWC, or KL penalties that need curated anchor data.

Critical weaknesses
  1. The headline claims don’t match the paper’s own tables, and the results are too noisy to support them. The abstract says the best baseline loses between −8.1% and −67.4% on its worst benchmark, but the appendix tables show otherwise:

    • MedBookVQA-2B: GSPO+EWC qualifies and loses only about −0.2%, the same as CFA (−0.3%). The −67.4% figure is the worst baseline in that cell, not the best.
    • Navigation-4B: GRPO loses only −1.2% (Table 13). No method needs mitigation in this cell.
    • Puzzle-4B: GRPO+KL (≈ −7.8%) is close to CFA (−5.5%) and beats RDA (≈ −11.5%).

    In addition, every result appears to come from a single run. The worst benchmark out of 10 noisy scores is especially sensitive to noise, and many of the deciding gaps are within the reported ±2–4 points. The rule for excluding baselines that “didn’t learn the task” is never specified. The evidence supports “CFA is consistently among the best.” It does not support “CFA beats every baseline everywhere.”

  2. It is unclear what the anchors actually are, and the theory does not explain why random anchors work. The paper describes the content-free anchors as random text tokens (Sections 3.3 and F, Algorithm 2) and also as Gaussian noise (Section 3.4, Appendix I). This matters because the forgetting is in visual skills: text-only anchors would never pass through the vision encoder. There are also confounds and contradictions:

    • Real anchors are short sequences padded to 2048 tokens, while random anchors contribute 2048 scored tokens each. The real-versus-random comparison may therefore reflect token count rather than content.
    • Algorithm 2 and Appendix G contradict each other on whether the reference model runs during training.
  3. On the theory side, the concentration bounds only show that any i.i.d. anchor set converges to its own average. They say nothing about whether that average protects capabilities, as Appendix D.4 itself admits. Yet the abstract and conclusion present the theory as explaining why anchor content does not matter.

Questions for authors
  1. Exactly what are the content-free anchors in the Qwen3-VL experiments: text tokens only, noise images, or both? Is the vision encoder trainable during RL? If anchors are text-only and the vision tower is trainable, how does CFA protect visual grounding?
  2. Is the reference model called during training (Algorithm 2) or not (Appendix G.4)? What are the true per-step FLOPs, including the anchor backward pass, and the measured wall-clock overhead?
  3. What is the precise in-domain-gain exclusion rule, and was it fixed before looking at OOD results?
  4. How do you reconcile GSPO+EWC on MedBookVQA-2B (≈ −0.2%, qualifying), GRPO (wd=0.1) on Navigation-4B (−1.2%) and GRPO+KL on Puzzle-4B (≈ −7.8%) with the abstract’s “−8.1% to −67.4% for the best baseline”?
  5. How many scored tokens do real and content-free anchors each contribute at equal N? Does the crossover survive at matched token budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer bL9N (Part 1/4)

We thank Reviewer bL9N for their thorough review, and for checking our claims against the per-benchmark tables. We are glad they found the problem practically important, the worst-k reporting convincing, the random-vs-curated result striking, and the method unusually easy to adopt. The reviewer identified the finding that anchor content is not load-bearing as the main reason the paper is interesting, conditional on it being robust. The revision is built around testing exactly that. We bring every claim in line with the tables, and test the finding against each threat the reviewer raised: seed noise, token count, the theory, and model family.

In brief,

  1. Headline claims now match the tables. The abstract no longer says "best baseline". It states the claim the reviewer's own reading supports: CFA is the only method without a severe collapse in any of the nine cells. We now report each method's worst case across cells.
  2. The finding survives seed noise. Three seeds per qualifying method leave the ranking unchanged. The CFA–RDA gap (about 10 points worst-1) is several times the seed-to-seed variation.
  3. The finding survives token matching. Real anchors contribute 4.2× fewer scored tokens. At matched token budgets the crossover persists on Qwen3-VL, and appears cleanly in the testbed, where token counts match exactly.
  4. The theory now explains it, and claims no more. A transfer theorem links the anchor loss to retention on real inputs. A new exchange bound shows that anchor content can improve on random anchors by at most a factor computable without held-out data, while missing coverage costs an unbounded amount.
  5. Procedure clarified. Anchors are random text tokens only. The reference runs once, before training. FLOPs include every backward pass, and CFA's measured overhead is 11.4% per step against 37.2% for GRPO+KL.

No experimental result changes.


Summary

Reviewer concernHow we addressed itWhere in the revised PDF
W1 / Q4. Headline numbers do not match the tablesAbstract corrected; each method's worst case across all cellsAbstract, Sec. 5.2, App. C.4 (Table 16)
W2 / Q3. Single runs; noise; exclusion rule unspecifiedThree-seed results; exclusion rule stated in full; one table flag correctedApp. C.3 (Table 15), Sec. 5.1, Table 3
W3 / Q1. What the anchors are; how text anchors protect visual skillsText tokens only, "Gaussian" removed; mechanism explained; scope relative to the vision encoder statedSec. 3.3, App. F, App. I, Sec. 5.4, App. J
W3 / Q2. Algorithm 2 vs. App. G; FLOPs; wall-clockReference evaluated once; Algorithm 2 corrected; FLOPs include all backward passes; wall-clock measuredSec. 3.4, Alg. 2, App. G.2, G.5
W3 / Q5. Token-count confoundTokens counted; crossover persists at matched tokens on Qwen3-VL and in the exactly matched testbedApp. F, App. C.7 (Table 22), App. D.8
W4. Theory does not explain why random anchors workTransfer theorem; exchange bound on the value of content; claims narrowed to what is provenSec. 3.5, App. D.5–D.8, Table 25

W1 / Q4: Reconcile GSPO+EWC on MedBookVQA-2B, GRPO on Navigation-4B, and GRPO+KL on Puzzle-4B with the abstract's "−8.1% to −67.4% for the best baseline".

The phrase "for the best baseline" was wrong, and the reviewer's reading of all three cells is right:

  • MedBookVQA-2B: GSPO+EWC's worst-1 change (−0.2%) ties CFA's (−0.3%). Under the exclusion rule this run does not in fact qualify (see W2 / Q3), but its retention is genuinely as good as CFA's in this cell.
  • Navigation-4B: GRPO loses only −1.2%; no method forgets substantially in this cell.
  • Puzzle-4B: GRPO+KL (−7.8%) is close to CFA (−5.5%) and better than RDA (−11.5%).

What these cells also show is that the best baseline in one cell is often catastrophic in another. GSPO+EWC, which matches CFA on MedBookVQA-2B, loses −94.1% on MedBookVQA-4B (DocVQA falls from 64.4 to 3.8). GRPO+KL, close to CFA on Puzzle-4B, loses −36.0% on MedBookVQA-2B. Picking the best baseline per cell therefore requires knowing the forgetting outcome in advance. We now report each method's worst case over the cells where it qualifies (App. C.4, Table 16):

MethodQualifying cellsWorst-1 change (cell)
SFT3/9−100.0% (MedBookVQA-8B)
GSPO+EWC8/9−94.1% (MedBookVQA-4B)
GRPO (wd=0)7/9−67.4% (MedBookVQA-2B)
CPO5/9−64.7% (MedBookVQA-8B)
GSPO9/9−53.2% (MedBookVQA-8B)
GRPO (wd=0.1)9/9−40.5% (Navigation-2B)
LoRA1/9−38.2% (Puzzle-8B)
GRPO+KL9/9−36.0% (MedBookVQA-2B)
RDA9/9−20.2% (Puzzle-2B)
CFA9/9−9.1% (Puzzle-2B)
AuthorsOpenReview ↗

Response to Reviewer bL9N (Part 2/4)

The abstract now reads: "Across all nine domain-scale combinations tested, CFA holds worst-decile forgetting to −0.1% to −9.1%, against baselines losing at least 36% on some benchmark." We agree with the reviewer's summary: the evidence supports "CFA is consistently among the best", not "CFA beats every baseline everywhere". The revised paper makes exactly the former claim. CFA's distinguishing property is that it is the only method with no severe collapse anywhere.

Revision in the paper: abstract, contributions, Sec. 5.2, App. C.4 (Table 16).


W2 / Q3: Every result is a single run, the worst of 10 noisy scores is sensitive to noise, and the exclusion rule is never specified. What is the precise rule, and was it fixed before looking at OOD results?

1. Seed variance (new App. C.3, Table 15). We retrained every qualifying method on Navigation, Qwen3-VL-2B, with three seeds at each method's selected hyperparameters. This is the cell containing the CountBenchQA result the reviewer highlights. For CFA, the seed also redraws the anchor set.

  • Means are stable. Every seed-averaged score lies within 1.3 points of the single-run score.
  • Seed noise is of the order the reviewer notes. Standard deviations across seeds are 1.6–4.8 points.
  • The ranking is unchanged:
MethodWorst-1, single runWorst-1, mean of 3 seeds
CFA−3.5%−3.8%
RDA−14.1%−13.7%
CPO−23.0%−23.6%
GSPO−27.9%−27.1%
GRPO+KL−33.2%−33.2%
GRPO (wd=0.1)−40.5%−41.1%

The CFA–RDA gap (about 10 points) and the gap to every other method (20+ points) are several times the seed-to-seed variation. The random-vs-curated result therefore does not depend on a favorable seed, or on a favorable anchor draw.

2. The exclusion rule (Sec. 5.1). A baseline enters the headline comparison for a cell if its in-domain gain (final minus base accuracy) is at least 80% of the median gain of the RL baselines in that cell.

  • It reads only in-domain accuracy, never OOD results, so it cannot be tuned toward any method's retention. The 80% threshold is unchanged from the submission; the revision states the full rule.
  • Why gain rather than final accuracy: base accuracy ranges from 3.5% to 49.8% across cells, and "did not learn the task" is a statement about how much was learned.
  • One corrected flag. GSPO+EWC on MedBookVQA-2B was mistakenly marked as qualifying in Table 3. Under the rule it does not qualify: its gain is +35.1 against a median RL gain of +49.6, i.e., 71%. It was never included in the corresponding figure, and it would qualify for any threshold below 71%. Complete per-benchmark results for every run, qualifying or not, remain in App. C.

Revision in the paper: App. C.3 (Table 15); Sec. 5.1; Table 3.


W3 / Q1: What exactly are the content-free anchors? Is the vision encoder trainable, and if so, how do text-only anchors protect visual grounding?

What the anchors are. Content-free anchors are text tokens only: sequences of 2048 tokens drawn i.i.d. and uniformly from the base vocabulary, excluding special tokens (Sec. 3.3, App. F). There are no noise images and no Gaussian inputs in any space. "Gaussian" was a leftover name from an earlier draft and has been removed from Sec. 3.4, App. I, Table 6, and Figure 7.

The vision encoder. The vision encoder is trainable in our runs. Because text anchors do not pass through it, CFA regularizes the language model that processes visual tokens, rather than the encoder that produces them (App. J; Cor. 5). This is where the forgetting arises in our setting (point 3 below). Where encoder drift does matter, the remedy is simple and compatible with CFA: freeze the encoder, or add image-side anchors.

Why text anchors still protect visual grounding.

  1. Image tokens are processed by the layers the anchors constrain. After the projector, each image patch is a vector in the language model's input space and passes through the same attention and MLP layers as text tokens. Uniform random token embeddings span the full input space of those layers (rank dd; Prop. 9). CFA therefore constrains how the model processes visual tokens, though not how they are produced.
  2. The observed failures are decoding behaviors. The grounding failures we document (App. H) all keep fluent, well-structured reasoning but stop checking it against the image. This is a change in the language model's policy, not in the visual features.
  3. Constraining the vision encoder alone does not help. EWC with a vision-encoder-only Fisher performs roughly like unregularized GSPO, whereas full-model EWC does better (App. I). Forgetting in this setting is therefore driven mainly by the language model, which is exactly what CFA regularizes.

Revision in the paper: Sec. 3.3–3.4, App. F, App. I, Table 6, Fig. 7, Sec. 5.4, App. J.


AuthorsOpenReview ↗

Response to Reviewer bL9N (Part 3/4)

W3 / Q2: Is the reference model called during training? What are the true per-step FLOPs, including the anchor backward pass, and the measured wall-clock overhead?

The reference is not called during training; App. G.4 was correct. Before training, the frozen reference runs one no-grad pass over the anchor pool. We cache only the expected embeddings and varentropy weights, then release the reference from GPU memory. Each step needs one forward and backward pass of the policy on a mini-batch of anchors. Sec. 3.4 and Algorithm 2 now state exactly this.

FLOPs, including every backward pass. The reviewer is right that App. G counted only the anchor forward pass. With Ffwd\mathcal{F}_{\text{fwd}} the forward cost of one 2048-token sequence and a backward pass costing about 2Ffwd2\mathcal{F}_{\text{fwd}}:

MethodAdded to base RLPer-step FLOPsRelative
GRPO / GSPO— (policy fwd + bwd on 8 rollouts)24 Ffwd24\,\mathcal{F}_\text{fwd}1.00×
GRPO+KLReference forward on the 8 rollouts (+8)32 Ffwd32\,\mathcal{F}_\text{fwd}1.33×
CFAPolicy forward (+8) and backward (+16) on 8 anchors48 Ffwd48\,\mathcal{F}_\text{fwd}2.00×

GRPO+KL adds no backward cost, because its gradient flows through the policy's log-probabilities on the same rollout tokens. CFA's anchors are additional sequences, so it adds a forward and a backward pass over them.

Measured wall-clock (new App. G.5). FLOP counts overstate the practical cost, because each RL step is dominated by autoregressive rollout generation:

MethodSeconds per stepOverhead
GRPO123.5—
GRPO+KL (β=0.01\beta=0.01)169.5+37.2% over GRPO
GSPO140.0—
GSPO + CFA156.0+11.4% over GSPO

Despite more FLOPs, CFA costs about a third of GRPO+KL's overhead. Its extra work is a fully parallel, teacher-forced pass on a small fixed batch with cached targets, whereas GRPO+KL keeps the reference resident and runs it on fresh rollouts every step.

Revision in the paper: Sec. 3.4; Algorithm 2; App. G.2 and Table 30; App. G.5.


W3 / Q5: How many scored tokens do real and content-free anchors contribute at equal N, and does the crossover survive at matched token budgets?

This is a fair concern, and the answer strengthens the finding. Padding is masked out of the anchor loss, so a real anchor contributes only its valid tokens: 486 on average over the full pool of 2,560, about 4.2× fewer than a random anchor. At equal NN, the paper's comparisons therefore gave random anchors more tokens. We remove this confound in two settings.

1. Approximately matched tokens on Qwen3-VL (App. C.7, Table 22). Because of the 4.2× ratio, our budget ablation (Navigation, Qwen3-VL-2B) already contains token-matched pairs, CFA at NN against RDA at 4N4N:

AnchorsScored tokensTargetWorst-1Worst-3
RDA, N=1280N=12800.62M85.7−14.1%−6.4%
CFA, N=320N=3200.66M83.8−14.6%−11.8%
RDA, N=2560N=25601.24M82.7−13.3%−9.8%
CFA, N=640N=6401.31M86.4−8.6%−5.6%

At about 0.65M tokens, curated anchors are as good or better. At about 1.3M tokens, content-free anchors win on worst-1, worst-3, and target accuracy. Curated anchors also stop improving with more tokens (worst-1 −14.1% → −13.3% when doubled), while CFA keeps improving.

2. Exactly matched tokens in the testbed (App. D.8). Each testbed anchor is a single input with no padding, so random and curated anchors at equal NN contribute exactly the same number of scored inputs (worst-capability forgetting in accuracy points, lower is better, 5 seeds):

Scored inputs (NN)3212825651210242048
Curated72.563.957.352.346.140.6
Random73.765.757.447.538.631.1

Curated anchors lead slightly at small budgets, the two tie at N=256N=256, and random anchors lead at every budget from N=512N=512 on, by a widening margin. Random anchors at N=1024N=1024 even beat curated anchors at twice the budget (38.6 vs. 40.6). Target accuracy is matched within about 1 point throughout.

Both settings show the same pattern: token count widens the gap at equal NN, but does not create it. This is the crossover Corollary 4 predicts. A curated pool's coverage is capped by the span of the pool, however many tokens it contributes, while random anchors keep adding directions.

Revision in the paper: App. F and App. C.7 (Table 22); App. D.8.


Reviewer_5gbSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies forgetting during RL post-training of VLMs and proposes CFA, which matches the current and reference models on random anchors using a varentropy-weighted MMD loss. It also reports worst-k forgetting curves to reveal degradation hidden by average scores. Experiments on Qwen3-VL at three scales and three target domains show improved retention with competitive target-task performance.

Critical strengths
  1. The paper addresses a practical problem: average scores can hide severe losses on individual capabilities. The worst-k curves make this easier to see.
  2. The method is simple, avoids real prior-task replay data, and shows promising retention across several tasks and model scales.
Critical weaknesses
  1. The concentration results concern the anchor distribution and do not establish retention on real inputs. Appendix D.4 acknowledges this, while Corollary 4 assumes that noise anchors have lower bias relative to the protective curvature. The broader claim that anchor relevance is unnecessary needs stronger support.
  2. The implementation is unclear. Appendix F describes uniform random tokens, whereas Appendix I describes Gaussian noise. Section 3.4 caches reference outputs once, but Algorithm 2 recomputes them every step. These inconsistencies make the actual experimental procedure difficult to reproduce.
  3. Table 6 lacks an unweighted-MMD control in the VLM setting. The number of independent training runs and the source of the reported standard deviations are also unclear. The curvature-proxy measurements supporting the claimed convergence rate are not shown in the cited Figure 6 and Table 5.
Questions for authors
  1. What evidence supports the lower-bias assumption in Corollary 4, and how does concentration on random anchors translate to capability retention on real inputs?
  2. Were the anchors random token sequences or Gaussian inputs? If Gaussian, in which space? Were reference outputs cached or recomputed at each step?
  3. Could the authors add an unweighted-MMD VLM control, clarify training repeats and standard deviations, and report the curvature-proxy measurements used to support the convergence claim?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5gbS (Part 1/3)

We thank Reviewer 5gbS for a careful and constructive review. We are glad the reviewer found the problem practical, the worst-k curves effective at exposing losses that average scores hide, and the method simple, free of real prior-task replay data, with promising retention across tasks and model scales. Each of the three points was well taken, and the revision addresses all of them.

In brief,

  1. Theory. A new transfer theorem (Theorem 1) links the anchor loss to drift on a capability's real inputs. Corollary 4 no longer assumes a lower bias; its condition is now stated in measurable quantities. A new exchange bound (Theorem 3) makes precise how much anchor relevance can matter.
  2. Implementation. Anchors are uniform random token sequences, reference outputs are cached once, and every section now describes this single procedure.
  3. Controls and statistics. We add the unweighted-MMD control, state the sweep, selection, and bootstrap procedure, add three-seed results, and report the convergence measurements directly.

Summary

Reviewer concernWhat we didWhere in the revised PDF
W1 / Q1. Concentration does not establish retention; Corollary 4's lower-bias assumption; "relevance is unnecessary" needs supportTransfer theorem; Corollary 4 restated with a measurable, provable condition; exchange bound on the value of content; testbed validationSec. 3.5, Remark 2, App. D.4–D.8, Table 25
W2 / Q2. Random tokens vs. Gaussian noise; cached vs. recomputed reference outputsUniform random tokens everywhere ("Gaussian" removed); reference outputs cached once; Algorithm 2 matches Sec. 3.4Sec. 3.3–3.4, Algorithm 2 (App. F), App. I, Table 6, Fig. 7
W3 / Q3. Unweighted-MMD control; runs and standard deviations; curvature-proxy measurementsUnweighted-MMD rows for both anchor types; procedure stated; three-seed results; measurements reported directlyTable 6, Fig. 7, Sec. 5.1, App. I, App. C.3, App. D.8, Fig. 11, Table 24

W1 / Q1: How does concentration on random anchors translate to retention on real inputs, and what supports the lower-bias assumption in Corollary 4?

We agree that concentration alone does not establish retention. The revision supplies the missing step and replaces the lower-bias assumption with statements that are either proven or measurable.

1. From concentration to retention on real inputs (Theorem 1). The concentration results (Props. 5–7) are now presented purely as estimation: a finite anchor set reproduces its own population curvature GνG_\nu at an N−1/2N^{-1/2} rate. The transfer step is Theorem 1. For a held-out capability τ\tau with curvature GτG_\tau computed on its own real inputs, define the transfer coefficient

κS(τ∥ν)=sup⁡u∈Su⊤Gτuu⊤Gνu,\kappa_S(\tau\|\nu)=\sup_{u\in S} \frac{u^\top G_\tau u}{u^\top G_\nu u},

where SS is the subspace in which training moves. Then the drift FτF_\tau on the capability's real inputs satisfies

Fτ  ≤  κS(τ∥ν) Lν,F_\tau \;\le\; \kappa_S(\tau\|\nu)\, L_\nu,

where LνL_\nu is the anchor loss. This holds to second order, and at any training step once measured remainder factors are included (Thm. 1(vi)). Concentration controls LνL_\nu, and κS\kappa_S converts it into retention on real inputs. If κS=∞\kappa_S=\infty, the anchors cannot see a direction the capability depends on, and no guarantee is possible (Thm. 1(v)). Each assumption is stated explicitly (App. D.5), and Table 25 lists which result uses which.

2. Corollary 4 no longer assumes a lower bias. The previous version compared anchors against an unobservable "protective curvature". The revised Corollary 4 is stated in measurable quantities (κS\kappa_S, the anchor coherence η\eta, and dim⁡S\dim S). Its key condition is not assumed but proven to hold in a structural case: a curated coreset's curvature can never leave the span of its pool, at any budget. So whenever some at-risk capability depends on a direction outside that span, random anchors carry a strictly smaller worst-case retention constant than the coreset at any budget, once NN exceeds an explicit threshold. A new second part makes this statement about forgetting itself, not only about the guarantee: along such a direction, the curated penalty exerts no second-order resistance at any budget or weight λ\lambda, while random anchors penalize the displacement in proportion to the forgetting it causes.

AuthorsOpenReview ↗

Response to Reviewer 5gbS (Part 2/3)

3. How much can relevance matter? (Theorem 3, new). This addresses the reviewer's broader point directly. For any curated anchor set ν\nu and random anchors μ\mu, every capability satisfies

κS(τ∥μ)  ≤  cS(ν∥μ) κS(τ∥ν),\kappa_S(\tau\|\mu)\;\le\;c_S(\nu\|\mu)\,\kappa_S(\tau\|\nu),

where cS(ν∥μ)c_S(\nu\|\mu) depends only on the two anchor sets and the update subspace, and uses no held-out data.

  • The bound is tight: it is attained on the capability the curated set itself represents.
  • The reverse gap is unbounded: if the curated set misses part of SS, some capability has κS=∞\kappa_S=\infty under curated anchors and a finite coefficient under random anchors.

Relevance can therefore help, but only by a bounded factor that needs no held-out data, while a lack of coverage can cost an unbounded amount. Theorem 2 identifies the one place where content is genuinely needed: interference with capabilities close to the target task, as opposed to collateral drift, which content-free anchors remove. The paper now claims exactly this, rather than that relevance never matters.

4. The condition is measured, not assumed (App. D.8). In a fully observable testbed where GτG_\tau is exact for every capability (5 seeds), we compute each anchor type's floor, its worst-case κS\kappa_S as N→∞N\to\infty:

CapabilitiesRandom anchors (CFA)Curated anchors (RDA)
Outside the curated pool, spread setting2.77.9
Outside the curated pool, concentrated setting9.488
Inside the curated pool, spread setting3.23.9
Inside the curated pool, concentrated setting24.75.8

Random anchors have the lower floor on every capability outside the curated pool, in every seed. They lose only on narrow capabilities inside the pool that overlap the target, exactly where Theorems 2 and 3 predict. In the same testbed:

  • κS\kappa_S, evaluated at the pretrained parameters on the subspace spanned by the training updates, rank-predicts each capability's accuracy drop after RL (Spearman ρ = 0.93).
  • With measured remainders, the bound of Theorem 1(vi) held at every checkpoint of all 1000 steps, for every capability, anchor set, and seed (8,000 pairs; drift never exceeded 0.91 of the bound).

On Qwen3-VL itself, random anchors beat curated ones under every divergence in Table 6, including the new unweighted-MMD rows.

Revision in the paper: Sec. 3.5, Remark 2; App. D.4 (concentration as estimation); D.5 (assumptions, Theorem 1); D.6 (Corollary 4); D.7 (Theorems 2–3); Table 25; D.8 (testbed).


W2 / Q2: Were the anchors random tokens or Gaussian inputs, and were reference outputs cached or recomputed at each step?

Thank you for catching these. Both were inconsistencies in the description, not in the experiments.

  • Anchors are uniform random token sequences. Each anchor is T=2048T=2048 tokens drawn i.i.d. and uniformly from the Qwen3-VL base vocabulary, excluding special tokens (Sec. 3.3, App. F). No Gaussian inputs are used in any space. "Gaussian" was a leftover name from an earlier draft. It has been removed from App. I, Table 6, and Figure 7, where the content-free variants are now labeled "Random".
  • Reference outputs are cached once. Before training, the frozen reference runs a single no-grad, teacher-forced pass over the anchor pool. We cache the expected embedding μi,t∗\mu^*_{i,t} and the varentropy weight w~i,t\tilde w_{i,t} for every anchor and position, then release the reference from memory. Each training step needs one forward and backward pass of the policy on a mini-batch of anchors, and no reference pass. Algorithm 2 (App. F) now shows exactly this and matches Sec. 3.4.

Revision in the paper: Sec. 3.3–3.4, Algorithm 2 (App. F), App. I, Table 6, Figure 7.


W3 / Q3: Add an unweighted-MMD VLM control, clarify training repeats and standard deviations, and report the curvature-proxy measurements.

1. Unweighted-MMD control (Table 6, Figure 7). Same setting as Table 6 (Navigation, Qwen3-VL-2B, N=1280N=1280). Values are worst-k average % change vs. base:

DivergenceAnchorsTargetWorst-1Worst-3Avg. (all 10)
Varentropy-weighted MMD (CFA)Random87.0−3.5−2.5+10.5
Unweighted MMDRandom88.1−11.6−7.5+7.7
Varentropy-weighted MMD (RDA)Real85.7−14.1−6.4+11.1
Unweighted MMDReal85.1−16.0−10.1+4.6

Removing the varentropy weighting lowers retention at every k, for both anchor types, at comparable target accuracy. The same holds for KL (worst-1 −5.8 vs. −14.6 with random anchors, −15.0 vs. −15.3 with real).

AuthorsOpenReview ↗

Response to Reviewer 5gbS (Part 3/3)

2. Training runs and standard deviations (Sec. 5.1, App. I).

  • Runs. For every method and cell, we sweep its hyperparameters (learning rate, and the regularizer strength where present), giving 3–18 configurations per method per cell. Each configuration is one 600-step run.
  • Selection. For each method we report the configuration with the highest in-domain accuracy, so selection never uses out-of-domain results.
  • Standard deviations. The ± values are bootstrap standard deviations over evaluation items, and every results-table caption states which interval is shown.
  • Seeds (new App. C.3, Table 15). We retrained every qualifying method on Navigation, Qwen3-VL-2B with three seeds. Seed-averaged scores lie within 1.3 points of the single run, seed standard deviations (1.6–4.8 points) match single-run evaluation noise, and the ordering by worst-1 forgetting is unchanged:
MethodWorst-1, single runWorst-1, mean of 3 seeds
CFA−3.5%−3.8%
RDA−14.1%−13.7%
CPO−23.0%−23.6%
GSPO−27.9%−27.1%
GRPO+KL−33.2%−33.2%
GRPO (wd=0.1)−40.5%−41.1%

3. Curvature-proxy measurements. The reviewer is right that these were not shown. One sentence in App. D.4 asserted a measured N−1/2N^{-1/2} rate while citing Figure 6 and Table 5, which show only the crossover. We removed it and now report the measurements directly:

  • Convergence rate (App. D.8). In the testbed, ∥G^N−G^2N∥op/∥G^2N∥op\|\hat G_N-\hat G_{2N}\|_{\mathrm{op}}/\|\hat G_{2N}\|_{\mathrm{op}} for random anchors decays with log–log slope −0.49, matching the N−1/2N^{-1/2} rate of Prop. 6. Curated subsets concentrate at a similar rate, which is why concentration alone cannot explain the crossover, and the transfer coefficient is needed.
  • Quadratic proxy on Qwen3-VL (Fig. 11, Table 24). Along a real Navigation run on Qwen3-VL-2B, the quadratic proxy of the anchor loss stays within 3–7% of the exact loss across all 600 steps.

Revision in the paper: Table 6, Figure 7; Sec. 5.1, App. I; App. C.3 (Table 15); App. D.8; Fig. 11, Table 24.


Summary. The revision addresses each weakness and question:

  • W1 / Q1: a transfer theorem connects the anchor loss to retention on real inputs. Corollary 4 no longer relies on a lower-bias assumption, and its condition is proven in the structural case and measured in a fully observable testbed. Theorem 3 bounds exactly how much anchor relevance can matter.
  • W2 / Q2: anchors are uniform random token sequences, reference outputs are cached once, and every section describes the same procedure.
  • W3 / Q3: the unweighted-MMD control is added; the sweep, selection, and bootstrap procedure is stated; three-seed results leave every conclusion unchanged; and the convergence and proxy measurements are reported directly.

The reviewer found the problem practical and the method simple and promising. Their remaining concerns were the soundness of the theoretical claim and the reproducibility of the procedure, and each now has a concrete answer in the paper. If the reviewer agrees that no critical blockers remain, we would be grateful for a corresponding increase in the score, and we would be glad to address any remaining question during the discussion period.

Dear Reviewer 5gbS, thank you again for your detailed review. As the discussion period closes in a few hours, we wanted to check whether our response addressed your concerns on the theory (Theorems 1 and 3, the revised Corollary 4), the implementation details, and the added controls ablations of unweighted-MMD. If anything remains unclear, we would be glad to clarify it before the deadline.

Reviewer_KeLkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out a real, under‑discussed failure mode of single‑domain RL post‑training for vision‑language models. Even when overall benchmark averages look fine, models can suffer huge, silent performance drops on unseen held‑out tasks; the average metric hides these severe tail degradations. The authors propose the worst‑k curve to properly measure this worst‑case forgetting, instead of only reporting suite‑wide averages.

Existing anti‑forgetting methods (EWC, replay, KL regularization, CPO) all assume you need real, meaningful reference data (“anchors”) to constrain the model and protect old capabilities. This work breaks that assumption: you do not need semantically meaningful real data for anchors. Pure random noise / random token sequences work well enough.

They introduce CFA (Content‑Free Anchoring), a regularizer using Fisher‑weighted MMD in token‑embedding space. CFA uses frozen reference‑model outputs computed over these random synthetic anchor inputs, no curated real‑world data required. The paper gives theoretical analysis unifying CFA and EWC under a shared Gauss‑Newton framework, and explains when classic second‑order approximations break down during RL updates.

Critical strengths
  1. Novel, counter‑intuitive core insight that content‑free random anchors can mitigate forgetting for RL post‑training. Almost all existing continual‑learning / RL‑alignment defenses require curated real reference data. Removing this requirement has large practical impact for real‑world LLM/VLM fine‑tuning: you avoid data licensing, storage, and dataset curation overhead. This shifts how people think about anchor‑based functional regularization.
  2. Thorough, multi‑faceted evaluation pipeline plus improved evaluation metric (worst‑k curve). The authors do not just propose a new loss. They identify a dangerous evaluation pitfall (relying only on aggregate benchmark averages which hide tail forgetting). They validate across toy setup and three model scales, multiple domains, many competitive baselines, rich ablations on anchor budget and loss variants, plus comparison against post‑hoc weight merging. The empirical evidence for CFA’s effectiveness is comprehensive.
Critical weaknesses
  1. Limited clarity on practical failure boundaries for CFA. The paper shows strong results for their tested setup, but it is unclear under what conditions content‑free anchors might stop working. For example: extremely long context windows, very large RL update magnitudes, different model families, or when protecting extremely niche fine‑grained capabilities. Without more boundary testing, practitioners cannot know safe operating limits when deploying CFA out‑of‑sample.
  2. Theory‑empirical gap. The concentration theorems show estimates converge regardless of anchor semantics, but they do not formally prove that these estimates will effectively protect arbitrary out‑of‑distribution capabilities. The strong real‑world performance is empirical, not theoretically guaranteed. The manuscript sometimes blurs the line between what is proven mathematically and what is observed empirically.
Questions for authors
  1. Your theory establishes that curvature estimates converge for any i.i.d anchor distribution, but does not guarantee good generalization to arbitrary held‑out real‑world tasks. Could you explicitly discuss what assumptions make random noise anchors effective in your setting, and what failure modes we might expect when those assumptions break?
  2. In your budget ablation, real curated anchors saturate quickly at small N while CFA needs larger anchor budgets to catch up. Have you tried mixing real anchors and content‑free noise anchors? Could hybrid anchoring get the best of both worlds (small N cost + strong protection)?
  3. You test on Qwen3‑VL only. Do you have preliminary results hinting whether CFA transfers to pure‑text LLMs for RLHF‑style fine‑tuning? If not, what key differences would you anticipate for text‑only models?
  4. Could you comment on sensitivity to the exact random token generation scheme? For instance: filtered token sets, different sampling temperatures, or non‑uniform token sampling — do these materially change CFA performance?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KeLK (Part 1/4)

We thank Reviewer KeLk for the encouraging review. We are glad the reviewer found the core insight novel and counter-intuitive with large practical impact ("you avoid data licensing, storage, and dataset curation overhead"), one that "shifts how people think about anchor-based functional regularization," and the evaluation thorough, multi-faceted, and comprehensive. The suggestions were constructive, and we have used each of them to strengthen the paper.

In brief,

  1. Theory. The paper now separates what is proven from what is observed, states every assumption together with the results that use it (App. D.5, Table 25), and adds a theorem bounding how much anchor content can ever add over random anchors (Theorem 3).
  2. Operating limits. A new Limitations section (App. J) covers each example the reviewer raised: model families (now tested), large updates, niche capabilities, and long context.
  3. Other model families. On InternVL3.5-1B and Gemma 4 E2B, CFA again forgets least of every method (worst-1 −5.1% and −3.6%, against −43.1% and −48.0% for GSPO alone).
  4. New experiments. Text-only models, hybrid anchors, five anchor samplers, SFT and LoRA, and three-seed robustness, all consistent with the main result.

Summary

Reviewer suggestionWhat we addedWhere in the revised PDF
W2 / Q1. Separate proof from observation; state assumptions and failure modesEstimation vs. transfer (Thm. 1); explicit assumptions mapped to results; a bound on what content can add (Thm. 3); fully observable testbedSec. 3.5, Remark 2, App. D.5, Table 25, App. D.7–D.8, App. J
W1. Practical operating limitsLimitations section covering each example; results on two other VLM familiesApp. J; Sec. 5.4, Fig. 8, App. C.5
Q2. Hybrid real + random anchorsHarmonic-mean bound (Prop. 11); 50/50 hybrid in the testbedSec. 5.3, App. D.6, App. D.8
Q3. Text-only LLMsRL on four text-only models, with and without CFASec. 5.4, App. C.6 (Tables 19–21)
Q4. Random-token generation schemeAblation over five samplersSec. 5.3, App. F.1 (Table 28)

W2 / Q1: Separate what the theory proves from what is observed, and state the assumptions behind random anchors and their failure modes.

We agree, and the revised theory makes the chain explicit in three steps.

  1. Estimation (Props. 5–7). A finite anchor set reproduces its own population curvature GνG_\nu at an N−1/2N^{-1/2} rate, for any i.i.d. anchor distribution. As the reviewer notes, this alone says nothing about held-out capabilities.
  2. Transfer (Theorem 1). Retention on a held-out capability τ\tau is governed by the transfer coefficient κS(τ∥ν)=sup⁡u∈Su⊤Gτuu⊤Gνu,\kappa_S(\tau\|\nu)=\sup_{u\in S} \frac{u^\top G_\tau u}{u^\top G_\nu u}, the largest ratio of the capability's curvature, measured on its own real inputs, to the anchors' curvature over the subspace SS in which training moves. An anchor loss of ε\varepsilon bounds drift on the capability by κS ε\kappa_S\,\varepsilon: to second order, and at any training step once measured remainder terms are included (Thm. 1(vi)). κS=∞\kappa_S=\infty marks a direction the anchors cannot see.
  3. How much content can add (Theorem 3, new). For any curated anchor set ν\nu and random anchors μ\mu, every capability satisfies κS(τ∥μ)≤cS(ν∥μ) κS(τ∥ν)\kappa_S(\tau\|\mu)\le c_S(\nu\|\mu)\,\kappa_S(\tau\|\nu). The factor cS(ν∥μ)c_S(\nu\|\mu) depends only on the two anchor sets and the update subspace, and uses no held-out data. The bound is tight, while the reverse gap is unbounded: a curated set has κS=∞\kappa_S=\infty on any capability that relies on a direction outside its pool.

What is proven. Anchors protect a capability exactly as far as κS\kappa_S allows; isotropic anchors minimize the worst case when the at-risk capability is unknown, and uniform random tokens attain this up to an explicit factor computable from the embedding table (Prop. 9); and no anchor content can improve on random anchors by more than cS(ν∥μ)c_S(\nu\|\mu) (Thm. 3).

What is empirical. Whether κS\kappa_S is small, not merely finite, for a given real capability. Computing it at multi-billion-parameter scale would require every held-out capability's curvature, so we validate the theory where every quantity is exact (testbed below) and test its predictions at scale: worst-1 forgetting under −10% in all nine domain–scale cells, the budget crossover (Fig. 6), and the sampler ablation (Q4).

Assumptions and failure modes. Each assumption is stated once (App. D.5, S1–S6), and Table 25 lists which result uses which. The three that make random anchors effective:

AuthorsOpenReview ↗

Response to Reviewer KeLK (Part 2/4)

What random anchors needWhy it holds hereFailure mode
An informative bound along the runRuns leave the conservative trust radius early (Fig. 11), but the any-step bound (Thm. 1(vi)) needs no trust radius: its anchor-side factor stays ≤ 1.08 over 600 steps on Qwen3-VL-2B (Table 24), and in the testbed it holds at every checkpoint of 1000 stepsMuch longer training or larger learning rates can inflate the remainder factors. CFA evaluates the exact anchor loss at every step, so this weakens the guarantee, not the training signal
Coverage: anchors excite every direction training movesUniform tokens give a full-rank first-layer second moment and are worst-case optimal up to an explicit factor (Prop. 9); content can add at most cS(ν∥μ)c_S(\nu\|\mu) (Thm. 3)Pathways the anchors never reach, e.g. the vision encoder under text anchors (Cor. 5); vocabularies with fewer than dd tokens (Prop. 9(b)); deep features random inputs barely excite
Forgetting is mostly collateralUnder a random-subspace model of the update (S6), a low-rank RL update overlaps a capability's curvature by only r0/sr_0/s of its trace (Thm. 2(c))Capabilities close to the target domain, where forgetting is interference that no content-free method can prevent (Thm. 2(b))

Validation (App. D.8). In a fully observable testbed where every quantity is exact (5 seeds):

  • κS\kappa_S, evaluated at the pretrained parameters on the subspace spanned by the training updates, rank-predicts each capability's accuracy drop after RL (Spearman ρ = 0.93).
  • The any-step bound of Thm. 1(vi) holds at every checkpoint of all 1000 steps, for every capability, anchor set, and seed (8,000 pairs; drift never exceeds 0.91 of the bound).
  • Random anchors attain κS\kappa_S = 1.7–4.2 on capabilities inside the curated pool and 2.0–3.3 outside it; curated anchors attain 3.1–8.5 and 7.4–19.3.
  • The theory also predicts where random anchors lose: on narrow capabilities overlapping the target, curated anchors have the lower floor (5.8 vs. 24.7).

Revision in the paper: Sec. 3.5, Remark 2; App. D.5 (assumptions, Thm. 1); D.6 (coverage, crossover); D.7 (Thms. 2–3, and a summary of what is proven vs. assumed); Table 25; D.8 (testbed); App. J.


W1: Clarify the conditions under which CFA may stop working, so that practitioners know its safe operating limits.

The new Limitations section (App. J) states what CFA guarantees, what we tested, and what we did not, organized around the reviewer's four examples:

  • Different model families (now tested). CFA, applied unchanged to InternVL3.5-1B and Gemma 4 E2B (Sec. 5.4, Fig. 8; App. C.5), has the smallest worst-k forgetting at every k: worst-1 −5.1% and −3.6%, against −43.1% and −48.0% for GSPO alone and at best −25.6% and −12.0% for any other method.

  • Very large RL updates. This limit is not specific to CFA. Every regularizer that keeps the model close to a reference, whether KL, EWC, replay, or anchoring, gives a weaker guarantee as the update grows. Two features make CFA the best placed of these methods:

    • CFA never relies on a local approximation. It evaluates the exact anchor loss at every step. EWC instead freezes its curvature at initialization, which is accurate only within a trust radius that our runs leave within about 120 steps (Prop. 4, Fig. 11). This is why GSPO+EWC collapses by −94.1% on MedBookVQA-4B.
    • CFA's guarantee degrades measurably, not silently. The any-step bound (Thm. 1(vi)) holds without a trust radius, through remainder factors that can be monitored during training. On Qwen3-VL-2B, the anchor-side factor stays at most 1.08 over 600 steps (Table 24).

    The largest updates we test support this. SFT moves the model furthest and is the most destructive method in our study, and adding CFA cuts its worst-1 forgetting from −78.8% to −9.1% (Table 13). Much longer or multi-stage training could inflate the remainder factors for every method; for CFA, this would show up in a quantity that can be checked during training.

  • Niche fine-grained capabilities. The theory shows that what matters is not how niche a capability is, but whether the anchors see the directions it relies on and how close it is to the training domain. Target-adjacent capabilities are most at risk, since content-free anchors remove collateral drift but not interference (Thm. 2); hybrid anchors (Prop. 11) can target them. Empirically, the most fragile fine-grained capability in our suite, counting (CountBenchQA, down up to −67.4% under GRPO), stays within 10% of base under CFA in every cell.

  • Extremely long context windows. We do not study long-context training; anchors are 2048 tokens, so much longer positions are constrained only indirectly. App. J states this explicitly.

AuthorsOpenReview ↗

Response to Reviewer KeLK (Part 3/4)

App. J also clarifies scope: text anchors regularize how the model processes visual tokens, not the vision encoder that produces them (Cor. 5). In our runs, forgetting arises downstream of the encoder (vision-encoder-only EWC performs roughly like unregularized GSPO; App. I), and freezing the encoder or adding image-side anchors would extend coverage if needed.

Within this envelope, the evidence is consistent: CFA holds worst-1 forgetting under −10% in all nine domain–scale cells at comparable in-domain gain, three-seed results leave every conclusion unchanged (App. C.3), and it adds only 11.4% wall-clock time per step (App. G.5).

Revision in the paper: App. J; Sec. 5.4, Fig. 8, App. C.5; Table 13.


Q2: Evaluate whether hybrid real and content-free anchors combine the strengths of both.

Yes, and the theory makes this precise. Since the curvature of a mixture, Gmix=αGreal+(1−α)GrandG_{\text{mix}}=\alpha G_{\text{real}}+(1-\alpha)G_{\text{rand}}, is linear in its components, the transfer coefficient obeys a harmonic-mean bound (Proposition 11):

κS(τ∥mix)−1≥α κS(τ∥real)−1+(1−α) κS(τ∥rand)−1\kappa_S(\tau\Vert\mathrm{mix})^{-1} \geq \alpha\,\kappa_S(\tau\Vert\mathrm{real})^{-1} + (1-\alpha)\,\kappa_S(\tau\Vert\mathrm{rand})^{-1}

so

κS(τ∥mix)≤min⁡{κS(τ∥real)α,κS(τ∥rand)1−α}\kappa_S(\tau\Vert\mathrm{mix}) \leq \min\left\lbrace \frac{\kappa_S(\tau\Vert\mathrm{real})}{\alpha}, \frac{\kappa_S(\tau\Vert\mathrm{rand})}{1-\alpha} \right\rbrace

A 50/50 hybrid is therefore never more than twice as loose as the better of its two components, on every capability, which is exactly the best-of-both behavior the reviewer anticipates:

  • Small-NN efficiency from real anchors on the capabilities the curated pool represents.
  • Coverage from random anchors on capabilities outside the pool, where curated anchors alone give κS=∞\kappa_S=\infty.

Theorem 3 also caps what the real component can contribute: at most a factor cS(real∥rand)c_S(\text{real}\|\text{rand}), on any capability. In the testbed (App. D.8), a 50/50 hybrid satisfied the bound in all 50 (seed, capability) pairs, retained nearly as well as CFA, and reached target accuracy between CFA and RDA. (At a fixed total budget, each component is estimated from fewer anchors; Props. 5–7.)

The trade-off is that hybrids reintroduce what CFA removes: real data, and knowing in advance which capabilities are at risk. We therefore see them as a complement to CFA for capabilities known to be at risk, especially target-adjacent ones (Thm. 2(b)), rather than a general replacement.

Revision in the paper: Sec. 5.3; Prop. 11 (App. D.6); App. D.8; App. J.


Q3: Assess whether CFA transfers to text-only LLMs, and what differences to expect.

Yes. CFA transfers to text-only models without any change, since its anchors are already random text tokens. We ran RL on four text-only models with and without CFA (App. C.6, Tables 19–21). Text-only RL forgets far less than VLM RL (worst-1 at most −7.2%, against up to −67.4% for Qwen3-VL), and CFA reduces even this, at unchanged target accuracy:

ModelRL taskTarget (RL)Target (RL + CFA)Worst-1 (RL)Worst-1 (RL + CFA)
Qwen2.5-1.5B-InstructArithmetic80.079.7−0.9%−0.2%
Qwen2.5-0.5B-InstructMedical ontology80.080.4−6.2%−1.5%
Qwen2.5-3B-InstructMedical ontology52.852.7−7.2%−1.7%
Qwen2.5-3B-InstructSearch query extraction85.085.4−4.2%−2.7%
Phi-tiny-MoE-InstructSearch query extraction42.742.5−0.1%−0.3%

Where RL alone forgets almost nothing (Phi-tiny-MoE), CFA leaves retention unchanged within noise. This agrees with recent reports that on-policy RL forgets little on standard text benchmarks (RL's Razor; Retaining by Doing; RFT naturally mitigates forgetting), and is why we focus on VLMs, where the same recipe collapses individual capabilities while suite averages stay flat. Safety behavior is also preserved: after GRPO on GSM8K, Qwen2.5-3B-Instruct refuses 98.3% of harmful prompts, the same as the base model.

Differences to expect. (1) No implementation change is needed. (2) Coverage is easier: anchors pass through every trainable parameter, so the blocked-pathway failure mode (Cor. 5) does not arise. (3) Two things to watch: anchor length should match the long reasoning traces common in text RL, and forgetting from a direct rule conflict (e.g., remapped arithmetic operators) is partly interference (Thm. 2), which hybrid anchors (Prop. 11) can target.

Revision in the paper: App. C.6 (Tables 19–21); Sec. 5.4; App. J.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all four reviewers for their careful and constructive feedback, which has substantially improved the paper. We have posted a detailed response to each review and uploaded a revised PDF. Since the reviews shared several themes, we summarize the main changes here.

1. Theory (all reviewers). Every reviewer noted that concentration of the anchor loss does not by itself establish retention on real capabilities. The revision adds the missing step:

  • Theorem 1 bounds drift on a held-out capability's own real inputs by the anchor loss, through a transfer coefficient.
  • Theorem 3 (new) bounds how much anchor content can ever add over random anchors, by a factor that requires no held-out data; the reverse gap is unbounded.
  • All assumptions are now stated explicitly and mapped to the results that use them (App. D.5, Table 25), and the abstract and conclusion claim only what is proven.

2. Generality (KeLk, V9L2). CFA again has the smallest worst-case forgetting on two further VLM families (InternVL3.5-1B, Gemma 4 E2B; Sec. 5.4), transfers unchanged to four text-only models (App. C.6), and remains best on six additional benchmarks outside the original suite (App. C.8).

3. Robustness and controls (5gbS, bL9N). We added three-seed results (App. C.3), a token-matched comparison of real and random anchors (App. C.7), the unweighted-MMD control (Table 6), each method's worst case across all cells (App. C.4), and the precisely stated exclusion rule (Sec. 5.1).

4. Clarity and reproducibility (5gbS, bL9N). Anchors are uniform random text tokens throughout, reference outputs are cached once before training, FLOP counts include every backward pass, and measured wall-clock overhead is reported (App. G). A new Limitations section (App. J) states the method's operating envelope.

No experimental result from the original submission has changed.

We thank Reviewer bL9N for engaging with our response and updating their assessment. We would be grateful if the other reviewers could let us know whether our responses resolve their concerns, or if any questions remain. With only a couple of days left in the discussion period, hearing about any open points soon would give us the best chance to address them properly, since new experiments would be difficult to complete at the last minute. Thank you again for all your time and effort in improving our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.