acceptodds
Under review as a conference paper at ICLR 2027

CANARY: Separating Behavioral Anchoring from Risk-Guided Selection in Continual LLM Post-Training

Abstract

Continual post-training often couples a retention objective with a policy for selecting the behaviors it protects, making their contributions difficult to separate. CANARY distinguishes cached stage-start response anchoring from risk-guided anchor allocation. Across six matched seed blocks with Qwen2.5-3B on a fixed tool-use, chemistry, and medical stream, gradient-ranked anchoring improves final stream performance, forgetting, and held-out accuracy over the implemented live-teacher on-policy self-distillation baseline in every block, with mean paired gains of , , and , respectively. Yet scored random and bottom-ranked anchors achieve stronger aggregate results. Across six saved states from one seed, gradient-conflict scores weakly agree with cloned one-microbatch AdamW loss changes, with mean Spearman correlation . Ranking by the cloned update yields favorable mean effects relative to both matched selector controls, with joint non-worsening in three of six runs against each. A four-seed risk-screened coverage follow-up improves mean held-out accuracy over random selection but worsens stream performance and forgetting. A separate supervised-replay comparison finds no benefit from adding the tested anchoring branch. These results distinguish the benefit of an anchored training policy, the fidelity of its local risk signal, and the downstream value of using that signal to allocate a fixed budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual learning, large language models, post-training, catastrophic forgetting, behavioral anchoring, risk-guided selection, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VgAXOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CANARY studies whether gains from cached-response anchoring actually require risk-guided anchor selection. In a controlled continual post-training stream, it compares gradient-ranked, random, bottom-ranked, and cloned-update selectors, then examines risk-screened coverage. The results distinguish improvements over the base learner from the incremental value of selection.

Critical strengths
  1. The motivation is clear and practically relevant. Matching initialization, scoring operations, and anchor budgets while including random and reversed-ranking controls directly addresses attribution, rather than treating overall improvement as evidence for the selector.
  2. Reporting is unusually careful: six paired seed blocks, uncertainty intervals, complete outcomes, and explicit separation of local diagnostics from training outcomes support interpretation. The negative and mixed results are informative, and the paper appropriately limits its claims.
Critical weaknesses
  1. Representativeness of the training regime. All principal comparisons use one 3B model, one three-stage stream, and 24 optimizer steps per stage. The implemented live-teacher baseline reaches only 0.0347 mean held-out accuracy (Table 1). Although the scope is acknowledged, it remains unclear whether the observed benefits and selector orderings persist under a more stable base learner.
  2. Meaning of the measured improvements. Stream retention uses token F1 on 32 examples per task, while several held-out evaluations allow only eight generated tokens and count parsing failures as errors (Appendix D.3). These measurements do not adequately distinguish retained task competence from output-format stabilization. The separate-model stress tests cannot diagnose the main baseline, as the authors correctly note.
  3. Anchor composition (secondary). Empty cached responses fall back to original targets, but the fallback frequency is unknown. Moreover, token-averaged anchoring gives responses different effective weights despite matching anchor counts. These disclosed limitations leave the contribution of target source and length composition unresolved; they do not invalidate the complete-policy comparisons.
Questions for authors
  1. Is there evidence from one stabilized learner that the relative outcomes of no anchoring, random anchoring, and gradient-ranked anchoring persist? A focused comparison would suffice.
  2. Could existing final checkpoints support a small audit of answer validity and correctness with adequate decoding, clarifying how much the main performance gap reflects parsing or format?
  3. Can available caches or logs recover fallback frequencies and effective response-token distributions by arm? Descriptive accounting would already help interpret the selector comparisons.
3 comments
AuthorsOpenReview ↗

Stable learner, format versus competence, cache accounting

Thank you. Pages refer to the revised PDF; new text in blue.

W1/Q1. Stabilized learner. We ran one on the submitted stream: supervised reference loss, 48 updates per stage, seeds 410–412, two backbones read separately; endpoint: reference NLL (Table 4, p. 7). Without anchors it forgot measurably in every seed (Qwen2.5-3B / Qwen3-4B: Chemistry 0.130 / 0.051 nats, ToolAlpaca 0.016 / 0.082), except ToolAlpaca in one Qwen2.5-3B seed. The pre-registered criterion, both anchored arms below no anchors on final ToolAlpaca and Chemistry NLL in every seed, is met on neither backbone. The anchored arms ended lower on Chemistry in 11 of 12 cells (by 0.043–0.046 / 0.011–0.016; both intervals below zero on Qwen2.5-3B only; Qwen3-4B gradient top reaches zero) but higher on ToolAlpaca in all 12 (by 0.122–0.130 / 0.086–0.106; every interval above zero), and acquired less Medicine in 11 of 12 (Qwen3-4B scored random reaches zero). Gradient top versus scored random: lower ToolAlpaca loss in all six runs, less Medicine acquired in five, mixed Chemistry and held-out signs: no consistent difference. Stream token F1 (pre-registered; backbones not compared; counts format tokens, not correctness): on Qwen2.5-3B every interval reaches zero, but anchored arms gained 0.03–0.04 less Medicine F1 (not pre-registered); on Qwen3-4B anchored final ToolAlpaca F1 was higher in every seed (seed 412 reaching zero) while ToolAlpaca NLL rose, and gradient top cost Medicine F1 (−0.028; scored random −0.001, reaching zero), below scored random in every seed. Held-out accuracy was mixed (backbones not compared; pp. 43–44). Qwen2.5-3B: anchored arms higher on SciKnowEval Chemistry in every seed (only seed 412's intervals exclude zero; label-likelihood accuracy within 0.4 pp; more capped unanchored outputs, not pre-registered), gradient top lower on PubMedQA in two seeds (intervals below zero). Qwen3-4B: anchored arms lower on PubMedQA, every interval below zero; the pretrained model scored higher than every trained checkpoint on MedMCQA and PubMedQA. Since finished: our general comment and App. X.1.

With the live teacher, Chemistry and Medicine overlap after each task's own stage was below its untrained Stage 1 level in 16 of 18 seed–arm cells (p. 29); a one-seed fixed-teacher learner failed its Medicine check (p. 30).

On three synthetic two-task streams (addition → reversed initials on Qwen2.5-3B and on Qwen3-4B; tool execution → arithmetic; pp. 31–37), supervised learners that passed an acquisition check (four earlier ones failed) lost 8 (and gained 1), 0 and 0 of 187, 186 and 192 old-task answers without anchors (cells: 64 questions × 3 seeds). On the first panel anchors recovered 4 of the 8 ([0.00, +4.69] pp, reaching zero; McNemar p = 0.22). New-task answers against no anchors: scored random −8, +2, −5 and gradient top −6, +1, +3 by panel; only the −8 excludes zero ([−7.29, −1.56] pp). Limits: the first and third learners passed at the threshold (16 of 32); the first panel's old task was mostly known beforehand (30 of 32); arithmetic ended below the check level in 8 of 9 runs. Intervals cover question sampling, not training seeds.

W2/Q2. Format or competence. Mostly format, but not only; the split is not identified. We regenerated seeds 410–412 (the submitted final checkpoints were lost; SeenAvg reproduces to four decimals, p. 30) and re-decoded 500 items per benchmark with caps raised from 8 to 256 tokens (GSM8K 96 to 512). Submitted-parser accuracy went 5.05 → 15.71% without anchors and 36.28 → 38.43% with scored random, valid answers 0.91 → 27.51% and 64.04 → 65.52%; every unanchored output still hits the longer cap. On SciKnowEval, MMLU and MedMCQA the unanchored model is also less accurate among valid answers (63.3/56.8/45.1% vs 77.7/67.4/48.6%) and in label-likelihood ranking; on PubMedQA it is more accurate in both (not planned; Appendix N, p. 29).

In these seeds 1,131 of 1,152 cached targets, and 148–156 of 160 stream outputs per anchored run, comment on a "reference answer" only the live teacher sees; unanchored outputs do so in 1, 85 and 71 of 160 and otherwise loop on the same template words. The surviving submitted seed-410 caches match the regenerated ones (pp. 29–30).

W3/Q3. Cache. Not recoverable for the six submitted seeds (no pre-fallback generations logged). The surviving seed-410 gradient caches hold 192 targets, each with 98 loss-bearing tokens, none equal to its dataset target (p. 30). In the regenerated runs the anchored arms used no fallback in 1,152 cache generations, and 575 of 576 selected targets had 98 loss-bearing tokens: every cached response hit the 96-token cap (p. 29).

What changed in the paper. With the live teacher, scored random had the stronger means in six seeds, and gradient top is still 2.48 pp lower at longer caps (three seeds). Added: the limits in Q1–Q2; no selector was consistently better in any setting (pp. 6–10).

Thanks to the authors for an unusually candid response. The new runs answer my questions directly, but they mostly confirm my concerns: with a stable supervised learner the pre-registered retention criterion is not met on either backbone, and the authors acknowledge that the main live-teacher gain largely reflects restored answer validity rather than retained competence.

I value the transparency and the careful attribution design, but the evidence for the central claims remains limited, so I keep my score of 2.

AuthorsOpenReview ↗

What changed on each earlier task under supervised training

We appreciate your follow-up and agree with your reading of the supervised experiment. In the submitted task order, neither backbone met the criterion that both anchor rules lower final reference loss on both earlier tasks in every seed (Table 38, p. 41). Across six Qwen2.5-3B seeds, scored random and gradient ranking raised final ToolAlpaca loss by 0.140 and 0.133 nats relative to no anchors. The same runs lowered Chemistry loss by 0.045 and 0.040 nats, respectively (Appendix X.1, pp. 45–46). An average over the two tasks would hide that difference.

Your concern about answer format also changes how we interpret the original live-teacher result. Across three matched reruns, longer decoding raised the share of unanchored outputs with an identifiable answer from 0.91% to 27.51%. Under the original parser, the accuracy gap between scored random and no anchors fell from 31.23 to 22.72 percentage points (Table 25, p. 29). We cannot infer how much skill was retained from the remaining gap. Thank you for pressing us to make that distinction clear.

Reviewer_XtFKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates catastrophic forgetting in continual LLM post-training from a practical and systematic perspective. It introduces CANARY, a framework that distinguishes the benefits of behavioral anchoring from those of risk-guided sample selection. Through controlled experiments, the authors evaluate different risk estimation and anchor selection strategies, revealing several meaningful findings. In particular, while behavioral anchoring can substantially mitigate forgetting, more sophisticated risk-based selection does not necessarily outperform random selection, and different risk estimation methods may exhibit limited agreement.

Overall, the paper provides useful empirical insights into important yet insufficiently examined aspects of continual learning. The experimental analysis is generally well motivated and conducted from multiple perspectives. However, some additional experiments would help strengthen the validation of the findings, particularly regarding the reliability of risk estimation and the generalizability of the conclusions.

Critical strengths
  1. The paper addresses an important and practical challenge in continual LLM post-training: understanding whether improvements in mitigating catastrophic forgetting arise from behavioral anchoring itself or from risk-guided sample selection. By explicitly separating these two factors, the authors provide meaningful insights into the effectiveness of existing strategies. In particular, the finding that sophisticated risk-guided selection does not necessarily outperform random selection challenges common assumptions and offers useful guidance for future research.

  2. The paper presents a well-structured empirical investigation, comparing different risk estimation and sample selection strategies under controlled experimental settings. The inclusion of random and low-risk selection baselines, multiple evaluation metrics, and multi-seed experiments strengthens the credibility of the findings. This systematic analysis is valuable because it helps distinguish improvements attributable to the anchoring mechanism from those genuinely introduced by the selection strategy.

Critical weaknesses
  1. Limited validation of risk estimation and score consistency. The paper uses Cloned-Update Risk to estimate the forgetting risk of historical samples. However, while the actual training employs gradient accumulation over four microbatches, the risk estimation simulates only a single-microbatch AdamW update. This approximation may not accurately reflect the actual parameter update, potentially affecting both risk estimation and sample ranking. A comparison between risk scores obtained from single-microbatch and full gradient-accumulation updates would help validate this approximation.

Furthermore, the score-consistency analysis in Section 6.2, which reports a low average Spearman correlation between Gradient Conflict and Cloned-Update Risk, is based on only six saved states from a single training seed. Although the main experiments report results across six seeds, extending this diagnostic analysis to multiple seeds would strengthen the reliability of this finding. These limitations are particularly important because understanding the relationship between different risk estimators is one of the paper’s central contributions.

  1. Limited generalizability of the experimental findings and coverage analysis.

The experiments are restricted to a single backbone (Qwen2.5-3B) and a fixed task sequence. While these controlled settings are useful for isolating different factors, they remain relatively limited compared with realistic continual LLM post-training scenarios. Additional experiments with different models, longer training schedules, or alternative task sequences would help establish whether the conclusions generalize beyond the current setting.

Moreover, the proposed Risk-screened Coverage strategy relies on lexical coverage rather than semantic diversity. Since lexically different prompts may express similar meanings, and semantically distinct prompts may share substantial vocabulary, lexical coverage may not adequately capture the diversity of historical behaviors. Comparing lexical coverage with embedding-based semantic coverage would provide a more informative evaluation of whether diversity-aware selection can improve retention.

Given that earlier continual learning studies, such as MIR and GSS, have already investigated the effectiveness of informed sample selection relative to random selection, broader validation would also help establish the distinct empirical contributions of this study in the context of continual LLM post-training.

Questions for authors
  1. Risk Estimation Reliability: How well does the single-microbatch Cloned-Update Risk approximate the actual four-microbatch accumulated AdamW update? Have the authors examined the consistency of the resulting risk scores and sample rankings?

  2. Stability of Score Agreement: The low Spearman correlation in Section 6.2 is based on six saved states from a single seed. Does this observation remain consistent across different training seeds?

  3. Lexical vs. Semantic Coverage: Would semantic coverage based on prompt embeddings lead to different sample selections or retention performance?

  4. Generalizability: To what extent do the main findings hold across different LLM backbones, task orders, and longer training schedules?

  5. Interpretation of Negative Results: What evidence supports the authors’ explanation for why risk-guided selection sometimes underperforms random selection? Could this result depend on the selection budget or redundancy among high-risk samples?

4 comments
AuthorsOpenReview ↗

Answers to Q1–Q3 and the relation to MIR and GSS (1/2)

Thank you for the five questions. Q1–Q3 are answered by replaying 18 saved training states and by a matched run on the submitted stream; Q4 only in part, and Q5 without a confirmed mechanism (part 2/2). Page numbers refer to the revised PDF; new text is blue.

W1/Q1. One versus four microbatches. At six saved states in each of three gradient-top runs we replayed the recorded AdamW step and scored the same 32 candidates with one and with all four task microbatches. "Excess loss recovered" is the share of the recorded top eight's loss increase over the pool mean that a score's own top eight captures (0% is a uniform draw's expectation).

Cloned risk computed fromSpearman with recorded stepRecorded top 8 recoveredExcess loss recovered
One task microbatch0.8636.00/889.6%
Four task microbatches0.8966.44/893.9%

The two rankings agree at mean Spearman 0.969 and share 7.11 of their top eight. The recorded step includes anchor gradients. The four-microbatch gain comes mostly from one run (+10.6, +0.9 and +1.3 pp) (§6.3, p. 8; Appendix N, p. 30; Appendix O, p. 31). With state and pool fixed, this measures ranking fidelity only.

End to end, the test is weak and does not answer the question for the submitted stream. We trained with each score on addition → reversed initials (Qwen2.5-3B), a synthetic two-task stream with a supervised learner; the same 64 held-out questions per task are scored in each of three seeds (192 answers per cell). Four- minus one-microbatch scoring gave +1 old-task answer ([0.00, +1.56] pp, which reaches zero; exact McNemar p = 1.00) and −1 new-task answer ([−3.12, +1.56]; p = 1.00). On that stream a mean 0.7 of 32 candidates per refresh has a predicted loss increase of at least 1e-4, and the scores agree less: Spearman 0.43, 0.45 and 0.84 at the first refresh, where state and pool are identical, and 4.97 of 8 anchors shared per refresh (Appendix W, p. 39).

W1/Q2. Agreement across seeds. The weak agreement replicates. With one microbatch, as submitted, mean Spearman between Gradient Conflict and cloned risk is 0.095, 0.114 and 0.069 by seed (submitted single seed: 0.101). With four it is 0.140, −0.047 and 0.234, and single states range from −0.49 to +0.89: the scores agree at some states and disagree at others, in every seed (pp. 8, 30).

W2/Q3. Lexical versus embedding-based coverage. The selections differ, but neither rule beat scored random jointly in any seed. We ran both, and scored random, on the submitted stream in seeds 410–412; seed 413 (lexical +4.60 pp over scored random) was not rerun. The coverage arms differ only in similarity: token-set Jaccard over the whole prompt, or cosine between all-MiniLM-L6-v2 embeddings of its first 256 tokens. That window holds most chemistry prompts but, of ToolAlpaca prompts (the only chemistry-stage candidates), only the API documentation before the user request. At the first refresh, with identical pools and scores, the two share 6, 5 and 4 of eight anchors.

Mean paired differenceSeenAvg (F1 points)Less forgetting (F1 points)True500 (pp)Joint wins
Semantic − lexical+0.66+1.10−2.872/3
Semantic − scored random+0.09−0.38−0.250/3
Lexical − scored random−0.56−1.49+2.610/3

Semantic coverage lost less stream F1 than lexical in every seed, by 0.56 to 1.92 points. Held-out accuracy moved by +0.32 and +0.28 pp in seeds 410 and 412 and by −9.20 pp in seed 411, where lexical coverage had its largest gain over scored random (+7.84 pp); there semantic coverage ended 1.36 pp below scored random. Three seeds, so no interval. The rerun lexical and scored-random arms reproduced the submitted selections at all 12 refreshes of each run, and the submitted endpoints (Appendix R, p. 35).

W2. MIR and GSS. Both already compare informed with random selection; we ran neither as a complete system (§9, p. 10). Cloned-update risk is MIR's virtual-update criterion carried to AdamW and cached LLM responses; Gradient Conflict equals it to first order under plain SGD (§4.1, p. 4). GSS uses gradient geometry to decide what enters the memory; our reservoir is fixed, and the closest rule here, prompt coverage, jointly beat scored random in two of four seeds without the risk screen and in none with it, as in Q3 (p. 25). On the three synthetic streams of part 2 (Q4), gradient top minus scored random was +2, −1 and +8 new-task answers; only the +8 excludes zero (p = 0.039, uncorrected, one of several such contrasts; pp. 33, 37, 37). What we add is a bottom-ranked control at matched scoring cost and a check of each score against the optimizer step: at eight anchors Gradient Conflict recovers 0.5% (one microbatch) or 10.4% (four) of the cloned update's excess loss, cloned risk 89.6% and 93.9% of the recorded step's (p. 31).

AuthorsOpenReview ↗

Q4–Q5: other settings, and what we know about ranking against random (2/2)

W2/Q4. Backbones, task orders, schedules. No matched live-teacher comparison exists on another backbone or schedule (stress tests of Qwen3-4B and Qwen2.5-7B and a 96-update fixed-teacher learner failed; pp. 28, 30), so we trained a supervised learner on the submitted stream (48 updates per stage, seeds 410–412, two backbones; reference NLL; pp. 7, 40–42). Unanchored forgetting (Qwen2.5-3B / Qwen3-4B: Chemistry 0.130 / 0.051 nats, ToolAlpaca 0.016 / 0.082) was measurable in every seed except ToolAlpaca in Qwen2.5-3B seed 411. The pre-registered criterion (both anchored arms below no anchors on final ToolAlpaca and Chemistry NLL in every seed) is met on neither backbone. Final Chemistry NLL was lower in 11 of 12 cells (both intervals below zero on Qwen2.5-3B only; Qwen3-4B gradient top reaches zero), but ToolAlpaca NLL was higher in all 12 (by 0.122–0.130 / 0.086–0.106; all intervals above zero). Cost: less Medicine acquired in 11 of 12 (0.022 and 0.025 of 0.174 / 0.003, reaching zero, and 0.007 of 0.123 nats). Gradient top minus scored random: lower final ToolAlpaca, higher final Medicine NLL in all six runs, mixed Chemistry and held-out signs: no consistent difference. Stream token F1 (pre-registered; backbones not compared; counts format tokens, not correctness): every Qwen2.5-3B interval reaches zero, but anchored arms gained 0.03–0.04 less Medicine F1 (not pre-registered); on Qwen3-4B anchored final ToolAlpaca F1 was higher in every seed (seed 412 reaching zero) while ToolAlpaca NLL rose, and gradient top cost Medicine F1 (−0.028; scored random −0.001, reaching zero). Held-out accuracy was mixed (backbones not compared; pp. 43–44). Qwen2.5-3B: anchored arms higher on SciKnowEval Chemistry in every seed (only seed 412's intervals exclude zero; label-likelihood accuracy within 0.4 pp; more capped unanchored outputs, not pre-registered), gradient top lower on PubMedQA in two seeds (intervals below zero). Qwen3-4B: anchored arms lower on PubMedQA, every interval below zero; the pretrained model scored higher than every trained checkpoint on MedMCQA and PubMedQA. Since finished: our general comment, App. X.1.

Three synthetic two-task streams used supervised learners that passed an acquisition check: addition → reversed initials on Qwen2.5-3B (first panel) and on Qwen3-4B, and tool execution → arithmetic on Qwen2.5-3B. Cells: 64 questions × 3 seeds (item-level 95% intervals in pp). Without anchors they lost 8 (and gained 1), 0 and 0 old-task answers. New-task answers against no anchors: scored random −8 ([−7.29, −1.56]), +2, −5; gradient top −6, +1, +3 (all but −8 reach zero; pp. 33, 37, 37). Limits: both Qwen2.5-3B learners qualified at the threshold (16 of 32), each after a failed attempt; arithmetic ended below that level in 8 of 9 cells; Qwen2.5-3B knew most addition answers beforehand.

Q5. Why ranking can lose with the live-teacher learner. We have no confirmed mechanism. Your candidates and those of §8:

  • Budget: 4, 8 and 16 anchors miss the criterion alike (supervised learner, Q4; App. X.1). At saved states Gradient Conflict recovers 0.5–3.4% (one microbatch) or 3.0–13.9% (four) of the cloned update's excess loss at k = 1, 4, 8 and 16 (p. 31). On the first panel gradient top and scored random differed by at most 3 answers per task at 4, 8 and 16 anchors (p. 34).
  • Redundancy: exact prompt reuse does not separate the arms here (63.50 vs 63.33 distinct of 96 slots; p. 30), but on Qwen3-4B gradient top reused more (37–46 vs 52–53; p. 37). Its anchors are 0.020 more alike in prompt embedding (−0.005 after adjusting for per-task counts), with lower same-stage coverage in every seed (−0.048; p. 35). Gradient redundancy is unmeasured.
  • Token weights: ruled out; 575 of 576 selected targets had 98 loss-bearing tokens (p. 10).
  • Target quality: at the 18 states Gradient Conflict's top eight have targets 0.022 F1 below their pools ([−0.035, −0.010]); partly task mix; the second pre-set measure showed no consistent difference for Gradient Conflict; overlap is not correctness (p. 36).
  • Wrong targets (first panel): gradient top used them in 21 of 288 slots, scored random 10 (p. 34); correcting them moved held-out answers by +2 per task, reaching zero (p. 38).
  • Placement: in the last stage gradient top put 33/15, 38/10 and 22/26 anchors on chemistry/tool prompts; scored random 70/74 pooled (p. 35). Outcomes disagree: gradient top ended below scored random on Chemistry in all three seeds (−1.3, −0.7, −2.2 F1), and its largest Tool deficit (−7.2) came in seed 412 (22/26).

What changed. With the live-teacher learner, gradient top still trails scored random, the anchoring gain is mainly in valid answers, and in seeds 410–412 most cached targets and anchored outputs comment on a "reference answer" only the live teacher sees (p. 29). No selector was consistently better; preserved capability is not shown on the submitted tasks.

Reviewer_XtFKOpenReview ↗

Main weak accept

I maintain my weak accept recommendation.

Strength: The revision strengthens the paper through additional validation of risk-score reliability and broader controlled experiments. I appreciate the distinction between the benefits of anchoring itself and the incremental value of risk-guided selection.

Weakness: The risk-screened lexical coverage strategy, however, shows trade-offs rather than consistent improvements over scored random selection or the embedding-based variant.

AuthorsOpenReview ↗

Does reordering the answer options change the seed 411 result?

Thank you for taking a closer look at the coverage comparison. Embedding coverage had less token-F1 forgetting than lexical coverage in all three seeds. Tables 31–32 (p. 35) compare average token F1 on seen tasks, forgetting, and held-out accuracy. In every seed, each coverage rule did worse than scored random on at least one of these measures.

Seed 411 had the largest gap in mean accuracy across the five held-out benchmarks. Lexical coverage scored 34.12%, embedding coverage 24.92%, and scored random 26.28%. Leaving out each benchmark in turn kept the lexical lead over embedding between 7.10 and 11.80 percentage points within that seed. The other two seeds slightly favored embedding.

We then took all 54 questions from the six math and logic subjects in the original MMLU panel. For each final checkpoint, we scored the most likely of the four answer letters directly. We also rotated the options so that each question’s correct answer appeared once in each position.

Lexical minus embedding, percentage pointsSeed 410Seed 411Seed 412
Original five-benchmark accuracy (Table 31)−0.32+9.20−0.28
54 math and logic questions, original option order+1.85+22.220.00
Same 54 questions, averaged over four option orders+0.93+6.02−1.85

On the original order, lexical, embedding and scored random answered 24, 12 and 16 of the 54 questions correctly in seed 411. Across the four orders they scored 86, 73 and 76 out of 216 presentations of those same questions. The original order accounted for 12 of the 13 net correct presentations separating lexical and embedding. Answer position therefore matters on this slice even when no generated answer has to be parsed. How much it contributes to the full five-benchmark gap remains open.

For the embedding comparison, ToolAlpaca’s API documentation filled the encoder’s 256-token window before the user request appeared (Appendix R, p. 35). We replayed 18 archived selection steps whose candidates all came from ToolAlpaca, keeping the model state, candidate pool and risk scores fixed at each step. Moving the request ahead of the documentation changed at least one of the eight anchors at 16 steps. Overlap with lexical selection rose at four, fell at eight and stayed the same at six. We have measured how the window changes selection. Its effect on accuracy and forgetting after retraining remains open, and we share your caution about preferring either coverage rule to scored random.

Reviewer_yQgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce CANARY, a controlled empirical framework, to separate behavioral anchoring from risk-guided selection in continual LLM post-training. More specifically, the authors separately research and test 3 subquestions: whether an anchored policy improves the base learner; whether a gradient-conflict score agrees with a cloned optimizer proposal; and whether allocating anchors using these signals improves complete training outcomes relative to matched selector controls. Through controlled experiments, the authors examine the contributions of behavioral anchoring and different selection strategies. The paper primarily emphasizes experimental attribution instead of introducing a new approach.

Critical strengths
  1. A meaningful research question that separates behavioral anchoring from risk-guided selection in continual LLM post-training.

  2. The experimental design is well structured within the evaluation setting. The experimental design helps isolate the contributions of different components.

Critical weaknesses
  1. The abstract focuses more on demonstrating results instead of clearly introducing the background and motivation of this work.

  2. An overall figure illustrating CANARY should be better included in the paper.

  3. The experimental scale is limited. The main experiments use only one LLM backbone, one continual learning setting, a single fixed sequence of three continual learning tasks, and relatively small training sets with only 24 optimizer steps per stage.

  4. Defensive writing is a bit too much.

  5. The paper evaluates continual-learning performance using token-level F1. However, token overlap does not necessarily reflect actual task accuracy.

Questions for authors
  1. Why only 1 microbatch is modeled instead of 4 in Gradient Conflict and Cloned-Update Risk?

  2. Since the cached responses for behavioral cloning are not necessarily correct, how to determine whether the observed reduction in forgetting reflects genuine preservation of task capabilities rather than consistency with potentially incorrect historical responses?

3 comments
AuthorsOpenReview ↗

Task correctness, cached targets, scale and presentation

Thank you. Pages refer to the revised PDF; new text in blue.

Q1. Why one microbatch? Only the first task-microbatch gradient exists when anchors are chosen. At 18 saved states of the submitted stream, one- and four-microbatch rankings agree at mean Spearman 0.969 for cloned risk and 0.656 for Gradient Conflict. Cloned risk tracks the recorded step; Gradient Conflict does not (p. 30). End to end on the first panel, four-microbatch cloned-risk scores changed held-out answers by +1 old-task ([0.00, +1.56], reaches zero; p = 1.00) and −1 new-task ([−3.12, +1.56]). A weak test: the score separates few candidates there (p. 39).

Q2. Capability, or agreement with cached errors? On the submitted stream we cannot show preserved capability. Scores use dataset references or answer keys, never the cache, but in seeds 410–412, 1,131 of 1,152 cached targets (pool-mean reference F1 0.11, p. 36) and 148–156 of 160 stream outputs per anchored run comment on a "reference answer" only the live teacher sees; unanchored ones loop on template words. The surviving submitted seed-410 caches match the regenerated ones (pp. 29–30). After each task's own stage, Chemistry and Medicine stream F1 was below its untrained Stage 1 level in 16 of 18 seed–arm cells (p. 29).

We scored exact answers on three synthetic two-task streams (addition → reversed initials on Qwen2.5-3B and on Qwen3-4B; tool execution → arithmetic) with a supervised learner that passes an acquisition check (64 questions × 3 seeds per cell). Without anchors it lost 8 (and gained 1), 0 and 0 of 187, 186 and 192 old-task answers. On the first panel anchors recovered 4 of the 8 ([0.00, +4.69] pp, item-level 95%, reaching zero; McNemar p = 0.22). New-task answers against no anchors: −8, +2, −5 (scored random) and −6, +1, +3 (gradient top) by panel (first panel: [−7.29, −1.56], [−6.77, +0.52]). Tool execution (learned from 0 of 32) stayed at 192 in every arm: no test of protection (pp. 33, 37, 37). Limits: two of three checks passed at the threshold (16 of 32); Qwen2.5-3B knew 30 of 32 addition questions beforehand (p. 32); arithmetic reached 32 of 64 in one of nine runs.

Do cached errors propagate? On the anchored prompts, yes. On the first panel gradient top trained on seven wrong cached answers and its final models repeated all seven; with only these targets corrected, all seven were right (no anchors: three). Held-out: +2 per task, both intervals reaching zero (p. 38).

W1, W2, W4. Abstract revised; new Figure 1 (p. 2); eleven sentences that only said what a result does not show cut; limits in §9.

W3. Scale. The live-teacher comparison has one backbone, one three-task order, 128 training examples and 24 updates per stage. Qwen3-4B collapsed in a live-teacher stress test, and a 96-update fixed-teacher learner failed its check (pp. 28, 30). A supervised learner on the submitted stream (48 updates per stage, seeds 410–412; p. 7) forgot measurably without anchors in every seed (ToolAlpaca 0.016 / 0.082 nats on Qwen2.5-3B / Qwen3-4B), except ToolAlpaca in one Qwen2.5-3B seed. The pre-registered criterion (both anchored arms below no anchors on final ToolAlpaca and Chemistry NLL in every seed) is met on neither backbone: anchored arms ended lower on Chemistry in 11 of 12 cells (both intervals below zero on Qwen2.5-3B only; Qwen3-4B gradient top reaches zero) but higher on ToolAlpaca in all 12 (by 0.122–0.130 / 0.086–0.106; every interval above zero), and acquired less Medicine in 11 of 12 (Qwen3-4B scored random reaches zero). Stream token F1 (pre-registered; backbones not compared): every Qwen2.5-3B interval reaches zero (not pre-registered: anchored arms gained 0.03–0.04 less Medicine F1); on Qwen3-4B gradient top cost Medicine F1 (−0.028; scored random −0.001, reaching zero). Held-out accuracy was mixed (backbones not compared; pp. 43–44). Qwen2.5-3B: anchored arms higher on SciKnowEval Chemistry in every seed (only seed 412's intervals exclude zero; label-likelihood accuracy within 0.4 pp; more capped unanchored outputs, not pre-registered), gradient top lower on PubMedQA in two seeds (intervals below zero). Qwen3-4B: anchored arms lower on PubMedQA, every interval below zero; the pretrained model scored higher than every trained checkpoint on MedMCQA and PubMedQA. Since finished: see our general comment and App. X.1.

W5. Token F1. We agree. New: Appendix D.3 on ToolAlpaca (p. 21). On held-out benchmarks (three rerun seeds), longer decoding cuts the scored-random advantage from 31.23 to 22.72 points; pooled accuracy among valid answers is then no higher with anchoring (on different items), so the clear effect is more valid answers (p. 29).

What a reader learns. On reference loss and exact answers, the live-teacher benefit did not carry over to learners that acquire tasks. No selector was consistently better; with the live teacher scored random had the stronger means.

I thank the authors for their detailed rebuttal and substantial additional experiments. I appreciate their efforts to address some of my concerns.

Yet, the authors acknowledge that the improvements in the original live-teacher setting largely reflect answer validity rather than demonstrated capability retention. Furthermore, the additional supervised-learning experiments do not satisfy the pre-registered retention criterion on either backbone. Also, the risk-guided selection does not consistently outperform random selection.

While these negative findings are informative, they also limit the strength and practical generalizability of the paper's conclusions. I appreciate the authors' transparency and careful empirical analysis, but I remain unconvinced that the current experimental evidence is sufficiently strong for acceptance.

I therefore maintain my original rating of 2 (Weak Rejection).

AuthorsOpenReview ↗

Which old answers did the anchors preserve?

Thank you for asking whether the model can retain a correct answer as well as reproduce a cached one. We traced the individual outcomes behind Table 28 (p. 33). The same 64 addition questions were scored in three seeds. After learning the new task, the unanchored model lost eight previously correct question–seed pairs. Both anchor rules kept the same three correct, each kept one more that the other missed, and both lost the remaining three. Each rule therefore preserved four of the eight. The net advantage over no anchors was four answers out of 192, with a 95% interval of 0.00 to 4.69 percentage points. On the new task, reversing initials, no anchors gave 99 correct answers out of 192, scored random gave 91 and gradient ranking gave 93.

The cache intervention lets us trace an error to its training target. Across the three seeds, we identified seven cases where gradient-ranked replay used a wrong addition answer. The final models repeated all seven errors. Replacing just those targets with the correct answers, while replaying the same selections, corrected all seven on the exposed prompts. On held-out questions, the corrected models gained two answers out of 192 on each task compared with the uncorrected models. Both intervals included zero (Appendix V, pp. 38–39).

On the submitted task order, neither supervised backbone passed the joint retention criterion (Table 38, p. 41). We also tried an executable catalogue task with a new counting convention. In a separate development pilot without anchors, the model went from 0/32 to 32/32 correct under the new convention and still answered all 32 cases correctly on each old operation (counting and lookup). With no old errors to prevent, we stopped before comparing anchor rules. The addition experiment gives us a small set of verifiable retained answers. We agree that the evidence for broader skill retention, or for a reliable benefit of gradient ranking, remains limited.

Comments on the paper

1 comment
AuthorsOpenReview ↗

What replay preserved across tasks, seeds, and anchor budgets

Thank you for asking what the accuracy gains with replay tell us about retention. With the original live teacher, allowing longer answers raised the share of unanchored outputs with an identifiable answer from 0.91% to 27.51%. The held-out accuracy gap between scored random replay and no anchors narrowed from 31.23 to 22.72 percentage points. We cannot tell how much retained skill accounts for the remaining gap (Table 25, p. 29).

The supervised runs give a clearer view of each task. In the submitted order, replay raised final reference loss on ToolAlpaca in all 12 comparisons with no anchors and lowered it on Chemistry in 11 (two selection rules, three seeds and two backbones, Table 38, p. 41). Three additional Qwen2.5-3B seeds showed the same trade-off. Neither reducing the budget from eight anchors to four nor raising it to sixteen met the criterion for improving both earlier tasks. Reversing the order met the sign criterion, but the Medicine intervals included zero, the model learned less on the new task, and two minibatch sequences also changed (Appendix X.1, pp. 45–46).

After the revision, we traced the individual answers behind Table 28 (p. 33). Across three seeds, the unanchored model lost eight previously correct addition answers. Each replay rule preserved four of those eight, including three that both preserved. Both rules also answered fewer new-task items correctly. Here we can identify specific answers that replay protected. Across the experiments, choosing examples by gradient ranking showed no consistent advantage over random selection at the same scoring cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.