- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CANARY studies whether gains from cached-response anchoring actually require risk-guided anchor selection. In a controlled continual post-training stream, it compares gradient-ranked, random, bottom-ranked, and cloned-update selectors, then examines risk-screened coverage. The results distinguish improvements over the base learner from the incremental value of selection.
Critical strengths
- The motivation is clear and practically relevant. Matching initialization, scoring operations, and anchor budgets while including random and reversed-ranking controls directly addresses attribution, rather than treating overall improvement as evidence for the selector.
- Reporting is unusually careful: six paired seed blocks, uncertainty intervals, complete outcomes, and explicit separation of local diagnostics from training outcomes support interpretation. The negative and mixed results are informative, and the paper appropriately limits its claims.
Critical weaknesses
- Representativeness of the training regime. All principal comparisons use one 3B model, one three-stage stream, and 24 optimizer steps per stage. The implemented live-teacher baseline reaches only 0.0347 mean held-out accuracy (Table 1). Although the scope is acknowledged, it remains unclear whether the observed benefits and selector orderings persist under a more stable base learner.
- Meaning of the measured improvements. Stream retention uses token F1 on 32 examples per task, while several held-out evaluations allow only eight generated tokens and count parsing failures as errors (Appendix D.3). These measurements do not adequately distinguish retained task competence from output-format stabilization. The separate-model stress tests cannot diagnose the main baseline, as the authors correctly note.
- Anchor composition (secondary). Empty cached responses fall back to original targets, but the fallback frequency is unknown. Moreover, token-averaged anchoring gives responses different effective weights despite matching anchor counts. These disclosed limitations leave the contribution of target source and length composition unresolved; they do not invalidate the complete-policy comparisons.
Questions for authors
- Is there evidence from one stabilized learner that the relative outcomes of no anchoring, random anchoring, and gradient-ranked anchoring persist? A focused comparison would suffice.
- Could existing final checkpoints support a small audit of answer validity and correctness with adequate decoding, clarifying how much the main performance gap reflects parsing or format?
- Can available caches or logs recover fallback frequencies and effective response-token distributions by arm? Descriptive accounting would already help interpret the selector comparisons.
3 comments
Thanks to the authors for an unusually candid response. The new runs answer my questions directly, but they mostly confirm my concerns: with a stable supervised learner the pre-registered retention criterion is not met on either backbone, and the authors acknowledge that the main live-teacher gain largely reflects restored answer validity rather than retained competence.
I value the transparency and the careful attribution design, but the evidence for the central claims remains limited, so I keep my score of 2.
What changed on each earlier task under supervised training
We appreciate your follow-up and agree with your reading of the supervised experiment. In the submitted task order, neither backbone met the criterion that both anchor rules lower final reference loss on both earlier tasks in every seed (Table 38, p. 41). Across six Qwen2.5-3B seeds, scored random and gradient ranking raised final ToolAlpaca loss by 0.140 and 0.133 nats relative to no anchors. The same runs lowered Chemistry loss by 0.045 and 0.040 nats, respectively (Appendix X.1, pp. 45–46). An average over the two tasks would hide that difference.
Your concern about answer format also changes how we interpret the original live-teacher result. Across three matched reruns, longer decoding raised the share of unanchored outputs with an identifiable answer from 0.91% to 27.51%. Under the original parser, the accuracy gap between scored random and no anchors fell from 31.23 to 22.72 percentage points (Table 25, p. 29). We cannot infer how much skill was retained from the remaining gap. Thank you for pressing us to make that distinction clear.
Stable learner, format versus competence, cache accounting
Thank you. Pages refer to the revised PDF; new text in blue.
W1/Q1. Stabilized learner. We ran one on the submitted stream: supervised reference loss, 48 updates per stage, seeds 410–412, two backbones read separately; endpoint: reference NLL (Table 4, p. 7). Without anchors it forgot measurably in every seed (Qwen2.5-3B / Qwen3-4B: Chemistry 0.130 / 0.051 nats, ToolAlpaca 0.016 / 0.082), except ToolAlpaca in one Qwen2.5-3B seed. The pre-registered criterion, both anchored arms below no anchors on final ToolAlpaca and Chemistry NLL in every seed, is met on neither backbone. The anchored arms ended lower on Chemistry in 11 of 12 cells (by 0.043–0.046 / 0.011–0.016; both intervals below zero on Qwen2.5-3B only; Qwen3-4B gradient top reaches zero) but higher on ToolAlpaca in all 12 (by 0.122–0.130 / 0.086–0.106; every interval above zero), and acquired less Medicine in 11 of 12 (Qwen3-4B scored random reaches zero). Gradient top versus scored random: lower ToolAlpaca loss in all six runs, less Medicine acquired in five, mixed Chemistry and held-out signs: no consistent difference. Stream token F1 (pre-registered; backbones not compared; counts format tokens, not correctness): on Qwen2.5-3B every interval reaches zero, but anchored arms gained 0.03–0.04 less Medicine F1 (not pre-registered); on Qwen3-4B anchored final ToolAlpaca F1 was higher in every seed (seed 412 reaching zero) while ToolAlpaca NLL rose, and gradient top cost Medicine F1 (−0.028; scored random −0.001, reaching zero), below scored random in every seed. Held-out accuracy was mixed (backbones not compared; pp. 43–44). Qwen2.5-3B: anchored arms higher on SciKnowEval Chemistry in every seed (only seed 412's intervals exclude zero; label-likelihood accuracy within 0.4 pp; more capped unanchored outputs, not pre-registered), gradient top lower on PubMedQA in two seeds (intervals below zero). Qwen3-4B: anchored arms lower on PubMedQA, every interval below zero; the pretrained model scored higher than every trained checkpoint on MedMCQA and PubMedQA. Since finished: our general comment and App. X.1.
With the live teacher, Chemistry and Medicine overlap after each task's own stage was below its untrained Stage 1 level in 16 of 18 seed–arm cells (p. 29); a one-seed fixed-teacher learner failed its Medicine check (p. 30).
On three synthetic two-task streams (addition → reversed initials on Qwen2.5-3B and on Qwen3-4B; tool execution → arithmetic; pp. 31–37), supervised learners that passed an acquisition check (four earlier ones failed) lost 8 (and gained 1), 0 and 0 of 187, 186 and 192 old-task answers without anchors (cells: 64 questions × 3 seeds). On the first panel anchors recovered 4 of the 8 ([0.00, +4.69] pp, reaching zero; McNemar p = 0.22). New-task answers against no anchors: scored random −8, +2, −5 and gradient top −6, +1, +3 by panel; only the −8 excludes zero ([−7.29, −1.56] pp). Limits: the first and third learners passed at the threshold (16 of 32); the first panel's old task was mostly known beforehand (30 of 32); arithmetic ended below the check level in 8 of 9 runs. Intervals cover question sampling, not training seeds.
W2/Q2. Format or competence. Mostly format, but not only; the split is not identified. We regenerated seeds 410–412 (the submitted final checkpoints were lost; SeenAvg reproduces to four decimals, p. 30) and re-decoded 500 items per benchmark with caps raised from 8 to 256 tokens (GSM8K 96 to 512). Submitted-parser accuracy went 5.05 → 15.71% without anchors and 36.28 → 38.43% with scored random, valid answers 0.91 → 27.51% and 64.04 → 65.52%; every unanchored output still hits the longer cap. On SciKnowEval, MMLU and MedMCQA the unanchored model is also less accurate among valid answers (63.3/56.8/45.1% vs 77.7/67.4/48.6%) and in label-likelihood ranking; on PubMedQA it is more accurate in both (not planned; Appendix N, p. 29).
In these seeds 1,131 of 1,152 cached targets, and 148–156 of 160 stream outputs per anchored run, comment on a "reference answer" only the live teacher sees; unanchored outputs do so in 1, 85 and 71 of 160 and otherwise loop on the same template words. The surviving submitted seed-410 caches match the regenerated ones (pp. 29–30).
W3/Q3. Cache. Not recoverable for the six submitted seeds (no pre-fallback generations logged). The surviving seed-410 gradient caches hold 192 targets, each with 98 loss-bearing tokens, none equal to its dataset target (p. 30). In the regenerated runs the anchored arms used no fallback in 1,152 cache generations, and 575 of 576 selected targets had 98 loss-bearing tokens: every cached response hit the 96-token cap (p. 29).
What changed in the paper. With the live teacher, scored random had the stronger means in six seeds, and gradient top is still 2.48 pp lower at longer caps (three seeds). Added: the limits in Q1–Q2; no selector was consistently better in any setting (pp. 6–10).