acceptodds
Under review as a conference paper at ICLR 2027

BEYOND ENDPOINT SCORES: WHEN CONTINUAL KNOWLEDGE-UPDATING RANKINGS REVERSE

Abstract

Continual knowledge-updating methods are usually compared at a single final checkpoint. On a retrospective 24-update factual stream we show that this reporting choice, the initialization of the replay baseline, and the query form can each reverse the outcome of one and the same comparison. The testbed pairs a fast/slow LoRA hierarchy—a slow adapter re-fitted every K updates plus a small monthly adapter—against cumulative LoRA replay on Llama-3.2-1B and Qwen2.5-1.5B. Baseline: carrying replay's adapter weights across updates raises held-out paraphrase retention by 12.8 points at rank 8 and turns the hierarchy's +7.3-point full-history lead into a −5.4-point deficit, while lowering unchanged-fact accuracy by 12–19 points. Window: because the final update is a consolidation state for every tested K, the endpoint favors the hierarchy by 9–11 points even where warm replay leads by 5–18 points on average; the reversal appears in 16 of 18 seed-level paraphrase comparisons. Query form: training-form queries sit near ceiling under warm replay and cannot discriminate. An exact decomposition of reporting error into within-cycle phase and between-cycle drift explains why averaging the last cycle still selects the wrong winner in four of 18 cases. A retrospective drift check passes in seven of 12 aggregates and withholds the one wrong last-cycle sign, and spreading the same number of checkpoints over the stream removes every sign error on the warm panel. We distill these findings into a reporting protocol for continual-updater comparisons.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual knowledge updating, LLM evaluation, temporal evaluation, LoRA, experience replay, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7HzPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends an evaluation protocol of continual knowledge updating methods. The existing evaluation protocol reveals biased comparisons between methods, when only the final, consolidated checkpoint is evaluated. The main extension is the addition of the warm replay (carrying over weights) method into the evaluated set of methods.

Critical strengths
  1. The paper adds the warm replay method into an existing evaluation protocol.
  2. The paper presents a more rigorous set of experiments than the existing protocol, using 3 random seeds instead of 1.
Critical weaknesses
  1. The content of this paper significantly overlaps with an existing paper (Choi, 2026), which the paper did also cite in Line 103. However, the writing of this paper is much harder to follow than the existing paper, which suggests that this is not from the same author. Assuming this paper is independent from the existing work, the contribution of this paper is very limited. The existing evaluation protocol already used endpoint sweeping to highlight the endpoint bias. Hence, endpoint sweeping is not new in this paper, as claimed in Line 104. Also, the existing framework evaluates 3 models on the fresh replay method. This paper added the evaluation of warm replay, but only for one model.
  2. The following inconsistencies make the paper hard to follow:
  • In Lines 261 to 262, the reversal also happens at T=1 for K=3 and K=6, just like for K=12. However, the result is mentioned for K=12, but not for K=3 and K=6.
  • The caption of Figure 1b says that the drop is 20 to 25 points within each cycle, but the first drop from T=1 to T=5 is obviously larger than 25 points.
  • In Line 397, CRAFT is the name of the dataset in the cited paper, whereas KEDAS is the proposed method in the cited paper.
  1. The proposed sparse reporting rule seems to defeat the exact purpose of the evaluation protocol. As shown in Figure 1b, the three compared methods show a complicated rank-flipping trajectory over the 24-month period. Hence, a rule for sampling only a very small subset from the 24 timepoints has the risk of over-simplifying the observed dynamics. Moreover, even the proposed method works in this single dataset, the generalizability to different number of timepoints is not supported.

  2. Also, it is questionable that "A natural remedy for a consolidation-aligned endpoint is to average the last full cycle" (Line 269). Averaging over the last cycle obviously produces the same endpoint bias when comparing warm and fresh replay, as shown in Figure 1b. This method is then an artificially weak baseline, which probably does not deserve the full-length theoretical justification of Propositions 1 and 2.

Questions for authors

I have no further clarification questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7HzP

Thank you for the concrete corrections and for questioning the reporting recommendation. We will correct the errors and clarify the contribution.

Contribution and attribution. L103–105 is inaccurate: it presents the reporting-month sweep and time-averaged unchanged-fact accuracy as additions relative to the cited report, which also contains them. We will remove that comparative claim. As in the abstract, the contribution of the paper is the audit as a whole: for one updater decision judged against a declared full-history target, replay initialization, the reporting window and the query form can each reverse the outcome; a phase–drift decomposition accounts for the endpoint and last-cycle errors; and sparse reports are evaluated against the same target. The revision adds a conditions study across recipes and models. We do not propose a new updater or claim temporal inspection itself is new. A newly logged, post-hoc comparison also changes a choice within warm replay itself: at equal exposure, the lower learning rate wins the endpoint but loses the history average in 3/3 paired seeds on Llama-3.2-1B (1/3 on Llama-3.2-3B). A compact conditions table and consolidated notation will replace the scattered qualifications.

Textual corrections. Against the fixed 24-state reference, the complete sign-opposing month sets are K=3 {1,18,19,21,22,24}, K=6 {1,18,19,24}, K=12 {1,24}; month 1 was omitted from the first two lists. Fig. 1b: the first decline (updates 1→5) is 30.9 percentage points (pp), and subsequent refit-to-trough declines are 19.0, 21.4 and 25.8; the blanket “20–25” range will be corrected. At L397, CRAFT is the dataset and KEDAS the method.

Sparse estimates versus trajectories. Estimating a specified time-average and reconstructing every rank-switching time are different tasks. The submitted protocol prioritizes the paired trajectory (L425–431, items ii/v); sparse scoring is only a fallback when scoring every state is infeasible. We agree that a fixed midpoint grid can miss or over-weight phases. Its original 0/36 warm-panel errors are not a general guarantee; an in-sample always-warm decision already errs only 2/36. Across nested horizons H=6…24 at six checkpoints, midpoint versus final-six errors are 14 versus 21 of 679 warm cell-horizons, but 23 versus 20 of 631 fresh cell-horizons. These dependent prefixes show that midpoint is not uniformly preferable, not hundreds of independent replications. Correctly weighted randomized time-stratified sampling is unbiased for the mean, not for the selected winner; at six checkpoints, assumption-free identification intervals exclude zero in 0/72 checked cells. Table 6 will state actual counts: endpoint 1, consolidation-only 24/K, and final-K/midpoint-K/random-K each K. Equal checkpoint counts need not mean equal evaluation cost.

Last-cycle and the propositions. Last-cycle averaging is a diagnostic control, not our proposed generally reliable estimator. It is not equivalent to the endpoint: on the submitted seed panels, endpoint versus last-K errors are 16/36 versus 4/36 (warm) and 16/33 versus 0/33 (fresh), and warm mean absolute error falls from 13.90 to 2.90 pp. Yet rank-32, K=3 GEN history/last-cycle/endpoint margins are −4.60/+4.29/+11.15 pp. The identity separates within-cycle phase from between-cycle drift in the observed sequence; it is not a training-dynamics theorem. We will shorten the main-text exposition of Propositions 1–2; their proofs remain in Appendix A.

New boundary tests (post-submission, same source). GEN means held-out wording of the same facts scored by multiple-choice likelihood. On Llama at six epochs/update and LR 1e-4, K=6 hierarchy-versus-warm endpoint reversals disappear (0/3), while K=6 hierarchy-versus-fresh reversals persist (3/3). The newly completed K=3 lower-LR test also removes both endpoint and last-three reversals against warm replay (0/3 each; 3/3 each at 3e-4 in the matched replication). Thus the short-period failure is recipe-conditional.

At T=24 with K=5/7, hierarchy-versus-warm endpoint reversals are 0/3 and 1/3; the positive K=7 endpoint is +0.96 pp with a within-seed 95% subject-bootstrap CI [−1.00,+2.86]. Changing K also changes training exposure. Qwen's prespecified three-paraphrase mean gives 0/3 versus warm and 2/3 versus fresh. Random and relation-quota resamplings each give 3/3 hierarchy-versus-warm GEN endpoint reversals (one random-panel endpoint CI includes zero), but no last-six reversal in all six comparisons; these remain same-pool controls. An O-LoRA-style partial adaptation gives 0/3 reversals against the hierarchy, not a complete reproduction of published O-LoRA. All counts are descriptive, not tests across independent streams.

Does separating time-average estimation from trajectory reconstruction, and treating last-cycle as a diagnostic with these explicit boundaries, resolve the concern about the evaluation objective?

Reviewer_7HzPOpenReview ↗

Concerns Addressed

Thanks for providing a response to address my concerns. The authors agreed on over-claiming their contributions in the novelty of the evaluation protocol and in the last-cycle propositions. Thus, I will maintain my score.

Thank you for your response. To remove any misunderstanding, we would like to clarify the intent of our reply. We did not concede that we over-claimed our contribution on either point you mention. On the first, our reply corrected one sentence (lines 103–105), which described the content of the cited report inaccurately. That correction does not change our contribution, which is the evaluation as a whole, as stated in our reply.

As for Propositions 1–2, our intent is to improve readability by shortening their main-text exposition and to present last-cycle averaging as a diagnostic, not to withdraw them. Propositions 1–2 are correct and unchanged: they give an exact phase–drift accounting of the observed margins, they remain part of the contribution, and we do not concede that we over-claimed their novelty.

Reviewer_NsouOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits how replay initialization, reporting windows, and query form change continual knowledge-updating rankings. A phase–drift decomposition explains endpoint discrepancies, and sparse checkpoint comparisons motivate reporting across the update stream.

Critical strengths
  • The matched fresh/warm replay comparison and separate preservation measurements expose meaningful evaluation trade-offs.
  • The exact decomposition provides a clear account of phase and drift, with explicit boundaries on interpreting correlated comparisons.
Critical weaknesses
  • The warm-replay drift results use a fixed recipe that repeatedly trains six epochs on cumulative data. Without epoch or learning-rate sensitivity, it remains unclear how strongly the last-cycle failures depend on this particular optimization regime.
  • The warm comparison is limited to Llama-1B on one relation-concentrated retrospective stream, with one held-out paraphrase template for the main generalization score. Independent validation across another stream or updater family would make the reporting recommendations more compelling.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Nsou

Thank you. Your concerns separate into optimization dependence and validation breadth; the new results address both without claiming that the original warm-replay reversal is universal. F/W denote fresh/warm replay and H_K the hierarchy with period K. GEN is same-fact held-out multiple-choice paraphrase scoring. μ denotes the across-update margin, positive for the left method; pp denotes percentage points. Counts use three paired seeds per new contrast; 95% subject-bootstrap CIs condition on each training seed and stream.

1. Optimization: epochs, learning rate and the last-cycle failure. Warm replay's GEN change from update 12 to 24 is +6.62 to +8.12 pp at one epoch/update, −12.04 to −2.96 at two, and −24.46 to −19.55 at six. With H6 fixed, one-epoch W yields positive H6−W history margins but negative last-six means (3/3 last-cycle reversals, 0/3 endpoint reversals); two-epoch W gives negative history and last-six means (0/3 last-cycle reversals, 3/3 endpoint reversals). The one-epoch training-form history score is only about 81.5–81.8%, versus 99.2–99.3% at six epochs; it is not a tuned-best baseline.

With all arms at the preset LR 1e-4 and six epochs, both H6−W and F−W endpoint reversals disappear (0/3 each), while H6−F persists (3/3). We also completed the direct K=3 lower-LR test: H3−W history margins are −13.25/−11.77/−11.97 pp and last-three margins are −13.99/−11.79/−15.62 pp; endpoint and last-three reversals are both 0/3, versus 3/3 each in the matched LR 3e-4 replication. For seed 0, the history CI is [−14.51,−12.01] pp and the last-three CI [−15.59,−12.46]. Thus the submitted short-period last-cycle failure does not reproduce at the lower rate. This closes the missing K=3 control; it does not establish that last-cycle reporting is universally reliable.

A practical consequence remains even without a hierarchy. In an explicitly post-hoc comparison of W at the two LRs, with equal record exposure, the lower rate loses on history but wins at the endpoint in 3/3 seeds on Llama-3.2-1B (1/3 on Llama-3.2-3B). Seed 0: history −6.06 pp, CI [−7.26,−4.93]; endpoint +10.50 pp, CI [+8.68,+12.36]. Lower LR also preserves unchanged facts better, so this is target-dependent recipe selection, not overall superiority of the higher LR. The revision will show these trade-offs and the non-reversals; neither tested rate is claimed optimal.

2. Generality: wording, backbone and sample selection. Across three held-out MC phrasings and three seeds, Llama H6−W history margins are negative in 9/9 correlated seed×form comparisons. The three-form mean margins are −13.36/−13.01/−12.73 pp, with history CIs [−14.38,−12.41], [−13.97,−12.07], [−13.68,−11.81]; their endpoints are positive in 3/3. An individual template does not always reverse at the endpoint. This broadens wording coverage, not unseen-fact or validated free-generation ability.

On Qwen2.5-1.5B, the prespecified three-form mean gives 0/3 H6−W endpoint reversals, with one near-tie endpoint, and 2/3 H6−F reversals. We keep this negative backbone result prominent.

For the relation-concentrated stream, new random and relation-quota resamplings each give 3/3 H6−W GEN endpoint reversals under the original recipe. Mean history margins are −11.40 and −8.36 pp, respectively. Random seed 0's endpoint is +1.04 pp, CI [−0.28,+2.46]; all three quota endpoint CIs exclude zero. Last-six means agree with full history in all six new comparisons. This shows recurrence beyond the exact file-head sample, not an independent dataset or a causal isolation of relation concentration. We will disclose the original nonrandom caps and remaining historical-gold limitations.

The new O-LoRA-style orthogonality-regularized non-replay arm gives no endpoint/history reversal against H6 (0/3). It is a partial adaptation, not a complete published-editor reproduction; WISE is unsupported after its development gate. Independent-source and fully validated editor generality therefore remain open. The preregistered Llama-3.2-3B panel (same recipe) is complete: H6−W history margins are −3.78/−10.54/−8.47 pp and endpoints +22.54/+15.83/+14.83 pp (3/3 reversals, all CIs excluding zero); last-six means oppose history in 2/3. At LR 1e-4 on 3B, as on 1B, H6−W endpoint reversals disappear (0/3) while H6−F remains 3/3.

The revision will organize these results into a conditions table rather than a universal recommendation. Do these controls and narrowed claims address the concerns underlying your assessment, and which specific optimization or validation issue remains unresolved?

The revised PDF is uploaded; Section 4 and Table 4 collect the controls for both of your weaknesses, non-reversals included. Numbers are GEN (held-out wording of the same facts) unless stated; W/F are warm/fresh cumulative replay, H6 the period-six hierarchy, pp percentage points. Counts are descriptive (three seeds, one stream). Items marked "new" are post-submission and post hoc, outside the PDF and not pooled with submitted results.

W1, epochs at 3B (new run for this discussion). We trained W at 3B with one and two epochs per update (LR 3e-4). Only W's epochs change; H6 and six-epoch W are the 3B runs of Table 4. At 3B, W's GEN change from update 12 to 24 is +5.84 to +11.17 pp at one epoch, −1.17 to +1.21 at two and −25.63 to −17.21 at six. H6−W endpoint reversals: 0/3, 1/3, 3/3. At one epoch H6 leads in both windows; at two W leads by 5.38–5.96 pp over the history, with H6−W endpoint margins within 1 pp (−0.83 to +0.25; the +0.25 seed is the reversal).

At 1B (first reply) two epochs still gave a decline (−12.04 to −2.96 pp) and 3/3 endpoint reversals. So the dependence on the optimization regime is strong: at 3B the late decline and the consistent endpoint reversal against W appear only at six epochs of the three settings tested, and with every arm at LR 1e-4 that reversal is absent at both sizes (0/3; first reply). Six epochs is the submitted recipe for every arm. One-epoch W is under-trained (training-form history accuracy 83.0–83.8%; 93.6–93.9% at two; 99.3–99.4% at six). The endpoint misstates the history margin at every setting: with H6 unchanged, H6−W endpoint minus history is −3.27 to −0.89 pp at one epoch, +4.67 to +6.21 at two and +23.30 to +26.37 at six. These are point estimates without intervals, and last-cycle means were not computed for this panel. The PDF's epoch results are 1B only; this run addresses its stated limitation "no epoch sensitivity at 3B".

W1, selection across recipes (new). Among six configurations, all six-epoch (W/F/H6 at LR 3e-4 and 1e-4), choosing by endpoint gives up 11.26–12.70 pp of full-history accuracy against the menu's best at 1B and 0–1.20 pp at 3B. Neither rate was validation-tuned.

W2 (new). At 1B, with both arms at LR 1e-4 and six epochs, H6 trails bounded-reservoir replay (BR, 528 stored records) by 4.18–7.79 pp over the history and leads by 7.54–10.08 pp at the endpoint (3/3 reversals; all within-seed intervals exclude zero). But BR itself trails W by 10.11–12.12 pp over the history and 17.75–19.88 pp at the endpoint, and has no epoch test. It is LoRA replay on the same stream; an independently sourced stream and a faithfully reproduced published editor are still missing.

A second model family (new; one seed, no F). With the original recipe (six epochs, LR 3e-4) on OLMo-2-0425-1B, H6 trails W by 14.06 pp over the history but leads by 13.04 pp at the refit-aligned endpoint; the margin peaks at every refit update (1, 6, 12, 18, 24), and the last-six mean (−7.29 pp) keeps the history sign. W's GEN falls by 14.12 pp from update 12 to 24. On SmolLM2-360M the endpoint does not reverse (history −16.82, endpoint −12.00 pp), and W falls by only 3.13 pp. Across Llama, OLMo, Qwen and SmolLM2, the reversal appears exactly where W declines late. This broadens backbones, not streams.

Our claim is conditional: endpoint and full-history reports can disagree, and whether they do depends on the recipe and on alignment. Please let us know if a specific concern about the optimization controls remains.

Reviewer_6mj7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This evaluation paper audits one comparison on a 24-update Wikidata stream: a fast/slow LoRA hierarchy vs. cumulative LoRA replay, on Llama-3.2-1B and Qwen2.5-1.5B. It shows that three choices can each reverse the winner: replay initialization (fresh vs. warm), the reporting window (endpoint vs. full history), and the query form (training form vs. paraphrase). An exact phase–drift decomposition explains why last-cycle averaging can still fail. Spreading checkpoints evenly removes all sign errors on the warm panel, and the paper proposes a reporting protocol.

Critical strengths
  1. A clear and practically relevant message with a concrete demonstration. On the same runs, the full-history margin is −11.11 while the endpoint margin is +11.15. Endpoint reversals persist against a stronger warm-replay baseline in 16 of 18 seed-level paraphrase comparisons.
  2. A simple explanation and an actionable remedy. The phase–drift decomposition explains the failure of last-cycle averaging, the spread-K rule is supported at equal budget (Table 6), and the claims are carefully scoped.
Critical weaknesses
  1. The scope of the research is relatively narrow. The paper employs only a single data stream and a specific hierarchical structure, while relying exclusively on the LoRA replay mechanism. Furthermore, the model scale is limited to the 1 billion to 1.5 billion (1–1.5B) parameter range, and the “warm-up” phase covers only the Llama-1B model.
  2. Some of the observed effects stem from the experimental setup itself. Since T=24T = 24 is perfectly divisible by all values ​​of KK, the experiment always concludes in a state of “consolidation.”
Questions for authors
  1. Does the reversal persist when T is not divisible by K, or for a published method such as WISE or O-LoRA on the same stream?
  2. What is the base model's accuracy without any adapter, and how sensitive is warm replay's late decline to epochs per update?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6mj7

Thank you for identifying the controls needed to interpret this comparison. We now have direct post-submission tests, including outcomes that narrow the claim. F/W denote fresh/warm cumulative replay; H_K is the hierarchy with period K. GEN is four-choice likelihood scoring of held-out wording for the same trained facts. μ is the equal-update full-history margin; positive A−B favors A; pp denotes percentage points. Each training comparison below has three paired seeds; quoted 95% CIs resample subjects within a seed, not independent streams.

Non-divisible endpoints. With T=24, H5−W has 0/3 endpoint sign reversals and H7−W has 1/3; both full-history comparisons favor W. The K=7 exception is only +0.96 pp at the endpoint, CI [−1.00,+2.86]. Against F, both K=5 and K=7 give 0/3. Your alignment concern is therefore supported, not dismissed. Exposure also changes with the schedule: H5/H7/H6 use 89,622/80,022/101,256 record-exposures, so this is not a pure equal-cost phase intervention. A contextual period-choice comparison remains consequential: H5−H6 has a +1.34-pp mean history margin but endpoint margins −27.50 to −17.00 pp, reversing the selected period in 3/3 seeds.

Epochs, learning rate and exposure. Warm replay's GEN change from update 12 to 24 is +6.62 to +8.12 pp at one epoch/update, −12.04 to −2.96 at two, and −24.46 to −19.55 at six. Holding H6 fixed, the full-history H6−W sign also switches: +2.05 to +4.59 pp against one-epoch W, versus −7.19 to −6.68 at two and −12.70 to −11.26 at six. These are recipe effects, not evidence that carrying weights inherently causes decline; one-epoch W is substantially less trained on the training-form score.

At the prespecified lower LR 1e-4, with six epochs for all arms, H6−W endpoint reversals disappear (0/3), while H6−F remains 3/3. The new K=3 lower-LR comparison also has 0/3 endpoint and 0/3 last-three reversals against W. Neither LR was validation-tuned. Matching W's record exposure to H6 leaves H6 behind over the history (−8.18 to −5.95 pp), but removes its endpoint reversal (0/3). Thus the revision will state recipe, exposure and schedule explicitly.

Frozen-base control and preservation. Without adapters, item-weighted GEN accuracy on the 2,400 changed evaluation facts is 38.58% for Llama-1B and 33.63% for Qwen-1.5B (chance 25%). On the 300 unchanged facts it is 75.00% and 67.67%. In the new original-recipe Llama panel, unchanged GEN is 48.36–50.44% for W, 57.89–59.26% for H6, and 68.69–69.86% for F, all below that base score. These selected-item controls do not establish absence of pretraining overlap or preservation of general capabilities.

Breadth, with limits. On Qwen, the prespecified three-paraphrase H6−W mean has 0/3 endpoint reversals (one near-tie endpoint); H6−F has 2/3. Without any hierarchy, Llama F−W reverses in 3/3 seeds at LR 3e-4: seed 0 history −4.68 pp, CI [−5.85,−3.56], versus endpoint +13.87, CI [+11.79,+15.83]. This disappears at LR 1e-4 (0/3). Our new O-LoRA-style non-replay arm shows 0/3 reversals against W, F or H6; it omits components of published O-LoRA and is not a faithful reproduction. WISE failed its frozen development gate and is unsupported in this setup, not a test-stream baseline. The independent published-editor question is therefore only partly addressed. Random and relation-quota resamplings each give 3/3 H6−W GEN endpoint reversals, with one random-panel endpoint CI including zero; last-six means agree with history in all six. These are same-pool controls, not independent datasets. The preregistered Llama-3.2-3B panel (same recipe) is complete: H6−W GEN endpoint reversals 3/3 with all endpoint CIs excluding zero, H6−F 3/3, F−W 2/3, H6−W training form 0/3. At LR 1e-4 on 3B, as on 1B, H6−W endpoint reversals disappear (0/3) while H6−F remains 3/3.

Finally, Table 6's “equal budget” wording needs correction despite the encouraging original result: endpoint uses 1 checkpoint, consolidation-only 24/K, and final-K/midpoint-K/random-K each K; equal checkpoint counts need not imply equal evaluation cost. We retain full trajectories and no universal sparse-rule claim.

Do these controls resolve the alignment and replay-recipe concerns for a scoped evaluation contribution, or is a specific remaining limitation decisive for your assessment?

Thank you for specifying these controls; the revision collects them in Section 4 and Table 4. Numbers are GEN (held-out wording of the same facts) unless stated; W/F are warm/fresh cumulative replay, H6 the period-six hierarchy, pp percentage points. Counts are descriptive (three seeds). Items marked "new" are post-submission and post hoc, outside the PDF and not pooled with submitted results.

Epochs at 3B (new; your second question). We trained W at 3B with one and two epochs per update (LR 3e-4). Only W's epochs change; H6 and six-epoch W are Table 4's 3B runs. At 3B, W's GEN change from update 12 to 24 is +5.84 to +11.17 pp at one epoch, −1.17 to +1.21 at two and −25.63 to −17.21 at six. H6−W endpoint reversals: 0/3, 1/3, 3/3. At one epoch H6 leads in both windows; at two W leads by 5.38–5.96 pp over the history, with H6−W endpoint margins within 1 pp (−0.83 to +0.25; the +0.25 seed is the reversal).

At 1B (first reply) two epochs still gave a decline (−12.04 to −2.96 pp) and 3/3 reversals. So the late decline is highly sensitive to epochs: at 3B it appears only at six epochs of the three tested, and so does the consistent reversal against W. Since H6 and its refit-aligned endpoint are the same in all three comparisons, alignment alone does not produce that reversal. This confirms your recipe concern; six epochs is the submitted recipe for every arm. One-epoch W is under-trained (training-form history accuracy 83.0–83.8%; 93.6–93.9% at two; 99.3–99.4% at six). The endpoint misstates the history margin at every setting: with H6 unchanged, H6−W endpoint minus history is −3.27 to −0.89 pp at one epoch, +4.67 to +6.21 at two and +23.30 to +26.37 at six. These are point estimates without intervals. The PDF's epoch results are 1B only; this run fills its stated 3B gap.

Bounded replay (new). At 1B, with both arms at LR 1e-4 and six epochs, H6 trails bounded-reservoir replay (BR, 528 stored records) by 4.18–7.79 pp over the history and leads by 7.54–10.08 pp at the endpoint (3/3 reversals; all within-seed intervals exclude zero). But BR itself trails W by 10.11–12.12 pp over the history and 17.75–19.88 pp at the endpoint, and has no epoch test.

Selection across recipes (new). Among six configurations, all six-epoch (W/F/H6 at LR 3e-4 and 1e-4), choosing by endpoint gives up 11.26–12.70 pp of full-history accuracy against the menu's best at 1B and 0–1.20 pp at 3B.

A second model family (new; one seed, no F). With the original recipe (six epochs, LR 3e-4) on OLMo-2-0425-1B, H6 trails W by 14.06 pp over the history but leads by 13.04 pp at the refit-aligned endpoint; the margin peaks at every refit update (1, 6, 12, 18, 24), and the last-six mean (−7.29 pp) keeps the history sign. W's GEN falls by 14.12 pp from update 12 to 24. On SmolLM2-360M the endpoint does not reverse (history −16.82, endpoint −12.00 pp), and W falls by only 3.13 pp. Across Llama, OLMo, Qwen and SmolLM2, the reversal appears exactly where W declines late.

How common is endpoint-only reporting? (published tables) Lifelong-editing papers usually score the model once, after the last edit: WISE evaluates “the final post-edit model” after the T-th edit, and GRACE reports metrics “computed after all sequential edits”. Where WISE reports several horizons (Table 2, ZsRE), 18 of 36 method pairs on LLaMA-2-7B and 16 of 36 on Mistral-7B change order in average score between T=1 and T=1000; most of these changes involve editors that degrade as edits accumulate.

An independent stream and a faithfully reproduced published editor remain missing. Our claim is conditional: endpoint and full-history reports can disagree, and whether they do depends on the recipe and on alignment. We therefore recommend reporting the paired trajectory with its endpoint error and every arm's recipe. Please let us know if a specific technical concern remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.