- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper extends an evaluation protocol of continual knowledge updating methods. The existing evaluation protocol reveals biased comparisons between methods, when only the final, consolidated checkpoint is evaluated. The main extension is the addition of the warm replay (carrying over weights) method into the evaluated set of methods.
Critical strengths
- The paper adds the warm replay method into an existing evaluation protocol.
- The paper presents a more rigorous set of experiments than the existing protocol, using 3 random seeds instead of 1.
Critical weaknesses
- The content of this paper significantly overlaps with an existing paper (Choi, 2026), which the paper did also cite in Line 103. However, the writing of this paper is much harder to follow than the existing paper, which suggests that this is not from the same author. Assuming this paper is independent from the existing work, the contribution of this paper is very limited. The existing evaluation protocol already used endpoint sweeping to highlight the endpoint bias. Hence, endpoint sweeping is not new in this paper, as claimed in Line 104. Also, the existing framework evaluates 3 models on the fresh replay method. This paper added the evaluation of warm replay, but only for one model.
- The following inconsistencies make the paper hard to follow:
- In Lines 261 to 262, the reversal also happens at T=1 for K=3 and K=6, just like for K=12. However, the result is mentioned for K=12, but not for K=3 and K=6.
- The caption of Figure 1b says that the drop is 20 to 25 points within each cycle, but the first drop from T=1 to T=5 is obviously larger than 25 points.
- In Line 397, CRAFT is the name of the dataset in the cited paper, whereas KEDAS is the proposed method in the cited paper.
-
The proposed sparse reporting rule seems to defeat the exact purpose of the evaluation protocol. As shown in Figure 1b, the three compared methods show a complicated rank-flipping trajectory over the 24-month period. Hence, a rule for sampling only a very small subset from the 24 timepoints has the risk of over-simplifying the observed dynamics. Moreover, even the proposed method works in this single dataset, the generalizability to different number of timepoints is not supported.
-
Also, it is questionable that "A natural remedy for a consolidation-aligned endpoint is to average the last full cycle" (Line 269). Averaging over the last cycle obviously produces the same endpoint bias when comparing warm and fresh replay, as shown in Figure 1b. This method is then an artificially weak baseline, which probably does not deserve the full-length theoretical justification of Propositions 1 and 2.
Questions for authors
I have no further clarification questions.
3 comments
Concerns Addressed
Thanks for providing a response to address my concerns. The authors agreed on over-claiming their contributions in the novelty of the evaluation protocol and in the last-cycle propositions. Thus, I will maintain my score.
Thank you for your response. To remove any misunderstanding, we would like to clarify the intent of our reply. We did not concede that we over-claimed our contribution on either point you mention. On the first, our reply corrected one sentence (lines 103–105), which described the content of the cited report inaccurately. That correction does not change our contribution, which is the evaluation as a whole, as stated in our reply.
As for Propositions 1–2, our intent is to improve readability by shortening their main-text exposition and to present last-cycle averaging as a diagnostic, not to withdraw them. Propositions 1–2 are correct and unchanged: they give an exact phase–drift accounting of the observed margins, they remain part of the contribution, and we do not concede that we over-claimed their novelty.
Response to Reviewer 7HzP
Thank you for the concrete corrections and for questioning the reporting recommendation. We will correct the errors and clarify the contribution.
Contribution and attribution. L103–105 is inaccurate: it presents the reporting-month sweep and time-averaged unchanged-fact accuracy as additions relative to the cited report, which also contains them. We will remove that comparative claim. As in the abstract, the contribution of the paper is the audit as a whole: for one updater decision judged against a declared full-history target, replay initialization, the reporting window and the query form can each reverse the outcome; a phase–drift decomposition accounts for the endpoint and last-cycle errors; and sparse reports are evaluated against the same target. The revision adds a conditions study across recipes and models. We do not propose a new updater or claim temporal inspection itself is new. A newly logged, post-hoc comparison also changes a choice within warm replay itself: at equal exposure, the lower learning rate wins the endpoint but loses the history average in 3/3 paired seeds on Llama-3.2-1B (1/3 on Llama-3.2-3B). A compact conditions table and consolidated notation will replace the scattered qualifications.
Textual corrections. Against the fixed 24-state reference, the complete sign-opposing month sets are K=3 {1,18,19,21,22,24}, K=6 {1,18,19,24}, K=12 {1,24}; month 1 was omitted from the first two lists. Fig. 1b: the first decline (updates 1→5) is 30.9 percentage points (pp), and subsequent refit-to-trough declines are 19.0, 21.4 and 25.8; the blanket “20–25” range will be corrected. At L397, CRAFT is the dataset and KEDAS the method.
Sparse estimates versus trajectories. Estimating a specified time-average and reconstructing every rank-switching time are different tasks. The submitted protocol prioritizes the paired trajectory (L425–431, items ii/v); sparse scoring is only a fallback when scoring every state is infeasible. We agree that a fixed midpoint grid can miss or over-weight phases. Its original 0/36 warm-panel errors are not a general guarantee; an in-sample always-warm decision already errs only 2/36. Across nested horizons H=6…24 at six checkpoints, midpoint versus final-six errors are 14 versus 21 of 679 warm cell-horizons, but 23 versus 20 of 631 fresh cell-horizons. These dependent prefixes show that midpoint is not uniformly preferable, not hundreds of independent replications. Correctly weighted randomized time-stratified sampling is unbiased for the mean, not for the selected winner; at six checkpoints, assumption-free identification intervals exclude zero in 0/72 checked cells. Table 6 will state actual counts: endpoint 1, consolidation-only 24/K, and final-K/midpoint-K/random-K each K. Equal checkpoint counts need not mean equal evaluation cost.
Last-cycle and the propositions. Last-cycle averaging is a diagnostic control, not our proposed generally reliable estimator. It is not equivalent to the endpoint: on the submitted seed panels, endpoint versus last-K errors are 16/36 versus 4/36 (warm) and 16/33 versus 0/33 (fresh), and warm mean absolute error falls from 13.90 to 2.90 pp. Yet rank-32, K=3 GEN history/last-cycle/endpoint margins are −4.60/+4.29/+11.15 pp. The identity separates within-cycle phase from between-cycle drift in the observed sequence; it is not a training-dynamics theorem. We will shorten the main-text exposition of Propositions 1–2; their proofs remain in Appendix A.
New boundary tests (post-submission, same source). GEN means held-out wording of the same facts scored by multiple-choice likelihood. On Llama at six epochs/update and LR 1e-4, K=6 hierarchy-versus-warm endpoint reversals disappear (0/3), while K=6 hierarchy-versus-fresh reversals persist (3/3). The newly completed K=3 lower-LR test also removes both endpoint and last-three reversals against warm replay (0/3 each; 3/3 each at 3e-4 in the matched replication). Thus the short-period failure is recipe-conditional.
At T=24 with K=5/7, hierarchy-versus-warm endpoint reversals are 0/3 and 1/3; the positive K=7 endpoint is +0.96 pp with a within-seed 95% subject-bootstrap CI [−1.00,+2.86]. Changing K also changes training exposure. Qwen's prespecified three-paraphrase mean gives 0/3 versus warm and 2/3 versus fresh. Random and relation-quota resamplings each give 3/3 hierarchy-versus-warm GEN endpoint reversals (one random-panel endpoint CI includes zero), but no last-six reversal in all six comparisons; these remain same-pool controls. An O-LoRA-style partial adaptation gives 0/3 reversals against the hierarchy, not a complete reproduction of published O-LoRA. All counts are descriptive, not tests across independent streams.
Does separating time-average estimation from trajectory reconstruction, and treating last-cycle as a diagnostic with these explicit boundaries, resolve the concern about the evaluation objective?