- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposed Randomised-Pass Replay (RPR) for reducing the rehearsal gap in continual learning. Experimental results show the effectiveness of the proposed RPR compared with i.i.d and fixed cycle though the improvement depends on batchsize and buffer capacity.
Critical strengths
The concept of rehearsal gap is interesting. Theoretical definition of upper bound of rehearsal gap is also the strength of this paper.
Critical weaknesses
Experimental results are insufficient.
1-1. The authors discuss only the relation between batchsize and the number of class. However, there is the relation among the number of class, batchsize, and buffer capacity. It would be better to include the analysis about it.
1-2. The dataset you used is too small to show the effectiveness of your method in real situation. In real applications, the number of class is large, the buffer capacity and batchsize are small. Will be the proposed RPR effective in the real situation? The authors should evaluate your method on larger datasets such as iNaturalist.
1-3. Table 3 shows that the proposed method is worse than the baseline when b=32. There is no discussion and analysis about it. The authors should clarify the reason why the proposed RPR decreases the accuracy.
1-4. Similary with 1-3, there is no ablation study regarding the batchsize and buffer capacity. For example, in Table 3, the authors evaluate the result only for buffer capacity of 500 when b=32. The authors show the accuracy while changing batchsize and buffer capacity, and clarify the limitation.
1-5. The experimental setting in Table 1 remains unclear. Implementation details noted that buffer capacities range is from 200 to 5,120. The authors should clarify buffer capacity of each dataset.
This paper is hard to read and interpret.
2-1. There is little discussion and analysis regarding experimental results.
2-2. There is little explanation of Figures and Tables.
2-3. This paper refers to the supplementary material as if they were part of the main text. The main manuscript should be self-contained and understandable without the supplementary material.
Questions for authors
Please respond to the Weakness I pointed out.
Dear Chairs and Reviewers,
Thank you for the time and care you have devoted to our submission. To respect your time, we briefly summarise the main concerns, the additional evidence, and its implications.
Contribution beyond reshuffling and fixed cycling (KgY7, yRZ2). We acknowledge that reshuffling is standard and clarify the replay-specific theoretical contribution. To test whether the advantage over numeric cycling merely reflects class-ID ordering, we added a permanent random-order comparator. RPR gains 0.527 percentage points across 30 paired seeds, providing evidence beyond the numeric-order comparison, although this does not isolate the benefit of renewed pairings.
Replay-frequency confounding (yRZ2). We report head/mid/tail replay counts and acknowledge that online counts are not exactly matched. An additional control fixes per-class replay counts, pair co-occurrence totals, and buffer-label trajectories. RPR order gains 0.367 points over permuted batches, supporting an effect of temporal order after matching these totals, without isolating gap length from other temporal dependencies.
Mechanism for cosine classifiers (KgY7, yRZ2). We added a conditional margin-stability analysis and direct weight-rotation and signed-margin measurements. Longer absence is associated with greater margin degradation, but RPR does not uniformly reduce rotation or aggregate episode-level margin loss. These diagnostics examine the proposed explanation directly; they do not establish causal mediation of the final accuracy gains.
Memory, budget, scale, and generality (EzZU, yRZ2, KgY7). A matched memory-by-batch experiment gives +1.060 points at memory 500/replay 8, addressing dependence on the large primary buffer; replay-32 effects remain inconclusive. With three updates per incoming batch, RPR retains a +0.624-point advantage over independent retrieval. Following EzZU’s request, a naturally imbalanced 1,000-species iNaturalist evaluation gives +0.960 points with memory 500, extending the evidence to a larger vocabulary. These exploratory results broaden the evaluation while retaining the negative and inconclusive findings across losses and backbones.
We also clarified experimental settings and expanded the main-text explanations, addressing EzZU’s and yRZ2’s presentation concerns. Additional comparisons are labelled exploratory with unadjusted paired intervals.
Thank you again for your constructive feedback, which has helped us strengthen the evaluation and state the contribution and limitations more precisely.
Kind regards,
The Authors