- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies \emph{calibration-memory allocation} for conformal prediction (CP) in class-incremental learning (CIL). The authors observe that under a fixed memory budget, calibration samples of old classes cannot be recovered once discarded, while the calibration demand is heterogeneous across classes and evolves over tasks. Existing work either assumes sufficient calibration resources or focuses on constructing conformal uncertainty estimates, leaving the problem of calibration-memory allocation under a fixed budget unexplored. The problem is formalized as the constrained optimization in Eq. (4) (a fixed total budget together with the irreversibility constraint ), and three theoretical results are provided:Theorem 1 (finite-calibration disagreement law ), Theorem 2 (oracle allocation , with equal allocation optimal only when all coincide), and Theorem 3 (a lower bound on irreversibility-induced future risk). Building on these, the authors propose \paper{}, a three-stage framework (Current Demand Estimation, Irreversibility-Aware Future Protection, Marginal Budget Allocation), and evaluate it on CIFAR-10/100 and Tiny-ImageNet with four CIL learners (ER, DER, MEMO, TagFex). Relative to Equal Allocation, \paper{} improves prediction-set stability (Jaccard, Disagreement) while leaving coverage and average set size essentially unchanged.
Critical strengths
Well-chosen and cleanly formalized problem. The two constraints in Eq.(4) a fixed total and a nested (monotone) sample store, capture the real tension in fixed-memory CIL, and are structurally the same as those of a replay buffer. The problem statement in Sec.3 is unambiguous.
Tight theory-to-algorithm correspondence. The three theorems map one-to-one onto the three stages (Thm. 1 -> Stage 1, Thm. 3-> Stage 2, Thm. 2 -> Stage 3), so this is not a method first, theory bolted on later paper. Theorem 2 yields a falsifiable claim: equal allocation is oracle-optimal only if all are equal. The 3/2 transformation is elegant. The authors transform the protected target support back into the demand space using the following formula: , making the third stage still a single allocation process.
Critical weaknesses
The authors claim that "different finite calibration samples may produce different prediction sets... These differences can change downstream decisions and make uncertainty quantification less trustworthy." However, the experiments in the entire paper (Table 1, Table 2, Fig. 3, Appendix C) measure only four internal CP metrics: Jaccard, Disagreement, Coverage, and Set Size. There is no experiment that measures the quality of any downstream decision. In other words, the paper claims that differences in prediction sets will change downstream decisions, but never demonstrates the consequences of those differences.
The paper treats as standing alongside an exogenous replay budget, yet never states the relationship between the two. Eq. (4) constrains only ; in the experiments is an independently given constant (Eq. (63) in Appendix C.1) that does not participate in gradient updates. The text never imposes a constraint of the form , nor does it clarify whether is disjoint from the replay buffer. Two consequences follow. (i) If is additional memory, this contradicts the paper's own fixed-memory premise ("unconstrained retention would make memory consumption grow continuously"). (ii) If is carved out of a fixed total, then every sample moved to calibration is one removed from replay, and accuracy must necessarily be affected. That is precisely the experiment capable of answering "is it useful for continual learning?", and it is omitted entirely.
Sec. 6.1 and Appendix C.1 state that Equal Allocation is the only baseline, on the grounds that "existing methods do not directly address calibration-budget allocation." That reasoning explains the absence of an off-the-shelf method for the same task, but it does not explain why existing allocation rules were not brought in as ablations. Using ER/DER/MEMO/TagFex partially addresses the training side and, to some extent, the demand for a comparison against replay methods; strictly speaking, however, the most directly opposed baselines are the memory-management strategies already established in the CIL literature (iCaRLherding, GDumb greedy balancing, Class-Balanced allocation) repurposed as rules for how many calibration samples each class receives.
Questions for authors
Q1. What exactly is the relationship between the calibration memory and the replay buffer ? Are they disjoint, i.e. are calibration samples partitioned out of the replay samples? If total memory is fixed, what is the accuracy cost of assigning memory to calibration?
Q2. Suppose part of is returned to training (for example, halving and doubling replay). Does the stability advantage still hold? Has this trade-off ever been attempted?
Q3. Please provide evidence on at least one downstream metric of the class-incremental learning task itself (for example, average accuracy at any point in time) showing that improved prediction-set stability translates into better decisions.
Q4. If each class is given a fixed per-class cap instead of an equal share, does the method still show a significant advantage? This bears directly on the practical value of Theorem 2.
3 comments
rebuttal improves clarity but does not substantially address the issues
I appreciate the authors’ detailed rebuttal and the clarification that replay memory and calibration memory are disjoint partitions of a fixed total memory budget. However, this clarification ultimately reinforces my main concern: every calibration sample necessarily comes at the expense of a replay sample, yet the paper does not evaluate this trade-off or show whether the stability gains justify the loss of replay capacity.
More importantly, the paper motivates BACA through downstream reliability and decision consistency, but the experiments only report CP-internal metrics such as Jaccard similarity and disagreement. While the rebuttal cites prior work showing why prediction-set instability may matter, it does not demonstrate any downstream benefit in the continual-learning setting considered here.
Overall, the rebuttal improves clarity but does not substantially address the practical significance or experimental completeness. Therefore, I prefer to maintain my score at 2.
Clarifying the Memory Trade-off and Practical Significance of Stability Improvements
We thank the reviewer for raising these important concerns.
Regarding the memory trade-off, BACA does not require additional calibration memory or reduce replay capacity compared with Equal Allocation. Both methods operate under the same fixed total memory budget and identical preallocated Replay Buffer and Calibration Memory sizes. They differ only in how the fixed calibration budget is distributed across classes. Therefore, BACA's stability gains are achieved without sacrificing additional replay capacity.
Importantly, our contribution is to improve calibration memory utilization through adaptive class-wise allocation under a fixed budget, rather than to increase memory consumption. BACA considers both current class-wise calibration demand and irreversibility-aware future protection when allocating calibration samples. Compared with Equal Allocation, BACA improves Jaccard similarity by 3.54 percentage points and reduces prediction-set disagreement by 7.76 percentage points on average, while maintaining comparable empirical coverage and prediction-set size. These improvements are achieved through more effective allocation of the existing calibration budget, without additional memory consumption.
Regarding downstream benefits, improved prediction-set stability can make subsequent decisions less sensitive to random calibration sampling. For example, consider an industrial monitoring system that determines inspection priorities based on prediction sets. For the same observation, different calibration subsets may produce different candidate sets, such as {} and {}. Under a policy that triggers urgent inspection whenever a critical fault is among the candidates, these outputs lead to different inspection priorities. By reducing such prediction-set variations, BACA can promote more consistent risk assessment, reduce unnecessary changes in inspection priorities, and make inspection resource allocation more predictable. Such consistency is also beneficial for reproducible risk assessment and operational auditing.
We acknowledge that quantifying the actual utility of these benefits requires application-specific decision policies and cost models, which we leave for future work.
Further Clarifications on Practical Implications and Calibration Memory Management
W1 and Q3
We thank the reviewer for raising this concern. Papamichalis et al. (2026) identify three practical consequences of prediction-set inconsistency : (1) hospitals may fail to establish a consistent predictive system for joint deployment; (2) regulators may be unable to verify whether a vendor's deployed system matches its audited version; and (3) inconsistent LLM answer sets may lead to unreliable cache reuse. Their experiments report changes in answer sets across independent calibrations for 45% of Llama-3.2-1B contexts and 76% of GPT-2 contexts. Furthermore, Cresswell et al. (ICML 2024) show that prediction sets affect human decision accuracy, while their ICLR 2025 study demonstrates that equalized coverage does not prevent disparate impacts on downstream decisions. These findings highlight the practical consequences of prediction-set inconsistency beyond coverage and set size.
Regarding ACC, BACA only reallocates calibration samples without modifying model training or parameters. Thus, its benefit is more consistent uncertainty information rather than improved classification accuracy.
W2 and Q1
We thank the reviewer for raising this concern. At initialization, we divide the fixed-size memory buffer into two fixed-size, non-overlapping sub-buffers: the CIL Replay Buffer and the CP Calibration Memory. The former retains representative training samples, while the latter stores randomly sampled calibration data independent of the training and test sets.
This separation is necessary because the two buffers serve fundamentally different purposes. The Replay Buffer supports model training and knowledge retention, whereas the Calibration Memory is exclusively used to estimate conformal quantiles and construct prediction sets, without participating in model training. In standard CP, random sampling is required to maintain exchangeability between calibration and test samples.
This design allows both buffers to serve their respective purposes within the fixed total memory budget. However, jointly optimizing their allocation under a shared budget is an interesting direction for future work, beyond the scope of this study. We also evaluate BACA under different calibration memory sizes, demonstrating its effectiveness across varying calibration budgets.
W3
We thank the reviewer for the suggestion. However, our objective differs fundamentally from CIL replay-buffer management. Replay buffers select informative and representative samples to improve model learning, whereas CP calibration memory is intended to reflect the test-data distribution. Therefore, calibration samples must be randomly selected rather than manually prioritized to preserve the exchangeability required by standard CP. Accordingly, BACA does not select individual samples; instead, it determines the number of calibration samples allocated to each class, with random sampling within each class. Therefore, existing CIL replay methods, which focus on selecting informative training samples, are not directly comparable to BACA.
Q2
Reducing the calibration memory may weaken the stability advantage of BACA. Nevertheless, BACA can retain its advantage over equal allocation under reduced calibration budgets, as Stage 3 adaptively distributes the available samples according to class-wise calibration demand and future protection requirements. When the budget becomes extremely small, BACA may approach equal allocation due to limited allocation flexibility, and its advantage may consequently diminish. Our experiments under different calibration memory sizes further evaluate this advantage across varying budgets.
Q4
We thank the reviewer for this question. We consider two settings of fixed per-class caps.
First, assigning the same fixed cap (X) to every class has two cases: (1) without a total budget constraint, the calibration memory grows as new classes arrive, violating our fixed-budget setting; (2) under a fixed total budget, fully utilizing all class caps reduces to equal allocation, while leaving some caps unused leads to budget underutilization.
Second, assigning different fixed caps (Xc) to individual classes requires prior knowledge of their respective calibration demands. However, these demands evolve as new classes arrive, making such predefined caps difficult to determine.
In contrast, BACA fully utilizes the fixed calibration budget through adaptive class-wise allocation, prioritizing classes with greater current calibration demands and future protection needs. This enables BACA to accommodate evolving class-wise requirements without relying on predefined per-class caps or uniform allocation.