acceptodds
Under review as a conference paper at ICLR 2027

Theory for group-robust continual learning

Abstract

To support the continuous adaptation of agentic AI systems as new task data arrives, continual learning (CL) has become increasingly important. However, rehearsal-based methods, a dominant class of approaches in CL, typically treat all past tasks equally to mitigate catastrophic forgetting. In this paper, we prove that this practice can destabilize learning by causing poor retention of knowledge from rare or difficult task groups in terms of large excess risk. Accordingly, we develop group-robust continual learning (GRCL), a new framework for achieving stable and robust performance across task groups, with a particular focus on improving the worst-group performance. We design two rehearsal-based GRCL methods guided by excess risk bounds: group-balanced replay allocates memory uniformly across groups to minimize a simplified bound, while difficulty-adaptive replay incorporates group learning difficulty estimated during CL into its replay buffer allocation and minimizes a tighter, difficulty-aware bound. We further extend GRCL to the setting where group information is unknown throughout CL. In this case, we propose two group identification methods based on maximum mean discrepancy and online expectation-maximization, respectively, and establish theoretical guarantees for their identification performance. We also derive a small excess-risk bound for GRCL using the estimated group identities. Experiments on real-world datasets show that our rehearsal-based GRCL methods significantly improve robustness over classicalbaselines by enhancing worst-group performance.

Primary area:
learning theory
Keywords:
Continual learning, group-robust learning, fairness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xkTcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper solves the problem of Group Robust Continual Learning (GRCL) by providing theoretical guarantee regarding the buffer size allocation targeted for solving worst group optimization. The paper first provides error bounds for GRCL comparing with normal replay based error minimization for CL and shows how normal replay based error minimization does not guarantee group robust error minimization. Then the paper provides the upper-bound for group robust error minimization in terms of adversarial group weight, replay buffer size and uncertainty term. The authors then provide realistic surrogate that motivates their buffer allocation strategies and show at what conditions the proposed method is more reliable followed by methods to predict group id when one is not available. Empirical analysis proves the improvement provided by proposed method in worst group accuracy for both Waterbirds and Celeb-A datasets.

Critical strengths

The theoretical contribution regarding Group Robust Continual Learning is novel for useful direction in the field of Continual Learning. This paper definitely adds value in this direction. The loss bound that depends on adversarial group weight and replay size is novel and useful. The proposed allocation methods and prediction of group-id is interesting and valuable. Empirical results show improvement in worst group performance of proposed methods compared to existing baselines.

Critical weaknesses
  • Major weakness is limited empirical evaluations. Please include more datasets, and recent replay-based baselines. Authors are also encouraged to experiment with powerful backbones (like pre-trained ViT) that may already solve the bias in the studied datasets to show the usefulness of proposed method in such settings.
  • The overall performance of proposed method is not the best for Celeb-A dataset and the proposed "Difficulty-adaptive GRCL" provides very modest improvement against "Group-balanced GRCL" especially for Waterbird dataset.
  • Lemma 1 is very interesting which shows that when mm grows, even when the error term decreases the gap between adversarial weight and buffer proportion may not shrink. However, there is always possibility of change in qq as the model changes which is not discussed sufficiently.
  • The proposed method only controls the size of replay buffer across groups but does not control what samples should be selected. The sample composition can also control the group difficulty, not only size.
  • Clarity issues:
  • Typo: classicalbaselines (line 27), classicalrehearsal (line 86)
  • Second paragraph of introduction is too vague.
    • Please add citations for line 43 and 45.
    • Please define task groups properly before using (line 44).
    • In Line 50, the sentence starts with "In the smart-home customer-support example ..." but such example is not introduced before. Please improve the clarity.
  • Lines 54 to 58, please add proper citations. Please define "minority groups" properly before using. What is the difference between task-group imbalance and class imbalance? It is not clear until the method section which makes it confusing while reading the introduction.
  • In line 63, the phrase starting with "largely overlooking..." is too vague/unclear. What is "data features"?
  • In line 67, what is "group structure"?
  • In line 80, what does it mean by "potentially adversarial manner"?
  • Line 98-101, what does "target allocation" and "target group allocation" mean?
  • In section 4.2, how is MMD computed between two datasets, do we use raw input pixels or hidden representation features?
Questions for authors

Please refer to the weakness section above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer xkTc (1/3)

Thank you for your careful review and thoughtful feedback. We address your comments point by point below and have revised the manuscript accordingly.

Response to W1

Following your suggestion, we have added experiments using ViT and included an additional experiment on a new dataset, UrbanCars.

We have added an experiment using a ViT backbone on the Waterbirds dataset, with the results reported in Figure 11 on page 39. Figure 11 shows that strong pretrained representations do not eliminate the performance gap between majority and minority groups during continual learning. At M=160M=160, group-balanced and difficulty-adaptive GRCL improve accuracy on the challenging waterbird-on-land group from 66.26%66.26\% with task-balanced ERM to 86.07%86.07\% and 86.57%86.57\%, respectively. These results demonstrate that our group-aware replay and robust model updates provide substantial worst-group improvements even with strong pretrained ViT features.

We have further conducted experiments using a ViT backbone on the UrbanCars (Li et al., 2023) dataset, comparing our methods with the baselines DER++ (Buzzega et al., 2020), MIR (Aljundi et al., 2019), and LPR (Yoo et al., 2024) in Appendix D.8. In UrbanCars, the task is to classify the car type as urban or country, while the background and a co-occurring object provide two potentially misleading shortcuts. Each attribute is binary, yielding eight groups defined by the combination of car type, background, and co-occurring object. The background and co-occurring object serve as potential shortcuts: models may use these contextual cues to predict the car type rather than relying on the car's own appearance.

Figure 12 on page 40 and Table 5 on page 41 of the revised manuscript show that substantial group disparities persist on UrbanCars even with a pretrained ViT-B/16 backbone. Task-balanced and class-balanced ERM achieve high accuracy on groups where both shortcuts align with the car label, but perform poorly when both the background and co-occurring object conflict with it. DER++, MIR, and LPR-adapted also exhibit low worst-group accuracy, ranging from 7.04%7.04\% to 13.28%13.28\% at M=160M=160. Group-aware replay substantially improves minority-group performance: group-balanced ERM and group-balanced GRCL achieve worst-group accuracies of 23.68%23.68\% and 36.16%36.16\%, respectively. Difficulty-adaptive GRCL attains the highest mean worst-group accuracy at both budgets, reaching 34.08%34.08\% at M=110M=110 and 44.80%44.80\% at M=160M=160. At the larger budget, this exceeds group-balanced GRCL by 8.648.64 percentage points, although the paired confidence interval includes zero across five seeds. These results support the usefulness of group-aware replay and robust updates with frozen ViT representations, while indicating that substantial challenges remain for groups affected by multiple conflicting shortcuts.

Response to W2

In the original CelebA experiments, the modest improvement of difficulty-adaptive GRCL over group-balanced GRCL can be attributed in part to their similar actual replay-buffer allocations. The update rule in Algorithm 2 limits how closely the buffer can track changes in the target allocation. Since hard groups arrive infrequently, there are insufficient opportunities to adjust their actual storage to the target, leading to similar actual allocations under the two methods. We therefore introduce Algorithm 3, which updates the replay buffer to track the target allocation more closely.

We conduct new experiments on CelebA using Algorithm 3 and report the results in Table 6 and Figures 14 and 15 in Appendix D.9. Table 6 shows that difficulty-adaptive replay substantially outperforms uniform replay on both seeds 80 and 81. For seeds 80 and 81, it improves the final worst-group accuracy from 20.73% to 49.39% and from 18.75% to 29.88%, respectively, while reducing the GRCL loss from 2.0781 to 1.0208 and from 2.6938 to 1.5586.

The allocation trajectories in Figures 14 and 15 show that difficulty-adaptive GRCL produces final storage allocations that differ from those of group-balanced GRCL, helping explain these improvements. Difficulty-adaptive replay allocates more storage over the training trajectory to challenging groups, including the blond female groups with and without eyeglasses. For seed 80, cumulative storage, measured as the sum of end-of-task buffer counts, increases by 64.6% for the group without eyeglasses and 39.2% for the group with eyeglasses relative to uniform replay. Their final accuracies increase from 47.90% to 58.13% and from 76.09% to 80.43%, respectively. The same pattern holds for seed 81: both groups receive more cumulative storage, and their final accuracies increase from 58.55% to 66.35% and from 84.78% to 89.13%. These results are consistent with the intended mechanism: adapting the replay allocation provides greater representation for challenging groups, helping mitigate forgetting and improve group-robust performance.

AuthorsOpenReview ↗

Response to Reviewer xkTc (2/3)

Response to W3

We have added a discussion after Lemma 1 clarifying that, even when the model is well trained, the mismatch term in the first line need not vanish. Although the adversarial weights qt(wt)q_t(w_t) can change as the model evolves, their normalization, ∑k∈[Kt]qtk(wt)=1\sum_{k\in[K_t]}q_t^k(w_t)=1, does not ensure alignment with the effective group proportions ptk=(mtk+nt1({k=It}))/(M+nt)p_t^k=(m_t^k+n_t\mathbf{1}(\{k=I_t\}))/(M+n_t). Model improvements alone therefore do not guarantee that this mismatch disappears. Furthermore, Lemma 3 in Appendix A rigorously establishes a worst-case example in which the excess risk remains bounded away from zero, further demonstrating that the issue when qq and the effective group proportions mismatch.

Response to W4

Our approach to controlling group sizes in the replay buffer is directly motivated by our theoretical analysis, and the proposed methods come with rigorous theoretical guarantees in Sections 3 and 4. In contrast, the sample-selection methods discussed here are primarily heuristic and lack comparable theoretical guarantees.

These sample-selection strategies could also be incorporated into our framework, although doing so would introduce substantial additional complexity. As the first theoretical study of group-robust learning in continual learning, our work provides theoretically grounded methods with simple implementations that already achieve significant improvements in worst-group accuracy over the benchmarks.

Response to clarity issues

Thank you for these detailed suggestions. We have revised the Introduction to clarify the terminology, distinguish task-group imbalance from class imbalance, and make the motivating examples more concrete. We have also added references and clarified the meanings of adversarial task ordering and target replay allocation. The specific revisions are described below.

We have revised the two typos.

We have added citations to Kim et al. (2020) and Wang et al. (2021) to support the discussion of imbalanced continual learning and differences in learning difficulty, on page 1, lines 45–46 of the revised manuscript.

We have introduced an explicit definition of task groups at their first discussion in the Introduction: “a task group is a collection of task arrivals whose data are sampled from the same underlying distribution” (page 1, lines 45–46). This definition is consistent with the distribution-based formulation in Section 2.1.

The smart-home customer-support scenario is now introduced in the second paragraph in more details (pages 1–2, lines 52–56). Specifically, a smart-home customer-support agent may continually learn from requests ranging from routine device setup to troubleshooting safety-critical equipment, such as smoke detectors. Frequent, straightforward requests dominate both training and replay, leaving rare and ambiguous fault reports underrepresented. The agent may consequently associate familiar phrases such as “the detector keeps beeping” with routine battery issues and fail to diagnose a recurring smoke-detector malfunction, potentially compromising user safety.

We have added citations to Hashimoto et al. (2018) and Lahoti et al. (2020) in the fairness discussion on page 2, lines 58–59 of the revised manuscript. We also clarify how average performance can mask poor minority-group performance and explicitly define minority groups in lines 59–61. We have also moved the distinction between class imbalance and task-group imbalance into the Introduction (page 2, lines 65–74). Class imbalance concerns unequal frequencies of class labels yy, whereas task groups can also be induced by data feature attributes in xx, which serves as input in neural networks, allowing subpopulations within the same class to exhibit different arrival frequencies, learning difficulties, or forgetting behavior.

We have replaced the vague reference to “data features” with an explicit distinction between class labels yy and feature attributes in xx. The revised text explains that feature attributes can induce subpopulations with different learning difficulties or forgetting behavior, and that relevant groups may therefore depend on both features and labels (x,y)(x,y), rather than class labels alone (page 2, lines 68–73). For example, images with the same bird label can belong to different groups depending on whether the background is land or water.

We have replaced the "largely overlooking" to "without explicitly accounting for" in line 68–69, page 2.

We have replaced the ambiguous phrase “group structure” with the explicitly defined term “task groups.” The revised sentence now refers directly to whether replay heuristics account for task groups and protect rare or difficult groups, rather than invoking an undefined notion of structure (page 2, lines 75–77).

AuthorsOpenReview ↗

Response to Reviewer xkTc (3/3)

We have clarified that “potentially adversarial” refers to task arrival orders that may pose challenges for continual learning. Specifically, we now give the example of sequences containing prolonged absences of certain task groups, which make knowledge of those groups difficult to retain (page 2, lines 90–91). This makes the intended challenge concrete: the task sequence need not provide regular opportunities to revisit every group.

We have explicitly defined the target group allocation as the desired number of replay samples assigned to each group by the proposed replay buffer allocation strategies (page 3, lines 110–112). We also clarify that this planned allocation may not be immediately achievable because buffer updates are restricted to the currently available data. Allocation-Track Reservoir Sampling is then introduced to make the realized buffer allocation track this target (page 3, lines 113–114).

MMD can be computed on either raw inputs or extracted feature representations by defining the kernel on the corresponding space (Gretton et al., 2012; Liu et al., 2020). The empirical formula for comparing two datasets is provided in Eq. (26) of Appendix C.2. In our Waterbirds experiments, we compute MMD on feature representations extracted by a frozen, ImageNet-pretrained ResNet-50. Specifically, the 2048-dimensional features are normalized and mapped to 64 dimensions using a fixed random projection. MMD is then computed using an RBF kernel in this projected space, with the bandwidth selected by the median heuristic on the incoming batch. Further implementation details are provided in Appendix D.1.

Reviewer_5xFpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies group robustness in rehearsal-based continual learning, motivated by the observation that standard replay strategies can disproportionately forget rare or difficult groups despite maintaining strong average performance. The authors formulate Group-Robust Continual Learning using a regularized group-DRO objective and theoretically characterize the excess risk of conventional rehearsal under group imbalance. Based on the derived bounds, they propose group-balanced replay and difficulty-adaptive replay, where the latter allocates more memory to groups estimated to be difficult. The paper further considers the setting where group identities are unavailable and introduces Buffer-MMD and EM-MMD for online group identification, together with theoretical identification and excess-risk guarantees under stated assumptions. The proposed framework is evaluated on a synthetic benchmark and two group-robust image classification datasets, Waterbirds and CelebA. The experiments report improved worst-group performance over standard rehearsal and several CL baselines.

Critical strengths
  1. The paper studies an important and relatively underexplored issue in continual learning: catastrophic forgetting can be highly heterogeneous across latent groups, and strong average performance can hide severe degradation on rare or difficult groups. The distinction between class imbalance and group-level difficulty/imbalance is meaningful and well motivated.

  2. The theoretical development is substantial. Rather than proposing replay heuristics alone, the paper derives excess-risk bounds for classical rehearsal, uses the resulting bounds to motivate group-balanced and difficulty-adaptive replay. The treatment of unknown group identities through Buffer-MMD and EM-MMD further broadens the scope of the framework.

Critical weaknesses
  1. My main concern is the gap between the theoretical development and the algorithms actually evaluated. In particular, the difficulty-adaptive replay strategy is theoretically motivated by minimizing the surrogate in Eq. (12), whereas the experiments use considerably simpler heuristic approximations based on the estimated adversarial weights. The paper does not quantify how well these approximations solve the theoretically derived allocation problem. Indeed, on Waterbirds, the appendix notes that the group-balanced and difficulty-adaptive allocations are nearly identical, making it difficult to assess the empirical benefit of the difficulty-aware theory itself.

  2. A second concern is that several theoretical guarantees rely on assumptions whose relevance to the experimental settings is unclear. Proposition 2 relies on a local quadratic-growth condition and a covariance assumption for the estimated adversarial weights, neither of which is empirically examined. Similarly, the EM-MMD analysis assumes a fixed number of groups, correct parametric specification, and local consistency of online EM, while the practical Waterbirds implementation uses projected deep features, diagonal Gaussian models, and an additional new-group activation mechanism.

Questions for authors
  1. The objective in Eq. (4) is a KL-regularized group-DRO objective rather than the exact worst-group loss when λ>0\lambda>0. Could the authors clarify in what precise sense minimizing this objective guarantees improved worst-group performance? It would be useful to provide a sensitivity analysis with respect to λ\lambda, including its effect on both the GRCL objective and actual worst-group accuracy.

  2. Could the authors provide controlled experiments varying the MMD separation between groups? This seems particularly important for empirically validating Proposition 4, whose guarantee depends explicitly on group separation and replay/sample sizes.

  3. Could the authors provide a more comprehensive and quantitative ablation study to disentangle the contributions of the GRCL update, group-balanced replay, and difficulty-adaptive replay? In particular, please quantify their differences across varying group imbalance, group difficulty, and memory budgets, with multiple seeds and statistical significance.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5xFp (1/3)

Thank you for your careful review and thoughtful feedback. We address your comments point by point below and have revised the manuscript accordingly.

Response to W1

We would like to clarify that the implemented allocation is not heuristic. It follows directly from the leading-order allocation problem underlying Eq. (12). Specifically, the dominant allocation-dependent term is ∑kq~tk/mtk\sum_k \tilde q_t^k/\sqrt{m_t^k}. Under a fixed replay budget ∑kmtk=M\sum_k m_t^k=M, its continuous optimum is

mtk∗=M(q~tk)2/3∑j(q~tj)2/3.m_t^{k^*}=M\frac{(\tilde q_t^k)^{2/3}}{\sum_j(\tilde q_t^j)^{2/3}}.

Thus, allocating replay according to the estimated adversarial weights has a precise optimization basis. The remaining allocation-dependent terms in Eq. (12) decay faster as the replay counts grow, so retaining the dominant term yields a principled leading-order approximation to the theoretically derived allocation problem.

Moreover, Eq. (12) is an upper bound that incorporates uncertainty in estimating the adversarial weights. If the true weights E[qt]\mathbb{E}[q_t] were available, the leading term E[∑kqtk/mtk]\mathbb{E}[\sum_k q_t^k/\sqrt{m_t^k}] in the allocation objective of Proposition 1 would directly yield the same 2/32/3-power rule. Using q~t\tilde q_t as a plug-in estimate in Proposition 1 and solving the resulting allocation problem in place of Eq. (12) therefore still approximately optimizes the GRCL objective.

The similar realized allocations can be attributed to the limited number of minority-group samples and the infrequent opportunities to replenish them in the Waterbirds stream. These constraints make it difficult for the actual buffer composition to fully track the targets prescribed by group-balanced and difficulty-adaptive replay. Nevertheless, difficulty-adaptive replay retains slightly more samples than group-balanced replay from the challenging waterbird-on-land group at the final round, which is already sufficient to improve worst-group performance. At M=160M=160, the average counts under difficulty-adaptive and group-balanced replay are 40.38540.385 and 40.00040.000, respectively, with Buffer-MMD, and 36.66036.660 and 36.24836.248, respectively, with EM-MMD. These differences are accompanied by improvements in mean worst-group accuracy: from 7676% to 7777% under Buffer-MMD and from 7575% to 7676% under EM-MMD. Our additional synthetic-data ablations further show that the worst-group benefits of difficulty-adaptive replay become more pronounced as the number of task arrivals increases.

Response to W2

While the quadratic-growth condition may not hold in practice, we can still approximately minimize the GRCL upper bound in Proposition 1. The quantity q~\tilde q defined in Proposition 2 serves as an estimator of the expected qq in Proposition 1. Thus, even without the quadratic-growth condition required by Proposition 2, we can substitute q~\tilde q for the expected qq in Eq. (8) and use the resulting expression in the objective in Eq. (12) to solve for mtkm_t^k, thereby approximately minimizing the GRCL upper bound.

We clarify that the EM-MMD analysis is not restricted to raw inputs and can be applied in a fixed feature space. In Waterbirds, both the pretrained feature extractor and the random projection are frozen throughout learning. Thus, using projected deep features is compatible with the theoretical framework, provided that the modeling, separation, and estimation-consistency assumptions hold in that feature space. The diagonal Gaussian model is a computationally efficient choice within the parametric framework. The additional new-group activation rule, borrowed from Buffer-MMD, initializes at most four latent groups and enables practical operation without supplying their identities, following our distribution-based identification design. Our Waterbirds experiments therefore demonstrate the effectiveness of this practical implementation, while the theoretical result characterizes its behavior under the stated assumptions.

AuthorsOpenReview ↗

Response to Reviewer 5xFp (2/3)

Response to Q1

Although Eq. (4) does not exactly minimize the worst-group loss for λ>0\lambda>0, it approximates worst-group loss minimization while balancing worst-group and average performance. Moreover, for every finite λ>0\lambda>0, it assigns strictly greater weight to a worst-performing group than the average group loss does, provided the group losses are not all equal.

Specifically, with a uniform reference distribution, the objective satisfies

=\lambda\log\left(\frac{1}{K}\sum_{g=1}^{K}e^{L_g(w)/\lambda}\right), \qquad F_\lambda(w)\leq\max_g L_g(w)\leq F_\lambda(w)+\lambda\log K.$$ Consequently, an $\varepsilon$-optimal solution $w_\lambda$ satisfies $$\max_g L_g(w_\lambda) \leq \inf_w\max_g L_g(w)+\lambda\log K+\varepsilon.$$ Moreover, for every finite $\lambda>0$, the objective assigns greater weight to a worst-performing group than the average group loss does, provided the group losses are not all equal. For any fixed $w$ and $\lambda>0$, the inner maximization in Eq. (4) admits the closed-form solution $$q_g^\star=\frac{e^{L_g(w)/\lambda}}{\sum_{j=1}^{K}e^{L_j(w)/\lambda}}.$$ For a worst-performing group with the largest loss $L_g(w)$, $q_g^\star$ is strictly greater than the uniform weight $1/K$, provided the group losses are not all equal. The parameter $\lambda$ controls the tradeoff between average and worst-group loss. We aim to improve worst-group performance without excessively sacrificing average performance. As $\lambda$ increases from $0$ to $\infty$, the GRCL objective transitions from the exact worst-group loss to the average group loss. Choosing $\lambda$ therefore allows us to balance these two objectives. In continual learning, limited replay samples can also lead to overfitting and inaccurate estimates of population group losses. Since $q_g^\star\propto e^{L_g/\lambda}$, small $\lambda$ makes the weights highly sensitive to these estimation errors. Increasing $\lambda$ smooths the weights and can stabilize updates and replay allocations, although it does not eliminate estimation bias. We report a sensitivity analysis in Appendix D.6, Figure 9, using 40 paired seeds. For difficulty-adaptive GRCL, increasing $\lambda$ from $0.03$ to $0.1$ improves worst-group accuracy from $77.97\%$ to $79.43\%$ and mean accuracy from $82.87\%$ to $84.36\%$. This is consistent with reduced sensitivity to loss fluctuations, although we do not directly verify this mechanism. Beyond $\lambda=0.1$, the average–worst-group tradeoff becomes apparent: at $\lambda=0.5$, mean accuracy increases to $85.27\%$, while worst-group accuracy decreases to $71.84\%$. Near $\lambda=0.1$, difficulty-adaptive GRCL also achieves lower GRCL loss than group-balanced GRCL, consistent with its higher worst-group accuracy. The overall decrease in GRCL loss across $\lambda$ additionally reflects the changing regularized objective. The nearly identical realized allocations can be attributed to the limited number of minority-group samples and the infrequent opportunities to replenish them in the Waterbirds stream. These constraints make it difficult for the actual buffer composition to fully track the targets prescribed by group-balanced and difficulty-adaptive replay. Nevertheless, difficulty-adaptive replay retains slightly more samples from the challenging waterbird-on-land group at the final round. At $M=160$, the average counts are $40.385$ versus $40.000$ under Buffer-MMD and $36.660$ versus $36.248$ under EM-MMD. These differences are accompanied by modest improvements in mean worst-group accuracy: from $76.28\%$ to $76.55\%$ under Buffer-MMD and from $75.06\%$ to $75.61\%$ under EM-MMD. Our additional synthetic-data ablations further show that the worst-group benefits of difficulty-adaptive replay become more pronounced as the number of task arrivals increases. **Response to Q2** Following your suggestion, we have added experiments varying the MMD separation to evaluate Buffer-MMD’s group identification accuracy and the resulting worst-group accuracy. These experiments appear in Appendix D.6 (page 37), with results reported in Table 4. Buffer-MMD achieves near-perfect group identification and high worst-group accuracy when the MMD separation exceeds 0.2, demonstrating its effectiveness in exploiting distributional differences between groups. Below 0.2, identification becomes more challenging, which reduces worst-group accuracy.
AuthorsOpenReview ↗

Response to Reviewer 5xFp (3/3)

Response to Q3

Following your suggestion, we have added ablation experiments across varying levels of group imbalance, group difficulty, and memory budgets to disentangle the contributions of the GRCL update, group-balanced replay, and difficulty-adaptive replay. The results are presented in Appendix D.6, Figure 10, on page 38.

Contribution of the GRCL update. The first row compares GRCL with ERM while retaining difficulty-adaptive replay. GRCL improves mean worst-group accuracy in 33 of the 36 settings, with statistically significant improvements in 16 settings. Its contribution is particularly pronounced when the hard group is rare and memory is limited. For example, under arrival ratio 16:4:116:4:1, s=0.25s=0.25, and M=90M=90, the GRCL update improves worst-group accuracy by 17.8917.89 percentage points. At M=360M=360 under the same imbalance and difficulty, the improvement remains 4.504.50 percentage points. These results show that allocating replay memory adaptively does not by itself replace a group-robust update: explicitly emphasizing high-loss groups provides an additional benefit.

Contribution of group-balanced replay. The second row compares group-balanced with task-balanced replay under the same GRCL update. The benefit increases markedly with arrival imbalance. Under balanced arrivals, the differences are generally small, because task-balanced replay already gives similar expected representation to each group. Under the 16:4:116:4:1 ratio, group-balanced replay yields improvements between 13.4513.45 and 23.5923.59 percentage points across the tested signal scales and memory budgets. The largest improvement occurs at s=0.25s=0.25 and M=360M=360. This pattern is consistent with group-balanced replay protecting rare groups from underrepresentation, particularly when the rare group is also difficult.

Contribution of difficulty-adaptive replay. The third row compares difficulty-adaptive with group-balanced replay under the same GRCL update. Difficulty-adaptive replay improves mean worst-group accuracy in 32 settings, with significant improvements in nine. Its advantage is strongest when the difficult group is rare and the replay budget is constrained. For arrival ratio 16:4:116:4:1 and s=0.25s=0.25, the improvements are 11.5111.51, 7.297.29, 4.634.63, and 1.551.55 percentage points as total memory increases from 90 to 180, 360, and 720. This decreasing advantage is consistent with adaptive allocation being most valuable when memory is scarce: equal quotas can leave the difficult group undersupplied, whereas larger budgets reduce the need to redistribute memory. Under balanced arrivals, the differences are generally small, showing that adaptive allocation is most useful when learning difficulty and data availability jointly create unequal replay needs.

These comparisons identify distinct contributions: group-balanced replay addresses unequal group representation, difficulty-adaptive replay further adjusts memory to unequal learning needs, and the GRCL update emphasizes poorly performing groups during optimization. At the reference setting (16:4:1,s=0.25,M=360)(16:4:1,s=0.25,M=360), all three comparisons show significant improvements, of 23.5923.59, 4.634.63, and 4.504.50 percentage points, respectively.

Reviewer_e6KyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies group-robust continual learning with limited replay memory. It develops allocation-dependent excess-risk bounds, proposes group-balanced and difficulty-adaptive replay, and introduces MMD-based methods for identifying unknown groups.

Critical strengths
  • The lower bound and allocation-dependent analysis provide a useful theoretical motivation for balancing replay across groups.
  • Group-balanced replay achieves substantial improvements on Waterbirds: worst-group accuracy rises from 59.24% with Group-ERM to 76.28%.
Critical weaknesses
  • The benefit of adaptive replay is not established. Appendix D.4 explicitly states that the two proposed strategies use the same storage allocation on CelebA and attributes their performance difference to single-run randomness. This substantially weakens the claimed adaptive advantage. Moreover, the experiments use heuristic allocations without establishing their relationship to the optimization in Eq. (12).
  • Proposition 1 fixes replay counts, whereas the algorithm changes allocations based on observed data. Appendix C.4 introduces a buffer discrepancy term, but establishes its convergence to zero only when target allocations eventually become fixed and are attained. This condition is not established for the proposed adaptive policy.
Questions for authors
  • How closely do the implemented allocation heuristics approximate Eq. (12)?
  • What guarantee applies when adaptive allocations continue changing?
  • Can you repeat CelebA experiments with matched seeds and report variability, while verifying that the two strategies actually produce different allocations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer e6Ky

Thank you for your careful review and thoughtful feedback. We address your comments point by point below and have revised the manuscript accordingly.

Response to W1 & Q3 on CelebA

In the original experiments on CelebA, group-balanced GRCL and difficulty-adaptive GRCL yield similar actual allocations because the replay-buffer update in Algorithm 2 limits how closely the buffer can follow changes in the target allocation. Because the hard groups arrive infrequently, the continual learning process provides insufficient opportunities to adjust their actual storage to the target allocation, resulting in similar actual replay-buffer allocations for difficulty-adaptive and group-balanced GRCL. We therefore introduce Algorithm 3, which updates the replay buffer to track the target allocation more closely.

We conduct new experiments on CelebA using Algorithm 3 and report the results in Table 6 and Figures 14 and 15 in Appendix D.9. Given the computational cost of the CelebA experiments and our limited resources, we report additional results for two seeds. Table 6 shows that difficulty-adaptive replay substantially outperforms uniform replay on both seeds 80 and 81. For seeds 80 and 81, it improves the final worst-group accuracy from 20.73% to 49.39% and from 18.75% to 29.88%, respectively, while reducing the GRCL loss from 2.0781 to 1.0208 and from 2.6938 to 1.5586.

The allocation trajectories in Figures 14 and 15 show that difficulty-adaptive GRCL produces final storage allocations that differ from those of group-balanced GRCL, helping explain these improvements. Difficulty-adaptive replay allocates more storage over the training trajectory to challenging groups, including the blond female groups with and without eyeglasses. For seed 80, cumulative storage, measured as the sum of end-of-task buffer counts, increases by 64.6% for the group without eyeglasses and 39.2% for the group with eyeglasses relative to uniform replay. Their final accuracies increase from 47.90% to 58.13% and from 76.09% to 80.43%, respectively. The same pattern holds for seed 81: both groups receive more cumulative storage, and their final accuracies increase from 58.55% to 66.35% and from 84.78% to 89.13%. These results are consistent with the intended mechanism: adapting the replay allocation provides greater representation for challenging groups, helping mitigate forgetting and improve group-robust performance.

Response to W1 & Q1 on Eq. (12)

We would like to clarify that the implemented allocation is not heuristic. It follows directly from the leading-order allocation problem underlying Eq. (12). Specifically, the dominant allocation-dependent term is ∑kq~tk/mtk\sum_k \tilde q_t^k/\sqrt{m_t^k}. Under a fixed replay budget ∑kmtk=M\sum_k m_t^k=M, its continuous optimum is

mtk∗=M(q~tk)2/3∑j(q~tj)2/3.m_t^{k^*}=M\frac{(\tilde q_t^k)^{2/3}}{\sum_j(\tilde q_t^j)^{2/3}}.

Thus, allocating replay according to the estimated adversarial weights has a precise optimization basis. The remaining allocation-dependent terms in Eq. (12) decay faster as the replay counts grow, so retaining the dominant term yields a principled leading-order approximation to the theoretically derived allocation problem.

Moreover, Eq. (12) is an upper bound that incorporates uncertainty in estimating the adversarial weights. If the true weights E[qt]\mathbb{E}[q_t] were available, the leading term E[∑kqtk/mtk]\mathbb{E}[\sum_k q_t^k/\sqrt{m_t^k}] in the allocation objective of Proposition 1 would directly yield the same 2/32/3-power rule. Using q~t\tilde q_t as a plug-in estimate in Proposition 1 and solving the resulting allocation problem in place of Eq. (12) therefore still approximately optimizes the GRCL objective.

Response to W2 & Q2 on Eq. (12)

Because EM-MMD requires stronger assumptions, including stationary, i.i.d. group arrivals, we present it as a complementary approach to the Buffer-MMD group-identification method described in Section 4.2 and Appendix C.3. Buffer-MMD does not require stationary group arrivals, and its group-identification guarantees remain applicable when this assumption fails, when adaptive allocations continue changing.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer e6Ky,

As the discussion period is drawing to a close, we would be grateful for any further feedback you may have to help us improve our paper. We sincerely thank you for your time, thoughtful feedback, and careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.