- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper solves the problem of Group Robust Continual Learning (GRCL) by providing theoretical guarantee regarding the buffer size allocation targeted for solving worst group optimization. The paper first provides error bounds for GRCL comparing with normal replay based error minimization for CL and shows how normal replay based error minimization does not guarantee group robust error minimization. Then the paper provides the upper-bound for group robust error minimization in terms of adversarial group weight, replay buffer size and uncertainty term. The authors then provide realistic surrogate that motivates their buffer allocation strategies and show at what conditions the proposed method is more reliable followed by methods to predict group id when one is not available. Empirical analysis proves the improvement provided by proposed method in worst group accuracy for both Waterbirds and Celeb-A datasets.
Critical strengths
The theoretical contribution regarding Group Robust Continual Learning is novel for useful direction in the field of Continual Learning. This paper definitely adds value in this direction. The loss bound that depends on adversarial group weight and replay size is novel and useful. The proposed allocation methods and prediction of group-id is interesting and valuable. Empirical results show improvement in worst group performance of proposed methods compared to existing baselines.
Critical weaknesses
- Major weakness is limited empirical evaluations. Please include more datasets, and recent replay-based baselines. Authors are also encouraged to experiment with powerful backbones (like pre-trained ViT) that may already solve the bias in the studied datasets to show the usefulness of proposed method in such settings.
- The overall performance of proposed method is not the best for Celeb-A dataset and the proposed "Difficulty-adaptive GRCL" provides very modest improvement against "Group-balanced GRCL" especially for Waterbird dataset.
- Lemma 1 is very interesting which shows that when grows, even when the error term decreases the gap between adversarial weight and buffer proportion may not shrink. However, there is always possibility of change in as the model changes which is not discussed sufficiently.
- The proposed method only controls the size of replay buffer across groups but does not control what samples should be selected. The sample composition can also control the group difficulty, not only size.
- Clarity issues:
- Typo: classicalbaselines (line 27), classicalrehearsal (line 86)
- Second paragraph of introduction is too vague.
- Please add citations for line 43 and 45.
- Please define task groups properly before using (line 44).
- In Line 50, the sentence starts with "In the smart-home customer-support example ..." but such example is not introduced before. Please improve the clarity.
- Lines 54 to 58, please add proper citations. Please define "minority groups" properly before using. What is the difference between task-group imbalance and class imbalance? It is not clear until the method section which makes it confusing while reading the introduction.
- In line 63, the phrase starting with "largely overlooking..." is too vague/unclear. What is "data features"?
- In line 67, what is "group structure"?
- In line 80, what does it mean by "potentially adversarial manner"?
- Line 98-101, what does "target allocation" and "target group allocation" mean?
- In section 4.2, how is MMD computed between two datasets, do we use raw input pixels or hidden representation features?
Questions for authors
Please refer to the weakness section above.
5 comments
Response to Reviewer xkTc (2/3)
Response to W3
We have added a discussion after Lemma 1 clarifying that, even when the model is well trained, the mismatch term in the first line need not vanish. Although the adversarial weights can change as the model evolves, their normalization, , does not ensure alignment with the effective group proportions . Model improvements alone therefore do not guarantee that this mismatch disappears. Furthermore, Lemma 3 in Appendix A rigorously establishes a worst-case example in which the excess risk remains bounded away from zero, further demonstrating that the issue when and the effective group proportions mismatch.
Response to W4
Our approach to controlling group sizes in the replay buffer is directly motivated by our theoretical analysis, and the proposed methods come with rigorous theoretical guarantees in Sections 3 and 4. In contrast, the sample-selection methods discussed here are primarily heuristic and lack comparable theoretical guarantees.
These sample-selection strategies could also be incorporated into our framework, although doing so would introduce substantial additional complexity. As the first theoretical study of group-robust learning in continual learning, our work provides theoretically grounded methods with simple implementations that already achieve significant improvements in worst-group accuracy over the benchmarks.
Response to clarity issues
Thank you for these detailed suggestions. We have revised the Introduction to clarify the terminology, distinguish task-group imbalance from class imbalance, and make the motivating examples more concrete. We have also added references and clarified the meanings of adversarial task ordering and target replay allocation. The specific revisions are described below.
We have revised the two typos.
We have added citations to Kim et al. (2020) and Wang et al. (2021) to support the discussion of imbalanced continual learning and differences in learning difficulty, on page 1, lines 45–46 of the revised manuscript.
We have introduced an explicit definition of task groups at their first discussion in the Introduction: “a task group is a collection of task arrivals whose data are sampled from the same underlying distribution” (page 1, lines 45–46). This definition is consistent with the distribution-based formulation in Section 2.1.
The smart-home customer-support scenario is now introduced in the second paragraph in more details (pages 1–2, lines 52–56). Specifically, a smart-home customer-support agent may continually learn from requests ranging from routine device setup to troubleshooting safety-critical equipment, such as smoke detectors. Frequent, straightforward requests dominate both training and replay, leaving rare and ambiguous fault reports underrepresented. The agent may consequently associate familiar phrases such as “the detector keeps beeping” with routine battery issues and fail to diagnose a recurring smoke-detector malfunction, potentially compromising user safety.
We have added citations to Hashimoto et al. (2018) and Lahoti et al. (2020) in the fairness discussion on page 2, lines 58–59 of the revised manuscript. We also clarify how average performance can mask poor minority-group performance and explicitly define minority groups in lines 59–61. We have also moved the distinction between class imbalance and task-group imbalance into the Introduction (page 2, lines 65–74). Class imbalance concerns unequal frequencies of class labels , whereas task groups can also be induced by data feature attributes in , which serves as input in neural networks, allowing subpopulations within the same class to exhibit different arrival frequencies, learning difficulties, or forgetting behavior.
We have replaced the vague reference to “data features” with an explicit distinction between class labels and feature attributes in . The revised text explains that feature attributes can induce subpopulations with different learning difficulties or forgetting behavior, and that relevant groups may therefore depend on both features and labels , rather than class labels alone (page 2, lines 68–73). For example, images with the same bird label can belong to different groups depending on whether the background is land or water.
We have replaced the "largely overlooking" to "without explicitly accounting for" in line 68–69, page 2.
We have replaced the ambiguous phrase “group structure” with the explicitly defined term “task groups.” The revised sentence now refers directly to whether replay heuristics account for task groups and protect rare or difficult groups, rather than invoking an undefined notion of structure (page 2, lines 75–77).
Response to Reviewer xkTc (3/3)
We have clarified that “potentially adversarial” refers to task arrival orders that may pose challenges for continual learning. Specifically, we now give the example of sequences containing prolonged absences of certain task groups, which make knowledge of those groups difficult to retain (page 2, lines 90–91). This makes the intended challenge concrete: the task sequence need not provide regular opportunities to revisit every group.
We have explicitly defined the target group allocation as the desired number of replay samples assigned to each group by the proposed replay buffer allocation strategies (page 3, lines 110–112). We also clarify that this planned allocation may not be immediately achievable because buffer updates are restricted to the currently available data. Allocation-Track Reservoir Sampling is then introduced to make the realized buffer allocation track this target (page 3, lines 113–114).
MMD can be computed on either raw inputs or extracted feature representations by defining the kernel on the corresponding space (Gretton et al., 2012; Liu et al., 2020). The empirical formula for comparing two datasets is provided in Eq. (26) of Appendix C.2. In our Waterbirds experiments, we compute MMD on feature representations extracted by a frozen, ImageNet-pretrained ResNet-50. Specifically, the 2048-dimensional features are normalized and mapped to 64 dimensions using a fixed random projection. MMD is then computed using an RBF kernel in this projected space, with the bandwidth selected by the median heuristic on the incoming batch. Further implementation details are provided in Appendix D.1.
Response to Reviewer xkTc (1/3)
Thank you for your careful review and thoughtful feedback. We address your comments point by point below and have revised the manuscript accordingly.
Response to W1
Following your suggestion, we have added experiments using ViT and included an additional experiment on a new dataset, UrbanCars.
We have added an experiment using a ViT backbone on the Waterbirds dataset, with the results reported in Figure 11 on page 39. Figure 11 shows that strong pretrained representations do not eliminate the performance gap between majority and minority groups during continual learning. At M=160, group-balanced and difficulty-adaptive GRCL improve accuracy on the challenging waterbird-on-land group from 66.26% with task-balanced ERM to 86.07% and 86.57%, respectively. These results demonstrate that our group-aware replay and robust model updates provide substantial worst-group improvements even with strong pretrained ViT features.
We have further conducted experiments using a ViT backbone on the UrbanCars (Li et al., 2023) dataset, comparing our methods with the baselines DER++ (Buzzega et al., 2020), MIR (Aljundi et al., 2019), and LPR (Yoo et al., 2024) in Appendix D.8. In UrbanCars, the task is to classify the car type as urban or country, while the background and a co-occurring object provide two potentially misleading shortcuts. Each attribute is binary, yielding eight groups defined by the combination of car type, background, and co-occurring object. The background and co-occurring object serve as potential shortcuts: models may use these contextual cues to predict the car type rather than relying on the car's own appearance.
Figure 12 on page 40 and Table 5 on page 41 of the revised manuscript show that substantial group disparities persist on UrbanCars even with a pretrained ViT-B/16 backbone. Task-balanced and class-balanced ERM achieve high accuracy on groups where both shortcuts align with the car label, but perform poorly when both the background and co-occurring object conflict with it. DER++, MIR, and LPR-adapted also exhibit low worst-group accuracy, ranging from 7.04% to 13.28% at M=160. Group-aware replay substantially improves minority-group performance: group-balanced ERM and group-balanced GRCL achieve worst-group accuracies of 23.68% and 36.16%, respectively. Difficulty-adaptive GRCL attains the highest mean worst-group accuracy at both budgets, reaching 34.08% at M=110 and 44.80% at M=160. At the larger budget, this exceeds group-balanced GRCL by 8.64 percentage points, although the paired confidence interval includes zero across five seeds. These results support the usefulness of group-aware replay and robust updates with frozen ViT representations, while indicating that substantial challenges remain for groups affected by multiple conflicting shortcuts.
Response to W2
In the original CelebA experiments, the modest improvement of difficulty-adaptive GRCL over group-balanced GRCL can be attributed in part to their similar actual replay-buffer allocations. The update rule in Algorithm 2 limits how closely the buffer can track changes in the target allocation. Since hard groups arrive infrequently, there are insufficient opportunities to adjust their actual storage to the target, leading to similar actual allocations under the two methods. We therefore introduce Algorithm 3, which updates the replay buffer to track the target allocation more closely.
We conduct new experiments on CelebA using Algorithm 3 and report the results in Table 6 and Figures 14 and 15 in Appendix D.9. Table 6 shows that difficulty-adaptive replay substantially outperforms uniform replay on both seeds 80 and 81. For seeds 80 and 81, it improves the final worst-group accuracy from 20.73% to 49.39% and from 18.75% to 29.88%, respectively, while reducing the GRCL loss from 2.0781 to 1.0208 and from 2.6938 to 1.5586.
The allocation trajectories in Figures 14 and 15 show that difficulty-adaptive GRCL produces final storage allocations that differ from those of group-balanced GRCL, helping explain these improvements. Difficulty-adaptive replay allocates more storage over the training trajectory to challenging groups, including the blond female groups with and without eyeglasses. For seed 80, cumulative storage, measured as the sum of end-of-task buffer counts, increases by 64.6% for the group without eyeglasses and 39.2% for the group with eyeglasses relative to uniform replay. Their final accuracies increase from 47.90% to 58.13% and from 76.09% to 80.43%, respectively. The same pattern holds for seed 81: both groups receive more cumulative storage, and their final accuracies increase from 58.55% to 66.35% and from 84.78% to 89.13%. These results are consistent with the intended mechanism: adapting the replay allocation provides greater representation for challenging groups, helping mitigate forgetting and improve group-robust performance.