- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper develops a conditional interpretation of forgetting using the reference volume of low risk solutions for prior-task data. A selection hypothesis and label permutation symmetry yield a lower bound on distal risk and on final mixture risk. A separate result bounds exact optimum selection under an additional classwise condition. CIFAR-100 experiments across class disjoint and overlapping tasks compare empirical risk CDFs with the reference and report plug-in risk floors.
Critical strengths
-
The distinction between the existence of a shared solution and whether training selects it is worth studying. Framing forgetting as a selection problem gives the paper a clear conceptual focus and connects its analysis to an important limitation of simple existence arguments.
-
The conditional arguments are developed carefully, and the paper is explicit about the assumptions needed for its risk bounds. I also appreciate the additional experiment that chooses thresholds independently of the risk outcomes, since it provides a useful check on the main diagnostic procedure.
Critical weaknesses
-
The paper is clear that the bound is on final risk, but I still want to see how much performance the model actually loses on old tasks. Please report error on the old examples selected in H.1 just before and after the final task, keeping the mixture weights and evaluation label space fixed. A matched model trained only on that task would help show what the earlier training contributes to the result.
-
The paper already allows different risk thresholds in Definition 1 and the CDF plots, but the factorial bound only covers exact optima. A small model with a known optimum and a clearly defined reference would let us see how the solution volume grows as we allow a little more error. I'm interested in whether models with useful accuracy are still rare in that example.
Questions for authors
-
Does the relation to the same reference change between the pre-final and final checkpoints on H.1's fixed final subset, and how does the endpoint compare with a matched current-task-only baseline?
-
How quickly does reference volume grow as the allowed error increases above the optimum in a small explicit model, and which tolerances still represent useful absolute accuracy?
5 comments
Thank you for your patience while we completed the additional analysis and experiments.
W1-Q1: Changes in past risk and the contribution of prior training.
We thank the reviewer for asking us to connect final risk to the performance actually lost during the final task. We compare the pre-final checkpoint , the sequential endpoint , and a matched current-task-only baseline over 30 seeds per protocol. The baseline uses the same initialization, final-task training set, and final-task training recipe. Across the three models, we hold the evaluation examples, uniform past-task mixture weights, and final evaluation label space fixed. We retain the distal construction from H.1 (I.1 in the revised draft), without reselecting the metric or threshold using these results.
Below, denotes empirical 0-1 risk, and is the normalized distal component of the past mixture relative to the final task. Tables report means over seeds. Risks are percentages and differences are percentage points.
Table 1. Past risk before and after final-task training.
| Statistic | CIL | Blurry1 | Blurry2 | gCIL-TV | gCIL-U | gCIL-L |
|---|---|---|---|---|---|---|
| 62.47 | 61.90 | 60.71 | 80.00 | 80.21 | 77.42 | |
| 100.00 | 99.14 | 97.89 | 93.48 | 91.01 | 90.06 | |
| 62.47 | 58.90 | 54.10 | 77.36 | 80.79 | 75.56 | |
| 100.00 | 99.80 | 99.59 | 99.79 | 99.93 | 99.61 | |
| +37.53 | +37.24 | +37.18 | +13.48 | +10.80 | +12.64 | |
| +37.53 | +40.90 | +45.49 | +22.43 | +19.14 | +24.05 |
Here, and measure the empirical risk increase across the final task transition, from to , on the past mixture and its normalized distal component, respectively. Positive values indicate deterioration.
For CIL, the two comparisons coincide because the entire past mixture is distal. In Blurry and gCIL, final risk is higher on the distal component than on the full past mixture and approaches one. The increase during final-task training is also larger (). In four of these five protocols, the distal component has lower pre-final risk despite its higher final risk. These comparisons show both greater endpoint error and a larger loss of predictive performance on the fixed distal component.
Table 2. Effect of prior training on final risk.
| Statistic | CIL | Blurry1 | Blurry2 | gCIL-TV | gCIL-U | gCIL-L |
|---|---|---|---|---|---|---|
| 100.00 | 99.14 | 97.89 | 93.48 | 91.01 | 90.06 | |
| 100.00 | 99.28 | 98.36 | 94.07 | 91.91 | 91.48 | |
| 17.39 | 17.85 | 17.82 | 62.28 | 53.56 | 53.34 | |
| 18.81 | 20.29 | 20.87 | 64.79 | 57.32 | 58.40 | |
| 0.00 | −0.14 | −0.47 | −0.59 | −0.90 | −1.42 | |
| −1.43 | −2.44 | −3.05 | −2.51 | −3.76 | −5.06 |
$ \Delta_{\mathrm{past}} =\widehat{R}(\theta_N;P_{1:N-1})-\widehat{R}(\theta_{\mathrm{only},N};P_{1:N-1}), $
$ \Delta_{\mathrm{current}} =\widehat{R}(\theta_N;P_N)-\widehat{R}(\theta_{\mathrm{only},N};P_N). $
Negative differences favor the sequential endpoint. Prior training yields modest improvements in final-task performance across the six settings and a small advantage on the past mixture in Blurry and gCIL. These gains coexist with the substantial distal risk increases in Table 1.
Q1: Change relative to the same volume reference.
The eCDF figure in Appendix I.1, Figure 7 compares all three models against the same volume reference on the fixed distal subset. Before final-task training, the models achieve markedly lower risks than this reference in every protocol. After final-task training, this advantage largely disappears: the endpoint CDF comparisons are consistent with the selection ordering in Hypothesis 2, with no rejection by the one-sided KS diagnostic at the 0.05 level. The sequential endpoint and the current-task-only baseline exhibit similar mean risks and empirical risk distributions on this subset, despite the substantially lower risk of the pre-final checkpoint. Together, these comparisons suggest that prior training may offer a modest benefit on the final task, while much of its earlier predictive advantage is lost on distal data.
W2-Q2: Solution volume beyond exact optimality.
Thank you for this suggestion. The following small model allows us to calculate how reference volume grows with the permitted excess risk and to relate each tolerance to absolute accuracy.
Consider equally likely classes, each with a single fixed input vector: for , where is the -th standard basis vector. Use a bias-free linear softmax classifier with prediction , parameter region , and uniform reference , as in our normalized Lebesgue construction. The optimal 0-1 risk is , attained, for example, by .
For input , the logits are column of . Under , each column predicts the correct label with probability , independently across columns. Thus, the number of misclassified inputs satisfies , and . The reference volume allowing excess risk is therefore
In particular, . For , the model has 100 parameters and chance accuracy is %. The exact formula gives:
| Excess risk | Required absolute accuracy (%) | Reference volume |
|---|---|---|
Allowing two mistakes increases the reference volume by a factor of 3,736 relative to exact optimality. Nevertheless, parameters achieving at least % accuracy still occupy only of the reference region. This example shows that, although the solution volume grows substantially with the allowed error, rarity under the reference can persist at useful absolute accuracies beyond exact optimality. We have added this example and its analysis to Appendix K.2 of the revised manuscript.
We hope these additions address your concerns and provide a clearer basis for assessing the contribution of the solution-density perspective. Thank you for the constructive suggestions, which helped us strengthen the empirical evidence and examine solution rarity beyond exact optimality.
Thank you for your thoughtful and detailed review, especially the concrete suggestions on measuring performance loss and examining near-optimal solution volume. We appreciate your recognition of our framing of forgetting as a selection problem, distinguishing the existence of shared solutions from the probability that sequential training selects them. We are planning to provide a detailed response within the next 48 hours altogether with the requested experimental results and a toy example illustrating how reference solution volume grows with the allowed error. Thank you for your patience.