- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces GYRO, a continual self-supervised learning method that reduces rotational invariance by anchoring representations to persistent reference directions.
The goal is that different views or augmentations of the same image should be similarly represented in the embedding space, while different images should be spread across the embedding space.
The method is an extension of MMCR, where each task receives a randomly oriented simplex frame that remains fixed after initialization. Every time an image is presented, its centroid is mixed with the nearest prototype of its task’s frame, and the nuclear norm is evaluated on the mixed centroids.
GYRO does so by introducing fixed reference directions, arranged as a regular simplex, for each task. Each image’s representation is mixed with its nearest reference before calculating the existing spectral objective, combining an image mean representation, selected reference, and anchoring strength.
With /beta=0, GYRO represents MMCR exactly.
Evaluated on two natural image benchmarks: Split CIFAR-100 (50 tasks) and Split Tiny ImageNet (100 tasks)
Critical strengths
The paper has a clear methodological contribution. The modifications of MMCR are easy to follow and understand.
Preserving learned geometry is a plausible mechanism for reducing catastrophic forgetting. The paper provides evidence that GYRO achieves greater geometric stability than comparable methods. More specifically, the method outperforms replay, distillation, and regularization baselines.
The reported final accuracies show where the contribution is strongest. As the sequence becomes longer, replay becomes increasingly beneficial, and GYRO achieves the highest final accuracy on both the 20- and 50-task sequences
Critical weaknesses
The authors provide empirical evidence that geometric stability minimizes this drift by anchoring representations to the nearest persistent simplex frame, which is achieved with GYRO. However, the authors do not convincingly establish a causal explanation for the superior performance. What is the reason that a network that flexibly adapts its parameters while maintaining separable object manifolds, performs better.
The authors write: “Hyperparameters were selected on test accuracy for every method, ours included”. This means that the test set has therefore participated in model selection. Reported accuracies can be (and probably are) too optimistic. A convincing evaluation requires choosing configurations on validation data, freezing them, and then reporting final test performance.
Figure 2 uses one run per method for ResNet, the best of three seeds for Tiny ImageNet, with the baseline ER using a different seed. The differences in the results are small. I would have expected more runs and a more statistical analysis in this paper, for the strong claims in the paper.
Questions for authors
-
Can you rerun Tables 1–2 with hyperparameters selected on validation data, comparable tuning budgets, and an untouched test set? Would the results survive independent model selection?
-
What evidence establishes the proposed rotational mechanism? Can you separate common rotation, task-dependent movement, and residual deformation, for example using orthogonal Procrustes alignment,and show which changes predict forgetting?
-
How much of the improvement comes from task identity? Can you compare against a baseline with the same per-image task IDs and test streams where task boundaries are less closely aligned with class membership?
3 comments
Response to Reviewer Me4G (2/2)
Q3. How much of the improvement comes ....
Task identity by itself does not explain the gain. We gave ER and LUMP the same per-image task IDs that GYRO stores (task-balanced replay), and we gave GYRO the wrong IDs. Below, we report Final Class-IL accuracy (%), mean over three seeds, with the change from the same method in brackets (*: 95% interval excludes zero).
| CIFAR-100, 50 tasks | Tiny ImageNet, 100 tasks | |
|---|---|---|
| GYRO | 57.45 | 31.83 |
| GYRO, stored images anchored to a random earlier task's frame | 53.82 (−3.63*) | 25.65 (−6.18*) |
| ER | 53.42 | 19.36 |
| ER with the same task IDs | 54.04 (+0.62) | 19.55 (+0.19) |
| LUMP | 55.15 | 27.90 |
| LUMP with the same task IDs | 54.96 (−0.19) | 27.50 (−0.40) |
The same task IDs do not help ER or LUMP. In GYRO the ID matters only because it selects each stored image's own anchor, and pointing it to another task's frame costs 3.6 and 6.2 points. The ID need not be a task label either: frames opened on a fixed clock, with no knowledge of task boundaries, stay within 1.3 points of true boundaries. The improvement comes from anchoring each task to its own fixed reference, not from the extra task information.
On streams less aligned with classes, we moved 20% of each class's images to the next task (blurry stream, CIFAR-100, 50 tasks), and GYRO keeps the highest mean accuracy (57.56, against 56.03 for LUMP). When every task contains every class there are no task-specific classes to keep apart, and per-task frames bring no gain.
Streams, CIFAR-100, 50 tasks
| Class-incremental | Blurry (20% to next task) | Every task has every class | |
|---|---|---|---|
| GYRO | 57.45 | 57.56 | 58.03 |
| LUMP | 55.15 | 56.03 | 61.24 |
| ER | 53.42 | 54.82 | 59.05 |
| Fine-tune | 49.60 | 51.10 | 57.84 |
Action: These comparisons are now in App. K.5 of the revised paper, which also states that GYRO targets class-incremental streams.
Thank you again for your precise questions. We hope our answers on the causal explanation, the model selection and the statistics behind Fig. 2 were as direct as the questions. If they resolve your concerns, we would appreciate it if you would consider raising your score.
Follow up response to Reviewer Me4G
Thank you again for your precise questions. We have uploaded the revised paper, and it includes the evidence you asked for. The abstract and introduction now explain the gain as fewer confusions between tasks, Fig. 2 uses the same three seeds for every method with error bars, and the interventions, the independent model selection and the task-identity controls are in App. K. Following the discussion with Reviewer JXmq, the abstract, the contributions and the discussion also state the claims more precisely, and these changes are marked in red. We would be glad to know whether this addresses your concerns, and we are happy to answer any further questions in the remaining time.
Response to Reviewer Me4G (1/2)
We thank you for reading our paper closely and for pointing out where our explanation fell short. It was helpful to know that the method itself was clear to you. We address your weaknesses and questions in order below.
We agree that Figure 2 in the submission showed a correlation on single runs, and that a causal explanation calls for further evidence. The main reason GYRO performs better is that it reduces confusion between tasks. In every method we tested, 98 to 99% of the remaining errors assign a test image to a class of another task (50 CIFAR-100 tasks and 100 Tiny ImageNet tasks). MMCR optimises the geometry of the current batch, and it contains no term that ties the representation of an earlier task to a persistent reference across tasks. GYRO adds such a reference by assigning each task a fixed frame in the embedding space. Because the images of a task remain aligned with their own frame, this helps keep the tasks separated from one another. As the table below shows, GYRO has the fewest cross-task errors, while accuracy within the right task is the same for every method. To verify that the frame is the cause, we ran targeted interventions that change one ingredient at a time:
Action: In the revised paper, the abstract and introduction give this explanation in place of "rotate freely", and the error analysis and the interventions are in App. K.1 and K.2.
We ran the Procrustes analysis you suggest and separated the three parts. A common rotation accounts for most of the drop of an old task's probe, with or without the anchor, and it is harmless because re-fitting the probe absorbs it. Non-rigid change is the smaller part. On which change predicts forgetting, how far a task moved relative to the others does not predict its accuracy loss, but how much separation from other tasks it lost does. We kindly invite you to refer to our CQ2 (response to common questions).
Yes, the results survive independent model selection. We kindly request you to refer to our CQ1 (response to common questions)
We agree. Fig. 2 now uses the same three seeds for every method and reports means with error bars. As you point out, the differences in disruption are modest, so we treat the figure as a description of the geometry and base our explanation on other evidence (please refer to our responses to W1 and CQ2). The accuracy differences on these same seeds are clear. GYRO is ahead of every other method with a paired 95% interval above zero, for example by +2.30 points over LUMP on CIFAR-100 (50 tasks) and by +3.93 points on Tiny ImageNet (100 tasks). This lead is also maintained across five random class orders. Fig. 3 now has three seeds in every cell, and every cell is measured directly.
Action: Fig. 2 is replaced by the matched-seed version with error bars in the revised paper, and the paired intervals on these seeds are in App. K.1.