- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work tries to assert that sequential learning in class incremental learning scenarios can leave the task blocks on incompatible scales resulting in a cross task score problem. The paper claims to study this problem and goes on to combine progressive Equiangular Tight Frame (ETF) and feature preservation with classifier standardisation before score fusion and additive task bias fitting afterwards. The pre fusion standardisation tries to control classifier contribution relative to ETF evidence. While post fusion task offsets adjust the cross task competition along with preserving within task rankings. The paper establishes their claims through experiments across many contemporary baseline works and also does a detailed ablation study.
Critical strengths
- The gap between Class Incremental Learning and Task Incremental learning accuracy, i.e the scores that are not comparable across tasks is a recognised bottleneck. Targeting this explicitly is a worthwhile contribution even though the relevant diagnostics is missing.
- The authors have compared across broad set of baselines spanning replay, contrastive, neural collapse methods, etc. This wide range of comparisons allows to get a better picture of the proposed approach.
- The standardisation, ETF based alignment and post fusion task offset etc are light weight components introduced for CIL framework.
- The combination of training time ETF alignment and post training corrections is a good design to target the cross task score problem.
Critical weaknesses
- Eventhough the cross task score problem is asserted as the motivation in the paper, proper diagnostics or results supporting this is not provided. This would provide with better insight into the problem in hand.
- The paper shows improvement over CIL and TIL across multiple baselines, but this alone is not a sufficient metric to show that the cross task score problem is mitigated. Evidences like error decomposition in Class Incremental Learning, that is within task error and cross task error would indicate how much improvement has been achieved. Standard calibration improvement measures like ECE etc would also help strengthen the calibration claims in the paper.
- The main table (Table 1) shows many values in bold especially under CIFAR-10 which are not the best numbers but are still shown in bold. (SeqCIFAR10 shows 66.86 (24.83) in bold against CTSC eventhough the best number corresponds to LODE 68.01 (24.63), other similar discrepancies also exist in the Table).
- Proposed CTSC shares the ETF alignment framework which is a significant part of the approach with ProNC. This suggests that the main contributions of this work is how this alignment approach is combined with post fusion task offset and classifier standardisation. Given this a direct plugin of the suggested novel component over the ProNC approach would have been more insightful. The ablation results in Figure 4 further strengthen this as the major influence on the accuracy seem to be coming from the ETF and Feature alignment part.
- Related to the above observation, the accuracy metric used in the paper as the major evidence sits very close to the accuracy numbers of the ProNC framework. Given this standard deviation across multiple runs would help us understand how much real improvement has been achieved with respect to the ProNC framework.
- The method uses ETF progressively by appending the new directions to the existing basis. This in turn would change the old anchors. The paper claims to use feature alignment to align the anchors. But this raises the natural question that aligning with a previous checkpoint would only result in the features aligning with the old anchors corresponding to the old ETF. How does the change in the ETF upon addition of new directions to the basis is accounted for while aligning the anchors.
Questions for authors
- Can authors provide diagnostics tat substantiate the cross-tasks core problem and its mitigation? For example decompose CIL errors into within task and cross task errors together with a task confusion matrix for the proposed CTSC and a representative baseline.
- As the paper also claims calibration improvement, can the authors provide standard calibration measures such as ECE for CTSC and main baselines?
- Please correct the Table 1 bold entries to match correctly with the actual results in the table.
- Can the authors also apply the proposed components as a plug in on top of ProNC under identical settings? Further can they also compare the task offset step against established bias correction methods such as BiC and WA? This would help us to isolate the contribution beyond the gains from ETF and feature alignment components.
- Please report the standard deviation over multiple seeds for CTSC and ProNC, so that the significance of the improvement over ProNC can be properly assessed.
- When new directions are amended to the basis, the old anchors change. How are the old class features realigned to the updated anchor, given that the alignment to previous checkpoint keeps features consistent with old ETF? Can th cosine similarity between the old class features to their respective current anchors be reported to properly judge the suitability of the alignment approach used.
I thank the authors for their targeted and detailed response. Most of my concerns have been addressed, and I believe this warrants an upgrade of my score.
I have a few minor follow up questions with regard to the section "6. Realignment when the ETF expands".