- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the stability–plasticity trade-off in continual learning for multimodal large language models. The authors propose ASAP, which employs MANOVA-based gradient effect decomposition to select shared, audio-specific, video-specific, and interaction parameter blocks. These blocks are selectively activated according to the available modalities, while gradient-based importance estimation is used to protect parameters critical to previously learned tasks. Experiments on MUSIC-AVQA and CMU-MOSI, covering six task orders and different backbone architectures, demonstrate improvements in new-task learning, final accuracy, and knowledge retention.
Critical strengths
- Well-motivated and coherent framework. The paper integrates modality-aware parameter selection, shared adaptation, and importance-based protection to balance stability and plasticity under a fixed adaptation budget. The three components are logically connected and form a coherent continual learning framework.
Critical weaknesses
- The necessity and distinctive contribution of MANOVA-based parameter selection are not sufficiently established. MANOVA-guided block selection is a central technical contribution of this paper. Although the random-selection ablation demonstrates its effectiveness, this comparison alone does not establish the necessity of MANOVA-based modality decomposition over conventional gradient-driven parameter selection. Existing methods such as SMT and SPU also employ gradient information for selective parameter updates. However, the paper lacks a controlled comparison with simpler gradient-importance-based or similar sparse block-selection strategies under the same parameter budget and sharing/protection mechanisms. Furthermore, the proposed modality-specific block assignments are derived from gradient differences across input conditions, but more direct evidence is needed to demonstrate their advantages for the corresponding modality-specific tasks. Consequently, the unique contribution of MANOVA decomposition remains insufficiently isolated.
- The task setting and applicability of the protection mechanism are relatively limited. The framework associates task identities with three predefined modality configurations (A, V, and AV) and uses these configurations to determine parameter activation and protection. Although six training orders are evaluated, the experiments remain restricted to these three task types. The paper does not investigate scenarios where different learning objectives share the same modality configuration. In such cases, distinct tasks may share the same activated parameter blocks and protection records, raising questions about whether their previously acquired knowledge can be adequately preserved. Additionally, the experiments involve only five training stages, leaving the long-term behavior of selective protection insufficiently explored. As more tasks are learned, accumulated protection constraints may reduce the trainable capacity available for subsequent adaptation. The current evaluation therefore provides limited evidence of the framework's effectiveness in more complex and extended continual learning scenarios.
Questions for authors
- Could the authors provide a controlled comparison between MANOVA-based selection and conventional gradient-importance-based or SMT-style block selection, while maintaining the same parameter budget, sharing mechanism, and protection strategy? Such an experiment would help clarify the necessity and distinctive advantages of MANOVA decomposition.
- How would ASAP distinguish and protect knowledge from different tasks that share the same modality configuration but have different learning objectives? Additionally, in longer continual learning sequences, could the accumulated protection constraints progressively reduce the number of trainable blocks available for subsequent tasks?
6 comments
Response to Reviewer WS9A (2/3)
[W2 / Q2] Objectives with the same modality configuration, and protection in longer sequences
Thank you for these two questions, which identified a case the submission did not test and led us to extend ASAP and to measure how protection grows. Both depend on one design choice, how ASAP keeps its importance records, so we answer them together with one new experiment.
The two record rules. As submitted, ASAP identifies a task by its modality combination and keeps one importance record per task (ASAP-M). Two objectives with the same modality configuration then share a record, and because Equation 8 (page 6) excludes the current task's own record, the earlier objective is unprotected while the later one trains, which we believe is the scenario you describe. When objective identity is known during training, each objective can instead keep its own record, with only the current objective's record excluded (ASAP-O). Nothing else changes. The two objectives still use the same active blocks, selection and the budget are the same, and inference needs no task or objective identifier.
The experiment. We tested both rules and PCLR, the second-best method on MUSIC-AVQA by Final.ACC in Table 1 (page 7), on a new stream of nine objectives built from the MUSIC-AVQA question types, two audio, two visual and five audio–visual. For example, audio counting at stage is followed by audio comparison at stage , which has the same modality configuration.
Result 1, retention. One record per objective protects the earlier objective. In Table R9, when a same-modality successor is learned, the earlier objective drops by about 0.2 points under ASAP-O, against 1.6 to 3.6 points under ASAP-M and a similar drop under PCLR. The whole stream benefits as well. In Table R5, ASAP-O has the highest Final.ACC and halves Avg.FOR (0.93 against 1.89 for ASAP-M and 2.03 for PCLR), while its New.ACC stays close to ASAP-M's and above PCLR's.
The next part reports the second result, on trainable capacity, and the trade-off between the two rules.
Table R9. Retention of each modality's first objective when a successor with the same modality is learned (Appendix D.9, Table 20, page 40).
| Objective | Method | After own stage | Before successor | After successor | Forgetting |
|---|---|---|---|---|---|
| A-Counting (, successor ) | ASAP-O | 66.568 | 65.782 | 65.585 | 0.197 |
| ASAP-M | 66.568 | 65.782 | 62.144 | 3.638 | |
| PCLR | 65.290 | 63.717 | 60.275 | 3.441 | |
| V-Counting (, successor ) | ASAP-O | 77.945 | 77.694 | 77.527 | 0.167 |
| ASAP-M | 77.945 | 78.195 | 76.608 | 1.587 | |
| PCLR | 76.274 | 77.611 | 75.439 | 2.172 | |
| AV-Existential (, successor ) | ASAP-O | 81.781 | 81.781 | 81.579 | 0.202 |
| ASAP-M | 81.579 | 81.579 | 79.858 | 1.721 | |
| PCLR | 80.668 | 80.668 | 79.555 | 1.113 |
Table R5. Nine-objective stream. to give the mean accuracy over the objectives learned so far, with as Final.ACC (Appendix D.9, Table 21, page 40).
| Method | (A) | (V) | (A) | (V) | (AV) | (AV) | (AV) | (AV) | (AV), Final.ACC | Avg.FOR | New.ACC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| PCLR | 65.290 | 69.995 | 66.950 | 69.335 | 71.908 | 71.345 | 70.013 | 66.751 | 67.101 | 2.028 | 68.739 |
| ASAP-M | 66.568 | 71.863 | 68.104 | 70.050 | 72.520 | 72.355 | 70.907 | 69.899 | 68.900 | 1.894 | 70.281 |
| ASAP-O | 66.568 | 71.863 | 69.196 | 71.302 | 72.543 | 72.906 | 71.377 | 70.020 | 69.516 | 0.928 | 70.179 |
Response to Reviewer WS9A (3/3)
[W2 / Q2, continued] Trainable capacity and the trade-off between the two record rules
Result 2, capacity. Table R10 shows that the frozen fraction of active blocks depends on the record rule.
With one record per task (ASAP-M), it is bounded. By Equation 8 (page 6), the blocks protected while a task trains are those of its active set that lie in other tasks' importance records, and a task's record is replaced, not accumulated, when the task recurs. The frozen fraction is therefore bounded by the union of the other tasks' records, however many stages follow. In the nine-stage stream it stays at 14.6% from the fifth stage on, because these stages all train the audio–visual task and the records in force no longer change. Over the thirty stages of the paper's six five-stage orders, where tasks recur, it is between about 5% and 23% at later stages (Appendix D.8, Figure 6, page 38).
With one record per objective (ASAP-O), each new objective adds a record, so the frozen fraction grows with the number of distinct objectives, as you anticipate. It grows slowly, because each record is the smallest set of blocks that covers the fraction of the objective's importance and the records overlap. The increments shrink from the sixth objective to the ninth, and the frozen fraction reaches about 45% after nine objectives, so more than half of the active blocks remain trainable. ASAP-O's new-objective accuracy stays within half a point of ASAP-M and above PCLR at every stage. A smaller protects fewer blocks for the same record, which is a direct way to slow this growth (Appendix D.8, page 38).
The trade-off. The two rules therefore trade retention against capacity. ASAP-M keeps the frozen fraction bounded but does not separate objectives that share a modality, whereas ASAP-O protects them at the cost of a frozen fraction that grows with the number of objectives and of objective identity during training. On the benchmarks of the paper each task has one objective, so the two rules coincide there and Table 1 is unchanged.
In the revised manuscript, Section 3.4 (page 6) now states that Equation 8 can use objective-specific records, Appendix D.8 (page 38) contains the record bound, the stage-wise frozen fractions and the diagnostic, Section 5.4 (page 10) states the bound, and the Limitations paragraph (page 10) notes the requirement of objective identity and reports the frozen fraction at nine objectives.
Table R10. Frozen fraction of active blocks and new-objective accuracy at each stage of the nine-objective stream (Appendix D.9, Table 21, page 40).
| Stage | Frozen (%), ASAP-M | Frozen (%), ASAP-O | New-obj. acc. (%), ASAP-M | New-obj. acc. (%), ASAP-O | New-obj. acc. (%), PCLR |
|---|---|---|---|---|---|
| A-Count | 0.00 | 0.00 | 66.568 | 66.568 | 65.290 |
| V-Count | 8.39 | 8.39 | 77.945 | 77.945 | 76.274 |
| A-Compare | 7.60 | 23.60 | 63.973 | 64.310 | 62.963 |
| V-Loc | 8.21 | 25.09 | 78.204 | 77.714 | 76.163 |
| AV-Exist | 14.60 | 24.80 | 81.579 | 81.781 | 80.668 |
| AV-Count | 14.60 | 32.40 | 72.490 | 72.411 | 70.593 |
| AV-Loc | 14.60 | 37.60 | 62.500 | 62.065 | 60.435 |
| AV-Compare | 14.60 | 41.20 | 63.942 | 63.851 | 62.398 |
| AV-Temporal | 14.60 | 44.80 | 65.328 | 64.964 | 63.869 |
We hope our responses have answered all your concerns. Please let us know if any point needs further detail, and we would be glad to discuss it.
Response to Reviewer WS9A (1/3)
We sincerely thank Reviewer WS9A for the careful review and for the constructive questions. Table R1 below reports the controlled comparison you ask for, Table R8 below adds the direct per-task evidence you request, and Tables R9 and R10 in the second and third parts of this reply answer your two questions on the task setting.
[W1a / Q1] Controlled comparison with gradient-importance or SMT-style selection
Thank you for this question, which helped us isolate what the MANOVA-guided selection contributes. To show its necessity and contribution, we ran the controlled comparison you describe as a new experiment, ranking blocks by the mean absolute gradient (MAG, SMT's statistic) inside ASAP with the same parameter budget, sharing mechanism and protection strategy. Table R1 shows that MAG is below ASAP on both datasets.
To separate the two steps of ASAP's score, we also tested the mean squared gradient norm (MSG) and the unstandardized factorial effect (UFE), which uses the factorial contrasts without the standardization.
Together, these experiments show two things. First, gradient magnitude does not explain the gains. Replacing ASAP's standardized effect score with MAG lowers Final.ACC and New.ACC by about 1.5 points on both datasets, and replacing it with MSG lowers them by about 1.4 points. Second, both steps of ASAP's score contribute. Removing only the standardization (UFE) lowers accuracy by about 0.8 points, and UFE in turn exceeds MAG by about 0.7 points. Avg.FOR is close for all selection scores, since they use the same protection mechanism.
MANOVA-guided selection is therefore necessary, and its contribution is clear from two standard statistical principles. The factorial contrast compares a block's gradients with and without a modality, which identifies the block's modality role. Standardization judges each effect relative to its variation across samples, which favors blocks whose effect is consistent. The General response gives the details, and its Table R2 gives the overlap between the selections.
Table R1. Selection scores with everything else in ASAP fixed (CMU-MOSI averaged over orders P1, P3 and P5, MUSIC-AVQA on order P3).
[W1b] Direct evidence for the modality-specific block assignments
Thank you for asking for more direct evidence, which led us to verify the modality roles at the level of individual tasks and block sets. Table R8 below gives three pieces of evidence.
Evidence 1, per-task gains. In Table R8a, ASAP improves the audio and video tasks by about 2 points each relative to MAG, against less than 1 point for the audio–visual task. This is the pattern expected if the roles are meaningful. By Equation 5 (page 5), the roles decide which blocks a unimodal task uses, whereas the audio–visual task activates all four block sets whatever their roles.
Evidence 2, removing a block set. In Table R8b, removing the learned updates of the audio or the video block set lowers its own task, by about 3 points for audio and 8 points for video. Each unimodal task therefore relies on the set assigned to its modality, and not only on the shared set.
Evidence 3, swapping two block sets. Swapping the audio and video updates lowers the video task about as much as removing its updates, and the audio task by most of that loss. One set's updates therefore largely cannot stand in for the other's, so each set holds knowledge specific to its modality.
Together with the results in our response to W1a, we have shown the unique contribution of the MANOVA decomposition and the advantages of its modality-specific assignments.
Both comparisons are in the revised manuscript, in Section 5.3, Table 3 (page 9) and Appendix D.6, Tables 14 and 15 (page 35). The next part turns to your questions on the task setting.
Table R8. Per-task accuracy and block interventions on MUSIC-AVQA order P3.
(a) Selection comparison, final accuracy (%)
(b) Interventions on the ASAP checkpoint, change in percentage points