- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BRACE, a framework for multimodal continual instruction tuning that aims to balance knowledge retention, reuse, and parameter efficiency. BRACE consists of three components: (1) FixedAtom estimates historical Atom reuse benefits through loss changes and selects reusable Atoms while freezing historical knowledge; (2) AdaAtom adaptively allocates new Atoms based on historical reuse gains and estimated residual adaptation demand; and (3) Input Diversity Loss reduces directional redundancy among newly introduced Atoms. Experiments on UCIT and MLLM-DCL with LLaVA-1.5 and InternVL demonstrate promising continual learning performance. On LLaVA-1.5, BRACE improves final performance over previous baselines by 6.22 and 8.53 percentage points, respectively, while reducing cumulative adaptation parameters by 38.3% and 29.2% compared with its non-adaptive variant.
Critical strengths
-
Clear motivation: The paper addresses important challenges in multimodal continual learning, including catastrophic forgetting, inefficient historical knowledge reuse, and increasing adaptation costs. The overall motivation is clear.
-
Fine-grained knowledge management: BRACE provides a unified framework for historical Atom selection, knowledge protection, and adaptive capacity allocation. The proposed components are reasonably designed and work together to improve knowledge reuse and parameter efficiency.
-
Promising empirical results: The method achieves strong results on multiple benchmarks and backbones. The ablation studies and storage comparisons provide useful evidence of its effectiveness, particularly regarding the performance-efficiency trade-off.
Critical weaknesses
-
W1. Heuristic design choices: Although BRACE introduces three components for Atom-level knowledge reuse and adaptation, several key design choices remain insufficiently justified. Related works, such as PCLR, MoRAM, and AdaLoRA, have already explored fine-grained knowledge management and importance-based capacity allocation. The authors should better clarify the technical distinctions.
- FixedAtom: The method estimates Atom reuse benefits using first-order approximations of individual loss changes. However, the effectiveness of this importance criterion remains insufficiently justified, especially considering potential interactions and conflicts among Atoms. The authors could refer to prior studies on gradient-based data selection, such as LESS (ICML 2024), FisherSFT (ICML 2025), SPICE (ICLR 2026), and GRAD-MATCH (ICML 2021), which explore various gradient-based importance scoring strategies. Although these methods focus on data selection, their underlying ideas may provide useful insights into Atom importance estimation. An additional ablation comparing the proposed criterion with alternative importance scoring strategies inspired by these studies, as well as relevant prior methods, would help better validate the effectiveness and necessity of the proposed design.
- AdaAtom: Section 3.3 assumes that weaker historical support indicates greater residual adaptation demand. However, lower historical support does not necessarily imply higher marginal gains from additional parameters. This heuristic assumption lacks sufficient theoretical or empirical justification.
- Input Diversity Loss: The proposed loss only considers input-direction similarity among newly introduced Atoms, without explicitly accounting for functional redundancy or gradient conflicts. Given the limited adaptation budget, it would be valuable to investigate whether gradient-aware diversity measures could improve the effective utilization of new Atoms.
-
W2. Computational overhead and scalability: Although BRACE reduces adaptation parameters and storage costs, FixedAtom requires historical Atom scoring and reuse configuration validation. As the historical Atom pool grows, these operations may introduce substantial computational overhead. The paper lacks sufficient analysis of end-to-end training efficiency and scalability with increasing numbers of tasks.
-
W3. Limited interpretability of Atom selection: Although the paper provides visualizations of reuse depth, layer-wise capacity allocation, and qualitative prediction examples, the actual Atom selection mechanism remains difficult to interpret. Given the multimodal setting, it would be helpful to provide more intuitive case studies showing which historical Atoms are selected for particular visual tasks, their importance scores, and how their removal affects model predictions. Such analysis could provide stronger evidence that FixedAtom identifies genuinely useful historical knowledge.
Questions for authors
Q1. How does BRACE fundamentally differ from existing methods such as PCLR, MoRAM, and AdaLoRA, particularly regarding Atom-level selection and adaptive capacity allocation?
Q2. How reliably do the first-order loss-based benefit estimates reflect the actual reuse value of historical Atoms? Could the authors provide additional ablations comparing alternative importance scoring strategies, including gradient-based criteria inspired by prior data selection studies? It would also be interesting to investigate whether accounting for interactions, gradient conflicts, or redundancy among Atoms could improve selection effectiveness.
Q3. What evidence supports the assumption in Section 3.3 that weaker historical support indicates greater residual adaptation demand? Could the authors further investigate whether the proposed residual scores correlate with the actual marginal performance gains from allocating additional Atoms to different layers?
Q4. Why is the Input Diversity Loss restricted to input-direction similarity among newly introduced Atoms? Could considering the complete rank-one updates, as well as potential gradient redundancy or conflicts between historical and new Atoms, lead to more effective capacity utilization?
Q5. Could the authors provide a more comprehensive analysis of the computational overhead introduced by Atom scoring, reuse-depth validation, and adaptive capacity allocation? In particular, how does the end-to-end adaptation cost scale with the number of historical tasks and accumulated Atoms?
Q6. Could the authors provide more intuitive visualizations or qualitative case studies to better illustrate the benefit-aware Atom selection process? For example, showing the importance scores and source tasks of selected historical Atoms, together with their effects on multimodal predictions, would help demonstrate what knowledge is actually being reused.
If the authors can address my concerns and questions, I would be willing to raise my score.
5 comments
Response to Reviewer ZPJX (2/3)
W1.3 & Q3: Historical support and the rationale for capacity allocation
Historical support is an allocation heuristic, not a measured marginal benefit. It directs capacity toward weak-support modules with slightly higher scores at matched budgets. A separate ablation shows that AdaAtom reduces new parameters while maintaining comparable performance. We will clarify the heuristic's role in §3.3.
(1) Historical support directs capacity toward weak-support modules. We compare it with random allocation at matched new-parameter budgets, using the same historical checkpoint, reused Atoms, training settings, and 3,000 hard-routed evaluation samples per task. Weak-support modules are the bottom quartile by pre-training historical support.
| Dataset | Allocation | New parameters (M) | Weak-support capacity share | Task score |
|---|---|---|---|---|
| ArxivQA | Historical support | 8.538 | 57.11% | 95.3000% |
| ArxivQA | Random | 8.538 | 20.41% | 95.2333% |
| VizWiz | Historical support | 20.405 | 36.73% | 62.1661 |
| VizWiz | Random | 20.405 | 26.19% | 62.1435 |
Capacity control supports effective adaptation. The capacity share allocated to weak-support modules increases from 20.41% to 57.11% on ArxivQA and from 26.19% to 36.73% on VizWiz, with slightly higher task scores.
(2) AdaAtom reduces new parameters while maintaining performance. In a separate ArxivQA ablation, we toggle only AdaAtom, retaining FixedAtom and ID with the same parent-model Atoms, historical references, and training settings:
| AdaAtom | Accuracy after task 2 (%) | New parameters (M) | New Atoms |
|---|---|---|---|
| Disabled | 93.43 | 19.99 | 1,792 |
| Enabled | 95.10 | 8.54 | 752 |
AdaAtom reduces new parameters by 57.28% while maintaining comparable performance. Together, the two experiments support the practical value of this heuristic allocation mechanism through layer-wise capacity allocation and overall capacity control.
W1.4 & Q4: Heuristic design of Input Diversity and alternative constraints
The original ID loss heuristically reduces redundancy among new Atoms while allowing them to complement historical knowledge. Additional experiments support your suggestion to regularize complete rank-one updates. However, the additional accuracy benefits of gradient redundancy and conflict constraints are modest and inconsistent across tasks. We will investigate these extensions in future work.
(1) ID prioritizes redundancy within newly added capacity. Historical Atoms are frozen; new Atoms with similar input directions can still complement them through different output effects. We therefore constrain new–new pairs without directly penalizing new–historical alignment. On 64 samples per task from the final UCIT model, the local responses of co-active Atoms, , show higher overlap for new–new pairs across all five tasks:
| Input task | Matched modules | Mean squared response correlation: new–new | Mean squared response correlation: new–historical |
|---|---|---|---|
| ArxivQA | 94 | 0.079 | 0.000056 |
| VizWiz | 224 | 0.049 | 0.00010 |
| IconQA | 77 | 0.098 | 0.00013 |
| CLEVR-Math | 224 | 0.035 | 0.000094 |
| Flickr30k | 79 | 0.049 | 0.00012 |
For nonzero rank-one updates ,
=\cos^2(a_i,a_j)\cos^2(b_i,b_j) \leq\cos^2(a_i,a_j).$$ Input-direction similarity thus bounds the squared cosine similarity of complete updates. ID is also applicable at initialization, when $A$ is random and $B$ is zero. These observations motivate the heuristic. **(2) Regularizing complete updates improves accuracy in controlled experiments.** Each task uses the same parent model, initial capacity, and training settings, with two initialization seeds and 3,000 test examples under hard routing. | Regularization | IconQA accuracy (%) | New parameters (M) | CLEVR-Math accuracy (%) | New parameters (M) | | --- | --- | --- | --- | --- | | Original input-side ID | 69.40 | 12.788 | 72.22 | 34.118 | | Complete rank-one updates | 70.27 | 12.990 | **72.63** | 34.053 | | Complete updates + gradient redundancy | 70.50 | 13.089 | 72.42 | **33.978** | | Complete updates + gradient conflicts | **70.60** | 13.028 | 72.27 | 34.151 | | Complete updates + both constraints | 69.67 | **12.727** | 72.18 | 34.058 | M denotes final added parameters in millions. Complete-update regularization improves mean accuracy by **0.87/0.41 percentage points** on IconQA/CLEVR-Math. On IconQA, individual constraints add **0.23/0.33 percentage points** with slightly more parameters. On CLEVR-Math, neither improves accuracy over complete-update regularization; combining both constraints also provides no consistent accuracy gain. We therefore regard your suggestions as promising extensions and will further study their integration with adaptive capacity allocation in future work.Response to Reviewer ZPJX (3/3)
W2 & Q5: Computational overhead and scalability
BRACE's selection takes only 0.53%–0.91% of training-loop time. Across the six-task sequence, the historical pool grows by 149.67%, while selection time increases by only 9.04% from the first reuse stage to the last.
(1) Scoring and validation add little overhead; allocation takes milliseconds. On UCIT with LLaVA-1.5-7B/NVIDIA A100, we time each of the five reuse stages three times and report medians. Selection takes 83.64–96.93 s per stage, including only 8.38–10.09 ms for allocation:
| Current task | Historical tasks | Cumulative historical Atoms | Candidates actually scored | Scoring and ranking (s) | Reuse-depth validation (s) | Capacity allocation (ms) | Total selection time (s) |
|---|---|---|---|---|---|---|---|
| ArxivQA | 1 | 3,584 | 3,584 | 36.54 | 52.30 | 8.38 | 88.89 |
| VizWiz | 2 | 4,336 | 2,544 | 36.97 | 46.09 | 8.84 | 83.81 |
| IconQA | 3 | 6,108 | 3,805 | 42.99 | 40.64 | 8.78 | 83.64 |
| CLEVR-Math | 4 | 6,724 | 3,421 | 45.97 | 41.87 | 10.09 | 88.31 |
| Flickr30k | 5 | 8,948 | 5,048 | 53.41 | 43.50 | 9.16 | 96.93 |
(2) Selection remains inexpensive as the pool grows in this sequence. Historical tasks increase from 1 to 5; stored Atoms grow from 3,584 to 8,948 (+149.67%), but scored candidates grow from 3,584 to 5,048 (+40.85%). Selection time increases by only 8.04 s (9.04%). Candidates comprise shared Atoms and private Atoms from the most recent task.
(3) Joint scoring and a fixed validation search limit computation. Discovery/stability each use 64 probes; one backward pass per probe scores all candidate gates. A separate image-disjoint set of 64 probes evaluates zero reuse and depths 1/2/4/8. These steps run during adaptation; inference loads the saved configuration.
W3 & Q6: Atom provenance and contributions of historical reuse
FixedAtom reuses components from multiple historical tasks. Removal interventions show their joint contributions to counting, scene details, and action descriptions; aggregate loss changes support their reuse value.
(1) Reuse sources vary with the current task. ImageNet-R's share falls from 100.0% on ArxivQA to 23.9% on Flickr30k, where VizWiz/CLEVR-Math jointly contribute 68.9% of reused Atoms:
| Current task | ImageNet-R | ArxivQA | VizWiz | IconQA | CLEVR-Math | Total reused Atoms |
|---|---|---|---|---|---|---|
| ArxivQA | 100.0% | — | — | — | — | 1,792 |
| VizWiz | 73.1% | 26.9% | — | — | — | 896 |
| IconQA | 51.4% | 5.5% | 43.1% | — | — | 1,792 |
| CLEVR-Math | 61.4% | 5.8% | 28.6% | 4.2% | — | 448 |
| Flickr30k | 23.9% | 6.7% | 26.4% | 0.5% | 42.5% | 1,792 |
(2) Source-group removal changes predictions in concrete examples. Other components and decoding settings are fixed. Scores are pre-adaptation standardized ranking scores, . Source denotes the task that trained the parameters; L is zero-based. Listed Atoms are examples; the intervention removes the indicated source group or single Atom:
| Target task/scenario | Historical source | Example selected Atom | Selection score | Atoms removed | Full-model prediction | Prediction after removal |
|---|---|---|---|---|---|---|
| IconQA/star counting | ArxivQA | L8.mlp.gate_proj, slot1 | 5.087722 | Source group: 99 Atoms | 79; correct | 77; incorrect |
| Flickr30k/painting a brick wall | VizWiz | L31.self_attn.v_proj, slot0 | 3.621361 | Source group: 473 Atoms | A man standing on a ladder painting a brick wall | A man painting a brick wall; the ladder is omitted |
| Flickr30k/gymnastics | CLEVR-Math | L20.self_attn.o_proj, slot7 | 2.897515 | Source group: 762 Atoms | Flipping on a balance beam | Airborne over the uneven bars |
| Flickr30k/hiking | IconQA | L5.self_attn.o_proj, slot7 | 3.537621 | Source group: 9 Atoms | A group of people hiking through a rocky area | A woman standing in a rocky area |
| IconQA/comparing pencils and paper | ImageNet-R | L31.mlp.gate_proj, slot8 | 7.396224 | One Atom | No; correct | No; correct, unchanged |
The first four source-group interventions cause counting errors, omitted details, or altered action/person descriptions. Removing one Atom in the last example leaves the answer correct, showing that the remaining active components suffice for that example.
(3) Aggregate loss increases after source-group removal. Removing the VizWiz source group increases mean reference-answer CE by 0.2940 over 256 Flickr30k samples. The image-resampling 95% CI is [0.2792, 0.3102], entirely above zero, supporting this group's contribution to reducing task loss.
Response to Reviewer ZPJX (1/3)
Thank you for recognizing the motivation behind BRACE, its fine-grained reuse mechanism, and its performance–efficiency trade-off. We address your comments and questions point by point below.
BRACE's key distinction is that it freezes historical Atoms and task catalogs to reduce forgetting caused by overwriting historical knowledge, uses the current multimodal task loss to identify and validate beneficial reuse, and controls and allocates new capacity before training to reduce redundant learning and parameter growth. We apologize that the lack of clarity in the original manuscript led to your misunderstanding. The rank-one representation provides the basis for fine-grained management in FixedAtom and AdaAtom; we do not claim this representation itself as a primary contribution.
The evaluation scope also differs. The original MoRAM paper evaluates continual recognition with CLIP and continual learning on text tasks, while AdaLoRA primarily evaluates text fine-tuning. Our paper further evaluates multimodal continual instruction tuning, including visual question answering, visual reasoning, and caption generation. PCLR already addresses this setting; its main differences from BRACE lie in the protection, reuse, and capacity management mechanisms summarized above.
(1) First-order scores accurately rank actual conditional removal benefits.
On IconQA/CLEVR-Math, we evaluate 24 Atoms using 96 held-out images for each task. The Spearman correlations between the original BF16 first-order scores and actual removal benefits evaluated in FP32 using the same weights are 0.9965/1.0000. Higher scores correspond to larger loss increases after removal, providing direct evidence for benefit-aware selection.
In Table 3 of the paper, with the same parent model, candidate pool, and reuse budget, Top-k achieves higher held-out benefits than Random-k/Bottom-k at all five stages. The Top-k/Random-k benefits are 0.072/−0.040 on VizWiz and 0.021/−0.026 on CLEVR-Math, indicating that benefit ranking identifies configurations that reduce the task loss.
(2) Full-pipeline comparisons support the practical value of the original criterion. Following your suggestion, we adapt four alternative strategies for Atom selection and compare them over the six-task continual adaptation sequence. Compared with FisherSFT-Atom, BRACE's MFT is only 0.0232 points lower, while using 23.50% fewer parameters and reducing training job time by 9.84%. Compared with the other three strategies, BRACE achieves higher MFT with 8.73%–25.10% fewer parameters. This shows that BRACE achieves competitive adaptation performance with fewer parameters and lower time costs throughout continual adaptation:
FisherSFT-Atom's set-level complementarity objective achieves slightly higher MFT but uses more parameters. SPICE-Atom introduces a penalty for conflicts in probe effects, yet achieves lower MFT and uses more parameters than BRACE.