acceptodds
Under review as a conference paper at ICLR 2027

BRACE: Benefit-aware Reuse and Adaptive Capacity Expansion for Multimodal Continual Instruction Tuning

Abstract

Multimodal continual instruction tuning requires models to learn new tasks while retaining existing knowledge. Existing methods primarily rely on parameter-level protection or module-level management, facing two limitations: 1) Insufficient historical knowledge protection and reuse, as preserving parameters or modules alone struggles to ensure both stable knowledge retention and accurate selection of beneficial components, potentially causing forgetting, negative transfer, and redundant learning; 2) Inefficient capacity allocation and utilization, as residual demand after reuse and component complementarity are insufficiently considered, leading to imbalanced capacity allocation and directional redundancy. To address these limitations, we propose BRACE, which organizes rank-one LoRA components into independently manageable Atoms. The FixedAtom component freezes historical Atoms and task catalogs and enables selective reuse through benefit-aware selection. The AdaAtom component adjusts the number and layer-wise distribution of new Atoms according to residual demand after reuse under a global capacity constraint. An input diversity loss further reduces directional redundancy among new Atoms within each layer. We evaluate BRACE on UCIT and MLLM-DCL using two multimodal backbones. On LLaVA-1.5, BRACE improves mean final performance over the best prior methods by 6.22 and 8.53 percentage points on the two benchmarks, respectively. Compared with BRACE-NA, a variant without AdaAtom or input diversity loss, BRACE reduces cumulative adaptation parameters by 38.3% and 29.2%, respectively, with comparable performance.

Primary area:
learning theory
Keywords:
Multimodal Large Language Models, Continual Instruction Tuning, Selective Knowledge Reuse, Adaptive Capacity Allocation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZPJXOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRACE, a framework for multimodal continual instruction tuning that aims to balance knowledge retention, reuse, and parameter efficiency. BRACE consists of three components: (1) FixedAtom estimates historical Atom reuse benefits through loss changes and selects reusable Atoms while freezing historical knowledge; (2) AdaAtom adaptively allocates new Atoms based on historical reuse gains and estimated residual adaptation demand; and (3) Input Diversity Loss reduces directional redundancy among newly introduced Atoms. Experiments on UCIT and MLLM-DCL with LLaVA-1.5 and InternVL demonstrate promising continual learning performance. On LLaVA-1.5, BRACE improves final performance over previous baselines by 6.22 and 8.53 percentage points, respectively, while reducing cumulative adaptation parameters by 38.3% and 29.2% compared with its non-adaptive variant.

Critical strengths
  • Clear motivation: The paper addresses important challenges in multimodal continual learning, including catastrophic forgetting, inefficient historical knowledge reuse, and increasing adaptation costs. The overall motivation is clear.

  • Fine-grained knowledge management: BRACE provides a unified framework for historical Atom selection, knowledge protection, and adaptive capacity allocation. The proposed components are reasonably designed and work together to improve knowledge reuse and parameter efficiency.

  • Promising empirical results: The method achieves strong results on multiple benchmarks and backbones. The ablation studies and storage comparisons provide useful evidence of its effectiveness, particularly regarding the performance-efficiency trade-off.

Critical weaknesses
  • W1. Heuristic design choices: Although BRACE introduces three components for Atom-level knowledge reuse and adaptation, several key design choices remain insufficiently justified. Related works, such as PCLR, MoRAM, and AdaLoRA, have already explored fine-grained knowledge management and importance-based capacity allocation. The authors should better clarify the technical distinctions.

    • FixedAtom: The method estimates Atom reuse benefits using first-order approximations of individual loss changes. However, the effectiveness of this importance criterion remains insufficiently justified, especially considering potential interactions and conflicts among Atoms. The authors could refer to prior studies on gradient-based data selection, such as LESS (ICML 2024), FisherSFT (ICML 2025), SPICE (ICLR 2026), and GRAD-MATCH (ICML 2021), which explore various gradient-based importance scoring strategies. Although these methods focus on data selection, their underlying ideas may provide useful insights into Atom importance estimation. An additional ablation comparing the proposed criterion with alternative importance scoring strategies inspired by these studies, as well as relevant prior methods, would help better validate the effectiveness and necessity of the proposed design.
    • AdaAtom: Section 3.3 assumes that weaker historical support indicates greater residual adaptation demand. However, lower historical support does not necessarily imply higher marginal gains from additional parameters. This heuristic assumption lacks sufficient theoretical or empirical justification.
    • Input Diversity Loss: The proposed loss only considers input-direction similarity among newly introduced Atoms, without explicitly accounting for functional redundancy or gradient conflicts. Given the limited adaptation budget, it would be valuable to investigate whether gradient-aware diversity measures could improve the effective utilization of new Atoms.
  • W2. Computational overhead and scalability: Although BRACE reduces adaptation parameters and storage costs, FixedAtom requires historical Atom scoring and reuse configuration validation. As the historical Atom pool grows, these operations may introduce substantial computational overhead. The paper lacks sufficient analysis of end-to-end training efficiency and scalability with increasing numbers of tasks.

  • W3. Limited interpretability of Atom selection: Although the paper provides visualizations of reuse depth, layer-wise capacity allocation, and qualitative prediction examples, the actual Atom selection mechanism remains difficult to interpret. Given the multimodal setting, it would be helpful to provide more intuitive case studies showing which historical Atoms are selected for particular visual tasks, their importance scores, and how their removal affects model predictions. Such analysis could provide stronger evidence that FixedAtom identifies genuinely useful historical knowledge.

Questions for authors

Q1. How does BRACE fundamentally differ from existing methods such as PCLR, MoRAM, and AdaLoRA, particularly regarding Atom-level selection and adaptive capacity allocation?

Q2. How reliably do the first-order loss-based benefit estimates reflect the actual reuse value of historical Atoms? Could the authors provide additional ablations comparing alternative importance scoring strategies, including gradient-based criteria inspired by prior data selection studies? It would also be interesting to investigate whether accounting for interactions, gradient conflicts, or redundancy among Atoms could improve selection effectiveness.

Q3. What evidence supports the assumption in Section 3.3 that weaker historical support indicates greater residual adaptation demand? Could the authors further investigate whether the proposed residual scores correlate with the actual marginal performance gains from allocating additional Atoms to different layers?

Q4. Why is the Input Diversity Loss restricted to input-direction similarity among newly introduced Atoms? Could considering the complete rank-one updates, as well as potential gradient redundancy or conflicts between historical and new Atoms, lead to more effective capacity utilization?

Q5. Could the authors provide a more comprehensive analysis of the computational overhead introduced by Atom scoring, reuse-depth validation, and adaptive capacity allocation? In particular, how does the end-to-end adaptation cost scale with the number of historical tasks and accumulated Atoms?

Q6. Could the authors provide more intuitive visualizations or qualitative case studies to better illustrate the benefit-aware Atom selection process? For example, showing the importance scores and source tasks of selected historical Atoms, together with their effects on multimodal predictions, would help demonstrate what knowledge is actually being reused.

If the authors can address my concerns and questions, I would be willing to raise my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZPJX (1/3)

Thank you for recognizing the motivation behind BRACE, its fine-grained reuse mechanism, and its performance–efficiency trade-off. We address your comments and questions point by point below.

W1.1 & Q1: Atom selection, historical knowledge protection, and capacity management

BRACE's key distinction is that it freezes historical Atoms and task catalogs to reduce forgetting caused by overwriting historical knowledge, uses the current multimodal task loss to identify and validate beneficial reuse, and controls and allocates new capacity before training to reduce redundant learning and parameter growth. We apologize that the lack of clarity in the original manuscript led to your misunderstanding. The rank-one representation provides the basis for fine-grained management in FixedAtom and AdaAtom; we do not claim this representation itself as a primary contribution.

MethodHistorical knowledge protectionComponent selection/reuseCapacity management
PCLRCompresses historical representations and compensates for performance degradation through teacher–student KL distillation and knowledge integrationImage–text query–key matchingCompresses old parameters to release capacity before learning new tasks
MoRAMFreezes historical rank-one memory unitsInput-driven sparse self-activation and dynamic mixingAdds new units, with sparse activation and optional pruning
AdaLoRAFreezes the pretrained backbone, without specifically protecting historical configurations across tasksEvaluates components for the current fine-tuning task using parameter sensitivityDynamically allocates rank during training under a given budget
BRACEFreezes historical Atoms and the Atom references registered in previous task catalogsUses current-task benefit ranking and independent configuration-level validation to determine Atom identities and reuse depthControls new capacity before training using validation reuse gains and allocates it heuristically according to layer-wise historical support

The evaluation scope also differs. The original MoRAM paper evaluates continual recognition with CLIP and continual learning on text tasks, while AdaLoRA primarily evaluates text fine-tuning. Our paper further evaluates multimodal continual instruction tuning, including visual question answering, visual reasoning, and caption generation. PCLR already addresses this setting; its main differences from BRACE lie in the protection, reuse, and capacity management mechanisms summarized above.

W1.2 & Q2: Reliability of first-order benefit ranking and alternative selection strategies

(1) First-order scores accurately rank actual conditional removal benefits.

  • On IconQA/CLEVR-Math, we evaluate 24 Atoms using 96 held-out images for each task. The Spearman correlations between the original BF16 first-order scores and actual removal benefits evaluated in FP32 using the same weights are 0.9965/1.0000. Higher scores correspond to larger loss increases after removal, providing direct evidence for benefit-aware selection.

  • In Table 3 of the paper, with the same parent model, candidate pool, and reuse budget, Top-k achieves higher held-out benefits than Random-k/Bottom-k at all five stages. The Top-k/Random-k benefits are 0.072/−0.040 on VizWiz and 0.021/−0.026 on CLEVR-Math, indicating that benefit ranking identifies configurations that reduce the task loss.

(2) Full-pipeline comparisons support the practical value of the original criterion. Following your suggestion, we adapt four alternative strategies for Atom selection and compare them over the six-task continual adaptation sequence. Compared with FisherSFT-Atom, BRACE's MFT is only 0.0232 points lower, while using 23.50% fewer parameters and reducing training job time by 9.84%. Compared with the other three strategies, BRACE achieves higher MFT with 8.73%–25.10% fewer parameters. This shows that BRACE achieves competitive adaptation performance with fewer parameters and lower time costs throughout continual adaptation:

MethodMFT with known task identitiesAtom parameters (M)Cumulative AtomsSelection time (min)Training job time (h)Training + evaluation GPU-hours
LESS-Atom74.40132.4211,8057.1018.8621.42
FisherSFT-Atom74.51143.2812,7867.0118.2320.50
SPICE-Atom74.25146.3513,0787.4517.9920.19
GRAD-MATCH-Atom74.30120.1010,69210.3218.0720.52
BRACE74.49109.619,7766.4316.4418.68

FisherSFT-Atom's set-level complementarity objective achieves slightly higher MFT but uses more parameters. SPICE-Atom introduces a penalty for conflicts in probe effects, yet achieves lower MFT and uses more parameters than BRACE.

AuthorsOpenReview ↗

Response to Reviewer ZPJX (2/3)

W1.3 & Q3: Historical support and the rationale for capacity allocation

Historical support is an allocation heuristic, not a measured marginal benefit. It directs capacity toward weak-support modules with slightly higher scores at matched budgets. A separate ablation shows that AdaAtom reduces new parameters while maintaining comparable performance. We will clarify the heuristic's role in §3.3.

(1) Historical support directs capacity toward weak-support modules. We compare it with random allocation at matched new-parameter budgets, using the same historical checkpoint, reused Atoms, training settings, and 3,000 hard-routed evaluation samples per task. Weak-support modules are the bottom quartile by pre-training historical support.

DatasetAllocationNew parameters (M)Weak-support capacity shareTask score
ArxivQAHistorical support8.53857.11%95.3000%
ArxivQARandom8.53820.41%95.2333%
VizWizHistorical support20.40536.73%62.1661
VizWizRandom20.40526.19%62.1435

Capacity control supports effective adaptation. The capacity share allocated to weak-support modules increases from 20.41% to 57.11% on ArxivQA and from 26.19% to 36.73% on VizWiz, with slightly higher task scores.

(2) AdaAtom reduces new parameters while maintaining performance. In a separate ArxivQA ablation, we toggle only AdaAtom, retaining FixedAtom and ID with the same parent-model Atoms, historical references, and training settings:

AdaAtomAccuracy after task 2 (%)New parameters (M)New Atoms
Disabled93.4319.991,792
Enabled95.108.54752

AdaAtom reduces new parameters by 57.28% while maintaining comparable performance. Together, the two experiments support the practical value of this heuristic allocation mechanism through layer-wise capacity allocation and overall capacity control.

W1.4 & Q4: Heuristic design of Input Diversity and alternative constraints

The original ID loss heuristically reduces redundancy among new Atoms while allowing them to complement historical knowledge. Additional experiments support your suggestion to regularize complete rank-one updates. However, the additional accuracy benefits of gradient redundancy and conflict constraints are modest and inconsistent across tasks. We will investigate these extensions in future work.

(1) ID prioritizes redundancy within newly added capacity. Historical Atoms are frozen; new Atoms with similar input directions can still complement them through different output effects. We therefore constrain new–new pairs without directly penalizing new–historical alignment. On 64 samples per task from the final UCIT model, the local responses of co-active Atoms, ri(h)=bi(ai⊤h)r_i(h)=b_i(a_i^\top h), show higher overlap for new–new pairs across all five tasks:

Input taskMatched modulesMean squared response correlation: new–newMean squared response correlation: new–historical
ArxivQA940.0790.000056
VizWiz2240.0490.00010
IconQA770.0980.00013
CLEVR-Math2240.0350.000094
Flickr30k790.0490.00012

For nonzero rank-one updates ΔWi=biai⊤Δ W_i=b_i a_i^\top,

=\cos^2(a_i,a_j)\cos^2(b_i,b_j) \leq\cos^2(a_i,a_j).$$ Input-direction similarity thus bounds the squared cosine similarity of complete updates. ID is also applicable at initialization, when $A$ is random and $B$ is zero. These observations motivate the heuristic. **(2) Regularizing complete updates improves accuracy in controlled experiments.** Each task uses the same parent model, initial capacity, and training settings, with two initialization seeds and 3,000 test examples under hard routing. | Regularization | IconQA accuracy (%) | New parameters (M) | CLEVR-Math accuracy (%) | New parameters (M) | | --- | --- | --- | --- | --- | | Original input-side ID | 69.40 | 12.788 | 72.22 | 34.118 | | Complete rank-one updates | 70.27 | 12.990 | **72.63** | 34.053 | | Complete updates + gradient redundancy | 70.50 | 13.089 | 72.42 | **33.978** | | Complete updates + gradient conflicts | **70.60** | 13.028 | 72.27 | 34.151 | | Complete updates + both constraints | 69.67 | **12.727** | 72.18 | 34.058 | M denotes final added parameters in millions. Complete-update regularization improves mean accuracy by **0.87/0.41 percentage points** on IconQA/CLEVR-Math. On IconQA, individual constraints add **0.23/0.33 percentage points** with slightly more parameters. On CLEVR-Math, neither improves accuracy over complete-update regularization; combining both constraints also provides no consistent accuracy gain. We therefore regard your suggestions as promising extensions and will further study their integration with adaptive capacity allocation in future work.
AuthorsOpenReview ↗

Response to Reviewer ZPJX (3/3)

W2 & Q5: Computational overhead and scalability

BRACE's selection takes only 0.53%–0.91% of training-loop time. Across the six-task sequence, the historical pool grows by 149.67%, while selection time increases by only 9.04% from the first reuse stage to the last.

(1) Scoring and validation add little overhead; allocation takes milliseconds. On UCIT with LLaVA-1.5-7B/NVIDIA A100, we time each of the five reuse stages three times and report medians. Selection takes 83.64–96.93 s per stage, including only 8.38–10.09 ms for allocation:

Current taskHistorical tasksCumulative historical AtomsCandidates actually scoredScoring and ranking (s)Reuse-depth validation (s)Capacity allocation (ms)Total selection time (s)
ArxivQA13,5843,58436.5452.308.3888.89
VizWiz24,3362,54436.9746.098.8483.81
IconQA36,1083,80542.9940.648.7883.64
CLEVR-Math46,7243,42145.9741.8710.0988.31
Flickr30k58,9485,04853.4143.509.1696.93

(2) Selection remains inexpensive as the pool grows in this sequence. Historical tasks increase from 1 to 5; stored Atoms grow from 3,584 to 8,948 (+149.67%), but scored candidates grow from 3,584 to 5,048 (+40.85%). Selection time increases by only 8.04 s (9.04%). Candidates comprise shared Atoms and private Atoms from the most recent task.

(3) Joint scoring and a fixed validation search limit computation. Discovery/stability each use 64 probes; one backward pass per probe scores all candidate gates. A separate image-disjoint set of 64 probes evaluates zero reuse and depths 1/2/4/8. These steps run during adaptation; inference loads the saved configuration.

W3 & Q6: Atom provenance and contributions of historical reuse

FixedAtom reuses components from multiple historical tasks. Removal interventions show their joint contributions to counting, scene details, and action descriptions; aggregate loss changes support their reuse value.

(1) Reuse sources vary with the current task. ImageNet-R's share falls from 100.0% on ArxivQA to 23.9% on Flickr30k, where VizWiz/CLEVR-Math jointly contribute 68.9% of reused Atoms:

Current taskImageNet-RArxivQAVizWizIconQACLEVR-MathTotal reused Atoms
ArxivQA100.0%————1,792
VizWiz73.1%26.9%———896
IconQA51.4%5.5%43.1%——1,792
CLEVR-Math61.4%5.8%28.6%4.2%—448
Flickr30k23.9%6.7%26.4%0.5%42.5%1,792

(2) Source-group removal changes predictions in concrete examples. Other components and decoding settings are fixed. Scores are pre-adaptation standardized ranking scores, min⁡(zdiscovery,zstability)\min(z_{\mathrm{discovery}},z_{\mathrm{stability}}). Source denotes the task that trained the parameters; L is zero-based. Listed Atoms are examples; the intervention removes the indicated source group or single Atom:

Target task/scenarioHistorical sourceExample selected AtomSelection scoreAtoms removedFull-model predictionPrediction after removal
IconQA/star countingArxivQAL8.mlp.gate_proj, slot15.087722Source group: 99 Atoms79; correct77; incorrect
Flickr30k/painting a brick wallVizWizL31.self_attn.v_proj, slot03.621361Source group: 473 AtomsA man standing on a ladder painting a brick wallA man painting a brick wall; the ladder is omitted
Flickr30k/gymnasticsCLEVR-MathL20.self_attn.o_proj, slot72.897515Source group: 762 AtomsFlipping on a balance beamAirborne over the uneven bars
Flickr30k/hikingIconQAL5.self_attn.o_proj, slot73.537621Source group: 9 AtomsA group of people hiking through a rocky areaA woman standing in a rocky area
IconQA/comparing pencils and paperImageNet-RL31.mlp.gate_proj, slot87.396224One AtomNo; correctNo; correct, unchanged

The first four source-group interventions cause counting errors, omitted details, or altered action/person descriptions. Removing one Atom in the last example leaves the answer correct, showing that the remaining active components suffice for that example.

(3) Aggregate loss increases after source-group removal. Removing the VizWiz source group increases mean reference-answer CE by 0.2940 over 256 Flickr30k samples. The image-resampling 95% CI is [0.2792, 0.3102], entirely above zero, supporting this group's contribution to reducing task loss.

Reviewer_yXdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal continual instruction tuning and proposes BRACE, which decomposes LoRA updates into rank-one components, termed Atoms, trained on earlier tasks are frozen and kept in a global pool for later reuse. For each new task, it estimates the removal benefit of each historical Atom via gradients, ranks the candidates within each module, and reuses the top-k Atoms, where the reuse depth k is chosen on a held-out validation set. It then derives the task-level active budget from the relative validation reuse gain and allocates the remaining capacity preferentially to layers with weaker historical support. The new Atoms are trained with a loss penalizing the similarity of their input directions within each layer. At inference, a prototype-based router selects one task catalog per sample.

Critical strengths
  • Reuse is implemented through references rather than parameter copies, which makes storage accounting transparent.
  • The selection procedure is designed with statistical robustness in mind: ranking and validation rely on image-disjoint subsets, each Atom is scored by the smaller of its two standardized benefits, and the reuse depth follows a one-standard-error rule.
  • The experiments cover 2 backbones, 3 benchmarks, and 11 baselines including several recent methods, and the paper reports task- and layer-wise capacity allocation.
Critical weaknesses
  1. The near-zero forgetting appears to stem mainly from parameter isolation combined with hard routing rather than from the proposed components. Since historical Atoms and task catalogs are fully frozen and each test sample activates exactly one catalog, the parameters serving a task never change after registration; forgetting can therefore arise only from routing errors, and thus the BWT values are largely guaranteed by construction. But paper reports neither routing accuracy nor a comparison with oracle task identities, and it omits the most informative baseline, ie an independent LoRA per task paired with the same router

  2. The premise that an Atom constitutes a meaningful knowledge unit is not justified, and the novelty is limited. An Atom is simply a column of B paired with the corresponding row of A in LoRA, closely resembling the rank-one units in PCLR and MoRAM and the rank-component importance scoring in AdaLoRA. More importantly, the LoRA factorization is not unique, since BA = (BQ)(Q⁻¹A) for any invertible Q, and Atoms within a task are trained jointly and co-adapt, so an individual Atom carries no canonical meaning.

  3. Several results require clarification. In Tab 1, the first task involves no reuse, so BRACE-NA trains what is effectively a plain rank-32 LoRA on it, yet BRACE-NA and BRACE reach 81.42 and 81.25 on RS after the full sequence, both exceeding Individual 78.15. BRACE also exceeds Individual by about 7.7 on Med and 8.2 points on Sci. Weere they trained under comparable settings? In addition, the seed-42 UCIT results in Tab 17 do not match the main results in Tab 2.

  4. The computational cost of the method is missing. Each new task requires gate-gradient probing over all historical candidates and loss evaluation for every candidate depth, a cost that hardly scales with the pool, yet no training time or memory overhead is given.

Questions for authors
  • Why is a single reuse depth k shared across all modules, and how would performance change if each module could select its own depth?
  • The historical pool is dominated by first-task Atoms in early stages, and these still account for about 37% of all Atoms after the UCIT sequence. How are the reused Atoms distributed across their source tasks? Is the drop in UCIT MAA under the reverse order related to the initial pool provided by the first task?
8 comments
AuthorsOpenReview ↗

Response to Reviewer yXdz (1/4)

Thank you for recognizing BRACE's parameter-sharing mechanism, the robustness of its selection procedure, and its comprehensive experimental evaluation. We address your comments and questions point by point below.

W1: Routing analysis and independent LoRA baseline

Additional experiments show that this result does not depend entirely on hard routing: (i) hard routing achieves 99.85% routing accuracy, with MFN and BWT differing from oracle routing by only 0.02 and 0.01, respectively; (ii) even with soft routing that activates all catalogs simultaneously, BWT is only −1.321, substantially better than most prior methods (e.g., DISCO: −8.87; PCLR: −5.09), indicating limited interference across catalogs with BRACE's validated Atom sharing; and (iii) under identical routing decisions, BRACE achieves 99.0% of the MFN of independent LoRA experts using only 45.0% of their adaptation parameters. We agree that freezing historical Atoms and registered task catalogs, together with activating only one catalog at inference, is the main reason for BRACE's near-zero forgetting. BRACE's broader value lies in fine-grained reuse and capacity allocation at the Atom level, substantially reducing parameter requirements while maintaining accuracy.

(1) Routing accuracy, oracle comparison, and soft routing.

Inference modeRouting accuracyMFNMAABWT
Hard routing99.85%74.3482.62−0.063
Image–text soft routing99.84%73.2782.43−1.321
Oracle—74.3682.63−0.053
  • Routing errors have a negligible impact: Among the 18,000 samples used for routing diagnostics, hard routing selects the wrong catalog for only 27 samples, an error rate of 0.15%. Relative to oracle routing, MFN is only 0.02 lower and BWT only 0.010 lower. Even when routing is incorrect, the selected and correct catalogs often share validated historical Atoms, allowing the model to produce a correct answer. Of the incorrectly routed samples, 4 of 6 on ArxivQA and 2 of 4 on CLEVR-Math are still answered correctly.
  • Forgetting remains limited under soft routing: Soft routing activates all catalogs simultaneously, removing isolation between catalogs. Nevertheless, BWT decreases only to −1.321 and MFN drops by only 1.07 points, with forgetting remaining substantially lower than for most prior methods.
  • Hard routing balances accuracy, forgetting, and inference overhead: Compared with oracle routing, hard routing requires no ground-truth task ID. Under this practical inference setting, its MFN and MAA are only 0.02 and 0.01 lower, respectively, and BWT differs by only 0.010, nearly matching ideal routing. Compared with soft routing, the routing accuracies are almost identical (99.85% vs. 99.84%), but hard routing activates only the selected catalog, avoiding interference from the updates associated with other catalogs. Moreover, hard routing loads only the Atoms referenced by one catalog per sample, so its inference overhead does not grow with the number of tasks; soft routing must compute the updates associated with all catalogs simultaneously.

(2) Independent LoRA expert baseline. We train a separate rank-16 LoRA for each task and use exactly the same routing decision as BRACE for every sample:

MethodHard-routing MFNOracle MFNCumulative parameters
Independent LoRA experts75.0875.10239.86M
BRACE74.3474.36107.84M

The MFN gap is 0.74 under both hard and oracle routing, showing that it does not arise from routing. Independent experts add a complete LoRA for every task. BRACE instead operates at the Atom level, reusing only validated historical components and adding new Atoms only to meet the remaining adaptation needs. This substantially reduces parameter requirements while maintaining accuracy.

AuthorsOpenReview ↗

Response to Reviewer yXdz (2/4)

W2: Atom interpretation and novelty

(1) Non-uniqueness of the decomposition does not undermine selection reliability. The identity BA=(BQ)(Q−1A)BA = (BQ)(Q^{-1}A) holds, and a general invertible transformation can indeed change individual Atoms. However, the score

Δ^u=−⟨buau⊤,∇ΔWL⟩\hat{\Delta}_u = -\left\langle b_u a_u^\top, \nabla_{\Delta W}\mathcal{L}\right\rangle

depends only on the rank-one matrix buau⊤b_u a_u^\top itself. Each Atom is a concrete matrix that is frozen and stored after training, and reuse and catalog references operate on these specific matrices. Although the decomposition is not unique, once chosen, it is fixed in the task catalog, giving each Atom an operational meaning. The decomposition determines which partial configurations are available for selection, but each candidate configuration is evaluated against zero reuse using the actual loss on an independent validation set. This validation safeguard applies to any fixed decomposition.

(2) The design accounts for joint training and co-adaptation. An individual Atom cannot be interpreted independently of the other Atoms, and BRACE explicitly accounts for this dependence. All candidates are active during scoring, so the score estimates a conditional removal effect rather than a contribution attributable to an Atom in isolation. First-order scores are used only for ranking; the final reuse depth is determined by the actual loss of the complete configuration.

(3) BRACE freezes historical Atoms from a fixed decomposition and their task catalogs, using Atoms as the operational units to reduce forgetting caused by overwriting historical knowledge; uses the current multimodal task loss to identify and validate beneficial reuse; and controls and allocates new capacity before training to reduce redundant learning and parameter growth. We apologize that these distinctions were not sufficiently clear in the original manuscript. The rank-one representation provides the basis for fine-grained management in FixedAtom and AdaAtom; we do not claim this representation itself as a primary contribution.

W3: MLLM-DCL settings and seed-42 results

After rerunning under the common training settings, BRACE achieves an MFN of 66.34 on MLLM-DCL, still exceeding the best prior method, ModalPrompt, by 4.56 points; the main conclusions remain unchanged. The seed-42 results reported in Table 17 were obtained with Oracle routing. We sincerely apologize and will correct them in the revised manuscript.

(1) Training settings on MLLM-DCL. Our review confirmed that BRACE and BRACE-NA in the original table used a learning rate of 2×10−42\times10^{-4}, whereas Individual and all baselines used 2×10−52\times10^{-5}, as specified in Appendix C.4. We have rerun the experiments on LLaVA-1.5-7B with a learning rate of 2×10−52\times10^{-5}, rank 32, and hard routing:

MethodRSMedADSciFinMFTMFNMAABWTCumulative Atom parameters
Individual78.1558.2052.7749.3288.0265.29————
BRACE-NA75.5461.2253.0752.1991.3066.3366.6666.38+0.421319.82M
w/o ID75.7060.6553.2751.9790.2266.0266.3666.21+0.430287.12M
BRACE75.7860.7953.2751.7990.0965.9366.3466.19+0.518285.60M

After this correction, performance on RS, the first task, is lower than Individual, consistent with your analysis. BRACE still outperforms Individual on Med, AD, Sci, and Fin because these tasks are learned with validated historical Atom reuse: Individual trains each task's adapter independently from scratch, whereas BRACE can benefit from positive transfer from preceding domains. All three continual adaptation variants also have positive BWT, and the main conclusions remain unchanged.

(2) Seed-42 results in Table 17. When compiling the supplementary results for seed 42, we mistakenly reported the Oracle-routing results (74.40 / 74.36 / 82.63 / −0.053) as our hard-routing results. The corresponding hard-routing results are exactly those in Table 2 (74.39 / 74.34 / 82.62 / −0.063). The differences are only 0.01–0.02 points and do not affect our conclusions.

AuthorsOpenReview ↗

Response to Reviewer yXdz (3/4)

W4: Computational cost

Selection takes only 0.53%–0.91% of the training-loop time at each stage. As the historical pool grows to 2.5 times its initial size, total selection time increases by only 9.0%. Peak GPU memory usage is 19.41 GB during scoring and 21.39 GB during training. We measure selection overhead at each UCIT stage using LLaVA-1.5-7B on a single NVIDIA A100 80GB GPU:

Current taskCumulative historical AtomsScoring and ranking (s)Reuse-depth validation (s)Capacity allocation (ms)Total selection time (s)
ArxivQA3,58436.5452.308.3888.89
VizWiz4,33636.9746.098.8483.81
IconQA6,10842.9940.648.7883.64
CLEVR-Math6,72445.9741.8710.0988.31
Flickr30k8,94853.4143.509.1696.93

(1) Selection accounts for only a small fraction of training time. Total selection time ranges from 83.64 to 96.93 seconds per stage.

(2) GPU memory usage is modest. Measured peak GPU memory usage is 19.41 GB during scoring and 21.39 GB during training.

Q1: Shared reuse depth across modules

A shared depth reduces a combinatorial search space that grows exponentially with the number of modules to a small set of candidate depths. This allows complete configurations across modules to be compared directly using their actual validation losses and reduces the risk of overfitting to a limited validation set. The MFN gains from module-specific depths are within the magnitude of observed variation across seeds, while requiring 19.1%–35.6% more parameters. BRACE selects the task-level reuse depth ktk_t by validating complete configurations and shares this depth across modules within the task. Different tasks can select different depths, each module still selects its own Atoms according to its own scores, and new capacity is allocated separately to each module.

(1) Comparison with module-specific effective reuse depths. We compare BRACE's shared depth with two variants that allow the effective reuse depth to vary by module. Both retain complete-configuration validation, AdaAtom, and ID loss:

  • Module-wise Depth (MD): Each module determines its own effective reuse depth by retaining the ranked prefix satisfying min⁡(zdis,zsta)>0\min(z_{\mathrm{dis}}, z_{\mathrm{sta}})>0, capped at 8 Atoms.
  • Task-Capped Module-wise Depth (TCMD): BRACE first selects the task-level depth ktk_t. Each module then retains only positively scored Atoms among its top-ktk_t candidates. Effective reuse depths differ across modules.

We conduct these experiments on UCIT with LLaVA-1.5. All three methods use Oracle routing to exclude routing effects:

MethodMFTMFNMAABWTCumulative Atom parametersCumulative Atoms
BRACE74.4074.3682.63−0.053107.84M9,576
MD74.5574.6782.79+0.150128.45M11,402
TCMD74.6474.3982.74−0.307146.18M13,055

Shared depth provides a better balance between accuracy and storage. MD and TCMD improve MFN by only 0.31 and 0.03 points, respectively, both within the magnitude of observed variation across seeds (MFN ranges from 73.96 to 74.36 in Table 17), but increase cumulative parameters by 19.1% and 35.6%, respectively. TCMD also lowers BWT to −0.307. These results support that shared depth captures the main benefits of reuse with substantially fewer parameters.

Reviewer_moNBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BRACE, an atomized continual adaptation framework for multimodal continual instruction tuning (MCIT). The setting is sequential multimodal instruction tuning without historical data replay and without ground-truth task IDs at inference. The paper identifies two main problems: (1) historical knowledge protection and reuse are insufficiently coordinated, since module-level reuse can introduce unsuitable directions or discard useful components; and (2) additional capacity is often allocated and utilized inefficiently, causing imbalanced layer-wise allocation and redundant low-rank components.

Critical strengths
  1. The atom-level formulation directly addresses the granularity problem in continual LoRA reuse: module-level reuse is too coarse, while rank-one Atoms allow independent scoring, selection, and referencing. The combination of immutable historical protection, benefit-aware reuse, adaptive residual allocation, and redundancy reduction is logically integrated.

  2. Using first-order gate-gradient removal benefits to rank historical Atoms is efficient and practical. Stabilizing rankings across discovery and stability subsets and validating reuse depth as a complete multi-layer configuration are good design choices. This is more principled than arbitrary top-k or random reuse.

  3. AdaAtom connects reuse evidence to residual capacity demand rather than using fixed per-task ranks. The layer-wise residual score and global capacity cap are reasonable mechanisms for balancing historical support and new learning. The storage reductions are meaningful.

Critical weaknesses
  1. The adaptive components do not clearly improve accuracy over the non-adaptive variant. This is the most important concern. BRACE-NA, which retains FixedAtom but removes AdaAtom and ID loss, often has higher MFN/MAA than full BRACE. For example, on UCIT, BRACE-NA reaches MFN 74.72 versus BRACE 74.34; on MLLM-DCL, 70.59 versus 70.31. On InternVL, BRACE-NA is also slightly better in MFN/MAA. The ablation shows that adding AdaAtom alone decreases MFN, and ID loss only partially recovers it. Thus, the main accuracy gains over prior methods appear to come largely from FixedAtom and atom-level reuse, while AdaAtom and ID loss mainly reduce storage and directional redundancy. The paper should frame this trade-off more explicitly rather than presenting adaptive allocation as a uniform accuracy improvement.

  2. Task-ID-free routing is under-analyzed and confounds the results. BRACE relies on a prototype/calibration router to select a task catalog at inference. The paper does not report router accuracy, misrouting rates, or an error analysis separating routing errors from adaptation errors. Since old Atoms are frozen and old catalogs are immutable, near-zero BWT is partly expected by construction if routing is correct. MFN/MAA/BWT therefore reflect the combined effect of catalog selection and answer generation, not purely parameter-level forgetting. This is a major interpretive issue: the strong BWT result may be driven by frozen catalogs and routing behavior rather than by better continual learning in shared parameters. Router robustness should be evaluated directly.

  3. Benefit estimation is approximate and potentially brittle. The removal benefit is a first-order Taylor approximation computed on finite current-task probes and conditional on the jointly active candidate pool. It estimates the effect of removing one Atom while others remain active, not the gain from independently adding it. The reuse-depth rule uses validation configuration benefit with a standard-error threshold, which is reasonable but introduces several hyperparameters: candidate depth set, probe sizes, discovery/stability/validation splits, standardization floor, and threshold. The paper provides limited sensitivity analysis for these choices. The method could be sensitive to probe representativeness and task distribution.

Questions for authors

The study uses only 7B backbones, three benchmarks, and finite task sequences. MLLM-ACL gains are modest and mixed: BRACE improves MFN over BRACE-NA but lowers MAA, and the text/table BWT values appear inconsistent. Larger models, longer task streams, and more diverse multimodal capabilities are not evaluated.

6 comments
AuthorsOpenReview ↗

Response to Reviewer moNB (1/3)

Thank you for recognizing BRACE’s fine-grained reuse, benefit validation, and storage benefits from adaptive capacity allocation. We address your comments and questions point by point below.

W1: Limited accuracy gains from adaptive components

AdaAtom and ID loss reduce new capacity and improve capacity utilization while maintaining comparable accuracy; BRACE’s accuracy gains over prior methods primarily come from FixedAtom’s historical Atom freezing and selective Atom-level reuse. The abstract and conclusion accordingly use “comparable performance” and “only small decreases”; we do not claim that adaptive allocation raises the accuracy ceiling.

The incremental UCIT ablations (Tables 4 and 12) support these roles.

  • AdaAtom controls capacity. Adding AdaAtom alone reduces cumulative new Atoms from 15,680 to 11,144 (−28.9%). It shrinks the new-capacity budget according to the validation loss reduction from reuse, ete_t (Eq. 6). Greater validated historical support leads to less new capacity.
  • ID loss improves capacity utilization. Adding ID loss to AdaAtom increases MFN from 73.69 to 74.34 (+0.65), further reduces cumulative Atoms to 9,576, and lowers the mean similarity between new Atom input directions from 0.0539 to 0.0229 (Table 12). ID loss primarily regularizes these input directions. Since a layer’s update satisfies rank⁡(BA⊤)≤rank⁡(A)\operatorname{rank}(BA^\top)\leq\operatorname{rank}(A), encouraging diverse input directions helps prevent the limited new capacity from collapsing into a lower-rank representation.
  • Overall trade-off. Compared with BRACE-NA, BRACE’s MFN is only 0.38 points lower (74.34 vs. 74.72), while cumulative Atom parameters decrease from 174.90M to 107.84M (−38.3%). BRACE thus achieves 99.5% of BRACE-NA’s MFN using approximately 60% of its adapter parameters.

W2: Routing analysis and attribution of results

Routing errors have a very small effect on the results: hard routing achieves 99.85% Top-1 accuracy, with MFN only 0.02 points and BWT only 0.01 points below Oracle routing using ground-truth task catalogs. The contribution of parameter reuse is reflected in learning new tasks with fewer parameters, rather than in BWT alone. To distinguish routing effects from adaptation effects, we compare hard routing, image–text soft routing, and Oracle routing under matched conditions on UCIT:

Inference methodRouting accuracyMFTMFNMAABWT
Hard routing (BRACE)99.85%74.3974.3482.62−0.063
Image–text soft routing (DISCO-style weighting)99.84%74.3773.2782.43−1.321
Oracle (ground-truth task catalog)—74.4074.3682.63−0.053

(1) High routing accuracy and separately measurable routing effects. We draw 18,000 samples from the task test sets for routing diagnostics. Hard routing selects an incorrect catalog for only 27 samples, yielding an error rate of 0.15%; outputs for the remaining 17,973 correctly routed samples are identical to those under Oracle routing. Oracle routing eliminates catalog-selection errors, so its comparison with hard routing measures their effect. The BWT of −0.053 remains under Oracle routing, showing that the small end-to-end performance change is not entirely attributable to routing errors.

(2) Near-zero BWT also holds without task IDs. Near-zero BWT is an expected outcome of freezing historical Atoms and registered catalogs. The comparison above demonstrates that this behavior is also maintained under task-ID-free inference and does not require Oracle catalog selection.

(3) The comparison supports single-catalog hard routing. Soft routing achieves almost the same Top-1 accuracy as hard routing, but MFN decreases by 1.07 points and BWT drops to −1.321. Soft routing simultaneously activates multiple task catalogs; even when the highest-weight catalog is correct, adaptation increments from other catalogs can interfere with predictions.

(4) Independent experts with the same router. Table 3 shows that benefit-based reuse outperforms random reuse at all five evaluated stages. We further add a baseline that trains one independent rank-16 LoRA per task and uses exactly the same per-sample routing decisions as BRACE:

MethodHard-routing MFNOracle MFNAtom parameters
Independent LoRA experts75.0875.10239.86M
BRACE74.3474.36107.84M

BRACE achieves 99.0% of the independent experts’ MFN using only 45.0% of their adapter parameters. The performance gap is approximately 0.74 points under both hard and Oracle routing, showing that this gap is not driven by routing errors. Shared Atoms therefore substantially reduce cumulative adapter storage across tasks while maintaining near-zero forgetting.

AuthorsOpenReview ↗

Response to Reviewer moNB (2/3)

W3: The first-order benefit estimate is an approximation and may lack robustness

BRACE uses only the signs and relative ranking of first-order estimates; the final reuse depth is determined by the actual loss of complete configurations on a validation set. Performance remains stable when the size and distribution of the probe samples change. The first-order estimate approximates the conditional loss change caused by removing Atom u while keeping all other candidate Atoms active, measuring the Atom's alignment with the negative gradient of the current task. It belongs to the same family of proxy metrics as Taylor importance in pruning, but retains the sign to capture the relative benefit of an Atom.

(1) The final decision does not rely on the first-order approximation. First-order scores are used only for ranking; the reuse depth is determined by the actual validation loss of complete configurations, which already accounts for interactions among Atoms. For ranking, we take the smaller of the standardized scores computed on two subsets with no overlapping images. For depth selection, we use a threshold based on the standard error. Both mechanisms help suppress finite-sample noise. Existing results in the paper support this design: on an evaluation set disjoint from the scoring data, the actual configuration benefits consistently satisfy Top-k > Random-k > Bottom-k across all five tasks.

(2) Robustness to probe samples. We designed a set of controlled experiments: probe samples were randomly drawn from an independent candidate pool, and new Atoms were initialized using a separate random seed so that shared parameters had identical initial values across probe conditions. Thus, differences in scores arose solely from the probe settings. Under each setting, we reselected the Atoms to reuse and trained the current task.

TaskProbe setting (discovery / stability / validation)Current-task scoreNew Atom parameters
IconQARandom-probe baseline (64 / 64 / 64)68.956.89M
IconQAHalf the samples (32 / 32 / 32)68.777.06M
IconQASampling biased toward longer samples (64 / 64 / 64)69.077.65M
Flickr30kRandom-probe baseline (64 / 64 / 64)58.057.21M
Flickr30kHalf the samples (32 / 32 / 32)58.097.05M
Flickr30kSampling biased toward longer samples (64 / 64 / 64)58.127.12M

Compared with the random-probe baseline, BRACE is insensitive to changes in both probe sample size and distribution:

  • Sample size: Reducing all three probe sets (discovery, stability, and validation) from 64 to 32 samples decreases the IconQA score by only 0.18 and increases the Flickr30k score by only 0.04. Changes in added parameters remain within +/-2.5% (IconQA: +2.5%; Flickr30k: -2.2%). Thus, reuse selection and capacity estimation remain stable even when the number of probe samples is halved.
  • Sample distribution: Biasing sampling toward longer samples changes the scores by no more than 0.12 on either task (IconQA: +0.12; Flickr30k: +0.07). Added parameters remain nearly unchanged on Flickr30k (-1.2%) and increase by 11.0% on IconQA. These results suggest that, under a shift in the probe distribution, BRACE uses AdaAtom to adjust the added capacity to the estimated remaining demand, while final performance remains largely unaffected.
AuthorsOpenReview ↗

Response to Reviewer moNB (3/3)

Q1: Improvements on MLLM-ACL and model scale

On MLLM-ACL, the gap between Zero-shot and Individual is only 8.88 points (MFT of 33.05 vs. 41.93 on LLaVA-1.5), leaving limited room for improvement. Within this range, BRACE achieves an MFN of 41.55 on LLaVA-1.5, reaching 99.1% of Individual's performance, and 42.26 on InternVL, exceeding Individual (41.78). BRACE outperforms all prior methods on both backbones. Compared with BRACE-NA, BRACE achieves a 0.32 higher MFN and a 0.21 lower MAA on LLaVA because MAA weights all stages equally: BRACE has slightly lower accuracy immediately after learning each of the first three tasks, but exhibits less forgetting (BWT improves from -0.54 to -0.02) and scores higher on the last task. On InternVL, both metrics improve. BRACE also remains effective on the larger Qwen3-VL-8B-Instruct model.

(1) BRACE outperforms prior methods on both backbones and achieves accuracy comparable to BRACE-NA.

  • Compared with prior methods: On LLaVA-1.5, BRACE improves MFN and MAA over the best prior results by 2.48 and 1.14 points, respectively (41.55 vs. DISCO's 39.07, and 37.73 vs. SEFE's 36.59; Table 15). On InternVL, the corresponding gains are 3.12 and 1.65 points (42.26 and 38.38, compared with DISCO; Table 16).
  • Compared with BRACE-NA: MFN considers only the final stage. BRACE exhibits less forgetting and scores 1.50 points higher on APP, yielding a 0.32 higher MFN. MAA weights all four stages equally: BRACE has slightly lower accuracy immediately after learning OCR, Math, and VP (-0.30, -1.20, and -0.26), with a combined contribution of approximately -0.30 to MAA. The final-stage advantage contributes only +0.08, resulting in a net MAA decrease of 0.21. These differences reflect only small performance variations, with overall accuracy remaining comparable to BRACE-NA.

(2) Effectiveness on a new backbone and a larger model. We conducted the six-task UCIT experiment on Qwen3-VL-8B-Instruct, which has a larger parameter count (8B) and a different architecture from LLaVA and InternVL:

MethodImageNet-RArxivQAVizWizIconQACLEVR-MathFlickr30kMFTMFNMAABWTParameters
Zero-shot32.1093.9326.6544.6795.6329.0153.67--------
Individual93.9095.7767.8498.5099.3762.6586.34------422.38M
BRACE-NA93.9795.8067.9397.5399.4062.5386.3786.1990.14-0.21348.72M
BRACE93.8795.9367.8796.6099.4361.8686.0085.9390.00-0.09267.45M
  • Compared with Zero-shot: BRACE's MFN exceeds the Zero-shot average by 32.26 (85.93 vs. 53.67), demonstrating that continual adaptation remains effective on the new backbone.
  • Compared with Individual: BRACE's MFN reaches 99.5% of Individual's average performance (85.93 vs. 86.34), with 36.7% fewer added parameters (267.45M vs. 422.38M). At the task level, BRACE slightly outperforms Individual on ArxivQA, VizWiz, and CLEVR-Math and performs nearly identically on ImageNet-R. The gaps mainly arise from IconQA (-1.90) and Flickr30k (-0.79).
  • Compared with BRACE-NA: MFN and MAA decrease by only 0.26 and 0.14, respectively, while BWT is closer to zero (-0.09 vs. -0.21) and added parameters decrease by 23.3%.

On Qwen3-VL-8B-Instruct, BRACE achieves performance comparable to Individual with near-zero forgetting and substantially fewer added parameters, consistent with the findings on LLaVA-1.5 and InternVL.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

AuthorsOpenReview ↗

Discussion Summary for the Area Chair

Dear Area Chair,

Thank you for your attention during the discussion period and for the reviewers’ evaluations and constructive comments. We appreciate their positive assessments of BRACE’s fine-grained reuse, parameter sharing, and performance–efficiency trade-off.

We addressed their concerns through additional experiments, quantitative analyses, and more carefully scoped claims. We summarize the issues and responses below.

Reviewer moNB

Q1: Do AdaAtom and ID loss improve accuracy? We clarified that they reduce new capacity and directional redundancy while maintaining comparable accuracy, rather than raising the accuracy ceiling.

Q2: Routing and independent LoRA baseline. We compared hard routing (99.85%), soft routing, and Oracle routing, and added independent LoRA experts under identical routing decisions to disentangle routing, isolation, and reuse.

Q3: Robustness of benefit estimates. Sensitivity analyses with half-sized and length-biased probe sets produced stable conclusions.

Q4: Different MFN and MAA trends on MLLM-ACL. MFN considers only the final stage, whereas MAA averages all stages equally. BRACE forgets less and performs slightly better at the final stage, but has slightly lower immediate accuracy after the earlier tasks; MFN therefore rises slightly while MAA falls slightly. This follows from the metrics’ aggregation schemes, and performance remains comparable.

Q5: Limited evaluation scope. We added experiments on Qwen3-VL-8B-Instruct, showing that BRACE remains effective on a larger model with a different architecture.

Reviewer yXdz

Q1: Near-zero forgetting may mainly come from parameter isolation and hard routing. We accepted this observation and clarified that freezing and task isolation protect historical knowledge, while BRACE improves parameter efficiency through fine-grained Atom reuse and capacity allocation. With matched training, hard routing, and active rank, shared-pool reuse with adaptive allocation reduces cumulative parameters by about 31% versus task-private atomized LoRA, with comparable final performance.

Q2: Atom meaning and novelty. An Atom is a rank-one component of the trained LoRA decomposition and an operational unit for selection, reuse, and capacity management. We do not claim that an Atom has canonical semantic meaning, and rank-one decomposition itself is not our contribution.

Q3: MLLM-DCL training settings. We reran MLLM-DCL with a unified learning rate of 2e-5 and updated all related results and analyses; the main conclusions remain unchanged.

Q4: Computational cost. Selection accounts for only 0.53%–0.91% of training time; peak GPU memory is 19.41 GB during scoring and 21.39 GB during training.

Q5: Shared versus module-specific reuse depth. MD and TCMD improve MFN by only 0.03–0.31, while increasing parameters by 19.1% and 35.6%, respectively. Shared depth therefore provides a better balance between performance and storage.

Q6: Atom sources and reversed task order. Position-weighted analysis shows that about 92% of the MAA difference comes from task order rather than the initial historical pool. We report detailed statistics of reuse sources.

Reviewer ZPJX

Q1: Differences from PCLR, MoRAM, and AdaLoRA. We compared historical knowledge protection, component selection/reuse, and capacity management.

Q2: First-order estimates and gradient-based data selection. The correlations between first-order scores and actual removal benefits are 0.9965/1.0000. We also compared four Atom-selection strategies inspired by LESS, FisherSFT, SPICE, and GRAD-MATCH, supporting our method’s effectiveness and parameter efficiency.

Q3: Historical support and expansion gains. Equal-capacity controlled experiments show that their relationship varies across tasks and metrics and cannot reliably predict marginal gains. This further clarifies layer-wise allocation as a heuristic for determining allocation priority.

Q4: ID loss and gradient redundancy/conflict. We added ablations on complete rank-one-update regularization and gradient-redundancy/conflict constraints; these constraints did not provide consistent gains across tasks.

Q5: Cost and scalability. We added overhead measurements and analyzed how selection cost changes as the candidate pool grows.

Q6: Interpretability. We added Atom-source analysis and concrete cases showing how removing source groups changes model predictions.

Following Reviewer yXdz’s suggestion, we further clarified the paper’s contribution and revised the relevant parts of the PDF, with all changes highlighted in blue. For the other reviewers’ suggestions—including gradient-based data-selection comparisons and further consideration of gradient redundancy and conflict constraints—we will add discussion and analysis in the Camera-Ready version. We again thank you and all reviewers for your attention, guidance, and valuable feedback.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.