acceptodds
Under review as a conference paper at ICLR 2027

COLORS: Consolidated Orthogonal LoRA Subspaces For Continually Learning Tasks

Abstract

We introduce Consolidated Orthogonal LoRA Subspaces (COLORS), a continual learning framework that enables foundational models to learn an expanding set of tasks while consolidating shared skills, with minimal parameter growth and without forgetting previously learned tasks. COLORS makes no assumptions about task ordering and provably preserves previously learned tasks exactly. Under a cheaply verifiable incoherence condition between tasks, we further show that each new task can be learned to near-oracle quality. Our approach addresses a growing challenge created by the widespread use of low-rank adapters (LoRAs): while LoRAs make it easy to adapt foundation models to individual tasks, they provide no principled mechanism for organizing or consolidating knowledge across tasks. Existing continual learning methods either repeatedly update a shared subspace, causing task drift, or add new parameters for each new task, which increases model size and training costs as more tasks are learned. COLORS instead organizes LoRA updates into orthogonal subspaces that preserve previously learned task solutions while consolidating reusable structures within a compact parameter budget, smaller than separate adapters. Across challenging image-to-image transformation task sets, including image restoration, we demonstrate that COLORS continually adapts multiple foundational-model backbones while often outperforming state-of-the-art continual learning methods in reconstruction quality.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Low-rank adaptation, Orthogonal Task sharing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WskeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents COLORS, a LoRA-based method for learning image-to-image tasks sequentially on a frozen diffusion backbone. Each task is stored as coefficients over a shared set of rank-one atoms plus a private low-rank residual. The residual is trained to be orthogonal to the stored atoms and residuals through a cheap Gram–Schmidt projection. After each task, a direction that recurs across residuals can be promoted to a new shared atom. The authors claim exact zero forgetting, a learnability bound under task incoherence, and strong results on restoration, stylization and video across six backbones.

Critical strengths

The append-only basis guarantees that stored adapters never change as the basis grows. This improves on methods that recompute a PCA basis and silently alter old tasks. The rank-space Gram–Schmidt trick keeps training cost flat as tasks accumulate. The experiments cover many backbones and task types, and the consolidation of public FLUX adapters is convincing and reproducible. As a way to store many task adapters compactly, the method is practical and well engineered.

Critical weaknesses

W1: The comparison with O-LoRA and InfLoRA in Table 2 is unfair. The introduction criticizes both methods (Sec. 1), but they are class-incremental methods: they have no task ID at inference, so all adapters stay active for every input. COLORS is evaluated as task-incremental learning. At inference it uses the task ID and a task-specific prompt to load only the matching coefficients and residual (Table 2 caption, Table 7). The independent-adapters baseline, which keeps one separate LoRA per task, reaches almost the same quality as COLORS (Tables 2 and 9). The large gap therefore comes from the task ID, not from the proposed method.

W2: The paper is positioned as continual learning, but the method works as multi-adapter compression. At inference, each task loads only its own scaling of the shared part and its own residual. This is equivalent to keeping one LoRA per task, as in Share, and the authors confirm that the task must be specified at inference (Sec. 5). Zero forgetting comes from storing each adapter exactly (Prop. 1, App. D.2), not from orthogonality, and removing Gram–Schmidt barely changes the results (Table 12). The method therefore does not address forgetting when the task is unknown at inference.

W3: The orthogonality that the theory relies on is not maintained. The shared part is only a task-specific diagonal scaling of a shared U and V (Eq. 3). The main text requires each residual to be orthogonal to all atoms and to all earlier residuals (Eq. 2). However, adding a new atom subtracts its component from every stored residual (Eq. 8), so residuals that were orthogonal start to overlap. The appendix confirms that residuals overlap and that orthogonality to the atoms is only approximate (App. D.1). The theoretical basis set up in the main text therefore does not hold.

W4: Atom selection does not optimize its own criterion. Eq. 7 scores a candidate atom by squared projections of the residuals, which is a second-order, PCA-style objective. The candidate itself, however, is the top singular pair of the arithmetic mean of the residuals (Sec. 2.3), which is a first-order statistic. The mean cancels components of opposite sign and can be dominated by one large residual, so it can miss shared directions that Eq. 7 would favor. In addition, if the residuals were orthogonal as Eq. 2 claims, little shared energy would exist to consolidate.

Questions for authors

Q1: Could the authors report O-LoRA and InfLoRA with access to the task ID, and COLORS without the task ID and with all adapters active?

Q2: Do the residuals remain orthogonal after new atoms are added, and do stored tasks drift when Gram–Schmidt is removed?

Q3: Could the authors compare COLORS with independent LoRAs and with existing LoRA compression methods at the same parameter count?

Q4: Could the authors explain why Eq. 7 uses squared projections while the candidate atom comes from the first-order mean, and whether a second-order extraction would select different atoms?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wske

W1 & Q1. Task identity and baseline fairness. We address both concerns through identical task-aware comparisons (General Response, Sec. 1). COLORS outperforms O-LoRA and InfLoRA, even when they are given task-ID access, confirming that task identity alone does not explain the performance gap.

W2. Continual learning or multi-adapter compression? COLORS is a continual learning method, not merely post-hoc adapter compression: tasks arrive sequentially without revisiting earlier data, and the shared basis is built and expanded online. Unlike independent adapter storage, later tasks actively reuse previously learned knowledge. In our experiments, appended atoms explain 25-99% of later residual energy, reducing the residual rank from 16 to 4 (Sec. 2.3).

How is zero forgetting guaranteed? We agree that zero-forgetting follows from append-only consolidation, which preserves previously learned adapters without modifying their reconstructed updates. However, orthogonality is what separates shared and private directions and establishes the relative storage-error bound in Proposition 1 (App. D.2; General Response, Sec. 2). Unlike SHARE, which also consolidates task adapters but introduces up to 104% drift, COLORS preserves stored adapters to numerical precision (Fig. 5d, Table 3). The revised abstract and Sec. 2 now make these roles explicit.

Q2. Do residuals stay orthogonal after appends? Do tasks drift without GS? Residuals remain orthogonal to shared atoms, while their mutual overlap remains minimal (median absolute cosine 9.8e-10) (App. D.1; General Response, Sec. 2). However, this does not affect zero forgetting, since append-only consolidation preserves each stored adapter exactly (Eq. 8). Even without Gram-Schmidt, our 5k-step run preserved stored adapters with error 5.8e-8. However, without GS, the relative storage-error bound of Prop. 1 is no longer guaranteed because residual-to-atom orthogonality may fail (App. D.2).

GS also ensures clean shared/private separation: Median private residual to shared atom overlap drops from 0.37 without GS to 7.5e-9 with GS. A separate ablation in Table 12 also shows substantial overlap without GS (0.77). Importantly, this improved separation comes at no measurable cost to restoration quality, with nearly identical PSNR (20.32 with GS vs 20.34 dB without GS on identical inputs), consistent with Theorem A under its assumptions.

W3. Orthogonality after appends. As clarified in the General Response (Sec. 2) and Q2 above, the orthogonality required by Prop. 1 holds up to numerical precision (App. C). Appending an atom can change the inner product between residuals by −γi,c+1γj,c+1-\gamma_{i,c+1}\gamma_{j,c+1} (Eq. 8), but the measured overlap remains negligible (median abs. cosine 9.8e-10). Theorem A assumes residual incoherence rather than exact mutual orthogonality (App. D.1). The revision states this after Eq. 2.

W4 & Q4. Choosing mean candidate vs squared-energy score (Eq. 7). We use the mean to identify shared directions because directly optimizing Eq. 7 instead favors task-specific directions. We verified this on the final 16-task Sana model. In all 80 layers, the candidate obtained by alternating maximization drew 100% of its score from a single task (median score of 0.197, close to the theoretical upper bound of 0.215) (Bessel inequality). In contrast, the mean-based candidate draws on approximately three tasks per layer, with the largest task contributing a median of 50% of the Eq. 7 score.

Why does direct optimization favor task-specific directions? Gram-Schmidt trains each residual to be orthogonal to earlier ones (Eq. 2), with only small overlaps after appends (W3). For near-orthogonal residuals, Bessel's inequality (which bounds the total squared projection onto orthogonal directions) caps the Eq. 7 score near the largest residual's energy share, so direct optimization favors private directions. Opposite signs (+A, -A) can cancel in the mean but also imply residual overlap, which GS suppresses. Thus, the mean proposes atoms and Eq. 7 accepts or rejects them (Sec. 2.3). An atom's value lies in reuse by later tasks, not overlap between stored residuals (W2).

Does COLORS outperform independent adapters and SHARE at matched storage budgets? COLORS matches compressed independent adapters in quality while enabling mixed-task composition. Rank-6 independent adapters match COLORS' 34.4M parameters (retaining at least 99.98% of their energy) and score 21.19 vs 21.30 dB. On mixed inputs, COLORS gives the highest composed PSNR of all methods and beats the hindsight-best single task (+0.12 dB), while summed independent adapters fall below it (-0.19 dB; table in reply to yUhn).

Against SHARE at matched storage, COLORS has lower error at every budget on 16 public FLUX adapters (22.7% vs 34.1% at 0.34G; 2.2% vs 9.5% at 1.5G; Tables 3, 14), with zero drift versus SHARE's 13-52%.

Thanks to the authors for their response. W1 changes the inference Settings of O-LoRA and infLoRA, and it seems that the authors confuse the difference between "Task-aware" and "Ground-truth Task-ID". Furthermore, W3/W4 seems to confirm my concerns that while the overlap is down to a negligible magnitude, this does not translate into performance gains in the end. Therefore, I tend to keep my score.

AuthorsOpenReview ↗

Follow-up to Reviewer Wske's comment

We thank the reviewer for the follow-up. We would like to clarify two points that may not have been sufficiently clear in our previous response.

Task-ID comparison. Our additional experiment deliberately provides the ground-truth task ID to O-LoRA and InfLoRA. The purpose is to test whether their lower performance in the original evaluation is caused by keeping all adapters active, including those learned after the evaluated task.

For a test image belonging to task k, we activate adapters 1 through k, matching the configuration under which that task was trained. This is an additional controlled comparison, not a replacement for the original all-adapters-active evaluation in Table 2. We do not claim that this setting evaluates the baselines' ability to identify tasks without supervision.

Even with access to the correct task identity, O-LoRA and InfLoRA achieve 19.89 and 19.74 dB, compared with 21.30 dB for COLORS. Thus, the performance difference cannot be attributed solely to the absence of task identity. The matched-budget experiments further support this finding.

Role of orthogonality. Gram-Schmidt is important for maintaining the consolidated representation. It prevents shared and private components from redundantly representing the same directions.

The ablation shows that residual-to-shared overlap decreases from 0.37 to 7.5e-9, while the fraction of task energy represented by the shared basis increases from 54% to 79%, with nearly identical PSNR (20.32 vs. 20.34 dB).. We note that this separation also supports the relative storage-error guarantee in Proposition 1.

Atom selection. We note that the performance of COLORS is on par with independent adapters. Atom selection is primarily intended to influence consolidation, as described in Sec.2.3. Moreover, directly optimizing Eq. 7 through alternating maximization produces candidates dominated by a single task, whereas our mean-based candidate draws contributions from multiple tasks. The results in Appendix E.9 (Table R6) further support our choice of candidates for consolidating reusable knowledge rather than selecting directions that maximize the immediate energy score.

Together, orthogonal decomposition and append-only consolidation enable COLORS to share knowledge across tasks while maintaining competitive performance, reducing storage, and preserving previously learned adapters without drift.

We hope these clarifications address the reviewer's remaining concerns.

Reviewer_yUhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses continual learning of LoRA adapters on frozen generative backbones without access to past data. Existing methods either update a shared subspace and let old tasks drift, or keep separate adapters and grow storage linearly. The authors propose COLORS: each task is stored as coefficients over a shared basis of orthonormal rank-one atoms plus a private low-rank residual. The residual is trained with a Gram-Schmidt projection, done in rank space for efficiency, that keeps it orthogonal to all atoms and all earlier residuals. After each task, a consolidation step appends at most one new atom to the basis and re-expresses old tasks exactly, so stored adapters never change. Experiments on a 16-task image restoration stream over six diffusion backbones, plus style, video, and consolidation of 16 public FLUX adapters, show flat quality as tasks accumulate, zero drift, flat training cost, and much less storage than separate adapters.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The idea is simple and sound. Freezing shared directions and only appending to them makes forgetting impossible by construction rather than something to regularize, and the rank-space projection is what makes this affordable at scale.
  3. The method is tested on many backbones and long task streams across restoration, style, and video.
Critical weaknesses

1- The paper makes two claims about its results, zero forgetting and storage/speed/efficiency generally. Efficiency is clear and well proved. But the zero-forgetting headline is trivial under the paper's own protocol. Task identity is given at inference and only the current task's adapter is merged. In that setting independent adapters also have zero forgetting, flat cost, and base-model speed, and the paper's own results show they tie with COLORS at every stream length (Table 2). Meanwhile O-LoRA, InfLoRA, and prefix-sequential Gram-Schmidt are run with all adapters active. Comparing a task-aware method against task-agnostic runs of the baselines is not like for like.

2- Fairness of the all-in-one comparison is unclear. The five baselines (OneRestore, InstructIR, MoCE-IR, BioIR, AdaIR) all have released pretrained checkpoints, but the paper trains them from scratch on its own data for 15k steps, which shows them weaker than their published strength. COLORS, in contrast, adapts SD3.5-M, a 2.5B parameter model pretrained on huge image data. The quality advantage claimed over them could therefore come from the pretraining gap rather than from the continual learning method.

Questions for authors

1- If O-LoRA, InfLoRA, and prefix-sequential Gram-Schmidt are run with task identity at inference, as COLORS is, what are their numbers? 2- In the all-in-one comparison, how would you isolate the contribution of the continual learning method from that of the pretrained 2.5B backbone?

3 comments
AuthorsOpenReview ↗

Response to Reviewer yUhn

W1. Is zero forgetting trivial with task IDs, and are the baseline comparisons fair? Independent adapters also achieve zero forgetting with known task IDs. However, COLORS additionally consolidates shared knowledge while preserving earlier adapters, using 3.5x less storage than full independent adapters (Fig. 10d). We address the O-LoRA/InfLoRA fairness concern through task-aware and all-adapters-active evaluations under identical inference settings (General Response, Sec. 1).

Note that COLORS offers advantages over independent adapters beyond zero forgetting. It achieves comparable restoration quality with 3.5x less storage (108M vs 382M parameters on SD3.5, Fig. 10d).

The shared representation also enables better task composition. We evaluate 210 images (CDD-11, OneRestore) with two or three simultaneous degradations (low-light, haze, rain/snow) across 7 mixed sets. For each method, we apply all relevant stored tasks together and compare the result against the best result obtained by applying each stored task individually and selecting the highest-PSNR output for each image.

MethodComposed PSNR (dB)vs. best single (dB)
Degraded input14.88-
O-LoRA13.81-0.04
InfLoRA13.25-0.15
Independent (summed)15.66-0.19
COLORS15.83+0.12

COLORS achieves the highest composed PSNR and is the only method that improves upon the best single-task result. Its average improvement is +0.12 dB, reaching +2.71 dB on low-light + haze.

Q1. O-LoRA, InfLoRA and prefix-sequential GS with task identity. We evaluate all three baselines with the true task ID on the same 787 images (General Response 1):

Method, task ID givenPSNRSSIMLPIPS
O-LoRA19.890.6060.278
InfLoRA19.740.6010.288
Prefix-sequential GS21.360.6310.218
COLORS21.300.6250.231

At a matched 15k budget the ordering against O-LoRA and InfLoRA is unchanged (20.83 and 20.34 against 21.40 and 21.42 over two seeds). Prefix-sequential GS matches COLORS in quality, but at substantially higher training cost. It retains all 16 rank-16 adapters (91.8M parameters vs 34.4M for COLORS) and must execute all preceding adapters for each task, causing training time to increase fivefold over the stream (Fig. 10b). In contrast, COLORS maintains nearly constant training cost and samples at base-model speed (Fig. 10c).

W2 & Q2. All-in-one fairness; isolating COLORS from the 2.5B backbone. The reviewer questioned whether COLORS' performance advantage comes from its pretrained 2.5B backbone rather than the continual learning method. To isolate this, we compare methods using the same SD3.5 backbone and training budget (40k steps per task, Tables 7, 8). On identical images, COLORS trained sequentially achieves 24.24 dB, matching SD3.5 trained jointly on all tasks (24.16 dB; SSIM 0.756 vs 0.759, LPIPS 0.176 vs 0.178; Table 8). In contrast, naive sequential finetuning of the same backbone falls to 6.98 dB (Table 2), with its first task dropping from 26.5 to 4.9 dB (Fig. 5b). Thus, the pretrained backbone alone cannot explain COLORS' retention or its performance under sequential learning.

Why did we retrain the all-in-one baselines instead of using their published checkpoints? We retrained the all-in-one methods on the same data to fairly compare their performance under identical tasks. Table 1 evaluates these methods under the easier setting of joint training on all tasks. Despite learning tasks sequentially, COLORS still outperforms four of the five methods (25.0 vs 23.6–24.4 dB mean PSNR). When the same architectures are trained sequentially (Fig. 5c, Fig. 12), all five forget earlier tasks and end at 6.3-6.7 dB after the 16-task stream. The revised Table 1 caption now states this.

Reviewer_yUhnOpenReview ↗

Follow up to Authors Response

Thank you very much for the additional experiments and the explanation.

With the additonal experiments and the explanation, I believe the paper main points and contibution are well conveyed now.

AuthorsOpenReview ↗

Follow-up to Reviewer yUhn's comment

We thank the reviewer for the positive follow-up. We are glad that the additional experiments and explanations have clarified the main contributions of COLORS and addressed the earlier concerns.

We appreciate the reviewer's time in evaluating the new evidence. Given that the reviewer now finds the main points and contributions well conveyed, we would be grateful if they would consider updating their original rating to reflect their revised assessment.

Reviewer_5byYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This draft focused on the continual learning with parameter-efficient fine-tuning (PEFT), especially CL with LoRA adapters. In this work, the authors considered to consolidate the shared knowledge during the incremental process. Specifically, the proposed method, named COLORS, decomposed each task adapter into two parts, a shared orthonormal basis and a task-specific residual. Then, new residuals are constrained to be orthogonal to the previously stored directions through a Gram-Schmidt process using rank-space factorization. After learning each task, an append-only consolidation mechanism promotes recurring residual directions into the shared basis while preserving previously learned adapters. The authors provide theoretical analysis on adapter preservation under storage truncation and the learnability of new tasks under an incoherence assumption. Experiments with multiple diffusion backbones on diverse tasks were conducted to demonstrate the competitive performance of the proposed method.

Critical strengths
  1. In this draft, the authors present a principled analysis for storage-efficient knowledge consolidation. The proposed append-only shared basis and task-specific residual representation provide a principled mechanism for reorganizing stored knowledge without substantially changing previous adapter weights. This is particularly valuable compared with shared-subspace methods that repeatedly recompute the basis and may introduce representation drift.
  2. This draft enables the efficient implementation of the rank-space factorization of Gram–Schmidt projection to avoid explicitly constructing dense weight updates.
  3. In the experimental part, the authors evaluated COLORS across multiple diffusion backbones and diverse task streams, including image restoration, stylization, video restoration, and pretrained adapter consolidation. Instead of classification tasks which was usually considered in previous work, the authors considered more diverse practical tasks.
Critical weaknesses
  1. Insufficient differentiation from existing LoRA-based continual learning methods. Several key ideas in this draft, including orthogonal subspace learning and shared/task-specific representations, have been explored in prior work. However, closely related methods such as CL-LoRA (CVPR 2025) and LoDA (ICML 2026) are not discussed.
  2. Insufficiently controlled comparisons with existing LoRA-based methods. Although this draft provides extensive experiments and computational efficiency analyses, the comparisons with existing LoRA-based continual learning methods are not fully controlled in terms of training budgets. This makes it difficult to isolate the performance and efficiency gains attributable to the proposed method.
Questions for authors
  1. How does COLORS differ from closely related methods such as CL-LoRA [1], LoDA [2], SplitLoRA [3], particularly regarding shared/task-specific subspace decomposition and knowledge consolidation? What are the key advantages of the proposed append-only consolidation mechanism?
  2. Equation (2) requires private residuals to be mutually orthogonal, whereas Appendix D.1 acknowledges that consolidation may cause them to lose mutual orthogonality. Could the authors clarify which orthogonality properties remain guaranteed? Additionally, how does COLORS perform when related tasks have substantial overlap in their optimal private updates?
  3. Although this draft provides computational efficiency analyses in Figure 10, the comparisons mainly focus on recursive Gram–Schmidt, prefix-sequential GS, and different inference configurations. Could the authors provide a more comprehensive comparison of training time, FLOPs, and peak GPU memory against existing LoRA-based continual learning methods, particularly O-LoRA and InfLoRA, under matched experimental settings? This would help clarify the computational overhead introduced by the proposed orthogonal projection and consolidation mechanisms.

References:

[1] CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning. CVPR 2025.

[2] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning. ICML 2026.

[3] SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting. ICLR 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5byY

W1 & Q1. How does COLORS differ from CL-LoRA, LoDA, and SplitLoRA? The reviewer noted that orthogonal learning and shared/task-specific representations already exist in prior work. We clarify that our main contribution is not orthogonality alone, but also append-only consolidation that shares knowledge without modifying previously stored adapters.

Existing methods repeatedly adjust how new tasks are learned: CL-LoRA continually updates a task-shared adapter through distillation and gradient reassignment, providing only approximate retention; SplitLoRA confines new-task updates to the minor gradient subspace of earlier tasks to reduce interference, without guaranteeing exact preservation of stored adapters; and LoDA, the closest to COLORS, recalibrates its general LoRA update before merging it into the backbone. None explicitly enforces append-only preservation of stored task adapters.

Unlike these prior methods, COLORS does not recalibrate or modify existing shared directions. Instead, it appends new atoms and transfers their components from stored residuals into shared coefficients, preserving every previously learned adapter (Sec. 2.3, Eq. 8, Prop. 1; worst numerical error 3.1e-6 across 14 runs, App. C). As a result, COLORS can reuse knowledge across tasks while preserving earlier solutions. Notably, appended atoms explain 25-99% of later residual energy, reducing the residual rank from 16 to 4 (Sec. 2.3). Unlike methods that recompute their shared basis, such as SHARE (up to 104% adapter drift, Fig. 5d), COLORS retains previously stored adapters up to numerical precision. We have added CL-LoRA, LoDA, and SplitLoRA to the related work (App. B) and clarify this distinction.

W2. Controlled budgets. As the reviewer pointed, our step counts (Table 4) and the baselines' (Table 7) are stated but not matched: on Sana we trained 25k steps per task and the continual baselines 15k. At a matched 15k budget (8 tasks, identical inputs), COLORS reaches 21.40 and 21.42 dB over two seeds, against 20.83 (O-LoRA), 20.34 (InfLoRA), 21.68 (SHARE) and 21.89 (independent adapters). The SD3.5 comparisons are already matched at 40k steps per task (Table 7, Table 8). We report the matched numbers in a new table (Table R3) and state the budget of every method in the appendix.

Q2. Which orthogonality properties remain guaranteed, and what if tasks overlap? COLORS maintains the two key orthogonality properties throughout the stream: shared atoms remain orthonormal, and every stored residual remains orthogonal to all shared atoms. Both hold to numerical precision across all runs: the largest deviation from atom orthonormality is 2.5e-8, and median residual-atom overlap is 7.5e-9 (verified after every stage, App. C). Importantly, these properties are sufficient for the relative storage-error bound in Prop. 1 (App. D.2) (zero-forgetting), while mutual orthogonality between residuals is not required.

Residuals are trained orthogonal to earlier ones, but later atom appends can introduce small pairwise overlaps (App. D.1). By Eq. 8, each append changes their inner product by −γi,c+1γj,c+1-\gamma_{i,c+1}\gamma_{j,c+1}. Empirically, this overlap remains negligible (median 9.8e-10 after all appends). The revision clarifies this right after Eq. 2, consistent with App. D.1.

Q2 (second part): what if tasks overlap? For highly overlapping tasks, Theorem B characterizes the potential capacity loss, while our pre-training probe identifies such cases before training. Deliberately duplicated tasks score 0.74-0.78 on the probe (Sec. 3.2), above all 120 natural task pairs (maximum 0.60; Table 13). Of the three flagged tasks, two are absorbed using only one new shared atom, while one loses 44% of its independently trained gain. We discuss this limitation in Sec. 5.

Q3. Training time, FLOPs and peak memory against O-LoRA and InfLoRA. We benchmarked all methods on the same A100, backbone, batch and resolution, using their actual configurations after 15 stored tasks.

MethodTrain s/stepPeak memTFLOPs/stepSampling s/step
Independent0.114.0 GB2.400.015
COLORS0.2610.8 GB3.020.015
O-LoRA0.4413.8 GB2.540.103
InfLoRA0.4414.9 GB2.530.103

COLORS trains 1.7x faster per step than O-LoRA and InfLoRA (0.26 vs 0.44 s/step) with less peak memory (10.8 vs 13.8 and 14.9 GB), and samples 6.8x faster than their all-adapters-active configurations. The projection costs about 20% more FLOPs per step (3.02 vs 2.54 TFLOPs). This complements Fig. 10: our projection stays at 3.2-3.7 ms per layer from 2 to 32 stored tasks (10a), training time per stage stays flat while prefix-sequential GS slows down fivefold (10b), and running all 16 adapters at inference takes 5.08 s vs 0.94 s merged on SD3.5 (10c).

Reviewer_5byYOpenReview ↗

Follow-up After Author Response

I appreciate the authors for their response and the additional experiments. The rebuttal has clarified several important points:

  • Novelty and differentiation from prior work. The authors have clarified that the main contribution of COLORS lies in its append-only consolidation mechanism, rather than orthogonal subspace learning alone. The distinction from CL-LoRA, LoDA, and SplitLoRA is now clearer, and I appreciate the inclusion of these methods in the related work.

  • Controlled experimental comparisons. The additional experiments under matched training budgets help address my concern about the fairness of the original comparisons. The results provide stronger evidence for the effectiveness of COLORS relative to O-LoRA and InfLoRA.

  • Orthogonality properties. The clarification regarding which orthogonality properties are preserved throughout consolidation is helpful. In particular, distinguishing the preservation of shared-atom orthonormality and residual-to-shared-basis orthogonality from the potentially imperfect mutual orthogonality of residuals resolves much of my concern about the consistency between the formulation and implementation.

  • Computational efficiency. The new comparisons of training time, FLOPs, and peak GPU memory against O-LoRA and InfLoRA are valuable and provide a more comprehensive understanding of the computational trade-offs.

Nevertheless, I have two remaining observations:

  • Positioning of the contribution. While the append-only consolidation mechanism provides a meaningful distinction from prior methods, I encourage the authors to emphasize that the primary contribution lies in storage-efficient knowledge consolidation with preservation of previously learned adapters, rather than orthogonality or zero forgetting alone.

  • Performance-efficiency trade-offs. The matched-budget experiments show that COLORS does not consistently outperform independent adapters or SHARE in reconstruction quality. Furthermore, the orthogonality ablation suggests that Gram-Schmidt primarily contributes to subspace separation and theoretical guarantees, rather than directly improving reconstruction quality. I encourage the authors to make these trade-offs explicit and avoid overly broad claims of superiority.

Overall, I find the rebuttal constructive and believe that the authors have adequately addressed most of my original concerns. The remaining issues are primarily related to the positioning and interpretation of the contributions rather than fundamental flaws in the proposed method.

AuthorsOpenReview ↗

Follow-up to Reviewer 5byY's comment

We thank the reviewer for the encouraging follow-up and for acknowledging that the additional experiments have addressed most of the original concerns.

We appreciate the suggestions on positioning and performance-efficiency trade-offs, which we have already incorporated into the revised manuscript. We have emphasized that the core contribution of COLORS is storage-efficient, append-only knowledge consolidation with preservation of previously learned adapters.

Independent adapters provide a strong reference for reconstruction quality because each task is trained independently, without constraints from previously learned tasks. Achieving comparable quality while also consolidating knowledge across tasks is therefore a meaningful result for a continual learning method. We have discussed this explicitly in Sec. 4.2 of the revised manuscript.

While independent adapters and SHARE achieve competitive performance in some settings, their trade-offs differ: full independent adapters require 3.5x more storage on SD3.5, while SHARE can introduce substantial drift in previously stored adapters (Fig. 5d, Table 3). COLORS, in contrast, combines competitive performance with compact storage, knowledge sharing, and exact preservation of earlier task updates.

We appreciate the reviewer recognizing that the remaining concerns are primarily about positioning rather than fundamental limitations, and would be grateful if the reviewer would consider revisiting their assessment.

Comments on the paper

1 comment

Dear Reviewers,

Could you take a moment to review the authors' responses and let them know whether any concerns remain or require further clarification?

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.