- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents COLORS, a LoRA-based method for learning image-to-image tasks sequentially on a frozen diffusion backbone. Each task is stored as coefficients over a shared set of rank-one atoms plus a private low-rank residual. The residual is trained to be orthogonal to the stored atoms and residuals through a cheap Gram–Schmidt projection. After each task, a direction that recurs across residuals can be promoted to a new shared atom. The authors claim exact zero forgetting, a learnability bound under task incoherence, and strong results on restoration, stylization and video across six backbones.
Critical strengths
The append-only basis guarantees that stored adapters never change as the basis grows. This improves on methods that recompute a PCA basis and silently alter old tasks. The rank-space Gram–Schmidt trick keeps training cost flat as tasks accumulate. The experiments cover many backbones and task types, and the consolidation of public FLUX adapters is convincing and reproducible. As a way to store many task adapters compactly, the method is practical and well engineered.
Critical weaknesses
W1: The comparison with O-LoRA and InfLoRA in Table 2 is unfair. The introduction criticizes both methods (Sec. 1), but they are class-incremental methods: they have no task ID at inference, so all adapters stay active for every input. COLORS is evaluated as task-incremental learning. At inference it uses the task ID and a task-specific prompt to load only the matching coefficients and residual (Table 2 caption, Table 7). The independent-adapters baseline, which keeps one separate LoRA per task, reaches almost the same quality as COLORS (Tables 2 and 9). The large gap therefore comes from the task ID, not from the proposed method.
W2: The paper is positioned as continual learning, but the method works as multi-adapter compression. At inference, each task loads only its own scaling of the shared part and its own residual. This is equivalent to keeping one LoRA per task, as in Share, and the authors confirm that the task must be specified at inference (Sec. 5). Zero forgetting comes from storing each adapter exactly (Prop. 1, App. D.2), not from orthogonality, and removing Gram–Schmidt barely changes the results (Table 12). The method therefore does not address forgetting when the task is unknown at inference.
W3: The orthogonality that the theory relies on is not maintained. The shared part is only a task-specific diagonal scaling of a shared U and V (Eq. 3). The main text requires each residual to be orthogonal to all atoms and to all earlier residuals (Eq. 2). However, adding a new atom subtracts its component from every stored residual (Eq. 8), so residuals that were orthogonal start to overlap. The appendix confirms that residuals overlap and that orthogonality to the atoms is only approximate (App. D.1). The theoretical basis set up in the main text therefore does not hold.
W4: Atom selection does not optimize its own criterion. Eq. 7 scores a candidate atom by squared projections of the residuals, which is a second-order, PCA-style objective. The candidate itself, however, is the top singular pair of the arithmetic mean of the residuals (Sec. 2.3), which is a first-order statistic. The mean cancels components of opposite sign and can be dominated by one large residual, so it can miss shared directions that Eq. 7 would favor. In addition, if the residuals were orthogonal as Eq. 2 claims, little shared energy would exist to consolidate.
Questions for authors
Q1: Could the authors report O-LoRA and InfLoRA with access to the task ID, and COLORS without the task ID and with all adapters active?
Q2: Do the residuals remain orthogonal after new atoms are added, and do stored tasks drift when Gram–Schmidt is removed?
Q3: Could the authors compare COLORS with independent LoRAs and with existing LoRA compression methods at the same parameter count?
Q4: Could the authors explain why Eq. 7 uses squared projections while the candidate atom comes from the first-order mean, and whether a second-order extraction would select different atoms?
3 comments
Thanks to the authors for their response. W1 changes the inference Settings of O-LoRA and infLoRA, and it seems that the authors confuse the difference between "Task-aware" and "Ground-truth Task-ID". Furthermore, W3/W4 seems to confirm my concerns that while the overlap is down to a negligible magnitude, this does not translate into performance gains in the end. Therefore, I tend to keep my score.
Follow-up to Reviewer Wske's comment
We thank the reviewer for the follow-up. We would like to clarify two points that may not have been sufficiently clear in our previous response.
Task-ID comparison. Our additional experiment deliberately provides the ground-truth task ID to O-LoRA and InfLoRA. The purpose is to test whether their lower performance in the original evaluation is caused by keeping all adapters active, including those learned after the evaluated task.
For a test image belonging to task k, we activate adapters 1 through k, matching the configuration under which that task was trained. This is an additional controlled comparison, not a replacement for the original all-adapters-active evaluation in Table 2. We do not claim that this setting evaluates the baselines' ability to identify tasks without supervision.
Even with access to the correct task identity, O-LoRA and InfLoRA achieve 19.89 and 19.74 dB, compared with 21.30 dB for COLORS. Thus, the performance difference cannot be attributed solely to the absence of task identity. The matched-budget experiments further support this finding.
Role of orthogonality. Gram-Schmidt is important for maintaining the consolidated representation. It prevents shared and private components from redundantly representing the same directions.
The ablation shows that residual-to-shared overlap decreases from 0.37 to 7.5e-9, while the fraction of task energy represented by the shared basis increases from 54% to 79%, with nearly identical PSNR (20.32 vs. 20.34 dB).. We note that this separation also supports the relative storage-error guarantee in Proposition 1.
Atom selection. We note that the performance of COLORS is on par with independent adapters. Atom selection is primarily intended to influence consolidation, as described in Sec.2.3. Moreover, directly optimizing Eq. 7 through alternating maximization produces candidates dominated by a single task, whereas our mean-based candidate draws contributions from multiple tasks. The results in Appendix E.9 (Table R6) further support our choice of candidates for consolidating reusable knowledge rather than selecting directions that maximize the immediate energy score.
Together, orthogonal decomposition and append-only consolidation enable COLORS to share knowledge across tasks while maintaining competitive performance, reducing storage, and preserving previously learned adapters without drift.
We hope these clarifications address the reviewer's remaining concerns.
Response to Reviewer Wske
W1 & Q1. Task identity and baseline fairness. We address both concerns through identical task-aware comparisons (General Response, Sec. 1). COLORS outperforms O-LoRA and InfLoRA, even when they are given task-ID access, confirming that task identity alone does not explain the performance gap.
W2. Continual learning or multi-adapter compression? COLORS is a continual learning method, not merely post-hoc adapter compression: tasks arrive sequentially without revisiting earlier data, and the shared basis is built and expanded online. Unlike independent adapter storage, later tasks actively reuse previously learned knowledge. In our experiments, appended atoms explain 25-99% of later residual energy, reducing the residual rank from 16 to 4 (Sec. 2.3).
How is zero forgetting guaranteed? We agree that zero-forgetting follows from append-only consolidation, which preserves previously learned adapters without modifying their reconstructed updates. However, orthogonality is what separates shared and private directions and establishes the relative storage-error bound in Proposition 1 (App. D.2; General Response, Sec. 2). Unlike SHARE, which also consolidates task adapters but introduces up to 104% drift, COLORS preserves stored adapters to numerical precision (Fig. 5d, Table 3). The revised abstract and Sec. 2 now make these roles explicit.
Q2. Do residuals stay orthogonal after appends? Do tasks drift without GS? Residuals remain orthogonal to shared atoms, while their mutual overlap remains minimal (median absolute cosine 9.8e-10) (App. D.1; General Response, Sec. 2). However, this does not affect zero forgetting, since append-only consolidation preserves each stored adapter exactly (Eq. 8). Even without Gram-Schmidt, our 5k-step run preserved stored adapters with error 5.8e-8. However, without GS, the relative storage-error bound of Prop. 1 is no longer guaranteed because residual-to-atom orthogonality may fail (App. D.2).
GS also ensures clean shared/private separation: Median private residual to shared atom overlap drops from 0.37 without GS to 7.5e-9 with GS. A separate ablation in Table 12 also shows substantial overlap without GS (0.77). Importantly, this improved separation comes at no measurable cost to restoration quality, with nearly identical PSNR (20.32 with GS vs 20.34 dB without GS on identical inputs), consistent with Theorem A under its assumptions.
W3. Orthogonality after appends. As clarified in the General Response (Sec. 2) and Q2 above, the orthogonality required by Prop. 1 holds up to numerical precision (App. C). Appending an atom can change the inner product between residuals by −γi,c+1γj,c+1 (Eq. 8), but the measured overlap remains negligible (median abs. cosine 9.8e-10). Theorem A assumes residual incoherence rather than exact mutual orthogonality (App. D.1). The revision states this after Eq. 2.
W4 & Q4. Choosing mean candidate vs squared-energy score (Eq. 7). We use the mean to identify shared directions because directly optimizing Eq. 7 instead favors task-specific directions. We verified this on the final 16-task Sana model. In all 80 layers, the candidate obtained by alternating maximization drew 100% of its score from a single task (median score of 0.197, close to the theoretical upper bound of 0.215) (Bessel inequality). In contrast, the mean-based candidate draws on approximately three tasks per layer, with the largest task contributing a median of 50% of the Eq. 7 score.
Why does direct optimization favor task-specific directions? Gram-Schmidt trains each residual to be orthogonal to earlier ones (Eq. 2), with only small overlaps after appends (W3). For near-orthogonal residuals, Bessel's inequality (which bounds the total squared projection onto orthogonal directions) caps the Eq. 7 score near the largest residual's energy share, so direct optimization favors private directions. Opposite signs (+A, -A) can cancel in the mean but also imply residual overlap, which GS suppresses. Thus, the mean proposes atoms and Eq. 7 accepts or rejects them (Sec. 2.3). An atom's value lies in reuse by later tasks, not overlap between stored residuals (W2).
Does COLORS outperform independent adapters and SHARE at matched storage budgets? COLORS matches compressed independent adapters in quality while enabling mixed-task composition. Rank-6 independent adapters match COLORS' 34.4M parameters (retaining at least 99.98% of their energy) and score 21.19 vs 21.30 dB. On mixed inputs, COLORS gives the highest composed PSNR of all methods and beats the hindsight-best single task (+0.12 dB), while summed independent adapters fall below it (-0.19 dB; table in reply to yUhn).
Against SHARE at matched storage, COLORS has lower error at every budget on 16 public FLUX adapters (22.7% vs 34.1% at 0.34G; 2.2% vs 9.5% at 1.5G; Tables 3, 14), with zero drift versus SHARE's 13-52%.