acceptodds
Withdrawn from ICLR 2027

ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study continual image editing, which aims to progressively expand the capability boundary of diffusion models while preserving previously acquired editing skills. In this paper, we propose ACE-LoRA, an adaptive regularization framework that captures state-dependent task interference through an online interference vector. To enable scalable continual learning, we introduce a rank-invariant historical information compression strategy that consolidates accumulated LoRA updates into a fixed-rank representation while preserving principal task-relevant subspaces. Moreover, we present CIE-Bench, a benchmark comprising six practical image editing tasks and a domain-aware evaluation protocol. Experiments show that ACE-LoRA not only outperforms existing baselines in instruction fidelity and visual realism, but also demonstrates the strongest robustness over long task sequences.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, image editing, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V3EpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies continual image editing, where a diffusion-based editing model sequentially learns new editing skills while retaining earlier ones. The proposed ACE-LoRA penalizes alignment between the current LoRA-B and an online interference vector, defined as the gradient of the current-task loss with respect to historical LoRA-B. It compresses historical LoRAs into a fixed-rank pair via truncated SVD and adopts a two-stage schedule that first trains without constraints and then refines with orthogonality and norm regularizers. The authors also introduce CIE-Bench, which has six editing tasks evaluated by a task-specific MLLM judge, and report gains over merging, architecture-based and regularization-based baselines.

Critical strengths
  • Timely problem and a useful benchmark. Continual image editing is largely unexplored in continual learning. CIE-Bench spans physical perception, entity manipulation and cognitive reasoning, with task-specific evaluation prompts and a human-agreement study, making it a potentially reusable testbed.
Critical weaknesses
  • The core mechanism is conceptually disconnected from forgetting. Forgetting means an increase in old-task loss, yet Eq. (7) measures the sensitivity of the current-task loss to historical parameters, and no old-task loss or gradient enters the procedure. Moreover, the penalty is a single scalar per sample summed over the rank dimension, so it can vanish even when individual columns are aligned with the interference vector.
  • The experiments have several issues. Hyperparameters and task order appear to be selected on the test set, as the defaults are the best entries in Tab. 11/12 and no validation split is described. λo is further tuned per task. Key ablations such as λo = 0 and λn = 0 are missing. The introduction claims to preserve the base model's original abilities, but there is no capability-retention evaluation on general editing benchmarks such as GEdit or ImgEdit.
  • Reported results do not match the stated definition. The paper describes BWT as the average degree of forgetting. Computing BWT = mean(Last) − mean(Imd.) on Tab. 5 exactly reproduces the reported values for Random and Summation, but gives −0.3369 for ACE-LoRA instead of the reported −0.1702. This raises concerns about the reliability of the reported results.
Questions for authors
  • Please give the precise definition of BWT. Under mean(Last) − mean(Imd.), ACE-LoRA should be −0.3369, while Random, Summation and the Static row in Tab. 1 are all exactly reproduced by this formula. How was −0.1702 obtained?
  • Please report ablations with λo = 0 and λn = 0 to separate the contributions of the interference vector, the norm term and the training schedule.
  • Why does orthogonality to an interference vector derived from the current-task loss protect old tasks? Eq. (8) effectively constrains ⟨∇_W ℓ_n, B_n A_i⟩, and Eq. (10) constrains ⟨∇_W ℓ_n, B_n A_his⟩, rather than the actual update B_n A_n. Why does the derivation use the squared form while the implementation uses the absolute value?
  • How were the hyperparameters and task order selected, and was a held-out validation split used?
  • How well is the base model's general editing ability on GEdit or ImgEdit retained after the full task sequence?
Reviewer_HP7DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies rehearsal-free continual image editing and proposes ACE-LoRA. The method constructs an online orthogonality regularizer using gradients of the current-task loss with respect to historical LoRA parameters. It combines this regularizer with SVD-based compression of historical updates and a two-stage fine-tuning schedule. The paper also introduces CIE-Bench, covering six editing tasks, and reports improvements over continual learning baselines on two image-editing backbones.

Critical strengths
  1. Relevant problem and useful evaluation setting. Continually extending an image-editing model without revisiting historical training data is practically relevant. CIE-Bench covers complementary capabilities, including refocusing, relighting, text editing, and virtual try-on.

  2. Broad empirical support for the overall framework. The reported improvements are consistent across Flux2-Klein-9B and Qwen-Image-Edit-2509. Evaluation on additional public datasets, an alternative MLLM evaluator, and human assessments strengthens the evidence beyond the proposed benchmark and its primary evaluator.

Critical weaknesses
  1. The methodological contribution and its connection to the proposed setting are not sufficiently established. The framework combines orthogonality regularization, SVD compression, and staged fine-tuning, with the online gradient signal being its main distinguishing component. Its connection to image editing is primarily through the application and evaluation: the regularizer does not explicitly exploit editing-specific structure, while the timestep-based motivation applies to diffusion models more generally. Broad applicability is not itself a weakness, but the paper provides limited direct evidence that timestep-dependent interference explains the observed improvement. More fundamentally, Eqs. (7) and (10) compute gradients of the current-task loss on historical parameters. This measures sensitivity to the new objective and does not directly establish which directions would damage previous tasks. A diagnostic connecting the proposed signal to actual historical-task degradation would substantially strengthen the central argument.

  2. The independent contribution of the regularizer and the claimed reduction in forgetting remain insufficiently isolated. Table 1 supports adaptive over static constraints, but the paper lacks a comparison that removes the orthogonality term while retaining the same incremental LoRA architecture, frozen historical parameters, two-stage schedule, norm regularization, and training budget. This matters because Sequential FT updates a single LoRA, whereas ACE-LoRA adds task-specific adapters, and the two-stage schedule itself has a substantial effect. The approximately doubled training cost also makes budget matching relevant. Additionally, the BWT values reported for ACE-LoRA in Tables 1 and 5 appear inconsistent with the displayed immediate and final scores, introducing uncertainty into the claim of superior forgetting mitigation.

Overall assessment: The application is worthwhile, and the empirical results are encouraging. However, the current evidence does not sufficiently establish the independent contribution and interpretation of the proposed online constraint beyond the surrounding continual LoRA framework. The missing controlled ablation and the unresolved BWT discrepancy also limit confidence in the explanation of the gains. I would be open to revising my assessment if these points are convincingly addressed.

Questions for authors
  1. What evidence shows that the current-task gradients in Eqs. (7) and (10) reliably identify directions that interfere with historical tasks? Can the authors provide a diagnostic relating this signal to measured historical-task degradation, including its dependence on diffusion timesteps?

  2. How does ACE-LoRA perform with the orthogonality weight set to zero, while keeping the incremental adapters, frozen history, two-stage schedule, norm regularization, and training budget unchanged? Please also clarify how training budgets are matched across baselines.

  3. Please clarify the BWT definition and the ACE-LoRA entries in Tables 1 and 5. The reported mean scores give 8.8685−9.2054=−0.33698.8685-9.2054=-0.3369, whereas the reported BWT is −0.1702-0.1702. Other rows appear consistent with the difference between these means. How should this discrepancy be resolved, and does it affect the conclusion regarding forgetting?

Reviewer_mBNGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACE-LoRA, an adaptive regularization framework for continual image editing with diffusion models. The method addresses catastrophic forgetting by constructing state-dependent interference vectors from gradients of historical LoRA parameters under current-task data and enforcing orthogonality with new LoRA updates. To improve scalability, the authors introduce SVD-based historical information compression and a two-stage fine-tuning strategy that balances plasticity and stability. The paper also introduces CIE-Bench, a benchmark covering six image editing tasks. Experiments on two diffusion backbones demonstrate improvements over existing continual learning baselines.

Critical strengths

1. Relevant problem and adaptive interference modeling.\textbf{1. Relevant problem and adaptive interference modeling.} The paper addresses catastrophic forgetting in continual image editing, an important but relatively underexplored problem. Modeling task interference dynamically based on training states is a reasonable extension of conventional static orthogonality constraints, particularly given the timestep-dependent behavior of diffusion models.

2. Practical framework with scalability considerations.\textbf{2. Practical framework with scalability considerations.} The combination of adaptive orthogonalization, SVD-based historical compression, and two-stage fine-tuning provides a coherent framework for balancing plasticity and stability. The compression strategy is particularly useful for reducing the computational overhead associated with accumulated historical adapters.

3. Comprehensive experimental evaluation.\textbf{3. Comprehensive experimental evaluation.} The paper evaluates ACE-LoRA on two diffusion backbones and compares it with diverse continual learning baselines. The introduction of CIE-Bench, together with ablation studies and additional evaluations, provides useful empirical evidence for the effectiveness of the proposed framework.

Critical weaknesses

1. Insufficient justification of the interference vector.\textbf{1. Insufficient justification of the interference vector.} Equation (7) defines the interference vector using gradients of historical LoRA parameters with respect to the current-task loss. However, it is unclear why sensitivity to the current task necessarily reflects interference with previously learned tasks. The paper does not directly establish the relationship between the proposed interference signal and actual forgetting, leaving the central design choice insufficiently justified.

2. Gap between theoretical motivation and practical objective.\textbf{2. Gap between theoretical motivation and practical objective.} Equations (4)–(6) motivate adaptive orthogonalization using squared inner products and a covariance decomposition, whereas the actual regularization loss in Equations (8) and (10) uses absolute inner products. The theoretical analysis therefore does not directly characterize the implemented objective. Furthermore, constraining LoRA-B alone may introduce sensitivity to the non-unique factorization of LoRA updates, which is not discussed.

3. Limited validation of timestep-dependent interference.\textbf{3. Limited validation of timestep-dependent interference.} Although the primary motivation is that interference varies across diffusion timesteps, the experiments mainly compare adaptive and static orthogonalization. Since the proposed interference vector also depends on data samples and noise, these results do not isolate the contribution of timestep-dependent modeling. A more direct analysis of interference directions across timesteps would strengthen the central motivation.

4. Insufficient analysis of compression and computational efficiency.\textbf{4. Insufficient analysis of compression and computational efficiency.} The SVD-based compression is evaluated through reconstruction similarity and comparisons with alternative compression strategies. However, the paper does not directly compare its performance against full historical orthogonalization without compression. Moreover, the scalability experiments are limited to relatively short task sequences, while the proposed method requires approximately twice the training time of sequential fine-tuning.

5. Limited benchmark scale and evaluation reliability.\textbf{5. Limited benchmark scale and evaluation reliability.} CIE-Bench contains relatively few training samples per task, and some data are generated using proprietary editing models. In addition, the evaluation relies heavily on MLLM-based judgments, while the human study is limited in scale. These factors raise questions about the generalizability and reliability of the reported improvements.

6. Inconsistent and unclear reference formatting in Tables 2 and 3.\textbf{6. Inconsistent and unclear reference formatting in Tables 2 and 3.} The baseline references in Tables 2 and 3 are not formatted according to standard citation conventions. For example, entries such as "TIES Yadav et al. (2023)" and "RegLoRA Chen et al. (2025)" directly concatenate method names and citations without clear separation. In addition, the same references are repeated in both the Avg. and Last sections, making the tables unnecessarily cluttered. A consistent citation format, such as "TIES (Yadav et al., 2023)," would improve readability and presentation.

Questions for authors

1. Interference vector justification:\textbf{1. Interference vector justification:} Could the authors clarify why gradients of historical parameters under the current-task loss provide a reliable measure of interference with previous tasks? Would it be possible to analyze the correlation between the proposed interference signal and actual forgetting on historical tasks?

2. Orthogonality objective:\textbf{2. Orthogonality objective:} Why is the absolute inner product used in Equations (8) and (10), despite the squared formulation used in the theoretical analysis? Could the authors compare these alternatives and discuss whether the loss is sensitive to different but equivalent LoRA factorizations?

3. Timestep-dependent interference:\textbf{3. Timestep-dependent interference:} Could the authors provide a more direct analysis of how interference directions vary across diffusion timesteps? For example, comparing timestep-specific, timestep-averaged, and timestep-independent constraints would help isolate the benefit of state-dependent modeling.

4. Compression and scalability:\textbf{4. Compression and scalability:} How does SVD-based compression compare with full historical orthogonalization in terms of forgetting, computational cost, and interference estimation accuracy? Could the authors also evaluate longer sequences of distinct tasks and report performance under matched computational budgets?

5. Benchmark reliability:\textbf{5. Benchmark reliability:} Could the authors provide additional evidence for the reliability of CIE-Bench, such as evaluations with more human annotators or multiple random seeds? It would also be helpful to clarify whether the synthetic data generation and evaluation procedures will be fully reproducible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.