acceptodds
Under review as a conference paper at ICLR 2027

DCHA: Dynamic Compositional Adaptation with Hierarchical Drift Constraints for Incremental Vision-Language Object Detection

Abstract

Incremental vision-language object detection (IVLOD) aims to continuously adapt vision-language detectors to emerging domains while preserving previously acquired knowledge and open-vocabulary generalization. While current methods typically optimize LoRA experts to achieve efficient adaptation, these training paradigms exhibit poor plasticity, as empirically evidenced by highly homogeneous parameter update directions across different categories. This rigidity stems from sharing restricted parameter spaces. Specifically, task-level LoRA methods force all categories to share unified experts, while existing category-level LoRA methods rigidly assign experts based solely on labels, completely ignoring intra-class feature variance and imposing a fixed routing capacity. To break this rigidity, we propose category-adaptive composition (CAC) equipped with learnable category-adaptive routing (LCAR). Inspired by Sparsegen, LCAR utilizes learnable keys and thresholds to dynamically select and assemble the most matched fine-grained parameter vectors into exclusive category LoRA experts. This demand-aware routing ensures that update directions are highly tailored to category-specific feature requirements, thereby significantly enhancing parameter plasticity and reducing training interference. Furthermore, because CAC introduces a hierarchical parameter structure where vectors compose category experts and category experts aggregate into task experts, parameter deviations inevitably accumulate across levels. To mitigate this severe error accumulation, we introduce a hierarchical drift constraint (HDC) that performs layer-wise regularization. Extensive experiments on ODinW-13 and ODinW-O demonstrate that our framework consistently outperforms state-of-the-art methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision-language object detection, continual learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eeDCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses Incremental Vision-Language Object Detection (IVLOD), where a pre-trained VLOD model must sequentially adapt to new domains while preserving prior knowledge and zero-shot generalization. The authors identify that existing LoRA-based IVLOD methods suffer from a plasticity bottleneck: parameter updates across categories are highly homogeneous due to rigid expert assignment and fixed routing capacity. They propose DCHA, which comprises: (1) Category-Adaptive Composition (CAC) that dynamically assembles category-specific LoRA experts from a shared vector bank; (2) Learnable Category-Adaptive Routing (LCAR) using Sparsegen-inspired demand-aware selection with per-category sparsity thresholds; and (3) Hierarchical Drift Constraint (HDC) that regularizes at vector, category, and task levels to prevent compositional error accumulation. Experiments on ODinW-13 and ODinW-O show DCHA outperforms SOTA methods with only 20.8M parameters (vs. DitHub's 147M) and near-zero forgetting.

Critical strengths
  1. A well-motivated problem-solution pipeline with clean ablation evidence. The paper presents a tightly integrated logical flow from problem identification to method design, forming a cohesive and well-motivated narrative. Moreover, the strategy of extracting shared category representations within the continual learning paradigm is both reasonable and well-justified.
  2. Compared to prior methods, the proposed approach is significantly more parameter-efficient, achieving reduced catastrophic forgetting with substantially fewer adaptation parameters.
  3. The experimental evaluation is relatively comprehensive and well-supported.
Critical weaknesses

The methodology section leaves room for improvement. The dense concentration of mathematical notation throughout this section hampers readability. Additionally, the paper could benefit from a more detailed exposition of how category-level low-level features are extracted and leveraged within the GroundingDINO backbone, as this aspect remains insufficiently clarified in the current manuscript.

Questions for authors

Can DCHA be adapted for use with LLMs such as Qwen?

1 comment

W1. We will revise the methodology section to improve readability. We also clarify what “category-level low-level features” means in DCHA. Here, “low-level” refers to trainable sub-vectors in a shared LoRA bank. The three levels in DCHA are the bank sub-vectors, the category-specific LoRA updates, and their task-level aggregate. LCAR uses learned sparse coefficients indexed by the prompted categories to compose category-specific updates. DCHA aggregates these updates and applies the result to GroundingDINO’s vision–language fusion projections, adapting how visual and textual features interact for downstream tasks.

Q1. We evaluate DCHA on two Qwen3.5 variants, Qwen3.5-2B and Qwen3.5-9B. The table below reports their zero-shot object detection AP on COCO val2017 (ZCOCO).

ModelZCOCO
Qwen3.5-2B1.0
Qwen3.5-9B1.7

Both variants score substantially below GroundingDINO under our COCO evaluation protocol. We therefore use Avg, the mean AP50–95 across the 13 ODinW-13 datasets, as the primary metric for downstream adaptation. The following tables report AP immediately after learning each task and after completing the full task sequence. The final results also include mean forgetting (Fˉ\bar{F}).

Qwen3.5-2B — AP at acquisition

VariantAvgAeAqCoEgMuPaPvPiPoRaShThVe
Zero-shot13.410.17.632.09.42.033.01.317.13.140.21.712.24.9
+CAC23.326.219.930.053.33.916.114.132.34.355.114.718.514.1
w/o HDC25.628.122.632.455.46.118.716.335.06.857.116.721.216.0
Full29.429.231.139.456.617.019.218.441.75.457.019.727.819.9

Qwen3.5-2B — Final AP and forgetting

VariantAvgFˉ↓\bar{F}\downarrowAeAqCoEgMuPaPvPiPoRaShThVe
Zero-shot13.4—10.17.632.09.42.033.01.317.13.140.21.712.24.9
+CAC17.76.018.715.422.939.73.211.411.324.43.239.914.714.711.0
w/o HDC19.36.819.817.324.540.74.913.013.026.14.940.816.716.712.4
Full28.21.327.729.937.854.116.418.217.839.95.154.319.726.819.1

Qwen3.5-9B — AP at acquisition

VariantAvgAeAqCoEgMuPaPvPiPoRaShThVe
Zero-shot31.012.311.266.748.78.045.78.351.412.558.413.445.121.4
+CAC43.427.534.564.866.021.659.021.663.412.072.126.657.038.3
w/o HDC45.629.636.666.868.623.961.024.265.614.074.129.359.040.5
Full50.034.539.161.168.826.459.945.064.323.974.542.960.448.7

Qwen3.5-9B — Final AP and forgetting

VariantAvgFˉ↓\bar{F}\downarrowAeAqCoEgMuPaPvPiPoRaShThVe
Zero-shot31.0—12.311.266.748.78.045.78.351.412.558.413.445.121.4
+CAC38.25.723.430.556.957.319.550.019.455.310.461.826.650.934.0
w/o HDC39.66.524.931.957.958.721.450.921.556.511.962.629.352.135.6
Full49.10.933.838.560.167.526.058.644.463.123.473.042.959.547.9

On ODinW-13, full DCHA improves final mean AP over the zero-shot baselines by 14.8 points on Qwen3.5-2B and 18.1 points on Qwen3.5-9B, reaching 28.2 and 49.1, respectively. CAC alone reaches mean acquisition AP of 23.3 and 43.4 on 2B and 9B, respectively, compared with their zero-shot AP of 13.4 and 31.0. Adding LCAR raises acquisition AP further to 25.6 and 45.6. Adding HDC brings acquisition AP to 29.4 and 50.0 and reduces mean forgetting from 6.8 to 1.3 points on 2B and from 6.5 to 0.9 points on 9B. These trends are consistent with the component-wise results for DCHA on GroundingDINO.

Reviewer_L6RtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies incremental vision-language object detection (IVLOD), aiming to adapt a pretrained Grounding DINO detector to sequentially arriving domains while preserving prior task performance and zero-shot detection. It proposes category-adaptive composition (CAC), which constructs category-specific LoRA matrices from a shared bank of fine-grained parameter vectors; learnable category-adaptive routing (LCAR), which uses Sparsegen with category-dependent learned sparsity to choose a variable number of vectors; and a hierarchical drift constraint (HDC), which regularizes changes at the bank-vector, category-update, and task-update levels. Experiments on ODinW-13 and the overlapping-category ODinW-O benchmark compare with prior incremental detectors and ablate the routing and drift components. On full-data ODinW-13, the method reports 62.3 average mAP, 47.8 COCO zero-shot mAP, and a harmonic mean of 54.1, while using 20.8M adaptation parameters versus 147M for the reported rank-16 DitHub variant.

Critical strengths
  1. Parameter-efficient compositional adaptation. The shared vector bank and block-wise construction provide a concrete way to reuse adaptation parameters across categories rather than storing a full category-specific LoRA for each class. The reported reduction from 147M to 20.8M adaptation parameters at rank 16 is substantial, while the full-data ODinW-13 average mAP is approximately maintained (62.3 versus 62.2 for DitHub). This is a practically meaningful accuracy–storage trade-off.

  2. Relevant task setting and component analyses. The submission studies both heterogeneous incremental domains (ODinW-13) and repeated categories across domains (ODinW-O), includes few-shot and full-data settings, and examines bank size, block size, sparsity parameterization, routing choices, and the HDC terms. These experiments help reveal the behavior and costs of the proposed parameterization, notwithstanding the concerns below.

Critical weaknesses
  1. Key empirical claims are difficult to reconcile with the stated benchmark and metric definitions. In Table 2, DitHub is reported at 59.2 average mAP on ODinW-O, whereas the official DitHub results report 62.38 on a benchmark with the same name (https://aimagelab.github.io/DitHub/). Since DCHA reports 60.2, the claimed advantage depends on explaining this discrepancy. The reported Grounding DINO and ZiRa values are close to those on the DitHub project page, making a clear protocol explanation particularly important. This discrepancy could change the relative ranking of the methods. Separately, Eq. (15) defines average forgetting using the maximum historical score minus the final score for each earlier task. Table 16 nevertheless attributes the reported value of 0.0167 to gains on some tasks offsetting declines on others, including Aquarium (−1.9), Pothole (−4.4), and Raccoon (−2.6). Such cross-task cancellation is inconsistent with Eq. (15). Using the rounded entries in Table 16, the sum of positive After−Before differences is about 10.8 mAP points, against about 11.0 points of declines. Even excluding the largest decline to allow conservatively for the final task, the remaining declines imply a lower bound of approximately 0.55 mAP, assuming that “Before” denotes performance immediately after learning each task, as stated in the appendix. In contrast, the signed sum of all After−Before differences, divided by 12, is approximately 0.0167, matching the reported value. This suggests that a signed average may have been used rather than the maximum-based metric; confirmation requires the stage-wise scores and evaluation code. These issues materially affect the forgetting and state-of-the-art claims.

  2. The incremental methodological novelty and the mechanism behind the gains are not yet sufficiently isolated. Sparsegen-based adaptive expert allocation is related to LD-MoLE (ICLR 2026), while C²Path (arXiv:2608.21937) also uses class-conditioned LoRA composition from an expert library for incremental vision-language detection. The evaluation protocols of C²Path and DCHA are not identical, so their published numbers should not be directly compared. DCHA differs through block-wise vector composition and hierarchical drift regularization, but its distinct technical contribution relative to these prior mechanisms should be made clearer. A controlled, matched-budget comparison of block-wise composition against rank-wise or whole-expert composition would better isolate the value of the proposed granularity. Moreover, the learned routing depends on category-specific parameters rather than input-image features and therefore does not directly address the within-class variation emphasized in the motivation. The category updates are then averaged into a task-level LoRA, leaving the extent of category-specific forward computation unclear. Finally, in Table 4 CAC alone reduces Avg from 56.7 to 52.0, with the largest improvement appearing only after HDC is added. The current ablations do not fully separate the contribution of LCAR from that of regularization.

Questions for authors
  1. Forgetting metric (critical clarification). Can the authors provide the full task-by-stage score matrix R_{t,i}, the exact evaluation code for Eq. (15), and a corrected or explicitly justified forgetting value? Table 16 appears to yield approximately 0.0167 when signed After−Before differences are averaged, whereas the maximum-based definition in Eq. (15) should not permit cross-task cancellation. How were the reported scores computed?

  2. ODinW-O baseline comparability (critical clarification). Why is DitHub reported at 59.2 Avg in Table 2 rather than the 62.38 reported on its official project page? Please specify any differences in task or category definitions, data splits, evaluation scripts, training protocols, or detector settings, and provide a directly comparable result. The discrepancy could change the ranking because DCHA reports 60.2 Avg.

  3. Novelty and matched-budget controls. Please clarify how DCHA differs technically from LD-MoLE’s learnable sparse expert routing and C²Path’s class-conditioned LoRA composition, while acknowledging their different benchmark settings. Can the contribution of sub-vector composition be isolated using rank-wise or full-expert composition within the same detector and task protocol, at comparable trainable-parameter budgets? A direct reproduction of C²Path on ODinW is not essential to answer this narrower question.

  4. Factorial ablation of the core modules. What are the results of CAC + HDC without LCAR, and of adding an appropriate drift regularizer to the independent category-LoRA baseline? These controls are needed to determine whether LCAR is essential and whether HDC explains most of the final performance gains.

  5. Claimed category specificity and practical efficiency. Since LCAR is category-indexed rather than image-conditioned, how does it account for intra-class feature heterogeneity? Can the authors provide a route-swap or category-routing perturbation test? Also, does the reported parameter reduction translate into lower peak memory and comparable training/inference cost on the same hardware?

3 comments
AuthorsOpenReview ↗

Response (1/2)

W1/Q1. Thank you for pointing out this error. We corrected Eq. (15) to match negative backward transfer (Lopez-Paz and Ranzato, 2017):Fˉ=−BWT=1T−1∑i=1T−1(Ri,i−RT,i).\bar{F}=-\mathrm{BWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}(R_{i,i}-R_{T,i}). We report the original peak-based definition separately as Fˉpeak=1T−1∑i=1T−1(max⁡i≤t<TRt,i−RT,i).\bar{F}_{\mathrm{peak}}=\frac{1}{T-1}\sum_{i=1}^{T-1}\left(\max_{i\le t<T}R_{t,i}-R_{T,i}\right). See the Metrics subsection of Experiments for the notation.

We show the full stage-by-task AP matrices RR for ODinW-O below. Rows are training stages, and columns are evaluation tasks. Tables 21–23 provide further details. DitHub on ODinW-O

StageAeVeThSdHwPv
1 (Ae)51.6—————
2 (Ve)50.961.5————
3 (Th)51.461.579.3———
4 (Sd)50.860.579.347.8——
5 (Hw)50.860.880.947.872.7—
6 (Pv)49.252.974.738.766.972.9

DCHA on ODinW-O

StageAeVeThSdHwPv
1 (Ae)49.8—————
2 (Ve)49.160.3————
3 (Th)52.560.378.9———
4 (Sd)49.759.378.943.3——
5 (Hw)49.762.381.748.070.6—
6 (Pv)45.856.675.742.067.873.0

We also recomputed Fˉpeak\bar{F}_{\mathrm{peak}} for both benchmarks and report it alongside Fˉ\bar{F} below.

BenchmarkMethodMean peak task AP ↑Fˉ\bar{F} ↓Fˉpeak\bar{F}_{\mathrm{peak}} ↓
ODinW-13DitHub64.732.802.80
ODinW-13DCHA63.340.01670.925
ODinW-ODitHub64.576.106.42
ODinW-ODCHA64.683.005.44

DCHA exhibits less forgetting than DitHub on both benchmarks under both definitions. The two metrics provide complementary perspectives on forgetting. Thank you for your feedback and for pointing out the need to clarify this distinction.

W1/Q2. The reported DitHub results come from our own reproduction. Our ODinW-13 results closely match the published results. On ODinW-O, Avg is 2.95 points lower and ZCOCO is 0.52 points higher than reported in the paper. We suspect that these differences stem from the different task order used during training. DitHub uses the expert learned in earlier tasks to initialize each new round of specialization for a recurring category, merging it with the current task’s warmup weights. DitHub also merges the shared (B) matrix after every task, carrying earlier updates across categories. Since every category on ODinW-O recurs, changing the training order can alter both the final category experts and the shared (B), affecting Avg and ZCOCO. To assess sensitivity to task order, we compare both methods across four seeds with different task orders (see QZuE W4). Computing (H) from mean ZCOCO and Avg gives DCHA gains of 0.55 and 0.53 points over reproduced DitHub on ODinW-13 and ODinW-O, respectively.

To further evaluate DCHA under substantial cross-domain category recurrence, we also tested on AODRaw (Li et al., CVPR 2025), which spans nine domains.

MethodZCOCOAvgHFog and Low LightFog and RainFogIndoor Low LightIndoor Normal LightLow Light and RainLow LightNormal LightRain
GD ZS47.415.623.513.319.619.415.824.811.716.35.814.1
ZiRA44.432.337.427.636.329.328.343.230.731.730.533.5
DitHub45.234.038.830.335.531.431.047.831.432.930.935.1
DCHA46.034.839.630.536.533.230.749.532.933.630.535.7

On AODRaw, DCHA improves ZCOCO, Avg, and H by 0.8 points each over DitHub. DCHA achieves higher AP in seven of the nine domains.

AuthorsOpenReview ↗

Response (2/2)

W2Q3. DCHA learns Sparsegen weights indexed by category and composes LoRA from sub-vectors within individual AA rows or BB columns. LD-MoLE uses input-token features to compute Sparsegen weights and mixes the outputs of complete LoRA experts. C²Path uses contextual text-token features for Top-K routing and combines complete AA and BB factors from selected experts to synthesize token-specific adapters.

We compare these designs on ODinW-13 at comparable trainable-parameter budgets of approximately 20.8 M, including routing parameters. In each of the 36 vision–language fusion projections, Paired-rank + LCAR uses category-indexed Sparsegen weights to compose LoRA updates from a bank of 76 rank pairs. Each pair consists of a full row of AA and its corresponding column of BB. LD-MoLE-style uses seven rank-16 LoRA experts and mixes their outputs using Sparsegen weights computed from input-token features. C²Path-style uses seven rank-16 LoRA experts and combines the AA and BB factors of the two experts selected for each token using contextual text features. DCHA uses a bank of 40 sub-vectors of dimension 256 per projection. We disable HDC in DCHA to isolate the effects of composition and routing.

MethodParam(M)ZCOCOAvgH
Paired-rank + LCAR20.746.553.649.8
LD-MoLE-style20.316.738.323.3
C²Path-style19.946.950.448.6
DCHA w/o HDC20.847.055.751.0

Our method achieves the highest ZCOCO/Avg/H among the variants with results. With the same LCAR routing, sub-vector composition raises H from 49.8 to 51.0 on ODinW-13 and from 52.0 to 52.6 on ODinW-O.

W2/Q4. LCAR improves Avg both when added to CAC alone and when HDC is already enabled. See QZuE W2 for the full ablation. Adding LCAR to CAC alone raises final Avg from 52.0 to 55.7 and increases Avgbef\mathrm{Avg}_{\mathrm{bef}}, the mean AP measured immediately after learning each task, from 56.9 to 61.3. With HDC enabled, adding LCAR raises final Avg from 59.5 (CAC+HDC) to 62.3. Full DCHA also exceeds independent category-specific LoRA with HDC by 1.3 points (62.3 vs. 61.0).

Q5. DCHA handles intra-class variation by applying task-level LoRA update to instance-specific images, and LCAR determines which shared bank vectors receive gradients through each category’s LoRA update (see QZuE W1). To test category specificity, we replace a recurring category’s route with that of another category from an earlier task during inference on ODinW-O. Task AP is the mean AP over all categories in the evaluated task, and Target AP is the AP of the category whose route is replaced, measured on the same evaluation set. The results are shown below.

TaskRoute swap (target ← source)Task APTarget AP
Hwperson ← Th dog67.8 → 65.467.8 → 65.4
Sdcar ← Ae boat42.0 → 39.848.6 → 45.8
Pvcar ← Ve truck73.0 → 70.168.1 → 64.4

Both Task AP and Target AP decrease in all three swaps, supporting the category specificity of the learned routes.

For training/inference cost, we benchmarked DCHA and DitHub on the same hardware(L20) with a training batch size of 2 on ODinW-13. DCHA reduces peak reserved GPU memory from 21.4 to 16.5 GiB during training and from 2.5 to 1.5 GiB during inference. It also reduces inference latency from 123.6 to 85.4 ms/image and increases throughput from 7.9 to 8.5 images/s.

Dear Reviewer L6Rt,

Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.

Best,

AC

Reviewer_QZuEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DCHA, a parameter-efficient adaptation framework for incremental vision-language object detection, which requires models to continually adapt to new domains while preserving performance on previous tasks and the open-vocabulary generalization acquired during pretraining. The authors argue that parameter sharing and fixed capacity allocation in existing task-level or category-level LoRA methods limit the flexibility of category adaptation, while continual updates to shared parameters cause historical knowledge to drift. DCHA therefore composes category-specific LoRA modules from a shared vector bank, using category-specific routing parameters and learnable sparsity coefficients to select different vector combinations. Category updates are then aggregated into task updates, while drift constraints are imposed at the vector, category, and task levels. Experiments on ODinW-13 and ODinW-O demonstrate favorable adaptation and knowledge retention.

Critical strengths
  1. DCHA starts from differences in adaptation requirements across categories and identifies fixed capacity allocation as a potential limitation on adaptation flexibility. This provides a concrete starting point and reasonable motivation for category-adaptive design.
  2. Constructing category-specific adaptation parameters from a shared vector bank enables knowledge reuse across categories while allowing different categories to select different combinations. The design is straightforward and offers a degree of flexibility.
  3. The experiments demonstrate a favorable trade-off between performance and parameter efficiency. On ODinW-13, the method reduces adaptation parameters from DitHub’s 147M to 20.8M while achieving comparable or slightly better detection performance, illustrating the practical value of shared vector composition.
Critical weaknesses
  1. The paper motivates the method partly through intra-class variation, but LCAR routing is determined by category-specific parameters rather than the current image or instance features. It therefore remains unclear how the method addresses differing adaptation requirements among samples within the same category.
  2. In the component ablations, the main performance improvement appears after introducing HDC. However, there is no comparison with independent category-specific LoRA modules trained under the same constraints, making it difficult to separate the contributions of compositional parameterization and regularization.
  3. The paper does not clearly explain how task updates are selected during inference, how unseen categories are handled, or how historical routing is updated when categories recur. These omissions make it difficult to assess the experimental protocol and the method’s applicability.
  4. Some accuracy gains over the strongest baseline are small, while all experiments use a single random seed. Without repeated runs and evaluations under different task orders, the stability of these gains remains uncertain.
Questions for authors
  1. It is suggested to clarify the core mechanisms introduced by DCHA beyond existing vector-bank LoRA methods and verify their independent benefits under the same parameter budget.
  2. Please compare learning the sparsity coefficient with directly adjusting the scale of the routing logits, to establish whether LCAR provides benefits beyond parameter rescaling.
  3. Please further explain how hierarchical drift constraints differ from conventional parameter-preservation regularization and verify their necessity through comparisons that control for regularization strength.
3 comments
AuthorsOpenReview ↗

Response (1/2)

W1. DCHA handles intra-class variation through capacity-adaptive multi-subspace composition, using the same route across images within each category. LCAR activates more basis vectors for categories with greater intra-class variation and fewer for simpler categories. DCHA then averages the resulting category-specific LoRA updates into a shared task-level update. Within a task, LCAR uses one route per category, and DCHA aggregates the category-specific LoRA updates into a shared update ΔWt\Delta W_t. The residual of ΔWt\Delta W_t can differ across instances because their input features differ. During training, gradients from the detection loss flow back through the aggregation to each category-specific update. The sparse LCAR coefficients determine which vectors in the shared bank receive gradients through each category branch. Fig. 4 visualizes the gradients of target-category scores for queries matched to GT instances. In the examples shown, DCHA’s responses are more concentrated within the target-category boxes than DitHub’s.

To test the effect of aligning category-specific updates with same-category GT supervision during training, we compare DCHA with three variants that differ in which updates they aggregate and which GT instances supervise detection. a) All GT categories. For each training image, we aggregate the LoRA updates of all annotated categories and supervise detection with all GT instances. b) One GT category (all GT). Following DitHub’s sampling strategy, we randomly sample one annotated category per image, use only its LoRA update, and retain all GT instances for supervision. c) One GT category (matched GT). We use the same sampling strategy and supervise detection only with GT instances of the sampled category.

VariantZCOCOZ_{\mathrm{COCO}}AvgHH
DCHA (all task categories)47.862.354.1
All GT categories47.561.253.5
One GT category (all GT)47.460.653.2
One GT category (matched GT)47.058.152.0

DCHA leads on all three metrics, outperforming the DitHub-style One GT category (all GT) by 0.4/1.7/0.9 points in ZCOCOZ_{\mathrm{COCO}}/Avg/HH, respectively.

W2. We added an independent category-specific LoRA baseline trained with HDC to the ODinW-13 ablation (second row of the table below) to demonstrate that HDC regularization can reduce forgetting (Fˉ\bar F 4.5 → 0.2). CAC and LCAR are also effective. CAC’s compositional parameterization reduces trainable parameters for category-specific adaptation from 123.8M to 20.7M. Adding LCAR costs only 0.1M parameters and improves new-task AP, raising Avgbef\mathrm{Avg}_{\rm bef} from 56.9 to 61.3, compared with 60.9 and 61.2 for independent LoRA without and with HDC, respectively.

CACLCARHDCParams (M)ZCOCOAvgHFˉ\bar FAvgbef\mathrm{Avg}_{\rm bef}
123.848.256.752.14.560.9
✓123.847.861.053.60.261.2
✓20.747.552.049.65.356.9
✓✓20.747.859.553.01.460.8
✓✓20.847.055.751.06.161.3
✓✓✓20.847.862.354.10.062.3

Avgbef=113∑i=113Ri,i\mathrm{Avg}_{\mathrm{bef}}=\frac{1}{13}\sum_{i=1}^{13}R_{i,i}, where Ri,iR_{i,i} is the AP on task ii immediately after training on that task, as a measure of how well the model adapts to new tasks.

W3. The composition of the task-level update is described in lines 208–209 of the main text. Following ZiRa and DitHub, we use a text prompt listing the target categories for each IVLOD task. We average the LoRA updates for those prompted categories to form the task-level update.

We distinguish between training and inference when handling unseen categories. During training, we initialize a learnable route for each category when it first appears. At inference, the LoRA average includes only categories with learned routes. If all prompted categories are unseen, we use the original pretrained detector.

For recurring categories, reusing and freezing the learned route. Its update can still evolve as the shared vector bank adapts under HDC. We compare updating and freezing the routing parameters below.

ODinW-13ODinW-O
RoutingZCOCOAvgHHZCOCOAvgHH
Updated47.461.353.547.459.552.8
Frozen47.862.354.147.660.253.1

Freezing the learned routes improves ZCOCO, Avg, and HH on both benchmarks, supporting the decision to preserve learned route when categories recur.

AuthorsOpenReview ↗

Response (2/2)

W4. We shuffled the task order for each of the four seeds (0, 42, 1024, and 2027). Our method outperforms the baseline on average across these runs, as shown below.

BenchmarkMethodZCOCOAvgH
ODinW-13DitHub47.25±0.4561.75±0.5553.54±0.24
ODinW-13DCHA47.80±0.1662.28±0.4254.09±0.22
ODinW-ODitHub47.03±0.1759.43±0.7452.51±0.19
ODinW-ODCHA47.25±0.2560.45±0.6453.04±0.20

DCHA improves mean H over DitHub by 0.55 points on ODinW-13 and 0.53 points on ODinW-O. On ODinW-13, DCHA reduces parameter storage from 147M to 20.8M and lowers training/inference cost (see L6Rt Q5). DCHA also achieves state-of-the-art performance in our IVLOD evaluation on AODRaw (Li et al., CVPR 2025) (see L6Rt W1/Q2).

Q1. DCHA uses a separate vector bank for each layer and category-specific Sparsegen routes, while existing vector-bank LoRA methods(VB-LoRA) uses a global vector bank and layer-specific top-2 routes shared across categories. At a matched vector-bank budget, CAC improves zero-shot AP, category-specific Sparsegen routing improves adaptation to new tasks, and HDC reduces forgetting. All variants adapt the same 36 fusion projections at rank 16 and use 1,440 vectors of dimension 256. Starting from VB-LoRA, we introduce CAC with top-2 routing, then replace top-2 routing with category-specific Sparsegen routing (LCAR), and finally add HDC.

VariantBank size (vectors)ZCOCOAvgHFˉ\bar FAvgbef\mathrm{Avg}_{\rm bef}
VB-LoRA (top-2)1,440 (global)46.552.049.19.260.5
CAC (top-2)1,440 (36 layers × 40)47.552.049.65.356.9
CAC + LCAR (Sparsegen)1,440 (36 layers × 40)47.055.751.06.161.3
CAC + LCAR + HDC (DCHA)1,440 (36 layers × 40)47.862.354.10.062.3

Q2. We compare learning the sparsity coefficient with directly adjusting the scale of the routing logits. LCAR achieves the highest ZCOCO, Avg, and H. See Tables 7 and 16 in the paper for the experimental settings.

VariantZCOCOAvgH
Learning the sparsity coefficient directly (Direct, Table 16)47.760.153.2
Directly adjusting the scale of the routing logits (S\mathcal{S}-Fixed, Table 7)47.860.653.4
LCAR (ours)47.862.354.1

In Direct, we use λc=min⁡(ηc,1−ϵ)\lambda_c=\min(\eta_c,1-\epsilon) because Sparsegen requires λc<1\lambda_c<1. The upper bound keeps the denominator 1−λc1-\lambda_c at least ϵ\epsilon and prevents division by zero.

Q3. HDC preserves previously learned adaptations at three levels. Conventional methods such as EWC (Kirkpatrick et al., 2017) and Synaptic Intelligence (Zenke et al., 2017) penalize changes to individual parameters. HDC is most effective when it jointly constrains drift in the shared bank vectors, the category-specific LoRA updates composed from them, and their task-level aggregates.

Drift constraintHH ↑Fˉ\bar F ↓
None51.06.1
Vector only (Lv\mathcal{L}_v)52.54.1
Category only (Lc\mathcal{L}_c)53.23.6
Task only (Lt\mathcal{L}_t)53.03.5
All three (Lv+Lc+Lt\mathcal{L}_v+\mathcal{L}_c+\mathcal{L}_t)54.10.0

Each single-level constraint reduces forgetting, with 3.5–4.1 points of mean forgetting remaining. The joint constraints reduce mean forgetting to 0 and raise HH from 51.0 to 54.1, showing the benefit of preserving adaptations at all three levels.

For regularization strength, Appendix Table 18 shows that the best results are obtained with regularization weights of 0.70.7, 0.50.5, and 0.70.7 for Lv\mathcal{L}_v, Lc\mathcal{L}_c, and Lt\mathcal{L}_t, respectively.

Dear Reviewer QZuE,

Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.