acceptodds
Withdrawn from ICLR 2027

Adapters Do More Than Adapt: Coupling Parameter-Efficient Fine-Tuning with Recovery-Aware Structured Pruning for Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual grounding aims to localize an image region specified by a natural-language expression. Large pretrained vision-language models have substantially improved grounding accuracy but incur considerable adaptation and inference costs. Parameter-efficient fine-tuning reduces trainable parameters while leaving the deployed backbone intact, whereas structured pruning produces smaller models but is typically decoupled from task adaptation, leading to representation shifts and degraded task-relevant responses. To bridge this gap, we propose Adapter-Conditioned Recovery Pruning (ACRP), which couples parameter-efficient adaptation with recovery-aware structured pruning in a unified optimization framework. The key idea is to counterfactually supervise adapters to approximate compact signatures of candidate-removed responses, thereby providing recovery-aware guidance for pruning. ACRP combines modality-aware joint budget allocation across attention and MLP structures with a three-stage optimization strategy, allowing adaptation and compression to interact before physical pruning. Moreover, parallel multi-level feature branches integrate hierarchical visual semantics, while an evidence-preserving token condenser retains expression-relevant tokens and aggregates the remaining tokens into regional context. Experiments on four public datasets demonstrate that ACRP achieves effective task adaptation and retains competitive localization performance, while updating merely 0.38% of the pre-trained encoder parameters and physically pruning approximately 40% of the backbone parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Grounding, Vision-Language Models, Parameter-Efficient Fine-Tuning, Recovery-Aware Structured Pruning, Adapter-Based Recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HDyoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adapter-Conditioned Recovery Pruning (ACRP) for visual grounding. ACRP augments frozen DINOv3 and CLIP encoders with visual and textual recovery adapters. During joint training, the adapters are supervised to approximate compact signatures of candidate-removed attention-head and MLP responses, and the resulting recoverability signal is combined with task and structural criteria to allocate a global pruning budget. The model also includes multi-level visual fusion and an evidence-preserving token condenser. Experiments on four visual-grounding benchmarks, together with component and pruning ablations, report competitive accuracy with a 0.38% adapter update ratio relative to the pretrained encoders and approximately 40% physical backbone pruning.

Critical strengths
  1. The paper addresses an important and practical problem. Standard PEFT reduces the number of updated parameters but generally leaves inference cost unchanged, whereas structured pruning improves deployment efficiency but may cause task-specific representation shifts. Coupling the two objectives is therefore well motivated.
  2. The experimental evaluation is broad and reasonably comprehensive. The method is evaluated on four public visual-grounding benchmarks and compared with visual-grounding, PEFT, and structured-pruning baselines. The paper also includes several components, pruning, recovery, and token-condensation ablations.
  3. The deployment-oriented reporting is relatively thorough. Beyond grounding accuracy, the paper reports parameter count, storage, GFLOPs, peak inference memory, and latency under multiple physical pruning ratios, providing a useful view of the accuracy–efficiency trade-off.
Critical weaknesses
  1. The efficiency and attribution claims are not fully supported. The 0.38% ratio excludes other trainable components, while heterogeneous backbones and insufficiently specified pruning baselines make it difficult to attribute the gains specifically to recovery-aware pruning.
  2. The recovery objective is only indirectly validated. The paper does not establish that its compact signature error correlates with downstream IoU degradation, and the relevant evidence is limited to one dataset and one random seed.
Questions for authors
  1. The reported 0.38% is clearly defined as adapter-only overhead. Could you also report the total trainable parameters and a module-wise breakdown, and clarify whether the comparison in Table 2 remains favorable under the complete trainable-parameter budget?
  2. Table 1 provides a useful overview, but the compared methods use different backbones. Could you provide the same backbone results for several representative baselines and report variation across multiple random seeds?
Reviewer_xdUsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose Adapter-Conditioned Recovery Pruning (ACRP) for visual grounding. They propose a framework for parameter-efficient adaption, and recovery-aware pruning. As part of the design they propose multi-level feature branches and a token consider to better improve visual semantics, improve regional context and remove redundancy and improve visual cues.

Contributions of the work are a framework to adapt, and prune the architecture, with token compression, and SOTA results on benchmarks.

Authors show results on 4 Benchmarks (RefCOCO/RefCOCO+/RefCOCOg/ReferItGame) with SOTA performance. This is impressive given the approach only updates 0.38% of pre-trained parameters (the least given published comparisons with like-backbones, and the least update ratio!) , and prunes ~40% of the backbone parameters.

Critical strengths

Results are quite strong when compared to same backbone, heads, supporting the case that the architecture adaptors has merit compared to SOTA approaches on key RefCOCO datasets. The approach is also well ablated, showing improvements across key elements/variables of the proposed approach The approach has best in class - having very few needed updates (smallest update ratio), and able to do signficant pruning.

Critical weaknesses

Qualitative claims are not well supported with proper ablation or visual examples - for example, claims for visual grounding are that terms improve expressions involving attributes, scale, spatial relationship, etc (and is better able to deal w/ object sizes and cluttered expressions), but the stronger way to present this is to ablate on this in the dataset itself (filter by claimed attribute types - for example object size).

Questions for authors

Aggressive pruning in Table 5 does not appear to have significant impact on latency suggesting that while model footprint is smaller, parameter size of the model has diminishing returns on inference time. Any thoughts/position on this?

What are the failure cases of the model - additional discussion and analysis would help strengthen the claims/results in the paper. The writing in the figures additionally are small so difficult/if not impossible to read.

Reviewer_7krsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ACRP combines parameter-efficient adaptation with recovery-aware structured pruning for visual grounding. The motivation is reasonable: efficient adaptation alone does not reduce deployment costs, while pruning can damage task-relevant representations.

Critical strengths
  1. The reported performance is competitive, with relatively small accuracy losses after approximately 40% backbone pruning.
  2. Using adapter recoverability to guide structural selection is interesting and reasonably novel.
Critical weaknesses
  1. Poor readability and positioning. The related-work section does not clearly explain recent developments or distinguish ACRP from closely related methods. Terms such as “FiLM-modulated” lack sufficient context. The correspondence between figures, equations, and descriptions is difficult to follow, and the captions provide insufficient guidance.

  2. Insufficient reproducibility. Several central operations, including candidate-removed response construction, per-structure recoverability scoring, global budget allocation, and associated losses, are not precisely specified. The notation also requires clarification and consistent use.

  3. Confounded baseline comparisons. Table 1 compares DINOv3 ViT-L/CLIP-L against substantially different backbones, including DARA’s ResNet-50/BERT. These comparisons cannot isolate ACRP’s contribution. Several results also correspond to unlabelled weaker variants, including HiVG-B and LoA-Trans-S, while stronger published versions are omitted without justification.

  4. Reporting inconsistency. Table 2 lists DARA’s update ratio as 1.63%, whereas DARA’s original paper reports 2.13%. Any change in the denominator should be explained.

Questions for authors
  1. Could the authors describe the method more clearly and in greater detail, revise the notation for consistency, and improve the figure and table captions to explain their contents and connection to the method?

  2. Could the authors provide comparisons under matched settings, such as using the same backbone or comparing models with similar parameter counts and computational costs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.