- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Deformable Vision Query (DVQ) connector, a vision-language connector that represents visual information using a fixed number of vision queries associated with spatial reference points. DVQ modules are inserted into selected LLM layers and use deformable cross-attention over multi-level vision features to retrieve visual information around these reference points. The reference points are initialized across the image and progressively refined across LLM layers, allowing the visual queries to adapt their spatial focus. The same representation is further extended to continuous reference boxes, enabling differentiable bounding-box inputs and outputs. The method is evaluated in two settings: small-scale task-specific fine-tuning on GQA, TextVQA, and RefCOCOg, and general-purpose vision-language instruction tuning following the LLaVA-1.5 protocol across several standard VLM benchmarks. The main empirical claim is that DVQ maintains performance more gracefully than Patch Project and DeepStack when the number of LLM-side vision tokens is reduced. Additional experiments study high-resolution inputs, difference LLM backbones and scales, DVQ layer placement, parameter sharing, hierarchical vision features, and differentiable bounding-box representations.
Critical strengths
-
The paper provides a clear and practically motivated spatial inductive bias for vision-language connectors. Associating vision queries wiith explicit reference points and refining them using deformable attention is an intuitive way to retain spatial coverage while allowing image-adaptive visual sampling. The approach is techinically clean and integrates naturally with hierarchical vision features.
-
The empirical evaluation is thorough. The paper evaluates multiple token budgets, different LLMs and LLM scales, high-resolution inputs, DVQ placement, parameter sharing, feature levels, and computational cost. In particular, the observation that DVQ degrades relatively gracefully under aggressive reductions in the number of vision tokens is potentially practical in resource-contrained settings. In additions, the general-purpose experiments show that DVQ mostly matches rather than consistently surpasses DeepStack at the full token budget.
Critical weaknesses
-
The standard DVQ is not actually language-conditioned, which weakens both the visual querying interpretation and its conceptual distinction from existing visual token compression. My main concern is that DVQ is presented as allowing the LLM to dynamically query visual information, but the spatial refinement of the standard vision queries does not use previous textual tokens. The authors explicitly acknowledge this limitation. This distinction is important, since for many VLM tasks, the subset of visual information that should be preserved is inherently conditional on the user query. For example, the same image may require inspecting a small sign for one question, or a global scene context for the other. If the reference-point trajectories are determined independently of the question, DVQ cannot selectively allocate its limited visual-query budget toward the evidence required by the current instruction. It is therefore closer to an image-adaptive compression mechanism than to genuinely language-conditioned visual querying. I consider this particularly consequential because the claimed advantage of DVQ becomes most important precisely under aggressive token compression, where allocating a small number of queries to task-relevant locations should matter most. The limitation is also structural rather than a missing minor ablation: making the lookup genuinely text-conditioned would require changing the interaction between text and vision queries and may affect the caching and efficiency properties emphasized by the paper. The reported weakness on dense-text/OCR tasks further demonstrates that sparse, task-independent sampling can fail to preserve information required downstream. This weakens the motivation for DVQ. Once the visual selection precess is decoupled from the language query, the distinction from conventional visual token pruning/compression becomes much less consequential, since both mechanisms ultimately attempt to retain a small subset of image information before or during language reasoning. The paper does not demonstrate that DVQ preserves task-relevant evidence better than modern token pruning/compression methods under matched computational budgets.
-
The experiments do not cleanly establish either computational efficiency or the claimed causal role of position grounding. The main comparisons are organized around the number of LLM-side vision tokens, but equal token counts do not correspond to equal computational budgets. DVQ introduces deformable-attention modules repeatedly throughout the LLM and has more connector parameters than DeepStack. Thus, the observation that DVQ performs better than DeepStack at a given value of does not by itself demonstrate a superior accuracy-efficiency trade-off. A compute, memory, or latency matched comparison would be substantially more convincing. The paper also acknowledges that DVQ adds computational and serving complexity relative to Patch Project and DeepStack. More importantly, I do not fine the "DVQ without grounding" experiment sufficient to support the strong claim that "position grounding is reponsible for overcoming the limitations of cross-attention." The ablation replaces deformable attention with standard dot-product attention. This simultaneously changes the positional inductive bias, sparse/local sampling mechanism, attention operator, and computational structure. Consequently, the large performance difference does not isolate position grounding as the causal factor. The paper nevertheless interprets this comparison as evidence that explicit position grounding itself overcomes the limitations of cross-attention. This is important because deformable attention and reference-point refinement are themselves established mechanisms from prior vision architectures. Without an ablation that isolates the positional component while keeping the underlying attention/sampling mechanism controlled, it remains unclear which part of DVQ is actually responsible for the reported gain.
Questions for authors
-
Can the authors provide a genuinely text-conditioned DVQ variant in which reference-point refinement has access to the user question or previously generated language tokens? I consider this more than a future-work suggestion because it directly tests the paper's core interpretation of DVQ as allowing an LLM to query visual information. In particular, I would like to know whether question-conditioned refinement improves performance under aggressive token budgets such as or , where task-dependent visual selection should matter most. If this variant substantially improves compression while retaining reasonable computational cost, it would affect my assessment.
-
Can the authors provide a controlled experiment that separates position grounding from the change in attention operator, as well as an actual compute-matched efficiency comparison? For example, could the authors compare variants that retain the same deformable-attention operator but remove or randomize the sptial grounding/refinement mechanism? Conversely, an experiment comparing DVQ and competing connectors at matched FLOPs, latency, or memory rather than only match numbers of LLM vision tokens would clarify whether DVQ provides a better practical efficiency-performance trade-off. These results would materially affect my interpretation of the paper's two central claims.
-
Can the authors provide an actual end-to-end experiment demonstrating why differentiable bounding-box inputs/outputs matter in practice? The paper motivates differentiable boxes primarily through their potential use in end-to-end trainable multi-model systems, yet the current experiments evaluate box prediction and box-conditioned tasks only in isolation. Can the authors demonstrate a downstream setting in which gradients are actually propagated through the predicted bounding boxes across model components, and show that this differentiability provides a meaasureable advantage over conventional text- or token-based box representations?
5 comments
Rebuttal (Part 2)
2a. Equal computational budgets
We want to highlight that the end-to-end efficiency is task-, implementation-, and hardware-dependent. While equal token counts do not correspond to equal computational budgets, we decided to focus on matched token budgets, because this is a good initial estimator of the potential performance across tasks (question/answer lengths), implementations, and hardware. To further estimate the practical costs, we now added a new analysis, comparing the FLOP premiums of our DVQ method (compared to DeepStack) with the resulting accuracy improvements on different tasks (see extended Sec. A.8 with new Fig. 6):
| Token Budget | FLOPs | GQA | TextVQA | RefCOCOg |
|---|---|---|---|---|
| 1024 | +9.3% | +2.5% | -11.9% | +22.7% |
| 256 | +5.0% | +3.2% | -0.0% | +16.7% |
| 64 | +3.0% | +5.0% | +22.5% | +20.2% |
| 16 | +2.4% | +4.2% | +66.9% | +27.4% |
| 4 | +2.3% | +3.7% | +14.3% | +65.5% |
At full budget DeepStack retains the TextVQA/OCR edge, but already at |V|=256 RefCOCOg buys +16.7% score and GQA +3.2% for +5.0% FLOPs. At |V|<=64 DVQ improves every task by a larger relative margin than the relative extra FLOPs costs (4-67% improvement vs. 2.3-3.0% extra FLOPs), while the extra cost itself shrinks as budgets gets smaller. Note that when the text length |T| (question+answer) increases, the extra FLOPs of the vision connector become less relevant: At |V|=256, the FLOPs premium of DVQ (against DeepStack) is +5.0% at |T|=1024 but only +1.6% at |T|=4096. Due to this, a perfectly FLOP-matched comparison is not possible, but the table above (and the new Fig. 6) should provide some context.
Additionally, we now also report the GPU memory and inference-speeds (see extended Sec. A.8 ). At 1024 tokens our peak memory usage is 10.7 GB (vs. 9.2 GB for DeepStack), while for 256 tokens it is 10.1 GB (vs. 8.71 GB for DeepStack). Our inference prefill (TTFT) is also a bit slower than DeepStack (260.8 ms vs. 164.1 ms at 1024 vision tokens, 134.2 ms vs. 70.8 ms at 256 tokens). After prefill, token generation of DVQ and DeepStack are identical (using only cached LLM hidden features, no need to use the DVQ layer after prefill). Note that we did not optimise our implementation ( generation loop, caching, ...) for inference and did not use any inference-frameworks or inference-optimized hardware. Therefore, the numbers above need to be treated with caution and the FLOPs may be a better estimate for fair comparison.
Finally, our methods also enables other compute cost reduction potentials, e.g. being well-suited for small-scale fine-tuning (see Sec. 4.1) and more robust to smaller LLMs (see A.1).
2b. Causal role of position grounding
We agree that our "position grounding" claim could be sharpened. We point out that the "standard cross-attention variant" only replaces the attention mechanism itself but keeps all other aspects of DVQ intact (including reference points, refinement, and reference position embeddings). While this variant has access to positional information (via reference pos embeddings), it is not explicitly enforced to use it (it is not explicitly grounded). For better clarity, we now explicitly refer to this variant in the paper as "DVQ with standard dot-product attention" instead of "no grounding". To further verify the relevance of the reference points as a central mechanism, we now also added an ablation of DVQ where reference points are randomly resampled at each layer (new Tab. 2 in Sec. 4.1). This lets us disentangle whether DVQ only benefits from locally sparse sampling or whether it actually benefits from this explicit position grounding. Furthermore, we also add an ablations study with "fixed reference points" (with optionally also fixed sampling offsets, new Tab. 2 in Sec. 4.1), to study the relevance of the refinement component.
| Method | GQA | TextVQA | RefCOCOg |
|---|---|---|---|
| DVQ (default) | 64.2 | 49.8 | 63.3 |
| -> random reference points | 61.8 | 21.9 | 47.5 |
| -> fixed reference points | 61.6 | 46.9 | 63.4 |
| -> fixed reference points + sampling offsets | 61.8 | 28.2 | 59.7 |
| -> standard-cross attention | 56.8 | 24.9 | 28.7 |
Performance collapses (especially on TextVQA) when randomly sampling reference points in each DVQ layer, demonstrating that DVQ relies in "positional grounding" via reference points to function. Using fixed reference points, i.e. not refining them over layers, leads to a small performance drop, and if sampling offsets are fixed as well, TextVQA performance collapses.
Rebuttal (Part 3, Questions)
Q1. Language-conditioning
See our answer 1a (above). Question-conditioning can improve the performance if enough query tokens remain before the question (as this is where the LLM expects context). Under more aggressive token budgets, using some of the token budget for "dynamic" tokens after the question, degrades the performance, as the remaining "static" (before-question) queries seem to be too few. Note that, unlike many existing "language-conditioned" token compressors, DVQ is directly integrated into the LLM layers, such that it needs to be designed with the LLM's expectations (e.g., context-before-question) in mind.
Q2. Position-grounding and compute-matched
See our answers 2b and 2a, respectively.
Q3. End-to-end differentiable bounding-box experiment
Such an end-to-end differentiable bounding-box experiment would evaluate the DVQ method as part of a larger system. As our primary contribution is an architectural component (the DVQ connector) and not the design of an end-to-end system, we consider such an experiment out-of-scope. In Sec. 4.3, we demonstrated that our DVQ component does enable differentiable bounding box inputs and outputs. Whether this property is needed or beneficial depends on the specific end-to-end system. In Appendix B, we already describe example use cases, but use cases are not limited to these. However, we extended Sec. 5, clarifying that we did not evaluate the end-to-end benefit of differentiable bounding boxes.
Rebuttal (Part 1)
Summary
We thank the reviewer for their detailed and insightful feedback. Below, we provide additional experimental evidence and explanations to sharpen our contributions, including:
1a. Language-conditioning
In its default form, DVQ does not have access to question context as vision query tokens are placed before any question tokens. However, the DVQ module still "queries" the vision information per token and layer, even if these queries are only conditioned on prior vision information and not on the question. We follow this design because placing image context before the question is common practice and easily compatible with multi-round conversations (e.g. in Llava training): each new question can directly look up the relevant information from the image. Placing the image (query) tokens after the question typically performs worse (confirmed by preliminary experiments) as it does not follow the LLMs typical conditioning structure (context first, then question/prompt) . However, DVQ's connector architecture is agnostic to placement in the sequence and also supports placing tokens after the question. To demonstrate this, we have now introduced "dynamic query" tokens that can be placed after the question (see new Tab. 8 in new A.4), with initial reference points predicted based on features after the first LLM layer and refined as always. We explored different mixtures of "static" queries (before the question) and "dynamic" queries (after), finding that a small fraction of "dynamic" queries can improve the performance on some tasks, especially on OCR-heavy tasks. However, the performance can drop if the "dynamic" fraction is too large or the total token budget is too small.
This demonstrates that "dynamic" queries can provide informative question-conditioned image-context, but a sufficiently large static query budget is still required (probably because LLMs are used to such a context).
This trade-off between "static" and "dynamic" queries (caused by its placement inside the LLM) leads to DVQ not benefiting from question-conditioning when the token budget is low. Note that caching the static query tokens is still possible but the dynamic queries need to be cached per question.
1b. Comparison with modern token pruning/compression
We note that the already included Patch Project, DeepStack, and Perceiver baselines all apply token compression when token budgets <1024 are used. Patch Project and DeepStack rely on pixel shuffle (see Appendix C.3), which is conceptually close to the C-Abstractor from Honeybee, while Perceiver relies on query-based resampling. Additionally, we now ran Honeybee's D-Abstractor, MQT-LLaVA, TokenPacker, and AcFormer (see new Tab. 7 in A.3) in our small-scale setup. All of these are learnable token compressors. Furthermore, we ran FastV as a non-trainable token pruning method, applied to Patch Project (|V|=1024) and pruned to 256 tokens.
In general, resampling-based methods (Perceiver, MQT, TokenPacker, D-Abstractor) seem to not work well in our small-scale setting, probably because they need way more training data to perform well. FastV shows strong performance on 2/3 tasks but still underperforms our method, especially on RefCOCOg and GQA.