acceptodds
Under review as a conference paper at ICLR 2027

Deformable Vision Query Connector: Letting LLMs Query Vision Information through Position-Grounded Lookup

Abstract

Large Vision-Language Models (LVLMs) represent images as long sequences of visual tokens. However, preserving fine-grained visual content under tight to- ken budgets remains challenging: spatial aggregation often lacks adaptivity, while query-based resampling can collapse without extensive training and miss impor- tant regions. We propose the *Deformable Vision Query (DVQ) Connector*, a position-grounded and spatially adaptive interface embedded directly within LLM layers. DVQ associates each visual query with a reference point and uses de- formable cross-attention to refine its spatial focus. Uniformly sampled reference points encourage image coverage, while learned refinements allow adaptive con- centration on salient regions, overcoming limitations of cross-attention connectors through this position-grounding. DVQ also natively encodes positional informa- tion within its queries, enabling differentiable bounding box encodings. Exper- iments on small-scale per-task fine-tuning and general-purpose vision-language modeling demonstrate that DVQ is competitive with strong vision connectors like DeepStack, while DVQ intrinsically enables token compression, making it partic- ularly effective under reduced vision token budgets and in high-resolution settings. Ablations further demonstrate DVQ’s robustness and flexibility. We provide our source code at https://anonymous.4open.science/r/dvq.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language modeling, Vision-Language Integration, Vision Connector, Large Vision-Language Models, Deformable Attention, Spatial Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rwjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Deformable Vision Query (DVQ) connector, a vision-language connector that represents visual information using a fixed number of vision queries associated with spatial reference points. DVQ modules are inserted into selected LLM layers and use deformable cross-attention over multi-level vision features to retrieve visual information around these reference points. The reference points are initialized across the image and progressively refined across LLM layers, allowing the visual queries to adapt their spatial focus. The same representation is further extended to continuous reference boxes, enabling differentiable bounding-box inputs and outputs. The method is evaluated in two settings: small-scale task-specific fine-tuning on GQA, TextVQA, and RefCOCOg, and general-purpose vision-language instruction tuning following the LLaVA-1.5 protocol across several standard VLM benchmarks. The main empirical claim is that DVQ maintains performance more gracefully than Patch Project and DeepStack when the number of LLM-side vision tokens is reduced. Additional experiments study high-resolution inputs, difference LLM backbones and scales, DVQ layer placement, parameter sharing, hierarchical vision features, and differentiable bounding-box representations.

Critical strengths
  • The paper provides a clear and practically motivated spatial inductive bias for vision-language connectors. Associating vision queries wiith explicit reference points and refining them using deformable attention is an intuitive way to retain spatial coverage while allowing image-adaptive visual sampling. The approach is techinically clean and integrates naturally with hierarchical vision features.

  • The empirical evaluation is thorough. The paper evaluates multiple token budgets, different LLMs and LLM scales, high-resolution inputs, DVQ placement, parameter sharing, feature levels, and computational cost. In particular, the observation that DVQ degrades relatively gracefully under aggressive reductions in the number of vision tokens is potentially practical in resource-contrained settings. In additions, the general-purpose experiments show that DVQ mostly matches rather than consistently surpasses DeepStack at the full token budget.

Critical weaknesses
  • The standard DVQ is not actually language-conditioned, which weakens both the visual querying interpretation and its conceptual distinction from existing visual token compression. My main concern is that DVQ is presented as allowing the LLM to dynamically query visual information, but the spatial refinement of the standard vision queries does not use previous textual tokens. The authors explicitly acknowledge this limitation. This distinction is important, since for many VLM tasks, the subset of visual information that should be preserved is inherently conditional on the user query. For example, the same image may require inspecting a small sign for one question, or a global scene context for the other. If the reference-point trajectories are determined independently of the question, DVQ cannot selectively allocate its limited visual-query budget toward the evidence required by the current instruction. It is therefore closer to an image-adaptive compression mechanism than to genuinely language-conditioned visual querying. I consider this particularly consequential because the claimed advantage of DVQ becomes most important precisely under aggressive token compression, where allocating a small number of queries to task-relevant locations should matter most. The limitation is also structural rather than a missing minor ablation: making the lookup genuinely text-conditioned would require changing the interaction between text and vision queries and may affect the caching and efficiency properties emphasized by the paper. The reported weakness on dense-text/OCR tasks further demonstrates that sparse, task-independent sampling can fail to preserve information required downstream. This weakens the motivation for DVQ. Once the visual selection precess is decoupled from the language query, the distinction from conventional visual token pruning/compression becomes much less consequential, since both mechanisms ultimately attempt to retain a small subset of image information before or during language reasoning. The paper does not demonstrate that DVQ preserves task-relevant evidence better than modern token pruning/compression methods under matched computational budgets.

  • The experiments do not cleanly establish either computational efficiency or the claimed causal role of position grounding. The main comparisons are organized around the number of LLM-side vision tokens, but equal token counts do not correspond to equal computational budgets. DVQ introduces deformable-attention modules repeatedly throughout the LLM and has more connector parameters than DeepStack. Thus, the observation that DVQ performs better than DeepStack at a given value of ∣V∣|V| does not by itself demonstrate a superior accuracy-efficiency trade-off. A compute, memory, or latency matched comparison would be substantially more convincing. The paper also acknowledges that DVQ adds computational and serving complexity relative to Patch Project and DeepStack. More importantly, I do not fine the "DVQ without grounding" experiment sufficient to support the strong claim that "position grounding is reponsible for overcoming the limitations of cross-attention." The ablation replaces deformable attention with standard dot-product attention. This simultaneously changes the positional inductive bias, sparse/local sampling mechanism, attention operator, and computational structure. Consequently, the large performance difference does not isolate position grounding as the causal factor. The paper nevertheless interprets this comparison as evidence that explicit position grounding itself overcomes the limitations of cross-attention. This is important because deformable attention and reference-point refinement are themselves established mechanisms from prior vision architectures. Without an ablation that isolates the positional component while keeping the underlying attention/sampling mechanism controlled, it remains unclear which part of DVQ is actually responsible for the reported gain.

Questions for authors
  • Can the authors provide a genuinely text-conditioned DVQ variant in which reference-point refinement has access to the user question or previously generated language tokens? I consider this more than a future-work suggestion because it directly tests the paper's core interpretation of DVQ as allowing an LLM to query visual information. In particular, I would like to know whether question-conditioned refinement improves performance under aggressive token budgets such as ∣V∣=64|V|=64 or 256256, where task-dependent visual selection should matter most. If this variant substantially improves compression while retaining reasonable computational cost, it would affect my assessment.

  • Can the authors provide a controlled experiment that separates position grounding from the change in attention operator, as well as an actual compute-matched efficiency comparison? For example, could the authors compare variants that retain the same deformable-attention operator but remove or randomize the sptial grounding/refinement mechanism? Conversely, an experiment comparing DVQ and competing connectors at matched FLOPs, latency, or memory rather than only match numbers of LLM vision tokens would clarify whether DVQ provides a better practical efficiency-performance trade-off. These results would materially affect my interpretation of the paper's two central claims.

  • Can the authors provide an actual end-to-end experiment demonstrating why differentiable bounding-box inputs/outputs matter in practice? The paper motivates differentiable boxes primarily through their potential use in end-to-end trainable multi-model systems, yet the current experiments evaluate box prediction and box-conditioned tasks only in isolation. Can the authors demonstrate a downstream setting in which gradients are actually propagated through the predicted bounding boxes across model components, and show that this differentiability provides a meaasureable advantage over conventional text- or token-based box representations?

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Summary

We thank the reviewer for their detailed and insightful feedback. Below, we provide additional experimental evidence and explanations to sharpen our contributions, including:

  • Additional experiments on question-conditioning and comparison with more pruning/compression baselines
  • Additional token efficiency analysis and ablation studies on position grounding While we think that these significantly improve the strength and clarity of our contributions, we thank the reviewer for already acknowledging our thorough empirical evaluation and the strong inductive bias of our method.

1a. Language-conditioning

In its default form, DVQ does not have access to question context as vision query tokens are placed before any question tokens. However, the DVQ module still "queries" the vision information per token and layer, even if these queries are only conditioned on prior vision information and not on the question. We follow this design because placing image context before the question is common practice and easily compatible with multi-round conversations (e.g. in Llava training): each new question can directly look up the relevant information from the image. Placing the image (query) tokens after the question typically performs worse (confirmed by preliminary experiments) as it does not follow the LLMs typical conditioning structure (context first, then question/prompt) . However, DVQ's connector architecture is agnostic to placement in the sequence and also supports placing tokens after the question. To demonstrate this, we have now introduced "dynamic query" tokens that can be placed after the question (see new Tab. 8 in new A.4), with initial reference points predicted based on features after the first LLM layer and refined as always. We explored different mixtures of "static" queries (before the question) and "dynamic" queries (after), finding that a small fraction of "dynamic" queries can improve the performance on some tasks, especially on OCR-heavy tasks. However, the performance can drop if the "dynamic" fraction is too large or the total token budget is too small.

Token BudgetStatic QueriesDynamic QueriesGQATextVQARefCOCOg
102410240 (default)64.653.567.9
102489612863.457.069.2
102476825663.054.361.6
2562560 (default)64.249.863.3
2562243261.444.062.9
64640 (default)63.238.155.4
6456860.034.653.3

This demonstrates that "dynamic" queries can provide informative question-conditioned image-context, but a sufficiently large static query budget is still required (probably because LLMs are used to such a context).

This trade-off between "static" and "dynamic" queries (caused by its placement inside the LLM) leads to DVQ not benefiting from question-conditioning when the token budget is low. Note that caching the static query tokens is still possible but the dynamic queries need to be cached per question.

1b. Comparison with modern token pruning/compression

We note that the already included Patch Project, DeepStack, and Perceiver baselines all apply token compression when token budgets <1024 are used. Patch Project and DeepStack rely on pixel shuffle (see Appendix C.3), which is conceptually close to the C-Abstractor from Honeybee, while Perceiver relies on query-based resampling. Additionally, we now ran Honeybee's D-Abstractor, MQT-LLaVA, TokenPacker, and AcFormer (see new Tab. 7 in A.3) in our small-scale setup. All of these are learnable token compressors. Furthermore, we ran FastV as a non-trainable token pruning method, applied to Patch Project (|V|=1024) and pruned to 256 tokens.

Method (at |V|=256)GQATextVQARefCOCOg
DVQ64.249.863.3
Patch Project56.237.118.2
DeepStack62.249.854.2
Perceiver48.610.29.1
MQT (new)53.411.27.7
Honeybee D-Abstractor (new)54.210.511.3
TokenPacker (new)44.88.76.5
Honeybee C-Abstractor (new)59.915.713.6
AcFormer (new)53.210.08.0
FastV (on Patch Project |V|=1024)60.248.225.6

In general, resampling-based methods (Perceiver, MQT, TokenPacker, D-Abstractor) seem to not work well in our small-scale setting, probably because they need way more training data to perform well. FastV shows strong performance on 2/3 tasks but still underperforms our method, especially on RefCOCOg and GQA.

AuthorsOpenReview ↗

Rebuttal (Part 2)

2a. Equal computational budgets

We want to highlight that the end-to-end efficiency is task-, implementation-, and hardware-dependent. While equal token counts do not correspond to equal computational budgets, we decided to focus on matched token budgets, because this is a good initial estimator of the potential performance across tasks (question/answer lengths), implementations, and hardware. To further estimate the practical costs, we now added a new analysis, comparing the FLOP premiums of our DVQ method (compared to DeepStack) with the resulting accuracy improvements on different tasks (see extended Sec. A.8 with new Fig. 6):

Token BudgetFLOPsGQATextVQARefCOCOg
1024+9.3%+2.5%-11.9%+22.7%
256+5.0%+3.2%-0.0%+16.7%
64+3.0%+5.0%+22.5%+20.2%
16+2.4%+4.2%+66.9%+27.4%
4+2.3%+3.7%+14.3%+65.5%

At full budget DeepStack retains the TextVQA/OCR edge, but already at |V|=256 RefCOCOg buys +16.7% score and GQA +3.2% for +5.0% FLOPs. At |V|<=64 DVQ improves every task by a larger relative margin than the relative extra FLOPs costs (4-67% improvement vs. 2.3-3.0% extra FLOPs), while the extra cost itself shrinks as budgets gets smaller. Note that when the text length |T| (question+answer) increases, the extra FLOPs of the vision connector become less relevant: At |V|=256, the FLOPs premium of DVQ (against DeepStack) is +5.0% at |T|=1024 but only +1.6% at |T|=4096. Due to this, a perfectly FLOP-matched comparison is not possible, but the table above (and the new Fig. 6) should provide some context.

Additionally, we now also report the GPU memory and inference-speeds (see extended Sec. A.8 ). At 1024 tokens our peak memory usage is 10.7 GB (vs. 9.2 GB for DeepStack), while for 256 tokens it is 10.1 GB (vs. 8.71 GB for DeepStack). Our inference prefill (TTFT) is also a bit slower than DeepStack (260.8 ms vs. 164.1 ms at 1024 vision tokens, 134.2 ms vs. 70.8 ms at 256 tokens). After prefill, token generation of DVQ and DeepStack are identical (using only cached LLM hidden features, no need to use the DVQ layer after prefill). Note that we did not optimise our implementation ( generation loop, caching, ...) for inference and did not use any inference-frameworks or inference-optimized hardware. Therefore, the numbers above need to be treated with caution and the FLOPs may be a better estimate for fair comparison.

Finally, our methods also enables other compute cost reduction potentials, e.g. being well-suited for small-scale fine-tuning (see Sec. 4.1) and more robust to smaller LLMs (see A.1).

2b. Causal role of position grounding

We agree that our "position grounding" claim could be sharpened. We point out that the "standard cross-attention variant" only replaces the attention mechanism itself but keeps all other aspects of DVQ intact (including reference points, refinement, and reference position embeddings). While this variant has access to positional information (via reference pos embeddings), it is not explicitly enforced to use it (it is not explicitly grounded). For better clarity, we now explicitly refer to this variant in the paper as "DVQ with standard dot-product attention" instead of "no grounding". To further verify the relevance of the reference points as a central mechanism, we now also added an ablation of DVQ where reference points are randomly resampled at each layer (new Tab. 2 in Sec. 4.1). This lets us disentangle whether DVQ only benefits from locally sparse sampling or whether it actually benefits from this explicit position grounding. Furthermore, we also add an ablations study with "fixed reference points" (with optionally also fixed sampling offsets, new Tab. 2 in Sec. 4.1), to study the relevance of the refinement component.

MethodGQATextVQARefCOCOg
DVQ (default)64.249.863.3
-> random reference points61.821.947.5
-> fixed reference points61.646.963.4
-> fixed reference points + sampling offsets61.828.259.7
-> standard-cross attention56.824.928.7

Performance collapses (especially on TextVQA) when randomly sampling reference points in each DVQ layer, demonstrating that DVQ relies in "positional grounding" via reference points to function. Using fixed reference points, i.e. not refining them over layers, leads to a small performance drop, and if sampling offsets are fixed as well, TextVQA performance collapses.

AuthorsOpenReview ↗

Rebuttal (Part 3, Questions)

Q1. Language-conditioning

See our answer 1a (above). Question-conditioning can improve the performance if enough query tokens remain before the question (as this is where the LLM expects context). Under more aggressive token budgets, using some of the token budget for "dynamic" tokens after the question, degrades the performance, as the remaining "static" (before-question) queries seem to be too few. Note that, unlike many existing "language-conditioned" token compressors, DVQ is directly integrated into the LLM layers, such that it needs to be designed with the LLM's expectations (e.g., context-before-question) in mind.

Q2. Position-grounding and compute-matched

See our answers 2b and 2a, respectively.

Q3. End-to-end differentiable bounding-box experiment

Such an end-to-end differentiable bounding-box experiment would evaluate the DVQ method as part of a larger system. As our primary contribution is an architectural component (the DVQ connector) and not the design of an end-to-end system, we consider such an experiment out-of-scope. In Sec. 4.3, we demonstrated that our DVQ component does enable differentiable bounding box inputs and outputs. Whether this property is needed or beneficial depends on the specific end-to-end system. In Appendix B, we already describe example use cases, but use cases are not limited to these. However, we extended Sec. 5, clarifying that we did not evaluate the end-to-end benefit of differentiable bounding boxes.

Reviewer_wrnYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper puts a deformable-attention module inside every second LLM layer to feed it image information. Each vision token has a point on the image, fixed at a random location at the start; at each of these layers it reads image features around its point at six resolutions, adds them to its hidden state, and moves the point a little. It is compared with Patch Project and DeepStack at 1024, 256 and 64 vision tokens, and also used to take and predict bounding boxes as coordinates.

Critical strengths
  1. It loses less accuracy than DeepStack when vision tokens are cut: at 64 tokens it ties or beats DeepStack on 5 of 10 benchmarks, and on RefCOCOg it drops 13.4% without box pretraining versus 29.7% for DeepStack.
  2. The evaluation is careful and honest: same data and backbone for all connectors, token-budget sweeps, ablations, reported compute cost and stated limitations.
Critical weaknesses
  1. Little new beyond moving an existing module into the LLM, and it does not look where the question asks. The attention is standard deformable attention from Deformable DETR over a standard feature pyramid; the paper itself says the novelty is placing it inside the LLM. The paper states that DVQ "does not consider previous textual tokens when refining its reference points", so for a given image the model samples the same places whatever the question is. That undercuts the title's claim that the LLM "queries" visual information.

  2. No gain at equal cost. With 1024 tokens DVQ scores below DeepStack on 9 of 10 benchmarks and below plain Patch Project on 7 of 10, while using 18% more parameters and 9% more compute. Its smaller drop when tokens are cut starts from a lower score and is helped by VisWiz, where DVQ improves from 51.3 to 58.1 as tokens are removed; without VisWiz the 64-token average drop is 4.9% for DVQ versus 7.7% for DeepStack, not 2.9% versus 9.2%. Its large RefCOCOg gain disappears when DeepStack injects at the same 18 layers (67.9 vs 67.3).

  3. The claims about coverage and about grounding are not tested. The paper says random starting points "prevent spatial query collapse", but the points then move freely over 18 layers, and the paper's own visualizations show them gathering on the main object, with no measurement of how much of the image they still cover. The "no grounding" baseline replaces deformable attention with full attention, which changes several things at once, so it cannot show that the points themselves are what helps.

Questions for authors
  1. How much of the image do the points still cover at each layer, and do results change if they start on a regular grid instead of random positions, or with a different random layout, especially at 64 tokens?
  2. What are the measured speed and memory of DVQ versus DeepStack, and which wins when both are given the same compute rather than the same number of tokens?
  3. Why does VisWiz accuracy go up as vision tokens are removed, and how much do results vary between training runs?
  4. Would letting the question steer where the points move, as the paper already does for its box-prediction tokens, improve results?
  5. Does performance drop if the points are frozen in place while keeping deformable attention, which would show whether moving the points matters?
  6. What are the DVQ width, number of heads and sampling points, and how far from its point can a query read at each resolution?
4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Summary

We thank the reviewer for their detailed and insightful feedback. Below, we provide additional experimental evidence and explanations to sharpen our contributions, including:

  • Additional experiments on question-conditioning
  • Additional analyses including image coverage and token efficiency
  • Additional ablation studies on position grounding While we think that these significantly improve the strength and clarity of our contributions, we thank the reviewer for already acknowledging our careful evaluation and the token reduction robustness of our method.

1a. Novelty

Our key contribution is a novel vision connector for LLMs. While this idea builds on existing components (like deformable attention) and ideas (e.g. multi-scale feature injection into LLMs, using cross-attention to condition the LLM), we think that this combination of ideas is a valuable contribution. Especially note that, to the best of our knowledge, deformable attention has so far not yet been studied inside the LLM, and instead only as an external abstraction component (e.g. D-Abstractor). The in-LLM placement of deformable attention does not only provide robustness to token reduction (see. Fig. 2, Fig. 3) and better high-resolution support (now Tab. 3), but also enables new usage patterns like differentiable bounding box inputs/outputs (now Tab. 4) or question-conditioning (see our answer 1b).

1b. Conditioning on the question

In its default form, DVQ does not have access to question context as vision query tokens are placed before any question tokens. However, the DVQ module still "queries" the vision information per token and layer, even if these queries are only conditioned on prior vision information and not on the question. We follow this design because placing image context before the question is common practice and easily compatible with multi-round conversations (e.g. in Llava training): each new question can directly look up the relevant information from the image. Placing the image (query) tokens after the question typically performs worse (confirmed by preliminary experiments) as it does not follow the LLMs typical conditioning structure (context first, then question/prompt) . However, DVQ's connector architecture is agnostic to placement in the sequence and also supports placing tokens after the question. To demonstrate this, we have now introduced "dynamic query" tokens that can be placed after the question (see new Tab. 8 in new A.4), with initial reference points predicted based on features after the first LLM layer and refined as always. We explored different mixtures of "static" queries (before the question) and "dynamic" queries (after), finding that a small fraction of "dynamic" queries can improve the performance on some tasks, especially on OCR-heavy tasks. However, the performance can drop if the "dynamic" fraction is too large or the total token budget is too small.

Token BudgetStatic QueriesDynamic QueriesGQATextVQARefCOCOg
102410240 (default)64.653.567.9
102489612863.457.069.2
102476825663.054.361.6
2562560 (default)64.249.863.3
2562243261.444.062.9
64640 (default)63.238.155.4
6456860.034.653.3

This demonstrates that "dynamic" queries can provide informative question-conditioned image-context, but a sufficiently large static query budget is still required (probably because LLMs are used to such a context).

2. No gain at equal cost

Yes, our DVQ method does not outperform DeepStack at the full token budget in general-purpose vision-language modeling. However, even at full scale it often closely matches DeepStack's performance, reaching at least 95% of DeepStack's performance on 8/10 tasks (see Tab. 9). We have clarified the claim in the contributions, replacing "is competitive" by "closely matches or exceeds [...] on most tasks".

However, note that our primary contributions are:

  • Robustness in smaller-scale settings, such as our small-scale scenario in Sec. 4.1, smaller token budgets, and smaller LLMs (Appendix A1)
  • A mechanism that is easy to train and overcomes the limitations of cross-attention-based connectors We think that we could demonstrate these with our experiments (see also Discussion in Sec. 5).
AuthorsOpenReview ↗

Rebuttal (Part 2)

3a. Coverage claim not tested

To quantitatively validate our coverage claim, we added a new analysis measureing the spatial query collapse in the standard attention variant and comparing it to DVQ (new Tab. 5 in Sec. 4.4). We measured the cell-hit ratio (cells@32, fraction of 32x32 grid cells with at least one query) at DVQ layers 1, 9, and 17, at 1024 vision tokens. For standard cross-attention we consider cells covered if they are in the top1, top3, and top5 highest attention probabilities of a query token:

MethodL1 [cells@32]L9 [cells@32]L17 [cells@32]
DVQ0.4310.4470.448
Cross-att (top-1)0.0380.0320.017
Cross-att (top-3 tokens)0.0790.0650.036
Cross-att (top-5 tokens)0.1120.0900.051

We observe that DVQ always has a significantly better cell coverage, i.e. it receives information from more regions of the image. Coverage remains stable, i.e. refinement adapts individual queries but does not decrease general coverage. For Standard cross-attention, coverage is much lower and degrades (collapses) over the layers.

3b. Grounding claim not tested

We agree that our "grounding" claim could be sharpened. We point out that the "standard cross-attention variant" only replaces the attention mechanism itself but keeps all other aspects of DVQ intact (including reference points, refinement, and reference position embeddings). While this variant has access to positional information (via reference pos embeddings), it is not explicitly enforced to use it (it is not explicitly grounded). For better clarity, we now explicitly refer to this variant in the paper as "DVQ with standard dot-product attention" instead of "no grounding". To further verify the relevance of the reference points as a central mechanism, we now also added an ablation of DVQ where reference points are randomly resampled at each layer (new Tab. 2 in Sec. 4.1). This lets us disentangle whether DVQ only benefits from locally sparse sampling or whether it actually benefits from this explicit position grounding.

MethodGQATextVQARefCOCOg
DVQ (default)64.249.863.3
-> random reference points61.821.947.5
-> standard-cross attention56.824.928.7

Performance collapses (especially on TextVQA) when randomly sampling reference points in each DVQ layer, demonstrating that DVQ relies in "positional grounding" via reference points to function.

AuthorsOpenReview ↗

Rebuttal (Part 3, Questions 1-3)

Q1. Coverage and regular grid initialization

We now compare the coverage (cells@32, fraction of 32x32 grid cells with at least one query) between our default initial reference points (random uniform sampling) with grid initialization, at token budgets 256 and 64.

MethodL1 [cells@32]L9 [cells@32]L17 [cells@32]
DVQ (256, default)0.1670.1550.159
DVQ (256, grid init)0.1810.1890.183
DVQ (64, default)0.0490.0490.049
DVQ (64, grid init)0.0530.0540.054

As expected, coverage decreases when reducing the token budget (as we then have fewer query tokens than grid cells). Between our default DVQ and DVQ with grid initialization, there is only a minor difference, with grid initialization resulting in slightly better coverage. However, note that this has minimal and task-dependent impacts on accuracy:

  • 256 tokens (default -> grid): GQA 64.2 -> 63.0, TextVQA 49.8 -> 54.1, RefCOCOg 63.3 -> 65.2
  • 64 tokens (default -> grid): GQA 63.2 -> 61.8, TextVQA:38.1 -> 36.3, RefCOCOg 55.4 -> 56.3 Also see our answer 3a for more coverage analyses.

Q2. Speed and Memory

To further estimate the practical costs, we added a new analysis (see extended Sec. A.8 with new Fig. 6), comparing the FLOP premiums of our DVQ method (compared to DeepStack) with the resulting score improvements on different tasks:

Token BudgetFLOPsGQATextVQARefCOCOg
1024+9.3%+2.5%-11.9%+22.7%
256+5.0%+3.2%-0.0%+16.7%
64+3.0%+5.0%+22.5%+20.2%
16+2.4%+4.2%+66.9%+27.4%
4+2.3%+3.7%+14.3%+65.5%

At full budget DeepStack retains the TextVQA/OCR edge, but already at |V|=256 RefCOCOg buys +16.7% score and GQA +3.2% for +5.0% FLOPs. At |V|<=64 DVQ improves every task by a larger relative margin than the relative extra FLOPs costs (4-67% improvement vs. 2.3-3.0% extra FLOPs), while the extra cost itself shrinks as budgets gets smaller. Note that when the text length |T| (question+answer) increases, the extra FLOPs of the vision connector become less relevant: At |V|=256, the FLOPs premium of DVQ (against DeepStack) is +5.0% at |T|=1024 but only +1.6% at |T|=4096. Due to this, a perfectly FLOP-matched comparison is not possible, but the table above (and the new Fig. 6) should provide some context.

Additionally, we now also report the GPU memory and inference-speeds (see extended Sec. A.8). At 1024 tokens our peak memory usage is 10.7 GB (vs. 9.2 GB for DeepStack), while for 256 tokens it is 10.1 GB (vs. 8.71 GB for DeepStack). Our inference prefill (TTFT) is also a bit slower than DeepStack (260.8 ms vs. 164.1 ms at 1024 vision tokens, 134.2 ms vs. 70.8 ms at 256 tokens). After prefill, token generation of DVQ and DeepStack are identical (using only cached LLM hidden features, no need to use the DVQ layer after prefill). Note that we did not optimise our implementation ( generation loop, caching, ...) for inference and did not use any inference-frameworks or inference-optimized hardware. Therefore, the numbers above need to be treated with caution and the FLOPs may be a better estimate for fair comparison.

We want to highlight that the end-to-end efficiency is task-, implementation-, and hardware-dependent. We decided to focus on matched token budgets, because this is a good initial estimator of the potential performance across tasks (question/answer lengths), implementations, and hardware.

Q3. VisWiz accuracy

The improvement of VisWiz seems to be monotonic over the reduction 1024->256->64 and we have observed similar behaviour in early exploration runs. We assume it is caused by noise in the images and that a reduction in token budget forces the model to focus on the more relevant (less-noisy) parts, thus improving performance.

AuthorsOpenReview ↗

Rebuttal (Part 4, Questions 4-6)

Q4. Question steering

See answer 1b. In brief: we can place some query tokens after the question and this "question-steering" can help, but only in some specific cases, i.e. if enough query tokens are still placed before the question (probably as that is where the LLM expects context).

Q5. Freezing reference points

To validate whether whether moving the points matters, we now added an ablations study with "fixed reference points" (new Tab. 2 in Sec. 4.1), where the initial reference points cannot be refined over the layers, and an ablation study with additionally fixed (non-learnable) sampling offsets:

MethodGQATextVQARefCOCOg
DVQ (default)64.249.863.3
-> fixed reference points61.646.963.4
-> fixed reference points + sampling offsets61.828.259.7

Fixed reference points lead to performance drops on 2/3 tasks (while having no impact on RefCOCOg). If we fix the sampling offsets as well, then TextVQA performance completely collapses. This shows the relevance of refinement.

Q6. DVQ Configuration

In all our experiments, we use d=1024 inside the DVQ module (DVQ width), which is slightly smaller than the image backbone dimension (d=1152) and much smaller than the LLM hidden size (d=2560). We use 8 attention heads with 4 sampling points per head and level, leading to 192 feature locations (8 heads x 6 levels x 4 points) per query and layer. The sampling range is initially restricted to a 0.5x0.5 window centered on each query's reference point (so half of the image size in each direction), but the model can also refine the sampling range (similar to the reference points). In practice, however, the model stays extremely focused, reducing the sampling range to 0.025x0.025 until layer 9 and to 0.014x0.014 until the last layer (on average). Note that the sampling range is resolution-agnostic and defined relative to the image resolution.

Reviewer_TTQYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Deformable Vision Query (DVQ) Connector to provide an LLM with fine-grained image information under a limited visual-token budget. Each query starts from a spatial reference point, reads nearby multi-scale features through deformable cross-attention at selected LLM layers, and updates its reference point across layers; reference boxes also support differentiable bounding-box inputs and outputs. The experiments cover connector pretraining on COCO Captions followed by task-specific fine-tuning on GQA, TextVQA, and RefCOCOg with the vision encoder and LLM frozen, as well as general-purpose VLM training using the LLaVA-1.5 recipe. DVQ retains more performance than DeepStack as the token budget shrinks, but does not consistently surpass it at the full budget or on OCR-related tasks.

Critical strengths
  1. Well-motivated problem. As image resolution increases, passing a large number of patch tokens directly to the LLM raises computational and context costs, while fixed spatial aggregation may discard small objects or local details. Achieving both image coverage and content-adaptive sampling with a small number of queries is a practically important problem.

  2. Coherent method with a clear interface. DVQ combines reference points, sparse deformable sampling, cross-layer query updates, and visual information injection within LLM layers. Reference points give visual queries explicit spatial meaning, while a fixed number of queries controls the visual-token budget. Extending reference points to continuous reference boxes for spatial inputs and outputs is also a promising direction.

  3. Good robustness to reduced token budgets. In the general-purpose VLM experiments, DVQ exhibits a smaller average performance drop than DeepStack when the visual-token budget is reduced from 1,024 to 256 or 64.

Critical weaknesses
  1. Insufficient attribution of the core mechanism. The paper largely attributes the substantial performance drop of the standard cross-attention variant to the absence of position grounding. However, this replacement simultaneously changes other mechanisms, including global dense attention versus local sparse sampling, so the contribution of position grounding cannot be isolated. The paper also does not directly quantify whether spatial query collapse occurs in this variant. Moreover, the existing ablations do not systematically disentangle the effects of fixed versus layerwise-updated reference points, reference-point positional encodings, random uniform versus regular-grid initialization, and predicted sampling offsets.

  2. Limited controlled comparisons with related position-aware and token-compression connectors. Beyond DeepStack, Honeybee [1], TokenPacker [2], and AcFormer [3] use locality, regional information, or visual anchors, respectively, to aggregate visual features, while MQT-LLaVA [4] studies variable token budgets. Although their architectures differ from DVQ, they address related problems. At least a few strong baselines should be compared under the same LLM backbone, vision encoder, training data, and visual-token or compute budget.

[1] Honeybee: Locality-enhanced Projector for Multimodal LLM
[2] TokenPacker: Efficient Visual Projector for Multimodal LLM
[3] Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
[4] Matryoshka Query Transformer for Large Vision-Language Models

  1. Robustness to token reduction does not translate into consistently stronger absolute performance. At the full visual-token budget, DVQ trails DeepStack trained under the same setting on most general-purpose VLM benchmarks, particularly TextVQA. Its gains at reduced budgets are task-dependent, while its RefCOCOg advantage narrows or reverses when DeepStack uses the same number of injection layers. The results therefore support greater robustness to token reduction more clearly than superior absolute performance.

  2. The end-to-end efficiency benefit of being “token-efficient” remains unclear. At the same visual-token budget, DVQ still performs deformable sampling and visual information injection at multiple LLM layers. Although Appendix Table 8 compares parameter counts and estimated TFLOPs for DVQ and DeepStack, the paper does not report actual inference latency, peak memory use, training throughput, end-to-end costs at different resolutions, or accuracy under matched FLOPs or latency. Its practical efficiency gains therefore remain uncertain.

  3. Limited validation under larger-scale training. The general-purpose VLM experiments use the LLaVA-1.5 pretraining and instruction-tuning data. This standard recipe helps control experimental variables, but the current results do not establish whether DVQ would retain the same relative advantages with larger and more diverse vision-language training data. A controlled evaluation under a more recent data regime would help, and the conclusions should be limited to the training scales tested.

  4. Insufficient targeted evaluation of fine-grained visual capabilities. The paper covers nine general-purpose VLM benchmarks, but lacks targeted tests of capabilities that spatially adaptive sampling may affect. For counting, CountBench and PixMo-Count could be added. Ref-L4 would provide a more challenging test of 2D referring-expression grounding. DocVQA and InfoVQA could assess text-rich document understanding, while AI2D and ChartQA could assess diagram and chart understanding.

Questions for authors
  1. The paper evaluates inputs up to 1,024 × 1,024. As resolution increases further, or when an image contains many small objects and complex relations among regions, can DVQ's sparse sampling retain the information needed for the task? Do the authors observe systematic omissions on visually dense tasks such as dense captioning?

  2. In Table 2, increasing the input resolution decreases DVQ's RefCOCOg score from 67.9 to 66.3, and DeepStack's from 55.3 to 49.2. How do the authors explain the decline in localization performance for both methods? After the features from four high-resolution crops are reassembled, how are DVQ's normalized reference coordinates aligned with the original image?

4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Summary

We thank the reviewer for their detailed and insightful feedback. Below, we provide additional experimental evidence and explanations to sharpen our contributions, including:

  • Additional ablation studies on the core mechanism and image coverage analysis
  • Additional position-aware baselines and benchmarks
  • Additional token efficiency analysis While we think that these significantly improve the strength and clarity of our contributions, we thank the reviewer for already acknowledging the soundness of our paper (soundness: 3) and highlighting our strong motivation as well as our coherent method and robustness to reduced token budgets.

1a. Insufficient attribution of the core mechanism (grounding)

We agree that the attribution to the core mechanism could be sharpened. We point out that the "standard cross-attention variant" only replaces the attention mechanism itself but keeps all other aspects of DVQ intact (including reference points, refinement, and reference position embeddings). While this variant has access to positional information (via reference pos embeddings), it is not explicitly enforced to use it (it is not explicitly grounded). For better clarity, we now explicitly refer to this variant in the paper as "DVQ with standard dot-product attention" instead of "no grounding". To further verify the relevance of the reference points as a central mechanism, we now also added an ablation of DVQ where reference points are randomly resampled at each layer (new Tab. 2 in Sec. 4.1). This lets us disentangle whether DVQ only benefits from locally sparse sampling or whether it actually benefits from this explicit position grounding.

Additionally, we now provide further ablation studies on

  • fixed versus layerwise-updated reference points ("fixed reference points")
  • reference-point positional encodings ("no pos embeddings")
  • random uniform vs. regular-grid initialization ("grid init") vs. hierarchical pyramid grid ("pyramid grid init")
  • predicted sampling offsets ("fixed reference points + sampling offsets") (see new Tab. 2 in Sec. 4.1 and new Tab. 11 in A.6)
MethodGQATextVQARefCOCOg
DVQ (default)64.249.863.3
-> random reference points61.821.947.5
-> fixed reference points61.646.963.4
-> fixed reference points + sampling offsets61.828.259.7
-> no pos embeddings63.251.662.9
-> grid init63.054.165.2
-> pyramid grid init62.854.464.8
standard-cross attention56.824.928.7

Performance collapses (especially on TextVQA) when randomly sampling reference points in each DVQ layer, demonstrating that DVQ relies on "positional grounding via reference points" to function. Fixing reference points (i.e. preventing refinement) leads to performance drops on 2/3 tasks. If we fix the sampling offsets as well, then TextVQA performance completely collapses, showing the relevance of refinement. Removing the position embeddings for reference points has minimal impact on performance, probably as DVQ also uses the reference points to directly steer deformable attention. Grid and pyramid grid initialization perform slightly worse on GQA, similar on RefCOCOg, but better on TextVQA. Overall, the impact of initialization does not seem to be drastic, but grid-based initializations may be beneficial for fine-grained tasks like OCR.

1b. Coverage/collapse analysis

We now added an analysis to quantify the spatial query collapse in the standard attention variant and compare it to DVQ (new Tab. 5 in Sec. 4.4). We measured the cell-hit ratio (cells@32, fraction of 32x32 grid cells with at least one query) at DVQ layers 1, 9, and 17, at 1024 vision tokens. For standard cross-attention we consider cells covered if they are in the top1, top3, and top5 highest attention probabilities of a query token:

MethodL1 [cells@32]L9 [cells@32]L17 [cells@32]
DVQ0.4310.4470.448
Cross-att (top-1)0.0380.0320.017
Cross-att (top-3 tokens)0.0790.0650.036
Cross-att (top-5 tokens)0.1120.0900.051

We observe that DVQ always has a significantly better cell coverage, i.e. it receives information from more regions of the image. Coverage remains stable, i.e. refinement adapts individual queries but does not decrease general coverage. For standard cross-attention, coverage is much lower and degrades over the layers. This indicates a clear collapse of region coverage, i.e. attention focuses on localized parts of the image.

AuthorsOpenReview ↗

Rebuttal (Part 2)

2. Limited controlled comparisons with related position-aware and token-compression connectors

We highlight that the already included Patch Project and DeepStack both already rely on position-aware token compression when token budgets <1024 are used. Both rely on pixel shuffle (see Appendix C.3), which is conceptually close to the C-Abstractor from Honeybee [1]. However, we have now also included the C- and D-Abstractor from Honeybee [1], TokenPacker[2], AcFormer [3] and MQT-LLaVA [4] in our small-scale experiments at 256 tokens (see new Tab. 7 in A.3):

MethodQuery Position RestrictionLLM injectionGQATextVQARefCOCOg
Patch Projectraster (pixel shuffle)embedding layer only56.237.118.2
Honeybee C-Abstractor (new)rasterembedding layer only59.915.713.6
Perceivernoneembedding layer only48.610.29.1
MQT (new)none (but uses pos-embedding)embedding layer only53.411.27.7
AcFormer (new)none (but uses patch-token-baed queries)embedding layer only53.210.08.0
TokenPacker (new)none (but uses patch-token-baed queries)embedding layer only44.88.76.5
Honeybee D-Abstractor (new)localityembedding layer only54.210.511.3
DeepStackraster (pixel shuffle)LLM layers62.249.854.2
Standard cross attentionnone (but uses pos-embedding)LLM layers56.824.928.7
DVQlocalityLLM layers64.249.863.3

Performance is influenced primarily by two factors: (i) are the query position-grounded (i.e. locally restricted) and (ii) does the connector directly inject information into the LLM layers. Injecting information directly into the LLM layers seems to perform better, especially on the TextVQA and RefCOCOg tasks. Raster-based compression (spatial aggregation) methods are very robust, while locality-restriction (position-grounding) via deformable attention also seems to be beneficial. In general, resampling-based methods (Perceiver, MQT, TokenPacker, D-Abstractor) seem to not work well in our small-scale setting, probably because they need way more training data to perform well. Reproducing these baselines on the full Llava-scale is not feasible, but in Tab. 9 (A.7) we had already included reproductions of DeepStack and Perceiver, and showed the originally published scores for MQT-LLaVA [4] and TokenPacker[2], which should provide context to judge the performance of our DVQ method. (Also consider that existing works rarely provide full reproductions of baselines and our primary focus is the small-scale setup.)

To further extend the comparison, we have now also added the training-free token-pruning method FastV (see extened Tab. 1 in Sec. 4.1) in our small-scale experiment (applied to Patch Project at |V|=1024 and pruned to 256 tokens):

Method (at |V|=256)GQATextVQARefCOCOg
DVQ64.249.863.3
Patch Project (pixel shuffle)56.237.118.2
FastV (on Patch Project |V|=1024)60.248.225.6
FastV shows strong performance, outperforming pixel shuffle based compression (Patch Project), but still underperforms our method, especially on RefCOCOg and GQA.

3. Robustness to token reduction does not translate into consistently stronger absolute performance.

Yes, our DVQ method does not outperform DeepStack at the full token budget in general-purpose vision-language modeling. However, even at full scale it often closely matches DeepStack's performance, reaching at least 95% of DeepStack's performance on 8/10 tasks (see old Tab. 9, now Tab. 14). We have clarified the claim in the contributions, replacing "is competitive" by "closely matches or exceeds [...] on most tasks".

However, note that our primary contributions are:

  • Robustness in smaller-scale settings, such as our small-scale scenario in Sec. 4.1, smaller token budgets, and smaller LLMs (Appendix A1)
  • A mechanism that is easy to train and overcomes the limitations of cross-attention-based connectors We think that we could demonstrate these with our experiments (see also Discussion in Sec. 5).
AuthorsOpenReview ↗

Rebuttal (Part 3)

4. The end-to-end efficiency benefit of being “token-efficient” remains unclear

We want to highlight that the end-to-end efficiency is task-, implementation-, and hardware-dependent. We decided to focus on matched token budgets, because this is a good initial estimator of the potential performance across tasks (question/answer lengths), implementations, and hardware. To further judge the efficiency-accuracy trade-offs, we **added a new analysis, comparing the FLOP premiums of our DVQ method (compared to DeepStack) with the resulting accuracy improvements on different tasks (see extended Sec. A.8 with new Fig. 6) **:

Token BudgetFLOPsGQATextVQARefCOCOg
1024+9.3%+2.5%-11.9%+22.7%
256+5.0%+3.2%-0.0%+16.7%
64+3.0%+5.0%+22.5%+20.2%
16+2.4%+4.2%+66.9%+27.4%
4+2.3%+3.7%+14.3%+65.5%

At full budget DeepStack retains the TextVQA/OCR edge, but already at |V|=256 RefCOCOg buys +16.7% score and GQA +3.2% for +5.0% FLOPs. At |V|<=64 DVQ improves every task by a larger relative margin than the relative extra FLOPs costs (4-67% improvement vs. 2.3-3.0% extra FLOPs), while the extra cost itself shrinks as budgets gets smaller. Note that when the text length |T| (question+answer) increases, the extra FLOPs of the vision connector become less relevant: At |V|=256, the FLOPs premium of DVQ (against DeepStack) is +5.0% at |T|=1024 but only +1.6% at |T|=4096. Due to this, a perfectly FLOP-matched comparison is not possible, but the table above (and the new Fig. 6) should provide some context.

Additionally, we now also report the GPU memory and inference prefill speeds (see extended Sec. A.8). At 1024 tokens our peak memory usage is 10.7 GB (vs. 9.2 GB for DeepStack), while for 256 tokens it is 10.1 GB (vs. 8.71 GB for DeepStack). Our inference prefill (TTFT) is also a bit slower than DeepStack (260.8 ms vs. 164.1 ms at 1024 vision tokens, 134.2 ms vs. 70.8 ms at 256 tokens). After prefill, token generation of DVQ and DeepStack are identical (using only cached LLM hidden features, no need to use the DVQ layer after prefill). Note that we did not optimise our implementation ( generation loop, caching, ...) for inference and did not use any inference-frameworks or inference-optimized hardware. Therefore, the numbers above need to be treated with caution and the FLOPs may be a better estimate for fair comparison. Regarding image resolutions: our DVQ-method is invariant to the original image resolution, only the number of feature-map levels (but not their size) and the number of queries matter. Therefore, increasing the image resolution only adds extra cost at the image encode but not at the DVQ module.

Finally, our methods also enables other compute cost reduction potentials, e.g. being well-suited for small-scale fine-tuning (see Sec. 4.1) and more robust to smaller LLMs (see A.1).

5. Limited validation under larger-scale training

Our primary contributions focus on smaller-scale setups, including small-scale fine-tuning, reduced token budgets, and small LLMs. We included the LLaVA experiments because they enable standard benchmarking against other published results, and the LLaVA dataset is not small (about 1M total samples). Scaling further is well beyond the scope of this paper (which does not primarily target large-scale scenarios). It is neither feasible within the rebuttal period nor are there common standard training datasets for larger scales (these vary a lot between publications). However, we now made this limitation explicit in Sec. 5.

AuthorsOpenReview ↗

Rebuttal (Part 4)

6. Insufficient targeted evaluation of fine-grained visual capabilities

While our tasks already cover global image understanding (e.g. GQA, VQAv2), OCR-heavy tasks (e.g. TextVQA, VisWiz), referring-expression grounding (RefCOCOg), and diagram/chart understanding (SciQA), we have now additionally evaluated DVQ (and DeepStack) on the following proposed benchmarks (see new Tab. 15 in A.10):

BenchmarkTypeDVQ (V=1024)DeepStack (V=1024)DVQ (V=256)DeepStack (V=256)DVQ (V=64)DeepStack (V=64)
CountBenchcounting97.097.097.498.898.697.8
PixMo-Countcounting46.656.444.951.140.639.9
Ref-L4Referring-expression grounding77.282.768.678.050.371.5
DocVQAtext-rich document29.541.724.627.817.219.6
InfoVQAtext-rich document23.625.122.322.321.620.1
AI2Ddiagram/chart66.867.166.165.764.464.4
ChartQAdiagram/chart16.127.015.916.813.715.3

On these new benchmarks, we observed similar patterns as on OCR-heavy tasks: On the full token budget, DVQ typically underperforms DeepStack, while with reduced token budgets DVQ remains more stable, closing the gap to DeepStack. However, note that, even at full token budget, DVQ closely matches DeepStack on CountBench, InfoVQA, and AI2D, demonstrating that DVQ can provide counting, text-rich, and diagram/chart understanding capabilities. While DeepStack seems stronger in these capabilities at full token budget, its pixel-shuffle compression demonstrates similar behaviour as DVQ at smaller token budgets, showing that grid-based compression is not generally better than DVQ's adaptive sampling. Only on Ref-L4, we see a slightly different result: while on the full budget DVQ is only slightly underperforming DeepStack, that gap widens with token reduction.

Q1. Information Retainment with High-resolution and Dense Tasks

DVQ's deformable attention mechanism can directly sample at any feature map size across several scales at once. In our high-resolution scenario we therefore simply added another feature map scale based on the reassembled high-resolution crops. The reference coordinates are defined relative to the reassembled crops, i.e. relative to the full image size, not relative to the size of individual crops. This, this reassembled feature map can be handled like any other scale. As the low-resolution scales are still available, we expect DVQ's sparse sampling to retain information even at high resolutions (scores even improve on 2/3 tasks and barely drop on the third). However, retaining information for many small objects can in some cases be challenging. While counting on CountBench performs well, performance on PixMo-Count is lower than DeepStack at 1024 tokens. The performance on visually dense tasks is also inconsistent, but some diagram/chart or text-rich document tasks seem to be challenging (see answer 6) However, DVQ's sparse sampling seems to be better at retaining information than grid-based compression (DeepStack with Pixel shuffle at <=256 tokens).

Q2

In both cases it may be challenging to combine the high-resolution and low-resolution features for the RefCOCOg task, while the benefits of high-resolution features may be marginal (unlike for TextVQA where they are beneficial). However, DVQ's drop is marginal, probably, because it can directly sample from both low- and high-resolution feature maps (see Q1). For DeepStack, the high-resolution feature maps replace the low-resolution features in the deep-stacked layers, such that accessing low-resolution information may be harder, explaining the larger drops.

For DVQ: The reference coordinates are defined relative to the reassembled crops, i.e. relative to the full image size, not relative to the size of individual crops. Thus, these feature maps can be used as an additional scale for deformable attention (see also Q1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.