acceptodds
Under review as a conference paper at ICLR 2027

Once Is Enough: Interaction Timing in Dual-Core Vision– Language Detection

Abstract

Vision–language detectors typically intertwine two architectural decisions: when visual and textual representations interact and how often this interaction is repeated. We study these variables independently through a controlled dual-core detector that first constructs semantic text cores and reference-aware visual cores, and then varies only the placement and frequency of cross-modal exchange. Holding representation size, decoder depth, initialization, and optimization fixed, a single interaction immediately after core construction consistently outperforms early fusion, middle or last-layer interaction, and four interactions distributed across the decoder. Importantly, this result persists when core capacity is increased: with 128 text cores, the single-interaction model matches the dense detector on COCO, whereas repeated interaction reduces AP by 1.23 points. The same ordering transfers to a larger Grounding DINO backbone, where repeated interaction incurs a 1.27-point drop, and remains compatible with COCO-to-LVIS zero-shot transfer. Mechanism ablations show that semantic text resampling and reference-aware visual refinement are both necessary, while preserving the original text tokens at the matching head protects the pretrained vision–language alignment interface. These findings establish interaction timing as a first-order design variable independent of interaction frequency and support a phase-separated decoder paradigm: construct modality-specific representations first, exchange information once, and continue visually grounded refinement afterward.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision–Language Detection, Multimodal Compression, Dual-Core Representation, Efficient Cross-Modal Interaction, Open-Vocabulary Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qzVyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two coupled design choices in vision-language object detectors: where visual and textual representations first interact, and how many times cross-modal interaction is repeated. The proposed Dual-Core architecture constructs learned text cores using a two-layer resampler and visual cores using top-ranked Grounding DINO proposals followed by reference-aware refinement. The main empirical finding is that one post-core interaction in the first decoder layer performs better than a pre-core bridge, a middle-layer interaction, a last-layer interaction, or four interactions spread over the decoder.

Critical strengths

The interventions of when and how often to let vision and language interact, is carefully experimented with. The finding that one interaction can suffice, is interesting. Keeping dense visual memory for deformable attention and retaining the original text tokens at the final matching head looks technically reasonable.

Critical weaknesses

The method figure seems oversimplified and doesn't really help in understanding the method better. The figure has limited value in this way. I want to understand the mechanisms better and the figure doesn't show that.

That one interaction between vision-text can suffice, is interesting, but also a bit disappointing. It is basically what several models do already. The question is whether the finding of retaining text cross-attention only in decoder layer 1, and that this works better than four alternatives, is interesting enough for the ICLR audience.

Only one detector is tested: Grounding DINO architecture. This is insufficient to establish transfer of the claimed principle to other architectures.

The paper does not discuss sufficiently the prior work on gating, sharing or sparsifying cross-modal attention across transformer layers. E.g. query bottlenecks and Q-Former are not considered.

Questions for authors

Why is {1,2,4,6} the representative repeated schedule?

What exactly is new in the “visual core” relative to Grounding DINO’s existing proposal/query?

Can you demonstrate the proposed mechanism directly, for example by measuring how localization, class evidence, or representation similarity evolves after each decoder interaction?

Can you expand related work on cross-modal interactions, multimodal bottlenecks, in order to clearly state the closest prior architecture and the difference.

Reviewer_6TMyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies two architectural variables in vision–language detectors: when visual and textual representations interact (timing) and how often they interact (frequency). Starting from a pretrained Grounding DINO, the authors build a "Dual-Core" decoder. A two-layer learnable resampler compresses the encoded prompt into KtK_t semantic text cores (64 by default). The top KvK_v=300 encoder proposals, refined by one gated, reference-aware self-attention layer, serve as visual cores. Text cross-attention is kept in only one decoder layer and physically removed from the others, while dense deformable visual-memory attention and the original full-text matching head are preserved. With core size, decoder depth, initialization and a decoder-scoped continuation recipe held fixed, the paper compares single-interaction schedules at different positions ({pre}, {1}, {3}, {6}) and a four-interaction schedule {1,2,4,6}. On COCO with Grounding DINO-T, interacting once in decoder layer 1 performs best (47.74 AP vs. 46.61–47.00 for other single-interaction placements and 47.10 for four interactions). With KtK_t=128 ("High-Config"), the single-interaction model matches the dense detector (49.45 vs. 49.36 AP), while four interactions lose 1.23 AP; a similar 1.27-AP gap is reported on a larger Grounding DINO-S (Swin-L) backbone. Additional experiments cover COCO-to-LVIS transfer, mechanism ablations of both core branches, a core-capacity sweep, prompt-distractor stress tests, and comparisons with ToMe/DynamicViT-style visual compression. The paper concludes that interaction timing is a first-order design variable independent of frequency, and advocates a phase-separated decoder that forms modality-specific representations first and exchanges information once.

Critical strengths
  1. A clean question with a reasonably controlled design. Separating the timing of cross-modal exchange from its frequency is a well-posed and potentially useful question for vision–language detector design. The paper fixes core size, depth, initialization and training budget while varying only the schedule, and reports the high-capacity control (KtK_t=128) that addresses the obvious "weak core" alternative explanation.

  2. Careful preservation of the pretrained interface, with honest scoping. Keeping dense visual memory and the full-text matching head while compressing only decoder communication is a sensible design, supported by mechanism ablations (visual refinement, resampler depth). The authors are also candid about limitations: they explicitly say the +0.09 AP differences are parity rather than superiority, and that there is no wall-clock gain.

Critical weaknesses
  1. The frequency effect may be an artifact of the training recipe rather than an architectural property. All variants are trained with a restricted decoder-only continuation objective (no auxiliary decoder losses, no denoising) starting from a checkpoint whose text cross-attention modules were pretrained on full text tokens. In {1,2,4,6}, three additional pretrained (or newly gated) cross-attention modules must adapt to the new core representation under the same limited budget, which by itself could explain the drop. Notably, the dense model, which interacts in all six layers, is not harmed by frequency (49.36 AP). Without varying the training recipe or budget, or testing repeated interaction with full text tokens, the causal claim that "repeated fusion is detrimental after core formation" is not established.

  2. The statistical evidence is thin and shows unusual regularities. The timing study (Table 3), which supports the central claim, reports means only. With standard deviations around 0.15–0.17 elsewhere, the differences among {pre}, {3} and {6} (0.11–0.39 AP) are likely within noise, and only {1} clearly stands out. In addition, High-Config exceeds the dense baseline by almost exactly the same margin in every setting (+0.09 AP on COCO-T, DINO-S and LVIS, with +0.08–0.11 on each sub-metric), with identical standard deviations for both models in each table. Per-seed results would help confirm these numbers.

  3. The timing conclusion is drawn in a capacity-limited regime and is not verified where it matters. With KtK_t=64 cores for an 80-category prompt, the compact model has fewer cores than categories. The capacity sweep shows a disproportionate jump from 64 to 128 cores (+1.71 AP vs. +0.62 from 32 to 64). Yet the timing sweep is run only at KtK_t=64 on DINO-T. Since the paper itself argues that weak cores can confound the frequency result, the timing ordering should also be shown at KtK_t=128 and on the second backbone.

  4. The scope is too narrow for the generality of the claims. The evidence comes from one detector family, one training recipe, five schedules, and only two frequency values (mm=1 vs. one specific mm=4 set). Intermediate frequencies (mm=2, 3, 6) and other placements at mm>1 are missing, so "timing is independent of frequency" is not actually tested as an interaction. The title and phrases such as "establish interaction timing as a first-order design variable" overgeneralize from this evidence.

  5. No practical benefit, so the paper rests entirely on the empirical finding. Appendix C states that Compact shows no reliable wall-clock gain over dense fusion and that the interaction count is not a FLOP or latency measure; High-Config only matches dense accuracy. The method is therefore not a contribution on its own, and the paper's value depends on the generality of the timing/frequency finding, which (per points 1–4) is not yet convincingly established.

  6. Several experimental details are unclear. It is unclear which public checkpoint "Grounding DINO-S with a Swin-L backbone" refers to in Hugging Face Transformers, and how the dense baselines relate to the numbers reported in the original Grounding DINO paper. The LVIS evaluation is called zero-shot, although the pretrained model was trained on large detection/grounding corpora whose vocabularies overlap LVIS. Parts of the related work (e.g., NuWa on class-specific ViTs, PeakPatch on negation in CLIP, EdgeCrafter on distillation) are only tangentially related to the question studied.

Questions for authors
  1. Per-seed results and variance. Please report per-seed AP for all main tables, standard deviations for Table 3 and Table 6, and significance tests for the timing comparisons. Can you explain why High-Config exceeds the dense baseline by nearly the same +0.09 AP, with identical standard deviations, on COCO-T, COCO-S and LVIS?

  2. Source of the frequency penalty. In {1,2,4,6} and the other single-interaction schedules, are the text cross-attention modules initialized from the pretrained weights, newly initialized, or gated? Does the ordering persist with a longer training budget or the full Grounding DINO objective (auxiliary losses, denoising)? What happens with four interactions using the full text tokens instead of cores?

  3. Timing at high capacity. Does the timing ordering ({1} > {6}, {3}, {pre}) hold with KtK_t=128 and on the Swin-L backbone? Results for intermediate frequencies (mm=2, 3, 6) would also show whether the effect is monotonic.

  4. Checkpoints and baselines. Which checkpoint is used for "Grounding DINO-S/Swin-L," and how do your dense baselines compare with the published zero-shot and fine-tuned COCO numbers of the corresponding models? What is the prompt length TT for the 80-category COCO prompt?

  5. Intended contribution. Given that the architecture yields no wall-clock gain and at best matches dense accuracy, what practical design recommendation should a practitioner take from this work, and under what conditions would you expect the finding to transfer to other fusion mechanisms (e.g., GLIP, OWL-ViT, YOLO-World)?

Reviewer_7tkMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Dual-Core, a controlled study of when and how often text and image features should interact inside an open-vocabulary object detector. Detectors such as Grounding DINO add text cross-attention to every decoder layer, which combines two choices: where the modalities interact and how many times they do. Dual-Core first builds a compact summary of each modality, 64 learned text vectors and the top 300 refined object candidates, then lets them interact only once in the first decoder layer. Final labels still use the full original text. The authors compare different interaction positions and counts on Grounding DINO with Swin-T and Swin-L backbones, training only the decoder on COCO and also testing zero-shot on LVIS. A single interaction in the first decoder layer works best, and with 128 text vectors it matches the original detector's accuracy, while four interactions lose about 1.2 AP. The design gives no speed or accuracy gain, so the contribution is the design principle: build each modality's summary first, exchange once, then refine.

Critical strengths
  1. How text and image features should interact inside a multimodal model, and what principles should guide that choice, is an important and under-studied question. The paper makes it testable by separating two choices that usually change together: where the modalities interact (timing) and how many times they do (frequency).

  2. Some of the architectural ideas are interesting. The compact text cores are used only for communication within the decoder, while the final labels still use the full original text tokens, thereby keeping the pretrained word-to-box matching intact. Visual cores keep their reference boxes and full access to the image in every layer, so compressing the queries doesn't lose spatial detail. New modules start near zero, so every variant begins from the pretrained detector.

  3. The proposed principle is simple: build each modality's summary first, exchange once, then refine. The evaluation tries to support it from several directions: timing with frequency fixed; frequency with higher core capacity to rule out weak cores; a second backbone; zero-shot transfer to LVIS; distractor prompts; and comparisons with visual token compression.

Critical weaknesses
  1. The evaluation section could be organized better. It consists of 9 small subsections, with insufficient detail in several tables. Some tables have very large font sizes. This space could be better used to present the key insights more clearly.
  2. The paper proposes a design principle. However, it has been evaluated on a single architecture for an open-vocabulary detector, with empirical results. A more thorough evaluation, including a discussion of different open-vocabulary detector architectures, is required to draw any conclusion here.
  3. The best model only matches dense accuracy [Tables 1, 4, 8], and there is "no reliable wall-clock gain". It is unclear when a practitioner should adopt this design.
  4. "Once" is overstated. Text already mixes with image features in Grounding DINO's 6-layer encoder, and it decides which locations become visual cores. The single interaction is one interaction in the decoder, on top of fusion that already happened.
Questions for authors
  1. The best model matches dense accuracy [Tables 1, 4, 8] with "no reliable wall-clock gain" [App. C]. What should a practitioner gain from this design? Can you report parameter counts and memory, and test a setting where fewer interactions might matter, such as long prompts or LVIS-scale vocabularies?

  2. All results use Grounding DINO [Sec. 5]. Does the ordering hold for other detectors, such as MM-Grounding-DINO, or for detectors from other families such as GLIP and YOLO-World?

  3. In the four-interaction variant, are the cross-attention layers initialized from pretrained weights? If so, four layers must adapt to the new core inputs instead of one. How do you rule out that this, rather than frequency, causes the 1.23 AP drop [Table 2]?

  4. What AP do you get with no decoder text cross-attention (m = 0), keeping encoder fusion and the full-text matching head? Grounding DINO's own ablation shows that removing decoder text cross-attention costs only 0.6 AP on COCO.

  5. Can you report standard deviations for Tables 3, 5, 6, and 9? The timing gaps are 0.1–1.1 AP. Can you also test more schedules, such as {1, 2}, {1, 6}, or all six layers? The claim that timing and frequency are independent rests on only m = 1 and m = 4 [Sec. 3.6, Table 3].

  6. Text already mixes with image features in the 6-layer encoder and chooses which locations become visual cores [Sec. 3.3]. In what sense is the decoder exchange the "single interaction," and does the result depend on this earlier fusion?

Minor

  1. Some of the tables have very large text
  2. Figure 2 can be improved by adding reference baseline model architecture to highlight the novelty in this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.