acceptodds
Under review as a conference paper at ICLR 2027

A Tale of Two Signals: Open-vocabulary Dense Perception with Frozen MLLMs

Abstract

Multimodal large language models (MLLMs) encode spatial semantics in decoder hidden states and provide image-conditioned responses about category presence. We investigate how these signals complement each other for spatial recognition without task-specific fine-tuning. Our analyses show that same-layer visual–text pairing is not necessarily optimal, and selected visual-layer aggregation can improve recognition beyond the strongest single-layer counterpart in our diagnostic. We further distinguish the contextual role of jointly encoded categories from their role as prediction alternatives: textual context affects discrimination, while category competition can alter recognition even with fixed representations. Building on these observations, we introduce TALE (Semantic Aggregation and Selective Verification), which aggregates visual states, jointly constructs contextual category representations, and independently verifies leading candidates in ambiguous regions. Valid negative responses guide similarity refinement without reconstructing the classification representations or restricting the full prediction vocabulary. Experiments on regional classification, open-vocabulary detection, and segmentation demonstrate the utility of TALE, while controlled ablations characterize its recognition gains and inference costs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Cross-Layer Visual Aggregation, Category Presence Verification, Dense Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PTSaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether visual representations and image-level category-presence judgments from a frozen MLLM can jointly improve open-vocabulary dense perception. TALE aggregates visual features from different decoder layers and matches them against contextual category representations constructed from the full vocabulary. For ambiguous regions, it independently queries the model about the presence of leading candidate categories and uses valid negative responses to adjust regional similarities. The representations remain fixed, and all vocabulary categories remain eligible for prediction. Experiments cover regional classification, open-vocabulary detection, and semantic segmentation, with additional analyses of component contributions and inference costs.

Critical strengths
  1. The analysis of cross-layer representations is reasonably thorough. The paper goes beyond comparing visual and textual features from the same layer. It also examines the strongest single visual layer for a fixed textual layer and compares this against aggregation. The appendix extends the analysis across textual depths and reports results after excluding development images. These experiments provide concrete evidence for cross-layer complementarity and a useful basis for the aggregation design.

  2. The segmentation results demonstrate practical value. In Table 5, TALE improves over FSeg-LLaVA with the same LLaVA-1.5-Vicuna-7B backbone by 7.6, 6.2, and 5.6 mIoU points on PC60, ADE150, and Stuff171, respectively. These are meaningful improvements and support the usefulness of the approach on several benchmarks with more complex category sets.

Critical weaknesses
  1. The contribution of verification to the final dense prediction results remains unclear relative to its cost. The component ablations in Tables 6 and 14 primarily evaluate classification within ground-truth regions, whereas Tables 4 and 5 report complete detection and segmentation results. The paper does not provide the corresponding component comparison within the same dense prediction pipeline, making it difficult to determine how much of the final AP or mIoU improvement comes from verification. This matters given its computational cost: in the timing experiment in Table 17, adding verification increases latency from approximately 1.00 to 3.90 seconds per image. A comparison using the final task metrics is needed to establish whether this additional cost is justified.

  2. Negative verification causes substantial damage to initially correct Stuff predictions. Under the configuration evaluated in Appendix B.2, Table 15 shows that No-only refinement turns 27.3% of initially correct Stuff regions that trigger verification into errors. It corrects 453 Stuff regions while damaging 391 initially correct ones. These counts use different weighting from class-macro accuracy and should not replace that metric, but they reveal a considerable correction–damage trade-off. In addition, Table 14 shows little improvement from weighting the penalty by the response preference rather than using a fixed penalty, leaving limited evidence for this particular design choice.

  3. Vocabulary-order sensitivity has not been sufficiently evaluated for the complete method. In the diagnostic in Appendix A.2, changing only category order produces Things Top-1 accuracy ranging from 57.6% to 65.3% and Stuff Top-1 accuracy ranging from 33.0% to 38.3%. These ranges should not be interpreted as the variability of the complete TALE pipeline, but they show that its textual representations can be sensitive to ordering. The paper does not establish whether TALE consistently improves over aggregation alone across these permutations, or how ordering affects verification frequency and query cost.

  4. Some supporting experiments use a different refinement rule from the main method. Equation (6) applies penalties only after valid No responses, leaving Yes and unresolved responses unchanged. However, Appendix A.1 states that both valid Yes and No responses contribute penalties in the textual-depth scan, while the corresponding curves are labeled TALE. Appendix B.2 shows that these rules have different correction and damage behavior. The paper should clarify which rule is used in Figures 4, 9, and 10 and establish whether the conclusions supporting the final method also hold under No-only refinement.

Questions for authors
  1. With the SAM configuration, spatial refinement, and post-processing held fixed, what detection AP and segmentation mIoU are obtained using the strongest single visual layer, aggregation alone, and aggregation with verification? Please also report complete pipeline latency on the same images so that the additional benefit of verification can be assessed against its cost.

  2. Why does No-only refinement damage 27.3% of initially correct Stuff regions that trigger verification in Table 15? Can the existing response records identify how often queried categories known to be present receive incorrect No responses, and which Stuff categories account for most corrections and damage? Is the small advantage over a fixed penalty in Table 14 consistent across evaluation settings?

  3. Across the ten vocabulary permutations in Figure 11, what are the mean accuracy, variation, and query counts for complete TALE and aggregation alone when layer choices, fusion weights, and verification hyperparameters remain fixed? This would help establish whether the reported gains depend on a particular category ordering.

  4. Do Figures 4, 9, and 10 use No-only or Yes/No refinement? If they use Yes/No refinement, do the main conclusions across textual depths still hold when applying the No-only rule in Equation (6)? Please clarify the configurations used for these scans and their relationship to the main results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PTSa (Part 1/2)

We thank the reviewer for the careful review and constructive feedback.

1. Component ablations on OVSS and OVOD

To isolate the contribution of verification to the final detection and segmentation results, we evaluated three variants within each task pipeline: the best single visual layer, aggregation alone, and aggregation with verification (TALE).

Relative to aggregation alone, verification added 0.8 novel AP50 points on OV-COCO, 0.1 mIoU points on VOC21, and 0.5 mIoU points on ADE150 (Table R1).

Table R1. Ablations on OVOD (Novel AP50) and OVSS (mIoU)

MethodOV-COCOVOC21ADE150
Best single layer30.866.821.8
Aggregation alone30.967.622.0
TALE (aggregation with verification)31.7 (+0.8)67.7 (+0.1)22.5 (+0.5)

We measured end-to-end latency for all three variants on the same images within each task (Table R2). Verification added 6.35 s/image on OV-COCO, 0.83 on VOC21, and 3.91 on ADE150.

Table R2. End-to-end latency on OVOD and OVSS (s/image)

MethodOV-COCOVOC21ADE150
Best single layer19.870.983.25
Aggregation alone21.161.013.30
TALE (aggregation with verification)27.51 (+6.35)1.84 (+0.83)7.21 (+3.91)

Table R3 breaks down the cost of generating proposals, extracting features, and verifying categories. Image-level verification was the most time-consuming of the three reported stages. Feature extraction encodes the full vocabulary jointly for each image view, whereas verification generates a separate answer for each selected category. On the profiled images, TALE queried 1.6 categories per image on VOC21 and 6.5 on ADE150, with corresponding verification latencies of 0.79 and 3.40 s/image. The overall measurement also includes preprocessing, spatial reconstruction, and decoding.

Table R3. computational cost on OVSS

TaskStageLatency (s/image)TFLOPs/imageMemory (GB)
VOC21SAM proposal generation0.172.0616.62
VOC21MLLM feature extraction0.5336.3415.53
VOC21Image-level verification0.7978.4315.43
VOC21TALE overall1.84118.0016.62
ADE150SAM proposal generation0.182.0616.62
ADE150MLLM feature extraction1.49114.7115.74
ADE150Image-level verification3.40335.5615.80
ADE150TALE overall7.21455.5516.62

To put this cost in context, we compared the original TALE configuration and the complete FSeg-LLaVA pipeline under the same protocol. Table R4 shows that TALE improved mIoU by 1.8 on VOC21 and 6.9 on ADE150, with lower latency and GPU memory usage on both. TALE scores the full vocabulary jointly and shares each selected category’s verification answer across regions, limiting repeated decoding and helping explain its lower latency.

Table R4. End-to-end comparison with FSeg-LLaVA

TaskMethodmIoULatency (s/image)TFLOPs/imagMemory (GB)
VOC21TALE67.81.84118.0016.62
VOC21FSeg-LLaVA66.07.02200.9820.82
ADE150TALE22.47.21455.5516.62
ADE150FSeg-LLaVA15.544.281468.3720.82

2. Sensitivity to vocabulary permutations

We evaluated aggregation alone and full TALE on the same images under each of the ten vocabulary permutations in Figure 11, keeping layer choices, fusion weights, and verification hyperparameters fixed. Across permutations, mean Top-1 accuracy (± SD) for aggregation alone was 62.7 ± 2.2% on Boxes, 64.9 ± 2.0% on Things, and 38.4 ± 1.6% on Stuff. Full TALE achieved 65.3 ± 1.8%, 67.6 ± 1.6%, and 39.4 ± 1.3%, respectively, using 10.1 ± 0.3 verification queries per image. Absolute accuracy varied with vocabulary order. Verification improved Top-1 accuracy for all three region types under every permutation.

One possible contributor is causal attention in the language decoder. Each category token can attend to the tokens before it, so reordering the vocabulary changes which other category names it can use as context. This may alter the category’s text representation and, in turn, its similarity score for the same visual region. The cause of the observed sensitivity remains open. In future work, we plan to examine why vocabulary order affects dense prediction and explore ways to reduce this sensitivity.

Table R5. Top-1 accuracy (%) and queries per image across ten vocabulary permutations (mean ± SD)

MethodBoxesThingsStuffQueries/image
Aggregation only62.7 ± 2.264.9 ± 2.038.4 ± 1.60.0 ± 0.0
Full TALE65.3 ± 1.867.6 ± 1.639.4 ± 1.310.1 ± 0.3
AuthorsOpenReview ↗

Response to Reviewer PTSa (Part 2/2)

3. No-only refinement details

We examined the stored image-level answers for the regions evaluated in Table 15. The GT-No rate is the fraction of initially correct, triggered regions whose ground-truth category received a No. 'Corrected' counts initially incorrect regions whose Top-1 prediction became correct. The harm rate is the fraction of initially correct, triggered regions whose Top-1 prediction became incorrect (Table R6).

Table R6. Details of image-level answers

RegionInitially correct and triggeredGT category receives NoGT-No rateCorrectedHarmedHarm rate
Boxes2,0431969.59%705924.50%
Things2,0521989.65%1,225984.78%
Stuff1,43196267.23%45339127.32%

The main difference is how often the verifier rejects a true category. Among initially correct regions that triggered verification, the true class received a false No in 67.23% of Stuff regions, compared with 9.59% of Boxes and 9.65% of Things. Once a false No occurred, it overturned the correct prediction in 40.64% of Stuff regions, 46.94% of Boxes, and 49.49% of Things.

One reason the verifier gives false No answers for Stuff is the wording of its category names. Some Stuff labels include the dataset-specific suffix -merged, as in sky-other-merged. We evaluated 78 image–category pairs in which the Stuff class was annotated as present but the verifier had answered No. Keeping the image and the rest of the question fixed, removing only -merged changed 45 of 78 answers to Yes and restored 20 of 36 correct predictions that verification had overturned.

Table R7. Corrections and harms by Stuff category

Stuff categoryRegionsCorrectedHarmedCorrected − Harmed
wall-other-merged1,7988269+13
sky-other-merged1,677737−30
floor-other-merged7372035−15
building-other-merged1,0301533−18
fence-merged583727−20
grass-merged986627−21
window-other519551+54
table-merged8624712+35
snow178160+16
wall-wood318166+10

In Table R7, the six categories with the most harmed regions all carry the -merged suffix and together account for 228 of 391 harms (58.31%). The largest net loss occurred for sky-other-merged, with 37 harms and only 7 corrections. Corrections were also concentrated: wall-other-merged, window-other, and table-merged accounted for 184 of 453 corrections (40.62%). In particular, wall-other-merged had both the most harms (69) and the most corrections (82), illustrating why both counts matter for frequent categories.

Table R8. Fixed penalty vs. TALE

Task / RegionAggregation onlyFixed penaltyTALE
COCO recognition / Boxes69.170.270.2
COCO recognition / Things67.369.369.4
COCO recognition / Stuff39.340.240.5
VOC21 OVSS67.267.667.7
ADE150 OVSS22.022.222.4
OV-COCO OVOD30.931.631.7

With the same candidate categories and QA responses, a fixed No penalty improved performance over aggregation alone in all six settings. At the reported precision, TALE outperformed the fixed penalty in five of the six settings and tied it on COCO Boxes (Table R8). The largest gain was on COCO Stuff, where Top-1 accuracy increased from 40.2% to 40.5%. On VOC21, mIoU increased from 67.6 with the fixed penalty to 67.7 with TALE.

4. Configuration of Figures 4, 9, and 10

Figure 4 used No-only refinement. Figures 9 and 10 used Yes/No refinement. To clarify whether the textual-depth conclusion also holds for the No-only rule used by the main method in Equation (6), we applied it to the same COCO images across textual depths and visual-layer budgets from 0 to 5. For each configuration, we kept the visual and textual representations, nominated categories, and cached QA answers fixed. Table R9 reports mean Top-1 and Top-5 accuracy across textual depths and visual-layer budgets. No-only refinement improved both metrics on average for all three region types.

Table R9. No-only refinement in the textual-depth scans of Figures 9 and 10

Figure / MetricVariantBoxesThingsStuff
Figure 9 · Top-1No refinement65.768.237.0
No-only (Eq. 6)67.3 (+1.6)69.6 (+1.4)38.3 (+1.3)
Figure 10 · Top-5No refinement86.887.766.8
No-only (Eq. 6)87.1 (+0.3)87.8 (+0.1)67.0 (+0.2)
AuthorsOpenReview ↗

Follow-up: Summary of Manuscript Revisions for Reviewer PTSa

Thank you again for your careful review and constructive feedback. Following our earlier responses, we have revised the manuscript to address your concerns, with the changes highlighted in blue. Specifically, we have added:

(1) Component ablations on OVSS and OVOD to Section 4.3 (lines 372–387, Table 7).
(2) Computational cost analyses and the FSeg-LLaVA comparison to Appendix D.3 (lines 1404–1452 and 1458–1467).
(3) Vocabulary-order comparisons to Appendix A.2 (lines 958–989, Table 12).
(4) Fixed versus response-weighted penalty controls to Appendix B.2 (lines 1180–1197, Table 20).
(5) Analyses of false No responses and Stuff-category corrections and harms to Appendix B.2 (lines 1246–1290, Tables 22–23).
(6) Clarifications of the refinement rules and additional No-only comparisons to Appendix A.1 (lines 820–825 and 857–899, Figures 9–10 and Table 11).

We would appreciate any further feedback and would be happy to address any remaining questions. Thank you for your time and consideration.

Reviewer_UHaVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reads hidden states from a frozen MLLM and matches them to category tokens for regional recognition. It then asks the same model whether a category is present, and uses a negative answer to penalize ambiguous regions. The resulting method, TALE, is evaluated on COCO region classification, OV-COCO detection, and five segmentation benchmarks.

Critical strengths

Section 2 separates joint encoding from the prediction set. Changing the context changes the category representation, while deleting competitors changes accuracy even when the representations stay fixed. The oracle removal is correctly treated as a diagnostic ceiling.

The cross-layer measurement is the substantive finding. The best visual layer and the best text layer are not the same, and a development-set ranking still gives mean Top-1 gains of about 3.7, 2.4, and 1.6 points on Boxes, Things, and Stuff over same-layer pairing.

The paper reports the cost and the damage. Verification takes about 8 queries and roughly 3 of the 3.9 seconds per image. A large penalty drops Stuff from 40.3 to 32.8, and 27.3% of triggered correct Stuff regions are flipped. The authors do not call the Yes preference a calibrated probability.

Critical weaknesses

The aggregation claim uses a weak reference. The single-layer baseline in Table 11 is L31, and Figure 1(a) already shows that the last layer falls to 44.5, so the jump from 42.3 to 65.1 is mostly a layer choice. Against the best single layer, the fusion in Figure 1(c) is selected on the evaluation results. In Table 6 the auxiliary branch alone reaches 67.1 on Things, above the fused 67.0. On LLaVA-1.6-Vicuna-7B, Stuff aggregation is slightly worse than the single layer.

The two-signal story is close to existing work, and the second signal is small. FSeg-LLaVA already queries presence and reads intermediate visual–text hidden states from frozen LLaVA. Liu et al. (COLM 2025) already show that perception peaks in middle layers and drops later. After aggregation, verification adds about two points and costs eight generated answers. Vocabulary order alone moves Things Top-1 from 57.6 to 65.3, which is as large as several of the method gains.

The dense-perception results are not yet attributable to TALE. The detection gain of 7.7 AP is over CLIP-B/16 on shared SAM proposals. Trained and training-free numbers from other detectors are not in that setup. The path from 24×24 token similarities to a VOC mIoU of 69.0, including upsampling, the background class, and any external refinement, is absent from the main text. Against FSeg with the same LLaVA, VOC moves from 68.0 to 69.0 and Object81 from 42.0 to 42.3. The larger gaps are on benchmarks where FSeg was already weak.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer UHaV (Part 1/2)

We thank the reviewer for the careful review and constructive feedback.

1. Aggregation vs. the Best Single Layer

To isolate the contribution of aggregation, we selected the best single visual layer for each task on the development set. We then fixed that layer as the main layer for both aggregation alone and full TALE.

Table R1. TALE with the best single layer fixed as the main layer

MethodCOCO BoxesCOCO ThingsCOCO StuffVOC21ADE150
Development-selected main layerL24L24L17L20L28
Best single layer64.367.238.866.821.8
Aggregation only64.567.542.067.622.0
Full TALE66.569.342.967.722.5

With the main layer held fixed, aggregation alone improves Top-1 accuracy by 0.2, 0.3, and 3.2 percentage on COCO Boxes, Things, and Stuff, respectively. It also improves mIoU by 0.8 on VOC21 and 0.2 on ADE150. Full TALE outperforms both the best single layer and aggregation alone in all five settings. TALE’s gains persist even against the strongest single-layer baseline.

2. TALE gains across MLLMs

We agree that aggregation alone slightly reduced COCO Stuff Top-1 accuracy on LLaVA-1.6-Vicuna-7B, while full TALE reached 42.7% (Table 11). The benefit of combining layers may depend on both the backbone and the task. We therefore examined how well TALE performs on two segmentation benchmarks with the same backbone and whether its gains extend to Qwen2-VL-7B-Instruct and InternVL2.5-8B.

Full TALE achieved 69.8 mIoU on VOC21 and 23.3 on ADE150 with LLaVA-1.6-Vicuna-7B. Table 5 of the paper reports 69.0 and 23.1, respectively, for LLaVA-1.5-Vicuna-7B. On VOC21, full TALE improved over the best single layer by 0.7 mIoU on Qwen2-VL-7B-Instruct and 1.0 on InternVL2.5-8B. It also improved COCO Boxes and Things Top-1 accuracy on both backbones (Table R2). Full TALE’s gains over a strong single-layer baseline thus extend to both model families in the settings tested.

Table R2. TALE on other MLLMs

ModelVariantBoxesThingsVOC21
Qwen2-VL-7B-InstructBest single layer65.966.047.6
TALE66.767.448.3
InternVL2.5-8BBest single layer70.269.057.0
TALE72.072.158.0

3. Comparison with FSeg-LLaVA and Liu et al.

FSeg-LLaVA [1] is the closest prior work to TALE. Both use intermediate representations and image-level answers from a frozen LLaVA. FSeg-LLaVA queries which classes are present, derives class-specific activation maps, and uses them to prompt SAM. TALE first scores the full vocabulary at each spatial location by visual–text similarity, then queries candidate categories and revises their scores after valid No responses. With the same LLaVA-1.5-Vicuna-7B backbone, TALE achieves higher mIoU than FSeg-LLaVA on all five OVSS benchmarks in Table 5.

Liu et al. [2] showed that intermediate language-model representations retain perceptual information and that this information can degrade in later layers. Figure 1(a) shows a similar pattern in regional recognition. Their layer probes address representation quality and do not provide an end-to-end segmentation baseline.

4. Computational Cost

In the original TALE configuration, adding verification to aggregation alone increases latency by 0.83 s/image and improves mIoU by 0.6 on VOC21. On ADE150, it adds 3.91 s/image and improves mIoU by 0.4. TALE averages 1.30 verification queries per image on VOC21 and 7.32 on ADE150.

Compared with FSeg-LLaVA, TALE achieves higher mIoU on both datasets with lower latency and GPU memory usage (Table R3). TALE scores the vocabulary jointly and generates answers only for selected categories, limiting the number of decoding calls.

Table R3. Computational cost comparison

TaskMethodmIoULatency (s/image)TFLOPs/imageMemory (GB)
VOC21Aggregation only67.21.0139.5716.62
VOC21TALE67.81.84118.0016.62
VOC21FSeg-LLaVA66.07.02200.9820.82
ADE150Aggregation only22.03.30119.9816.62
ADE150TALE22.47.21455.5516.62
ADE150FSeg-LLaVA15.544.281468.3720.82

All variants in Table R3 were evaluated on the same hardware under the same evaluation conditions.

[1] Bingfeng Zhang et al. The Power of Prior: Training-Free Open-Vocabulary Semantic Segmentation with LLaVA. CVPR 2026.

[2] Benlin Liu et al. Visual Representations inside the Language Model. COLM 2025.

AuthorsOpenReview ↗

Response to Reviewer UHaV (Part 2/2)

5. Sensitivity to vocabulary order

We agree that vocabulary order affects absolute accuracy. We compared aggregation alone and full TALE on the same images across the ten vocabulary permutations in Figure 11, holding layer choices, fusion weights, and verification hyperparameters fixed. Verification improved Top-1 accuracy for Boxes, Things, and Stuff under all ten permutations. The differences between the reported mean accuracies were +2.6, +2.7, and +1.0 percentage points, respectively. TALE averaged 10.1 verification queries per image. The size of the gain varied with order, and Things Top-1 with full TALE ranged from 65.4% to 70.0% (Table R4).

One possible contributor is causal attention in the language decoder. Each category token can attend to preceding tokens, so reordering the vocabulary changes which other category names it can use as context. This may alter the category’s text representation and, in turn, its similarity score for the same visual region. The cause of the observed sensitivity remains open. In future work, we plan to examine why vocabulary order affects dense prediction and explore ways to reduce this sensitivity.

Table R4. Verification gains under each of the ten vocabulary permutations

OrderBoxes AggregationBoxes TALE (Δ)Things AggregationThings TALE (Δ)Stuff AggregationStuff TALE (Δ)TALE queries/image
163.765.0 (+1.3)65.466.9 (+1.5)38.238.8 (+0.6)10.3
259.363.0 (+3.7)62.265.5 (+3.3)37.137.5 (+0.4)10.0
361.063.8 (+2.8)64.067.6 (+3.6)35.737.7 (+2.0)9.4
464.967.2 (+2.3)67.570.0 (+2.5)37.838.9 (+1.1)10.1
562.764.8 (+2.1)64.266.8 (+2.6)40.841.3 (+0.5)10.1
665.867.4 (+1.6)67.169.2 (+2.1)38.039.3 (+1.3)10.0
760.563.1 (+2.6)62.465.4 (+3.0)38.940.8 (+1.9)10.4
861.366.3 (+5.0)63.467.4 (+4.0)40.340.9 (+0.6)10.1
963.265.1 (+1.9)66.167.4 (+1.3)39.839.9 (+0.1)9.9
1065.067.7 (+2.7)67.069.4 (+2.4)36.738.7 (+2.0)10.4
Mean62.765.3 (+2.6)64.967.6 (+2.7)38.439.4 (+1.0)10.1

6. Attribution of the detection gains

The original table reports a 7.7-point gain in novel AP50 (from 24.1 to 31.8) when comparing two complete scoring pipelines that use the same SAM proposals. To isolate TALE’s contribution within frozen LLaVA, we added a component ablation on OV-COCO. All variants use the same SAM proposals and detection post-processing. They differ only in how they compute and refine category scores for each proposed region.

Table R5. Novel AP50 on OV-COCO with shared SAM proposals

MetricCLIP-B/16 + SAMLLaVA L31 + SAMLLaVA best layer (L21) + SAMAggregation + SAMTALE + SAM
Novel AP5012.88.430.830.931.7

Under these shared conditions, full TALE gains 0.9 novel AP50 over the strongest single-layer LLaVA baseline (30.8 to 31.7). Adding verification to aggregation alone accounts for 0.8 (30.9 to 31.7).

7. OVSS implementation details

We will clarify in the main text how the 24×24 crop features produce the final VOC21 segmentation, including upsampling, background handling, and external refinement. For each crop, we aggregate visual features using patch affinities constrained by class-agnostic SAM masks, then compute category scores. We bilinearly upsample each crop’s score maps to the crop resolution, average scores where crops overlap, and bilinearly resize the merged maps to the original image resolution.

Image-level verification filters only foreground categories, so background remains eligible throughout. At each pixel, we apply a softmax over the retained query scores and use the maximum probability among the 26 background terms as the background score. We assign the highest-scoring class, defaulting to background if the winning score is below 0.2. For the final spatial refinement, each SAM region receives its majority pixel label. This pipeline yields 69.0% mIoU across 21 classes on all 1449 VOC2012 validation images.

AuthorsOpenReview ↗

Follow-up: Summary of Manuscript Revisions for Reviewer UHaV

Thank you again for your careful review and constructive feedback. Following our earlier responses, we have revised the manuscript to address your concerns, with the changes highlighted in blue. Specifically, we have added:

(1) Best-single-layer controls to Section 4.3 (lines 372–387, Table 7).
(2) A same-backbone detection baseline to Section 4.2 (lines 316–318, Table 4).
(3) Additional backbone evaluations to Section 4.3 (lines 456–466) and Appendix B.1 (lines 1110–1118).
(4) Vocabulary-order comparisons to Appendix A.2 (lines 958–989).
(5) OVSS implementation details to Appendix C.1 (lines 1298–1325).
(6) Computational cost analyses to Appendix D.3 (lines 1404–1452 and 1458–1467).
(7) Distinctions from FSeg-LLaVA and Liu et al. to Appendix E (lines 1480–1489).

We would appreciate any further feedback on the revised manuscript and would be happy to address any remaining questions. Thank you for your time and consideration.

Reviewer_VzwCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TALE, a training-free open vocabulary detection framework using frozen MLLMs. The proposed method aggregates features from different decoder layers, and uses image-level category verification to further surpass the negative predictions. The proposed method achieves results that surpass several methods finetuned on specific tasks.

Critical strengths
  1. This paper conducts empirical study of the effectiveness of representation feature from different layers for detection and find the optimal features do not come from the same layer, which is inspiring.
  2. The author conducts experiments on comprehensive datasets and tasks and shows that the training-free framework can even surpass methods fine-tuned on specific tasks.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. This paper combines several techniques which are already explored before, such as multilayer feature aggregation, contextual text representation construction and image-level category verification. Although the combination is effective and achieves good performance, the novalty is limited.
  2. The overall pipeline, i.e., SAM proposal generation, MLLM feature extraction, and image-level verification, is complex and requires multiple inferences which may hinder its practical appliance.
  3. This paper mainly conducts evaluation on LLaVA, does the author conducts experiments on other more recent MLLM backbones and investigate whether the performance can be further improved.
Questions for authors
  1. Can the author further elaborates the difference of multi-layer feature aggregation with other similar feature aggregation techniques which has been widely applied in open-vocabulary detection.
  2. Does the author tests if the method can applied to other MLLM backbones.
2 comments
AuthorsOpenReview ↗

Response to Reviewer VzwC

We thank the reviewer for the careful review and constructive feedback.

1.Computational Cost. We profiled all methods on an RTX 4090. Tables R1–R3 report latency (s/image), compute (TFLOPs/image), and GPU memory (GB).

Table R1. COCO computational cost.

StageLatencyComputeMemory
MLLM feature extraction1.4415.2214.50
Image-level verification5.22415.1114.29
TALE overall6.70430.3314.50

Verification accounts for 5.22 of 6.70 s/image. TALE scores the full vocabulary jointly, then generates one answer per selected category and shares it across regions in the same image.

Table R2. TALE stage costs on OVSS

VOC21ADE150
StageLatencyComputeMemoryLatencyComputeMemory
SAM proposal generation0.172.0616.620.182.0616.62
MLLM feature extraction0.5336.3415.531.49114.7115.74
Image-level verification0.7978.4315.433.40335.5615.80
TALE overall1.84118.0016.627.21455.5516.62

Verification was the slowest of the three stages. On the profiled images, TALE averaged 1.6 queries/image on VOC21 and 6.5 on ADE150, accounting for 43% and 47% of end-to-end latency, respectively.

Table R3. End-to-end cost on OVSS.

VOC21ADE150
MethodLatencyComputeMemoryLatencyComputeMemory
FSeg-LLaVA7.02200.9820.8244.281468.3720.82
TALE1.84118.0016.627.21455.5516.62

TALE achieves higher mIoU on both datasets while using less GPU memory and running inference faster. TALE scores the vocabulary jointly and verifies only selected categories, reducing decoding cost while potentially improving accuracy through visual context and relationships among category names.

2.Comparison with multilayer feature aggregation methods. Multilayer methods differ in how they select layers and use the fused features. TALE ranks visual layers by regional recognition accuracy at a fixed text depth. At each location, it aligns and separately normalizes the main-layer feature and the mean auxiliary feature, then scores each category by cosine similarity between their weighted sum and the category’s text embedding from the same frozen MLLM.

(1) CLIPSelf/F-ViT[1] uses predefined layer indices for each visual backbone. For ViT-B/16, it maps features from layers 3, 5, 7, and 11 to four resolutions. Its feature pyramid combines coarse and fine features to detect objects at different scales. TALE selects decoder layers by their measured regional recognition accuracy.

(2) NoOVD/K-FPN[2] fuses predefined ViT-B/16 layers 5, 7, and 11 from deep to shallow and builds a five-level pyramid. It matches proposal features to foreground and background descriptions to discover novel objects. TALE scores the full target vocabulary at each location by matching one fused visual descriptor to category text embeddings.

(3) YOLO-World/RepVL-PAN[3] fuses three Darknet stages. CLIP category embeddings modulate the visual features, and pooled image features update the text embeddings. TALE directly matches fused visual features to category text embeddings from the same frozen MLLM.

These detectors build multiscale feature maps. TALE uses measured layer performance to construct one visual descriptor per location for direct matching to the target vocabulary. We apply the same selection procedure to each backbone and fix its selected layers and weights for inference.

3.Applied to other MLLM backbones. We evaluated TALE on Qwen2-VL-7B-Instruct [4] and InternVL2.5-8B [5]. Table R4 reports COCO regional Top-1 accuracy for boxes and thing masks, and VOC21 mIoU.

Table R4. TALE on other MLLMs.

ModelVariantBoxesThingsVOC21
Qwen2-VL-7B-InstructBest single layer65.966.047.6
TALE66.767.448.3
InternVL2.5-8BBest single layer70.269.057.0
TALE72.072.158.0

Full TALE improves all three metrics on both backbones, gaining 0.7–3.1 points over the best single layer.


[1] Size Wu et al. CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction. ICLR 2024.

[2] Yupeng Zhang et al. NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection. CVPR 2026.

[3] Tianheng Cheng et al. YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR 2024.

[4] Peng Wang et al. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. 2024.

[5] Zhe Chen et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling.

AuthorsOpenReview ↗

Follow-up: Summary of Manuscript Revisions for Reviewer VzwC

Thank you again for your careful review and constructive feedback. Following our earlier responses, we have revised the manuscript to address your concerns, with the changes marked in blue.

Specifically, we have added comparisons with related feature aggregation and verification methods to Appendix E (lines 1454–1457 and 1470–1489); stage-level and end-to-end computational cost analyses to Appendix D.3 (lines 1404–1452 and Table 31, lines 1458–1467); and experiments on Qwen2-VL-7B-Instruct and InternVL2.5-8B to Section 4.3 (lines 456–466, Table 10).

We would appreciate any further feedback on the revised manuscript and would be happy to address any remaining questions. Thank you for your time and consideration.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. If you haven't already, we would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.