- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether visual representations and image-level category-presence judgments from a frozen MLLM can jointly improve open-vocabulary dense perception. TALE aggregates visual features from different decoder layers and matches them against contextual category representations constructed from the full vocabulary. For ambiguous regions, it independently queries the model about the presence of leading candidate categories and uses valid negative responses to adjust regional similarities. The representations remain fixed, and all vocabulary categories remain eligible for prediction. Experiments cover regional classification, open-vocabulary detection, and semantic segmentation, with additional analyses of component contributions and inference costs.
Critical strengths
-
The analysis of cross-layer representations is reasonably thorough. The paper goes beyond comparing visual and textual features from the same layer. It also examines the strongest single visual layer for a fixed textual layer and compares this against aggregation. The appendix extends the analysis across textual depths and reports results after excluding development images. These experiments provide concrete evidence for cross-layer complementarity and a useful basis for the aggregation design.
-
The segmentation results demonstrate practical value. In Table 5, TALE improves over FSeg-LLaVA with the same LLaVA-1.5-Vicuna-7B backbone by 7.6, 6.2, and 5.6 mIoU points on PC60, ADE150, and Stuff171, respectively. These are meaningful improvements and support the usefulness of the approach on several benchmarks with more complex category sets.
Critical weaknesses
-
The contribution of verification to the final dense prediction results remains unclear relative to its cost. The component ablations in Tables 6 and 14 primarily evaluate classification within ground-truth regions, whereas Tables 4 and 5 report complete detection and segmentation results. The paper does not provide the corresponding component comparison within the same dense prediction pipeline, making it difficult to determine how much of the final AP or mIoU improvement comes from verification. This matters given its computational cost: in the timing experiment in Table 17, adding verification increases latency from approximately 1.00 to 3.90 seconds per image. A comparison using the final task metrics is needed to establish whether this additional cost is justified.
-
Negative verification causes substantial damage to initially correct Stuff predictions. Under the configuration evaluated in Appendix B.2, Table 15 shows that No-only refinement turns 27.3% of initially correct Stuff regions that trigger verification into errors. It corrects 453 Stuff regions while damaging 391 initially correct ones. These counts use different weighting from class-macro accuracy and should not replace that metric, but they reveal a considerable correction–damage trade-off. In addition, Table 14 shows little improvement from weighting the penalty by the response preference rather than using a fixed penalty, leaving limited evidence for this particular design choice.
-
Vocabulary-order sensitivity has not been sufficiently evaluated for the complete method. In the diagnostic in Appendix A.2, changing only category order produces Things Top-1 accuracy ranging from 57.6% to 65.3% and Stuff Top-1 accuracy ranging from 33.0% to 38.3%. These ranges should not be interpreted as the variability of the complete TALE pipeline, but they show that its textual representations can be sensitive to ordering. The paper does not establish whether TALE consistently improves over aggregation alone across these permutations, or how ordering affects verification frequency and query cost.
-
Some supporting experiments use a different refinement rule from the main method. Equation (6) applies penalties only after valid No responses, leaving Yes and unresolved responses unchanged. However, Appendix A.1 states that both valid Yes and No responses contribute penalties in the textual-depth scan, while the corresponding curves are labeled TALE. Appendix B.2 shows that these rules have different correction and damage behavior. The paper should clarify which rule is used in Figures 4, 9, and 10 and establish whether the conclusions supporting the final method also hold under No-only refinement.
Questions for authors
-
With the SAM configuration, spatial refinement, and post-processing held fixed, what detection AP and segmentation mIoU are obtained using the strongest single visual layer, aggregation alone, and aggregation with verification? Please also report complete pipeline latency on the same images so that the additional benefit of verification can be assessed against its cost.
-
Why does No-only refinement damage 27.3% of initially correct Stuff regions that trigger verification in Table 15? Can the existing response records identify how often queried categories known to be present receive incorrect No responses, and which Stuff categories account for most corrections and damage? Is the small advantage over a fixed penalty in Table 14 consistent across evaluation settings?
-
Across the ten vocabulary permutations in Figure 11, what are the mean accuracy, variation, and query counts for complete TALE and aggregation alone when layer choices, fusion weights, and verification hyperparameters remain fixed? This would help establish whether the reported gains depend on a particular category ordering.
-
Do Figures 4, 9, and 10 use No-only or Yes/No refinement? If they use Yes/No refinement, do the main conclusions across textual depths still hold when applying the No-only rule in Equation (6)? Please clarify the configurations used for these scans and their relationship to the main results.
3 comments
Response to Reviewer PTSa (Part 2/2)
3. No-only refinement details
We examined the stored image-level answers for the regions evaluated in Table 15. The GT-No rate is the fraction of initially correct, triggered regions whose ground-truth category received a No. 'Corrected' counts initially incorrect regions whose Top-1 prediction became correct. The harm rate is the fraction of initially correct, triggered regions whose Top-1 prediction became incorrect (Table R6).
Table R6. Details of image-level answers
| Region | Initially correct and triggered | GT category receives No | GT-No rate | Corrected | Harmed | Harm rate |
|---|---|---|---|---|---|---|
| Boxes | 2,043 | 196 | 9.59% | 705 | 92 | 4.50% |
| Things | 2,052 | 198 | 9.65% | 1,225 | 98 | 4.78% |
| Stuff | 1,431 | 962 | 67.23% | 453 | 391 | 27.32% |
The main difference is how often the verifier rejects a true category. Among initially correct regions that triggered verification, the true class received a false No in 67.23% of Stuff regions, compared with 9.59% of Boxes and 9.65% of Things. Once a false No occurred, it overturned the correct prediction in 40.64% of Stuff regions, 46.94% of Boxes, and 49.49% of Things.
One reason the verifier gives false No answers for Stuff is the wording of its category names. Some Stuff labels include the dataset-specific suffix -merged, as in sky-other-merged. We evaluated 78 image–category pairs in which the Stuff class was annotated as present but the verifier had answered No. Keeping the image and the rest of the question fixed, removing only -merged changed 45 of 78 answers to Yes and restored 20 of 36 correct predictions that verification had overturned.
Table R7. Corrections and harms by Stuff category
| Stuff category | Regions | Corrected | Harmed | Corrected − Harmed |
|---|---|---|---|---|
| wall-other-merged | 1,798 | 82 | 69 | +13 |
| sky-other-merged | 1,677 | 7 | 37 | −30 |
| floor-other-merged | 737 | 20 | 35 | −15 |
| building-other-merged | 1,030 | 15 | 33 | −18 |
| fence-merged | 583 | 7 | 27 | −20 |
| grass-merged | 986 | 6 | 27 | −21 |
| window-other | 519 | 55 | 1 | +54 |
| table-merged | 862 | 47 | 12 | +35 |
| snow | 178 | 16 | 0 | +16 |
| wall-wood | 318 | 16 | 6 | +10 |
In Table R7, the six categories with the most harmed regions all carry the -merged suffix and together account for 228 of 391 harms (58.31%). The largest net loss occurred for sky-other-merged, with 37 harms and only 7 corrections. Corrections were also concentrated: wall-other-merged, window-other, and table-merged accounted for 184 of 453 corrections (40.62%). In particular, wall-other-merged had both the most harms (69) and the most corrections (82), illustrating why both counts matter for frequent categories.
Table R8. Fixed penalty vs. TALE
| Task / Region | Aggregation only | Fixed penalty | TALE |
|---|---|---|---|
| COCO recognition / Boxes | 69.1 | 70.2 | 70.2 |
| COCO recognition / Things | 67.3 | 69.3 | 69.4 |
| COCO recognition / Stuff | 39.3 | 40.2 | 40.5 |
| VOC21 OVSS | 67.2 | 67.6 | 67.7 |
| ADE150 OVSS | 22.0 | 22.2 | 22.4 |
| OV-COCO OVOD | 30.9 | 31.6 | 31.7 |
With the same candidate categories and QA responses, a fixed No penalty improved performance over aggregation alone in all six settings. At the reported precision, TALE outperformed the fixed penalty in five of the six settings and tied it on COCO Boxes (Table R8). The largest gain was on COCO Stuff, where Top-1 accuracy increased from 40.2% to 40.5%. On VOC21, mIoU increased from 67.6 with the fixed penalty to 67.7 with TALE.
4. Configuration of Figures 4, 9, and 10
Figure 4 used No-only refinement. Figures 9 and 10 used Yes/No refinement. To clarify whether the textual-depth conclusion also holds for the No-only rule used by the main method in Equation (6), we applied it to the same COCO images across textual depths and visual-layer budgets from 0 to 5. For each configuration, we kept the visual and textual representations, nominated categories, and cached QA answers fixed. Table R9 reports mean Top-1 and Top-5 accuracy across textual depths and visual-layer budgets. No-only refinement improved both metrics on average for all three region types.
Table R9. No-only refinement in the textual-depth scans of Figures 9 and 10
| Figure / Metric | Variant | Boxes | Things | Stuff |
|---|---|---|---|---|
| Figure 9 · Top-1 | No refinement | 65.7 | 68.2 | 37.0 |
| No-only (Eq. 6) | 67.3 (+1.6) | 69.6 (+1.4) | 38.3 (+1.3) | |
| Figure 10 · Top-5 | No refinement | 86.8 | 87.7 | 66.8 |
| No-only (Eq. 6) | 87.1 (+0.3) | 87.8 (+0.1) | 67.0 (+0.2) |
Follow-up: Summary of Manuscript Revisions for Reviewer PTSa
Thank you again for your careful review and constructive feedback. Following our earlier responses, we have revised the manuscript to address your concerns, with the changes highlighted in blue. Specifically, we have added:
(1) Component ablations on OVSS and OVOD to Section 4.3 (lines 372–387, Table 7).
(2) Computational cost analyses and the FSeg-LLaVA comparison to Appendix D.3 (lines 1404–1452 and 1458–1467).
(3) Vocabulary-order comparisons to Appendix A.2 (lines 958–989, Table 12).
(4) Fixed versus response-weighted penalty controls to Appendix B.2 (lines 1180–1197, Table 20).
(5) Analyses of false No responses and Stuff-category corrections and harms to Appendix B.2 (lines 1246–1290, Tables 22–23).
(6) Clarifications of the refinement rules and additional No-only comparisons to Appendix A.1 (lines 820–825 and 857–899, Figures 9–10 and Table 11).
We would appreciate any further feedback and would be happy to address any remaining questions. Thank you for your time and consideration.
Response to Reviewer PTSa (Part 1/2)
We thank the reviewer for the careful review and constructive feedback.
1. Component ablations on OVSS and OVOD
To isolate the contribution of verification to the final detection and segmentation results, we evaluated three variants within each task pipeline: the best single visual layer, aggregation alone, and aggregation with verification (TALE).
Relative to aggregation alone, verification added 0.8 novel AP50 points on OV-COCO, 0.1 mIoU points on VOC21, and 0.5 mIoU points on ADE150 (Table R1).
Table R1. Ablations on OVOD (Novel AP50) and OVSS (mIoU)
We measured end-to-end latency for all three variants on the same images within each task (Table R2). Verification added 6.35 s/image on OV-COCO, 0.83 on VOC21, and 3.91 on ADE150.
Table R2. End-to-end latency on OVOD and OVSS (s/image)
Table R3 breaks down the cost of generating proposals, extracting features, and verifying categories. Image-level verification was the most time-consuming of the three reported stages. Feature extraction encodes the full vocabulary jointly for each image view, whereas verification generates a separate answer for each selected category. On the profiled images, TALE queried 1.6 categories per image on VOC21 and 6.5 on ADE150, with corresponding verification latencies of 0.79 and 3.40 s/image. The overall measurement also includes preprocessing, spatial reconstruction, and decoding.
Table R3. computational cost on OVSS
To put this cost in context, we compared the original TALE configuration and the complete FSeg-LLaVA pipeline under the same protocol. Table R4 shows that TALE improved mIoU by 1.8 on VOC21 and 6.9 on ADE150, with lower latency and GPU memory usage on both. TALE scores the full vocabulary jointly and shares each selected category’s verification answer across regions, limiting repeated decoding and helping explain its lower latency.
Table R4. End-to-end comparison with FSeg-LLaVA
2. Sensitivity to vocabulary permutations
We evaluated aggregation alone and full TALE on the same images under each of the ten vocabulary permutations in Figure 11, keeping layer choices, fusion weights, and verification hyperparameters fixed. Across permutations, mean Top-1 accuracy (± SD) for aggregation alone was 62.7 ± 2.2% on Boxes, 64.9 ± 2.0% on Things, and 38.4 ± 1.6% on Stuff. Full TALE achieved 65.3 ± 1.8%, 67.6 ± 1.6%, and 39.4 ± 1.3%, respectively, using 10.1 ± 0.3 verification queries per image. Absolute accuracy varied with vocabulary order. Verification improved Top-1 accuracy for all three region types under every permutation.
One possible contributor is causal attention in the language decoder. Each category token can attend to the tokens before it, so reordering the vocabulary changes which other category names it can use as context. This may alter the category’s text representation and, in turn, its similarity score for the same visual region. The cause of the observed sensitivity remains open. In future work, we plan to examine why vocabulary order affects dense prediction and explore ways to reduce this sensitivity.
Table R5. Top-1 accuracy (%) and queries per image across ten vocabulary permutations (mean ± SD)