acceptodds
Under review as a conference paper at ICLR 2027

Textual Residual Prompt for Fine-Grained Real-Time Open-Vocabulary Detection

Abstract

Real-time open-vocabulary detectors combine open-category recognition with efficient inference, but still lag significantly behind heavy open-vocabulary detectors on fine-grained detection tasks. We observe an interesting property in the textual embedding space: semantic residuals induced by the same fine-grained attribute remain highly consistent across different base categories, suggesting that fine-grained attributes correspond to relatively stable and transferable semantic directions. Motivated by this observation, we propose , a lightweight adaptation method for fine-grained real-time open-vocabulary detection. Instead of modifying the detector or relearning complete textual representations, TRP learns textual residuals to enhance fine-grained discriminability. The resulting textual representations can still be cached before inference, preserving the efficient prompt-then-detect paradigm. Experiments show that TRP achieves superior fine-grained detection performance among parameter-efficient fine-tuning methods while maintaining real-time inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
fine grained object detection, real time object detection, parameter efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zTxZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Textual Residual Prompt to learn the fine-grained attributes for fine-grained open vocabulary detection.

Critical strengths
  1. The paper is well-written and easy to follow. 2.The observation of fine-grained attributes is informative.
Critical weaknesses

1.The proposed TRP method appears closely related to compositional zero-shot learning (CZSL), which aims to recognize novel attribute–object compositions. The FG-OVD benchmark primarily evaluates fine-grained object attributes, such as color, material, and pattern, rather than fine-grained taxonomic categories such as dog breeds or bird species. The assumption that semantic displacement vectors are transferable across base categories is closely related to the attribute–object compositionality studied in prior CZSL research, including Misra et al. (2017) and Geng et al. (2025). However, the paper does not sufficiently discuss these connections or clearly distinguish its contribution from existing CZSL approaches.

[1]Ishan Misra, Abhinav Gupta, and Martial Hebert. 2017. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1160–1169.

[2]Geng, Yuxia, et al. "Graph-guided cross-composition feature disentanglement for compositional zero-shot learning." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

2.Given the conceptual similarity between TRP and CZSL, the paper would benefit from comparisons with representative CZSL methods adapted to the FG-OVD setting. Such comparisons could help establish whether the proposed semantic residual modeling provides advantages over existing attribute–object composition techniques. The authors should also discuss the differences between CZSL classification and FG-OVD localization when designing these comparisons.

3.Minor

  • TCR appears to be one of the strongest baselines in the experimental results, but its original reference is not clearly provided.
  • ? in line 140
Questions for authors

1.Please discuss the distinction between TRP and CZSL 2.Please provide experimental comparisons between TRP and CZSL 3.Please indicate the reference for TCR

8 comments

We sincerely thank the reviewer for the positive evaluation and constructive feedback. We appreciate the reviewer’s recognition that our paper is well-written and easy to follow, and that our observations on fine-grained attributes are informative. We take the reviewer’s concerns regarding the soundness of our method seriously and address them point by point below.

W1 & Q1: TRP is closely related to compositional zero-shot learning (CZSL), but the paper does not sufficiently discuss their connections and differences

1. Connections and differences between TRP and compositional CZSL

We agree that, at a higher level, TRP and the studies by Misra et al. (2017) and Geng et al. (2025) all involve transferable compositional structures between attributes and objects. However, the specific patterns identified and the modeling approaches adopted by the three studies differ considerably.

① Misra et al. (2017) found that the visual appearance of the same attribute across different objects is strongly context-dependent. For example, the color red has different visual appearances in red wine and *red tomato*.

② Geng et al. (2025) proposed that the visual features of the same attribute vary significantly across different compositions.

③ TRP focuses on the observation that, in the text embedding space, the semantic displacements induced by the same attribute across different base categories exhibit a high degree of directional consistency, unlike the preceding two studies, both of which involve interactions between visual information and textual semantics.

It should be emphasized that TRP’s contribution is not to introduce an entirely new attribute–object compositional learning paradigm, but to identify and exploit the directional consistency of these semantic displacements for parameter-efficient adaptation of real-time open-vocabulary detectors.

2. The modeling structures of existing CZSL methods are not suitable for direct transfer to real-time open-vocabulary detectors

Although TRP is related to existing CZSL methods at a higher level, the attribute–object modeling structures adopted by existing methods are not suitable for direct application to real-time open-vocabulary detectors.

① Geng et al. (2025) perform cross-composition disentanglement of textual and visual features through L-Adapter and V-Adapter, respectively. L-Adapter constructs a global compositional graph containing attribute, object, and composition nodes. The total number of nodes is N = A + O +C, where A denotes the attribute set, O denotes the object category set, and C denotes the set of attribute–object compositions currently involved in graph construction.

However, open-vocabulary detection involves numerous attributes and object categories, resulting in a large number of combinations that can cause the compositional graph to grow rapidly. Geng et al. (2025) explicitly acknowledge this issue in the Limitations section on page 9 of their paper. Therefore, this modeling approach, which relies on a global compositional graph, is not suitable for direct transfer to real-time open-vocabulary detectors.

② Misra et al. (2017) rely on pretrained, fixed attribute and object visual classifiers for composition. Their modeling approach is restricted to a predefined set of visual classifiers, making it difficult to apply directly to query categories dynamically specified through natural language in open-vocabulary detection.

3. TRP’s research objective and core contribution

We further emphasize that our research objective is to improve fine-grained open-vocabulary detection, rather than to validate compositional zero-shot recognition capabilities(Examples include recognizing “green tomatoes” without training samples of “green tomatoes” after seeing “red tomatoes” and “green apples”; or learning the attribute “red” and the category “tomato” separately and then recognizing the combination “red tomatoes,” which was absent during training).

TRP’s distinctive contribution is not to introduce a new attribute–object compositional learning paradigm, but to identify the high directional consistency of textual semantic displacements induced by the same attribute across different base categories and apply this prior to parameter-efficient adaptation for real-time open-vocabulary detection. Our research focuses on comparisons with other parameter-efficient fine-tuning methods in the same family, rather than comparisons with CZSL methods in compositional zero-shot recognition capabilities.

[1] Misra I, Gupta A, Hebert M. From red wine to red tomato: Composition with context[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 1160-1169.

[2] Geng Y, Zhu R, Chen J, et al. Graph-guided cross-composition feature disentanglement for compositional zero-shot learning[C]//Findings of the Association for Computational Linguistics: ACL 2025. 2025: 2678-2690.

W2 & Q2: Lack of Experimental Comparisons with Representative CZSL Methods

1. Fundamental differences between CZSL classification and FG-OVD localization.

CZSL focuses on generalization to unseen compositions, whereas FG-OVD focuses on the accuracy of fine-grained semantic discrimination and object localization. CZSL primarily investigates zero-shot generalization to attribute–object compositions, namely recognizing unseen compositions based on attributes and objects seen during training. FG-OVD primarily investigates matching fine-grained textual descriptions to specific object regions in images and localizing them. It requires not only recognizing object categories but also distinguishing differences in attributes such as color, material, and pattern within the same category, while accurately predicting object bounding boxes.

2. How to reasonably adapt CZSL methods originally designed for classification to object detection to ensure fair experimental comparisons.

We draw inspiration from L-Adapter’s approach of introducing trainable residual modules within a frozen text encoder. Considering that the computational and maintenance costs of its global compositional graph and GNN may grow with the vocabulary and composition scale, directly applying it to real-time open-vocabulary detection with dynamic queries presents certain challenges. Therefore, we omit components such as the compositional graph and GNN, insert only a shared residual MLP after the final CLIP text Transformer block and before the final LayerNorm. We name this simplified text-tower adaptation baseline T-Adapter to distinguish it from the original L-Adapter.

The mean ± standard deviation of mAP (%) across three random seeds is shown below (the results in the table differ slightly from those in the paper because they summarize results from three random seeds):

BenchmarkT-AdapterTCRTRP
Hard48.45 ± 0.2646.31 ± 0.4850.14 ± 0.38
Medium56.44 ± 0.2154.90 ± 0.5357.13 ± 0.20
Easy59.72 ± 0.2758.56 ± 0.6859.81 ± 0.35
Trivial58.59 ± 0.2258.54 ± 0.3959.38 ± 0.10
Color45.75 ± 0.1243.53 ± 0.0948.05 ± 0.72
Material53.70 ± 0.3251.47 ± 0.5254.07 ± 0.10
Pattern55.32 ± 1.1652.76 ± 1.0255.60 ± 0.96
Transparency63.05 ± 1.0061.18 ± 1.5263.24 ± 1.30
Overall55.13 ± 0.3853.41 ± 0.6055.93 ± 0.26

Each benchmark is first averaged over valid N values, and Overall is the macro-average across the eight benchmarks. TRP’s Overall is 0.80 percentage points higher than T-Adapter, with gains of 1.69 and 2.30 percentage points on Hard and Color, respectively. We attribute the improvement to the semantic displacement conditioning used by TRP, which explicitly models the relationship between base and fine-grained descriptions, rather than directly introducing a residual adapter inside the text encoder. In addition, because CLIP is an integrated pretrained model and the T-Adapter module is located inside the text tower, it cannot directly reuse the standard CLIP final text embedding cache. Consequently, T-Adapter cannot directly reuse the precomputed final text embeddings during training and must execute the text encoder at each iteration, introducing additional computational overhead. We further report parameter efficiency and training overhead (the results in the table differ slightly from those in the paper because they summarize results from three random seeds):

MethodUpdated componentTrainable params (M)Model ratio (%)F+B time (ms/step)FG-OVD mAP (%)COCO mAP (%)
TRPTextual adapter0.3282.51163.255.93 ± 0.2637.3
T-AdapterText-tower adapter0.3292.51518.455.13 ± 0.3837.3
Head-FTDetection head2.31717.71139.255.38 ± 0.242.6
Full-FTBackbone, neck and head12.7697.49223.154.45 ± 0.212.5

TRP can directly reuse precomputed frozen text embeddings, avoiding repeated execution of the complete text tower during training. Based on the values in the table, T-Adapter’s forward–backward time per step is approximately 3.18 times that of TRP.

Revision: We have clarified the differences between CZSL classification and FG-OVD localization, introduced T-Adapter as a simplified baseline inspired by L-Adapter, and added comparisons of detection performance across three random seeds, parameter efficiency, and training overhead (Appendix B.5).

W3 & Q3: Other weaknesses

1. Sources of the comparison baselines

TCR is a comparison baseline inspired by CoCoOp and constructed for fine-grained open-vocabulary detection. Lines 291–292 of the original manuscript explicitly state this design inspiration. Therefore, TCR is not a direct reproduction of an existing method with the same name, nor is there an original paper that directly proposes TCR. Specifically, we draw on CoCoOp’s conditional prompt learning approach, conditioning on fine-grained text embeddings and predicting textual residuals through a shared generator to construct a comparison method suitable for detection tasks. Since research on parameter-efficient fine-tuning for real-time open-vocabulary detectors remains limited, our work also includes introducing multiple published parameter-efficient fine-tuning strategies into the same detection framework for comparison with TRP. Similarly, TPT is a shared textual residual baseline inspired by TaskRes. Lines 290–291 of the original manuscript explicitly use “Inspired by TaskRes” to indicate its source, rather than presenting it as a direct reproduction of TaskRes.

2. Formatting issue

We thank the reviewer for pointing this out. The issue in line 140 was caused by a LaTeX formatting error and has now been corrected.

Reviewer_ddrDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TRP, a lightweight adaptation method for fine-grained real-time open-vocabulary detection. A shared residual generator takes the difference between frozen fine-grained and coarse-category text embeddings and produces an additive correction to the fine-grained embedding. Both the text encoder and detector remain frozen. The adapted query embeddings can be cached before image inference, preserving the existing detection pipeline. Experiments on eight FG-OVD benchmark settings report improvements over the tested adaptation baselines on YOLO-World, gains over zero-shot YOLOE, and nearly unchanged image-inference latency with cached query features.

Critical strengths

TRP provides a practical adaptation strategy: it trains only a small textual adapter and caches the adapted embeddings before inference, maintaining nearly unchanged image-inference latency. TRP consistently outperforms the closest internal baseline, TCR, across all eight YOLO-World benchmark settings.

Critical weaknesses

The external comparison uses inconsistent evaluation settings. In Figure 1 and Table 8, the reference-detector scores correspond to fixed N=5 for difficulty-based benchmarks and N=2 for attribute-based benchmarks, whereas TRP results average over all valid N. This mismatch prevents a direct comparison and weakens the claimed competitiveness with heavier detectors. The proposed transferability explanation lacks direct validation. Figure 2 shows similarities between textual residuals across categories, but the paper does not explicitly evaluate transfer to held-out base categories or attribute combinations. The current results therefore do not clearly distinguish transferable attribute learning from adaptation to the evaluated benchmark distribution. A controlled generalization evaluation would strengthen this interpretation.

Questions for authors
  1. Can you report compatible evaluation results for Figure 1 and Table 8, using the same negative counts or aggregation procedure for all methods? How does this affect the conclusions?
  2. Is a single adapter trained across all eight benchmark settings, or a separate adapter for each setting? What is the training–test overlap in base categories and attribute combinations, and do the gains persist on explicitly held-out categories or combinations?
  3. Are TCR and TRP matched in generator architecture, parameter count, initialization, and normalization? Are their performance differences consistent across multiple training seeds?
5 comments

We sincerely thank the reviewer for the positive evaluation and constructive feedback. We appreciate the recognition that TRP provides a practical adaptation strategy, maintaining nearly unchanged image-inference latency through a lightweight textual adapter and cached embeddings. We also appreciate the acknowledgment that TRP consistently outperforms all other parameter-efficient fine-tuning methods across all eight YOLO-World benchmark settings. We address the reviewer’s concerns point by point below.

W1 & Q1: On the fairness of comparisons with external methods

1. TRP’s performance advantage remains after unifying the evaluation settings.

We thank the reviewer for pointing out the inconsistencies in evaluation settings and the unclear labeling of training status. We have revised Table 8 according to the evaluation protocol of Bianchi et al.: N=5 for difficulty benchmarks and N=2 for attribute benchmarks. We have also explicitly labeled whether each method was trained on the FG-OVD training sets in both Figure 1 and Table 8.

BenchmarkNTRPTCRHead-FT
Hard549.5946.3646.78
Medium557.3356.3156.56
Easy560.1960.0060.04
Trivial559.5760.0960.24
Color253.0548.1151.52
Material257.7654.8756.50
Pattern255.7055.8556.22
Transparency260.3558.7457.32

At the matched evaluation operating points, TRP outperforms TCR by 3.23 and 4.94 percentage points on Hard and Color, respectively, and Head-FT by 2.81 and 1.53 percentage points, respectively. Therefore, the advantages on these two benchmarks do not arise solely from averaging over different N values.

2. The conclusion regarding TRP’s competitiveness relative to larger detectors remains after unifying the evaluation settings.

The external comparison results under the unified evaluation settings are shown below. Train indicates whether a method was trained on the FG-OVD training sets.

MethodTrainHard (N=5)Medium (N=5)Easy (N=5)Trivial (N=5)Color (N=2)Material (N=2)Pattern (N=2)Transp. (N=2)
OWL (B/16)No26.239.838.453.945.337.326.634.1
OWL (L/14)No26.539.344.065.143.844.936.029.2
OWLv2 (B/16)No25.338.540.052.953.336.923.312.2
OWLv2 (L/14)No25.441.242.863.253.336.923.312.2
DeticNo11.518.618.669.721.538.830.124.6
ViLDNo22.136.139.956.643.234.924.530.1
Grounding DINONo16.627.930.162.741.030.231.225.4
CORANo13.820.020.435.125.019.322.027.9
YOLO-Worldv2-S (zero-shot)No2.64.34.48.15.64.65.11.7
YOLO-Worldv2-S + TRPYes49.657.360.259.653.157.855.760.3

The comparison with larger open-vocabulary detectors is not intended to demonstrate that TRP outperforms these detectors, but serves the following two purposes:

  • To highlight the fine-grained detection performance gap: Existing real-time detectors, such as YOLO-World, still exhibit a substantial performance gap compared with larger open-vocabulary detectors on fine-grained detection tasks, motivating the need to enhance their real-time fine-grained perception capabilities.

  • To demonstrate the practical value of TRP: After adaptation with TRP, YOLO-World’s fine-grained detection performance improves substantially, suggesting that lightweight real-time detectors have the potential to serve as practical alternatives to larger detectors in FG-OVD-like scenarios.

The original conclusions remain valid after unifying the evaluation settings. The previous inconsistencies in evaluation settings and unclear labeling of training status were our oversight. We thank the reviewer again for pointing them out.

Revision: We have added experiments evaluating generalization to held-out base categories (Appendix B.3). All revisions are highlighted in blue.

W2 & Q2. The Claimed Transferability of the Method Lacks Direct Experimental Validation.

We thank the reviewer for pointing out the lack of experimental validation of cross-category transferability. To address this, we have added held-out base-category experiments to examine the generalization ability of TRP and TCR on base categories unseen during adaptation training.

We constructed two non-overlapping base-category holdout splits, each containing 10 base categories. For each split, we removed images containing positive instances of the held-out categories from all eight training subsets and filtered candidate texts mentioning these categories or their aliases, including negative candidates. TRP and TCR were each trained for 20 epochs with seeds 0, 1, and 2 under the same settings. We report the mean and standard deviation of mAP50:95 on Hard (N=5) and Color (N=2).

SplitHeld-out base categories (10 categories)
fold0laptop, ladder, watch, lamp, napkin, chair, bicycle, mirror, blade, basket
fold1vase, dog, knife, newspaper, earphone, cup, bucket, bottle, car, television

Results are reported as mean ± standard deviation of mAP50:95 (%) across seeds 0, 1, and 2, computed exclusively on test samples corresponding to the held-out base categories; gains are measured in mAP percentage points.

SplitBenchmarkTRPTCRTRP − TCR
fold0Hard, N=528.72 ± 0.7523.01 ± 0.55+5.70
fold0Color, N=236.58 ± 0.5826.41 ± 2.30+10.17
fold1Hard, N=546.56 ± 1.0538.46 ± 1.04+8.10
fold1Color, N=251.11 ± 1.2442.89 ± 0.81+8.22

The original manuscript did not include this controlled experiment because our research focused on efficiently adapting frozen detectors in practical open-vocabulary detection scenarios involving multiple base categories and attribute descriptions, rather than separately validating cross-category transferability. Following the reviewer's suggestion, we added held-out base-category experiments, which show that TRP consistently outperforms TCR even on base categories unseen during adaptation training. These results are consistent with the text embedding analysis in Figure 2 and support the view that attribute-induced semantic displacements help share adaptation patterns across base categories, further explaining TRP's performance advantage over TCR.

Revision: We have added experiments evaluating generalization to held-out base categories (Appendix B.3). All revisions are highlighted in blue.

Q2: On adapter training and cross-category generalization

1. Is a single adapter trained jointly, or is a separate adapter trained for each benchmark?

We jointly train a single shared adapter on all eight FG-OVD training subsets, rather than training a separate adapter for each benchmark or each number of negative descriptions, N. After training, the same adapter is evaluated on all eight benchmarks under their different N settings.

2. How much overlap is there between the training and test sets in base categories and attribute combinations?

We calculated the proportions of base categories and complete fine-grained descriptions in the test set that appear in the training set, where the training texts include positive descriptions and negative candidates.

BenchmarkBase categories appearing in trainingComplete combinations appearing in training
Hard89/104 (85.6%)692/1965 (35.2%)
Medium87/101 (86.1%)572/1734 (33.0%)
Easy78/89 (87.6%)113/947 (11.9%)
Trivial89/104 (85.6%)692/1965 (35.2%)
Color88/100 (88.0%)632/1801 (35.1%)
Material83/97 (85.6%)629/1767 (35.6%)
Pattern56/61 (91.8%)48/326 (14.7%)
Transparency24/30 (80.0%)30/190 (15.8%)

These results show that the training and test sets do not completely overlap in base categories or attribute combinations. To further directly examine transferability, we added held-out base-category experiments to evaluate the generalization ability of TRP and TCR on base categories unseen during adaptation training.

Q3: On the rigor and stability of the comparison between TCR and TRP.

1. Architectural consistency

We thank the reviewer for highlighting the importance of the comparison settings. TCR and TRP use identical generator architectures, parameter counts, and normalization strategies. The only difference is the generator input: TRP uses the difference between the fine-grained text embedding and the base-category text embedding, whereas TCR uses the fine-grained text embedding.

Comparison itemTRPTCR
Residual generator architecture512→256→128→256→512512\rightarrow256\rightarrow128\rightarrow256\rightarrow512512→256→128→256→512512\rightarrow256\rightarrow128\rightarrow256\rightarrow512
Trainable parameter count328,832328,832
Parameter initializationuniform distribution linear-layer initializationuniform distribution linear-layer initialization
Normalization strategyL2 normalization L2 normalization

2. Stability of the results

We have supplemented the experiments with 10 paired random seeds (seeds 0–9). The paired experimental results of TRP and TCR for seeds 0–9 are shown below, with mAP (%) as the metric. Each sub-benchmark is first averaged over valid N values, and Overall is the macro-average across the eight sub-benchmarks.

BenchmarkMean TRP − TCR Gain95% Paired t Confidence IntervalSeeds with Positive Gains
Overall+2.38[2.06, 2.71]10/10
Color+4.39[3.90, 4.89]10/10
Hard+3.65[3.34, 3.95]10/10

The paired statistics are shown below, with gains measured in mAP percentage points. Confidence intervals are calculated from the TRP − TCR differences under the same seed.

Therefore, TRP's advantage over TCR is stable and significant, especially on the Color and Hard benchmarks.

Reviewer_tvFbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TRP adapts a frozen real-time open-vocabulary detector by editing cached text embeddings. For each fine-grained description and a coarse base description, a shared MLP maps the embedding difference tf − tb to a residual, which is added to tf and L2-normalized. The detector and text encoder stay frozen, and the adapted embeddings are cached before inference. The paper trains this adapter on FG-OVD splits derived from COCO training images and evaluates it on the official FG-OVD tests with YOLO-Worldv2-S and YOLOE-v8s. The controlled comparisons are against full and head fine-tuning and against shared or text-conditioned residual prompts.

Critical strengths

S1. On YOLO-World, TRP is compared with full fine-tuning, head fine-tuning, visual prompts, and text residuals under the same FG-OVD data, detection loss, and schedule (Section 4.1, Tables 1 and 5). Conditioning on tf − tb reaches 56.2 overall mAP, against 54.1 for a residual generated from tf and 34.9 for one shared residual. Hard and Color improve by 3.9 and 5.3 points over the text-conditioned residual. Table 7 shows that a linear map scores 56.1, so the result is tied to the conditioning signal rather than to MLP depth.

S2. The efficiency measurements match the deployment claim that the adapter is offline. Table 6 times cached features on one A100 and reports 8.78 ms versus 8.71 ms per image for YOLO-Worldv2-S, or 113.9 versus 114.8 FPS. Table 4 trains 0.328M parameters, and the frozen detector can be returned to its original vocabulary by disabling the adapter.

Critical weaknesses

W1. Figure 1 and Table 8 are the evidence offered for making a real-time detector competitive with heavier open-vocabulary detectors. The reference entries match Bianchi et al., CVPR 2024 (arXiv:2311.17518), Tables 2 and 3 entry for entry: difficulty benchmarks there are reported at N=5 and attribute benchmarks at N=2, for detectors evaluated without FG-OVD training. The TRP row is instead the paper’s mAP averaged over valid N, including N=0, after 20 epochs on the FG-OVD training subsets (Table 1 and Appendix B.3). Appendix B.1 says the reference numbers were not rerun and are not a controlled comparison of model size or runtime. It does not state the difference in N or training status. This affects the significance claim in the introduction. Reporting every method at the same N, with training status labeled, would correct it. The current figure cannot support that claim.

W2. The adaptation itself is a small change over a text-conditioned residual, and the transfer story is untested. Table 5 gives a 2.1-point overall gain over TCR, from one logged seed (Table 9, seed 0), with no interval. Figure 2 motivates shared attribute directions, but there is no held-out attribute or held-out base-category evaluation. The implemented TPT baseline is one residual shared by all categories. It is not the per-category residual in TaskRes (Yu et al., CVPR 2023, arXiv:2211.10277), which the related work cites as the category-specific alternative. On YOLOE, Table 3 reports a rise from 38.1 to 51.3 overall mAP against the unadapted detector only, while Trivial falls from 59.7 to 54.3. A multi-seed comparison with TCR, plus a held-out vocabulary test, would address the claim. Another sentence in the related work would not.

Questions for authors

Q1. Please report TRP, TCR, head fine-tuning, and the reference detectors at the operating points used by Bianchi et al.: N=5 on the difficulty benchmarks and N=2 on the attribute benchmarks. Please also state whether any reference detector was trained on the FG-OVD training split. (W1.) If the Hard and Color margins remain large at those matched points, the practical comparison can be restated. If they depend on averaging over easy values of N or on comparing a trained adapter with zero-shot detectors, Figure 1 should not be used as evidence of competitiveness.

Q2. Across multiple seeds, what is the paired variability of TRP minus TCR on Hard, Color, and the eight-benchmark average? If the generator is trained without a held-out set of attributes or base categories, what accuracy does it reach on that held-out set? (W2.) Stable gains and positive held-out transfer would support the shared-direction claim and could move the rating toward weak acceptance. A gain within seed noise, or a failure on held-out attributes, would leave the contribution as in-domain prompt fitting.

Q3. On YOLOE-v8s, what do TCR and head fine-tuning score under the same 20-epoch schedule? The reported gain is relative to unadapted YOLOE, and Trivial moves in the opposite direction. A TCR control would show whether the YOLOE result isolates tf − tb.

5 comments

We sincerely thank the reviewer for the constructive feedback. We appreciate the reviewer's recognition of the effectiveness of our textual residual prompt design, the rigorous experimental comparisons, and TRP's negligible inference overhead. We address the concerns and questions point by point below.

W1. Inconsistent Evaluation Settings in Comparisons with Larger Open-Vocabulary Detectors in Figure 1 and Table 8

The original purpose of Figure 1 and Table 8 was twofold:

  1. To highlight the fine-grained detection gap: Existing real-time detectors, such as YOLO-World, still exhibit a substantial performance gap compared with larger open-vocabulary detectors on fine-grained detection tasks, motivating the need to enhance their fine-grained perception capabilities.

  2. To demonstrate the practical value of TRP: After adapting YOLO-World with TRP, its fine-grained detection performance improves substantially, suggesting that lightweight real-time detectors can serve as practical alternatives to larger detectors in FG-OVD-like scenarios.

We acknowledge that the original presentation did not clearly distinguish these motivations and may have implied a controlled comparison with larger detectors. Our primary claim is TRP's effective and parameter-efficient adaptation compared with other fine-tuning methods under matched settings, rather than superiority over larger open-vocabulary detectors.

We thank the reviewer for pointing out the inconsistencies in evaluation settings and the unclear labeling of training status. We have revised Table 8 to follow the evaluation protocol of Bianchi et al., using (N=5) for difficulty benchmarks and (N=2) for attribute benchmarks. We have also explicitly labeled the FG-OVD training status of each method in both Figure 1 and Table 8.

Revision: We have revised the potentially ambiguous statements in the original manuscript and updated Figure 1 and Table 8 accordingly, with all revisions highlighted in blue.

W2 & Q2. The TRP introduces only a modest modification to text-conditioned residuals, while its transferability remains untested.

1. Although TRP introduces only a modest modification to text-conditioned residuals, its advantage over TCR is stable and significant, especially on the Color and Hard benchmarks.

We thank the reviewer for pointing out the lack of uncertainty estimates in the single-seed experiment. We have supplemented the experiments with 10 paired random seeds (seeds 0–9), comparing TRP and TCR under the same seed.

SeedOverall TRP (Gain)Overall TCRColor TRP (Gain)Color TCRHard TRP (Gain)Hard TCR
056.20 (+2.15)54.0548.77 (+5.30)43.4650.58 (+3.93)46.65
155.89 (+2.59)53.3048.07 (+4.43)43.6449.90 (+3.37)46.53
255.69 (+2.82)52.8747.32 (+3.83)43.4949.93 (+4.17)45.76
356.21 (+2.44)53.7748.20 (+4.32)43.8750.52 (+4.20)46.32
455.75 (+1.70)54.0547.60 (+3.55)44.0450.05 (+3.21)46.84
555.71 (+1.96)53.7647.75 (+4.26)43.4949.62 (+3.05)46.57
656.20 (+2.57)53.6347.81 (+4.12)43.6950.25 (+3.74)46.51
756.75 (+3.17)53.5748.58 (+5.12)43.4650.35 (+3.96)46.39
855.71 (+1.91)53.8047.61 (+3.56)44.0649.83 (+3.14)46.70
956.04 (+2.52)53.5248.20 (+5.44)42.7650.22 (+3.70)46.52

The paired experimental results of TRP and TCR for seeds 0–9 are shown below, with mAP (%) as the metric. Each sub-benchmark is first averaged over valid N values, and Overall is the macro-average across the eight sub-benchmarks.

BenchmarkMean TRP − TCR Gain95% Paired t Confidence IntervalSeeds with Positive Gains
Overall+2.38[2.06, 2.71]10/10
Color+4.39[3.90, 4.89]10/10
Hard+3.65[3.34, 3.95]10/10

The paired statistics are shown below, with gains measured in mAP percentage points. Confidence intervals are calculated from the TRP − TCR differences under the same seed.

Therefore, TRP's advantage over TCR is stable and significant, especially on the Color and Hard benchmarks.

W2. (Continued from the previous response)

2. Validation of transferability further explains TRP's advantage over TCR

We thank the reviewer for pointing out the lack of experimental validation of cross-category transferability. To address this, we have added held-out base-category experiments to examine the generalization ability of TRP and TCR on base categories unseen during adaptation training.

We constructed two non-overlapping base-category holdout splits, each containing 10 base categories. For each split, we removed images containing positive instances of the held-out categories from all eight training subsets and filtered candidate texts mentioning these categories or their aliases, including negative candidates. TRP and TCR were each trained for 20 epochs with seeds 0, 1, and 2 under the same settings. We report the mean and standard deviation of mAP50:95 on Hard (N=5) and Color (N=2).

SplitHeld-out base categories (10 categories)
fold0laptop, ladder, watch, lamp, napkin, chair, bicycle, mirror, blade, basket
fold1vase, dog, knife, newspaper, earphone, cup, bucket, bottle, car, television

Results are reported as mean ± standard deviation of mAP50:95 (%) across seeds 0, 1, and 2, computed exclusively on test samples corresponding to the held-out base categories; gains are measured in mAP percentage points.

SplitBenchmarkTRPTCRTRP − TCR
fold0Hard, N=528.72 ± 0.7523.01 ± 0.55+5.70
fold0Color, N=236.58 ± 0.5826.41 ± 2.30+10.17
fold1Hard, N=546.56 ± 1.0538.46 ± 1.04+8.10
fold1Color, N=251.11 ± 1.2442.89 ± 0.81+8.22

The original manuscript did not include this controlled experiment because our research focused on efficiently adapting frozen detectors in practical open-vocabulary detection scenarios involving multiple base categories and attribute descriptions, rather than separately validating cross-category transferability. Following the reviewer's suggestion, we added held-out base-category experiments, which show that TRP consistently outperforms TCR even on base categories unseen during adaptation training. These results are consistent with the text embedding analysis in Figure 2 and support the view that attribute-induced semantic displacements help share adaptation patterns across base categories, further explaining TRP's performance advantage over TCR.

3. Other weaknesses

TPT is an adaptation scheme inspired by TaskRes and designed for open-vocabulary detection: We thank the reviewer for pointing this out. TaskRes learns a separate residual for each downstream category, whereas open-vocabulary detection involves numerous, dynamically changing category descriptions. Directly adopting per-category residuals would increase parameter storage requirements and complicate the handling of unseen categories. Therefore, we construct TPT using a residual shared across all categories. Lines 290–291 of the original manuscript explicitly describe it as a shared textual residual baseline “Inspired by TaskRes,” rather than a direct reproduction of TaskRes; this wording was already present in the original submission and was not added in response to this review.

The degradation on Trivial is not unique to TRP: Section 4.3 of the original manuscript reports the performance decline on Trivial and notes that the positive and negative descriptions in this subset do not differ in fine-grained attributes. Its evaluation requirements therefore differ from the attribute discrimination primarily targeted by TRP. Supplementary YOLOE comparisons under the same random seed show similar declines for TCR and Head-FT on Trivial. Compared with YOLO-World, unadapted YOLOE already achieves relatively high performance on Trivial. We hypothesize that forgetting of existing capabilities during adaptation may cause this decline.

MethodOverall mAP (%)Trivial mAP (%)
Original YOLOE38.0659.67
TRP51.3054.29
TCR48.5053.80
Head-FT51.1054.15

For each benchmark, mAP50:95 is averaged over the valid numbers of negative descriptions, N. Overall is the macro-average across the eight FG-OVD benchmarks.

Revision: We have added 10-seed paired comparisons with 95% confidence intervals (Appendix B.2), held-out base-category generalization experiments (Appendix B.3), and supplementary comparisons of TRP with TCR and Head-FT on YOLOE (Appendix B.4). We have also clarified the distinction between TPT and TaskRes and expanded the discussion of the performance decline on the Trivial benchmark. All revisions are highlighted in blue.

Q1: Report the results at fixed N values and clarify the meaning of the comparisons in Figure 1 and Table 8.

We thank the reviewer for the suggestion. We have reorganized the results according to the operating points used by Bianchi et al.: N=5 for the difficulty benchmarks and N=2 for the attribute benchmarks, no longer directly comparing metrics averaged across N with metrics at fixed N values. The corresponding results are shown below.

BenchmarkNTRPTCRHead-FT
Hard549.5946.3646.78
Medium557.3356.3156.56
Easy560.1960.0060.04
Trivial559.5760.0960.24
Color253.0548.1151.52
Material257.7654.8756.50
Pattern255.7055.8556.22
Transparency260.3558.7457.32

At the matched operating points, TRP outperforms TCR by 3.23 and 4.94 percentage points on Hard and Color, respectively, and Head-FT by 2.81 and 1.53 percentage points, respectively. Therefore, the advantages on these two benchmarks do not arise solely from averaging over different N values.

The original purpose of this comparison was to illustrate the performance gap of real-time detectors in fine-grained detection and the practical value of TRP in narrowing this gap while retaining real-time inference capabilities. We did not originally intend to use this comparison to demonstrate that TRP outperforms larger open-vocabulary detectors under identical training conditions. However, the wording and presentation of the figures and tables in the original manuscript did not sufficiently clarify this purpose and may have caused misunderstanding.

Q3: On the lack of necessary ablation experiments on YOLOE.

We have added YOLOE ablation experiments using the same random seed and the same 20-epoch training settings.

MethodOverall mAP (%)Trivial mAP (%)
Original YOLOE38.0659.67
TRP51.3054.29
TCR48.5053.80
Head-FT51.1054.15
  1. Overall performance. TRP achieves an overall mAP 2.80 percentage points higher than TCR and reaches performance comparable to Head-FT. This is consistent with the YOLO-World experimental results.

  2. The performance decline on Trivial is not unique to TRP. Section 4.3 of the original manuscript reports the performance decline on Trivial and notes that the positive and negative descriptions in this subset do not differ in fine-grained attributes. Its evaluation requirements therefore differ from the attribute discrimination primarily targeted by TRP. Supplementary YOLOE comparisons under the same random seed show similar declines for TCR and Head-FT on Trivial. Compared with YOLO-World, unadapted YOLOE already achieves relatively high performance on Trivial. We hypothesize that forgetting of existing capabilities during adaptation may cause this decline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.