acceptodds
Under review as a conference paper at ICLR 2027

SetOPD: From Few Visual Exemplars to Multimodal Candidate Sets for Remote-Sensing Open-Prompt Detection

Abstract

Open-prompt detection (OPD) allows users to specify targets with text, visual exemplars, or both. Existing multimodal OPD typically compresses multiple exemplars into a class-level prototype and combines text and vision in prompt or representation space, leaving modality-specific candidate states unavailable for explicit cross-modal comparison. We formulate these two limitations from a set perspective. SetOPD first introduces Base–Residual prompting, which reads boxed exemplars in scene context and maps pooled support evidence into a fixed-capacity anchor–correction visual memory that drives a dedicated visual pathway. We further introduce Paired-Query Arbitration (PQA), in which text and visual readers decode from a shared prompt-conditioned initialization, producing naturally paired candidate states that are arbitrated within each pair and subsequently refined as a set. Across 11 heterogeneous remote-sensing sources, including four fully held-out datasets, PQA outperforms representation-level joint prompting on every source and raises Macro11 AP from 54.00 to 55.57. It also improves over the text and Base–Residual visual readers by 1.66 and 2.02 AP, respectively, while adding only 0.63M trainable parameters. Detection-level analysis further shows that the two readers recover complementary objects and that PQA preserves a substantial fraction of this modality-exclusive evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-prompt detection, visual prompting, multimodal object detection, remote sensing, open-vocabulary detection, set prediction, few-shot learning, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u4HLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SetOPD to combine text and visual examples for remote sensing open-prompt detection. It encodes pooled example features into a compact visual memory, generates text-based and visual detection candidates separately, and combines paired candidate features to produce the final predictions. Across 11 datasets, including four excluded from adaptation training, it reports Macro11 AP of 55.57.

Critical strengths

The paper addresses a practical way for users to specify RS detection targets. The paper offers a useful perspective on combining different sources of evidence.

The main comparison has useful breadth and shares a base detector. PQA exceeds Joint Prompt Representation on all eleven sources. The experiments are solid.

The paper examines limitations directly.

Critical weaknesses

The practical flexibility of the open-prompt interface is insufficiently demonstrated. The introduction argues that "direct target specification becomes a central requirement" (p. 1), but the main evaluation uses a harmonized 153-category space and five boxed exemplars per category (p. 7). Held-out datasets provide useful evidence of source transfer, but do not by themselves demonstrate reliable control through varied user prompts. There are no controlled evaluations of ambiguous category descriptions, conflicting text and exemplars, or prompts selecting different subtypes within one category. The results therefore leave the central benefit of flexible target specification insufficiently tested.

The practical value of the added system complexity is unclear. The paper highlights a 1.57-point Macro11 AP gain with "only 0.63M trainable parameters" (p. 3), but the full prediction path includes separate text and visual decoding streams, support-example encoding, and candidate fusion. End-to-end latency and memory measurements under comparable prompting conditions is required.

Index consistency is not an object-correspondence guarantee. The introduction says PQA “resolves it structurally” about cross-modal assignment (p. 2). Shared initialization and permutation equivariance ensure consistent indexing, but two modality-conditioned decoders can still associate an index with different objects.

Questions for authors

See above.

4 comments

Dear Reviewer u4HL,

We sincerely thank you for your careful review, constructive feedback, and recognition of the strengths of our work. We particularly appreciate your positive assessment of the candidate-level multimodal collaboration framework, the breadth of our evaluation across 11 datasets, and the experimental evidence supporting PQA. Your comments have helped us further clarify the scope, computational efficiency, and candidate-pairing mechanism of SetOPD. We address your concerns point by point below.

Q1: On the practical flexibility of the open-prompt interface

We thank the reviewer for this comment. However, we disagree that evaluations involving ambiguous descriptions, contradictory text–visual prompts, or arbitrary intra-category subtype selection are necessary to validate the central contributions of SetOPD.

Our evaluation follows established practices in remote-sensing open-vocabulary and open-prompt detection. For example, OpenRSD (ICCV 2025) evaluates text- and image-prompt detection on public remote-sensing benchmarks, while LAE-DINO (AAAI 2025) evaluates open-vocabulary detection on DIOR, DOTA-v2.0, and LAE-80C. Similarly, RS-MPOD (2026) evaluates textual, visual, and multimodal prompting through standard, cross-dataset, and fine-grained detection benchmarks.

Importantly, specialized semantic capabilities constitute distinct research objectives. For instance, Hi-OPD (2026) specifically investigates hierarchical category prompting and introduces dedicated parent/grandparent-query evaluations to assess hierarchical retrieval. Such evaluations are appropriate when semantic hierarchy is the methodological focus, but they are not prerequisites for evaluating every open-prompt detection framework.

SetOPD addresses a different methodological question: how to encode a few visual exemplars into an effective visual memory and how to combine independently decoded text- and visual-conditioned detection candidates. It does not propose a natural-language ambiguity resolver or a mechanism for resolving contradictory user intentions.

The unified 153-category space serves as a consistent evaluation taxonomy rather than a fixed 153-class prediction head. Our experiments directly evaluate the proposed mechanisms through Text-only, Visual-only, and Text+Visual prompting, different support-example counts, and 11 heterogeneous datasets, including four held-out sources.

Therefore, we believe these experiments provide appropriate and comprehensive evidence for our stated contributions. While ambiguity resolution, conflicting prompts, and intra-category semantic control are valuable research directions, they address capabilities beyond the specific mechanisms investigated in this paper.

Q2: Computational overhead of the dual-reader SetOPD framework

We thank the reviewer for this comment. We clarify that the additional 0.63M trainable parameters refer to the PQA module, rather than the computational cost of the entire dual-reader system. To quantify the actual inference overhead, we conducted additional profiling under identical experimental conditions.

All methods were evaluated on a single NVIDIA RTX 3090 GPU using FP32, with the same 120 DOTA-v2.0 images, identical preprocessing (short side 800 pixels, long side at most 1333 pixels), 5 visual exemplars per class, and a maximum of 300 output detections.

Text and visual support prompt encodings were cached. After warm-up, latency measurements included image loading, preprocessing, model inference, and postprocessing. We measured computation, peak allocated GPU memory, inference latency, and throughput under the same conditions.

MethodGFLOPs/ImagePeak Memory (MiB)Latency (ms)FPS (B=1)FPS (B=60)
Joint Prompting372.66858.17113.468.8111.60
Late Fusion410.42858.17124.148.0612.16
Relation-Aware Selection411.481334.24136.707.3210.56
SetOPD (Ours)411.82860.56128.987.7511.80

GFLOPs were measured using the same PyTorch profiler scope, counting one multiply-add as two floating-point operations. Custom deformable-attention operations are not fully covered. Peak GPU memory was measured at batch size 1. The reported latency reflects image inference with cached prompt encodings rather than first-time support encoding.

Compared with Joint Prompting, SetOPD increases measured computation by 39.16 GFLOPs (10.5%), latency by 15.52 ms (13.7%), and peak GPU memory by 2.39 MiB.

More importantly, Late Fusion provides a direct computational reference because it also requires two modality-specific decoding streams. Compared with Late Fusion, SetOPD adds only 1.40 measured GFLOPs (0.34%) and 4.84 ms (3.9%) per image.

These results distinguish the computational cost of the complete dual-reader inference path from the incremental overhead of PQA. While maintaining separate modality-specific readers incurs additional computation compared with Joint Prompting, the candidate-level collaboration introduced by PQA adds relatively little overhead beyond the dual-reader architecture.

Q3: On index consistency and object-level correspondence

We thank the reviewer for this important observation. We clarify that shared query initialization provides a structural pairing prior rather than a strict guarantee of object-level correspondence. The phrase "resolves it structurally" refers to establishing index-aligned candidate pairs without requiring explicit cross-modal assignment.

Our original Table 5 already provides pairing-consistency analyses on DIOR, DOTA-v2.0, and HRSC2016. To further validate the reliability of shared initialization across additional data sources, we conducted a new, independent reader-to-reader correspondence analysis on three other datasets: FAIR1M, SODA-A, and HRRSD, including the held-out HRRSD dataset.

For each dataset, we evaluated 128 images and all 900 same-index query pairs per image. When both readers produced class-correct detections at IoU ≥0.5\geq 0.5, we examined whether they corresponded to the same ground-truth object.

DatasetSame GTDifferent GTSame-GT Rate
FAIR1M3,196999.72%
SODA-A3,859699.84%
HRRSD (Held-out)1970100.00%
Total7,2521599.79%

Among 7,267 jointly correct query pairs, only 15 pairs (0.21%) correspond to different GT objects, while 7,252 pairs (99.79%) refer to the same object.

These results provide further empirical support for our PQA design. Since the two readers independently decode from shared initial queries, the vast majority of jointly correct candidate pairs exhibit consistent object correspondence, enabling direct index-based arbitration without an additional cross-modal matching procedure.

We acknowledge that object-level mismatches can still occur. Therefore, PQA uses index alignment as an effective structural prior rather than assuming guaranteed object identity. The learned gate adaptively arbitrates within each candidate pair, while SetBlock enables interactions across the fused candidate set.

We sincerely thank the reviewer again for the constructive comments and valuable suggestions. The additional computational profiling and three-dataset candidate-correspondence analysis further strengthen the empirical evidence for SetOPD. We will incorporate these results and clarify the scope of our method and the role of shared query initialization in the revised manuscript.

Thank you for the clarifications and additional experiments. The rebuttal addresses my main concerns. I will keep my rating.

Thank you for your positive feedback and for considering our additional experiments and clarifications. We sincerely appreciate your time and thoughtful assessment.

Reviewer_1r3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SetOPD for open-prompt object detection in remote-sensing imagery, using text prompts, visual exemplars, or both. The method introduces Base–Residual (BR) prompting, which encodes a small set of boxed exemplars into a fixed-size visual memory, and Paired-Query Arbitration (PQA), which combines text- and visual-conditioned candidate states. PQA uses shared query initialization to establish candidate pairs, applies a learned gate within each pair, and refines the resulting candidates with a set-level attention block. Experiments across multiple remote-sensing sources evaluate the overall framework, support-example efficiency, and alternative visual-prompt and collaboration designs.

Critical strengths
  1. The paper addresses a relevant problem with a sensible approach to multimodal collaboration. Text descriptions and visual exemplars offer complementary ways to specify objects in remote-sensing imagery, where category names may not capture the appearance of the target adequately. Allowing the two readers to develop their own candidate states before combining them is a reasonable design choice. Shared initialization provides a straightforward correspondence prior for these candidates, making the proposed collaboration mechanism a practical extension of an existing prompt-conditioned detector.
  2. The evaluation provides useful evidence of the framework’s effectiveness across different sources. Table 2 shows consistent improvements over joint prompt representation, with SetOPD also outperforming the stronger unimodal reader on most sources. The inclusion of held-out sources strengthens the empirical evaluation by examining transfer beyond the training sources. Table 1 provides useful structural controls and examines performance with different numbers of support exemplars. In particular, the duplicated-mean baseline controls for prompt length, making the visual-prompt comparison more informative than a comparison against a single prototype alone.
Critical weaknesses
  1. The specific benefit of the Base–Residual factorization is not sufficiently isolated. Section 3.3 describes the default single-residual configuration as a mean-dependent anchor–correction mapping. Although Table 1 includes useful Base-only and duplicated-mean controls with matched training budgets, these comparisons do not fully distinguish the benefit of the proposed decomposition from that of learning a more expressive two-token representation. A parameter-matched nonlinear encoder that maps the pooled mean to two distinct tokens, using the same evidence-extraction and readout architectures and training budget, would provide a more direct control. This comparison is important for assessing BR as a central methodological contribution. Reporting variability across independently sampled support sets would also strengthen the evidence for the few-exemplar gains.

  2. The contributions of learned arbitration and set-level refinement remain unclear. Table 2 supports the effectiveness of the complete PQA pipeline, but does not isolate the contributions of its learned gate and subsequent SetBlock. The fixed-gate alternatives described in Section 3.3 provide natural controls. Comparing learned gating with fixed averaging, and separately removing the SetBlock while keeping the rest of the pipeline unchanged, would clarify the contribution of each component. Simple late fusion of the same readers would provide another useful reference. The architectures, trainable components, and optimization budgets of the collaboration baselines also need clearer specification. These comparisons and clarifications are important for attributing the reported gains to the arbitration and candidate-set reasoning mechanisms emphasized in the paper.

Questions for authors
  1. How does BR compare with a parameter-matched nonlinear encoder that maps the pooled evidence mean to two distinct tokens, using the same evidence-extraction architecture, readout, and training budget?
  2. How does the learned gate compare with fixed averaging when the SetBlock, final readout, and training protocol are kept unchanged?
  3. How much does the SetBlock contribute when the learned gate and the rest of the pipeline are held fixed?
  4. How does SetOPD compare with a simple late-fusion baseline that combines the same two readers’ final predictions?
  5. What are the architectures, trainable components, and optimization budgets of the joint-prompt and relation-aware baselines, and how do these differ from PQA?
  6. Are the improvements in Table 1 consistent across independently sampled support sets, particularly when only one or a few exemplars are available?
3 comments

Dear Reviewer 1r3t,

We sincerely thank you for your careful review, constructive feedback, and recognition of the strengths of our work. We particularly appreciate your positive assessment of our multimodal candidate collaboration framework, comprehensive evaluation across heterogeneous datasets, and structural analysis of visual prompting. Your comments have helped us further strengthen the experimental validation of Base–Residual prompting and Paired-Query Arbitration (PQA). We address your concerns point by point below.

Q1: How does BR compare with a parameter-matched nonlinear encoder that maps the pooled evidence mean to two distinct tokens?

We conducted an additional controlled experiment comparing Base–Residual (BR) with a parameter-matched nonlinear two-token encoder (Nonlinear Mean→2). Unlike the duplicated-mean baseline in Table 1, this encoder learns two distinct nonlinear representations from the pooled evidence mean. Both methods use identical evidence extraction, classification readout, training budgets, and evaluation protocols, with only the prompt encoding mechanism varied.

We further evaluated both methods across five independently sampled support sets to assess their performance and consistency.

DatasetBR AP (Mean ± Std.)Nonlinear Mean→2 AP (Mean ± Std.)ΔAP
DIOR58.03 ± 0.2058.08 ± 0.16−0.05
DOTA-v2.046.31 ± 0.4646.84 ± 0.90−0.54
FAIR1M28.42 ± 0.3328.05 ± 0.37+0.37
HRSC201677.97 ± 0.5075.56 ± 2.92+2.41
SIMD71.96 ± 0.2670.74 ± 0.19+1.22
SODA-A39.38 ± 0.2138.77 ± 0.13+0.61
ShipRSImageNet58.09 ± 0.7057.77 ± 0.78+0.32
WHU-Buildings78.91 ± 0.0279.11 ± 0.07−0.21
HRRSD32.02 ± 0.2229.35 ± 0.39+2.67
RSOD42.61 ± 0.7035.15 ± 2.97+7.46
VEDAI54.64 ± 0.7149.94 ± 1.76+4.70
Macro1153.48 ± 0.1051.76 ± 0.42+1.73

AP differences are calculated from unrounded values.

BR outperforms the parameter-matched nonlinear two-token encoder by 1.73 Macro11 AP, achieving higher average performance on 8 of the 11 datasets.

Notably, BR achieves improvements on all four held-out datasets, with an average gain of 4.31 AP, demonstrating stronger cross-source generalization.

These results provide direct experimental evidence that the proposed anchor–correction parameterization offers additional benefits beyond learning two independent nonlinear prompt tokens.

We will incorporate this comparison into the revised manuscript.

Q2: How does the learned gate compare with fixed averaging when the SetBlock, final readout, and training protocol are kept unchanged?

We conducted an additional ablation experiment, keeping the SetBlock, final readout, and training protocol identical, with only the gating mechanism varied.

PQA ConfigurationMacro11 AP
Fixed gate (gi=0.5g_i=0.5)55.29
Learned gate (Ours)55.57
Improvement+0.28

The learned gate improves Macro11 AP by 0.28, demonstrating the benefit of adaptive modality arbitration over fixed averaging.

Q3: How much does SetBlock contribute when the learned gate and the rest of the pipeline are held fixed?

We conducted a separate ablation experiment, retaining the learned gate, final readout, and training protocol, with only SetBlock removed.

PQA ConfigurationMacro11 AP
Without SetBlock54.82
With SetBlock (Ours)55.57
Improvement+0.75

SetBlock improves Macro11 AP by 0.75, demonstrating the effectiveness of set-level candidate interactions beyond independent pairwise arbitration.

Q4: How does SetOPD compare with a simple late-fusion baseline combining the same two readers' final predictions?

We conducted an additional comparison with a simple prediction-level Late Fusion baseline using the same Text Reader and Base–Residual Visual Reader. Unlike PQA, Late Fusion combines the final detection predictions without learned candidate-state arbitration or set-level refinement.

DatasetLate Fusion APSetOPD APΔAP
DIOR58.8659.96+1.10
DOTA-v2.048.8949.01+0.12
FAIR1M30.1830.53+0.35
HRSC201679.3579.95+0.60
SIMD72.4772.57+0.10
SODA-A43.9544.60+0.65
ShipRSImageNet60.3062.38+2.08
WHU-Buildings81.8383.06+1.23
HRRSD32.3332.30−0.03
RSOD40.6441.97+1.33
VEDAI55.6654.93−0.73
Macro1154.9555.57+0.62

SetOPD outperforms simple Late Fusion by 0.62 Macro11 AP, achieving higher AP on 9 of the 11 datasets. Notable improvements are observed on ShipRSImageNet (+2.08 AP), RSOD (+1.33 AP), WHU-Buildings (+1.23 AP), and DIOR (+1.10 AP).

These results demonstrate that candidate-state collaboration provides additional benefits beyond combining the final predictions of two complementary readers. In particular, the learned pairwise arbitration and subsequent SetBlock enable multimodal interactions before the final detection readout, rather than relying solely on prediction-level fusion.

We will include this comparison in the revised manuscript.

Q5: What are the architectures, trainable components, and optimization budgets of the collaboration baselines, and how do these differ from PQA?

All three methods share the same base detector and use identical training budgets, optimization settings, and evaluation protocols. They differ only in how text and visual information interact and which modules perform the fusion.

MethodArchitectureTrainable Components
Joint Prompt RepresentationConcatenates text tokens and BR visual tokens into a joint prompt, which is processed by a single decoder.BR visual-prompt encoder; prompt concatenation is parameter-free.
Relation-Aware SelectionCombines the Top-300 predictions from the Text and Visual Readers and applies class-wise relational reasoning based on appearance, ranking, and relative box geometry.Relation selector, including appearance/rank projections and geometric relation attention.
PQA (SetOPD)Uses shared initialization for two independently decoded candidate sets, followed by learned pairwise gating and SetBlock refinement.Learned pairwise gate and SetBlock.

All methods are evaluated under the same training budget and experimental protocol, ensuring that performance differences reflect their respective multimodal fusion mechanisms.

Q6: Are the improvements in Table 1 consistent across independently sampled support sets, particularly when only one or a few exemplars are available?

We conducted an additional evaluation using five independently sampled support sets, keeping the trained models and evaluation settings unchanged.

MethodMacro11 AP (Mean ± Std.)Wins Across 5 Draws
Nonlinear Mean→251.76 ± 0.420/5
Base–Residual (Ours)53.48 ± 0.105/5

BR consistently outperforms the nonlinear two-token baseline across all five support-set draws, while exhibiting substantially lower performance variability (0.10 vs. 0.42 standard deviation).

This advantage is particularly evident on held-out datasets. For example, BR achieves 42.61 ± 0.70 AP on RSOD, compared with 35.15 ± 2.97 for Nonlinear Mean→2, and 54.64 ± 0.71 AP on VEDAI, compared with 49.94 ± 1.76.

Furthermore, our original Table 1 evaluates different numbers of visual exemplars (K=1,3,5,10K=1,3,5,10). BR achieves 50.45 AP with only one exemplar, increasing to 52.08 at K=3K=3 and 53.55 at K=5K=5, with performance saturating thereafter.

Together, the original support-size analysis and the additional repeated-sampling experiments demonstrate the effectiveness and stability of Base–Residual prompting under few-exemplar settings.

We sincerely thank the reviewer again for the constructive comments and valuable suggestions. The additional parameter-matched comparison, controlled PQA ablations, late-fusion evaluation, and support-set robustness analysis further strengthen the experimental evidence for SetOPD. We will incorporate these results and clarify the corresponding implementation details in the revised manuscript.

Dear Reviewer 1r3t,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Reviewer_p4JoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal open-prompt object detection in remote sensing imagery, allowing users to specify target categories through text, a few bounding-box-annotated visual exemplars, or both. The authors propose SetOPD, a framework comprising two main components. Base–Residual prompting converts evidence from support exemplars into a fixed-capacity visual memory that drives a dedicated visual detection branch. Paired Query Arbitration (PQA) initializes the text and visual branches with shared queries, fuses paired candidate states through learned gating, and then refines the candidate set using a self-attention module. Experiments cover 11 remote sensing data sources, four of which were unseen during training. Under the same detection framework, SetOPD improves Macro11 AP from 54.00 for the Joint Prompt Representation baseline to 55.57.

Critical strengths

Preserving modality-specific candidate detection states before fusion is a reasonable approach to combining these two types of information.

The experiments cover multiple heterogeneous data sources, including four datasets unseen during training.

Critical weaknesses

The central PQA mechanism is insufficiently validated. PQA combines shared query initialization, pairwise gating, SetBlock, the (L_{\text{keep}}) retention loss, and subsequent localization and QFL classification stages. The experiments mainly evaluate the complete pipeline without isolating these components. Consequently, the improvement cannot be confidently attributed to candidate pairing or arbitration rather than accompanying training and readout choices.

Comparisons with OpenRSD, RS-MPOD, and Hi-OPD mainly use previously reported results, potentially involving different training data, category splits, support-exemplar counts, preprocessing, and evaluation protocols. The controlled experiments establish an advantage over internal baselines, but any broader claim of superiority over existing OPD methods remains insufficiently supported.

Questions for authors

Do PQA and the fusion baselines use the same localization, QFL training, and scoring settings? Could you show how much gating, SetBlock, and LkeepL_{\text{keep}} each contribute? Does shared query initialization work better than initializing the two branches independently?

Why combine text and visual scores at a fixed 1:1 ratio, and why not use the learned gating weights? On DOTA, using text-branch scores with the same PQA candidates improves AP from 48.25 to 50.14. Does this also happen on other datasets?

Beyond the extra 0.63M trainable parameters, how much computation does SetOPD add? Could you report FLOPs, peak GPU memory, inference latency, and throughput for SetOPD and the baseline under the same conditions?

4 comments

Dear Reviewer p4Jo,

We sincerely thank you for your careful review, constructive comments, and recognition of the strengths of our work. Your suggestions have helped us further clarify the methodology and strengthen the experimental validation of SetOPD. We address your concerns point by point below.

Q1: Do PQA and the fusion baselines use the same localization, QFL training, and scoring settings?

Yes. Our comparisons are strictly controlled, with only the fusion mechanism varied. All methods use identical localization strategies, QFL training protocols, final scoring rules, and evaluation settings. Therefore, the observed performance differences can be directly attributed to the different fusion mechanisms.

Q2: How much do the learned gate, SetBlock, and LkeepL_{\mathrm{keep}} contribute?

We conducted additional controlled ablation experiments to quantify the contribution of each PQA component. With only the corresponding component modified.

PQA ConfigurationMacro11 APContribution
Full PQA55.57—
Fixed gate (gi=0.5g_i=0.5)55.29+0.28
Without SetBlock54.82+0.75
Without LkeepL_{\mathrm{keep}}55.50+0.07

The results demonstrate that SetBlock contributes the largest improvement (+0.75 AP), highlighting the importance of set-level interactions among candidate representations. The learned gate provides an additional 0.28 AP, confirming the benefit of adaptive modality arbitration over fixed averaging.

LkeepL_{\mathrm{keep}} yields a smaller improvement of 0.07 AP. We will revise the manuscript to more accurately describe its role as an auxiliary regularization term.

Q3: Does shared query initialization outperform independent initialization?

We conducted an additional controlled experiment comparing shared and independent query initialization. In the shared setting, the Text Reader and Visual Reader start from the same 900 queries, whereas in the independent setting, each reader selects its own 900 initial queries. All other training and evaluation settings remain identical.

Query InitializationMacro11 AP
Independent initialization55.18
Shared initialization (Ours)55.57
Improvement+0.39

Shared initialization improves Macro11 AP by 0.39 AP, outperforming independent initialization on 10 out of 11 datasets.

These results demonstrate that shared initialization provides an effective structural prior for pairing modality-specific candidates, enabling direct index-wise collaboration without additional cross-modal matching.

Q4: Comparisons with OpenRSD, RS-MPOD, and Hi-OPD.

We thank the reviewer for this comment. Our comparisons follow the established evaluation practice in remote-sensing OPD, where methods are compared using their published results on common benchmark test sets and corresponding evaluation metrics. This practice is also adopted by previous studies, including RS-MPOD and Hi-OPD.

In Table 6, we explicitly distinguish text, visual, and text–visual prompting settings, as well as AP and AP50 metrics. For example, on DOTA-v2.0, SetOPD achieves 49.0 AP with 5 visual exemplars per class, compared with 48.0 AP reported by RS-MPOD using 32 visual exemplars.

Importantly, the external comparisons establish the competitiveness of SetOPD on shared benchmarks, while our controlled experiments across all 11 datasets, using the same detection framework and training protocols, directly demonstrate the effectiveness of the proposed fusion mechanism. These two forms of evidence provide complementary support for our conclusions.

Q5: Why combine text and visual scores at a fixed 1:1 ratio instead of using the learned gating weights?

The key motivation is that adaptive multimodal fusion has already been performed before the final classification stage.

Specifically, the learned gate first adaptively fuses the modality-specific candidate representations, followed by SetBlock for set-level interactions:

H^F=\operatorname{SetBlock}(H^P).$$ The final text-conditioned and visual-conditioned classification heads then compute their scores from the **same collaboratively refined candidate state** $h_i^F$, rather than from two independent Reader outputs. Therefore, we adopt a simple, symmetric 1:1 probability average at the final classification stage: $$p_{ic}^{F}=\frac{1}{2}p_{ic}^{T,F}+\frac{1}{2}p_{ic}^{V,F}.$$ The learned gate is designed for candidate representation arbitration, not classification-score calibration. Since adaptive modality integration has already occurred, reusing the gate weights for final scoring is not necessarily beneficial. To validate this design, we compared the two scoring strategies across all 11 datasets using **four random seeds (71–74)**. For each seed, the fused candidate states, localization boxes, and evaluation protocol were kept identical between the two scoring strategies. | Final Scoring Strategy | Seed 71 | Seed 72 | Seed 73 | Seed 74 | Mean Macro11 AP | Std. | |---|---:|---:|---:|---:|---:|---:| | **Equal-weight fusion (1:1, Ours)** | **55.57** | **55.53** | **55.49** | **55.47** | **55.52** | **0.04** | | Learned gate-weighted fusion | 55.50 | 55.40 | 55.36 | 55.33 | 55.40 | 0.07 | **Equal-weight fusion consistently outperforms learned gate-weighted fusion across all four seeds**, achieving a higher mean Macro11 AP (**55.52 vs. 55.40, +0.12 AP**) and lower performance variability (0.04 vs. 0.07 standard deviation). These results further support our choice of a **simple, symmetric, parameter-free final scoring rule following adaptive multimodal candidate fusion**, without reusing the representation-level gating weights for classification. **Q6: Does the improvement from Text Reader rescoring on DOTA generalize to other datasets?** The training-free experiments in Table 3 were designed to investigate whether the performance limitation originates from insufficient candidate generation or suboptimal classification scoring and ranking. By modifying the readout without additional training, we can assess whether useful multimodal candidates are already present. Specifically, replacing the PQA candidate scores with the original Text Reader scores improves DOTA-v2.0 AP from **48.25 to 50.14**. The second diagnostic, combining Text/PQA logits with Text Reader boxes, achieves **50.17 AP**. Both are training-free readout diagnostics. To investigate whether this phenomenon generalizes, we conducted an additional **training-free evaluation across all 11 datasets using the final SetOPD model**. We first reproduced the original Table 2 results, matching the archived AP values to four decimal places. We then **fixed the Top-300 query–class candidates selected by SetOPD, including their query indices, categories, and localization boxes, and replaced only their classification scores with the corresponding original Text Reader scores**. | Metric | DIOR | DOTA | FAIR1M | HRSC | SIMD | SODA-A | ShipRS | WHU | HRRSD | RSOD | VEDAI | Macro11 | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | **SetOPD AP** | 59.97 | 49.01 | 30.54 | 79.89 | 72.54 | 44.59 | 62.66 | 83.06 | 32.19 | 41.91 | 54.93 | **55.57** | | **Text Rescoring AP** | 59.29 | 50.20 | 29.26 | 78.55 | 72.45 | 44.26 | 60.57 | 82.99 | 33.02 | 40.57 | 54.29 | **55.04** | | **ΔAP** | −0.69 | **+1.20** | −1.28 | −1.33 | −0.09 | −0.33 | −2.09 | −0.08 | **+0.84** | −1.33 | −0.64 | **−0.53** | *AP differences are calculated from unrounded values.* The results demonstrate that **the improvement is dataset-dependent rather than universal**. Text Reader rescoring improves AP on only **2 of the 11 datasets (DOTA-v2.0 and HRRSD)**, while decreasing performance on the remaining nine. Overall, Macro11 AP decreases from **55.57 to 55.04 (−0.53 AP)**. On DOTA-v2.0, Text Reader rescoring improves the final SetOPD AP from **49.01 to 50.20**, consistent with the scoring sensitivity identified in Table 3. This further confirms that useful multimodal candidates are already present, while their classification scores can still be improved on certain datasets. However, directly replacing the final multimodal classification scores with original Text Reader scores does not consistently improve performance across datasets. Therefore, **the training-free experiments demonstrate the scoring sensitivity of existing multimodal candidates rather than the general superiority of Text Reader scoring**, supporting our decision to retain the unified multimodal classification strategy.

Q7: Beyond the additional 0.63M trainable parameters, what is the computational overhead of SetOPD?

We conducted additional computational profiling under identical experimental conditions.

All methods were evaluated on a single NVIDIA RTX 3090 GPU using FP32, with the same 120 DOTA-v2.0 images, identical preprocessing (short side 800 pixels, long side at most 1333 pixels), 5 visual exemplars per class, and a maximum of 300 output detections.

Text and visual support prompt encodings were cached. Inference latency includes image loading, preprocessing, model inference, and postprocessing. After warm-up, we measured latency, peak allocated GPU memory, and throughput under the same settings.

MethodGFLOPs / ImagePeak Memory (MiB)Latency (ms / Image)Throughput B=1 (img/s)Throughput B=60 (img/s)
Joint Prompting372.66858.17113.468.8111.60
Late Fusion410.42858.17124.148.0612.16
Relation-Aware Selection411.481334.24136.707.3210.56
SetOPD (Ours)411.82860.56128.987.7511.80

GFLOPs were measured using the same PyTorch profiler scope, counting one multiply-add as two floating-point operations. Custom deformable-attention operations are not fully covered. Peak GPU memory corresponds to peak allocated memory during batch-size-1 inference.

Compared with Joint Prompting, SetOPD increases the measured computation by approximately 39.16 GFLOPs (10.5%), inference latency by 15.52 ms (13.7%), and peak GPU memory by only 2.39 MiB.

More importantly, compared with Late Fusion, which also processes two modality-specific branches, SetOPD introduces only 1.40 additional measured GFLOPs (0.34%) and 4.84 ms (3.9%) of additional inference latency.

These results demonstrate that the additional cost of PQA candidate collaboration is relatively small compared with the computation already required by the dual-reader framework.

We will incorporate these computational results into the revised manuscript.


We sincerely thank the reviewer again for the constructive comments. The additional controlled ablations, cross-dataset scoring analysis, and computational profiling provide further evidence for the effectiveness and practical feasibility of SetOPD. We will incorporate these results and clarify the corresponding experimental protocols in the revised manuscript.

Dear Reviewer p4Jo,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,
Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall Response to All Reviewers

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful evaluations, constructive comments, and recognition of the strengths of SetOPD. We have carefully addressed the concerns raised in our individual responses and conducted additional controlled experiments to further strengthen the validation of our two main contributions: Base–Residual (BR) prompting and Paired-Query Arbitration (PQA).

We summarize the key additional findings below.

1. Base–Residual Prompting: Structural Effectiveness and Robustness (Reviewer 1r3t)

We introduced a parameter-matched nonlinear two-token encoder as a stronger baseline. Across five independently sampled support sets, BR achieves 53.48 ± 0.10 Macro11 AP, compared with 51.76 ± 0.42 for the nonlinear baseline, yielding +1.73 AP.

BR outperforms this baseline on 8 of 11 datasets, including all four held-out datasets, and consistently wins across all five support-set draws. These results provide direct evidence for the effectiveness and robustness of the proposed anchor–correction parameterization beyond a general nonlinear two-token representation.

2. PQA: Component Contributions and Candidate Collaboration (Reviewers p4Jo, 1r3t, u4HL)

Additional controlled ablations demonstrate that the learned gate contributes +0.28 AP, while SetBlock contributes +0.75 AP. Furthermore, SetOPD outperforms simple prediction-level Late Fusion by +0.62 Macro11 AP, achieving higher AP on 9 of 11 datasets.

Shared query initialization also improves performance on 10 of 11 datasets compared with independent initialization. Additional reader-to-reader correspondence analysis on three datasets shows that 99.79% of jointly correct same-index candidate pairs correspond to the same ground-truth object, providing empirical support for index-based candidate arbitration.

Together, these results substantiate the contributions of adaptive pairwise arbitration, set-level reasoning, and shared initialization.

3. Final Classification Scoring (Reviewer p4Jo)

Across four random seeds, equal-weight (1:1) scoring consistently outperforms learned gate-weighted scoring, achieving 55.52 vs. 55.40 mean Macro11 AP, with lower performance variability.

We also reproduced the original Table 2 results and conducted a training-free Text Reader rescoring analysis across all 11 datasets. With the Top-300 candidates and localization boxes fixed, replacing the final multimodal scores with original Text Reader scores decreases Macro11 AP from 55.57 to 55.04 (−0.53 AP), reducing performance on 9 of 11 datasets.

These findings support our choice of a simple equal-weight classification rule following adaptive multimodal candidate fusion.

4. Computational Efficiency and Evaluation Scope (Reviewers p4Jo, u4HL)

We additionally measured FLOPs, GPU memory, inference latency, and throughput under matched conditions. Compared with Late Fusion, which also uses two modality-specific decoding streams, SetOPD introduces only 0.34% additional measured GFLOPs and 3.9% inference latency under the cached-prompt evaluation setting.

We have also clarified the distinction between controlled internal comparisons and published external benchmark results, as well as the methodological scope of SetOPD within remote-sensing open-prompt detection.


Together, these additional experiments directly address the principal concerns regarding methodological attribution, experimental robustness, candidate correspondence, scoring design, and computational efficiency.

Detailed experimental results and explanations are provided in our individual responses to each reviewer. We will incorporate these additional findings and implementation clarifications into the revised manuscript.

We sincerely appreciate the reviewers' and Area Chair's time and consideration, and welcome further discussion.

AuthorsOpenReview ↗

Revised Manuscript — Main Contributions and Reviewer-Specific Rebuttal Summary

Dear Reviewers and Area Chair,

We have uploaded the revised manuscript. We summarize the three contributions and how the rebuttal addresses each review.

Main Contributions

1. Set-based formulation. We identify two bottlenecks in multimodal Open Prompt Detection(OPD): reducing visual support sets to category-level prototypes and fusing modalities before their detection hypotheses can be compared. SetOPD connects support encoding and candidate collaboration in a set-to-set framework respecting permutation invariance/equivariance.

2. Base–Residual (BR) prompting. A box-conditioned encoder reads exemplars in full scene context and produces a fixed two-token anchor–correction memory for an independent Visual-only reader, regardless of support-set size. BR improves Visual-only Macro11 AP from 47.11 (Base-only) to 53.55; one exemplar already achieves 50.45 AP versus 53.55 with five.

3. Paired-Query Arbitration (PQA). Text and visual readers decode independently from shared initial queries, yielding naturally paired candidate states without explicit assignment. A learned gate arbitrates within pairs, followed by set-level refinement. SetOPD achieves 55.57 Macro11 AP on 11 sources (four fully held out): +1.57 over Joint Prompting (11/11 wins), +1.66 over Text Reader, and +2.02 over Visual Reader, with only 0.63M trainable PQA parameters. It beats the stronger unimodal reader on 9/11 sources. Gains versus Joint Prompting extend to held-out sources (+0.89 AP on average).

Additional Rebuttal Evidence, by Reviewer Question

• 1r3t Q1/Q6 — BR specificity and support variability (A.8). Under matched evidence extraction, readout, parameter count, and training budget, BR scores 53.48 ± 0.10 vs. 51.76 ± 0.42 Macro11 AP for a nonlinear two-token baseline (+1.73). It wins all five independently drawn support sets and leads on all four held-out sources on average (+4.31 AP), directly addressing the concern that the benefit merely reflects a more expressive two-token mapping.

• 1r3t Q2–Q5; p4Jo Q1–Q3 — PQA attribution and controls (A.9). With matched localization, QFL, scoring, and evaluation, learned gating contributes +0.28 AP, SetBlock +0.75, and auxiliary Lkeep +0.07. Shared initialization adds +0.39 AP versus independent initialization (10/11 wins). SetOPD exceeds prediction-level Late Fusion of the same readers by +0.62 AP (9/11 wins). We also specify the architectures and trainable components of the competing fusion baselines.

• p4Jo Q5/Q6 — Scoring and rescoring (A.10). Across four seeds, equal-weight final scoring beats reuse of gate weights (55.52 vs. 55.40 mean AP). With the final Top-300 candidate list and boxes held fixed, Text Reader rescoring helps DOTA-v2.0 and HRRSD but lowers Macro11 AP from 55.57 to 55.04, showing that the original DOTA finding is not a universal advantage of Text scoring.

• u4HL Q3 — Pair correspondence (A.3, Table 5). We now state explicitly that shared initialization is a structural prior, not a guarantee of object identity. Across three additional sources, 7,252/7,267 (99.79%) jointly correct same-index pairs match the same GT; this figure is conditional on both readers being true positives, not on all queries.

• p4Jo Q7; u4HL Q2 — Computation (A.11). We report matched FLOPs, peak memory, latency, and throughput. Relative to single-decoder Joint Prompting, SetOPD incurs +10.5% measured FLOPs and +13.7% latency. Relative to dual-reader Late Fusion, PQA adds only +0.34% measured FLOPs and +3.9% latency under cached prompts.

• p4Jo Q4; u4HL Q1 — Scope and comparisons. We separate published external results from controlled within-framework comparisons and clarify that the paper tests few-exemplar encoding and candidate collaboration, not ambiguous/conflicting-prompt resolution.

Additional detection-level evidence (A.2). Both readers recover distinct correct objects; PQA retains 53.81–100% of their modality-exclusive GT union across four audited sources. This supports complementarity while exposing remaining retention limitations.

Summary. The new controls strengthen the attribution of BR and PQA, while the revised manuscript clarifies scoring, object correspondence, computational trade-offs, and evaluation scope. The core method remains unchanged, but the evidence for candidate-set collaboration is substantially stronger.

We sincerely thank the reviewers for their constructive comments and valuable suggestions, which have helped us strengthen the experimental validation, clarify our methodological contributions, and improve the overall quality of the manuscript. We also appreciate the Area Chair's time and consideration.

Best regards,
Authors

AuthorsOpenReview ↗

Update: Revised Manuscript and Rebuttal Summary

Dear Reviewers and Area Chair,

We would like to kindly highlight that we have updated our earlier comment, "Revised Manuscript — Main Contributions and Reviewer-Specific Rebuttal Summary."

The updated comment summarizes our three main contributions and explicitly maps the additional experiments and clarifications to each reviewer's specific concerns, with references to the corresponding sections of the revised manuscript.

We hope this consolidated summary will facilitate the final assessment.

We sincerely appreciate your constructive feedback, time, and consideration.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.