acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Object Detection: What If Visual Targets Diverge from Universal Taxonomies?

Abstract

Existing object detection methods typically assume that detection targets are specified in advance by semantic categories or language descriptions. Real-world applications, however, may require targets whose visual boundaries depend on application requirements and cannot be fully specified initially. We introduce Progressive Visual Target Alignment, a new detection task in which targets preserve their semantic identity while being defined according to application scenarios and progressively calibrated through interaction. We construct AST-50K, the first large-scale benchmark for this task, comprising 52,577 images from 43 application scenarios and 117 application-specific targets. We further establish a training-free agentic baseline that iteratively calibrates target and orchestrates complementary frozen visual tools through multimodal interaction. Experiments demonstrate that existing detection paradigms, including OVD, vision-language, and image-generation-based methods, suffer from inherent limitations on AST-50K, whereas our framework delivers substantial improvements.

Primary area:
datasets and benchmarks
Keywords:
visual alignment, visual agent, object detection, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Jq64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progressive Visual Target Alignment (PVTA), a visual recognition setting in which an initially under-specified application intent is progressively aligned with an operational visual target through additional visual evidence and interaction. The paper introduces AST-50K, containing 52,577 images from 43 real-world scenarios and covering 117 application-specific targets, and establishes a training-free agentic framework based on an MLLM Planner and a Target Calibration Critic (TCC). The framework iteratively analyzes visual results and feedback, refines target descriptions, and orchestrates complementary visual tools. Experiments show that providing additional visual examples and iterative calibration improves detection performance, and the TCC ablations support the usefulness of its feedback components. However, the paper does not yet convincingly establish why target specification must be progressive and multi-step rather than being directly expressed as a sufficiently detailed natural-language description. Moreover, the current experimental design does not fully disentangle the benefit of progressive target alignment from iterative prompt refinement and test-time workflow selection. Therefore, while the problem is interesting and the benchmark is potentially useful, the central novelty and necessity of the proposed formulation require stronger validation.

Critical strengths
  1. Interesting task formulation. PVTA focuses on the gap between semantic object identity and application-specific operational targets.
  2. Clear application motivation. The paper considers scenarios where the same semantic category can correspond to different operational targets.
  3. Large-scale benchmark. AST-50K contains 52,577 images, 43 scenarios, and 117 application-specific targets.
  4. Clear agentic framework. The MLLM Planner, TCC, and visual modules form a coherent workflow.
  5. Training-free design. The framework does not require updating the parameters of the underlying visual models.
Critical weaknesses
  1. The necessity of progressive target alignment is not sufficiently established. The central assumption is that the initial application intent is under-specified, but the paper does not clearly show why the complete operational target cannot simply be expressed using a sufficiently detailed natural-language description.
  2. The key comparison with direct target specification is missing. The paper compares coarse target names, examples, captions, iterative calibration, and an oracle condition, but it does not directly compare against a strong baseline that receives the complete target description in one step.
  3. The gains may partly come from additional target information rather than progressive alignment. The experiments show that adding visual examples and captions already produces a substantial improvement. This makes it difficult to determine how much of the final gain comes specifically from the progressive nature of the interaction.
  4. PVTA may overlap with iterative prompt refinement. The framework repeatedly uses intermediate visual results and feedback to refine target descriptions. The paper needs to more clearly distinguish this mechanism from a generic MLLM-based iterative refinement procedure.
  5. The contribution of TCC is not fully isolated. The final improvement can potentially come from multiple factors, including additional visual evidence, iterative prompting, workflow selection, tool composition, and TCC itself.
  6. A simpler iterative baseline is missing. A baseline using the same visual tools, demonstrations, and iteration budget but replacing TCC with direct MLLM-based target refinement would provide a much cleaner evaluation of the proposed contribution.
  7. The distinction from existing language-based target specification needs to be stronger. The paper itself recognizes that open-vocabulary and language-based recognition systems can specify targets beyond fixed semantic categories. The key difference therefore needs to be the establishment of the target itself, rather than simply providing richer target descriptions.
  8. The practical necessity of multi-step interaction remains under-validated. The paper motivates progressive alignment conceptually, but it provides limited evidence showing concrete cases where a single natural-language description is fundamentally insufficient while progressive interaction succeeds.
Questions for authors
  1. Why does target specification need to be progressive and multi-step? Can the authors compare the proposed method against a strong baseline that receives the complete operational target as a single natural-language description?
  2. How much of the improvement comes from progressive interaction itself? Can the authors separate the benefit of adding visual examples/captions from the benefit of iteratively calibrating the target?
  3. Can the authors compare TCC with a generic iterative MLLM refinement baseline?
  4. How does the proposed setting differ from test-time workflow selection?
  5. Can the authors provide concrete examples where a single natural-language description is insufficient but progressive visual interaction is necessary?
10 comments
AuthorsOpenReview ↗

Response to W1, W2, W3, W7 & Q1: Necessity of Progressive Target Alignment and Comparison with Direct Target Specification

A1: We agree that the necessity of Progressive Target Alignment should be clarified and validated through more direct experimental comparisons, and we thank the reviewers for pressing on this point.

Our key distinction is that PVTA focuses on progressively establishing an operational visual target for a specific application, rather than merely detecting an already specified target through language or visual interaction. In conventional language-based and referring visual tasks, the target is typically specified by a category or linguistic expression at inference time. Although such expressions may be complex and context-dependent, the detection target is generally assumed to be sufficiently determined. In contrast, PVTA starts from an application intent that may not uniquely determine the visual inclusion and exclusion criteria. Therefore, additional visual evidence, contextual information, and feedback are required to progressively align the application intent with an operational visual target.

New experiments. We acknowledge that the original manuscript does not establish whether a complete target description supplied directly can replace Progressive Target Alignment. We have therefore added two complementary controlled experiments (Exp. A and B). Both are run on a subset of AST-50K: we sample 10 tasks with at most 500 samples each, giving 3,922 samples (about 7% of the full benchmark). We use a subset rather than the full benchmark only for evaluation cost; this size keeps the comparison reliable while allowing us to report results within the rebuttal period.

Exp. A — direct target specification, no interaction. For each sampled task we take the four inputs that the planner receives at the best round of our own run, i.e. the four feedback channels of TCC at round k = k*: the box-annotated image f_vis^(k), the refined target description f_rtd^(k), the detailed language description f_fcd^(k), and the visual-module trial results f_ver^(k). These four inputs are given to the planner in a single pass, with no iterative interaction, and the planner produces one workflow whose boxes are taken as the prediction. This isolates the information that progressive interaction yields from the interaction process itself: the planner sees the same final feedback but never goes through the rounds. As Exp. A shows, our method retains a clear advantage, so the four feedbacks alone do not recover what progressive alignment provides.

Exp. B — complete linguistic specification, single-shot strong MLLM. We further test whether a sufficiently complete natural-language specification can replace progressive alignment. A strong multimodal MLLM, GPT-5.6-sol — the same model that serves as our planner — is given the full natural-language description accumulated over the ten rounds, namely the refined target descriptions and detailed language descriptions produced by TCC across rounds 1–K, so that it receives as complete an expression as possible and performs detection in one shot. Using the same model as our planner keeps this comparison about the interaction loop rather than about model capacity. Exp. B shows a clear drop: even a strong multimodal model equipped with a comprehensive description does not match progressive alignment.

Results of Exp. A

Method[email protected][email protected][email protected][email protected]
Ours0.40290.53690.52100.6474
Exp. A0.35860.48330.48120.5916

Results of Exp. B

Method[email protected][email protected][email protected][email protected]
Ours0.40290.53690.52100.6474
Exp. B0.37340.48910.45040.6278

These experiments validate the necessity of progressive target establishment from two directions: supplying the planner with the final aligned feedback but removing the interaction (Exp. A), and supplying a strong multimodal model with a complete one-shot description (Exp. B). In both cases, removing progressive alignment costs accuracy — precisely the regime PVTA is designed for. We will add these results and the corresponding discussion to the revised manuscript.

AuthorsOpenReview ↗

Response to Q2: Disentangling the Benefits of Progressive Interaction

A2: We thank the reviewer for raising this important issue. We agree that it is necessary to further distinguish the benefits brought by additional visual information from those brought by progressive interaction itself. The original experiments demonstrate that visual examples and captions can substantially improve detection performance, but this does not imply that all performance gains originate from Progressive Target Alignment.

To address this concern, we combine the results in Table 6 of the main manuscript with Exp. A of A1. By comparing different target-information input settings and iterative calibration processes, we further analyse the respective contributions of visual examples, target descriptions, and progressive interaction. This analysis helps more explicitly reveal the independent benefits of multi-turn interaction and progressive target calibration while distinguishing the effects of additional target information from those of the interaction process itself.

Beyond these controlled comparisons, the interaction records of AST-50K themselves indicate where the benefit of progressive alignment actually lies. As reported in Figure 7 of Appendix E.1, we record for each target the earliest round k* at which the AP-maximizing workflow is attained. Only 31 of the 117 targets (26.5%) reach their best workflow in the first round, whereas 86 targets (73.5%) require k* ≥ 2; coverage rises to 52.1% by round 3, 70.9% by round 5 and 88.9% by round 8, with the remaining targets extending to round 10. For the large majority of tasks, therefore, the workflow that best matches the application intent cannot be identified from the initial description alone, and is reached only after several rounds of visual feedback and calibration. We would emphasise that AST-50K is not constructed to manufacture difficulty: its targets are drawn from real application scenarios, and the spread of k* simply reflects how differently the visual boundaries of those real targets have to be pinned down. The distribution of the best round thus also supports the necessity and effectiveness of Progressive Target Alignment on realistic applications.

AuthorsOpenReview ↗

Response to W5, W6 & Q3: Comparison of TCC with Generic Iterative MLLM Refinement

A3: We thank the reviewer for pointing out that the contribution of TCC has not been sufficiently isolated. We agree that the final improvement may be influenced by several factors. Below we separate TCC from generic iterative MLLM refinement in two steps: the component ablation in the main manuscript (Table 6), and a new experiment in which generic refinement is run as an independent method (Exp. C).

Table 6 already isolates the mechanism. Each row removes one feedback type while keeping the remaining components unchanged. Two of these rows remove the channels that carry non-linguistic information: Failure Visualization, i.e. the FP/FN box-annotated image fed back to the agent, and Model-level Verification, i.e. probing of what each visual tool can actually do. Both removals cost accuracy, and removing Model-level Verification produces the largest drop in the table. With both channels gone, the feedback of TCC carries text only — the refined target description and the detailed language description — which is precisely what a generic iterative MLLM does, refining a description over turns using language alone (GPT-5.6-sol in our case). TCC degenerating into generic iterative MLLM feedback therefore costs a clear amount of accuracy, whereas removing a single textual channel changes the result only marginally.

Exp. C tests the same point as an independent method, on the subset used in A1. We take the textual output of the MLLM at the best round — the target description together with the detailed language description — and give it to the planner for single-pass orchestration, so that neither visual evidence nor tool-capability feedback reaches the planner. mAP and F1 drop substantially (Exp. C), while recall stays comparable (0.66 against 0.65) and precision roughly halves (0.52 to 0.33), i.e. the text-only variant over-detects.

Results of Exp. C

Method[email protected][email protected][email protected][email protected]
Ours0.40290.53690.52100.6474
Exp. C0.22650.31280.33290.6625

Together, the ablation and Exp. C show that the gain of TCC over generic iterative MLLM refinement comes from the feedback channels that are not purely linguistic — the FP/FN visual evidence and the tool-capability probing — rather than from iterating on prompts. We will add this analysis to the revised manuscript.

Reviewer_dyjEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a new visual recognition task, PVTA, addressing the challenge that target visual boundaries in real-world applications are difficult to fully specify in advance. The authors construct AST-50K, the first large-scale benchmark dataset, and propose a training-free agentic baseline that progressively calibrates visual targets through iterative interaction and tool orchestration, significantly outperforming existing paradigms in experiments.

Critical strengths

S.1 The proposed task, PVTA, reflects the demand in real-world scenarios for more fine-grained and precise object detection.

S.2 The constructed dataset, AST-50K, contributes a new benchmark to the community.

S.3 The proposed training-free baseline provides a reference for future research.

Critical weaknesses

W.1. Poor Readability. The paper has notable weaknesses in how the dataset and method are presented. It is difficult for readers to understand the key details of the proposed dataset and method from the main text. See W.2, W.3, and W.4.

W.2. Unclear Key Dataset Details. As a new dataset proposed for a new task, its data format should be described in detail in the main text to help readers better understand the dataset and task definition. In particular, readers should be able to easily understand what information the annotations contain and in what format this information is stored (see Q1 and Q3). This omission further hinders understanding of the proposed method—for example, where dd and PplanP_{\text{plan}} in Eq. (5) come from, whether they are directly converted from annotations, and if so, how the conversion is performed.

W.3. Insufficient Explanation of Notation. Many symbols are not adequately introduced when they first appear, which impedes understanding. For example, (a) dd in Eq. (6) is not explained in the text following the equation (it appears to correspond to the “initial description” in L299–300). (b) frtdk−1f_{\text{rtd}}^{k-1} in Eq. (5) and fviskf_{\text{vis}}^k, ffcdkf_{\text{fcd}}^k, and fverkf_{\text{ver}}^k mentioned in Line 311 are not briefly introduced immediately after their first appearance. Although these symbols are described later in the paper, their roles (e.g., semantic feedback, linguistic feedback), forms (e.g., textual descriptions or other formats), and the origins of the abbreviations (e.g., what “rtd” and “fcd” stand for) should be summarized at first mention.

W.4. Unclear Method Description. See Q2 and Q4. The paper does not clearly explain, for example, how the modules planned by the Planner are executed and interact, or what the format of the prediction results is.

Questions for authors

Q1: Figure 1 illustrates how the detection scope is progressively narrowed through multiple rounds of application requirements and human interpretation. However, the example in Figure 6 only provides a scene description and a set of fine-grained categories for the corresponding scene. (a) Is the multi-round dialogue in Figure 1 part of how samples are annotated in the dataset, or is it merely used to help readers understand PVTA? (b) Does Figure 6 show the annotation format, i.e., a text description of a scene plus several fine-grained categories?

Q2: The Planner mentioned in Section 4.1 plans the order of the candidate modules (Seed2Pro, SAM3, RexOmni, Fuser) (i.e., Πk\Pi^k in Lines 288-289). (a) Are these models run sequentially according to Πk\Pi^k ? If so, how does the output of the previous model affect the output of the next model? Is the previous model’s bounding boxes drawn on the image and fed as input to the next model? (b) If the Planner only plans the order, why does Seed2Pro appear twice in Round-final in Figure 5?

Q3: What exactly does tjt_j in Eq. (3) (the corresponding application-specific visual target) represent? Typically, a target is represented by its coordinates (bjb_j) and category (cjc_j). What additional information does tjt_j contain here?

Q4: What is the format of the prediction result BkB^k in Eq.(4)? Is it the same as in Eq. (3)?

5 comments
AuthorsOpenReview ↗

Response to W2 & W3, Q1 & Q3: Dataset Format and Task Definition, Explanation of Notation

W2 & Q1 & Q3: Unclear Key Dataset Details

A1: We thank the reviewer for pointing out that the descriptions of the dataset format, task definition, and their relationship to the relevant equations are not sufficiently clear. We agree that, as a dataset constructed for a new task, AST-50K should present its annotation contents and data organization more explicitly in the main text.

First, regarding the relationship between Fig. 1 and Fig. 6, we clarify that the multi-turn dialogue in Fig. 1 is primarily intended to help readers intuitively understand the concept of PVTA and the process of progressively narrowing the detection scope. It does not directly correspond to the actual annotation format of each sample in AST-50K. In contrast, Fig. 6 illustrates the annotation format of AST-50K, which consists of a scene text description and multiple application-specific detection categories associated with that scene. In the revised manuscript, we will further clarify the distinct purposes of these two figures and move Fig. 6 into the main text to make the dataset format more readily accessible to readers. This will also help prevent readers from mistakenly interpreting the interactive process illustrated in the task overview as part of the dataset annotations.

Second, regarding tt in Eq. (3), we further clarify that tt denotes the application-specific visual target to be recognized in the final detection scenario, rather than merely a bounding box or a conventional semantic category. Specifically, cc denotes the corresponding detection scenario or detection superclass, tt represents the application-specific detection target further defined within that scenario, and bb denotes the bounding box predicted by the model. Since different application requirements may exist under the same detection superclass cc, they can correspond to different application-specific targets tt. We will further clarify the relationships and semantic meanings of these variables in the revised manuscript to help readers better understand PVTA's target definition and its distinction from conventional category-based detection.

In summary, we will supplement the revised manuscript with a detailed description of the dataset format, clarify the distinct roles of Fig. 1 and Fig. 6, and provide a unified explanation of the key variables in Eq. (3); the variables in Eq. (5), including dd, are clarified in our separate response to W3 below.

W3: Insufficient Explanation of Notation

A2: We thank the reviewer for pointing out this issue and sincerely apologize for the oversight. Due to the page limitations of the original manuscript, we inadvertently omitted some essential explanations of the equations when condensing the main text. Consequently, several symbols lacked sufficient definitions and explanations upon their first appearance, making it more difficult for readers to understand the task formalization and methodological workflow.

We agree that mathematical notation should not only be formally defined but also be accompanied by clear explanations of its meaning, role, and corresponding information representation upon its first introduction. Therefore, in the revised manuscript, we will systematically review and revise all equations and related notation throughout the paper, ensuring that every symbol is explicitly and concisely defined at its first occurrence and that all abbreviations are expanded and appropriately explained when first introduced.

Regarding the sources of the relevant variables in Eq. (5), we thank the reviewer for this careful observation. We agree that the four feedback terms fvis(k)f_{vis}^{(k)}, frtd(k)f_{rtd}^{(k)}, ffcd(k)f_{fcd}^{(k)} and fver(k)f_{ver}^{(k)}, together with the feedback set f(k)f^{(k)} they constitute, should be briefly introduced immediately after their first appearance in Eq. (5). We further clarify the symbol dd in Eq. (5): dd denotes the target description from the user's initial input, i.e., the description before any TCC feedback has been applied. It therefore acts as the initial condition of the recursion, and after the first round of feedback its role is taken over by frtd(1)f_{rtd}^{(1)} (and by frtd(k)f_{rtd}^{(k)} thereafter); dd itself does not reappear in later rounds. We will incorporate all of the above changes into the revised manuscript.

AuthorsOpenReview ↗

Response to W4, Q2 & Q4: Workflow Orchestration and Output Consistency

W4 & Q2 & Q4

A3: We thank the reviewer for raising this question. We would like to clarify that the Planner does not simply determine a fixed execution order for candidate visual tools. Instead, it dynamically orchestrates the entire workflow according to the current application intent, target calibration requirements, and the functional characteristics of different visual tools.

Specifically, visual tools such as Seed2Pro, SAM3, RexOmni, and Fuser do not follow a fixed execution order, nor are their invocation counts predetermined. The Planner can flexibly determine the subsequent invocation order and whether a particular tool should be called repeatedly based on the current state. This design allows the workflow to go beyond sequentially executing all candidate modules and instead adapt dynamically to the requirements of each specific task.

For example, in certain application scenarios, Seed2Pro may generate complementary visual detection results using different prompts, while SAM3 may not provide additional useful target information in the current context. Consequently, the Planner may choose to invoke Seed2Pro first, followed by RexOmni, and then invoke Seed2Pro again based on the intermediate results to further improve the detection results or verify target boundaries. This also explains why Seed2Pro appears twice in the Round-final stage of Fig. 5.

We will further clarify the Planner's dynamic workflow orchestration mechanism in the revised manuscript. In addition, we provide statistics on visual tool invocations in the workflow, as summarized in Table 1, to illustrate the actual invocation patterns of different modules.

Table 1. Per-module invocation statistics in our benchmark — occurrence of each module across the 10-round interaction workflow orchestration of 117 targets.

ModuleTotal invocationsAvg. invocations per workflowWorkflows containing the moduleWorkflows with repeated invocations (>= 2)
Seed2Pro1,1911.0181,118 (95.6%)71 (6.1%)
SAM33530.302353 (30.2%)0
RexOmni3840.328384 (32.8%)0
Fuser5730.490573 (49.0%)0

Regarding the prediction output format in Eq. (4), we thank the reviewer for this helpful comment. The final prediction represented by Eq. (4) is consistent with the detection output defined in Eq. (3), with both corresponding to the final detection categories, targets, and their predicted bounding boxes. The output representations of the two equations in the original manuscript were not sufficiently consistent, which may have caused ambiguity. We will unify the output formulations of Eqs. (3) and (4) in the revised manuscript and provide the necessary explanations to ensure consistency between them.

Thank you for the detailed responses. I appreciate the authors' efforts to clarify the manuscript.

I acknowledge the motivation behind this work. Demonstrating that the proposed method can progressively refine target definitions according to application-specific requirements, thereby enabling more accurate target localization, would better align with the central objective of PVTA. However, the current manuscript does not clearly demonstrate this capability or adequately establish how the proposed benchmark evaluates it.

Therefore, I will maintain my original rating.

Reviewer_TnBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce Progressive Visual Target Alignment (PVTA), a visual detection task in which targets are defined by application-specific requirements rather than fixed and predefined semantic taxonomies and are progressively refined through interaction. To support this task, the authors further construct a benchmark, AST-50K. Then the paper proposes a training-free agentic framework that combines an MLLM-based planner with a target calibration critic (TCC) to iteratively refine target descriptions and orchestrate visual detection tools. Experiments on AST-50K demonstrate improvements over existing open-vocabulary detectors and VLMs.

Critical strengths
  1. The paper constructs an application-oriented benchmark, AST-50K. It provides a relatively large-scale benchmark with 52,577 images, 43 application scenarios, and 117 visual targets. Its emphasis on application-specific detection requirements offers a useful evaluation setting beyond conventional category-based detection.

  2. The authors propose a framework that combines iterative feedback with heterogeneous visual tools and achieves substantial improvements over individual detection models without additional training. The experimental results show the practical potential of a feedback-driven workflow for specialized detection tasks.

Critical weaknesses
  1. The distinction between PVTA and existing language-based (described object detection and omnilabel), referring object detection/segmentation, and interactive visual grounding tasks is unclear. These tasks already support intent-driven natural language that go beyond fixed semantic taxonomies. They allow objects to be selected based on attributes, contextual constraints, and user intents. Prior studies also explored multi-turn visual grounding and iterative refinement through interaction (e.g., RegionReasoner [1]). The examples shown in Fig. 6, such as distinguishing digital versus static billboards, appear to be straightforward fine-grained and attribute-based detection problems.

  2. Although PVTA is proposed to progressively resolve initially under-specified intents, AST-50K consists of predefined application-specific targets with fixed annotations. As a result, the experiments primarily demonstrate iterative detector adaptation rather than the progressive discovery or double-clarification of an initially ambiguous target. The paper does not directly evaluate whether the model can identify a user's intended target through successive interactions, nor does it establish that such interactions are necessary beyond providing additional supervision. Additionally, the comparison in Tab. 3 raises fairness concerns. The baselines only include pure semantic object detection models or general-purpose large vision models and do not include the aforementioned methods in weakness 1.

[1]. https://arxiv.org/pdf/2602.03733

Questions for authors
  1. What is the fundamental difference between the proposed PVTA and other language-based and referring visual tasks?
  2. How's the performance of those methods on AST-50K, and vise versa?
6 comments
AuthorsOpenReview ↗

Response to W1 & Q1 - Part 1

A1: We thank the reviewer for the careful and detailed feedback. As the reviewer points out, we also consider clarifying the distinction between PVTA and other visual tasks to be a crucial question, and we are glad to discuss it in detail. The core difference lies in what the system is trying to find. PVTA searches for the joint optimum of orchestration, tools and external priors, and, for cost reasons, we restrict it to a ten-round exploration space. Even though the PVTA task we propose poses a substantial challenge to the language, visual and reasoning ability of any single model, our focus is not on how to make a model follow instructions and perform visual reasoning in multi-turn interaction.

A single model has an inherent disadvantage on the PVTA task, for two reasons. (1) A fully complete linguistic description of a real application intent does not exist. From one's own understanding one can say as clearly as possible what one thinks one is looking at, but doing this by hand for a real application is expensive and unrealistic. (2) The knowledge of existing models comes from pretraining or from tuning for a particular scenario, and the cognitive bias introduced by either source prevents a model from directly producing a sufficiently complete and ideal description for a real application target. Having identified this problem, we provide what we currently consider a reasonable baseline given its characteristics.

Our multi-round progressive interaction provides the planner with a space it can explore. The TCC we propose probes model capability by independently testing FP/FN with visual tools, and jointly provides visualised FP/FN/TP boxes, refined target descriptions and linguistic feedback, so that the planner can, during the interaction stage, try different orchestrations of the available visual tool modules across multiple rounds and thereby find a better detection workflow, which is finally frozen and used for direct inference and detection.

In fact, we differ from existing OVD, language-based and interactive visual methods in the problem to be solved, the task definition, the method design, the role of the rounds and the output (Table A compares the paradigms in detail), and the detection metric itself is the quantity that best reflects the progressive establishment of alignment in this process; we therefore sampled ten tasks, counted their detection accuracy by round, and report the results in Table B, which is given in our next response (Response to W1 & Q1 - Part 2).

Table A: Comparison of Different Visual Task Paradigms

Comparison DimensionFine-grained DetectionLanguage-based and referring groundingInteractive / multi-turn visual taskPVTA
Primary objectiveDistinguish fine-grained semantic categoriesLocalize targets specified by languageLocalize different language-specified targets through multi-turn interactionProgressively establish and localize application-specific targets through multi-turn interaction
Target alignmentNot a primary concernGenerally assumed to be completed before detectionTargets to be detected in each round are specified before detectionProgressively establish and align targets while maintaining a consistent application intent
Cross-round target relationshipNoneNoneIndependent and different targets across roundsTargets are continuously associated across rounds
Target specificationFine-grained category labelsVocabulary, phrases and linguistic descriptionsLinguistic descriptionsStructured feedback and linguistic descriptions
Testing objective / criterionNo target switching involvedNo target switching involvedSwitch targets after multi-turn interactionIdentify and establish a specific target after multi-turn interaction
AuthorsOpenReview ↗

Response to W1 & Q1 - Part 2

Table B: Average performance of the ten sampled tasks over the ten orchestration rounds

Round[email protected]F1PR
10.0810.150.3590.116
20000
30.510.640.7620.573
40.3740.4070.4410.386
50.3950.4720.5820.443
60.4960.5460.5950.541
70.3020.3690.410.338
80.480.5720.6650.522
90.4430.5060.5390.483
100.4520.5490.6380.4

The results in Table B do not mean that round 3 is optimal for every task: the round at which the best workflow actually appears differs across application targets (Figure 7 in Appendix E.1 reports that statistics). Our multi-round exploration is unguided, which is markedly different from RegionReasoner, where explicit textual guidance is given. In the first round the planner orchestrates as many visual tool modules as possible according to its existing knowledge, typically from pretraining or additional tuning (for example SeedPro + SAM3 + RexOmni + Fuser); after the first multimodal FP/FN feedback from TCC, however, the planner tends to make a drastic change to SeedPro plus the FP/FN feedback — we counted the complete round-2 orchestration for all 117 targets and found that 78 of them perform this subtraction in the second round, keeping only SeedPro, the model the planner considers strongest; only in later rounds does it progressively find a better orchestration. In this multi-round progressive process the planner is therefore also progressively probing its joint understanding of the optional tool modules and of TCC's four feedback channels, thereby escaping the planner model's "cognitive boundary" over particular components and feedback. This is also one of the key differences between our multi-round setting and other single-round, multi-round and interactive visual tasks.

AuthorsOpenReview ↗

Response to W2 & Q2 - Part 1

A2: We thank the reviewer for raising these points. It is worth further emphasizing that the construction of application-specific targets in AST-50K was not a one-time process, but rather involved multiple rounds of human discussion, annotation feedback, and revision of decision boundaries. Table 9 in Appendix B.3 presents multi-round communications between researchers and annotators during the quality-control process. Our annotation experience indicates that even for tasks with explicit application purposes, researchers cannot fully and precisely define the visual boundaries of all target instances solely based on the application intent at the initial stage. Instead, target definitions often require repeated discussion and revision based on concrete visual instances, boundary ambiguities, and annotation feedback encountered during the annotation process. This process demonstrates the practical need for progressively aligning application intents with operational visual targets and provides real-world motivation for the progressive target establishment and multi-turn interaction design in PVTA.

Before turning to RegionReasoner, we would like to clarify the purpose of Table 3 and the way the comparison there is set up. Table 3 of the main manuscript already compares representative models of several paradigms on AST-50K: language-based detection with strong MLLMs from the Seed, Qwen and GPT families, open-vocabulary detection, referring grounding such as SAM3 and RexOmni, where the two together already exceed GroundingDINO, a widely adopted reference a few years ago, and generative detection such as Nano Banana Pro. These baselines are therefore strong rather than trivially beatable; yet none of them performs well on AST-50K without additional training. This is not because the benchmark is built to defeat existing models — AST-50K is drawn from real application scenarios — but because benchmarks such as RefCOCO+/g and OmniLabel do not cover the PVTA problem, in which an application intent first has to be turned into an operational target. Agreeing that a direct comparison with a multi-turn interactive method is desirable, we took the time to analyse and reproduce RegionReasoner.

RegionReasoner on AST-50K. We compare on the 20-images-per-scene subset used in the controlled comparison of Table 3 (lower part): 20 images for each of the 117 targets, i.e. 2,340 images; the results are in Table C. The comparison is restricted to this subset because RegionReasoner infers at a markedly lower speed and generating boxes for it alone takes close to a full day, so a full-benchmark comparison is not feasible within the rebuttal period. Both methods see the same images, and metrics are macro-averaged over the 117 targets. We instantiated RegionReasoner as a single workflow per target, with the same exemplar images and application description as ours but without TCC feedback, since its formulation is per-turn instruction following.

Table C: Comparison of Ours and RegionReasoner on AST-50K subset

IoUMethodmAPF1PrecRecall
0.15Ours0.58410.63240.61260.7794
0.15RegionReasoner0.27910.35180.30890.6250
0.50Ours0.50430.56920.55040.7051
0.50RegionReasoner0.20320.28190.24620.5051

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.