- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Progressive Visual Target Alignment (PVTA), a visual recognition setting in which an initially under-specified application intent is progressively aligned with an operational visual target through additional visual evidence and interaction. The paper introduces AST-50K, containing 52,577 images from 43 real-world scenarios and covering 117 application-specific targets, and establishes a training-free agentic framework based on an MLLM Planner and a Target Calibration Critic (TCC). The framework iteratively analyzes visual results and feedback, refines target descriptions, and orchestrates complementary visual tools. Experiments show that providing additional visual examples and iterative calibration improves detection performance, and the TCC ablations support the usefulness of its feedback components. However, the paper does not yet convincingly establish why target specification must be progressive and multi-step rather than being directly expressed as a sufficiently detailed natural-language description. Moreover, the current experimental design does not fully disentangle the benefit of progressive target alignment from iterative prompt refinement and test-time workflow selection. Therefore, while the problem is interesting and the benchmark is potentially useful, the central novelty and necessity of the proposed formulation require stronger validation.
Critical strengths
- Interesting task formulation. PVTA focuses on the gap between semantic object identity and application-specific operational targets.
- Clear application motivation. The paper considers scenarios where the same semantic category can correspond to different operational targets.
- Large-scale benchmark. AST-50K contains 52,577 images, 43 scenarios, and 117 application-specific targets.
- Clear agentic framework. The MLLM Planner, TCC, and visual modules form a coherent workflow.
- Training-free design. The framework does not require updating the parameters of the underlying visual models.
Critical weaknesses
- The necessity of progressive target alignment is not sufficiently established. The central assumption is that the initial application intent is under-specified, but the paper does not clearly show why the complete operational target cannot simply be expressed using a sufficiently detailed natural-language description.
- The key comparison with direct target specification is missing. The paper compares coarse target names, examples, captions, iterative calibration, and an oracle condition, but it does not directly compare against a strong baseline that receives the complete target description in one step.
- The gains may partly come from additional target information rather than progressive alignment. The experiments show that adding visual examples and captions already produces a substantial improvement. This makes it difficult to determine how much of the final gain comes specifically from the progressive nature of the interaction.
- PVTA may overlap with iterative prompt refinement. The framework repeatedly uses intermediate visual results and feedback to refine target descriptions. The paper needs to more clearly distinguish this mechanism from a generic MLLM-based iterative refinement procedure.
- The contribution of TCC is not fully isolated. The final improvement can potentially come from multiple factors, including additional visual evidence, iterative prompting, workflow selection, tool composition, and TCC itself.
- A simpler iterative baseline is missing. A baseline using the same visual tools, demonstrations, and iteration budget but replacing TCC with direct MLLM-based target refinement would provide a much cleaner evaluation of the proposed contribution.
- The distinction from existing language-based target specification needs to be stronger. The paper itself recognizes that open-vocabulary and language-based recognition systems can specify targets beyond fixed semantic categories. The key difference therefore needs to be the establishment of the target itself, rather than simply providing richer target descriptions.
- The practical necessity of multi-step interaction remains under-validated. The paper motivates progressive alignment conceptually, but it provides limited evidence showing concrete cases where a single natural-language description is fundamentally insufficient while progressive interaction succeeds.
Questions for authors
- Why does target specification need to be progressive and multi-step? Can the authors compare the proposed method against a strong baseline that receives the complete operational target as a single natural-language description?
- How much of the improvement comes from progressive interaction itself? Can the authors separate the benefit of adding visual examples/captions from the benefit of iteratively calibrating the target?
- Can the authors compare TCC with a generic iterative MLLM refinement baseline?
- How does the proposed setting differ from test-time workflow selection?
- Can the authors provide concrete examples where a single natural-language description is insufficient but progressive visual interaction is necessary?
10 comments
Response to Q2: Disentangling the Benefits of Progressive Interaction
A2: We thank the reviewer for raising this important issue. We agree that it is necessary to further distinguish the benefits brought by additional visual information from those brought by progressive interaction itself. The original experiments demonstrate that visual examples and captions can substantially improve detection performance, but this does not imply that all performance gains originate from Progressive Target Alignment.
To address this concern, we combine the results in Table 6 of the main manuscript with Exp. A of A1. By comparing different target-information input settings and iterative calibration processes, we further analyse the respective contributions of visual examples, target descriptions, and progressive interaction. This analysis helps more explicitly reveal the independent benefits of multi-turn interaction and progressive target calibration while distinguishing the effects of additional target information from those of the interaction process itself.
Beyond these controlled comparisons, the interaction records of AST-50K themselves indicate where the benefit of progressive alignment actually lies. As reported in Figure 7 of Appendix E.1, we record for each target the earliest round k* at which the AP-maximizing workflow is attained. Only 31 of the 117 targets (26.5%) reach their best workflow in the first round, whereas 86 targets (73.5%) require k* ≥ 2; coverage rises to 52.1% by round 3, 70.9% by round 5 and 88.9% by round 8, with the remaining targets extending to round 10. For the large majority of tasks, therefore, the workflow that best matches the application intent cannot be identified from the initial description alone, and is reached only after several rounds of visual feedback and calibration. We would emphasise that AST-50K is not constructed to manufacture difficulty: its targets are drawn from real application scenarios, and the spread of k* simply reflects how differently the visual boundaries of those real targets have to be pinned down. The distribution of the best round thus also supports the necessity and effectiveness of Progressive Target Alignment on realistic applications.
Response to W5, W6 & Q3: Comparison of TCC with Generic Iterative MLLM Refinement
A3: We thank the reviewer for pointing out that the contribution of TCC has not been sufficiently isolated. We agree that the final improvement may be influenced by several factors. Below we separate TCC from generic iterative MLLM refinement in two steps: the component ablation in the main manuscript (Table 6), and a new experiment in which generic refinement is run as an independent method (Exp. C).
Table 6 already isolates the mechanism. Each row removes one feedback type while keeping the remaining components unchanged. Two of these rows remove the channels that carry non-linguistic information: Failure Visualization, i.e. the FP/FN box-annotated image fed back to the agent, and Model-level Verification, i.e. probing of what each visual tool can actually do. Both removals cost accuracy, and removing Model-level Verification produces the largest drop in the table. With both channels gone, the feedback of TCC carries text only — the refined target description and the detailed language description — which is precisely what a generic iterative MLLM does, refining a description over turns using language alone (GPT-5.6-sol in our case). TCC degenerating into generic iterative MLLM feedback therefore costs a clear amount of accuracy, whereas removing a single textual channel changes the result only marginally.
Exp. C tests the same point as an independent method, on the subset used in A1. We take the textual output of the MLLM at the best round — the target description together with the detailed language description — and give it to the planner for single-pass orchestration, so that neither visual evidence nor tool-capability feedback reaches the planner. mAP and F1 drop substantially (Exp. C), while recall stays comparable (0.66 against 0.65) and precision roughly halves (0.52 to 0.33), i.e. the text-only variant over-detects.
Results of Exp. C
| Method | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| Ours | 0.4029 | 0.5369 | 0.5210 | 0.6474 |
| Exp. C | 0.2265 | 0.3128 | 0.3329 | 0.6625 |
Together, the ablation and Exp. C show that the gain of TCC over generic iterative MLLM refinement comes from the feedback channels that are not purely linguistic — the FP/FN visual evidence and the tool-capability probing — rather than from iterating on prompts. We will add this analysis to the revised manuscript.
Response to W1, W2, W3, W7 & Q1: Necessity of Progressive Target Alignment and Comparison with Direct Target Specification
A1: We agree that the necessity of Progressive Target Alignment should be clarified and validated through more direct experimental comparisons, and we thank the reviewers for pressing on this point.
Our key distinction is that PVTA focuses on progressively establishing an operational visual target for a specific application, rather than merely detecting an already specified target through language or visual interaction. In conventional language-based and referring visual tasks, the target is typically specified by a category or linguistic expression at inference time. Although such expressions may be complex and context-dependent, the detection target is generally assumed to be sufficiently determined. In contrast, PVTA starts from an application intent that may not uniquely determine the visual inclusion and exclusion criteria. Therefore, additional visual evidence, contextual information, and feedback are required to progressively align the application intent with an operational visual target.
New experiments. We acknowledge that the original manuscript does not establish whether a complete target description supplied directly can replace Progressive Target Alignment. We have therefore added two complementary controlled experiments (Exp. A and B). Both are run on a subset of AST-50K: we sample 10 tasks with at most 500 samples each, giving 3,922 samples (about 7% of the full benchmark). We use a subset rather than the full benchmark only for evaluation cost; this size keeps the comparison reliable while allowing us to report results within the rebuttal period.
Exp. A — direct target specification, no interaction. For each sampled task we take the four inputs that the planner receives at the best round of our own run, i.e. the four feedback channels of TCC at round k = k*: the box-annotated image f_vis^(k), the refined target description f_rtd^(k), the detailed language description f_fcd^(k), and the visual-module trial results f_ver^(k). These four inputs are given to the planner in a single pass, with no iterative interaction, and the planner produces one workflow whose boxes are taken as the prediction. This isolates the information that progressive interaction yields from the interaction process itself: the planner sees the same final feedback but never goes through the rounds. As Exp. A shows, our method retains a clear advantage, so the four feedbacks alone do not recover what progressive alignment provides.
Exp. B — complete linguistic specification, single-shot strong MLLM. We further test whether a sufficiently complete natural-language specification can replace progressive alignment. A strong multimodal MLLM, GPT-5.6-sol — the same model that serves as our planner — is given the full natural-language description accumulated over the ten rounds, namely the refined target descriptions and detailed language descriptions produced by TCC across rounds 1–K, so that it receives as complete an expression as possible and performs detection in one shot. Using the same model as our planner keeps this comparison about the interaction loop rather than about model capacity. Exp. B shows a clear drop: even a strong multimodal model equipped with a comprehensive description does not match progressive alignment.
Results of Exp. A
Results of Exp. B
These experiments validate the necessity of progressive target establishment from two directions: supplying the planner with the final aligned feedback but removing the interaction (Exp. A), and supplying a strong multimodal model with a complete one-shot description (Exp. B). In both cases, removing progressive alignment costs accuracy — precisely the regime PVTA is designed for. We will add these results and the corresponding discussion to the revised manuscript.