acceptodds
Withdrawn from ICLR 2027

Hierarchical Reasoning-Guided Progressive Aerial Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Complex aerial visual grounding requires not only understanding what a referring expression describes, but also coordinating how its different semantic components guide visual localization. This challenge is amplified in UAV imagery by small objects, dense same-category instances, and complex spatial layouts. We propose Hierarchical Reasoning-Guided Aerial Visual Grounding (HRVG), a unified framework that organizes semantic guidance throughout a progressive grounding process. The key idea is to align different semantic roles with corresponding visual computations, rather than treating semantic reasoning as an auxiliary language representation. HRVG derives hierarchical scene-, entity-, and relation-level semantic states from language reasoning and progressively integrates them into visual grounding, enabling contextual understanding, candidate grounding, and relational decision-making within a single connected framework. We further introduce RefScene-20K, a benchmark comprising 20,000 referring expressions over 8,550 UAV images across relational, ordinal, extremal, multi-target, and no-target tracks, with outputs uniformly represented as potentially empty sets of bounding boxes. Experiments on RefScene-20K demonstrate that HRVG outperforms existing grounding baselines. Further analyses, including semantic decomposition, reasoning-order, and stage-intervention studies, verify the effectiveness of organizing hierarchical semantics and aligning them with progressive visual computations. These results highlight semantic-role alignment as an effective modeling principle for complex aerial visual grounding.

Primary area:
datasets and benchmarks
Keywords:
Visual grounding, Aerial visual grounding, UAV banchmark, Progressive Visual Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_wxY8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies complex aerial visual grounding, where an expression may refer to zero, one or multiple objects. The proposed HRVG uses a frozen Qwen3-8B to decompose each expression into scene-, entity- and relation-level fields. These fields guide three successive stages built on RemoteSAM and Grounding DINO: scene-conditioned encoder attention, entity-based candidate selection, and pairwise relation reasoning that scores each candidate and outputs a thresholded box set. The paper also introduces RefScene-20K, with 20,000 expressions over 8,550 VisDrone images covering relational, ordinal, extremal, multi-target and no-target cases. The expressions are generated by Qwen3-VL-32B and checked by geometric rules, a second VLM and human reviewers. HRVG reaches 59.87 / 60.95 Avg on RefScene-20K (Swin-T / Swin-B), about 5 points above the strongest baseline, and reports smaller gains on AerialVG and RefDrone.

Critical strengths

The paper targets an important difficulty in aerial grounding: compositional expressions involving scene context, relations, ordering, and variable target cardinality. The proposed benchmark RefScene-20K covers relational, ordinal, extremal, multi-target and no-target expressions in a unified setting. The results reveal that the ordinal track remains hard for all evaluated models.

Critical weaknesses
  1. The experiments do not yet support the paper's central claim that aligning semantic roles with visual stages drives the improvement. Ablation results are duplicated across tables (Table 3 and Table 5), and two variants described as removing different components report identical numbers on every metric. The variant with the largest drop (w/o PI) is described only conceptually, and it is unclear what it removes. The stage-reordering experiment is hard to interpret, because the three modules act on different parts of the architecture. Taken together, it is hard to verify that the reported ablations correspond to distinct trained models.
  2. The benchmark and the method are closely matched, and the comparison is not controlled. RefScene-20K expressions are generated from scene restrictions, attribute groups and relation or ordinal templates, which are the same fields the parser extracts. HRVG also uses a frozen 8B language model and a frozen RemoteSAM backbone that no baseline has access to, and no MLLM baseline is evaluated. On RefDrone, the gains over NGDINO are small. It is therefore hard to separate the benefit of hierarchical reasoning from the benefit of extra models and of a parse schema that matches how the data was generated.
Questions for authors
  1. Please clarify the benchmark taxonomy. Fig. 1 shows six expression types, but the statistics and Table 1 use only five tracks. Do the relational, ordinal and extremal tracks together make up the single-target expressions?
  2. Do As\mathcal{A}_s, Ae\mathcal{A}_e and Ar\mathcal{A}_r correspond one-to-one to SGA, EGR and RGR? Is HeH_e simply the candidate set XeX_e in Eq. (7), or does it also pass dense visual features to the relation stage?
  3. What are the exact forms and loss weights of Lscene\mathcal{L}_{\mathrm{scene}}, Lgraph\mathcal{L}_{\mathrm{graph}} and Lenergy\mathcal{L}_{\mathrm{energy}}?
  4. Tables 3 and 5 report exactly the same rows, and "w/o EGR" in Table 2 matches "w/o Entity Reasoning" in Table 3 on all six metrics. Yet the first removes a visual module and the second removes a text field. Are these distinct trained models? What exactly is removed in "w/o PI", given that Section 4.3 does not specify a separate inheritance mechanism?
  5. In Table 4, SGA acts on encoder attention, EGR on token alignment and RGR on candidate pairs, so their execution order is fixed by the architecture. What does an ordering such as Entity → Relation → Scene mean in practice? Are the text fields simply routed to different modules?
  6. RefScene-20K was generated with Qwen3-VL-32B, but no MLLM baseline is evaluated. How do fine-tuned MLLMs such as Qwen2.5-VL-7B or Qwen3-VL-8B perform on it? How does a Grounding DINO baseline perform if it receives the same parsed fields and applies rule-based ordinal or extremal selection over its boxes?
  7. Do the supervision targets for scene responses, directed relations, ordering and soft counts come from the RefScene-20K construction metadata, and if so, how is HRVG trained on AerialVG and RefDrone, where such metadata is not available?
Reviewer_rrk4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies complex referring-expression grounding in UAV imagery and makes two contributions. (1) HRVG, a Grounding-DINO-based framework. A frozen Qwen3-8B parser first produces a chain-of-thought and a structured JSON of scene, entity and constraint fields. These fields then drive three successive visual stages. Scene-guided alignment injects a RemoteSAM-based heatmap prior into the deformable encoder attention. Entity-guided candidate grounding builds a shared candidate pool. Relation-guided constraint reasoning scores candidates with a directed pairwise support field, soft counts and an energy threshold, and outputs a possibly empty box set. (2) RefScene-20K, a benchmark of 20,000 expressions on 8,550 VisDrone2019-DET images. It has relational, ordinal, extremal, multi-target and no-target tracks. Expressions are generated by Qwen3-VL-32B from rule-derived target sets, then checked by a geometric verifier, InternVL3.5-38B and human reviewers. HRVG reaches 59.87/60.95 Avg. (Swin-T/B) on RefScene-20K, compared with 54.84 for the strongest baseline. It also shows gains on AerialVG and RefDrone. Ablations cover components, semantic decomposition, stage order and stage-specific flat guidance.

Critical strengths
  1. RefScene-20K is a useful benchmark. Ordinal and extremal references get their own tracks instead of being folded into generic "spatial" descriptions. Every query is cast as set prediction, so zero, one and many targets are handled uniformly. The construction pipeline is documented in detail (Appendix B): rule-derived target sets come before text generation, there are explicit geometric conventions with tie-breaking rules, a dual automatic verifier, and two-reviewer human checking with reported agreement (κ = 0.92). This level of auditability is better than in many LLM-generated grounding datasets.
  2. The idea of assigning distinct semantic roles to distinct visual stages is clearly motivated, and the paper tests it in several ways. Table 3 replaces structured semantics with flat expression guidance. Table 4 changes the stage order. Table 6 swaps in flat guidance one stage at a time. The relation stage gives a large gain on ordinal queries (21.96 → 30.17), and these queries are known to be hard for detector-style grounding models.
Critical weaknesses
  1. Missing controls for the source of improvement. HRVG uses resources the Table 1 baselines do not have: a frozen 8B LLM parser, an extra RemoteSAM backbone, and extra construction-time supervision (reference objects, ordering, counts) through L_graph and L_scene. The parser schema also closely mirrors the rule taxonomy used to generate RefScene-20K. Without a baseline that gets the same parsed fields, and without current MLLM grounders, it is hard to separate the contribution of the proposed semantic-to-visual alignment from these extra resources. The much smaller margins on RefDrone (0.06–1.07 points, no variance reported), which was not built with the authors' pipeline, make this question more pressing. I would be glad to raise my score if these controls support the claims.
  2. Clarity of the ablation evidence. Several ablation results are hard to interpret as independent evidence. Table 5 repeats the rows of Table 3. Table 2 "w/o EGR" is identical on all metrics to "w/o Entity Reasoning", which may be expected if removing R_e disables EGR, but the paper should say this explicitly. There are also smaller inconsistencies: 57.42 vs. 67.42 Top-1 between the Appendix A text and Table 7, examples in Sec. 5.2 that do not appear in Fig. 3, incorrect bolding in Table 1, and an empty Sec. D.1. These do not invalidate the results, but they weaken the ablations, which are the main support for the central claim.
Questions for authors

Clarifications that could change my assessment:

  1. Table 2 "w/o EGR" and Table 3 "w/o Entity Reasoning" report identical numbers. Please clarify whether these are the same experiment, and why Table 5 duplicates Table 3.
  2. How were the Table 1 baselines adapted to set prediction and to no-target queries? In particular, how do TransVG-style single-box models and Grounding DINO produce empty sets, and is their threshold tuned on validation as τ_E is for HRVG?
  3. Please report results for (a) Grounding DINO plus the same Qwen3-8B parsed fields and a simple rule-based selector, and (b) at least one strong MLLM grounder on RefScene-20K. Please also report HRVG without RemoteSAM.
  4. How often does the parser produce invalid output or hit the fallback path, and how often are its fields wrong? How does performance change with oracle (ground-truth) fields versus noisy fields?
  5. For Table 4, the stages are fixed modules. What exactly is reordered: the modules themselves, or which semantic field each module receives? If the latter, this tests mismatched inputs rather than reasoning order.
  6. How is L_graph supervised on AerialVG and RefDrone, which lack the reference, ordering and count annotations used for RefScene-20K?
  7. Please give mean ± std over several seeds for Tables 1, 7 and 8, given the sub-point margins on RefDrone.
  8. Avg. averages [email protected], set-level F1 and empty-set accuracy. Please also report how often HRVG wrongly predicts an empty set on positive queries, or give a trade-off curve over τ_E.

Suggestion (not required for my assessment): A human or MLLM study on the naturalness of the generated expressions, compared with human-written ones.

Reviewer_tBGxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HRVG, a hierarchical reasoning-guided framework for aerial visual grounding. It decomposes referring expressions into scene-, entity-, and relation-level semantics, which progressively guide visual encoding, candidate grounding, and relational selection. The authors also introduce RefScene-20K, a benchmark covering relational, ordinal, extremal, multi-target, and no-target expressions.

Critical strengths

(1) The paper addresses challenging aerial grounding scenarios involving dense objects and complex spatial relationships.

(2) The authors provide component ablations and evaluations on two additional UAV grounding benchmarks, offering some evidence of the framework's effectiveness.

Critical weaknesses

(1) The proposed framework mainly combines existing techniques, including LLM-based semantic decomposition, scene-guided attention, entity-level candidate selection, and graph-based relational reasoning. Similar ideas have been explored in prior work, such as MAttNet[1] and GrabVG[2]. Although HRVG integrates these components into a progressive framework, the methodological contribution beyond their combination is not sufficiently clear.

[1] Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., & Berg, T. L. MAttNet: Modular Attention Network for Referring Expression Comprehension. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1307–1315, 2018. [2] Wang, C., Di, Y., Sun, J., Liu, B., Tian, J., Li, Y., Guo, G., & Gao, S. GrabVG: Graph-Attentive Binding for Visual Grounding in UAV Imagery. arXiv preprint arXiv:2608.18996, 2026.

(2) HRVG incorporates several additional components, including Qwen3-8B, RemoteSAM, and relation-specific supervision. However, the ablations do not clearly isolate the benefits of hierarchical reasoning. Comparisons with simpler alternatives, such as direct structured parsing without chain-of-thought or Grounding DINO with the same semantic information and geometric rules, are missing.

(3) The main improvements are achieved on RefScene-20K, which relies on predefined geometric rules and LLM-generated expressions. This raises concerns about potential dataset-specific biases. Although the authors also evaluate on AerialVG and RefDrone, the gains are relatively small, particularly on RefDrone.

Questions for authors

Please refer to the weaknesses above.

Reviewer_6KaYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses referring-expression grounding in UAV images, where an expression can combine scene context, object attributes, and relational, ordinal, or extremal constraints, and can refer to zero, one, or several objects. The proposed HRVG uses a frozen Qwen3-8B to parse each expression into scene, entity, and constraint fields, which condition three stages built on Grounding DINO: a scene heatmap from RemoteSAM features that adds a gated residual to the encoder's deformable-attention logits, entity-based selection of a shared candidate pool, and a directed candidate graph whose energies are thresholded to give a possibly empty set of boxes. The authors also introduce RefScene-20K, built on VisDrone2019-DET, with 20,000 expressions over 8,550 images split into relational, ordinal, extremal, multi-target, and no-target tracks. Targets are fixed by geometric rules before an LLM writes the expressions, which are then checked by humans. HRVG reaches 59.87/60.95 Avg. (Swin-T/B) on RefScene-20K against 54.84 for the best baseline, and the paper also reports ablations and supervised results on AerialVG and RefDrone.

Critical strengths
  1. The benchmark is a useful addition. Ordinal and extremal references get their own tracks rather than being folded into general spatial relations, and all queries use a possibly empty set of boxes as output. Targets are fixed by explicit geometric rules before any expression is written (App. B.4), each record keeps the evidence behind its target set, and agreement between the two annotators is high (96.2% on target sets, κ = 0.92 on track labels; App. B.5). No baseline exceeds 26.37 on the ordinal track (Table 1), which points to a real weakness of current detection-based grounding models.

  2. Several design choices are sensible and clearly documented. The zero-initialized, bounded scene residual leaves the pretrained attention unchanged at the start of training (Eq. 11), and the antisymmetric branch in Eq. 15 is a natural bias for directional relations. The LLM parser is specified more fully than is common (prompt, decoding settings, validation and fallback, caching), and the efficiency analysis reports the uncached end-to-end latency as well as the cached one.

Critical weaknesses
  1. The ablations do not reliably support the central claim. This is my main concern. Table 5 repeats the first four rows of Table 3, and "w/o EGR" (Table 2) and "w/o Entity Reasoning" (Table 3) report identical numbers although they remove different components. Progressive Inheritance has the largest effect (59.87 → 53.29) but is not defined as a separate component, and without it HRVG-T is at the level of plain Grounding DINO-T (53.08). The numbers are hard to reconcile with Table 10: with 750 test expressions per single-target track and 300 no-target expressions, values such as 40.82 and 93.04 are not attainable. No variance is reported, although many of the differences are only 1–2 points. Until these issues are corrected and explained, the ablations do not show that semantic-role alignment is an effective modeling principle.

  2. It is unclear how much of the gain comes from the method itself. Besides target boxes, HRVG is trained with relation, ordering, and count supervision, uses reference objects as auxiliary targets (Sec. 4.4), and adds a frozen RemoteSAM backbone and an 8B parser. The paper does not say how the baselines were trained on RefScene-20K, and some of their numbers look low. AerialVG (Swin-L) is below Grounding DINO-T on the relational and ordinal tracks, and NGDINO is below Grounding DINO on the multi-target and no-target tracks, even though it beats Grounding DINO on every RefDrone metric (Table 8). Results with target-only supervision, and with baselines trained under the same schedule and threshold selection, are needed to show how much of the gain comes from the proposed design.

  3. Novelty over existing designs is modest. The stages are close to modular subject/location/relation decomposition (MAttNet), Grounding DINO with a relation module (AerialVG), and Grounding DINO with counting (NGDINO), and the entity stage still runs Grounding DINO on the full expression and takes a max over entity-token logits (Eq. 13). The "hierarchical reasoning" is a single text-only parse: the chain of thought is kept only for auditing, and only the extracted fields reach the visual model (App. C.1). The paper reports neither parser accuracy nor how often the fallback is used, although the Limitations section notes that parsing errors can propagate. The introduction also describes existing aerial methods as matching a single sentence embedding (lines 37–40), which does not fit Grounding DINO-based models such as NGDINO.

  4. Results outside RefScene-20K are mixed. The parser's output schema (App. C.1) closely follows the structured records used to generate RefScene-20K (App. B.3–B.4), which may favor HRVG on this benchmark. HRVG improves clearly on AerialVG, but on RefDrone the gain over NGDINO is only 0.06–1.07 points, and the training setup for neither external benchmark is described. Some details needed for reproduction are also missing: f_energy and the exact forms and weights of L_scene, L_graph, and L_energy are not given, although the Reproducibility Statement says the appendix includes the weights, and it is unclear what supervises the scene heatmap, since the paper describes no scene-region annotations. There are also smaller inconsistencies: Section 5.2 describes examples that do not appear in Figure 3, and App. A.1 reports 57.42 where Table 7 reports 67.42.

Questions for authors
  1. Main concern, What exactly does Progressive Inheritance refer to, and how was "w/o PI" implemented? Are the repeated rows in Tables 3 and 5 and the identical "w/o EGR" / "w/o Entity Reasoning" results reporting errors? If so, please provide the correct numbers. How many expressions were evaluated per track, and are the reported numbers averaged over several runs? Mean and standard deviation over a few seeds, or a paired significance test, for Tables 1–3 would make the comparisons much easier to judge.

  2. Attribution, Did the baselines get the same auxiliary supervision, number of epochs, and threshold selection as HRVG? How does HRVG perform when trained with target-set supervision only? It would also help to see a simple baseline that applies the benchmark's geometric rules (App. B.4) to the parser output and the Grounding DINO candidates, at least on the ordinal and extremal tracks.

  3. RefScene-20K and RefDrone (Sun et al., 2025) both draw on VisDrone2019-DET images, and RefScene-20K uses its own split. How many images do the two benchmarks share, and were any weights transferred between them? Separately, the text-only parser can output a "no-target" constraint type. How often does it do so on the true no-target expressions? A text-only baseline on the no-target and multi-target tracks would help rule out language priors. How accurate is the parser on RefScene-20K, how often is the fallback triggered, and what is used to supervise the scene heatmap (L_scene)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.