acceptodds
Under review as a conference paper at ICLR 2027

Parallel Instance Decoding for Language-Guided Localization in Aerial Images

Abstract

Earth observation (EO) localization is challenging due to large scene sizes, complex backgrounds, large variations in object scale and varying numbers of target objects. Existing remote-sensing vision-language models (RSVLMs) typically perform localization tasks, such as grounding, detection, and pointing by autoregressively generating bounding-box coordinates. As the object count increases, this step-by-step decoding becomes slow because boxes are produced one at a time and each prediction relies on earlier ones. To address this limitation, we introduce Parallel Instance Decoding (PID), a unified generative localization framework that separates instance identification from spatial localization. Given a textual reference (open vocabulary query), PID first generates a variable-length set of point addresses, with one address representing each matching object instance. It then predicts the bounding boxes for all identified instances in a single parallel decoding pass. To enable independent box prediction, we use an instance-isolated attention mask that allows each box to attend to the full image and its corresponding address while preventing attention to other boxes. Under the same training data and experimental setup, PID improves [email protected] over the direct-box baseline by 8.3 points on DIOR and 7.3 points on the DOTAv2 dataset. It also reduces complete-response latency by 22.4% and 43.6%, respectively. Moreover, PID performs favorably well against general-purpose VLMs and specialized RSVLMs on grounding and pointing tasks. We will publicly release our code and models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
remote sensing, vision-language models, visual grounding, object detection, pointing, parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v4N2OpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Parallel Instance Decoding (PID), a two-stage generative localization framework for aerial imagery. PID first autoregressively predicts a variable-length set of point addresses and then predicts all address-conditioned boxes in one packed forward pass using instance-isolated attention. It is evaluated on category-conditioned detection, referring-expression grounding, and pointing, with controlled comparisons against a finetuned direct-box baseline and explicit latency/throughput analysis.

Critical strengths
  1. The controlled comparison is unusually careful: the direct-box baseline uses the same training data, curriculum, trainable components, and checkpoint-selection procedure, and the paper additionally provides class-isolated controls.
  2. The paper evaluates both accuracy and complete-response efficiency, and the density-stratified analysis is aligned with the claimed motivation that parallel decoding should matter most in dense aerial scenes.
Critical weaknesses
  1. The instance-isolated attention mask is the core mechanism that enables independent parallel box decoding, yet it is never directly ablated. Without a packed-decoding variant that allows cross-instance attention, the paper does not show that masking itself is responsible for the accuracy gain.
  2. The architectural contribution is confounded with additional point-center supervision. PID receives explicit address targets while the direct-box baseline does not; the paper itself acknowledges that the effect of point supervision cannot be fully separated from instance addressing. A direct-box baseline with the same auxiliary point objective is necessary.
  3. The efficiency claim is not fully clean. The final detection numbers use a rotary-stabilized implementation, whereas the reported timing was measured before that numerical change. Moreover, specialized detectors remain dramatically faster in absolute throughput, so the claimed efficiency advantage is primarily within autoregressive VLM-style localizers rather than localization methods in general.
  4. The 'open-vocabulary' framing is not validated by the main detection protocol. Queries only contain categories known to be annotated in each image, so the evaluation does not test absent-category rejection or unseen-category generalization. The current evidence supports category-conditioned localization, not unrestricted open-vocabulary detection.
  5. There is an unresolved train/evaluation contamination risk. The training pool contains DIOR/DOTA-family sources, while the authors only remove exact image overlaps and explicitly state that geographic independence between nearby aerial tiles is not guaranteed. A near-duplicate/geospatial leakage analysis is needed before interpreting the gains as clean generalization.
Questions for authors
  1. What happens if box segments are allowed to attend to other boxes during the packed pass? Please provide a direct ablation of the instance-isolated attention mask.
  2. Can the direct-box baseline be trained with the same auxiliary point-center supervision to isolate the effect of PID's architecture from the extra supervision signal?
  3. Can the authors report latency/throughput using the exact rotary-stabilized implementation used for the final accuracy results?
  4. Can the authors provide quantitative absent-category and unseen-category evaluation, and a nearest-neighbor/geographic-overlap analysis between training and evaluation imagery?
4 comments

We thank the reviewer for their constructive comments. We have uploaded a revised manuscript; all changes are highlighted in blue. All results below use the same rotary-stabilized implementation as the final results in the paper.

W1, Q1: Ablation of the instance-isolated mask

In the parallel box pass, all boxes are predicted together in one forward pass, so no box sees the decoded coordinates of another box. Without the mask, however, the tokens of each box could still attend to the segments of the other boxes. The instance-isolated mask blocks this, so each box depends only on the image prefix and its own address (Sec. 3.2). We test the mask in two ways. (a) We train PID with cross-box attention: each box segment may also attend to the reference–address prompts of all other box segments. Attention between the geometry positions of different boxes cannot be trained this way, because during training these positions hold the target coordinates and would leak the answers; variant (b) covers this case. Everything else in (a), including the point targets and the loss, is the same as for PID. (b) We take the final PID model, trained with the mask, and open the mask only at test time, so that each box segment attends to all tokens of the earlier segments or of all other segments.

We report (a) in a cumulative ablation that adds the parts of PID one at a time to the class-isolated direct-box model LA-3B (FT, class-iso.)§ (Table 5A). The second row is the direct-box model with the same point supervision as PID, which answers W2:

Model ([email protected])Point lossBoxes from addressesIsolated maskDIORDOTAv2
LA-3B (FT, class-iso.)§ (Table 5A)NoNo–75.4561.66
 + point lossYesNo–77.7860.71
 + boxes from addresses, (a) cross-box attentionYesYesNo79.3263.45
 + instance-isolated mask (PID, Table 1)YesYesYes82.2466.90
(b) Stress test: PID trained with the mask, mask opened only at test time
 Each box also attends to all earlier boxes–––52.8426.66
 Each box also attends to all other boxes–––37.0914.51

The main evidence is (a): even when the model is trained with cross-box attention, the instance-isolated mask adds +2.9 points on DIOR and +3.5 points on DOTAv2 on top of addressing. Attending to other boxes therefore does not help box prediction, and it hurts more in the denser DOTAv2 scenes. Variant (b) is a stress test rather than a clean ablation: the model was never trained with cross-box attention, and all box segments share the same position indices (App. B.2). Opening the mask at test time reduces [email protected] by up to 45.2 points on DIOR and 52.4 points on DOTAv2, which shows that the trained model relies on the mask. The mask also makes each box independent of the number and order of the other objects. This is why all boxes can be predicted in one pass with the same accuracy as one at a time ([email protected] differs by only 0.06 on DIOR and 0.00 on DOTAv2). This is added in Sec. 4.4 (Table 6) and App. D.5 (Table 20).

W2, Q2: Direct-box baseline with the same point supervision

We trained this baseline (second row of the table above). It is the class-isolated direct-box model, which is stronger than LA-3B (FT)† and makes the same per-category requests as PID. We add the point targets of PID as separate point requests in the same training episodes, next to the per-category box requests and with the same image prefix. There is no extra head, and no point is written before a box. The two losses have equal weights, as in Eq. 4, and data, curriculum and checkpoint selection are the same as for PID. Its supervision is therefore the same as PID; the remaining difference is that PID predicts each box from its own address.

The point loss helps on DIOR (+2.3) but not on DOTAv2 (−1.0), where the point-supervised model returns about 1.7× as many boxes with the same number of correct boxes, i.e., more false positives. PID remains 4.5 / 6.2 points above this control: predicting boxes from addresses adds 1.5 / 2.7 points, and the instance-isolated mask a further 2.9 / 3.5. Most of the gain therefore comes from predicting each box from its own address with the isolated mask, not from the extra supervision. This is added in Sec. 4.4 (Table 6) and App. D.5.

W3, Q3: Efficiency with the rotary-stabilized implementation

The Table 4 timings were recorded with the rotary-stabilized implementation, in the same evaluation run that gives the Table 1 accuracy. The run log shows the FP32 rotary cache installed in all 36 rotary modules before any prefill, and the same outputs give both tables: the ratio of correct to valid boxes per second in Table 4 (5.50 / 6.99 on DIOR and 8.33 / 13.30 on DOTAv2) matches, up to rounding, the micro-precision of the Table 1 outputs (2,473 / 3,146 and 16,158 / 25,785). The statements in Sec. 4.3, App. B.3 and the caption of Table 18 (Table 15 in the submission) were incorrect and are corrected in the revision. For convenience, we repeat the comparison with the direct-box baseline, which uses the same backbone, data and training:

DIOR [email protected]DIOR s/imageDIOR Correct boxes/sDOTAv2 [email protected]DOTAv2 s/imageDOTAv2 Correct boxes/s
LA-3B (FT)†73.940.9902.9059.623.9303.27
PID (ours)82.240.7685.5066.902.2188.33
Speed-up1.29×1.77×

PID is both faster and more accurate. It reduces latency by 22.4% on DIOR and 43.6% on DOTAv2 and gives 1.90× and 2.55× more correct boxes per second. The advantage grows with scene density, as expected from Eq. 3. With the same addresses, predicting all boxes in one pass is 3.0× (DIOR) and 5.9× (DOTAv2) faster than predicting them one at a time, with the same accuracy (App. D.3, Table 19B).

We agree that specialized detectors are faster in absolute time (4–12×, Table 18), and the revision limits our efficiency claim to generative VLM-based localizers. Even among these, PID is not the fastest everywhere: on DOTAv2, the pretrained LA-3B has slightly lower latency (1.992 vs. 2.218 s, Table 18), but PID produces 2.8× more correct boxes per second (8.33 vs. 3.00). The statement in Sec. 4.3 that PID has the lowest latency on both datasets is replaced by the statement that it has the highest throughput among generative VLMs. However, the specialized detectors are not evaluated here on referring-expression grounding or pointing, which PID handles with the same model. In accuracy, PID is far above Grounding DINO (82.24 vs. 27.48 on DIOR and 66.90 vs. 17.17 on DOTAv2) and above LAE-DINO on DIOR at stricter thresholds ([email protected]: 69.83 vs. 58.98; mF1: 65.91 vs. 54.75). LAE-DINO, which is pretrained on the large LAE-1M dataset, remains stronger on DOTAv2 (Table 1).

W4, Q4 (a): Absent-category and unseen-category evaluation

We agree that the main protocol measures category-conditioned localization. It follows the annotated-category detection setting of Rex-Omni (App. C.1), and the revision states this scope and replaces "open vocabulary query" with "class name or phrase" (Abstract and Sec. 1). To test absent categories, we asked for all 20 DIOR and 18 DOTAv2 categories in every image, so every prediction for a category that is not in the image is a false positive:

Model (all categories requested, same run)[email protected] DIOR[email protected] DOTAv2Absent-category predictions DIORAbsent-category predictions DOTAv2
LA-3B (FT)†69.8355.124,31327,235
LA-3B (FT, class-iso.)§73.2359.961532,318
PID (ours)66.5656.542,61515,424

The class-isolated model is the most accurate in this setting and rejects absent categories best, and on DIOR PID is also below LA-3B (FT)†. PID makes 39% (DIOR) and 43% (DOTAv2) fewer absent-category predictions than LA-3B (FT)† and is more accurate on DOTAv2. These errors start in the address step: absent categories still receive addresses, and the box step does not reject all of them. Since predicting the boxes in one pass or one at a time gives the same result, they are not caused by parallel box prediction.

We additionally test robustness to new wordings of the seen categories. Every category is requested with one fixed new wording; the wordings were generated with an LLM assistant, checked with a case-insensitive substring search over all training records, fixed before any model was evaluated, and are identical for all models (full list in App. D.6, Table 23). The evaluation labels are unchanged.

ModelDIOR OriginalDIOR New wordingDIOR RetainedDOTAv2 OriginalDOTAv2 New wordingDOTAv2 Retained
LA-3B (FT)†73.9461.6683%59.6240.6268%
LA-3B (FT, class-iso.)§75.4550.6067%61.6630.4649%
PID (ours)82.2471.2187%66.9048.5173%

All values are [email protected]. Examples: storage tank → "cylindrical oil silo", helicopter → "rotorcraft", chimney → "industrial smokestack". "Original" is from Tables 1 and 5A.

PID is the most accurate model with the new wordings and keeps the largest part of its accuracy. This tests robustness to new wordings, not generalization to unseen categories. Both analyses are added in App. D.6 (Tables 21–23), with a pointer in Sec. 4.1.

W5, Q4 (b): Near-duplicate and geographic-overlap analysis

The overlap analysis leads to three main findings. First, on the 332 DIOR test images that share no ground area with any training image, PID keeps its full gain (+6.7 over the class-isolated model and +8.1 over LA-3B (FT)†). Second, most of the DOTAv2 overlap comes from the LAE DOTAv2 benchmark split itself: its training and test tiles are cut from the same source images, and 523 of the 567 source images of the test tiles also have tiles in its training file. The DOTAv2 subset of LAE-1M is sampled from this training file and contains tiles of 517 of these 567 source images, so the overlap is a property of the benchmark split and also affects models pretrained on LAE-1M, such as LAE-DINO. Third, the gain of PID remains large when scenes of similar density are compared: on the DOTAv2 images with less than 25% overlap, PID's gain over LA-3B (FT)† is +10.5 in both the 6–20 and the >20 object groups, comparable to its gain in the same groups on the overlapping images (+13.8 and +11.6). The details are given below.

For every DIOR and DOTAv2 test image, we searched all 106,318 training images using CLIP ViT-L/14 similarity (full image and five crops) and perceptual hashing. We then checked each candidate with SIFT + RANSAC homography and pixel-level correlation. This check accepts 0 of 4,000 random image pairs from the same dataset. Since DIOR and DOTA have no geographic coordinates, we use a verified homography as a proxy for showing the same ground area.

Test images that share ground area with a training imageDIORDOTAv2
Any overlap254 / 586792 / 874
Overlap covering ≥ 25% of the test image247 / 586603 / 874
Overlap covering ≥ 50% of the test image18 / 586393 / 874

All compared models are trained on the same data, so overlap affects them all, although not necessarily by the same amount. To check this, we removed the overlapping test images and evaluated again:

[email protected]ImagesLA-3B (FT)†LA-3B (FT, class-iso.)§PIDPID gain (vs. class-iso. / vs. LA-3B (FT)†)
DIOR, all images58673.9475.4582.24+6.79 / +8.30
DIOR, without any overlap33272.3173.7380.39+6.66 / +8.08
DOTAv2, all images87459.6261.6666.90+5.24 / +7.28
DOTAv2, without ≥ 50% overlap48161.1461.8566.17+4.32 / +5.03
DOTAv2, without ≥ 25% overlap27163.5163.4865.89+2.41 / +2.38

On DOTAv2, PID stays the best model after removing images with ≥50% or ≥25% overlap, although the gap becomes smaller and is no longer significant at the ≥25% threshold. The low-overlap DOTAv2 images are also much sparser (median 5 objects, compared with 9 over all DOTAv2 images). In the sparsest group (1–5 objects) of these images, LA-3B (FT)† is ahead by 5.0 points, so together with the third finding above, the smaller overall gain is mainly explained by the larger share of sparse images. Only 82 DOTAv2 images have no overlap at all. On this small subset, LA-3B (FT)† scores 70.29, PID 68.35 and the class-isolated model 67.01, but the difference to PID is not significant. The overlap-filtered DOTAv2 and overlap-free DIOR numbers are reported next to the main results (Sec. 4.2), the full analysis is in App. A.3 (Table 10), and we will release the overlap lists.

In summary, each of the five weaknesses is now addressed in the revised manuscript, either with a new experiment (mask ablation, point-supervised baseline, timing provenance, absent categories and new wordings, overlap analysis) or with a narrowed claim (efficiency among generative VLM localizers, category-conditioned localization). We hope this resolves the reviewer's concerns and would be grateful if the reviewer would reconsider the score.

Reviewer_NKUpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on remote-sensing vision-language models (RSVLMs). Existing generative RSVLM localization methods typically produce bounding boxes autoregressively, one instance at a time, limiting inference speed in multi-object scenes and potentially allowing errors in earlier predictions to affect subsequent ones. The proposed method decouples instance enumeration from bounding-box prediction. It first generates point addresses for matching instances, autoregressively within each textual reference and concurrently across different references, and then predicts all bounding boxes in parallel conditioned on these addresses. This design reduces sequential decoding overhead and eliminates inter-box generation dependencies through instance-isolated attention. Experiments show that the overall approach improves both inference speed and localization accuracy.

Critical strengths

The paper presents a coherent approach with clearly defined contributions. It decouples the RSVLM detection pipeline into point prediction and bounding-box prediction, improving both localization accuracy and inference speed.

Critical weaknesses
  1. The contributions of the individual design choices are not sufficiently disentangled. As the authors acknowledge, the effects of instance addressing and box-center point supervision cannot be fully separated in the current experiments. In addition, the efficiency evaluation reports only overall latency, without a breakdown of the time spent on address prediction and bounding-box prediction. Such an analysis would help clarify the sources of both the accuracy and efficiency gains.
  2. The evaluation provides limited evidence of open-vocabulary generalization. Although the method is formulated as accepting open-vocabulary queries, the main detection experiments request categories annotated in each image and do not explicitly evaluate generalization to categories unseen during training. An evaluation with a clearly defined seen/unseen category split would better substantiate this aspect of the method.
Questions for authors

Please refer to the Weaknesses section above.

5 comments

We thank the reviewer for the positive assessment and the helpful suggestions. We have uploaded a revised manuscript; all changes are highlighted in blue.

W1 (a): Instance addressing vs. point supervision

We trained the missing control and report it in a cumulative ablation. We start from the class-isolated direct-box model LA-3B (FT, class-iso.)§ (Table 5A) rather than LA-3B (FT)†, because it is the stronger direct-box baseline and makes the same per-category requests as PID. We then add the parts of PID one at a time. All new models use the same data, curriculum and checkpoint selection as PID.

  • Point loss. We add the point targets of PID as separate point requests in the same training episodes, next to the per-category box requests and with the same image prefix. There is no extra head, and no point is written before a box. The two losses have equal weights, as in Eq. 4, and only the box requests are used at inference. This model therefore receives the same supervision as PID, but its boxes are not predicted from the addresses.
  • Boxes from addresses. PID trained with cross-box attention, so that each box can also attend to the addresses of the other boxes. It uses the same point targets and loss as PID.
Model ([email protected])Point lossBoxes from addressesIsolated maskDIORDOTAv2
LA-3B (FT, class-iso.)§ (Table 5A)NoNo–75.4561.66
 + point lossYesNo–77.7860.71
 + boxes from addresses (cross-box attention)YesYesNo79.3263.45
 + instance-isolated mask (PID, Table 1)YesYesYes82.2466.90

The point loss helps on DIOR (+2.3) but not on DOTAv2 (−1.0). On DOTAv2, the point-supervised model returns about 1.7× as many boxes as the plain model, with the same number of correct boxes, so the additional boxes are false positives. Predicting boxes from addresses adds 1.5 / 2.7 points, and the instance-isolated mask a further 2.9 / 3.5 points. In total, PID is 4.5 / 6.2 points above the point-supervised control, so most of the gain comes from predicting each box from its own address with the isolated mask, not from the extra point supervision. This is added in Sec. 4.4 (Table 6) and App. D.5, and replaces the earlier statement there that the two effects cannot be separated.

W1 (b): Latency breakdown

The table below shows the time per image for each step, measured in the same run as Table 4:

Seconds per imageDIORDOTAv2
Image encoding + shared prefix0.4030.920
Address prediction0.3211.189
Box prediction (all boxes in one pass)0.0430.108
Total (same as Table 4)0.7682.218

Box prediction takes only 6% (DIOR) and 5% (DOTAv2) of the total time, even though it produces all boxes. To measure the effect of parallel box prediction directly, we used the same weights and the same addresses and predicted the boxes either in one pass or one address at a time, timing both settings together on the same images:

Box prediction, same addresses (s/image)DIORDOTAv2
One address at a time0.2830.888
All addresses in one pass (PID)0.0940.151
Speed-up (95% CI)3.0× [2.7, 3.3]5.9× [5.6, 6.1]
[email protected] difference0.060.00

This controlled run was made on a different machine from Table 4, which was timed on a shared server (App. D.2). All steps therefore run at different speeds, which is why one-pass box prediction takes 0.094 s here instead of 0.043 s on DIOR. Timings on the shared server are stable: an earlier timing run of the same checkpoint on the same server, made before the rotary change, gave 0.776 s per image and 0.044 s for the box step on DIOR, within 2% of Table 4. The rotary change therefore does not noticeably affect speed. Absolute times are comparable only within a run, so the speed-up compares the two settings within the controlled run.

Parallel box prediction keeps the same accuracy and is 3.0× to 5.9× faster, with a larger speed-up in denser scenes. In short, the speed gain of PID comes from replacing the long sequence of boxes with short address lists that are decoded concurrently, and from predicting all boxes in one pass; the accuracy gain comes from predicting each box from its own address (W1 (a)). The breakdown is summarized in Sec. 4.3, and both tables are added to App. D.3 (Table 19).

W2: Open-vocabulary generalization

We agree and have narrowed the claim. We used "open-vocabulary" in the same sense as prior remote-sensing work such as LAE-DINO (Pan et al., 2025): categories are given as free text instead of a fixed label set, and the test categories are covered by the training data. In the revised manuscript, we now replace "open vocabulary query" with "class name or phrase" for clarity (Abstract and Sec. 1).

We additionally test robustness to new wordings of the seen categories. Every category is requested with one fixed new wording, and the evaluation labels are unchanged. The wordings were generated with an LLM assistant, one per category, and checked with a case-insensitive substring search over all Stage 1 and Stage 2 training records, including all prompts, expressions and answers; the three wordings found there were replaced. The list was fixed before any model was evaluated and then remains the same for all models. The full list and the check are given in App. D.6 (Table 23).

ModelDIOR OriginalDIOR New wordingDIOR RetainedDOTAv2 OriginalDOTAv2 New wordingDOTAv2 Retained
LA-3B (FT)†73.9461.6683%59.6240.6268%
LA-3B (FT, class-iso.)§75.4550.6067%61.6630.4649%
PID (ours)82.2471.2187%66.9048.5173%

All values are [email protected]. Examples: storage tank → "cylindrical oil silo", helicopter → "rotorcraft", chimney → "industrial smokestack". "Original" is from Tables 1 and 5A.

PID is the most accurate model with the new wordings and keeps the largest part of its accuracy. The class-isolated model, which also asks for one category at a time, drops the most, so this robustness does not come from asking for one category at a time. This tests robustness to new wordings, not generalization to unseen categories. The analysis is added in App. D.6 (Tables 22 and 23), with a pointer in Sec. 4.1.

In summary, the revised manuscript adds the cumulative component ablation, the latency breakdown and the new-wording analysis, and narrows the open-vocabulary claim. We hope these changes address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

Dear Reviewer NKUp,

We sincerely appreciate the opportunity to address your comments. We understand you may have been busy, but we would be grateful if you could consider re-engaging with the discussion. We believe that some of the concerns raised could benefit from further clarification, and we would greatly welcome any additional feedback you might have.

Thank you for your time and efforts in supporting the review process.

Authors

Reviewer_TH55OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Parallel Instance Decoding (PID) method for language-guided localization in aerial images, which addresses the latency issue in dense object detection. Their finding is that using AR to predict object center locations and then regressing bounding-box locations in parallel reduces latency and is effective.

Critical strengths
  1. The paper is fluently written, and the figures are clear and easy to follow.
  2. The paper's main contribution is to decouple two processes: first, autoregressively predicting the center locations, and then predicting the bounding boxes in parallel, which is reasonable.
  3. The results compared with previous methods are clear and competitive, and the latency is good.
Critical weaknesses
  1. The main issue I am concerned about is that the paper lacks a fully AR baseline. I mean fully autoregressive prediction using the same backbone as the paper. I assume this would be an upper bound, as it could explain how much accuracy is sacrificed by parallel prediction. Although the results in the paper look good, we know there is no free lunch. How can the authors defend the parallel decoding approach in terms of performance rather than speed?
  2. This paper uses two-stage inference. I am afraid that if the center points are predicted incorrectly, the subsequent bounding-box predictions may also be affected. I would appreciate a clearer analysis of how center-point prediction errors affect the final localization performance, especially in dense scenes.
  3. Some new techniques, such as block diffusion models, can already handle parallel decoding. Can the authors discuss whether this approach might become unnecessary if these techniques become mature and widely adopted?
Questions for authors

Please refer to the weakness above.

3 comments

We thank the reviewer for their positive assessment and the useful questions. We have uploaded a revised manuscript; all changes are highlighted in blue.

W1: Fully autoregressive baseline

The closest baseline in the paper is LA-3B (FT)†. It uses the same backbone, training data, curriculum, trainable components and checkpoint selection as PID, and it predicts all boxes autoregressively in one sequence (Sec. 4.1). However, each box is decoded in one PBD step, so it is not autoregressive within a box, and the reviewer is right that this is not a fully autoregressive baseline. We therefore added a fully token-by-token variant, in which every coordinate is generated sequentially. It uses the same LA-3B (FT)† weights without retraining. LA-3B (FT)† is fine-tuned with next-token cross-entropy on every answer token, and the native PBD blocks are added only as extra supervised copies (App. A.2), so token-by-token decoding is within its training distribution. Table 5A also includes a class-isolated variant, which decodes each category in a separate sequence. PID performs better than all of them at [email protected], [email protected] and mF1 on both datasets:

Model (same backbone, data and training)DIOR [email protected]DIOR [email protected]DIOR mF1DOTAv2 [email protected]DOTAv2 [email protected]DOTAv2 mF1
LA-3B (FT)†, token-by-token decoding (new)70.3759.5355.9455.5532.6432.36
LA-3B (FT)†, one PBD step per box (Table 1)73.9463.7559.9459.6236.3335.28
LA-3B (FT, class-iso.)§ (Table 5A)75.4563.6960.0761.6638.5737.04
PID (ours)82.2469.8365.9166.9044.1542.68

Therefore, the fully autoregressive model is not an upper bound in our setting. In one autoregressive sequence, each box depends on all earlier boxes. An early mistake, such as a duplicate or a missed object, affects the boxes that follow, and this becomes worse in dense scenes: in images with more than 20 objects (App. D.1), PID reaches 74.95 (DIOR) and 67.51 (DOTAv2) [email protected], compared with 59.22 and 56.19 for LA-3B (FT)†. In PID, each box depends only on the image, the query and its own address, so errors do not propagate between boxes. The address prediction itself remains autoregressive, and most of the remaining errors come from this step: 74.6% (DIOR) and 83.4% (DOTAv2) of the missed objects never receive an address, while an object that receives an address usually gets a correct box.

Parallel decoding also does not reduce accuracy. With the same weights and the same addresses, predicting all boxes in one pass instead of one box at a time changes [email protected] by only 0.06 on DIOR and 0.00 on DOTAv2. The token-by-token row is added to Table 5A (Sec. 4.4, App. D.4), and the one-pass comparison to App. D.3 (Table 19B).

W2: Effect of center-point errors, especially in dense scenes

Thank you for this interesting suggestion. To carry out this analysis, we matched every predicted address with the ground truth and report the densest DOTAv2 images separately. An address is on a target object if it lies inside a ground-truth box of the requested category that is not already claimed by another address.

DIORDOTAv2DOTAv2 (>20 objects)
Correct box (IoU ≥ 0.5) when the address is on a target object93.9%86.3%84.8%
Missed objects that never received an address74.6%83.4%83.1%

When an object receives an address, its box is usually correct, also in dense scenes. Most errors come from objects that never receive an address, so the main source of error is finding the instances rather than predicting their boxes from the points. Table 5B is consistent with this: with annotation-derived addresses, [email protected] increases from 82.24 to 91.32 on DIOR and from 66.90 to 89.22 on DOTAv2.

The box step degrades only moderately when the point is less accurate. To show this, we moved each annotation-derived address from the box center to a random point inside the box:

Address position ([email protected])DIORDOTAv2
Annotation-derived, box center (Table 5B)91.3289.22
Annotation-derived, random point inside the box85.4179.94

Random points inside the objects lower [email protected] by 5.9 on DIOR and 9.3 on DOTAv2. Moreover, since each box is decoded independently from its own address, an inaccurate address affects only its own box. Both analyses are added in Sec. 4.4 and App. C.6 (Table 17).

W3: Relation to block diffusion

We see block diffusion as complementary to PID, not as a replacement. LocateAnything already predicts each box in parallel in one PBD step, but different boxes are still generated one after another (Sec. 3.1). Block diffusion works in a similar way: it decodes the tokens inside a block in parallel, but the tokens in a block still attend to each other, and each block still depends on the earlier ones. PID removes this dependence between objects. Once the addresses are known, all boxes are predicted in one pass, which takes only 0.043 s on DIOR and 0.108 s on DOTAv2 per image in the per-step timing of the Table 4 run (about 5–6% of the total time). The remaining time is spent on image encoding and address prediction, and a strong block-diffusion decoder could speed up the address step. The parallel box step of PID would still apply on top of such a decoder. This discussion is added to the related work (Sec. 2).

In summary, the revised manuscript adds the token-by-token baseline, the address-error and address-position analyses, and the discussion of block diffusion. We hope these additions resolve the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

Dear Reviewer TH55,

We sincerely appreciate the opportunity to address your comments. We understand you may have been busy, but we would be grateful if you could consider re-engaging with the discussion. We hope our responses and new experiments address your questions, and we would greatly welcome any additional feedback you might have.

Thank you for your time and efforts in supporting the review process.

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive feedback. The reviewers acknowledged the rationale for decoupling instance identification and box prediction (TH55, NKUp), the controlled comparisons (v4N2), and the gains in accuracy and latency (TH55, NKUp). We have uploaded a revised manuscript with all changes highlighted in blue. The new results are summarized below:

  • Addressing vs. point supervision (NKUp, v4N2). We trained the class-isolated direct-box model with the same point loss as PID to isolate the contribution of point supervision. Adding the point loss changes [email protected] by +2.3 / −1.0 points on DIOR / DOTAv2, respectively, while PID remains 4.5 / 6.2 points higher. The cumulative ablation further shows that most of the improvement comes from predicting each box from its own address with the instance-isolated mask, rather than from the additional supervision.

  • Instance-isolated mask (v4N2). We evaluated the effect of the instance-isolated mask by training PID with cross-box attention. This variant achieves 2.9 / 3.5 points lower [email protected] than PID on DIOR / DOTAv2, respectively, indicating that allowing boxes to attend to other boxes does not improve localization performance.

  • Fully autoregressive baseline and address errors (TH55). We added a fully autoregressive baseline that decodes the same model token by token. It achieves 70.37 / 55.55 [email protected] on DIOR / DOTAv2, respectively, compared with 73.94 / 59.62 using one PBD step per box and 82.24 / 66.90 for PID. The error analysis shows that most missed objects never receive an address. However, when an address falls inside a target object and a box is returned, the predicted box is correct in 93.9% / 86.3% of cases.

  • Efficiency (NKUp, v4N2). We provide a per-step latency breakdown, showing that box prediction accounts for only 5–6% of the total inference time. Using the same addresses, predicting all boxes in a single pass is 3.0× / 5.9× faster than predicting one box at a time, with no loss in accuracy. We also clarify that the timings reported in Table 4 were obtained from the same rotary-stabilized run as the accuracy results in Table 1, and correct the corresponding statements in the manuscript.

  • Scope of claims (NKUp, v4N2). We replace "open vocabulary query" with "class name or phrase", clarify that the main evaluation protocol considers category-conditioned localization, and restrict the efficiency claims to generative VLM-based localizers. We also add evaluations with new category wordings, where PID retains the largest proportion of its original accuracy, and report results on absent categories.

  • Train/test overlap (v4N2). We conduct an analysis of near-duplicates and geographic overlap between training and test images. The results show that PID retains its full performance gain on overlap-free DIOR images. Most of the DOTAv2 overlap originates from the LAE DOTAv2 benchmark split itself and also affects models pretrained on LAE-1M, such as LAE-DINO. On DOTAv2, PID remains the best-performing model after removing overlapping images, although its performance gain is smaller on the remaining, sparser images.

The new results are added where they belong: the token-by-token baseline, the cumulative component ablation (Table 6) and the address-error analysis in Sec. 4.4, the overlap-filtered results in Sec. 4.2, the latency breakdown in Sec. 4.3 and block diffusion in Sec. 2. Full protocols and tables are in App. A.3, C.6 and D.3–D.6. The Abstract, Sec. 1, App. A.2, App. B.3 and the caption of Table 18 (Table 15 in the submission) are updated accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.