acceptodds
Under review as a conference paper at ICLR 2027

WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language Grounding

Abstract

MLLM-based detectors formulate vision-language grounding as autoregressive coordinate generation, yet in dense scenes, they often collapse into a box repetition loop, repeatedly emitting identical boxes until the decoding budget is exhausted. We trace this failure to historical-answer attention takeover: Under full-history autoregression, repeated coordinate tokens in the growing answer history attract disproportionate attention, crowding out the attention sinks needed to attend to new visual evidence. % To break this loop, in this paper, we propose Compressed Window Auto-Regression (WinAR). This new mechanism partitions a long answer sequence into consecutive generation windows and restricts full token-level autoregression to the current window only while compressing each previous window into a fixed set of latent tokens that condition subsequent decoding. This design reduces the repeated exposure to raw historical coordinates that triggers the attention takeover, yet preserves cross-window dependencies essential for coherent multi-object generation. On RSDense, which contains 200-2,000 objects per image, our model WinARDet improves RexOmni F1-score (IoU=0.5) from 32.16 to 39.36 over the baseline, while reducing the redundancy rate from 35.48% to 13.00%. WinARDet also generalizes across decoding strategies and output formats, including parallel box decoding and polygon generation, providing a practical paradigm for long-horizon vision-language grounding across scene densities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Horizon Vision-Language Grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4WqFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies repetitive box generation in MLLM-based detectors under long decoding horizons, particularly in densely populated scenes. The authors associate this failure with increasing attention to historical answer tokens and propose Compressed Window Auto-Regression (WinAR). WinAR preserves exact autoregressive context within the current window while compressing completed windows into compact representations, retaining boundary tokens to maintain local structural continuity. Experiments cover RexOmni and LocateAnything, multiple decoding modes, and both bounding-box and polygon outputs. On RSDense, RexOmni-WinAR improves F1 from 32.16 to 39.36 and reduces the duplicate rate from 35.48% to 13.00% relative to the exposure-matched FHAR* baseline. The paper also reports lower training memory consumption and improved non-redundant prediction throughput.

Critical strengths
  1. On RSDense, WinAR improves RexOmni’s F1 by 7.20 points while reducing the duplicate rate from 35.48% to 13.00% against FHAR* . These results indicate meaningful practical benefits beyond a small improvement in aggregate detection accuracy.
  2. The approach is applicable across multiple generation settings and offers useful resource savings. The experiments include two detector frameworks, multiple decoding modes, and polygon outputs . The reported reductions in training memory and improvements in non-redundant prediction throughput strengthen the practical contribution.
Critical weaknesses
  1. Limited methodological novelty and missing controls. The method combines window-wise generation with familiar compression operations, but its distinction from existing history-management approaches is insufficiently established. Comparisons with history truncation, uniform sampling, and simple pooling are needed to isolate the contribution of the learned compressor.
  2. Insufficient experimental specification. Dataset splits and evaluation descriptions are inconsistent between the main text and appendix. Duplicate-box criteria and cross-window training details also remain unclear, limiting reproducibility and assessment of the reported gains and training fairness.
Questions for authors
  1. How does WinAR compare with history truncation, uniform sampling, and simple pooling under matched budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4WqF

W1/Q1. We compare WinAR with the 3 requested baselines using 300-token windows and a matched history budget of 20 tokens per window. Like WinAR, Pooling and Sampling retain 6 original tokens at each end and divide the middle 288 tokens into 8 equal intervals. Pooling averages the embeddings in each interval, while Uniform Sampling selects the right-middle token. History Truncation retains the last 20 tokens of each window. All variants use the same training setting in Table 8.

RexOmniDOTA2&SODARSDense
R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR*66.9468.2667.592.431.3936.8237.9232.1635.483.14
History Truncation66.5166.2766.351.531.4132.9234.2627.6621.413.97
Pooling65.8662.5864.161.761.1439.0332.1133.9926.072.32
Uniform Sampling65.3166.2465.761.480.7238.2834.0735.1920.851.95
WinAR68.3267.6367.971.260.5145.2338.9039.3613.000.87
  • History Truncation. On RSDense, it reduces predictions of identical-coordinates box from 35.48 to 21.41(RdupR_{dup}), but lowers recall from 36.82 to 32.92, false positives account for a larger share of predictions (precision: 37.92→34.26), and RoverR_{over} rises from 3.14 to 3.97. The decoder emits boxes$(x_1, y_1, x_2, y_2)$ in left-to-right order by x1x_1 and quantizes coordinates into 1000 bins. Dense objects can share the same x1x_1 bin, so the window tail indicates rough progress but leaves ambiguity about which objects were already reported. The decoder may then revisit earlier objects with slightly different coordinates, producing false positives and consuming decoding steps that could have covered new objects.
  • Pooling and Uniform Sampling. Both reduce the history visible to attention, but their fixed rules ignore box boundaries. In geometric task, mixing coordinates across boxes or retaining only fragments of a coordinate tuple can obscure which object regions have already been covered.
  • Novelty. We are the first to identify and characterize the box repetition loop in dense autoregressive detection. This discovery is a contribution in its own right. In designing WinAR, we prioritized simplicity and effectiveness. WinAR substantially outperforms heuristic alternatives on geometric tasks while retaining its generality.

W2. Data Split. We thank the reviewer for identifying this inconsistency and acknowledge that the split in Sec. 4.1 was described incorrectly. The current descriptions and their implied splits are summarized below:

LocationCurrent statementImplied split
Main paper, Sec. 4.1, p.6, 317training on the union of their training sets and evaluating on the union of their validation setsDOTA2-train + SODA-train for training; DOTA2-val + SODA-val for evaluation
Appendix A.1, p.13, 682-685SODA-A train and val splits are merged for training while test split for evaluationDOTA2-train + SODA-A-train/val for training; DOTA2-val + SODA-test for evaluation

All models and experiments consistently use the Appendix split, and the fairness is unaffected. The counts in Tables 6–7 are consistent with these splits. The SODA description in Sec. 4.1 is a wording error that we will correct.

AuthorsOpenReview ↗

Following Response to Reviewer 4WqF

W2. Evaluation. The main text in Sec. 4.1 summarizes the setup and the appendix in App. A.3 provides the details. All methods within each comparison use the same protocol, ensuring fair comparisons.

ProtocolEvaluation ruleReported results
COCO-StyleFollowing RexOmni, retain the first K generated boxes per image per category (K=1000 for DOTA2&SODA) and average P/R/F1 across categories.Tables 1,4,5(DOTA2&SODA)
All-StyleUse the same category-wise evaluation but retain all predictions. This captures the effect of redundant boxes, each of which consumes decoding budget.Table 9
RSDense-StyleRetain all predictions, perform class-agnostic matching within each image, and average per-image P/R/F1 to reduce sensitivity to category imbalance.Tables 1,4,5(RSDense) and Table 2

W2. Duplicate-box criterion. The abstract and the second paragraph of the Introduction describe the repetition loop as repeatedly decoding identical boxes. RdupR_{dup} quantifies coordinate-identical repeats; we will use “exact-duplicate rate” to avoid ambiguity. We already report Rover=(Npred−Ngt)/NgtR_{over}=(N_{pred}-N_{gt})/N_{gt} as a complementary measure of full over-prediction.

W2. Cross-window training. A window is a contiguous, non-overlapping block of answer tokens. A 3,000-token answer yields ten 300-token training instances. When targeting window 3, the model receives the multimodal prefix and compressed ground-truth windows 1–2, and predicts window 3 with causal teacher forcing. Only its 300 tokens contribute to cross-entropy loss; windows 4–10 are excluded. Sec. 3.2, Fig. 4(a), and App. A.2 provide the procedure and configuration.

We have revised the manuscript. We welcome any follow-up questions and will release the preprocessed dataset splits and training and evaluation code.

AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Dear Reviewer,

Thank you again for your constructive feedback. We have clarified the experimental protocols and comparison fairness, added comparisons with the requested baselines under matched history budgets, and revised the manuscript to improve clarity and consistency.

As the discussion deadline approaches, we would appreciate your feedback on whether our responses address your concerns and whether any questions remain. We would welcome the opportunity to provide further clarification.

Thank you for your time and consideration.

Reviewer_SNVTOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the box repetition loop in autoregressive MLLM-based object detection, particularly in densely populated scenes. The authors associate this failure mode with historical-answer attention takeover and propose Compressed Window Auto-Regression (WinAR), which partitions long output sequences into local generation windows while compressing previous outputs into compact latent representations. Experiments demonstrate that WinAR effectively reduces redundant predictions and improves detection accuracy in certain extremely dense scenarios, while also reducing peak training memory consumption.

Critical strengths
  1. Meaningful problem formulation. The paper identifies a practical failure mode of autoregressive MLLM-based detection in dense scenes. The authors provide systematic empirical observations through prediction redundancy, new-object discovery rates, and attention analysis.
  2. A reasonably complete framework with encouraging results. WinAR has a clear motivation and is evaluated across different scene densities, decoding paradigms, and polygon prediction tasks. On RSDense, it improves RexOmni's F1 from 32.16 to 39.36 while reducing the redundancy rate from 35.48% to 13.00%. The exposure-matched baselines and training memory analysis further strengthen the experimental evaluation.
Critical weaknesses
  1. Limited methodological novelty and insufficient justification of the proposed design. Window-based generation and historical context compression are established concepts in long-sequence modeling. This work mainly combines and adapts these ideas to dense object detection. However, comparisons with simpler alternatives, such as history truncation, sliding-window decoding, lightweight compression, and duplicate suppression, are insufficient. Moreover, the method introduces a multi-layer history compressor and requires additional training, but the effectiveness of training only the newly introduced module or using parameter-efficient fine-tuning has not been established. Consequently, the necessity and practical cost-effectiveness of the proposed design remain unclear.
  2. Inconsistent accuracy improvements and insufficient correspondence between qualitative and quantitative results. Although WinAR substantially reduces prediction redundancy, these improvements do not consistently translate into better detection accuracy. For example, the F1 improvement on the full DOTA2 & SODA-A benchmark is only 0.38 for RexOmni. More notably, under LocateAnything-NTP on RSDense, the redundancy rate decreases from 5.90% to 0.49%, while F1 slightly decreases. Furthermore, Figures 5–8 show visually substantial improvements in object coverage, which appear more pronounced than the overall quantitative gains. The representativeness of these examples and their correspondence to evaluation metrics require further clarification. The causal explanation linking attention redistribution to repetitive generation also remains insufficiently validated.

Minor weakness: The presentation of Figure 2 is not intuitive, making it difficult to understand the corresponding content. I encourage the authors to revise the figure and its accompanying explanations to present the key ideas more clearly and concisely.

Questions for authors
  1. Comparison with simpler alternatives: Could the authors compare WinAR with history truncation, sliding-window decoding, simple pooling-based compression, and inference-time duplicate suppression? Spatial image tiling could also reduce the number of objects generated in a single sequence. What are WinAR's distinctive advantages over these alternatives?
  2. Full versus parameter-efficient fine-tuning: Could the authors clarify which components are updated or frozen during training, and whether full-parameter fine-tuning is performed? If so, have the authors evaluated training only the Window History Compressor with a frozen backbone, or using parameter-efficient approaches such as LoRA? How do these alternatives compare in accuracy and training cost?
  3. Qualitative versus quantitative results: Figures 5–8 demonstrate visually substantial improvements in object coverage, whereas the overall F1 gains are relatively modest. Are these visualizations subject to NMS, duplicate filtering, or other post-processing? Could the authors provide per-image Precision, Recall, and F1 for the visualized examples, together with density-stratified evaluations, to clarify their representativeness?
  4. Redundancy versus detection accuracy: Why does the substantial reduction in redundancy not consistently improve Precision or F1, particularly for LocateAnything-NTP? Could the authors provide a breakdown of true positives, duplicate false positives, other false positives, and missed objects to quantify how much repetitive generation actually limits detection performance?
  5. Causal explanation and computational efficiency: The current attention analysis primarily establishes correlation. Could the authors provide interventions, such as historical-token perturbation or attention manipulation, to further validate the proposed causal explanation? Additionally, could they report total training GPU-hours and end-to-end inference latency to better establish the practical computational benefits of WinAR?

If the authors provide a convincing response to my concerns, I would consider raising my score.

9 comments
AuthorsOpenReview ↗

Response to Reviewer SNVT

W1/Q1. Comparison with simpler alternatives. We compare WinAR with the 5 requested baselines.Truncation uses the same window-wise setup and keeps the last 20 tokens of each 300-token window. Pooling keeps 6 original tokens at each end and averages the tokens within each of 8 equal intervals in the middle. The remaining baselines are training-free. Sliding Window adapts StreamingLLM’s strategy, preserving the full multimodal prefix and the latest 512 answer tokens. Duplicate Suppression changes the decoding repetition penalty from 1.05 to 1.15. Spatial Image Tiling splits each image into four equal tiles, runs on each tile and merges the results.

RexOmniDOTA2&SODARSDense
R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR*66.968.367.62.431.3936.837.932.235.483.14
History Truncation66.566.366.41.531.4132.934.327.721.413.97
Sliding Window58.955.356.20.661.6416.425.219.215.294.63
Pooling65.962.664.21.761.143932.13426.072.32
Duplicate Suppression6669.167.51.310.5336.84033.725.522.14
Spatial Image Tiling57.963.359.80.570.2742.540.140.210.010.66
WinAR68.367.667.91.260.5145.238.939.4130.87
  • History Truncation. On RSDense, truncation reduces exact duplicates RdupR_{\rm dup} (35.48% → 21.41%), but recall and precision fall while RoverR_{over} rises (3.14 → 3.97). The decoder generates boxes (x1,y1,x2,y2)(x_1,y_1,x_2,y_2) in left-to-right order by x1x_1, with coordinates quantized into 1,000 bins. Nearby objects can share an x1x_1 bin, so retaining each window’s final 20 tokens provides only a rough indication of progress while discarding much of the information about previously reported objects. The decoder may consequently revisit these objects with slightly shifted boxes, consuming decoding steps that could otherwise discover new objects.
  • Sliding Window. Restricting inference to the latest 512 answer tokens removes older detection history and creates a train–inference mismatch for a decoder trained with full answer histories, which contributes to its performance decline.
  • Repetition Penalty. Increasing the penalty reduces RdupR_{dup} on RSDense from 35.48% to 25.52%, but recall remains essentially unchanged. Thus, suppressing repeated tokens alone does not meaningfully improve object coverage in this setting.
  • Pooling. Although Pooling compresses the history, its fixed intervals ignore box boundaries and can mix coordinate embeddings from different objects. The averaging rule neither explicitly preserves box structure nor learns which geometric relationships to retain, potentially obscuring which object regions have already been covered.
  • Spatial Image Tiling. Splitting each image into four tiles reduces the number of objects generated per sequence, improving RSDense F1 from 32.2 to 40.2. However, on DOTA2&SODA, which includes scenes with widely varying object sizes, F1 drops from 67.6 to 59.8. Fixed tiling can split large objects across tile boundaries and remove surrounding context, making it less suitable for scenes where small and large objects coexist and limiting its general applicability.

W1/Q2. Full versus parameter-efficient fine-tuning. All main WinAR experiments use full-parameter fine-tuning, jointly updating RexOmni and the Window History Compressor (WHC). We additionally evaluate two alternatives with the pretrained RexOmni weights frozen: training WHC alone, and training WHC together with LoRA adapters. Both substantially reduce trainable parameters and training time, but yield markedly lower detection accuracy on both datasets.

TrainingTrainable params↓Wall time (h,8×A40)↓DOTA2&SODARSDense
R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
Full4.1B47.668.367.667.91.260.5145.238.939.4130.87
WHC+LoRA360.6M20.760.561.360.62.260.6222.925.617.638.683.67
WHC319.5M16.252.559.855.93.230.6910.416.86.952.294.63
AuthorsOpenReview ↗

Following Response to Reviewer SNVT

W2/Q3. Qualitative versus quantitative results. Fig 5–8 show outputs without any post-processing. They were selected to illustrate extremely dense RSDense and SKU110K cases. We have also updated Fig 5 to report per-image precision, recall, and F1 for both FHAR* and WinAR. FHAR’s repeated boxes overlap at identical coordinates and thus appear as one. The modest SKU110K gain in Tab 9 (Tab 13 in new PDF) also partly reflect annotation incompleteness: Fig 6 suggests that some small objects are unannotated, so detections on them are counted as false positives. DOTA2&SODA-A spans a wide range of object densities and is included to show that WinAR improves performance in dense scenes (RSDense) without sacrificing performance in ordinary scenes. The density-stratified evaluation results on DOTA&SODA-A are shown below.

GT objects per imageImagesPrecision (%)Recall (%)F1 (%)
0–19917,96969.51 → 68.84 (−0.67)69.32 → 70.14 (+0.82)69.42 → 69.49 (+0.07)
200–39948366.22 → 64.21 (−2.01)49.36 → 54.18 (+4.82)56.56 → 58.77 (+2.21)
400–59913557.85 → 53.19 (−4.66)36.02 → 40.68 (+4.66)44.40 → 46.11 (+1.71)
600–7994856.00 → 53.38 (−2.62)25.21 → 57.60 (+32.39)34.78 → 55.41 (+20.63)
800–999197.98 → 59.06 (+51.08)8.31 → 59.14 (+50.83)8.14 → 59.10 (+50.96)
1,000–1,199148.72 → 14.38 (+5.66)7.22 → 10.90 (+3.68)7.90 → 12.40 (+4.50)
1,200–1,399510.71 → 10.91 (+0.20)5.26 → 8.14 (+2.88)7.05 → 9.32 (+2.27)
1,400–1,599562.19 → 63.99 (+1.80)55.21 → 55.55 (+0.34)58.49 → 59.48 (+0.99)
1,600–1,799652.61 → 57.56 (+4.95)50.84 → 53.05 (+2.21)51.71 → 55.21 (+3.50)
1,800–1,999233.61 → 7.77 (−25.84)1.34 → 2.33 (+0.99)2.57 → 3.59 (+1.02)
2,200–2,399120.96 → 13.67 (−7.29)3.08 → 3.73 (+0.65)2.36 → 5.86 (+3.50)
Overall18,68768.26 → 67.63 (−0.63)66.94 → 68.32 (+1.38)67.59 → 67.97 (+0.38)

W2/Q4. Redundancy versus detection accuracy. F1 changes in DOTA&SODA-A of Tab1 are not strictly proportional to the changes in redundancy. As explained in App. A.3, DS follows RexOmni’s F1 protocol, which adopts the prediction cap used in COCO AP evaluation: only the first K boxes per category in each image are evaluated (K=100 on COCO, K=1000 on DS). Late decoded duplicate boxes therefore not affect the reported F1. Tab9 (Tab13 in new PDF) complements this protocol by retaining the complete boxes.

For LA-NTP, Rdup=Ndup/NpredR_{dup}=N_{dup}/N_{pred}, where NdupN_{dup} counts coordinate-identical boxes defined in abstract. Its reduction indicates that WinAR breaks repetition loop, while the nearly unchanged RoverR_{over} suggests that some predictions become unique false positives, hence the limited P/F1 gain. WinAR yields clear gains under LA-MTP, where repetitive generation and false positives are more severe, consistent with our claim.

Q5. Causal explanation and computational efficiency. We test whether answer tokens crowd out attention to the initial sink tokens (system tokens) and thereby promote box repetition loop on RSDense with RexOmni-FHAR*. For each image that enters a repetition loop, we denote the first loop-onset box index by BB. We adjust only the attention logits of the initial system tokens serving as attention sinks, separately for each layer and head:

  • Before loop onset. After decoding B/4B/4 or B/2B/2 boxes, we lower the sink attention mass to half that of the historical answer tokens to test whether weakening the sink promotes earlier repetition.
  • At loop onset. After decoding box BB and entering the box repetition loop, we raise the sink attention mass to match that of the historical answer tokens (1:1) to test whether the decoder can exit the loop and resume finding new objects.
RexOmniRecall (%)↑Precision (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR*36.8237.9232.1635.483.14
Weaken sink at B/4B/431.8529.3128.6847.894.30
Weaken sink at B/2B/234.7229.1029.1947.874.46
Strengthen sink at BB39.7537.3936.9616.981.32

Weakening sink attention increases exact repetition and reduces precision and recall, with earlier intervention causing larger recall and F1 losses. Conversely, restoring sink attention at loop onset improves recall and F1 while substantially reducing repetition.

These bidirectional effects support our explanation that historical-answer attention crowds out initial sink attention, promoting box repetition loops. They also support the rationale behind WinAR: compressing raw answer history curbs its dominance over attention while retaining useful coverage cues, helping the decoder continue discovering new objects.

We have revised the manuscript. We welcome any follow-up questions and will release the preprocessed dataset splits and training and evaluation code.

Dear Reviewer,

Thank you again for your detailed and constructive feedback. We have carefully worked through your concerns. Our responses now include:

  • Comparisons with simpler alternatives, including history truncation, sliding-window decoding, pooling, repetition penalties, and spatial image tiling.
  • Accuracy and training-cost comparisons between full fine-tuning, compressor-only training, and compressor training with LoRA.
  • Clarifications of why reduced redundancy does not always translate into proportional accuracy gains, including the evaluation caps and LA-NTP results, together with density-stratified evaluations.
  • Attention interventions examining how changes in sink attention affect repetitive generation and object discovery.

We have also revised Figure 2 and its explanation and incorporated additional analyses and clarifications into the manuscript.

As the discussion deadline approaches, we would appreciate your feedback on whether these additions address your concerns. We would welcome the opportunity to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_sJSROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WinARDet investigates the box repetition problem in long-horizon autoregressive visual grounding. The authors identify an attention shift toward historical coordinate tokens and propose a compressed window autoregression mechanism that replaces full answer histories with compact latent representations. By maintaining exact autoregression within each window while compressing previous windows, the method reduces redundant predictions and improves object discovery in dense scenes. Experiments demonstrate clear improvements on dense detection benchmarks, with additional benefits in training memory efficiency and compatibility with different decoding strategies.

Critical strengths
  • Clear problem identification supported by empirical observations. The paper identifies a specific failure mode in generation and provides experimental evidence to support its observations.
  • Simple and reasonably generalizable method. The proposed approach is straightforward and has the potential to generalize to other settings.
  • Well-designed experiments with clear efficiency gains. The experimental evaluation is reasonably rigorous and demonstrates substantial improvements in efficiency.
Critical weaknesses
  • The cause of repetitive generation could be further investigated. The observed attention shift suggests a correlation, but its causal role remains unclear.

  • Comparisons with simpler history management strategies would be helpful. Baselines such as history truncation, sliding-window attention, and KV-cache compression could better highlight the benefits of learned compression.

  • The History Compressor could be evaluated more directly. It would be useful to examine whether compressed tokens preserve key historical information, such as object locations, categories, and spatial coverage.

Questions for authors

The main strength of this paper lies in identifying a specific generation failure mode and proposing an effective engineering solution. However, one key question remains insufficiently addressed: Why is learned window-based compression necessary, and why would simpler strategies, such as history truncation or deduplication, be insufficient?

3 comments
AuthorsOpenReview ↗

Response to Reviewer sJSR

W1. Attention interventions to investigate the cause of repetitive generation. We test whether answer tokens crowd out attention to the initial sink tokens (system tokens) and thereby promote box repetition loop on RSDense with RexOmni-FHAR*. For each image that enters a repetition loop, we denote the first loop-onset box index by BB. We adjust only the attention logits of the initial system tokens serving as attention sinks, separately for each layer and head:

  • Before loop onset. After decoding B/4B/4 or B/2B/2 boxes, we lower the sink attention mass to half that of the historical answer tokens to test whether weakening the sink promotes earlier repetition.
  • At loop onset. After decoding box BB and entering the box repetition loop, we raise the sink attention mass to match that of the historical answer tokens (1:1) to test whether the decoder can exit the loop and resume finding new objects.
RexOmniRecall (%)↑Precision (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR*36.8237.9232.1635.483.14
Weaken sink at B/4B/431.8529.3128.6847.894.30
Weaken sink at B/2B/234.7229.1029.1947.874.46
Strengthen sink at BB39.7537.3936.9616.981.32

Weakening sink attention increases exact repetition and reduces precision and recall, with earlier intervention causing larger recall and F1 losses. Conversely, restoring sink attention at loop onset improves recall and F1 while substantially reducing repetition.

These bidirectional effects support our explanation that historical-answer attention crowds out initial sink attention, promoting box repetition loops. They also support the rationale behind WinAR: compressing raw answer history curbs its dominance over attention while retaining useful coverage cues, helping the decoder continue discovering new objects.

W2/Q. Comparison with three alternatives. We compare WinAR with three alternatives. History Truncation uses the same window-wise training setup and retains the last 20 tokens of each completed 300-token window, matching WinAR’s per-window history budget. The other two alternatives require no additional training. Sliding Window adapts StreamingLLM’s retention strategy, keeping the full multimodal prefix and the latest 512 answer tokens at each decoding step. Repetition Penalty increases the decoding penalty from 1.05 to 1.15.

RexOmniDOTA2&SODARSDense
R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓R (%)↑P (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR*66.968.367.62.431.3936.837.932.235.483.14
History Truncation66.566.366.41.531.4132.934.327.721.413.97
Sliding Window58.955.356.20.661.6416.425.219.215.294.63
Repetition Penalty66.069.167.51.310.5336.840.033.725.522.14
WinAR68.367.667.91.260.5145.238.939.413.000.87
  • History Truncation. On RSDense, truncation reduces exact duplicates RdupR_{\rm dup} (35.48% → 21.41%), but recall and precision fall while RoverR_{over} rises (3.14 → 3.97). The decoder generates boxes (x1,y1,x2,y2)(x_1,y_1,x_2,y_2) in left-to-right order by x1x_1, with coordinates quantized into 1,000 bins. Nearby objects can share an x1x_1 bin, so retaining each window’s final 20 tokens provides only a rough indication of progress while discarding much of the information about previously reported objects. The decoder may consequently revisit these objects with slightly shifted boxes, consuming decoding steps that could otherwise discover new objects.
  • Sliding Window. Restricting inference to the latest 512 answer tokens removes older detection history and creates a train–inference mismatch for a decoder trained with full answer histories, which contributes to its performance decline.
  • Repetition Penalty. Increasing the penalty reduces RdupR_{dup} on RSDense from 35.48% to 25.52%, but recall remains essentially unchanged. Thus, suppressing repeated tokens alone does not meaningfully improve object coverage in this setting.
AuthorsOpenReview ↗

Following Response to Reviewer sJSR

W3/Q. Why learned history compression helps. Object detection predicts an unordered set of boxes, but autoregressive detectors impose a left-to-right order to serialize this set. To investigate whether WinAR learns implicit representations of spatial coverage from historical detections, we shuffle the historical box order while preserving each box’s category and coordinates, keeping the spatial coverage information unchanged. we randomly shuffle complete boxes within each history window (300 tokens) at inference for both FHAR* and WinAR. FHAR* uses the shuffled windows as raw history, whereas WinAR compresses them. We compare each method’s performance with its unperturbed baseline on RSDense, averaging the shuffled results over three runs.

RexOmniRecall (%)↑Precision (%)↑F1 (%)↑RdupR_{\rm dup} (%)↓RoverR_{\rm over}↓
FHAR* (Base)36.837.932.235.483.14
FHAR*, shuffled boxes24.729.725.942.143.67
WinAR45.238.939.413.000.87
WinAR, shuffled boxes42.435.536.419.531.21

Shuffling historical boxes reduces FHAR*’s recall/F1 from 36.8/32.2 to 24.7/25.9. WinAR experiences smaller declines, from 45.2/39.4 to 42.4/36.4, and still outperforms the unperturbed baseline. Although the original order remains helpful to both models, WinAR is substantially less sensitive to its disruption.

WinAR’s robustness supports the interpretation that its learned summaries retain useful information about which objects have already been covered, beyond their original order. These findings indicate that WinAR not only compresses raw answer history but also learns implicit representations of which objects have already been covered.

We have revised the manuscript. We welcome any follow-up questions and will release the preprocessed dataset splits and training and evaluation code.

AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Dear Reviewer,

Thank you again for your constructive feedback. Your questions helped us examine more closely both the cause of repetitive generation and the role of learned history compression. In our responses, we have added:

  • Attention interventions testing how weakening or restoring sink attention affects repetition and continued object discovery.
  • Comparisons with history truncation, sliding-window decoding, and repetition penalties to assess what simpler strategies can achieve and where they fall short.
  • Box-order-shuffling experiments to investigate whether the compressed history retains useful information about previously covered objects.

We have also revised the manuscript to clarify the analysis and experimental findings. As the discussion deadline approaches, we would appreciate your feedback on whether this additional evidence addresses your concerns about learned compression and whether any questions remain. We would welcome the opportunity to clarify these further.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.