acceptodds
Under review as a conference paper at ICLR 2027

CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinement

Abstract

Visual grounding localizes an object described by language with a bounding box. Most multimodal grounding models compress target identification, spatial reasoning, and boundary estimation into one terminal prediction. Free-form rationales make reasoning linguistically explicit but do not necessarily expose measurable, editable spatial states. Intermediate localization errors are therefore difficult to diagnose and correct, allowing incorrect region choices and imprecise boundaries to persist in the final box. We introduce CoEvolve, a construct-to-edit framework that separates grounding into explicit state construction and state editing. Region-Evolution Reinforcement (RER) organizes grounding analysis into a progressive semantic–spatial trajectory, with each reasoning step committing to an explicit candidate region. Bidirectional Denoising Refiner (BDR) treats the reasoning text as fixed semantic context and refines the trajectory's coordinate fields through bidirectional same-position reconstruction. Geometry- and behavior-level objectives provide target geometry and edit-preference signals for consolidating reliable candidates, preserving accurate inputs, or correcting toward annotations. Evaluations cover natural-image and remote-sensing grounding. With a 9B backbone, CoEvolve rivals models up to 241B parameters in grounding accuracy. Under controlled corruption, a single BDR pass improves mean box overlap by over 27 percentage points, demonstrating strong recovery from substantial localization errors. State-source comparisons further support the complementarity of explicit state construction and source-matched editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual grounding, Reinforcement Learning, Bidirectional denoising repairable localization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eiXHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoEvolve separates visual grounding into state construction and coordinate editing. Region-Evolution Reinforcement (RER) uses GRPO and terminal box annotations to generate reasoning steps with progressively refined candidate regions. Bidirectional Denoising Refiner (BDR) then updates coordinate tokens in place while retaining the reasoning text. The stages use separate adapters on a frozen Qwen3.5-9B backbone. Experiments cover RefCOCO-family grounding, controlled box correction, and remote-sensing adaptation.

Critical strengths
  1. Explicit, editable spatial states. Associating reasoning steps with candidate regions makes intermediate localization decisions measurable and provides a useful interface for correction.

  2. Systematic rewards without intermediate annotations. Equation 6 combines format, spatial-progress, and terminal-localization rewards. The progressive reward derives geometric feedback from final annotations without requiring labeled reasoning chains or region trajectories.

  3. A reasonable coordinate-editing mechanism. Bidirectional same-position reconstruction enables synchronous coordinate updates using the complete response context while preserving its text and structure.

  4. Strong final localization performance. Across five test splits, CoEvolve improves mIoU from 82.16% to 85.72% and [email protected] from 60.64% to 68.64% over native Qwen3.5-9B, supporting its effectiveness for precise localization.

Critical weaknesses
  1. Semantic–spatial consistency remains unclear. RER’s geometric rewards and BDR’s edit preferences refer to the final target annotation rather than each step’s textual meaning. In Figure 8, BDR replaces a scene-level region with a localized region while retaining the scene description. Thus, improved final coordinates do not necessarily imply faithful intermediate states. Appendix D acknowledges this limitation, but its implications for interpreting the trajectory are insufficiently examined.

  2. The contribution of RER trajectories is not isolated. In Table 3, RER starts below native output (80.70% versus 82.16% mIoU) but finishes above it after source-matched refinement (85.72% versus 84.57%). This supports the complete pipeline, but the editors differ in editable positions and supervision (Appendix A.6). A controlled comparison under matched training and inference budgets is needed among four settings: (i) refining only a terminal bounding box, without intermediate reasoning or regions; (ii) refining a terminal box accompanied by brief reasoning, without a multi-step region trajectory; (iii) refining the full RER reasoning-and-region trajectory; and (iv) directly predicting coordinates through bidirectional reconstruction or iterative denoising, without an autoregressive state-construction stage. This would clarify the contributions of explicit reasoning, intermediate regions, and the construct-to-edit design.

  3. Key design choices lack targeted ablations. Zero-Anchor BDR removes both the localization hypothesis and associated preservation/preference objectives, establishing only their joint value. The individual contributions of the progressive reward, bidirectional attention, quality-gated targets, and auxiliary losses remain unclear.

  4. Controlled correction does not isolate editing from re-localization. In Section 4.4, native Qwen3.5-9B reaches 83.85% output mIoU versus BDR’s 86.19%, but only BDR receives task-specific training. A model could also achieve a high repair rate by ignoring the anchor and locating the target again. The experiment therefore does not distinguish the benefits of state-dependent editing from additional localization training.

  5. The motivation and positioning could be clearer. The first introduction paragraph moves from prior grounding and reasoning methods to the need for editable intermediate states, without sufficiently explaining why this is an important limitation or how addressing it benefits localization. Meanwhile, although related work is discussed briefly in the introduction and preliminaries, a more structured review would help contextualize the approach and clarify its differences from existing progressive grounding, grounded reasoning, and diffusion-based localization methods.

Questions for authors
  1. Are edited historical regions expected to remain consistent with their original textual roles? Please clarify Figure 8 and provide more visualization cases.

  2. How much does BDR rely on the reasoning text and input boxes? Text-removal/shuffling and anchor-replacement controls would help distinguish state-dependent editing from re-localization.

  3. Does the full RER trajectory outperform a terminal-box-only state under comparable editor supervision and editing budgets? What accounts for RER’s lower initial accuracy than native output?

  4. How does Equation 8 compare with always using ground-truth targets? Targeted ablations of the progressive reward, regression, edit preference, and bidirectional attention would clarify the essential components.

4 comments

Thank you for your feedback. Unless noted, all tests use the 30,969 examples across the 5 test splits (invalid IoU=0\text{IoU}=0, 2 BDR passes).

Critical Weaknesses

  • W1 (Historical roles): Text retains original reasoning while Step boxes move toward b∗b^*; Answer-only mode keeps them 100% faithful (Q1).
  • W2 (Trajectory): Matched controls isolate read-only trajectory and multi-step editing gains (Q3).
  • W3 (Components): Ablations verify Eq. (8), losses, attention, and RprogR_{\text{prog}} (Q4).
  • W4 (Anchors vs. re-localization): In Sec. 4.4 (1 pass), zeroing the 58.80%-mIoU anchor in BDR drops mIoU/[email protected] by -2.93/-6.80 pp (below Native Qwen3.5), proving anchor editing:
Setting (Sec. 4.4)mIoU[email protected][email protected]
BDR, perturbed anchor86.1987.8968.32
Native Qwen3.5, same anchor83.8584.1662.86
Same BDR, zeroed anchor83.2683.4761.52
  • W5 (Positioning): We will expand Intro/Related Work to contrast CoEvolve with GRIT, Grounded-CoT, GeoSearcher, and Lavida-O.

Questions for Authors

Question 1: Historical Roles & Fig. 8

Last Step/Answer use CE+L1, General/Format CE keep text, and Preference pulls Step boxes toward b∗b^* while keeping accurate answers fixed (Fig. 8 broccoli vs. person in [0,1000][0,1000]):

RoleRERPass 1Pass 2
Broccoli Step 1[0,0,1000,1000][810,20,1000,400][810,20,1000,420]
Broccoli Ans (IoU)[605,30,1000,420] (.451)[815,20,1000,410] (.913)[815,20,1000,420] (.936)
Person Step 1[0,0,1000,1000][0,310,200,1000][0,310,250,1000]
Person Ans (IoU)[0,320,250,1000] (.936)[0,320,250,1000] (.936)[0,320,250,1000] (.936)

For strict text–box alignment, Answer-only editing (Q3.1) keeps Step boxes fixed (85.35 mIoU / 67.27 [email protected]).

Question 2: Text & Box Reliance

Intervening on inputs before Pass 1 (Replace uses a valid unrelated box):

StatemIoU[email protected]Valid%
Original (Complete BDR)85.7268.6499.93
No text (keep boxes)85.3568.2099.95
Shuffle tokens85.3668.2599.91
Replace all boxes84.7264.8399.95
No text + replace boxes84.1364.3999.92
Replace Answer (keep Steps)84.3365.7598.84
  1. Anchors: Perturbation training gives ~84% mIoU fallback, but replacing boxes drops [email protected] by 3.81–4.25 pp, proving reliance on RER anchors.
  2. Cross-step reading: Replace Answer beats Replace all boxes by +0.92 [email protected], showing BDR reads intact Step boxes (1.1% validity loss from Step KK/Answer conflict lowers mIoU).
  3. Text: Coordinates dominate (No text: -0.44 [email protected]), but under corrupted boxes, text recovers +0.59 mIoU / +0.44 [email protected].

Question 3: State Comparison & Initial Accuracy

3.1 Matched Four-Way State Comparison

Settings (i)–(iii) share the backbone, RER inputs, and Answer-only losses (text/Steps read-only); (iv) edits a zero box (83.79/63.40, matching Fig. 4's 83.49/62.16).

State & editing scopemIoU[email protected][email protected][email protected]
(i) Terminal box only84.8691.9586.6865.98
(ii) Brief + terminal85.0692.2086.8366.18
(iii) Trajectory (Answer edit)85.3592.3587.3467.27
(iii-b) Trajectory (All-box)85.7292.5087.9468.64
(iv) Direct prediction83.7990.5184.2763.40

Read-only trajectories (iii) beat (i)–(ii) (+1.29/+1.09 [email protected], +398 cases at IoU≥0.9\text{IoU}\ge 0.9 over (i)); all-box editing (iii-b) adds +1.37 [email protected].

3.2 Initial Accuracy: RER vs. Native

Multi-step output is OOD for JSON-pretrained Qwen3.5 (63.48/24.28 unadapted, Table 3). RER reaches 80.70/59.81, and its spatial trajectory doubles BDR's gain (+5.02/+8.83 to 85.72/68.64, vs. +2.41/+5.68 to 84.57/66.32 for Native+BDR).

Question 4: Targets, Losses, Attention & Rewards

4.1 Target Gating, Attention & Losses

SettingmIoU[email protected][email protected]Valid%
Complete BDR (Eq. 8)85.7287.9468.6499.93
Always-GT (w/o Eq. 8)84.9687.0166.7999.92
Causal masks85.3387.5267.5899.90
No L185.3387.3367.7499.77
No Preference85.2287.0566.9999.92
No General + Format CE———5.39
No hist. + final entropy85.5487.5668.1299.92

Eq. (8) adds +0.76 mIoU / +0.93 [email protected] / +1.85 [email protected] over Always-GT and lifts IoU≥0.9\text{IoU}\ge 0.9 retention (18,524 inputs) from 89.76% to 93.61%. Right context, L1, Preference, and Entropy add +1.06 / +0.90 / +1.65 / +0.52 [email protected] (Preference lifts IoU<0.7\text{IoU}<0.7 [email protected]: 49.25%$\rightarrow$51.50%); no General/Format CE drops validity to 5.39%.

4.2 Progressive Reward (RprogR_{\text{prog}})

RewardsmIoU[email protected]Step 1 →\rightarrow Ans IoU
w/o progressive reward78.6557.79+2.14
Complete reward (Eq. 6)80.7059.81+59.45

RprogR_{\text{prog}} prevents static boxes and adds +2.05 mIoU / +2.02 [email protected]. We will add all controls and clarify Fig. 8.

Thank you for asking how the edits relate to the original reasoning and what the trajectory and input anchors contribute to refinement.

  1. Historical boxes (Weakness 1 / Question 1). We added a second example to Appendix C.3 (pp. 28–29). In the first example (Figure 8), the first Step box tightens while its text still describes the full scene. In Figure 9 and Table 16, the first historical box moves toward the person and the Answer stays at IoU 0.936 in both passes. Appendix D (p. 36) explains how edits can change the spatial scope of a Step box. Answer-only decoding leaves earlier text and boxes unchanged.

  2. Trajectory context (Weakness 2 / Question 3). We compare three Answer-only editors with the same starting Answer and dedicated Answer losses in Section 4.5 and Table 4 (p. 9). The full trajectory gains 1.29 [email protected] points over the terminal box alone and 1.09 points over a brief explanation with the box. Table 20 (p. 34) gives the gains on all five splits for the full trajectory relative to the terminal box. Table 19 (p. 33) also reports direct prediction from an all-zero box and the full-state editor that supervises and updates earlier boxes.

  3. RER adaptation and progressive reward (Weakness 2 / Question 3; Weakness 3 / Question 4). Under the same structured interface, RER raises terminal mIoU from 63.48% to 80.70% (Section 4.5, p. 8). After refinement by their respective BDR editors, the RER and native outputs reach 85.72% and 84.57% mIoU (Tables 3 and 12, pp. 9 and 26). Removing the progressive reward lowers terminal mIoU by 2.05 points and [email protected] by 2.02 points. Table 13 (p. 26) also reports the IoU change from the first Step to the Answer.

  4. Targets, losses, and attention (Weakness 3 / Question 4). Section 4.5 (p. 9) reports the causal-mask, preservation, and repair results. Table 21 (p. 34) reports 92.06% [email protected] for Always-GT and gives the results of the loss ablations. For the General and Format CE ablation, we use the same decoding procedure in both conditions. Section 3.3 (p. 5) states the priority of the output branch after Eq. (8).

  5. Text and coordinate inputs (Question 2). Appendix C.8 and Table 22 (p. 35) compare removing or shuffling the reasoning text and replacing coordinate boxes. Replacing all boxes lowers [email protected] by 3.81 points; removing the text alone lowers it by 0.44 points. With the boxes replaced, retaining the text recovers 0.44 points in [email protected].

  6. Controlled correction with and without the input anchor (Weakness 4). We compare perturbed and zeroed anchors with the same BDR checkpoint, images, expressions, and one-pass budget (Section 4.4, p. 8; Table 23, p. 35). Zeroing the anchors lowers mIoU from 86.19% to 83.26% and [email protected] by 6.80 points.

  7. Relation to prior work (Weakness 5). The Introduction (p. 1) places RER alongside work on explicit regions in reasoning and sequential region search. CoEvolve passes those regions to BDR as starting coordinates and as a reference for evaluating edits. Section 2.2 (p. 3) discusses masked diffusion for grounding, including Lavida-O, and describes how BDR edits coordinates in an existing response.

Thanks for the authors' response. I think the authors have addressed my concerns, and I am willing to raise my score to weak accept. However, I am not very familiar with this research area, so I keep my confidence score at 2.

Dear Reviewer eiXH,

Thank you very much for taking the time to review our response and for raising your score. Your constructive questions regarding the matched trajectory controls, input interventions, and historical box behaviors were instrumental in helping us strengthen the experimental analysis and clarity of the revised manuscript. We truly appreciate your thoughtful engagement and feedback throughout the review process.

Best regards, The Authors

Reviewer_dXU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the visual grounding task. To this end, the authors propose CoEvolve, that aims to push the visual grounding capability to a higher performance. At the core of the method, there are two components: 1. Region-Evolution Reinforcement (RER); and 2. Bidirectional Denoising Refiner (BDR). In this way, the final visual grounding is achieved with a trajectory of visual grounding predictions. Through the step-by-step adjustment of the predicted bounding boxes, the authors show the proposed method achieves the state-of-the-art performance for visual grounding on benchmarks such as RefCOCO splits.

Critical strengths
  1. Visual grounding is an important task and problem to study. It is an essential task to evaluate the visual / spatial capabilities of VLMs. The visual grounding task can also benefit a lot of downstream applications, such as robotics and autonomous driving.

  2. The authors' method have achieved strong evaluation performance for visual grounding accuracy. On RefCOCO, it surpasses a variety of methods or models. The authors have also shown the experiment results for remote-sensing grounding and downstream segmentation, where the method also performs strongly.

Critical weaknesses
  1. The authors have not shown results regarding efficiency of their method. Visual grounding is very important, as it may influence downstream applications such as robotics and autonomous driving, while these tasks strongly require a low latency for visual grounding. Therefore, it would be good if the authors can show the efficiency results for their method.

  2. It would be better if the authors can show more ablations / analysis for us to better understand the improvement. First, how much improvement does the iterative visual grounding method contribute to the evaluation on RefCOCO splits? It makes sense that such iterative method can greatly improve the accuracy for visual grounding on benchmarks and scenarios as in Figure 11 in the appendix, but for RefCOCO, the target object location is relatively easy to identify, so how can the method help improve RefCOCO results? Second, as in Equation 11, there are multiple loss items in the training objective. It is better if the authors can show a break down of the effect of each loss item to help us better understand how each loss item actually functions.

Questions for authors
  1. As in weakness 1, could you please provide efficiency results for your method?

  2. As in weakness 2, could you please provide further analysis regarding how the iterative method contribute to the visual grounding accuracy improvement on RefCOCO?

  3. As in weakness 2, could you please provide the break down of the effect of each loss item?

2 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for recognizing the importance of visual grounding and CoEvolve's strong results.

Critical Weaknesses

  • W1 (Accuracy & runtime): Because BDR updates coordinates synchronously in one non-autoregressive pass, two BDR passes add only 0.465 s/sample (13.72%) at batch 8 (0.498 s at batch 1) while boosting mIoU/[email protected] by +4.00/+6.60 pp on the 500-sample timing subset (+5.02/+8.83 pp on all 30,969 test samples). With vLLM, RER latency drops 4.1×4.1\times to 0.830 s/sample (Q1).
  • W2 (RefCOCO refinement & losses): On RefCOCO, BDR tightens loose boundaries rather than repeating coarse search, raising overall [email protected] 59.81%→68.64%59.81\%\rightarrow 68.64\% (+8.83 pp) (Q2). Ablations verify each term in Eq. (11) (Q3).

Questions for Authors

Question 1: Efficiency Results

1.1 Matched accuracy and runtime (Batch size 8)

We use one A100-40GB (BF16), Transformers with KV caching, and 500 shared examples sampled across the test splits at batch 8 (3 runs after warm-up, mean ±\pm SD). Times cover loading, tokenization, execution, handoff, decoding, and parsing. One frozen backbone hosts both LoRAs: run all RER, switch once (31.6 ms/workload), then BDR.

SettingmIoU[email protected]Amortized time/sample (s)
Native Qwen3.5-9B82.9864.200.405±0.0010.405 \pm 0.001
RER83.2064.803.385±0.0103.385 \pm 0.010
RER + one BDR pass86.5370.003.623±0.0113.623 \pm 0.011
RER + two BDR passes87.2071.403.850±0.0113.850 \pm 0.011

One BDR pass adds 0.238 s/sample (7.03%) for +3.33 mIoU / +5.20 [email protected]; Pass 2 adds ∼0.23\sim 0.23 s/sample and raises [email protected] to 71.40%. Peak VRAM is 23.97 GiB.

1.2 Batch size 1 and vLLM deployment

Same unoptimized Transformers protocol on a 100-example subset at batch size 1:

SettingmIoU[email protected]Amortized time/sample (s)
Native Qwen3.5-9B81.8562.001.981±0.1241.981 \pm 0.124
RER83.5664.0021.474±0.06821.474 \pm 0.068
RER + one BDR pass85.0269.0021.729±0.06821.729 \pm 0.068
RER + two BDR passes85.5470.0021.972±0.06821.972 \pm 0.068

Sequential 3–5 step autoregressive decoding in Transformers dominates RER runtime, while synchronous BDR costs only 0.255 s in Pass 1 and 0.498 s (2.32%) over two passes (0.269 s mean / 0.298 s P95 per pass). With vLLM (500 examples, batch 8), Native reaches 0.134 s/sample, RER gains 4.1×4.1\times speedup (0.830 s/sample), and full CoEvolve takes ∼1.1–1.3\sim 1.1\text{--}1.3 s/sample. For latency-sensitive robotics, vLLM Native plus one BDR pass offers a high-precision <0.4<0.4 s option.

Question 2: Iterative Refinement on RefCOCO

We evaluate 30,969 natural RER outputs across the five test splits (invalid answers assigned IoU=0\text{IoU}=0):

Output statemIoU[email protected][email protected][email protected]
Initial RER prediction80.7087.2280.5659.81
First BDR pass85.4992.3787.4367.97
Second BDR pass (CoEvolve)85.7292.5087.9468.64

Why refinement helps RefCOCO: While coarse target search is easy (initial RER [email protected]: 87.22%), single-pass boxes suffer from loose margins, truncation, or occlusion drift. BDR refines boundaries using visual and trajectory context, yielding +5.02 / +5.28 / +7.38 / +8.83 points in mIoU / [email protected] / 0.7 / 0.9 overall, and +4.42 mIoU (82.80→87.2282.80\rightarrow 87.22) / +9.40 [email protected] (61.78→71.1861.78\rightarrow 71.18) on 10,752 RefCOCO testA/B examples (Tables 8 and 10). Pass 1 achieves 95.4% of the mIoU gain; Pass 2 adds +0.23 mIoU / +0.67 [email protected].

Question 3: Breakdown of Loss Items

3.1 Supervision roles & ablations (5 test splits, 2 passes, Coordinate CE kept)

Removing L1 sets regression weight to 0 while keeping hard-error CE weights; removing Preference retains quality gating.

Setting / Loss rolemIoU[email protected][email protected][email protected]Valid%
Complete BDR (All terms in Eq. 11)85.7292.5087.9468.6499.93
w/o Soft-coordinate L1 (continuous distance gradient)85.3392.2887.3367.7499.77
w/o Coordinate Preference (relative edit utility)85.2292.3687.0566.9999.92
w/o General + Format CE (text/tag preservation)————5.39
w/o Historical + Final entropy (digit sharpening)85.5492.3887.5668.1299.92
  • L1 & Preference add +0.39/+0.50 mIoU and +0.90/+1.65 [email protected]. On natural RER errors with initial IoU<0.7\text{IoU}<0.7, Preference raises [email protected] 49.25%→51.50%49.25\%\rightarrow 51.50\%.
  • Quality gating vs. Always-GT: Always-GT yields 84.96 mIoU / 66.79 [email protected]; quality gating (Eq. 8) adds +0.76/+1.85 points and raises retention on the 18,524 inputs at IoU≥0.9\text{IoU}\ge 0.9 from 89.76% to 93.61%.
  • Preservation & Entropy: Removing General/Format CE without freezing non-coordinate tokens corrupts tags/delimiters, dropping validity 99.93%→5.39%99.93\%\rightarrow 5.39\%. Entropy terms add +0.18/+0.12/+0.38/+0.52 points in mIoU/[email protected]/0.7/0.9. We will add all results to the revision.

Thank you for suggesting that we report accuracy alongside runtime and examine the effects of each refinement pass and training objective.

  1. Accuracy and runtime (Weakness 1 / Question 1). We measured accuracy and runtime on the same 500 examples (Section 4.6, p. 9; Table 5, p. 10). We used one A100-40GB in BF16 at batch size eight, with three timed runs after warm-up. Two BDR passes add 0.465 s per example to the 3.385 s for RER. On this subset, [email protected] rises from 64.80% to 71.40%.

  2. Batch size one and vLLM timings (Question 1). Appendix C.9 (p. 35) describes the timing boundaries, the single adapter switch, and peak memory allocation for the 500-example workload. Table 24 gives Transformers timings for a separate 100-example workload at batch size one. The last paragraph reports vLLM generation timings on the 500-example workload.

  3. Successive refinement passes (Weakness 2 / Question 2). Section 4.3 (p. 7) and Table 15 (p. 27) follow the same natural RER outputs through two BDR passes on all 30,969 test examples. mIoU rises from 80.70% initially to 85.49% after the first pass and 85.72% after the second. [email protected] rises from 59.81% to 67.97% and then to 68.64%. Across both passes, the gain is 5.28 points at [email protected] and 8.83 points at [email protected].

  4. Training objectives (Weakness 2 / Question 3). Table 21 (p. 34) includes the Always-GT target rule, the causal-mask control, and ablations for soft-coordinate L1, coordinate preference, General and Format CE, and the entropy terms. For Always-GT, [email protected] is 92.06%. Removing soft-coordinate L1 keeps the hard-error weights on coordinate CE; removing coordinate preference keeps the target gate. For the General and Format CE ablation, we use the same decoding procedure in both conditions. The valid Answer rate is 99.93% with these losses and 5.39% without them.

  5. Preserving accurate inputs and repairing inaccurate ones (Question 3). Section 4.5 (p. 9) and Appendix C.7 (p. 34) report both input groups. With quality gating, the share of the 18,524 inputs at IoU ≥ 0.9 that still meet the threshold rises from 89.76% to 93.61%. With quality gating in place, adding coordinate preference raises [email protected] from 49.25% to 51.50% for inputs below IoU 0.7.

Reviewer_bz87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage visual grounding framework built on Qwen3.5-9B. Region-Evolution Reinforcement (RER) trains the model to generate a sequence of explicit candidate boxes. A separately trained Bidirectional Denoising Refiner (BDR) then updates coordinate tokens at their existing positions while keeping the surrounding reasoning text fixed. Training combines geometric supervision, quality-dependent target selection, and edit-preference objectives. Experiments cover the RefCOCO family, controlled anchor corruption, domain-specific adaptation to remote sensing, and downstream segmentation. The method reports 92.41% mean split-level [email protected] and 85.72% pooled mIoU on the five test splits.

Critical strengths
  1. The separation between constructing an explicit localization state and editing its coordinates is a useful design. It makes intermediate boxes observable and permits parallel coordinate updates without regenerating the whole response. The evaluation demonstrates practical improvements at strict IoU thresholds: [email protected] reaches 68.64%, compared with 60.64% for the reproduced native backbone (Table 2).

  2. The paper goes beyond a single headline accuracy comparison. It includes natural upstream errors, controlled corruption, a native-output-plus-BDR control, and a zero-anchor ablation. In particular, the RER-plus-BDR pipeline reaches 85.72% mIoU versus 84.57% for the separately trained native-output-plus-BDR pipeline (Table 3 and Appendix C.2). This supports the usefulness of the complete pairing, although it does not isolate each proposed mechanism.

Critical weaknesses
  1. The experiments establish the performance of the complete pipeline more clearly than the necessity of its central design choices. The native-output comparison changes both the state representation and its separately trained editor; the zero-anchor comparison changes both the input state and the anchor-aware objectives, as the paper acknowledges. These controls therefore do not determine whether explicit intermediate regions and bidirectional editing outperform a simpler, equally trained refinement system. The final advantage over native-output-plus-BDR is 1.15 mIoU points. A matched editor comparison, such as retaining only the terminal RER box or training an autoregressive editor on the same states and supervision, would substantially strengthen the methodological contribution.

  2. The core BDR training recipe is incomplete in the supplied materials. Appendix A.7 directs readers to the supplementary code for complete optimization settings, but the supplied ZIP's README explicitly excludes BDR training, and the package contains BDR inference rather than a training implementation. The paper specifies the losses and target rules, and the inference configuration gives adapter settings, but the BDR optimizer, learning rate/schedule, training duration, and effective batch size are not provided there. This is consequential because most of the localization improvement comes from this stage. The concern is the missing reproducible recipe, rather than code availability alone.

Questions for authors
  1. Decision-relevant additional evidence: Do you have a matched comparison that isolates the value of the RER trajectory or bidirectional editing while keeping the backbone, training examples, supervision, and refinement budget comparable? A terminal-box-only BDR control or an equally trained autoregressive editor would be informative. Please distinguish any existing result from a proposed future experiment.

  2. Clarification: Please provide the complete BDR optimization configuration and explain the discrepancy between Appendix A.7 and the supplied compact release. In particular, what are the optimizer, learning rate and schedule, number of updates/epochs, effective batch size, and checkpoint-selection criterion?

3 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for recognizing the value of explicit localization states and coordinate editing. We address both weaknesses and questions with strictly controlled ablations and full training specifications below.

Weakness 1 / Question 1: Trajectory Context and Attention Direction

All controls use two refinement passes on the same 30,969 examples from RefCOCO testA/testB, RefCOCO+ testA/testB, and RefCOCOg test, with the same frozen Qwen3.5-9B backbone and LoRA capacity. All examples are included, with invalid answers assigned IoU=0\text{IoU}=0.

1.1 Full-trajectory vs. terminal-box states

Both Answer-only controls share the RER outputs, images, expressions, initial Answer boxes, target selection, and dedicated Answer coordinate losses (CE, soft-L1, Preference, final entropy). General CE reconstructs text and historical box digits; Format CE supervises structure. To match the terminal-box editor strictly, Full RER trajectory (Answer-only edit) updates only the 16 Answer digits and keeps historical fields fixed as read-only context. Complete CoEvolve BDR jointly edits and supervises both historical and Answer boxes.

Input state & editing scopemIoU[email protected][email protected][email protected]
Terminal box (Answer edit)84.8691.9586.6865.98
Full RER trajectory (Answer-only edit)85.3592.3587.3467.27
Complete CoEvolve BDR (All-box edit)85.7292.5087.9468.64
  1. Read-only trajectory context (+0.49 mIoU, +1.29 [email protected]): Under identical Answer-only supervision, exposing the full trajectory improves [email protected] on all five splits (+398+398 net examples at IoU≥0.9\text{IoU}\ge 0.9), confirming that intermediate spatial commitments aid precise refinement.
  2. Joint multi-step editing (+0.37 mIoU, +1.37 [email protected]): Allowing BDR to jointly update historical and Answer boxes keeps intermediate and terminal anchors consistent across passes, further raising mIoU to 85.72% and [email protected] to 68.64% (+0.86 mIoU / +2.66 [email protected] over terminal-only).

1.2 Attention direction in same-position editing

To isolate bidirectional right-context access under the same refinement budget, both editors share the BDR backbone, states, target selection, losses, and two-pass synchronous updates at existing token positions. The causal editor restores causal masks in the full-attention layers.

Attention settingmIoU[email protected][email protected][email protected]
Causal same-position editing85.3392.2787.5267.58
Bidirectional same-position editing (Ours)85.7292.5087.9468.64

Bidirectional attention improves all four metrics, including +0.39 percentage points in mIoU and +1.06 points in [email protected], as earlier box digits (x1,y1x_1, y_1) and intermediate steps can attend to later coordinates (x2,y2x_2, y_2) and downstream <answer> tokens during synchronous updates. We will add both ablations and split-level results to the revision.

Weakness 2 / Question 2: BDR Configuration and Supplement

The main experiment trains a separate BDR LoRA adapter on the frozen Qwen3.5-9B backbone. We verified this configuration against the training implementation and saved optimizer state:

ItemOriginal main experiment
OptimizerAdamW; β1=0.9,β2=0.999,ϵ=10−8\beta_1=0.9, \beta_2=0.999, \epsilon=10^{-8}, weight decay 0.010.01
Peak learning rate2×10−52\times 10^{-5}
Learning-rate scheduleLinear warm-up over first 3% of updates, then linear decay to 0
Training duration8 epochs on RefCOCOg train states (20,128 optimizer updates)
Effective global batch size32: 2 GPUs ×\times microbatch 4 per GPU ×\times accumulation 4
Gradient clipping & memoryMax global norm 1.0; gradient checkpointing enabled; max length 2,048
LoRA & precisionRank 128, α=256\alpha=256, dropout 0.05, all-linear modules; BF16 backbone
Objective warm-upFirst 3% updates: 10% token masking with Box CE + 0.2 General CE; then full Eq. (11)
Seed & checkpoint selectionSeed 42; final.pt saved deterministically at the end of epoch 8

The initial package covered RER training and BDR inference. We have re-uploaded the full code in the supplementary materials (including the BDR trainer, config, launch script, and README) and will add this table to Appendix A.7.

Thank you for the suggestions on trajectory and attention controls and for asking us to document the BDR training setup.

  1. Trajectory context (Weakness 1 / Question 1). In Section 4.5 and Table 4 (p. 9), we compare the terminal box, a brief explanation with the box, and the full RER trajectory. The three editors share the same initial Answer and dedicated Answer losses, and each updates only the 16 Answer digits. Compared with the terminal box alone, reading the full trajectory raises mIoU by 0.49 points and [email protected] by 1.29 points. Table 19 (p. 33) reports all four localization metrics, and Table 20 (p. 34) shows gains in [email protected] on all five splits. Supervising and editing the historical boxes in full-state BDR adds a further 1.37 points in [email protected] (Appendix C.7, p. 34).

  2. Attention direction (Weakness 1 / Question 1). For the causal editor, we restore the mask in the full-attention layers. Both editors use the same states and losses, and both update existing token positions in two passes. The bidirectional editor gains 0.39 points in mIoU and 1.06 points in [email protected] (Section 4.5 and Table 4, p. 9). Table 21 (p. 34) gives the remaining thresholds and Answer validity.

  3. Training and inference settings (Weakness 2 / Question 2). Appendix A.7 and Table 9 (p. 24) now give the optimizer, learning-rate schedule, and effective batch size of 32. They also specify that we use the checkpoint saved after eight epochs (20,128 updates). Appendix A.5 (p. 22) defines the editing scope for standard BDR and the Answer-only controls. The Reproducibility statement (p. 11) points to the anonymous supplement for RER and BDR training, inference, and evaluation code.

AuthorsOpenReview ↗

Gentle Follow-Up for Reviewer bz87

Dear Reviewer bz87,

As the author-reviewer discussion period is drawing to a close, we wanted to kindly check in and see whether our rebuttal and revised manuscript have adequately addressed your concerns. For your convenience, here is a brief summary of the updates made in response to your review:

  • Matched trajectory and attention controls (Section 4.5, Table 4; Appendix C.7, Tables 19-21): Under matched backbone, data, supervision, and refinement budgets, read-only RER trajectory context improves over terminal-box-only editing by +0.49 mIoU / +1.29 [email protected], joint all-box editing adds a further +0.37 mIoU / +1.37 [email protected], and bidirectional attention outperforms causal same-position editing by +0.39 mIoU / +1.06 [email protected].
  • Complete BDR training recipe and code (Appendix A.7, Table 9; updated supplementary ZIP): We have added the full BDR optimization configuration (AdamW, peak LR 2e-5 with 3% linear warm-up and linear decay, 8 epochs / 20,128 updates, effective batch size 32, LoRA settings, and checkpoint selection) and re-uploaded the complete supplementary package including the BDR trainer, configs, launch scripts, and updated README.

Please let us know if any questions remain or if further clarification would be helpful before the discussion window closes. Thank you again for your time and constructive review.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.