acceptodds
Under review as a conference paper at ICLR 2027

GeoOPD: Internalizing Visual Search from Privileged Evidence for Remote Sensing Visual Reasoning

11 views

Abstract

Applying multimodal large language models (MLLMs) to remote sensing visual reasoning remains challenging because task-relevant evidence is often fine-grained and sparse. Such evidence can be overwhelmed by extensive background content, leading to incomplete perception and visually unsupported answers. Recent “thinking with images” methods expose local details through iterative cropping, but incur additional inference cost and remain vulnerable to redundant or inaccurate region selection; local views are also insufficient for questions that depend on global context. Motivated by this observation, we introduce GeoOPD, a remote sensing post-training framework that uses Evidence-Privileged On-Policy Distillation (E-OPD) to transfer the benefits of training-time evidence localization into tool-free visual reasoning at deployment. We develop an evidence-driven data synthesis pipeline that constructs marked views and textual evidence privileges according to the spatial organization required by each question. During post-training, the teacher additionally observes the organized evidence and provides token-level distributional supervision over student-generated trajectories. Divergence gating further concentrates the learning signal on response positions where privileged evidence induces substantial predictive changes. Experiments across multiple benchmarks show that GeoOPD consistently improves its backbone and performs favorably against both general-purpose and remote-sensing-specific models. These results suggest that task-relevant evidence available only during training can be partially internalized by a model used for standard inference, complementing iterative visual tool use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aHXqOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes GeoOPD, which aims to internalize visual search capabilities without requiring inference-time cropping or additional tool calls. The method first constructs evidence-grounded QA data with five types of organized visual and textual evidence. It then uses E-OPD to transfer knowledge from a teacher with access to additional evidence to a student receiving only the original image and question. Experiments show improvements over the backbone model and competitive performance against existing methods.

Critical strengths

The paper presents a clear motivation for addressing the limitations of existing MLLMs and crop-based visual agents. The proposed framework uses evidence-grounded data synthesis to provide task-relevant supervision and divergence-based token selection to focus learning on informative tokens. The method achieves competitive results on both in-domain and zero-shot benchmarks. The paper is also clearly written and easy to follow.

Critical weaknesses
  1. The paper aims to internalize visual search and reduce the errors introduced by crop-based visual agents. However, the experiments mainly demonstrate improvements in downstream accuracy, without directly showing whether the model learns better evidence localization or reduces visual search errors. Although the paper compares against ZoomEarth, the use of different backbones makes it difficult to isolate the benefits of internalized visual search.

  2. Several related works have already explored privileged visual information and token-level distillation, as discussed in the related work section. It remains unclear how much improvement comes from the proposed evidence organization and divergence gating compared with existing OPD methods. Moreover, E-OPD achieves the same in-domain accuracy as GRPO and only a small improvement on MME-RealWorld-RS, making its distinct advantage less clear.

Questions for authors

1.The method is evaluated on only one student backbone. Could the authors include experiments with another backbone or model scale to demonstrate the generalizability of E-OPD?

2.The paper shows that selecting the top 50% of tokens performs best among the evaluated strategies, but does not explore different retention ratios. A sensitivity analysis of this hyperparameter would help justify the choice.

3.Could the authors further explain why combining Evidence and Gold Answer does not improve performance? Also, given the similar performance of E-OPD and GRPO, how much additional benefit does privileged teacher information actually provide? A comparison with distillation without privileged evidence would help clarify whether the improvements come from the teacher's additional information or the distillation process itself.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aHXq 1/2

Response to Reviewer aHXq 1/2

We thank the reviewer for the constructive suggestions on evidence use, cross-backbone generalization, and the training mechanism. We have added region interventions, a same-backbone comparison, and further ablations to clarify the contributions of evidence organization, token selection, and teacher privileges.

W1. Evidence use and comparison with tool-based inference

Using the supporting-region annotations in ID150, we conduct region interventions on 101 questions that depend on localized evidence. We gray out either supporting or non-supporting regions, jointly masking all supporting regions for multi-object questions, and measure answer retention relative to each model’s original-image prediction.

As shown in Table 1, GeoOPD’s original-image accuracy improves from 66.3% to 75.2%, while retention under non-supporting-region masking increases from 83.16% to 98.02%. GeoOPD substantially reduces answer changes caused by irrelevant background perturbations while retaining responsiveness to supporting evidence. This provides behavioral support for more selective visual evidence use, complementing the accuracy and heatmap analyses.

Table 1. Region intervention results on 101 ID150 questions. Accuracy and retention are reported in %.

ModelOriginal-image accuracySupporting-region masking: retentionNon-supporting-region masking: retentionRetention gap (pp)
Backbone66.341.5883.1641.58
GeoOPD75.248.5198.0249.51

Answer retention measures agreement with each model’s original-image answer, rather than accuracy after masking. The gap is non-supporting-region retention minus supporting-region retention.

To address the backbone difference in the ZoomEarth comparison, we additionally evaluate both methods with Qwen2.5-VL-3B. GeoOPD achieves 73.6%/45.3% on ID150/MME-RS, exceeding ZoomEarth by 28.8/5.3 percentage points without inference-time cropping tools (Table 2).

Table 2. Same-backbone comparison with ZoomEarth using Qwen2.5-VL-3B. Accuracy (%).

MethodInference-time cropping toolsID150MME-RS
ZoomEarthYes44.840.0
GeoOPDNo73.645.3

Q1. Generalization across backbones

The additional Qwen2.5-VL-3B experiment also shows consistent post-training gains. E-OPD improves its SFT initialization from 68.0%/40.6% to 73.6%/45.3% on ID150/MME-RS, yielding gains of 5.6/4.7 percentage points (Table 3). We attribute the substantial SFT improvement to the combined benefits of remote sensing domain adaptation and supervision on reasoning trajectories. Together with the main Qwen3-VL-4B results, this supports transfer across model generations. Further experiments at other model scales are ongoing under our available compute budget.

Table 3. Stage-wise results on the additional Qwen2.5-VL-3B backbone. Accuracy (%).

BackboneTraining stageID150MME-RS
Qwen2.5-VL-3BBackbone37.625.3
Qwen2.5-VL-3BSFT68.040.6
Qwen2.5-VL-3BSFT + E-OPD73.645.3

Q2. Sensitivity to the token retention ratio

We compare retention ratios of 25%, 50%, 75%, and 100% (Table 4). The default 50% achieves the highest ID150 accuracy and ties for the best MME-RS result. It focuses supervision on high-divergence tokens while retaining broader coverage than the 25% setting, which showed less stable optimization in our experiments. Compared with full-sequence supervision, 50% retention improves ID150/MME-RS by 1.6/1.3 percentage points.

Table 4. Sensitivity to the token retention ratio. Accuracy (%).

Retention ratioID150MME-RS
25%73.645.3
50% (default)76.047.3
75%75.247.3
100% (full sequence)74.446.0
AuthorsOpenReview ↗

Response to Reviewer aHXq 2/2

Response to Reviewer aHXq 2/2

W2 (1). Contribution of evidence organization

We compare evidence configurations using the same backbone, SFT initialization, training data, and training budget. Organized visual privileges alone achieve 75.2%/43.3% on ID150/MME-RS, outperforming single-crop OPD by 2.4/1.3 percentage points. Combining organized visual and textual privileges yields 76.0%/47.3%, corresponding to total gains of 3.2/5.3 percentage points over single-crop OPD (Table 5).

These results highlight the limitations of a uniform single-crop strategy for questions involving spatially dispersed evidence. Question-dependent organization provides more effective teacher supervision under otherwise identical training conditions. Text-only privileges match the complete configuration on ID150, while adding visual privileges further improves MME-RS by 1.3 percentage points.

Table 5. Contributions of evidence organization and privileged modalities. Accuracy (%).

ConfigurationID150MME-RS
Single-crop OPD72.842.0
Visual-only OPD (organized visual privileges)75.243.3
Text-only OPD (organized textual privileges)76.046.0
GeoOPD76.047.3

W2 (2) / Q3. Comparison with GRPO and practical training value

With the same SFT initialization, E-OPD and GRPO both achieve 76.0% on ID150, while E-OPD improves MME-RS from 46.0% to 47.3% (Table 6). E-OPD directly uses teacher token distributions to provide dense supervision, without requiring a task-specific scalar reward or an LLM-based scorer for open-ended answers. This facilitates a unified training procedure for remote sensing multiple-choice and open-ended questions.

Under our experimental configuration, E-OPD also achieves more than a twofold speedup in training and convergence relative to GRPO (the exact speedup depends on the data and training settings). Its competitive accuracy and practical training efficiency make it a promising post-training approach under limited computational resources.

Table 6. Post-training methods with the same SFT initialization. Accuracy (%).

Training settingID150MME-RS
SFT72.043.3
SFT + GRPO76.046.0
SFT + E-OPD76.047.3

Q3. Gold-answer privileges and self-distillation without privileges

Adding the gold answer provides no further improvement. Evidence and Evidence + Gold both achieve 76.0% on ID150, while their MME-RS results are 47.3% and 46.7%, respectively. We interpret this as evidence that the supporting information already enables useful teacher supervision; directly supplying the answer does not necessarily improve its transfer to the student. We therefore adopt the Evidence configuration without providing gold answers to the teacher.

Regarding self-distillation without privileges, the additional evidence is what gives the teacher an information advantage in our setting. Teacher and student start from the same SFT checkpoint. Without privileges, identical inputs and prefixes produce identical initial predictive distributions under the same forward conditions, leaving no initial corrective signal from additional information. Our method therefore relies on evidence-conditioned supervision to improve visual understanding; distillation alone does not supply this information advantage.

Reviewer_EBfVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoOPD is a post-training framework for remote sensing visual reasoning. It constructs visual and textual teacher privileges in five evidence organizations and distills a frozen, evidence-conditioned teacher into a student that receives only the image and question. The teacher supervises student-generated prefixes, and a divergence gate retains the 50% of response positions with the largest teacher-student discrepancies. Starting from Qwen3-VL-4B, the authors apply SFT and E-OPD using constructed remote sensing data and evaluate on their in-domain ID150 set, MME-RealWorld-RS, and XLRS-Bench.

Critical strengths
  1. The method transfers evidence-guided supervision into tool-free inference, improving the model's ability to identify and use task-relevant visual evidence without additional visual tool calls. The backbone improves from 42.0% to 47.3% on MME-RealWorld-RS and from 42.4% to 59.0% on XLRS-Bench.
Critical weaknesses
  1. The in-domain evaluation provides limited evidence of visual search. CHOICE supplies 4,000 training examples and 75 ID150 questions; the remaining ID150 questions come from LRS-GRO, another training source. Sharing a source does not itself imply leakage, but the paper needs to explain how crops of the same original scene are kept apart across splits. Only 125 questions are scored, and 41 of the 75 synthesized inputs are pre-cropped. These choices limit what ID150 establishes about searching full scenes.

  2. There is no comparison with single-crop OPD using the same backbone and training data, nor an ablation showing the benefit of the five evidence organizations. The existing ablations also omit XLRS-Bench, where the largest gain is reported. It remains unclear how much the proposed design contributes beyond the constructed data and the underlying OPD framework.

  3. The paper should explain what limitations existing OPD methods face when applied to remote sensing images and how the proposed method addresses them. It is also unclear what new ideas are introduced beyond applying OPD to the remote sensing setting. The motivation and novelty are not clear.

Questions for authors
  1. How were training data and ID150 separated at the original-image or scene level? Could different crops of the same scene occur in both sets?
  2. How does GeoOPD compare with single-crop OPD using the same backbone, data, and training budget?
  3. What is the SFT-only result on XLRS-Bench? This would show how much of the reported 16.6-point improvement comes from E-OPD.
3 comments
AuthorsOpenReview ↗

Response to Reviewer EBfV 1/2

Response to Reviewer EBfV 1/2

We thank the reviewer for the constructive suggestions on data separation, controlled comparisons, and training-stage contributions, which have helped us improve the paper. We have added a breakdown of the evaluation inputs, a controlled single-crop comparison, and stage-wise results on XLRS-Bench to address these concerns.

W1 / Q1. Original-image separation and evaluation input composition

LRS-GRO is partitioned at the original-image level before QA synthesis, derived cropping, and data filtering. At the beginning of data construction, we independently reserve 50 original images, internally named Core50, for evaluation. These images and all their derived crops are excluded from training. CHOICE likewise uses separate training and evaluation image sets. We additionally check image hashes across original images and crops to identify exact duplicate files and prevent inadvertent overlap.

Pre-cropping converts extremely large images, such as those of approximately 10,000 × 10,000 pixels, into high-resolution inputs of approximately 2,000 × 2,000 pixels during data preprocessing. The model receives the resulting image directly and does not invoke a cropping tool during inference. It must still locate and interpret fine-grained evidence within this high-resolution input.

Questions constructed from LRS full images tend to emphasize global understanding, since referring unambiguously to tiny objects in extremely large scenes is difficult. Questions constructed from pre-cropped images typically target finer details and require locating small objects within the input. These groups therefore differ in task content and difficulty. Their separate results clarify the evaluation composition; the accuracy differences are not interpreted as the effect of pre-cropping itself.

Table 1. Breakdown of scored ID150 questions by source and input type.

Input typeQuestionsCorrect answersGeoOPD Accuracy (%)
LRS full image161593.8
LRS pre-cropped image342676.5
CHOICE full image755472.0

W2 / Q2. Controlled comparison with single-crop OPD

We compare GeoOPD with single-crop OPD using the same backbone, SFT initialization, training data, and training budget. As shown in Table 2, single-crop OPD achieves 72.8%/42.0% on ID150/MME-RS. Organized visual privileges alone improve these results to 75.2%/43.3%, while the complete GeoOPD reaches 76.0%/47.3%.

Organized visual privileges alone outperform single-crop OPD by 2.4/1.3 percentage points; combining visual and textual privileges yields total gains of 3.2/5.3 percentage points. These results highlight the limitations of a uniform single-crop strategy for remote sensing questions involving spatially dispersed objects. By matching the evidence configuration to each question’s spatial requirements, the proposed organization provides more effective teacher supervision and improves distillation under otherwise identical training conditions.

Table 2. Single-crop and organized-evidence OPD under the same training conditions. Accuracy (%).

ConfigurationID150MME-RS
SFT72.043.3
Single-crop OPD72.842.0
Visual-only OPD (organized visual privileges)75.243.3
GeoOPD76.047.3
AuthorsOpenReview ↗

Response to Reviewer EBfV 2/2

Response to Reviewer EBfV 2/2

W3. Motivation and contribution of spatial evidence organization

Inference-time crop-and-zoom agents expose local details but incur additional latency through repeated tool calls and image re-encoding. Their effectiveness also depends on region selection: inaccurate crops may omit relevant evidence, while isolated local views can lose the broader context required for counting dispersed objects and understanding global layouts. These limitations motivate transferring evidence-guided search into post-training, enabling tool-free inference at deployment.

The remaining challenge is determining what evidence the teacher should receive. Remote sensing questions require different spatial scopes, and a uniform crop struggles to balance local detail with evidence coverage. GeoOPD therefore organizes teacher evidence into five question-dependent forms, transferring the resulting supervision to a student receiving only the input image and question. Table 2 supports the benefit of organized visual privileges over single-crop privileges. Divergence-based token selection further targets high-divergence response positions, while GeoOPD-10K and its data governance pipeline provide the supporting training resources.

Q3. Contributions of SFT and E-OPD on XLRS-Bench

The backbone, SFT, and SFT + E-OPD achieve 42.4%, 52.7%, and 59.0% on XLRS-Bench, respectively (Table 3). Of the total 16.6-percentage-point improvement, SFT contributes 10.3 points, and E-OPD provides a further 6.3 points over SFT. We attribute the substantial SFT gain primarily to high-quality remote sensing QA supervision and domain adaptation, while subsequent evidence distillation further strengthens the model’s ability to locate and reason over fine-grained visual cues.

Table 3. Stage-wise results on XLRS-Bench.

Training stageAccuracy (%)Gain over the preceding stage (pp)
Backbone42.4—
SFT52.7+10.3
SFT + E-OPD59.0+6.3

Table 4 further reports selected fine-grained XLRS-Bench categories. GeoOPD correctly answers 135, 10, and 53 more questions than SFT in object classification, counting with change detection, and spatial relationships, respectively, corresponding to gains of 16.9, 16.7, and 10.6 percentage points. These tasks require local object recognition, object comparison, or instance counting, consistent with our objective of improving fine-grained evidence use through distillation.

Table 4. SFT and GeoOPD on selected fine-grained XLRS-Bench categories.

CategoryQuestionsSFT correct (accuracy)GeoOPD correct (accuracy)Gain (pp)
Object classification798477 (59.8%)612 (76.7%)+16.9
Counting with change detection6013 (21.7%)23 (38.3%)+16.7
Object spatial relationships500128 (25.6%)181 (36.2%)+10.6

We thank the reviewer again for the suggestions that prompted these additional experiments and helped strengthen the evaluation.

I appreciate the clarifications and the additional experiments.

That said, the specific benefit of the five-form evidence organization is still only partly disentangled from textual privileges, particularly since there are no corresponding configuration ablations on XLRS-Bench.

The region interventions are useful for showing background robustness, but without an SFT intervention baseline it remains unclear how much of the behavioral improvement should be attributed to E-OPD.

These remaining uncertainties about the contribution of the proposed design lead me to keep my original rating of 2.

Reviewer_oLqBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on remote sensing visual reasoning, where evidence is fine-grained and sparse and is overwhelmed by extensive background content. The authors argue that “thinking with images” methods based on iterative cropping incur high inference overhead and depend on the accuracy of region selection. To address this, this paper propose GeoOPD, a remote sensing post-training framework centered on Evidence-Privileged On-Policy Distillation (E-OPD). During training, a teacher from the same checkpoint additionally observes visual and textual evidence organized into five spatial forms, and provides token-level distributional supervision along the trajectories generated by the student itself. The framework also introduces divergence gating, which retains only the top-50% of response positions with the largest teacher–student discrepancy for distillation, and is accompanied by a multi-stage verified, evidence-driven data synthesis pipeline (GeoOPD-10K). Experiments show that GeoOPD-4B, built on Qwen3-VL-4B, outperforms its backbone on ID150, MME-RealWorld-RS, and XLRS-Bench, performs favorably against general-purpose and remote-sensing-specific models without any tool calls at inference time, and has lower inference latency than the crop-based ZoomEarth.

Critical strengths
  1. The motivation is clear, and the proposed solution follows naturally from it. The paper identifies a concrete weakness of “thinking with images” in remote sensing: iterative cropping adds inference cost, depends on accurate region selection, and cannot handle questions that need global context.
  2. Privilege design is treated as the core problem in visual on-policy self-distillation, and the five-form evidence organization addresses a real limitation of prior work. Existing visual OPD methods such as Vision-OPD use a single region of interest as the teacher’s privileged context. The paper points out that this cannot serve questions about pairwise spatial relations, counting (dispersed instances are omitted), or global scene understanding.
Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the proposed evidence organization. The core E-OPD framework closely follows existing visual on-policy distillation methods, particularly Vision-OPD, where a teacher conditioned on privileged visual information provides token-level supervision along student-generated trajectories. The main extensions—five task-specific evidence organizations, multimodal teacher privileges, and divergence-based token selection—appear largely incremental. More importantly, the paper lacks controlled comparisons with existing methods such as Vision-OPD, VAD, and OPD-V. The current ablations also fail to isolate the contribution of structured evidence organization from simply providing additional visual crops or textual descriptions.

  2. Insufficient empirical evidence supporting the central claim of internalized visual search. The paper's central claim is that privileged evidence supervision enables the student to internalize visual search capabilities without inference-time tools. However, the experimental evidence primarily consists of downstream VQA accuracy improvements and a limited qualitative attention visualization. These results do not establish whether the model actually learns to localize task-relevant visual evidence more effectively, or merely benefits from additional domain-specific supervision and semantic priors. Furthermore, attention concentration does not necessarily imply causal reliance on the highlighted regions.

Questions for authors
  1. What are the key methodological differences between GeoOPD and existing visual on-policy distillation methods, particularly Vision-OPD, beyond domain-specific evidence construction and divergence-based token selection?

  2. Can the authors provide ablations comparing the proposed five evidence organizations with simpler privilege configurations, such as visual-only, text-only, and unified single-crop privileges, to demonstrate the necessity of structured evidence organization?

  3. Can the authors provide quantitative evidence localization results or causal intervention experiments to demonstrate that GeoOPD genuinely internalizes visual search capabilities rather than merely benefiting from additional domain-specific supervision?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oLqB 1/2

Response to Reviewer oLqB 1/2

We thank the reviewer for recognizing our motivation and for the encouraging comments and constructive suggestions on the methodological contribution, evidence organization, and visual search analysis. We have strengthened the paper with direct evaluations of existing methods, controlled ablations, and region intervention experiments.

W1 / Q1. Contribution and differences from existing visual OPD methods

GeoOPD’s core contribution is spatial evidence organization for remote sensing questions and its transfer to a full-image student. The required evidence has diverse spatial structures: local attributes depend on object details, relationship questions involve multiple objects, counting requires coverage of dispersed instances, and scene understanding relies on global layouts. A uniform cropping strategy struggles to balance evidence coverage and detail preservation. For example, a local crop may omit instances needed for counting, whereas a single crop covering two distant objects introduces substantial background and reduces the advantage of magnifying relevant details. We therefore organize evidence into five forms: single-object, paired-object, regional, instance-set, and global. Visual and textual privileges preserve the information required by each question, while selective distillation targets high-divergence positions along student-generated trajectories.

Vision-OPD emphasizes crop-based privileges, VAD focuses on visual attribution, and OPD-V emphasizes modality-related token selection. GeoOPD investigates how teacher evidence can accommodate the spatial structures of remote sensing questions, with divergence-based selection providing more targeted supervision. We also contribute GeoOPD-10K and its data governance pipeline, providing high-quality data for evidence organization and distillation in remote sensing.

Existing general-purpose visual OPD methods show limited direct transfer to remote sensing. As shown in Table 1, the Qwen3.5-based Vision-OPD-4B and VAD-4B achieve 44.0% and 27.2% on ID150, respectively, falling 27.2 and 44.0 percentage points below their Qwen3.5 backbone’s 71.2%. These results show that gains from general-purpose visual OPD do not automatically transfer to remote sensing, further highlighting the value of training tailored to its evidence requirements.

Table 1. Accuracy (%) on ID150. Vision-OPD and VAD are evaluated using their official checkpoints.

MethodMultiple choiceShort answerOverall
Qwen3.5-4B65.380.071.2
Vision-OPD-4B26.770.044.0
VAD-4B22.734.027.2
GeoOPD-4B72.082.076.0

Q2. Additional ablations on evidence organization and privileged modalities

We have added ablations to examine the contributions of evidence organization and privileged modalities. As shown in Table 2, organized visual privileges alone achieve 75.2%/43.3% on ID150/MME-RS, outperforming single-crop OPD by 2.4/1.3 percentage points. A single crop provides limited teacher advantages for questions requiring dispersed evidence or broader spatial context, motivating evidence organization tailored to each question. The complete GeoOPD, combining organized visual and textual privileges, achieves 76.0%/47.3%, improving over single-crop OPD by 3.2/5.3 percentage points.

Text-only privileges outperform visual-only privileges on both benchmarks. Combining the two retains the best ID150 accuracy of 76.0% and further improves MME-RS from 46.0% to 47.3%, supporting better generalization. These results demonstrate the value of question-specific visual evidence organization and the additional benefit of visual privileges alongside textual evidence.

Table 2. Ablations on privileged modalities and evidence configurations. Accuracy (%).

ConfigurationID150MME-RS
Backbone66.442.0
SFT72.043.3
Single-crop OPD72.842.0
Visual-only OPD (organized visual privileges)75.243.3
Text-only OPD (organized textual privileges)76.046.0
GeoOPD (visual + textual privileges)76.047.3
AuthorsOpenReview ↗

Response to Reviewer oLqB 2/2

Response to Reviewer oLqB 2/2

W2 / Q3. Behavioral validation of evidence use through region interventions

We agree that downstream accuracy and attention heatmaps alone do not establish the mechanism of internalized visual search. The original heatmaps serve as qualitative observations; we have therefore added region intervention experiments to directly examine the model’s use of visual evidence.

Using the supporting-region annotations provided by ID150, we evaluate 101 questions that depend on localized evidence and permit masking. We gray out either the supporting evidence or non-supporting regions; for multi-object questions, all supporting regions are masked jointly. We then measure answer retention relative to each model’s original-image prediction, testing whether predictions respond differently to relevant and irrelevant regions.

As shown in Table 3, GeoOPD retains its original answer in 98.02% of cases under non-supporting-region masking, exceeding the backbone by 14.86 percentage points. Retention under supporting-region masking is 48.51%, meaning that removing supporting evidence changes approximately half of its predictions. The retention gap increases from 41.58 to 49.51 percentage points, while original-image accuracy improves from 66.3% to 75.2%. The clearest improvement is reduced sensitivity to irrelevant background perturbations while retaining responsiveness to changes in supporting evidence. These results support more selective use of relevant visual information and provide complementary behavioral evidence for GeoOPD’s visual search capability.

Table 3. Region intervention results on 101 maskable questions from ID150. Accuracy and retention are reported in %.

ModelOriginal-image accuracySupporting-region masking: retentionNon-supporting-region masking: retentionRetention gap (pp)
Backbone66.341.5883.1641.58
GeoOPD75.248.5198.0249.51

Answer retention is the proportion of masked inputs for which the model preserves its own original-image answer, rather than accuracy after masking. The retention gap is non-supporting-region retention minus supporting-region retention.

Reviewer_oLqBOpenReview ↗

Official Comment by Reviewer oLqB

The response clarifies the motivation for spatial evidence organization but does not establish methodological novelty beyond existing vision-OPD methods. My concerns about methodological novelty remain unresolved, so I keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.