acceptodds
Desk rejected at ICLR 2027

Beyond Fixed Targets: Context-Adaptive Subspace Navigation for Mitigating Object Hallucinations in LVLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) generate fluent responses but often hallucinate objects that are absent from the input image. Existing subspace interventions reduce hallucinations by moving subspace components toward a target that remains fixed across inputs, but this uniform correction can also suppress mentions of real objects, creating a trade-off between hallucination reduction and object recall. To address this challenge, we propose **S**ubspace-**C**onstrained **A**nchor **N**avigation, termed SCAN, which retains a shared subspace but replaces the fixed target with an anchor adapted to each context. Specifically, SCAN separates the hidden state into a component within the hallucination-associated subspace and a residual orthogonal to it. Using only this residual as context, a lightweight Navigator predicts an anchor and per-direction strengths that control how far the subspace component moves toward the anchor, leaving the residual unchanged. With the LVLM backbone and subspace basis frozen, the Navigator is trained on preference pairs for captioning and object existence judgments to discourage hallucinated mentions while preserving object recall. Extensive experiments across multiple tasks and LVLM backbones demonstrate that SCAN consistently reduces object hallucinations and achieves a favorable hallucination–recall trade-off while preserving decoding efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
object hallucination, large vision-language models, subspace intervention, inference-time intervention, context-adaptive steering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a3GxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SCAN for mitigating object hallucination in LVLMs. It retains a frozen, backbone-specific hallucination-associated subspace and trains a lightweight Navigator to predict context-adaptive anchors and direction-wise weights from the orthogonal residual. A closed-form proximal update modifies the subspace coordinates while preserving the residual at the current intervention position. The squared-error analysis characterizes the approximation gap of fixed targets when the conditional mean of the desired anchor varies with context. Experiments report hallucination reductions, hallucination–recall trade-offs, and low decoding overhead in the measured configuration.

Critical strengths
  1. The paper provides useful empirical evidence for context-adaptive navigation within a shared subspace. Table 5 examines fixed versus adaptive anchors, uniform versus direction-wise strengths, and alternative basis constructions. Appendix G reports hallucination and recall jointly across intervention strengths. At nearly matched recall, SCAN achieves lower CHAIRs and CHAIRi than the uniform zero-target control. These comparisons support an improved hallucination–recall trade-off within the evaluated setting.

  2. Extensive experiments across models and tasks demonstrate that SCAN consistently reduces object hallucinations and achieves a favorable hallucination–recall trade-off. Additional efficiency measurements show that SCAN retains decoding throughput close to that of the original backbone. The results support selective intervention and low overhead under the reported protocols.

Critical weaknesses
  1. Comparative coverage is incomplete for closely related subspace-intervention methods. Nullu and HulluEdit appear in the 64-token CHAIR comparison, but not in the POPE or AMBER tables. The Qwen2.5-VL and Qwen3-VL results compare SCAN with Vanilla only. Thus, the experiments provide different levels of comparative evidence across backbones and benchmarks. Table 1 explains that some omitted configurations lack specified hyperparameters; this is a relevant constraint, but the comparative conclusions should remain tied to the configurations actually evaluated.

  2. The reporting of per-backbone adaptation and generalization scope could be more complete. The basis and preference pools are constructed from COCO images and annotations. AMBER provides broader benchmark evidence, so cross-dataset generalization should not be described as entirely untested. However, its aggregate results do not separately characterize non-COCO image sources or categories outside the training annotation inventory. Appendix D specifies backbone-specific basis construction and dimension-dependent Navigator architectures, while the detailed optimization settings and approximately 50-minute cost are primarily documented for LLaVA-1.5 7B. A concise per-backbone configuration and cost summary would make the practical adaptation requirements clearer.

Questions for authors
  1. Given the backbone-specific basis construction in Appendix D.1, could you provide a per-backbone summary of the intervention layer, rank, inference strength, Navigator training procedure, and offline cost? Please distinguish reused image pools from regenerated response pairs and clarify how the reported defaults were selected.

  2. Could you characterize the scope of generalization more explicitly while recognizing the AMBER results already provided? If available, performance separated by image source or categories outside the training inventory would help. Otherwise, please distinguish demonstrated cross-benchmark performance from broader domain robustness without implying that domain-specific rebuilding is necessarily required.

  3. Could you extend the Nullu and HulluEdit comparisons to POPE or AMBER where feasible, or explain the implementation constraints? Table 1 already notes unspecified hyperparameters for omitted configurations such as ASCD on Qwen-VL-Chat; please clarify whether a justified configuration can be evaluated and keep the comparative claims appropriately scoped if it cannot.

4 comments

Response to Reviewer a3Gx (Part 1/2): Comprehensive Baseline Comparisons

Thank you for your constructive comments. We have addressed your main concerns through a per-backbone configuration and training-cost summary, a category-level AMBER analysis, and expanded baseline comparisons on POPE, AMBER, and Qwen-VL-Chat. The revised manuscript has been uploaded to the submission system, with changes addressing your comments highlighted in green.

[W1 & Q3-1] Extended Comparisons with Nullu and HulluEdit

Following the reviewer's suggestion, we have extended the Nullu and HulluEdit evaluations to POPE and AMBER using their official implementations under the same prompts, decoding settings, and evaluation protocols as SCAN on LLaVA-1.5 7B.

Table R1: POPE comparison (Accuracy/F1, %)

MethodRandomPopularAdversarial
Vanilla85.1/83.783.7/82.480.9/80.0
Nullu84.4/82.783.2/81.680.6/79.3
HulluEdit82.0/80.381.2/79.878.1/77.2
SCAN89.2/88.687.1/86.783.8/83.7

Table R2: AMBER comparison (official metrics, %)

MethodCHAIR ↓Cover ↑Hal ↓Cog ↓
Vanilla8.351.837.84.5
Nullu7.448.532.43.4
HulluEdit4.143.725.12.2
SCAN5.650.423.91.6

SCAN consistently outperforms Nullu and HulluEdit across all POPE splits, whereas both baselines fall below Vanilla. On AMBER, SCAN outperforms Nullu across all four metrics. Although HulluEdit achieves lower CHAIR (4.1 vs. 5.6), SCAN preserves substantially higher object coverage (50.4 vs. 43.7) while achieving lower Hal and Cog, demonstrating a more favorable balance between hallucination mitigation and content preservation.

[W1 & Q3-2] Completed ASCD and VTI Comparisons on Qwen-VL-Chat

We have completed the previously unevaluated ASCD and VTI configurations on Qwen-VL-Chat using their official implementations. The previously cited VTI results have been replaced with our own evaluations, ensuring that all entries in Table 1 follow the same protocol.

Table R3: Qwen-VL-Chat comparison on POPE and CHAIR (%).

MethodRandom Acc/F1Popular Acc/F1Adversarial Acc/F1CHAIR_s / CHAIR_i ↓
Vanilla84.5/82.383.4/81.480.5/78.938.6/11.9
VTI83.5/81.282.9/80.780.3/78.629.8/8.6
ASCD86.1/84.585.0/83.582.6/81.327.2/9.2
SCAN88.4/87.686.6/86.083.3/83.124.4/7.5

SCAN achieves the highest POPE accuracy/F1 across all three splits and the lowest CHAIR_s/CHAIR_i, further supporting its effectiveness under consistent evaluation settings. For Qwen2.5-VL and Qwen3-VL, we clarify that the reported results demonstrate improvements over Vanilla rather than comparisons with untested baselines.

The updated AMBER and Qwen-VL-Chat comparisons are provided in Table 3 and Table 1 (page 8), respectively.

Response to Reviewer a3Gx (Part 2/2): Per-Backbone Adaptation and Generalization

[W2 & Q1] Per-Backbone Configuration and Training Cost

Following the reviewer's suggestion, we added a comprehensive per-backbone summary of intervention configurations, response-pair reuse, and training costs.

Table R4: Per-backbone configurations and Navigator training costs.

BackboneLayerPreference PairsResponse SourceαTime (min)
LLaVA-1.5 7B315,334Own0.1049
LLaVA-1.5 13B395,334LLaVA-1.5 7B0.10269
LLaVA-NeXT-7B315,334Own0.10146
Qwen-VL-Chat275,334Own0.0558
Qwen2.5-VL-7B285,334Own0.0544
Qwen3-VL-8B285,334Qwen2.5-VL0.0535

(1) Data Reuse and Training Cost. All backbones use the same MS COCO image pools, with disjoint images for basis construction (3,072 pairs) and Navigator preference learning (5,334 pairs). Each backbone constructs its own rank-128 basis and trains a separate Navigator for 500 steps. Four backbones generate their own response pairs, while LLaVA-1.5 13B and Qwen3-VL reuse those from LLaVA-1.5 7B and Qwen2.5-VL, respectively. Reusing response pairs still requires backbone-specific hidden-state extraction, basis construction, and Navigator training. The reported times cover complete Navigator training runs on one RTX 4090, excluding offline response generation, basis construction, and benchmark evaluation. LLaVA-1.5 13B additionally requires CPU offloading.

(2) Default Configuration Selection. The 500-step checkpoint achieves the lowest preference loss on the independent 533-pair development split of LLaVA-1.5 7B, and this training budget is adopted across backbones. Rank 128 and layer 31 are selected through CHAIR sensitivity analysis on LLaVA-1.5 7B. Other backbones retain rank 128 and use a layer in the last quarter of their decoder blocks without further layer tuning. The inference strength is selected at the knee of the CHAIR hallucination–recall curve (α = 0.10 for LLaVA and 0.05 for Qwen) and then fixed for the reported benchmark evaluations.

Full configurations and costs are provided in Appendix G.3 (Table 15, page 32), with selection details in Appendix G.1–G.2 (pages 30–31).

[W2 & Q2] Scope of Cross-Benchmark Generalization

We thank the reviewer for this insightful suggestion. SCAN demonstrates cross-benchmark transfer to AMBER without additional adaptation, reducing CHAIR from 8.3 to 5.6 and Hal from 37.8 to 23.9, with only a modest decrease in Cover (51.8 → 50.4; Table 3).

Category-Level Generalization Analysis. Since AMBER does not identify the source of each image, we further analyzed generalization by object-category coverage. Our preference data covers the 80 MS COCO categories, and 77 of AMBER's 340 labels map to them under the CHAIR synonym list. We classify AMBER object mentions into in-inventory and out-of-inventory groups and evaluate their hallucination counts and object coverage.

Table R5: Category-level hallucination and coverage analysis on AMBER

MetricVanillaSCANChange
In-inventory hallucinations514289−44%
Out-of-inventory hallucinations193191−1%
Cover (in-inventory)86.083.5−2.5
Cover (out-of-inventory)37.136.3−0.8

Note: Hallucination changes are relative percentages; Cover changes are percentage points.

SCAN reduces hallucinations by 44% on in-inventory categories, which account for 73% of Vanilla's hallucinations. In contrast, out-of-inventory hallucinations, mainly scene-related objects such as sun, beach, and sky, remain largely unchanged, with only a 0.8-point decrease in coverage. These findings demonstrate cross-benchmark effectiveness without additional adaptation, while clarifying that the improvements primarily concentrate on categories represented in the preference data. Broader cross-domain robustness remains to be evaluated.

The analysis is provided in Appendix L.2 (Table 21, page 37), with the generalization scope clarified in Section 4.2 (page 9).

Based on the above clarifications, we hope that our responses have adequately addressed your concerns and warrant a more favorable assessment.

Thank you for the additional experiments and clarifications.

  1. The new Nullu and HulluEdit results on POPE and AMBER, together with the ASCD and VTI results on Qwen-VL-Chat, address my main concern about baseline coverage.

  2. The revised appendix clarifies the intervention settings, response-pair reuse, training procedure, and Navigator training cost for each backbone. It also makes clear which offline stages are excluded from the reported timings.

  3. The AMBER category breakdown answers my question about the scope of transfer.

These additions address my main concerns and strengthen the evidence for SCAN’s hallucination–recall trade-off. I change my rating from 2 (weak rejection) to 3 (weak acceptance).

Thank you for your positive feedback and for raising your score. We are glad our responses have addressed your concerns, and your suggestions have truly helped us make the paper more complete and convincing. We greatly appreciate your time and thoughtful review!

Reviewer_fw6vOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCAN, a context-adaptive subspace intervention method for mitigating object hallucinations in LVLMs. The authors identify fixed-target mismatch as a limitation of existing subspace interventions and propose replacing fixed correction targets with context-dependent anchors. SCAN decomposes hidden states into a hallucination-associated subspace and an orthogonal residual, then uses a lightweight Navigator to predict adaptive anchors and direction-wise intervention strengths. Experiments across multiple LVLMs demonstrate hallucination reductions and improvements in the hallucination–recall trade-off while maintaining decoding efficiency.

Critical strengths
  • An interesting context-adaptive subspace intervention framework. SCAN replaces fixed correction targets with context-dependent anchors while preserving the orthogonal residual. This provides a flexible approach to hallucination mitigation and a meaningful extension of conventional subspace interventions.

  • Useful evaluation of the hallucination–recall trade-off. The paper evaluates SCAN across multiple LVLM architectures and benchmarks, considering both hallucination reduction and object coverage. The matched-recall comparisons and component ablations provide useful evidence for the effectiveness of the proposed approach.

Critical weaknesses
  • Limited empirical evidence for fixed-target mismatch. While the paper theoretically motivates context-dependent correction targets, it does not directly demonstrate that different image and textual contexts require distinct optimal targets. The existing experiments mainly establish improvements over the tested fixed-target baselines, leaving the central fixed-target mismatch hypothesis only indirectly supported.

  • Missing comparisons with related representation-space intervention methods. The evaluation lacks direct comparisons with relevant methods such as REVIS, which also explores selective representation-space interventions for hallucination mitigation. Such comparisons are important for establishing the distinctive advantages of SCAN over existing approaches.

Questions for authors
  • Empirical Validation of Fixed-Target Mismatch. Could the authors provide a PCA visualization showing how correction targets vary across different image and textual contexts? Such a visualization would offer more intuitive evidence for the proposed fixed-target mismatch hypothesis.

  • Comparison with Existing Intervention Methods. Could the authors compare SCAN with relevant methods under matched evaluation settings, particularly in terms of hallucination reduction, object recall, and inference efficiency?

5 comments

Thank you for your insightful comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with revisions addressing your comments highlighted in red.

[W1 & Q1] Direct Empirical Evidence for Fixed-Target Mismatch

We conducted a controlled analysis independently of the learned Navigator by directly optimizing context-specific and shared correction targets under the same grounding objective and edit-size penalty, with the backbone, intervention layer, and subspace basis frozen (Appendix I.1).

(1) PCA Visualization (Figure 2a–b, page 4). Using held-out images excluded from basis construction and Navigator training, we examine two settings:

Image Context: With a fixed text prefix, targets optimized for 120 images (60 dog-present and 60 dog-absent) exhibit image-dependent variation.

Text Context: For 77 images with three different prefixes each (231 contexts), changing only the preceding object mention (chair, cup, or person) produces systematic target shifts, yielding 94% prefix-identification accuracy (33% chance level).

(2) Grounding-Gain Analysis (Figure 2c). As shown in Table R1, context-specific targets substantially outperform shared and swapped targets. Shared targets recover only 20%/43% of context-specific gains. Even swapping targets between images with the same dog label and similar edit sizes (0.27 vs. 0.26) recovers only 21%, highlighting the importance of target–context alignment rather than intervention magnitude alone.

Table R1: Grounding gains of directly optimized correction targets.

TargetImage Varies ↑Text Varies ↑
Context-specific5.474.19
Context-swapped1.14 (21%)3.28 (78%)
Shared fixed1.10 (20%)1.82 (43%)
Shared (cross-validated)1.06 (19%)1.75 (42%)
Zero target−0.06−0.10

(3) Learned Navigator Validation (Figure 6, Appendix I.3). The Navigator-predicted anchors also capture contextual variation, achieving 0.75 AUC for object-presence discrimination. Replacing adaptive anchors with mean or context-shuffled anchors reduces hallucination selectivity from 1.20 to 0.73/0.77, respectively.

These results provide empirical support for context-dependent target variation and the fixed-target mismatch mechanism underlying Proposition 1. Full experimental details are provided in Appendix I (pages 33–34).

[W2 & Q2] Comparison with Existing Representation-Space Interventions

We have added comparisons with REVIS and RUDDER, two recent training-free representation interventions from ICML 2026. Unlike their visual-direction steering mechanisms, SCAN learns residual-conditioned anchors and direction-wise strengths within a fixed hallucination-associated subspace.

(1) Matched-Recall Comparison. We integrated their official implementations into the same framework and evaluated all methods on LLaVA-1.5 7B, LLaVA-NeXT-7B, and Qwen-VL-Chat, using identical CHAIR protocols (500 MS COCO images, greedy decoding, 64 tokens). To control for object omission, we sweep intervention strengths and compare hallucination rates at matched object recall, with interpolation between measured operating points.

Table R2: CHAIR_s reduction (%) relative to Vanilla at matched recall. Higher is better.

BackboneRecall (%)SCANREVISRUDDER
LLaVA-1.5 7B69189−3
64512412
LLaVA-NeXT-7B48463210
47543416
Qwen-VL-Chat71.5161314
7038−2–

Note: Representative operating points are shown; all 12 comparisons appear in Appendix K.2. “–” indicates a recall level outside the method's evaluated range.

SCAN achieves the largest CHAIR_s reduction across all 12 matched-recall operating points among the available methods, and the lowest CHAIR_i at 11 of them. At 64% recall on LLaVA-1.5 7B, SCAN reduces CHAIR_s by 51%, compared with 24%/12% for REVIS/RUDDER. The advantage also extends to LLaVA-NeXT-7B and Qwen-VL-Chat.

(2) Inference Efficiency. Under identical timing protocols, SCAN, REVIS, and RUDDER retain 99.4%, 99.9%, and 87.0% of Vanilla throughput, respectively, with nearly identical memory usage. Thus, SCAN achieves stronger hallucination mitigation while maintaining near-Vanilla inference efficiency. The 99.4% measurement differs slightly from Table 6 (98.4%) because of run-to-run GPU throughput variation; both remain within 2% of Vanilla.

Detailed results are provided in Appendix K.2 (Figure 8 and Table 18, page 36) and Appendix K.3 (Table 19, page 37).

Based on the above clarifications, we hope that our responses have adequately addressed your concerns and warrant a more favorable assessment.

Thank you for the detailed response and for adding the new experiments. I appreciate the additional empirical analysis of fixed-target mismatch and the new comparisons with related intervention methods.

I have one remaining question regarding the relationship between SCAN and MGAP (ICML 2026). The two methods appear closely related at the level of their subspace intervention mechanisms. MGAP constructs an SVD-based subspace, decomposes the hidden representation into projected and orthogonal components, and applies context-adaptive attenuation to the projected component while preserving the orthogonal residual. SCAN similarly decomposes the hidden state into subspace coordinates cc and an orthogonal residual zz, modifies only the subspace component, and preserves zz, with the main distinction that the Navigator predicts a context-dependent non-zero anchor aa together with direction-wise strengths.

More specifically, MGAP's update can be written as

= z + B(1-\beta)c,$$ whereas SCAN applies $$h' = h - B\left[\gamma \odot (c-a)\right].$$ Therefore, under the same subspace, setting $a=0$ and tying the direction-wise coefficients as $\gamma_1=\cdots=\gamma_k=\beta$ appears to reduce the SCAN update to the MGAP-style update. Could the authors discuss this relationship more explicitly and, if feasible, provide a matched comparison with MGAP? In particular, I would be interested in understanding the additional benefit of learning context-dependent non-zero anchors beyond existing context-adaptive subspace attenuation.

Thank you for the follow-up and for pointing us to MGAP. We agree with your reduction: under the same subspace, the MGAP update is the zero-anchor, tied-strength case of the SCAN update. MGAP adapts how much to attenuate but always moves toward zero, whereas SCAN also learns where to move through a context-dependent anchor. We therefore address your question through a matched comparison with MGAP and an analysis of the context-dependent anchor. The revised manuscript has been uploaded to the submission system, with revisions addressing your comments highlighted in red.

[Q1] Matched Comparison with MGAP

We have added a matched comparison with MGAP on LLaVA-1.5 7B, under the same 64-token CHAIR protocol as SCAN.

(1) Official MGAP (Appendix K.1 and Table 19, pages 35–38). We integrated the official MGAP code into our framework and verified token-level agreement with its outputs on 20 images. With its official code and configuration, MGAP does not mitigate hallucination, because its calibration yields a degenerate subspace when every input shares the captioning prompt. We therefore reconstructed this calibration following the description in the MGAP paper.

Table R3: Hallucination–recall trade-off of MGAP under our protocol.

MGAP αRecall ↑CHAIR_s ↓CHAIR_i ↓
0 (Vanilla)71.120.47.08
0.75 (official code)71.122.07.04
0.5070.520.86.59
0.7570.121.07.23
1.0070.020.26.94
1.2567.322.27.57
1.5059.918.87.31
2.0033.011.28.27

Note: The official-code row uses the original calibration, and the other rows use the reconstructed calibration.

As shown in Table R3, the reconstructed MGAP keeps CHAIR_s close to Vanilla while recall stays near 70. Larger strengths reduce recall sharply while CHAIR_i rises above Vanilla, so no lower-hallucination point is reached between recall 69 and 64.

(2) Same Subspace and Layer as SCAN (Appendix K.2, Figure 9 and Tables 19–20, pages 37–38). Because this subspace differs from that of SCAN in both its directions and its layer, the weak result could stem from the subspace rather than the update rule. Replacing it with VCD-difference directions keeps recall above 70 at every strength without a consistent reduction in CHAIR. We then instantiated your reduced form inside SCAN, with the same subspace B and layer, a = 0, and a single strength β given by the MGAP gate, so that the update rule is the only difference. Values at matched recall are interpolated between measured operating points.

Table R4: Hallucination–recall comparison between SCAN and MGAP in the same subspace and layer.

Metric (%) ↓MethodRecall ≈69≈67≈65≈64
CHAIR_sMGAP (SCAN subspace)19.217.714.314.6
SCAN16.714.212.010.0
CHAIR_iMGAP (SCAN subspace)6.516.496.245.96
SCAN6.115.264.554.02

SCAN achieves lower hallucination at every matched recall, by 2.3 to 4.6 points in CHAIR_s and 0.40 to 1.94 points in CHAIR_i.

[Q2] Benefit of Context-Dependent Non-Zero Anchors

Since the two methods in Table R4 share the subspace and layer, the remaining differences are the anchor and the direction-wise strengths. CHAIR_i is the fraction of mentioned objects that are hallucinated, and at matched recall both methods keep the same share of ground-truth objects. The lower CHAIR_i of SCAN therefore means that it removes more hallucinated objects for the same loss of grounded ones.

This advantage grows with the editing strength. From recall 69 to 64, MGAP in the SCAN subspace lowers CHAIR_i only from 6.51 to 5.96, whereas SCAN lowers it from 6.11 to 4.02, so the gap widens from 0.40 to 1.94. The reason lies in where each update leads. As β grows, the zero target pulls every subspace coordinate toward the origin, and in the limit the whole subspace component is erased, grounded content included. A context-dependent anchor avoids this, so a stronger edit removes more hallucination without discarding the grounded component.

These results show that adaptive attenuation decides how strongly to correct, while the non-zero anchor decides where the corrected state should lie, a role that strength alone cannot play.

Based on the above clarifications, we hope that our responses have adequately addressed your remaining question and warrant a more favorable assessment.

Reviewer_KrhzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCAN, a lightweight learned intervention for mitigating object hallucinations in LVLMs. SCAN decomposes hidden states into a fixed hallucination-associated subspace and its orthogonal residual. A Navigator uses the residual to predict context-dependent anchors and direction-wise strengths, updating only the subspace coordinates. The Navigator is trained on captioning and object-existence preference pairs while the backbone and basis remain frozen. Experiments across six LVLMs show reduced hallucination, an improved hallucination–recall trade-off, and near-baseline decoding throughput.

Critical strengths
  1. The distinction between sharing a correction subspace and sharing a correction target is meaningful. SCAN implements this idea through a simple closed-form update, while residual conditioning restricts the predictor’s dependence on the coordinates being edited. Fixed-anchor, subspace-basis, and Navigator-input ablations provide useful evidence for these design choices.

  2. The evaluation considers hallucination, object coverage, and computational efficiency together. At nearly matched recall, SCAN outperforms zero-target suppression on both CHAIR metrics (Table 14). Improvements extend across multiple backbones, and SCAN retains 98.4% of vanilla decoding throughput on LLaVA-1.5 7B. These results support its practical value as a lightweight intervention.

Critical weaknesses
  1. The advantage over other learned representation interventions remains unclear. SCAN receives task-specific preference supervision, whereas the principal baselines do not receive equivalent Navigator training. The fixed-anchor controls are informative, but a parameter-matched ReFT/LoReFT-style intervention trained on the same preference pairs would better distinguish the value of residual-conditioned navigation from that of additional supervision.

  2. Content preservation is only partially established. At the default setting, CHAIR object recall decreases from 71.1 to 64.0, and AMBER coverage also declines. The matched-recall analysis supports a better trade-off than zero-target suppression, but does not establish preservation of general visual competence. Evaluation focuses on object existence and captioning, leaving potential effects on attributes, spatial reasoning, and broader VQA insufficiently examined.

Questions for authors
  1. How does SCAN compare with a parameter-matched learned representation intervention trained using the same preference data and objective? This comparison would help establish the specific benefit of the proposed navigation structure.

  2. Do the improvements persist when object recall is constrained close to vanilla? Can the authors provide evidence that the intervention preserves performance on visual tasks beyond object existence and captioning?

  3. Appendix F explicitly excludes CHAIR evaluation images from the training-related pools. Are POPE and AMBER evaluation images also excluded? Please provide the corresponding overlap checks and clarify how checkpoints and inference strengths are selected without using benchmark test results.

3 comments

Thank you for your constructive comments. We have carefully addressed your main concerns with additional experiments and clarifications. The revised manuscript has been uploaded to the submission system, with revisions addressing your comments highlighted in blue.

[W1 & Q1] Matched-Supervision Comparison with LoReFT

Table R1: Hallucination–recall comparison between SCAN and LoReFT under identical preference supervision.

Metric (%) ↓MethodRecall ≈70≈67≈65≈64≈61
CHAIR_sLoReFT19.614.414.814.613.4
SCAN19.214.012.810.08.8
CHAIR_iLoReFT6.585.455.995.786.82
SCAN6.595.104.764.024.11

Note: Both methods share identical training data, objectives, regularization, intervention layers, and training steps, with only a 0.016% difference in trainable parameters.

As shown in Table R1, we evaluated seven LoReFT learning-rate configurations and matched five operating points to the closest measured SCAN recall levels. Vanilla achieves 71.1% recall, with CHAIR_s/CHAIR_i of 20.40/7.08. SCAN consistently achieves a more favorable hallucination–recall trade-off at these matched operating points, with larger gains at lower recall levels. The complete comparison is provided in Appendix J (Figure 7 and Table 17, page 35).

[W2 & Q2-1] Hallucination Reduction at Near-Vanilla Recall

To verify whether SCAN remains effective when object recall is close to Vanilla, we added three low-strength operating points (α = 0.025, 0.030, and 0.035). At α = 0.035, SCAN reduces CHAIR_s/CHAIR_i by 18.6%/12.7% relative to Vanilla, while object recall decreases by only 1.9 points (71.1 → 69.2). These results demonstrate that SCAN can reduce object hallucinations while maintaining recall close to Vanilla.

Table R2: Hallucination–recall trade-off under low intervention strengths.

SCAN αRecall ↑CHAIR_s ↓CHAIR_i ↓
0 (Vanilla)71.120.47.08
0.02070.018.86.88
0.02569.619.66.93
0.03069.117.66.46
0.03569.216.66.18
0.04068.315.25.55

[W2 & Q2-2] Preservation of Broader Visual Abilities

To evaluate whether SCAN affects visual capabilities beyond object existence and captioning, we conducted additional evaluations on MMBench (CircularEval) and AMBER attribute questions, covering attribute recognition, spatial understanding, and relation reasoning.

Table R3: General visual understanding performance of Vanilla and SCAN (accuracy%).

MethodMMBench OverallAttributeSpatialRelationAMBER Attribute
Vanilla65.274.420.053.079.5
SCAN65.073.920.053.979.4
Δ−0.2−0.50.0+0.9−0.1

As shown in Table R3, SCAN maintains comparable performance across the evaluated visual abilities, with only a 0.2-point decrease in overall MMBench accuracy (65.2% → 65.0%) and changes within 0.9 percentage points across the reported ability-specific metrics. These results indicate that SCAN mitigates object hallucinations without substantially compromising the evaluated broader visual capabilities. Detailed results are provided in Appendix L.1 (Table 20, page 37).

[Q3-1] Data Overlap Verification

We verified zero overlap between CHAIR/POPE evaluation images and the basis-construction pool (3,072 pairs), all three Navigator splits (5,334 pairs), and the mechanism-analysis set using COCO image IDs. AMBER images originate from the unannotated COCO 2014 test split and Unsplash, whereas our data pools use COCO images with object annotations, ensuring separation by data source. Details are provided in Appendix F (page 29).

[Q3-2] Checkpoint and Intervention Strength Selection

The Navigator checkpoint was selected using the independent 533-pair development split, where the preference loss reached its minimum at step 500. For intervention strength, α = 0.10 was selected based on the CHAIR hallucination–recall trade-off. Increasing α to 0.15 reduces recall by another 3.30 percentage points, while CHAIR_s improves by only 1.20 points and CHAIR_i slightly worsens (4.02 → 4.11).

We clarify that CHAIR results informed the selection of α, whereas neither POPE nor AMBER was used for hyperparameter selection. Both benchmarks were evaluated using the fixed configuration. Furthermore, our hallucination–recall findings are supported by comparisons across multiple intervention strengths and matched-recall operating points, rather than relying solely on α = 0.10. Details are provided in Appendix G.1 (page 30) and Table 14 (page 31).

Based on the above clarifications, we hope that our responses have adequately addressed your concerns and warrant a more favorable assessment.

The authors’ response has addressed most of my concerns, I will maintain my current rating.

Thank you for your reply. We are glad our responses have addressed most of your concerns, and your comments have helped us strengthen both the analysis and the presentation of the paper. Thank you again for your time and careful review!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

Thank you for your time and constructive feedback. We summarize the main updates below.

  • Baseline comparability. To test whether SCAN's gains persist under comparable conditions, we compared it with LoReFT trained on the same preference data and objective, and with REVIS, RUDDER, and MGAP at matched object recall. SCAN achieved lower CHAIR_s than each method at every matched recall. We also extended Nullu and HulluEdit to POPE and AMBER.
  • Context-dependent targets. To verify that correction targets vary with context, we directly optimized targets for individual images and text prefixes with the backbone and basis frozen. Context-specific targets yielded much larger grounding gains than shared or zero targets, and the Navigator-predicted anchors also captured this variation.
  • Hallucination–recall trade-off and general abilities. To assess whether SCAN reduces hallucination at the cost of content, we added low-strength operating points and evaluated MMBench and AMBER attribute questions. SCAN reduced hallucination with only a small drop in recall, and accuracy on these abilities changed by at most 0.9 points.
  • Configuration and generalization scope. We reported the layer, strength, data reuse, and training time of each backbone, and analyzed AMBER hallucinations by category coverage. SCAN transferred to AMBER without adaptation, with its reduction concentrated on categories covered by the preference data. We also clarified data overlap checks and checkpoint selection.

We hope this summary is helpful in your assessment of our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.