acceptodds
Under review as a conference paper at ICLR 2027

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

Abstract

Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have little leverage at some layers and may damage a correct answer even when it changes the output. We investigate where visual interventions retain leverage and when they improve predictions. First, we apply the same visual-residual edit at successive decoder layers, matching its RMS strength across depth. Across six checkpoints from three model families, we observe a persistent drop in intervention effect and define its operational onset for this edit as the Evidence Anchoring Boundary (EAB). With boundaries fixed on calibration queries, held-out effects on disjoint evaluation queries before EAB are – larger than those after it. Comparison with intermediate readouts shows that the edit can lose leverage before answer preferences stabilize. Second, we compare counterfactual subtraction across presence, attribute, and relation questions. Subtraction can correct a misleading preference, leave a decision unchanged, or remove a preference supporting the correct answer. Based on these observations, we develop calibrated selective routing, which preserves the original prediction unless calibration supports an intervention for the question type. It improves full-set attribute accuracy by – percentage points across three checkpoints, while universal subtraction is neutral or harmful.

Primary area:
interpretability and explainable AI
Keywords:
Multimodal Large Language Models (MLLMs), Multimodal Reasoning, Visual Grounding, Hallucination Mitigation, Mechanistic Interpretability, Inference-Time Intervention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UymdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where visual interventions retain influence on multimodal predictions and when counterfactual subtraction improves accuracy. An equal-RMS layer sweep across six checkpoints identifies an Evidence Anchoring Boundary (EAB), where the tested intervention gets persistently weak, potentially before answer-readout stabilization. The authors also examine corrective and harmful subtraction across question types and propose calibrated selective routing, which improves full-set attribute accuracy by 1.56–1.81 percentage points across three checkpoints.

Critical strengths
  • The layer-wise analysis includes useful controls and held-out validation. Matching intervention RMS across depth helps separate perturbation level from layer position. Selecting the boundary on calibration queries and evaluating it on disjoint queries strengthens the evidence for the early-versus-late effect contrast across six checkpoints.
  • The question-type analysis shows why subtraction should be applied selectively. The paper reports both corrective and destructive outcomes. The paper evaluates full-set attribute routing through image-disjoint cross-validation, and the reported transfer failures appropriately limit the scope of the correction methods.
Critical weaknesses
  • The definition and robustness of EAB are not sufficiently established. Section 3.3 and Appendix A.2 describe deleting the "visual residual contribution," but do not specify the exact tensor operation or insertion point. Thus, it is unclear which pathway is edited and whether image information previously propagated into text-token representations is preserved. This matters for interpreting the measured influence and its separation from ACP. Moreover, the authors select EAB using a 10% threshold on 48 calibration queries, without analyzing its location under nearby thresholds or resampled calibration sets. Appendix C.2 explicitly distinguishes the held-out effect-ratio intervals from uncertainty in the selected layer. Although the paper appropriately treats EAB as an operational boundary, the large pre/post contrast does not establish that its location is stable. These gaps limit the reproducibility and interpretation of the central finding.
  • The routing gains are not sufficiently separated from simpler calibration of subtraction strength. Section 5.3 and Appendix E.1 list counterfactual score spread, ranking disagreement, and sign consistency as routing signals, but do not specify the concrete rule mapping them to (\beta), including its thresholds and selection procedure. The main comparison uses preservation, full subtraction, and routing, without a constant (\beta) optimized under the same calibration protocol. The gains could therefore partly reflect stronger subtraction rather than the proposed selection rule. Table 3 also states that the stability safeguard’s separate contribution is not quantified. Without these details and controls, I find the additional contribution of the router insufficiently established to support acceptance.
Questions for authors
  • What are the ACP layer indices for the six checkpoints under the held-out tuned lens?
  • Are the equivalent-prompt and ablation probes used to assess sign consistency run for each evaluation claim set, or are these probes used only during calibration?
6 comments
AuthorsOpenReview ↗

EAB Specification, Robustness, and ACP

Thank you for identifying the missing operator specification, boundary-location robustness check, and router baseline. We agree these details are necessary to interpret the reported findings. We answer the two weaknesses and the specific questions directly.

W1. Which residual states are edited, how stable is EAB, and where is ACP?

The operator edits only visual-token residual states at decoder-block inputs; it does not remove visual information already propagated to text-token states. Let HjH_j be the residual input to decoder block jj before normalization/self-attention, and VV its visual-token positions. At every block j≥ℓj\geq\ell:

δj=mean⁡v∈V(Hj)−Hj,V\delta_j=\operatorname{mean}_{v\in V}(H_j)-H_{j,V}

Hj,V′=Hj,V+rδjmax⁡(RMS⁡(δj),10−8).H'_{j,V}=H_{j,V}+r\frac{\delta_j}{\max(\operatorname{RMS}(\delta_j),10^{-8})}.

The checkpoint-specific target rr is the median unnormalized edit RMS on 48 calibration images and is frozen for evaluation. Each injection is independently RMS-normalized; suffix length varies with ℓ\ell. Text-token states at insertion are not directly overwritten. ACP is evaluated separately on clean answer-position residual readouts, not through this intervention. We agree that “deleting visual residual contributions” was imprecise: EAB is an operator-specific sensitivity boundary, not evidence that all visual information disappears.

EAB's exact layer is threshold-dependent, but its separation from the answer-readout stabilization guard persists under the tested settings. All six checkpoints use 120 distinct POPE COCO-adversarial images (48 calibration, 72 held out). We reselected EAB at 5%, 10%, 15%, and 20% cutoffs and used 2,000 calibration-image bootstrap draws to estimate boundary-location uncertainty separately from held-out effect uncertainty.

CheckpointEAB at 5/10/15/20%10% EAB location 95% intervalTuned rank layer / consensus ACP median (resolved/72)
LLaVA-1.5-7B6 / 5 / 3 / 3[3, 6]12 / 14 (67/72)
LLaVA-1.5-13B10 / 8 / 7 / 6[7, 10]12 / 14 (64/72)
LLaVA-NeXT-34B25 / 24 / 23 / 22[22, 25]29 / 39 (68/72)
Qwen2.5-VL-7B12 / 11 / 11 / 10[11, 12]15 / 17 (70/72)
Qwen2.5-VL-32B31 / 30 / 28 / 27[27, 31]28 / 45 (70/72)
InternVL3.5-8B16 / 15 / 15 / 14[14, 16]19 / 24 (69/72)

Requested ACP indices: The final column gives the held-out tuned-rank layer / consensus ACP median, and in parentheses the number of images with resolved ACP out of 72. State kk is after kk decoder blocks, equivalently before zero-based block kk. Consensus ACP additionally requires raw/tuned sign stability and both rank guards. Across tested cutoffs, the held-out early/late KL ratios remain ≥17.80×. In 200 joint calibration resamples reselecting EAB and refitting the tuned lens, all six checkpoints retain positive 95% gap intervals relative to the joint raw/tuned rank guard at all cutoffs.

We agree that the earlier held-out effect intervals did not quantify uncertainty in the selected EAB location. Revision: We will specify the tensor operation and its limits in Section 3.3 and Appendix A.2, and add the cutoff sensitivity, boundary-location intervals, and ACP indices in Appendix C.2.

W2. Is routing distinct from global strength calibration, and are the probes used?

The evaluated router uses a calibration-derived, type-level binary β\beta; its additional benefit over a calibrated constant weight is supported on 13B, but not across all three checkpoints. For antonym type aa, with d=M(c1)−M(c2)d=M(c_1)-M(c_2) and dcf=Mcf(c1)−Mcf(c2)d_{\mathrm{cf}}=M_{\mathrm{cf}}(c_1)-M_{\mathrm{cf}}(c_2), define

sa=Pr⁡cal[sign⁡(d)≠sign⁡(d−dcf)∣a]s_a=\Pr_{\mathrm{cal}}[\operatorname{sign}(d)\ne\operatorname{sign}(d-d_{\mathrm{cf}})\mid a]

β=1[sa>τ]\beta=\mathbf{1}[s_a > \tau]

S(c)=M(c)−βMcf(c).S(c)=M(c)-\beta M_{\mathrm{cf}}(c).

For each held-out fold, sas_a is estimated on the other four image-disjoint calibration folds. Types with fewer than eight calibration pairs, or unseen types, use the global disagreement fraction. We choose τ\tau from 41 percentiles (0, 2.5, ..., 100) of calibration selector values to maximize calibration accuracy, breaking ties toward the smallest threshold. Folds are built from sorted image IDs shuffled with NumPy RandomState seed 0. Selector values use calibration scores; threshold selection uses calibration labels only.

Requested baseline: On these same folds, a checkpoint-specific constant β∈0,0.01,…,1\beta\in\\{0,0.01,\ldots,1\\} is tuned on the other four folds (smallest value on ties):

CheckpointFixed β\beta accuracyRouter accuracyPaired gain (pp), 95% image-cluster CI
LLaVA-1.5-7B77.04%77.75%+0.71 [−0.17, +1.57]
LLaVA-1.5-13B79.60%81.71%+2.10 [+1.45, +2.76]
InternVL3.5-8B87.67%87.33%−0.33 [−1.94, +1.23]

On 13B, the advantage persists after 200 bootstrap refits of both calibration rules (95% CI [+0.75, +2.65] pp; paired image sign-flip p<10−4p < 10^{-4} for the fitted rules). This separates a routing contribution from global-strength calibration for 13B only.

Are equivalent-prompt and alternative-ablation sign-consistency probes used on evaluation claims or only in calibration? Neither. They are diagnostic probes only and are used in neither calibration nor evaluation of the reported primary router. The evaluated selector uses DisFrac (sas_a) alone. We agree that Section 5.3 and Appendix E.1 did not make that distinction clear.

Stability safeguard: Its independent contribution has not been quantified. Therefore, we do not attribute the router's improvement to it or claim a demonstrated safeguard-specific gain.

Revision: We will state the actual β\beta selection and calibration rule, add the same-fold fixed-weight comparison and paired uncertainty, separate diagnostic probes from evaluated routing features, and explicitly qualify the unisolated safeguard contribution in Section 5.3 and Appendix E.1.

Thank you for the authors' detailed responses. Some of my concerns have been addressed. However, I remain concerned about the substantial discrepancies between the original manuscript and the rebuttal results. As the authors clarified, the original EAB results were based on the median of per-query boundaries. In contrast, the rebuttal uses a different analysis procedure to correct the EAB locations and effect estimates - the substantial change in the effect ratios from approximately 13–26× to 27–96×. This raises my concerns about the reliability of the original results. Although the updated results still support the main finding, I think this is more than a minor clarification considering that EAB is the main contribution of this paper. I would maintain my original rating.

Minor: It was quite difficult to identify which parts of the original manuscript should be replaced based on the rebuttal. I would suggest providing a revised manuscript with the changes clearly highlighted, or common responses that summarize the sections, figures, and tables changed (e.g., Table 6).

Reviewer_YF3FOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes between two issues: at which decoder layers visual edits can still alter the output of a MLLM, and when counterfactual corrections can improve the accuracy of its responses. The authors define an Evidence Anchoring Boundary based on layer-wise interventions of equal RMS strength. The study also examines when subtracting counterfactual scores helps or hurts different visual claims. The experimental section includes a layer-wise analysis across six model checkpoints, followed by tests on selective routing and targeted generation for a subset of these models.

Critical strengths
  1. The distinctions made between answers readable from intermediate states, intervention effects, and accuracy improvements are useful. By matching intervention intensities and establishing a fixed boundary prior to evaluating new queries, the authors’ layer-wise analysis becomes more convincing. The large differences in intervention effects observed before and after the boundary remain consistent across all six model checkpoints.
  2. I particularly appreciate the inclusion of negative results. For instance, while subtracting counterfactual scores may help with a subset of hard attributes, it can hurt performance on the full attribute set. The separate object-presence correction, PEC, also transfers poorly to several other tasks. The authors have given careful consideration to these failure cases. The routing evaluation uses disjoint images, and the uncertainty estimates account for image clusters. Both choices strengthen the evaluation. While the improvement of approximately 1.6 to 1.8 percentage points on attribute claim pairs is modest, it is supported by the reported statistical tests.
Critical weaknesses
  1. The specific boundary location is less well supported than the general decline in intervention effect with depth. The boundary uses a 10% persistence rule calibrated on 48 queries. Appendix C.2 explicitly acknowledges that sensitivity to other cutoffs and uncertainty in the selected location are not quantified. The held-out intervals concern the effect ratio with the boundary fixed. I agree that EAB is an operator-specific summary, but its relation to answer stabilization should still be checked across reasonable boundary choices.
  2. The routing comparison focuses primarily on two extremes: the original margin M and full subtraction G. A key baseline is missing here: a fixed mixing weight selected separately for each checkpoint using the same calibration folds. Without this control, the gains might come from calibrating the overall subtraction strength rather than choosing it for each input.
Questions for authors
  1. For each checkpoint, how much do the EAB location and its position relative to answer stabilization change under nearby cutoffs (like 5%, 15%, 20%) or different calibration samples? Please also specify exactly which residual tensors are edited and how equal RMS edit strength is enforced.
  2. As above, under the same five-fold image-disjoint split, does the attribute router outperform a constant mixing weight selected per checkpoint using only the calibration folds? The paired held-out difference and its uncertainty would help assess the routing contribution.
4 comments
AuthorsOpenReview ↗

EAB Robustness and Calibrated Routing Baselines

Thank you for identifying two specific controls that would strengthen the conclusions. We have evaluated both: sensitivity of the selected EAB location and a same-fold, checkpoint-specific constant-β\beta baseline.

W1. EAB location, threshold sensitivity, and answer stabilization

The precise EAB layer varies with the threshold and calibration sample; we do not claim an invariant location. However, the measured separation from the answer-readout stabilization guard persists across the tested choices.

We recalculated EAB at 5%, 10%, 15%, and 20% cutoffs. We also used 2,000 bootstrap resamples of the 48 calibration images to estimate uncertainty in the selected layer, rather than only the held-out intervention effect with a frozen boundary.

CheckpointEAB at 5/10/15/20%10% EAB location 95% intervalTuned rank layer / consensus ACP median (resolved/72)
LLaVA-1.5-7B6 / 5 / 3 / 3[3, 6]12 / 14 (67/72)
LLaVA-1.5-13B10 / 8 / 7 / 6[7, 10]12 / 14 (64/72)
LLaVA-NeXT-34B25 / 24 / 23 / 22[22, 25]29 / 39 (68/72)
Qwen2.5-VL-7B12 / 11 / 11 / 10[11, 12]15 / 17 (70/72)
Qwen2.5-VL-32B31 / 30 / 28 / 27[27, 31]28 / 45 (70/72)
InternVL3.5-8B16 / 15 / 15 / 14[14, 16]19 / 24 (69/72)

Across these cutoffs, the held-out early/late KL effect ratio stays at least 17.80×. In 200 joint calibration resamples that reselect EAB and refit the tuned lens, the 95% gap intervals relative to the joint raw/tuned rank guard remain positive for every checkpoint and cutoff. Varying persistence (2/3/4 layers) and the suffix quantile (0.5/0.75/0.9) shifts the 10% boundary by at most one layer. These results support the relative separation, not an exact threshold-independent layer index.

Requested tensor operation and equal-RMS control. At the input HjH_j of every decoder block j≥ℓj\geq\ell, before normalization/self-attention, we edit only the visual-token positions VV:

δj=mean⁡v∈V(Hj)−Hj,V\delta_j=\operatorname{mean}_{v\in V}(H_j)-H_{j,V}

Hj,V′=Hj,V+rδjmax⁡(RMS⁡(δj),10−8).H'_{j,V}=H_{j,V}+r\frac{\delta_j}{\max(\operatorname{RMS}(\delta_j),10^{-8})}.

The target rr is the checkpoint-specific median unnormalized edit RMS from the 48 calibration images, fixed before evaluation and enforced independently at each injection. Text-token states are not directly edited; thus EAB characterizes this operator, not the disappearance of all visual information. The tuned-rank layer and consensus ACP are reported separately because consensus additionally requires raw/tuned sign stability and both rank guards.

We agree that Appendix C.2 did not quantify uncertainty in the selected location. Revision: We will add the cutoff and bootstrap-location results and the comparison to the stabilization guard in Section 3.3 and Appendix C.2, while specifying the intervention operator and equal-RMS implementation in Appendix A.2.

W2. Does routing improve over a calibrated constant mixing weight?

Yes on LLaVA-1.5-13B, but not consistently on the other two checkpoints. We agree that comparison against only preservation and full subtraction cannot isolate selective routing from calibration of overall subtraction strength.

We added the requested constant-β\beta baseline, selecting β∈0,0.01,…,1\beta\in\\{0,0.01,\ldots,1\\} separately for each checkpoint and held-out fold using only the other four image-disjoint calibration folds. Ties select the smallest β\beta. The router uses the identical splits.

CheckpointCalibrated fixed β\beta accuracyRouter accuracyPaired gain (pp), 95% image-cluster CI
LLaVA-1.5-7B77.04%77.75%+0.71 [−0.17, +1.57]
LLaVA-1.5-13B79.60%81.71%+2.10 [+1.45, +2.76]
InternVL3.5-8B87.67%87.33%−0.33 [−1.94, +1.23]

The confidence intervals come from 10,000 paired whole-image bootstrap draws. On 13B, the gain remains positive after 200 bootstrap refits of both calibration rules (95% CI [+0.75, +2.65] pp; paired image sign-flip p<10−4p < 10^{-4} for the fitted rules). Thus, an additional routing benefit beyond calibrated constant mixing is supported on 13B, while the 7B interval includes zero and InternVL shows no advantage.

Revision: We will add this baseline and paired uncertainty in Section 5.3 and Appendix E.1, and qualify the routing claim accordingly.

Thank you. The threshold and bootstrap analyses substantially address my concerns about EAB robustness, and the calibrated constant-β baseline provides a useful clarification of the routing contribution.

One minor clarification would be helpful: the 10% EAB locations reported here appear to differ from the frozen EAB locations in Table 6 of the original submission. Could you briefly explain the reason for this discrepancy?

Thank you for catching this discrepancy. It reflects a correction to the analysis protocol that we should have made explicit in our previous response.

On auditing the original analysis, we found that the EAB entries in Table 6 were medians of per-query boundaries computed from a normalized margin/√KL composite, although the table described them as frozen calibration-selected boundaries. The rebuttal instead uses newly collected curves with each injection independently RMS-normalized, and selects one boundary from the mean full-vocabulary KL trajectory of 48 calibration images, before freezing it for evaluation on 72 explicitly disjoint images. Thus, the same nominal 10% cutoff is applied to different effect profiles and aggregation procedures; the discrepancy is not caused by bootstrap resampling.

The rebuttal values therefore supersede the original Table 6 locations and associated effect estimates. We will correct Table 6 and the corresponding text and figures to use this consistent protocol and explicitly document the correction. Under this protocol, the held-out pre/post KL ratios at 10% are 27.39–96.14× across the six checkpoints. We apologize for not explaining this distinction earlier.

Thank you. I will maintain my rating.

Reviewer_k7mkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks two important questions about visual interventions for reducing MLLMs’ hallucinations: (1) Where can a visual intervention still change the final answer? (2) When does a change improve the answer? To answer question 1, the authors control the effect of visual-residual edits across different layers and establish EAB, which indicates that after a certain layer, visual information is not prominent. The authors further use ACP to compensate for this finding. To answer question 2, the authors use mean visual feature ablation to construct M_cf and G, which indicate how an answer is related to visual information. The finding is that the participation level of visual information does not necessarily connect to the correct answer. Thus, the authors design a selective routing method that dynamically uses G and M_cf under different conditions. They further design PEC for a calibrated object-presence setting. Experiments show improvements across different tests, especially for PEC on POPE-adversarial. But PEC is limited when generalizing to other tasks.

Critical strengths
  1. I do believe I have learned something interesting and useful about how visual features affect the model’s output. The EAB and ACP analyses are very informative. M_cf and G also provide interesting and honest findings about the effects of visual features. The analysis itself is a good contribution of this paper.

  2. The evaluation includes different controls and results. The attribute router is also evaluated with image-disjoint calibration. The paper also reports both improvements and cases where performance decreased, which clarifies the scope of this paper.

Critical weaknesses
  1. I found that several key designs are not well specified in the algorithm and diagnosis. I have listed them below:
(1) How are “visual residual contributions” extracted and removed when obtaining EAB and ACP?
(2) How exactly is (\beta) obtained? Eq. 7 just gives the definition of S, and Figure 4 is also just a general pipeline, which is far from enough.

  2. PEC is in fact the most efficient design (correct me if I am wrong here), but it seems to me that it is very insufficiently established, with only one or two introductory sentences. It also looks quite irrelevant to the main findings of this paper. In my understanding, PEC is independent of EAB, so its relation to Q1 is very limited. It is also not very well established how it is related to Q2. Yet it brings quite a significant improvement on POPE-adversarial.

Questions for authors

The major questions are asked in the weaknesses. Additionally, the authors could provide the complete pseudocode for the router and its calibration procedure. They could also reveal which dataset is used for the analysis of Q1 (there seem to be 120 queries), including how the queries are samp

3 comments
AuthorsOpenReview ↗

Intervention Details, Router Calibration, and PEC Scope

Thank you for highlighting two places where the manuscript did not give readers enough information: the implementation details in W1 and the role of PEC in W2. We address these points directly below.

W1. Intervention mechanics, router calibration, and Q1 data

(1) What is the “visual residual contribution,” and how is it removed for EAB and ACP?

We do not extract and delete a separately identifiable visual-information component. The EAB experiment instead applies a specified mean-centering intervention to visual-token residual states. Let HjH_j be the residual input to decoder block jj, before normalization/self-attention, and VV the visual-token positions. For an intervention starting at layer ℓ\ell, at every block j≥ℓj\geq\ell we apply

δj=mean⁡v∈V(Hj)−Hj,V\delta_j=\operatorname{mean}_{v\in V}(H_j)-H_{j,V}

Hj,V′=Hj,V+rδjmax⁡(RMS⁡(δj),10−8).H'_{j,V}=H_{j,V}+r\frac{\delta_j}{\max(\operatorname{RMS}(\delta_j),10^{-8})}.

The checkpoint-specific rr is the median unnormalized edit RMS on the 48 calibration images and is fixed before evaluation. Each injection is normalized separately; equal RMS describes the strength of each injection, not the accumulated effect of editing a variable-length suffix. Text-token states are not directly edited, so visual information previously propagated into them is preserved at the insertion point. ACP is measured separately from clean answer-position residual readouts, using raw/tuned sign stability and rank reliability; ACP does not use this intervention.

Our phrase “removing visual residual contributions” was too broad. Revision: We will state the exact tensor operation, insertion point, and its limited interpretation in Section 3.3 and Appendix A.2.

(2) How is β\beta determined?

The evaluated router uses a binary, calibration-derived β\beta, shared by claim pairs of the same antonym type; it does not fit a continuous weight separately for each test pair. For a fixed order of claims c1,c2c_1,c_2, define

d=M(c1)−M(c2)d=M(c_1)-M(c_2)

dcf=Mcf(c1)−Mcf(c2)d_{\mathrm{cf}}=M_{\mathrm{cf}}(c_1)-M_{\mathrm{cf}}(c_2)

sa=Pr⁡cal[sign⁡(d)≠sign⁡(d−dcf)∣a].s_a=\Pr_{\mathrm{cal}}[\operatorname{sign}(d)\ne\operatorname{sign}(d-d_{\mathrm{cf}})\mid a].

The evaluated scoring rule is

β=1[sa>τ]\beta=\mathbf{1}[s_a > \tau]

S(c)=M(c)−βMcf(c).S(c)=M(c)-\beta M_{\mathrm{cf}}(c).

The complete router/calibration procedure is:

Sort image IDs; shuffle with NumPy RandomState(seed=0).
Partition images into five disjoint folds.
For each held-out fold:
    1. Use the other four folds for calibration.
    2. Compute full-image and mean-counterfactual claim margins.
    3. Estimate the disagreement fraction s_a for each antonym type.
       If a type has <8 calibration pairs or is unseen, use the
       global calibration disagreement fraction instead.
    4. Form 41 candidate thresholds from percentiles
       0, 2.5, ..., 100 of the calibration selector values.
    5. Select tau maximizing calibration-pair accuracy;
       break ties in favor of the smallest tau.
    6. Freeze s_a and tau. For each held-out claim pair, set
       beta = 1[s_a > tau] and score S(c) = M(c) - beta*M_cf(c).
    7. Predict the higher-scoring claim.
Aggregate the held-out predictions across folds.

Only calibration scores enter sas_a, and only calibration labels are used to choose τ\tau. Revision: We will replace the schematic description with this explicit β\beta rule and complete pseudocode in Section 5.3 and Appendix E.1.

(3) Where are the code and Q1 sampling details?

Code availability. The complete anonymized code was already included in the Supplementary Material (ZIP) accompanying the original OpenReview submission.

Q1 dataset and sampling. We use 120 distinct POPE COCO-adversarial images, one query per image. For image index b=0,…,119b=0,\ldots,119, we select zero-based row 6b+(b mod 6)6b+(b\bmod6) from its six-query block, rotating the chosen query position deterministically. The first 48 images calibrate EAB and the lens; the remaining 72 are held out for evaluation. Revision: We will clarify the dataset, sampling rule, and calibration/evaluation split in the experimental setup.

W2. Is PEC the most efficient design, and how is it related to Q1/Q2?

PEC is comparatively lightweight in its forward-pass requirements, but we have not established that it is the fastest method in wall-clock time. We agree that it is independent of EAB. PEC uses the presence-score adjustment

zPEC=zL+α(zL−zℓ⋆).z_{\mathrm{PEC}}=z_L+\alpha(z_L-z_{\ell^\star}).

It needs one clean forward pass plus an intermediate-state head projection, whereas the reported counterfactual router needs two scoring forwards per claim. Without matched runtime measurements, we do not claim a definitive latency advantage.

Relation to Q1: PEC does not apply the EAB intervention and does not establish where a visual edit ceases to influence the model. It should not be presented as a consequence or validation of EAB.

Relation to Q2: PEC is an auxiliary, task-specific score-calibration example. Its relevance is that a score change must be judged by whether it improves decisions; it is not itself evidence of a general correction mechanism.

The observed POPE-adversarial gain is real but limited in scope: with historical layers fixed and α\alpha calibrated on 50 images, accuracy on 450 disjoint images increases from 77.59% to 81.11% (7B), 79.74% to 83.56% (13B), and 79.59% to 86.56% (InternVL). PEC also transfers poorly to several other evaluated tasks.

We agree that the brief original description did not adequately support the prominence given to PEC. Revision: We will clarify the intermediate-state projection and the actual selection/calibration procedures for ℓ⋆\ell^\star and α\alpha in the PEC subsection, identify its forward-pass cost without making an unsupported runtime claim, and present it as an auxiliary task-specific result rather than as part of the EAB mechanism. We will state its transfer limitations explicitly.

Thank you for the detailed responses. They clarify my questions about router calibration and data sampling, but my concern about the EAB analysis remains. All three reviewers raised concerns about different aspects of EAB, and the discussion with Reviewer YF3F revealed that the original results used a different procedure from the one described in the manuscript. Since EAB is central to the paper, this goes beyond missing implementation details. Although the updated results support the qualitative finding, it remains insufficiently clear which original results and conclusions are superseded. The clarified scope of the router and PEC also limits the methodological contribution. I therefore maintain my weak reject rating.

Comments on the paper

1 comment

Dear reviewers, with the discussion period nearing its end, could you please share your thoughts on the authors’ rebuttal and whether your concerns have been addressed? Please also update your scores if warranted. Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.