- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies where visual interventions retain influence on multimodal predictions and when counterfactual subtraction improves accuracy. An equal-RMS layer sweep across six checkpoints identifies an Evidence Anchoring Boundary (EAB), where the tested intervention gets persistently weak, potentially before answer-readout stabilization. The authors also examine corrective and harmful subtraction across question types and propose calibrated selective routing, which improves full-set attribute accuracy by 1.56–1.81 percentage points across three checkpoints.
Critical strengths
- The layer-wise analysis includes useful controls and held-out validation. Matching intervention RMS across depth helps separate perturbation level from layer position. Selecting the boundary on calibration queries and evaluating it on disjoint queries strengthens the evidence for the early-versus-late effect contrast across six checkpoints.
- The question-type analysis shows why subtraction should be applied selectively. The paper reports both corrective and destructive outcomes. The paper evaluates full-set attribute routing through image-disjoint cross-validation, and the reported transfer failures appropriately limit the scope of the correction methods.
Critical weaknesses
- The definition and robustness of EAB are not sufficiently established. Section 3.3 and Appendix A.2 describe deleting the "visual residual contribution," but do not specify the exact tensor operation or insertion point. Thus, it is unclear which pathway is edited and whether image information previously propagated into text-token representations is preserved. This matters for interpreting the measured influence and its separation from ACP. Moreover, the authors select EAB using a 10% threshold on 48 calibration queries, without analyzing its location under nearby thresholds or resampled calibration sets. Appendix C.2 explicitly distinguishes the held-out effect-ratio intervals from uncertainty in the selected layer. Although the paper appropriately treats EAB as an operational boundary, the large pre/post contrast does not establish that its location is stable. These gaps limit the reproducibility and interpretation of the central finding.
- The routing gains are not sufficiently separated from simpler calibration of subtraction strength. Section 5.3 and Appendix E.1 list counterfactual score spread, ranking disagreement, and sign consistency as routing signals, but do not specify the concrete rule mapping them to (\beta), including its thresholds and selection procedure. The main comparison uses preservation, full subtraction, and routing, without a constant (\beta) optimized under the same calibration protocol. The gains could therefore partly reflect stronger subtraction rather than the proposed selection rule. Table 3 also states that the stability safeguard’s separate contribution is not quantified. Without these details and controls, I find the additional contribution of the router insufficiently established to support acceptance.
Questions for authors
- What are the ACP layer indices for the six checkpoints under the held-out tuned lens?
- Are the equivalent-prompt and ablation probes used to assess sign consistency run for each evaluation claim set, or are these probes used only during calibration?
6 comments
W2. Is routing distinct from global strength calibration, and are the probes used?
The evaluated router uses a calibration-derived, type-level binary ; its additional benefit over a calibrated constant weight is supported on 13B, but not across all three checkpoints. For antonym type , with and , define
For each held-out fold, is estimated on the other four image-disjoint calibration folds. Types with fewer than eight calibration pairs, or unseen types, use the global disagreement fraction. We choose from 41 percentiles (0, 2.5, ..., 100) of calibration selector values to maximize calibration accuracy, breaking ties toward the smallest threshold. Folds are built from sorted image IDs shuffled with NumPy RandomState seed 0. Selector values use calibration scores; threshold selection uses calibration labels only.
Requested baseline: On these same folds, a checkpoint-specific constant is tuned on the other four folds (smallest value on ties):
| Checkpoint | Fixed accuracy | Router accuracy | Paired gain (pp), 95% image-cluster CI |
|---|---|---|---|
| LLaVA-1.5-7B | 77.04% | 77.75% | +0.71 [−0.17, +1.57] |
| LLaVA-1.5-13B | 79.60% | 81.71% | +2.10 [+1.45, +2.76] |
| InternVL3.5-8B | 87.67% | 87.33% | −0.33 [−1.94, +1.23] |
On 13B, the advantage persists after 200 bootstrap refits of both calibration rules (95% CI [+0.75, +2.65] pp; paired image sign-flip for the fitted rules). This separates a routing contribution from global-strength calibration for 13B only.
Are equivalent-prompt and alternative-ablation sign-consistency probes used on evaluation claims or only in calibration? Neither. They are diagnostic probes only and are used in neither calibration nor evaluation of the reported primary router. The evaluated selector uses DisFrac () alone. We agree that Section 5.3 and Appendix E.1 did not make that distinction clear.
Stability safeguard: Its independent contribution has not been quantified. Therefore, we do not attribute the router's improvement to it or claim a demonstrated safeguard-specific gain.
Revision: We will state the actual selection and calibration rule, add the same-fold fixed-weight comparison and paired uncertainty, separate diagnostic probes from evaluated routing features, and explicitly qualify the unisolated safeguard contribution in Section 5.3 and Appendix E.1.
Thank you for the authors' detailed responses. Some of my concerns have been addressed. However, I remain concerned about the substantial discrepancies between the original manuscript and the rebuttal results. As the authors clarified, the original EAB results were based on the median of per-query boundaries. In contrast, the rebuttal uses a different analysis procedure to correct the EAB locations and effect estimates - the substantial change in the effect ratios from approximately 13–26× to 27–96×. This raises my concerns about the reliability of the original results. Although the updated results still support the main finding, I think this is more than a minor clarification considering that EAB is the main contribution of this paper. I would maintain my original rating.
Minor: It was quite difficult to identify which parts of the original manuscript should be replaced based on the rebuttal. I would suggest providing a revised manuscript with the changes clearly highlighted, or common responses that summarize the sections, figures, and tables changed (e.g., Table 6).
EAB Specification, Robustness, and ACP
Thank you for identifying the missing operator specification, boundary-location robustness check, and router baseline. We agree these details are necessary to interpret the reported findings. We answer the two weaknesses and the specific questions directly.
W1. Which residual states are edited, how stable is EAB, and where is ACP?
The operator edits only visual-token residual states at decoder-block inputs; it does not remove visual information already propagated to text-token states. Let Hj be the residual input to decoder block j before normalization/self-attention, and V its visual-token positions. At every block j≥ℓ:
δj=meanv∈V(Hj)−Hj,V
Hj,V′=Hj,V+rmax(RMS(δj),10−8)δj.
The checkpoint-specific target r is the median unnormalized edit RMS on 48 calibration images and is frozen for evaluation. Each injection is independently RMS-normalized; suffix length varies with ℓ. Text-token states at insertion are not directly overwritten. ACP is evaluated separately on clean answer-position residual readouts, not through this intervention. We agree that “deleting visual residual contributions” was imprecise: EAB is an operator-specific sensitivity boundary, not evidence that all visual information disappears.
EAB's exact layer is threshold-dependent, but its separation from the answer-readout stabilization guard persists under the tested settings. All six checkpoints use 120 distinct POPE COCO-adversarial images (48 calibration, 72 held out). We reselected EAB at 5%, 10%, 15%, and 20% cutoffs and used 2,000 calibration-image bootstrap draws to estimate boundary-location uncertainty separately from held-out effect uncertainty.
Requested ACP indices: The final column gives the held-out tuned-rank layer / consensus ACP median, and in parentheses the number of images with resolved ACP out of 72. State k is after k decoder blocks, equivalently before zero-based block k. Consensus ACP additionally requires raw/tuned sign stability and both rank guards. Across tested cutoffs, the held-out early/late KL ratios remain ≥17.80×. In 200 joint calibration resamples reselecting EAB and refitting the tuned lens, all six checkpoints retain positive 95% gap intervals relative to the joint raw/tuned rank guard at all cutoffs.
We agree that the earlier held-out effect intervals did not quantify uncertainty in the selected EAB location. Revision: We will specify the tensor operation and its limits in Section 3.3 and Appendix A.2, and add the cutoff sensitivity, boundary-location intervals, and ACP indices in Appendix C.2.