W1. Limited novelty: the pooling is standard attention-MIL pooling.
Eqs. 10 to 12 are the attention-based multiple-instance-learning pooling of [1] and closely match the span-attention representation of [2]; neither is cited. Because the risk head is linear and ∑tαt=1, the span logit decomposes exactly as
us=wr⊤∑tαtet+br=∑t=ab−1αt,ut,ut=wr⊤et+br,αt=softmaxt(wa⊤et).
SpanProbe is therefore a softmax-weighted average of per-token logits. It interpolates between mean pooling (wa→0) and max pooling (wa∝wr with a large scale) of token logits. The span labels have "at least one part is hallucinated" semantics (an MSCOCO sentence is positive if any object is hallucinated), so the principled baselines are max, log-sum-exp or noisy-OR pooling of token logits, none of which is compared. StatPool, which takes the maximum over representations rather than over token logits, is already within 2.27 AUPRC (95% CI [1.09, 4.29], Table 12).
W2. The motivating claim relies on the method's own attention weights.
Fig. 1(a) uses SpanProbe's own pooling weights as evidence that hallucination signals concentrate on a few tokens, but attention weights are not faithful explanations [3]. The independent analysis in App. H.2 clips and renormalizes noisy differences (vt=max(δt,0)) before computing the Gini coefficient and top-20% share. This procedure produces concentration even when the true signal is uniform, so comparing against a "uniform baseline" is not a valid null; a noise null is required, for example the same statistics computed between two images that both support the object. The remove-and-retrain effect in App. H.3 is small (1.86 and 2.51 AUPRC), uses one teacher per fold, and has no confidence intervals, while random removal improves AUPRC on LLaVA-v1.5, indicating a noise level of about 0.5 points.
W3. The mitigation comparison is supervised versus training-free, and the supervision matches the metric.
The MSCOCO mitigation detector is trained on COCO val2014 images with the same prompt and with labels produced by the CHAIR object-matching procedure (App. A.4), and the span constructor uses the same CHAIR vocabulary. In effect, the detector learns the evaluation metric's labeling function on the same distribution. On POPE, the detector is trained on 400 of POPE's 500 images with the same question templates, and because the answer is binary, a rollback with the rejected answer banned simply flips the answer, so the mitigation reduces to a supervised answer-correctness classifier. PAI, VCD, EAZY and beam search use no labels. A fair comparison requires supervised detector-guided baselines, such as HaloProbe-guided decoding [4] (cited but not compared), VIB-Probe mitigation under the same protocol (Table 10 is cross-study only), Lookback-Lens-guided decoding [5], M-HalDetect rejection sampling [6], and REVERSE [7], together with evaluation on benchmarks whose labels are not defined by CHAIR (e.g., AMBER, Object HalBench, MMHal-Bench).
In addition, banning only the exact rejected sequence (Eq. 15) leaves the model free to emit a synonym or a plural, or to omit the object, and nothing in the procedure favors a grounded replacement. The results suggest that omission contributes substantially: object recall drops from 65.19 to 61.10 on LLaVA-v1.6 and from 55.84 to 50.65 on Qwen2.5-VL, and objects per caption drop from 5.81 to 5.28 and from 5.26 to 4.57 (Table 7). Fewer mentions mechanically lower CHAIR, yet no CHAIR-versus-recall curves over τ are reported.
W4. Baseline fairness and missing baselines.
MetaToken, DHCP and VIB-Probe are matched reimplementations with span-adaptation rules chosen by the authors; in particular, VIB-Probe is restricted to endpoint readout, the very strategy the paper argues against. No reproduction of published numbers is shown, and SVAR and Internal Confidence perform near chance on M-HalDetect and MSCOCO (AUROC 51 to 56), which suggests poor adaptation. Strong simple baselines are missing: a probe on middle-layer hidden states with mean or attention pooling (the "w/o TEP" ablation uses only the final layer, which Table 15 suggests is the weakest depth), ContextualLens (cited but not compared), HaloProbe [4], Lookback Lens features [5], and HalLoc [8]. The POPE detection results add little evidence, because each question is scored with both "Yes" and "No", exactly one of which is wrong, so detection reduces to answering the question, involves a single token (the pooling is never active), and is near ceiling.
W5. Additional concerns.
Table 1 and all mitigation results come from a single run (mitigation on 500 images) without confidence intervals. Detectors are trained per backbone and dataset; transfer is tested only on LLaVA-v1.5 and cross-model transfer is not tested. The efficiency comparison uses eager attention for both methods because TEP requires materialized attention probabilities, so the overhead relative to a realistic FlashAttention or SDPA deployment is understated. Finally, detection is framed around arbitrary semantic spans, but mitigation acts only on COCO object mentions of one or two tokens, where adaptive pooling has little effect; the mitigation gains therefore come mainly from the TEP features and the rollback, not from the paper's central contribution.
Response to authors
Thank you for the detailed response. Some of my concerns are resolved; others remain.
Span boundaries. Thank you for clarifying that candidate boundaries are determined online during generation. I withdraw my comment that the method assumes span boundaries are known in advance. I note, however, that the online span constructor identifies only object mentions matching the CHAIR vocabulary, so the mitigation applies only to that type of span.
Rollback. My main concern remains. Because rollback restores generation to the start of the flagged span, the prefix that introduces the claim (e.g. "There is also a") is retained, and the model must still produce an object in that position. Re-detection only helps when the replacement is itself recognized as an object mention. If it falls outside the vocabulary, it is never re-evaluated, and it is also not counted by CHAIR. The example in Figure 2 ("traffic light" → "roadside pole") appears to be such a case. Could the authors report:
Cost. Thank you for the latency comparison. Two questions:
Novelty. I appreciate the clarification that other detectors can be plugged into the rollback loop, and that Table 9 compares them under a shared policy.
Perhaps my largest disagreement from the authors response is the claim that hallucination evidence is concentrated within spans. Figure 1 shows SpanProbe's own learned softmax weights, which are non-uniform by construction. A trained softmax will always favor some items over others, so I find Figure 1 uninformative. The independent evidence suggests modest concentration: in the remove-and-retrain analysis, removing the top-weighted token costs 1.9–2.5 AUPRC points. In the qualitative examples, the top-weighted token is usually the object noun. Could the authors compare against a fixed readout at the head noun of the span, and against max pooling? This would show whether the adaptive weighting learns more than "attend to the object word."
I am maintaining my score for now and will reconsider in light of answers to these questions.