- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper associates erroneous multimodal trajectories with longer low-entropy intervals and abrupt local entropy changes. SEED monitors tokens 5–60 for entropy below its previous mean together with an entropy increase greater than 0.25, then triggers two steps of visual-feature injection. Vision attention aggregates image tokens into a query-independent e_vis added to the language model's first-layer hidden state with alpha=0.5. Evaluation covers three backbones and six benchmarks/nine settings. Table 2 reports approximately 3.2–3.7 percentage-point average gains on reasoning tasks, and Table 6 reports approximately 5% additional latency.
Critical strengths
- The motivation includes population-level evidence rather than only selected examples. Pages 3–4 and A.3 align hallucination onset across 1,228 trajectories. Evaluation spans three backbones and both reasoning and hallucination tasks, which is more informative than interpreting an isolated attention curve.
- The method is lightweight and explicitly reports failures. Llama POPE and other regressions remain visible in the main tables. Hyperparameters are selected once and fixed across settings, with sensitivity and latency analyses. The limitations acknowledge correlation rather than causation, so the review should not incorrectly claim this distinction is absent.
Critical weaknesses
W1: Failure-analysis counts do not match the main-table scores, undermining the mechanism interpretation
- Table 3, p. 8, reports Llama POPE-Adversarial scores of 85.04 for the base model and 83.27 for SEED.
- B.5/Table 7, pp. 18–19, reports Correct→Wrong=454, Wrong→Correct=408, and Wrong→Wrong=729, with rates 15.13%, 13.60%, and 24.30%, implying N=3000.
- Under the same correctness definition, base errors are 408+729=1137, implying accuracy 62.10%. SEED errors are 454+729=1183, implying 60.57%. These do not match 85.04/83.27. The net change is -46/3000=-1.53 points rather than the main-table -1.77 points.
- Invalid-format outputs also increase from 610 to 646. If these count as errors, they alone imply an error rate above 20%, difficult to reconcile with 85% accuracy. Different runs, parsers, subsets, or metrics could explain the discrepancy, but these must be specified rather than guessed by the reviewer.
- This is not proof of fabrication. It does mean that the explanation attributing degradation mainly to formatting rather than hallucination cannot currently be verified. Versions and metrics need reconciliation.
W2: Experiments do not isolate entropy triggering; onset annotation and visual-vector implementation need to be auditable
- Table 4 compares different interventions: Equal attention, SVD, perturbation, and SEED. It does not hold e_vis, intervention count, and budget constant while comparing random/periodic triggers, low-entropy-only and spike-only triggers, no entropy gating, or mean visual embeddings. Gains could mainly come from inexpensive visual residual injection.
- On p. 15, hallucinated responses are aligned to their first hallucinated token, while controls use random positions. Annotation source, blinding/agreement, difficulty, token type, and position matching need explanation. Significant p-values do not remove selection/confounding. A full causal theory is unnecessary; online detector precision/recall and incremental benefit are the relevant questions.
- Equation 5 on p. 6 defines A_ref as n×n, whereas Eq. 6 uses A_ref,j as a scalar visual weight. Aggregation over query rows versus key columns must be explicit. Row sums of row-normalized attention may produce uniform weights; column sums imply another implementation. This is a specification ambiguity, not a claim that the code necessarily implements the wrong operation.
- Current-step entropy is available after final-layer logits. Injecting into the first layer of that same step requires recomputation; injecting into subsequent steps requires explicit indexing and cache rules. Does the 5% cost include recomputation, attention hooks, and any loss of efficient attention kernels?
- Page 7 tunes on a held-out M3CoT subset, while p. 17 evaluates on M3CoT validation. Disjoint task IDs are needed. Shared use of the name “validation” alone does not prove leakage.
Questions for authors
see weakness