- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes PARALLAX, a supervised linear probe for hallucination detection that uses intermediate layers selected automatically rather than fixed in advance. Specifically, it scores the layers by inner 5-fold cross-validated AUROC on the training split only, selects one layer or a small set, and fits the probe there. Experiments across five open-weight models and four corpora are provided.
Critical strengths
The problem considered in this work is important. Internal-state probes are among the cheapest hallucination detectors available at deployment time, and which layer they read affects the performance a lot.
Critical weaknesses
- A published, near-identically titled paper [1] on the same problem is not cited, discussed, or compared against. It studies exactly the problem this submission poses: automatically choosing the probe layer for hidden-state hallucination detection, across models and datasets. It seems the method proposed in this work is called validation loss there, as one of the baselines they explored but is shown to be less optimal than their proposed method. It would be helpful to provide a comparison with this work including detection performance, runtime etc.
- The multi-layer (k>1) component is not justified by the results. Table 4 shows k=1 and the adaptive rule are indistinguishable on three of four corpora.
- Three of the four corpora are not hallucination detection in the usual sense, and the fourth is not the probed model's text. In TruthfulQA, HaluBench and MedHallu the model is handed a candidate answer it did not write and the probe reads its states over that text; in RAGTruth the responses come from other LLMs. The task is therefore closer to supervised claim verification on supplied text than to detecting the model's own hallucinations, which is the setting of most of the cited baselines in this work.
- The abstract's "first reason" is unclear and not acted upon. The TXTEMB result is about benchmark construction, not about detectors being fragile, yet it is presented as one of two reasons detector numbers are fragile and the proposed method is not shown to mitigate this issue.
[1] "Automatic Layer Selection for Hallucination Detection" Wang, Cao, Wilson, Zeng, ICML 2026.
Questions for authors
- Can the authors provide a comparison with the related work mentioned above?
- What is the recommended k? Given Table 4 and Table 13, is there any setting where you would advise a practitioner to use k>1, and how would they know in advance?
- Can you report results on at least one setting where the probed model generates the scored text so the method is evaluated on the model's own hallucinations?
- Does the selected layer change if pooling is switched from mean over the scored text to last-token or first-sentence-token? Does the "stable across corpora" finding survive that change?
On weakness 4 in all three parts, and Q3. Thank you for the two ablations and the HaMI comparison. HaMI scores every token and pools the top tenth with a learned head, where our probe reads one pooled vector per response.
W4, first part: gains over SAPLMA's five preset depths at full scale. Our four-corpus median gain is 0.0006. Across 20 model-corpus pairs at five seeds the mean is +0.0038, with 17 of 20 positive and a range of -0.0054 to +0.0197. At seed level that is 72 wins, 7 ties and 21 losses over 100 seeds. The median sits inside run-to-run variation from re-extracting the same activations, which moves our numbers by 0.0004 to 0.0088. Two of the four corpora saturate both selection rules, and on the two that do not the medians are +0.0102 on TruthfulQA and +0.0034 on RAGTruth (see our second reply to reviewer QTiD).
W4, second part: label budget and transfer. HaMI leads in 18 of 24 transfer settings, and 19 of 24 on a fresh re-extraction. On the label curve, our probe beats single-pass HaMI at 50, 100 and 200 labels and trails the stronger HaMI variant at 400. The combination your Q5 proposes closes much of the transfer gap (see our third reply).
On the label curve we ran RAGTruth at 50, 100 and 200 labels on Llama-3.3-70B and DeepSeek-R1-32B over ten paired seeds, with each method seeing only its label budget. We lead single-pass HaMI at all three budgets on both models, by +0.0097 to +0.0708, on 10 of 10 seeds in five of the six settings and 4 of 10 on DeepSeek-R1-32B at 200 labels. The margin peaks at 100 labels (+0.0708 and +0.0659) and drops at 200 (+0.0318 and +0.0097).
At 400 labels, the full training fold, the stronger HaMI variant leads by 0.013 on Llama-3.3-70B and 0.026 on DeepSeek-R1-32B under our adaptive rule, and by 0.029 and 0.023 at k=1. Our probe beats single-pass HaMI with 200 or fewer labels, and the stronger HaMI variant beats ours at 400. HaMI's token-level reader may help where the labelled span is a small part of a long response.
W4, third part: our own HaMI comparison, recounted. Both methods use the same 500 rows per corpus (499 on TruthfulQA) and the same five-fold splits for seeds 0 to 9. The table shows our score minus the better HaMI variant, using our adaptive rule:
Our submission reported 21 of 25 wins by counting any positive difference as a win. We now count a difference as a lead only when it exceeds 0.0088, the upper end of run-to-run variation. That is a descriptive band, not a test of equivalence, and it gives 12 leads, 3 losses and 10 differences inside the band. Each pair takes the better of the two HaMI variants where both ran.
At the k=1 we recommend, the comparison is worse for us. On RAGTruth the column becomes -0.018, +0.025, -0.036, -0.023 and -0.029, so HaMI leads on 4 of 5 there instead of 3. Two pairs change category over the full 25, and the count moves to 11 leads, 4 losses and 10 inside the band.
Q3: why HaMI is absent from Tables 2 and 5. Extraction cost. Full HaMI now runs on all 25 within-corpus pairs of the 500-row sets, and the single-pass variant on 17, where it is the better variant on 12. The cross-corpus transfer run covers two models (Llama-3.3-70B and DeepSeek-R1-32B). On the remaining 8 pairs only full HaMI ran, so for a third of the comparison we do not know which variant is stronger, and the caption states that.
Revision. We will report the per-corpus breakdown of the SAPLMA-5 comparison in Section 5. Section 6 will carry the HaMI comparison at both values of k with the band counts, the label curve at all four budgets, with the opponent at each budget stated, the transfer counts, and the RAGTruth losses.