acceptodds
Under review as a conference paper at ICLR 2027

LLM Hallucination Detection via Low-rank Multilinear Decomposition

Abstract

Large language models (LLMs) frequently generate fluent yet factually incorrect responses, raising concerns about their reliability. Recent hallucination detectors increasingly exploit internal representations, but extracting informative signals from high-dimensional hidden states remains challenging. Existing approaches often rely on selected activations or pooling-based compression, potentially overlooking discriminative information distributed across Transformer layers, feature modes, and tokens. We propose LOTUS (Low-rank Order-statistic TUcker-based Subspace learning), a multilinear low-rank decomposition framework for hallucination detection that extracts compact representations from LLM hidden states while reducing dependence on labeled data. Our approach first summarizes variable-length token representations using complementary statistical descriptors that capture peak activations, activation distributions, and cross-layer changes. We then leverage low-rank multilinear decomposition to extract compact representations that capture the underlying structure of LLM hidden states, which are subsequently used to train a lightweight hallucination classifier. Experiments across multiple LLMs and question-answering benchmarks demonstrate improved hallucination detection performance over recent state-of-the-art baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Hallucination Detection, Hidden-State Representations, Low-Rank Multilinear Decomposition, Tucker Decomposition, Label-Efficient Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zjPmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LOTUS, a framework for hallucination detection using compact representations of LLM hidden states. The method summarizes token activations through peak values, quantiles, and cross-layer differences, then applies Tucker-2 projections and a random-forest classifier. Experiments across two LLMs and four QA benchmarks demonstrate competitive detection performance with reduced representation storage.

Critical strengths
  1. The proposed framework is simple and well motivated, producing compact representations while incorporating information across layers and token positions.
  2. LOTUS consistently outperforms HARP and remains competitive with ACT-ViT across the evaluated settings. The summary ablations and cross-model alignment experiments provide additional insight into the method’s behavior.
Critical weaknesses
  1. Limited methodological contribution. LOTUS combines established statistical summaries, Tucker-2 decomposition, and a random-forest classifier. While practically useful, the current evidence does not sufficiently establish the advantage of the proposed multilinear structure over simpler alternatives. The paper reports that equal-dimensional PCA performs within 0.80 AUROC points with full training data, while single-summary variants often remain close to the full method.

  2. Practical efficiency. Reduced projection size and representation storage do not directly establish lower runtime or peak memory. A matched comparison of activation extraction, projection fitting, and detector inference would help substantiate the efficiency claim and distinguish offline costs from per-response overhead.

  3. Interpretation of the spectral analysis. The analysis supports compressibility but does not directly establish that the retained directions preserve the most useful information for hallucination detection. The connection between variance preservation and discriminative performance should be supported more directly or stated more cautiously. \end{itemize}

Questions for authors
  1. With summaries, layers, classifier, and output dimensionality held fixed, what advantages does Tucker-2 provide over flat PCA in detection performance and computational cost?

  2. How sensitive are the results to the number of sampled responses KK? Since KK affects both training-set size and the identification of model-known questions, separating these effects would help assess the robustness of the evaluation protocol.

  3. Can the authors compare detection using the retained subspace with detection using the discarded residual, to establish whether the low-rank projection preserves the relevant discriminative information?

Unless I have overlooked supplementary material, the version available to me does not include the appendices referenced in the main text. Could the authors clarify where these can be accessed? I would be happy to reconsider my score if the additional material and author response adequately address my concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zjPm (1/2)

We thank the reviewer for the detailed review and for the offer to reconsider. We apologize for the missing appendices: they were in the supplementary material (Appendices A–Q). The revised PDF now includes them after the references, together with new Appendices R–W. Table and appendix numbers below refer to the revised PDF, and each new result is given here in full.

W1 / Q1. Advantage of Tucker-2 over flat PCA with summaries, layers, classifier and width fixed.

Table 8 reports exactly this comparison: the same summaries, layer window, random forest and 896 output features (pooled AUROC, %).

Qwen TQAQwen TyDiLLaMA TQALLaMA TyDiFitted parametersMultiply-adds per response
Tucker-2 (LOTUS)87.7687.8289.2282.471.15M / 1.31M9.9M / 11.3M
Flat PCA88.0987.4588.4281.9645.6M / 52.2M45.6M / 52.2M
Random projection82.8785.3884.3277.24none45.6M / 52.2M
  • With full training data, the two reductions are within 0.80 AUROC points. The multilinear structure does not improve accuracy in this regime, and the revised text states this.

  • Tucker-2 requires about 40× fewer fitted parameters and 4.6× fewer multiply-adds per response.

  • With fewer labelled questions, Tucker-2 is more accurate (Table 11). At 25% and 50% of the training questions, where flat PCA can still supply all 896 components, Tucker-2 improves AUROC over flat PCA in 3 of 4 settings, by 0.71–2.91 points.

  • The fitted bases matter: a random projection of equal width lowers AUROC by 2.44–5.23 points.

  • Measured cost (new, Appendix V; TruthfulQA, 8 CPU cores, Qwen / LLaMA):

    • fitting the Tucker-2 bases takes 15.8 / 16.6 s, against 15.0 / 16.5 s for flat PCA;
    • peak memory is 12.9 / 14.6 GB, against 12.7 / 14.8 GB.

    The smaller projection therefore does not reduce runtime or memory at this scale, where robust scaling of the summaries dominates both. The revised text states the advantage as model size and robustness to scarce labels.

Against the strongest supervised baseline, the same property shows more clearly (new, Appendix T): with 5–10% of the training questions, LOTUS exceeds ACT-ViT by 4.42–28.22 AUROC points in all four settings.

On single-summary variants: we agree that each summary alone comes close to the full method. The full configuration is the most consistent, being the best or within 0.74 points of the best variant in every setting (Table 3). The revised text claims no more than this.

W2. Runtime and peak memory, offline versus per response.

New, Appendix V. TruthfulQA; one H100 for GPU stages and 8 CPU cores otherwise; Qwen / LLaMA.

StageLOTUSACT-ViT
LLM forward pass with hidden states (shared)6–44 ms per response6–44 ms per response
Pooling to the stored input, on GPU18.2 / 4.0 ms0.4 / 0.4 ms
Stored per response (float16)301 / 344 KB1,120 / 1,280 KB; 280 / 320 KB reduced (Appendix U)
Offline fitting2.4 / 2.6 min on CPU8.4 / 7.4 min of GPU training (15 epochs)
Peak memory during fitting12.9 / 14.6 GB host10.4 / 11.7 GB host, 1.6 GB GPU
Per-response scoring1.4 / 1.7 ms (CPU)1.4 / 1.6 ms (GPU)

The forward-pass times vary between runs on shared nodes, so we give the range over two runs. The cost profile is mixed:

  • Extraction: LOTUS's token-quantile summaries cost more to compute than ACT-ViT's max-pooling (4–18 ms against 0.4 ms per response on GPU), which is a fraction of the forward pass.
  • Storage: LOTUS stores 3.7× less per response than ACT-ViT as configured in the paper. ACT-ViT reduced to LOTUS's size loses at most 1.61 AUROC points in all but one setting (Appendix U), so we no longer claim a storage advantage.
  • Fitting: LOTUS fits on a CPU, with no GPU, in about a third of ACT-ViT's training time.
  • Scoring: the two detectors' per-response costs are equal.

The revised paper reports this profile in full instead of storage alone.

AuthorsOpenReview ↗

Response to Reviewer zjPm (2/2)

W3 / Q3. Detection with the retained subspace versus the discarded residual.

New, Appendix R. For each summary XX, the residual is R=X−ULUL⊤XUFUF⊤R = X - U_L U_L^\top X U_F U_F^\top. The same random forest is trained on four inputs:

  1. the retained Tucker-2 core;
  2. a Tucker-2 core of RR at the same ranks (896 features);
  3. a random projection of RR to 896 features;
  4. the retained and residual cores together.

Pooled AUROC (%), mean ± std over 5 seeds; within-question AUROC in brackets.

Qwen TQAQwen TyDiLLaMA TQALLaMA TyDi
Retained (LOTUS)87.76 ± 0.9 [82.1]87.82 ± 2.0 [88.1]89.22 ± 1.4 [82.7]82.47 ± 1.2 [81.3]
Residual, Tucker-2 core86.34 ± 0.9 [80.1]86.77 ± 2.1 [88.2]88.87 ± 1.2 [81.7]82.05 ± 1.4 [80.8]
Residual, random projection76.43 ± 1.1 [74.0]80.10 ± 2.0 [83.0]79.08 ± 1.1 [73.8]72.71 ± 3.0 [70.6]
Retained + residual87.45 ± 0.7 [81.2]87.48 ± 2.0 [87.9]89.30 ± 1.2 [82.7]82.55 ± 1.2 [81.4]
  • Nothing usable is discarded. Adding the residual to the retained core changes AUROC by −0.34 to +0.08 points, within one seed standard deviation.
  • The retained directions are the best slice. The retained core exceeds the residual's best same-size core in all four settings, by 0.35–1.42 points.
  • The signal is redundant. The residual's leading directions remain informative, whereas a random projection of the residual lowers AUROC by 7.7–11.3 points. The hallucination signal is therefore distributed redundantly across many directions. This redundancy is why a low-rank projection loses nothing.

The revised spectral paragraph states that the spectra establish compressibility and that this experiment provides the evidence on discriminative content. Consistent with it, increasing the feature rank from 64 to 128 changes AUROC by at most 0.46 points (Table 13).

Q2. Sensitivity to the number of sampled responses K.

New, Appendix S. For k ∈ {5, 3}, a seeded random k of the 10 responses per question is drawn. Three variants then change one factor at a time:

  • training-set size only: model-known questions and the split come from all 10 responses;
  • identification of model-known questions only: from the k responses, keeping every response;
  • both: the protocol as if K = k had been sampled.

Pooled AUROC (%), change from K = 10 in parentheses.

Qwen TQAQwen TyDiLLaMA TQALLaMA TyDi
K = 10 (reported)87.7687.8289.2282.47
Training size only, k = 587.18 (−0.59)87.10 (−0.73)88.64 (−0.57)80.97 (−1.50)
Training size only, k = 386.75 (−1.01)87.29 (−0.54)87.82 (−1.40)80.42 (−2.05)
Identification only, k = 587.50 (−0.26)86.66 (−1.16)88.62 (−0.60)81.95 (−0.52)
Identification only, k = 387.53 (−0.24)85.36 (−2.47)85.92 (−3.29)80.27 (−2.20)
Both, k = 587.55 (−0.21)85.19 (−2.63)89.13 (−0.09)82.34 (−0.13)
Both, k = 387.71 (−0.05)84.47 (−3.36)85.84 (−3.37)81.13 (−1.35)
  • With k = 5, each factor alone changes AUROC by at most 1.50 points. The full protocol stays within 0.21 points in three settings and is 2.63 points lower on Qwen/TyDiQA-GP, the smallest dataset (seed standard deviation 2.0). 5–8% of model-known questions are no longer identified.
  • With k = 3, 11–17% of model-known questions are no longer identified. Identification becomes the larger factor, lowering AUROC by up to 3.37 points.

The protocol is therefore stable down to five responses per question. Appendix S reports the known-question counts and train/test sizes for every row.

We hope these additions address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the assessment.

I thank the authors for their detailed responses and the comprehensive experimental results presented in the rebuttal. Although I remain unconvinced that Tucker2 offers a significant advantage over simpler clustering methods, the paper provides useful insights into the effectiveness of low-rank multilinear decomposition for hallucination detection. I have therefore decided to raise my score from 2 to 3.

We thank the reviewer for engaging with the new results and for raising the score. We agree that, with all training data, Tucker-2 does not outperform simpler reductions: it matches flat PCA (Table 8). The revised paper therefore states its advantages as about 40 times fewer fitted parameters and a lead over flat PCA that averages 0.9–2.7 AUROC points when 5–50% of the labelled training questions are available (Table 11), not higher accuracy with all labels.

Reviewer_fgjaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows a white box hallucination detection method based on taking all hidden representations of all tokens across all layers of the examined model. Preprocessing by reducing the response length dimension to a constant via statistical values (with no very convincing motivation) and then compressing to an even smaller representation. Then learning a detector on this compact representation.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The paper studies a very timely and important problem.
Critical weaknesses
  1. There is no consistent improvement over the strongest baseline (ACT-ViT). LOTUS gets better results in 6 out of 8 cases, but in two of them the improvement is only 0.18 and 0.12 AUC points, which is almost no change. On TriviaQA, ACT-ViT gets better results with both models, by 1.08 and 3.11 points. There are good improvements in some cases, but in others the method gives almost the same results or worse. Also, the experiments include only two models of similar size and four question-answering datasets. This makes it hard to know how useful the method would be for other models and tasks. Generally, the improvement in detection alone is not enough to make the contribution strong.

  2. The storage efficiency improvement also needs a better comparison. The paper uses ACT-ViT with fewer tokens and still shows that LOTUS needs less storage. However, I believe that ACT-ViT can also use fewer layers and even fewer tokens. It would be useful to compare both methods when they use about the same amount of storage. This is important because on Triviaqa, LOTUS uses less storage but also gets worse results. The current comparison does not make it clear how much LOTUS is better than simply making the ACT-ViT representation smaller.

Questions for authors
  1. What is the real motivation for the feature selection method?
  2. Minor note: in table 2 in LlamaXTruthfulQA cell you highlight in bold a score that is not the strongest.
4 comments
AuthorsOpenReview ↗

Response to Reviewer fgja (1/2)

We thank the reviewer for the review and for noting that the paper is clear and the problem timely. Table and appendix numbers refer to the revised PDF.

W1. No consistent improvement over ACT-ViT; detection gains alone are not enough.

We agree with the reading of the numbers and have revised the wording.

  • Pooled AUROC: LOTUS is higher in 6 of 8 settings. Two of those margins are within 0.2 points (Qwen NQ-Open, LLaMA TruthfulQA), and ACT-ViT is higher on TriviaQA by 1.08 and 3.11 points.
  • Within-question AUROC, which compares responses to the same question: LOTUS is higher in 3 of 4 settings (Table 2; the revision corrects the bold the reviewer pointed out).

The revision describes LOTUS as comparable to or better than ACT-ViT, not consistently better.

Beyond detection accuracy, the revision makes four contributions explicit:

  1. The evaluation protocol, moved into the main text. With 10 sampled responses per question, an answer-level split places 92.6–95.7% of model-known questions in both the training and the test set. Every trained detector gains from this overlap: HARP by 3.75–13.30 AUROC points, ACT-ViT by 0.82–5.29, LOTUS by 2.36–11.85. The overlap also reverses the HARP/ACT-ViT ordering on TyDiQA-GP for both models. The released HARP and ACT-ViT code splits at the answer level: on the same generations and labels, HARP's released code reaches 85.16 on Qwen/TruthfulQA under its own split and 77.54 under a question-level split (Appendix O).
  2. Accuracy with few labels, a practical gain over the strongest baseline (new, Appendix T). The representation is learned without labels. With 5–10% of the labelled training questions, LOTUS exceeds ACT-ViT by 4.42–28.22 AUROC points in all four settings, consistently across seeds. In deployment, labelled hallucinations are the scarce resource.
  3. Lower training cost. LOTUS fits on a CPU in 2.4–2.6 minutes, with a 1.15M-parameter projection; ACT-ViT trains for 7.4–8.4 minutes on a GPU (Appendix V).
  4. Detector transfer across architectures without target labels. It now includes a third architecture family (below).

Models and tasks. We added Falcon-H1-7B, a hybrid Mamba-2/attention model with different depth and hidden size. All settings were fixed in advance.

  • In-model LOTUS AUROC is 89.69 on TruthfulQA and 90.21 on TyDiQA-GP. Some of Falcon's sampled responses are empty (the first generated token is a stop token); as for the other models, these count as hallucinated (Appendix W gives the rate).
  • Transfer between Falcon and each of Qwen and LLaMA, without target labels, stays within 0.40–6.24 AUROC points of the target-trained detector in 7 of 8 directions, comparable to Qwen ↔ LLaMA (1.78–6.03). The exception is LLaMA → Falcon on TyDiQA-GP (12.19 points).

We agree that larger models and non-QA tasks would strengthen the evaluation. The revision lists them as a limitation; they were not feasible within the discussion period.

Minor. Corrected in the revision: in Table 2, LLaMA × TruthfulQA, ACT-ViT's 83.50 is the higher value. Thank you.

AuthorsOpenReview ↗

Response to Reviewer fgja (2/2)

W2. Storage comparison at a matched budget.

We agree, and we ran it (new, Appendix U). Per response, LOTUS stores 43·D values. ACT-ViT stores (layer slots) × (token slots) × D values: 160·D at the 8 × 20 setting reported in the paper. We evaluated ACT-ViT at two settings with 40·D values, slightly below LOTUS:

  • fewer tokens: 8 layer slots × 5 token slots;
  • fewer layers and tokens: 4 × 10. This needs a new extraction, so it was not run on TriviaQA (99,600 responses per model).

The 8 × 5 inputs are derived exactly from the stored 8 × 20 tensors, because ACT-ViT's max-pooling blocks nest. Question-level pooled AUROC (%), mean over 5 seeds:

Qwen TQAQwen TyDiQwen TriviaQALLaMA TQALLaMA TyDiLLaMA TriviaQA
LOTUS (43·D)87.8387.8691.6989.1982.4984.69
ACT-ViT 8 × 20 (160·D)85.4082.8292.7789.0779.4387.80
ACT-ViT 8 × 5 (40·D)84.3759.6892.5488.7877.8286.46
ACT-ViT 4 × 10 (40·D)84.5572.20—89.9179.87—

Our responses are short (median 6–17 tokens across models and datasets), so most of ACT-ViT's 20 token slots hold zero padding, and the paper's storage comparison overstated LOTUS's advantage:

  • At LOTUS's storage, ACT-ViT loses at most 1.61 AUROC points in every setting except Qwen/TyDiQA-GP. There it falls by 23.14 (8 × 5) and 10.62 (4 × 10) points and varies widely across seeds.
  • The ranking does not change. At 8 × 5, the two detectors rank as at 8 × 20 in all six settings: LOTUS is higher on TruthfulQA and TyDiQA-GP, and ACT-ViT on TriviaQA. At 4 × 10, ACT-ViT is also higher on LLaMA/TruthfulQA, by 0.72 points, within one seed standard deviation.
  • The revision removes storage from LOTUS's claimed advantages.

The full cost comparison is in Appendix V:

  • LOTUS fits on a CPU in 2.4–2.6 minutes, while ACT-ViT trains for 7.4–8.4 minutes on a GPU;
  • scoring costs are equal;
  • LOTUS's token-quantile pooling costs more per response than ACT-ViT's max-pooling (4–18 ms against 0.4 ms on GPU).

Q. Motivation for the summaries.

  • Responses vary in length, so the token mode must be reduced to a fixed size before any decomposition.
  • The per-layer peak retains short, localized activations that averaging would dilute.
  • The quantile range retains their spread across tokens.
  • The cross-layer difference retains the layer-to-layer update of each token's representation.
  • Each summary alone carries most of the signal, and their combination is the most consistent across settings (Table 3).

Averaging over tokens performs comparably (Table 10).

Thank you for the detailed response and for taking the time to run the additional experiments. I appreciate the authors’ efforts in addressing my comments. However, I am not sure that my two main concerns have been fully resolved: (i) the improvement over ACT-ViT still appears to be limited, and (ii) if I understand the results correctly, this also seems to hold when the methods are compared at a matched budget.

We thank the reviewer for reading the new results closely. The reviewer's reading is correct on both points, and the revised paper states them the same way:

  • With all training questions, LOTUS is higher than ACT-ViT in six of eight settings, by 0.12–5.04 AUROC points, and ACT-ViT is higher on TriviaQA, by 1.08 and 3.11 points (Section 4.2).
  • At the storage budget of LOTUS, ACT-ViT loses at most 1.61 points except on Qwen/TyDiQA-GP, and at 8 × 5 the ranking does not change (Appendix U).

The revision therefore describes LOTUS as comparable to or better than ACT-ViT and no longer claims a storage advantage. Its claims rest on three other results:

  1. Accuracy with few labels (Appendix T). With 5–10% of the labelled training questions, LOTUS exceeds ACT-ViT by 4.42–28.22 AUROC points in all four settings, and in every seed. The representation of LOTUS is learned without labels, and labelled hallucinations are the costly resource in practice.
  2. The evaluation protocol (Section 4.2). The answer-level split used by the released HARP and ACT-ViT code inflates every trained detector, by up to 13.30 AUROC points, and reverses the ordering of HARP and ACT-ViT on TyDiQA-GP for both models.
  3. Transfer without target labels (Appendices J and W), now across three model families, including a hybrid Mamba-2/attention model.

We would be grateful if the reviewer would weigh these revised claims in the final assessment.

Reviewer_pjMeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LOTUS, a low-rank multilinear framework for detecting hallucinations from LLM hidden states. The method uses peak activations, activation quantiles, and cross-layer differences, and then applies Tucker-2 decomposition to reduce dimensionality. It only requires training a lightweight random forest classifier. The method is compared with six label-free methods and two supervised methods using two language models and four question-answering benchmarks.

Critical strengths
  1. The paper proposes a structured low-rank representation of LLM hidden states that combines complementary activation summaries and Tucker-2 projection.
  2. The paper compares LOTUS with a wide range of baselines, providing solid empirical evidence for its effectiveness.
Critical weaknesses
  1. Neither the comparison with PCA nor the use of a lightweight classifier directly supports the abstract's claim of reducing dependence on labeled data.
  2. The transfer experiments involve only two LLMs, limiting the evidence for the method's generalizability across a broader range of models and architectures.
Questions for authors
  1. Although LOTUS achieves promising empirical performance, could the authors further explain why the proposed statistical summaries and Tucker-2 decomposition are effective for hallucination detection, and provide evidence that the learned low-rank representations preserve hallucination-discriminative information?
  2. Why do the authors make the assumption that “Our approach is motivated by the hypothesis that hallucination-discriminative information can be captured by compact, low-rank multilinear representations of LLM hidden states.”
  3. Could the authors clarify the computational cost of learning the cross-model feature mapping?
  4. In the “Performance with limited labeled data” section, the authors examine Tucker-2 under reduced training data. Could the authors also compare supervised baselines under varying amounts of labeled data to further assess LOTUS's dependence on training sample size?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pjMe (1/2)

We thank the reviewer for the thoughtful questions. Each is answered below with new experiments; table and appendix numbers refer to the revised PDF.

W1 / Q4. Dependence on labelled data, and supervised baselines with fewer labels.

We agree that the comparison with PCA alone did not support the claim against other detectors. The claim has two parts:

  • the representation (robust scaling and both Tucker-2 bases) is learned from unlabelled hidden states, and labels are used only by the final classifier;
  • as a consequence, the detector remains accurate when labelled questions are scarce.

The second part requires a supervised baseline. As the reviewer suggests, we trained ACT-ViT on 5–100% of the training questions (new, Appendix T). The subsets are identical to those LOTUS used in Table 11: the same subsampling function and seeds, with whole questions kept or removed and the test sets fixed. Pooled AUROC (%), LOTUS / ACT-ViT, mean over 5 seeds:

Training questions5%10%25%50%100%
Qwen TQA77.10 / 63.4081.32 / 69.3784.89 / 78.5586.12 / 84.0587.76 / 85.40
Qwen TyDi74.05 / 47.4984.00 / 55.7885.01 / 71.4086.31 / 64.5887.82 / 82.82
LLaMA TQA81.60 / 74.5379.51 / 75.0984.02 / 81.4587.49 / 87.2989.22 / 89.07
LLaMA TyDi68.48 / 50.0773.82 / 54.3077.38 / 62.0279.84 / 70.5682.47 / 79.43
  • LOTUS is more accurate at every fraction in all four settings.
  • With 5–10% of the training questions, the margin is 4.42–28.22 AUROC points, and LOTUS is higher in all 5 seeds of every such setting.
  • With all training questions, the margin is 0.15–5.00 points.
  • ACT-ViT's values at 100% reproduce Table 1.

The revised abstract states the claim with these numbers.

W2. Transfer experiments with only two LLMs.

We added Falcon-H1-7B, a hybrid Mamba-2/attention model from a different architecture family. It has 44 layers and hidden size 3072, against 3584 (Qwen) and 4096 (LLaMA). The layer window and all settings were fixed before any Falcon result was computed.

  • In-model LOTUS: AUROC is 89.69 ± 0.53 on TruthfulQA and 90.21 ± 0.95 on TyDiQA-GP. Some of Falcon's sampled responses are empty (the first generated token is a stop token); as for the other models, these count as hallucinated (Appendix W gives the rate).
  • Transfer without target labels, in both directions with each of Qwen and LLaMA (new, Appendix W). Pooled AUROC (%) on the target's test responses, mean over 5 seeds, with the target-trained detector of the same runs in parentheses:
Source → targetTruthfulQATyDiQA-GP
Falcon → Qwen84.24 (87.76)82.26 (87.82)
Qwen → Falcon89.30 (89.70)83.98 (90.22)
Falcon → LLaMA87.90 (89.22)77.19 (82.47)
LLaMA → Falcon88.11 (89.70)78.03 (90.22)
  • Aligned transfer stays within 0.40–6.24 points of the target-trained detector in 7 of the 8 directions, comparable to Qwen ↔ LLaMA in the paper (1.78–6.03).
  • The exception is LLaMA → Falcon on TyDiQA-GP (12.19 points). There, LLaMA reading Falcon's responses itself scores no higher (77.03), so the gap lies in the source detector, not in the mapping.
  • Without alignment, AUROC falls to 39.03–66.62.
  • In all 12 directions, the mapped detector scores within −2.03 to +3.86 points of the source model reading the responses itself, without running the source model at test time.
AuthorsOpenReview ↗

Response to Reviewer pjMe (2/2)

Q1. Why the summaries and Tucker-2 are effective, and whether the discriminative information is preserved.

  • Summaries. Responses vary in length, so the token mode must be reduced to a fixed size. The three summaries retain complementary aspects of each layer's activations across tokens:

    • the peak retains short, localized activations that averaging would dilute;
    • the quantile range retains the spread of activations across tokens;
    • the cross-layer difference retains the layer-to-layer update of each token's representation (cf. ICR Probe).

    Each summary alone carries most of the signal, and their combination is the most consistent configuration across settings (Table 3).

  • Tucker-2. Each summary is a layer × feature matrix. Tucker-2 learns separate layer and feature bases, so the projection requires LrL+FrFL r_L + F r_F parameters instead of LFrLrFL F r_L r_F (about 40× fewer here). This is what keeps it accurate when labelled data are scarce (Table 11).

  • Preservation (new, Appendix R). We trained the same classifier on the retained Tucker-2 core and on the discarded residual R=X−ULUL⊤XUFUF⊤R = X - U_L U_L^\top X U_F U_F^\top.

    • Adding the residual to the retained core changes AUROC by −0.34 to +0.08 points, within one seed standard deviation.
    • The retained core exceeds the residual's best same-size core in all four settings, by 0.35–1.42 points.
    • A random projection of equal width lowers AUROC by 2.44–5.23 points, and feature directions beyond the 64th add nothing (Table 13).

Q2. The low-rank hypothesis.

The hypothesis follows prior evidence that truthfulness and hallucination information is encoded along a small number of directions in LLM hidden states (Azaria & Mitchell, 2023; Marks & Tegmark, 2024; Li et al., 2023; Du et al., 2024; Hu et al., 2026). The paper treats it as a hypothesis and tests it in three ways:

  • the covariance spectra: the leading 64 of 3,584 directions explain 52–66% of the variance;
  • the rank sweep: AUROC plateaus between rF=32r_F = 32 and 128128;
  • the new retained-versus-residual experiment.

That experiment also refines the hypothesis. The residual's own leading directions remain informative, while a random projection of the residual lowers AUROC by 7.7–11.3 points. The signal is therefore spread redundantly across many directions rather than confined to a few, and this redundancy is why a compact projection loses nothing. The revised text states the hypothesis in these terms.

Q3. Computational cost of learning the cross-model mapping.

New, Appendix V; TruthfulQA, Qwen ↔ LLaMA.

  • The map is a ridge regression from the target's 896 features to the source's, with 805,504 parameters and the penalty chosen by 5-fold question-grouped cross-validation over 11 values.
  • The data are the target's unlabelled training responses, read by both models (3,790–4,990 responses on TruthfulQA).
  • Fitting takes 24–30 s on CPU, including the cross-validation.
  • Applying the map takes about 10 µs per response.
  • The dominant one-time cost is the source model reading the target's training responses: 34–51 ms per response on one GPU, or 2–4 minutes for TruthfulQA.

No label of the target model is used at any stage.

Dear Reviewer pjMe, as the discussion period closes on November 18, we would be grateful to know whether our responses address your questions, in particular the comparison with ACT-ViT when fewer labels are available (W1/Q4) and transfer to a third model family (W2). We are glad to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading. The reviews raise four common points:

  1. what the multilinear structure provides beyond simpler reductions;
  2. measured cost rather than storage;
  3. evidence beyond two models;
  4. the motivation for the summaries and for low rank.

We have revised the paper and added experiments for each. The revised PDF contains the following changes.

  1. Appendices in the main PDF. They were submitted as a separate supplementary archive (Appendices A–Q), which remains attached unchanged. They now follow the references; only Appendices F, J and K differ from the archive (items 2 and 5).
  2. Precise claims.
    • At equal width, Tucker-2 matches flat PCA in AUROC with full training data.
    • It requires about 40× fewer fitted parameters and is more accurate when labelled training questions are scarce.
    • The representation is learned without labels; only the classifier uses them.
    • With 5–10% of the labelled training questions, LOTUS exceeds ACT-ViT by 4.4–28.2 AUROC points in all four settings.
    • Against ACT-ViT, LOTUS is described as comparable or better, not consistently better.
    • Storage is no longer claimed as an advantage: ACT-ViT reduced to LOTUS's storage loses at most 1.61 AUROC points in all but one setting (U). Appendix K is revised accordingly.
  3. New experiments (Appendices R–W):
    • (R) detection from the retained subspace versus the discarded residual;
    • (S) sensitivity to the number of sampled responses, separating training-set size from the identification of model-known questions;
    • (T) ACT-ViT trained on 5–100% of the labelled questions, with the same subsets as LOTUS;
    • (U) ACT-ViT at LOTUS's storage budget, with fewer tokens and with fewer layers;
    • (V) measured runtime and peak memory for extraction, fitting and per-response scoring, and the cost of the cross-model mapping;
    • (W) a third LLM from a different architecture family, Falcon-H1-7B (hybrid Mamba-2/attention), with in-model detection and cross-model transfer in both directions with each of the other two models.
  4. The evaluation-protocol finding in the main text. Splitting at the answer level, as the released HARP and ACT-ViT code does, places 92.6–95.7% of model-known questions in both the training and the test set. This inflates the AUROC of every trained detector by 0.82–13.30 points and reverses the HARP/ACT-ViT ordering on TyDiQA-GP.
  5. Corrections. The bold in Table 2, typos in the abstract, two broken references (Appendices F and J), and Appendix J's description of Table 4, which does not include the source model reading the target's responses; Table 28 now reports this for all twelve directions.

The individual responses give the numbers for each point.

AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

We thank the Area Chair and the reviewers. This note summarizes the discussion and the revised paper.

Changes in the revision. The appendices, previously a separate supplementary archive, now follow the references, and Appendices R–W report the new experiments. The claims are narrower: LOTUS is described as comparable to or better than ACT-ViT, and storage is no longer claimed as an advantage.

Reviewer positions after the discussion.

  • zjPm raised the score from 2 to 3. The remaining reservation is the advantage of Tucker-2 over simpler reductions. We agree that it matches flat PCA with all training data. The revision claims about 40 times fewer fitted parameters and an average lead of 0.9–2.7 AUROC points with 5–50% of the training questions (Table 11).
  • pjMe acknowledged the empirical contributions and practical value and kept the score, citing the stability of the advantages and the limited theory for the low-rank hypothesis. Our reply reports run-level counts and sensitivity results. We agree that the paper gives no theoretical guarantee; low rank is stated and tested as an empirical hypothesis.
  • fgja's remaining concern is that the improvement over ACT-ViT is limited, including at a matched storage budget. We agree with this reading, and the revised paper states it. The reviewer's reply does not comment on the three results below, on which the revised claims rest.

What the revised paper claims.

  1. Accuracy with few labels (Appendix T). With 5–10% of the training questions, LOTUS exceeds ACT-ViT by 4.42–28.22 AUROC points in all four settings, and is higher in 40 of 40 runs. Its representation is learned without labels.
  2. An evaluation-protocol finding (Section 4.2). With 10 responses per question, an answer-level split, as in the released HARP and ACT-ViT code, places 92.6–95.7% of model-known questions in both the training and the test set. This inflates the AUROC of every trained detector, by up to 13.30 points, and reverses the HARP/ACT-ViT ordering on TyDiQA-GP for both models.
  3. Detector transfer without target labels (Appendices J and W). We added a hybrid Mamba-2/attention model. Aligned transfer stays within 0.40–6.24 AUROC points of the target-trained detector in 7 of 8 new directions. The exception is LLaMA to Falcon on TyDiQA-GP (12.19 points).

We thank the Area Chair for considering these points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.