- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presented a structure-level intervention framework to mitigate hallucinations in LLMs by localizing hallucination-related units via contrastive attribution and causal activation patching, aggregating them to the layer level, and modifying block-level execution trajectories. Experiments across several small-size models and three benchmarks were conducted in both intra-task and cross-task transfer settings to compare structure-level operations against unit-level interventions.
Critical strengths
-
This paper investigated an important granularity question in mechanistic interpretability: whether layer-level interventions offer a better trade-off between behavioral control and general capability preservation than unit-level ablations.
-
Experimental evaluation spans 5 models and includes out-of-distribution benchmarks as well as random perturbation controls to test cross-task robustness.
Critical weaknesses
-
The primary TruthfulQA evaluation is conducted under a hallucination-only prompt where all candidate options are incorrect, so reducing the probability of the initial argmax hallucinated token only measures output distribution flattening or shifting between false answers rather than true hallucination mitigation.
-
The cross-task transfer comparison in Figure 2 and Appendix K is unfair because unit-level transfer overwrites target-task, i.e., CoLA, activations with raw activation values recorded from unrelated source-task, i.e., TruthfulQA, prompts, whereas structure-level transfer only transfers the layer index without corrupting target activations.
-
There are severe contradictions and potential oracle leaks in target layer selection: Appendix D.1 defined target layer selection per test sample using ground-truth contrastive prompts, and Section 6.3 claimed Stage 3 selects Layer 10 for Pythia-2.8B on TruthfulQA, whereas Appendix J.1 and Figure 5d state that Pythia-2.8B peaks in the deepest layers.
-
Several critical implementation details were missing or inconsistent with the figures, including how the auxiliary layer is chosen for Swap and Insert, and how contrastive prompts are constructed for CoLA and ASDiv to achieve implausible intra-task accuracies, such as 99.25% on CoLA and 92.93% on ASDiv with GPT-2.
-
The unit-type distribution analysis in Appendix J.2 didn’t normalize by the total number of units per component type, reflecting the fixed 4-to-1 architectural ratio of MLP neurons to residual stream dimensions rather than a mechanistic finding.
Questions for authors
-
Under the hallucination-only prompt setting in TruthfulQA, how does the model perform on standard multiple-choice accuracy after applying Freeze, Swap, and Insert?
-
How does unit-level cross-task transfer perform on CoLA and ASDiv when implemented via standard zero-ablation or mean-ablation rather than overwriting target activations with values from TruthfulQA prompts?
-
Was the target layer selected globally on a held-out validation split or per test sample using oracle prompts, and what accounts for the discrepancy between selecting Layer 10 for Pythia-2.8B in Table 2 compared to the deepest-layer peak shown in Figure 5d?
-
What exact rule is used to select the auxiliary layer for Swap and Insert, and how were the prompts and interventions constructed on CoLA and ASDiv without leaking ground-truth test labels?
-
It would be helpful to normalize the unit-type distribution by the baseline parameter counts of each module type and evaluate open-ended generation fluency alongside truthfulness.
We thank the reviewer for these comments and address each point below.
1. TruthfulQA: Accuracy and Hallucination Mitigation
(1) Rationale for separating correct and incorrect options. In our preliminary TruthfulQA experiments, we presented correct and incorrect options together. The model selected a correct answer on roughly half of the questions, leaving only the other half available for extracting activations associated with incorrect predictions. To retain every question for localization, we constructed two prompts per question: one with only correct options and one with only incorrect options. We extracted activations separately under each condition.
(2) Motivation for the controlled prompts. Our design is inspired by Li et al. (2023), who compared activations for truthful and false answers on TruthfulQA, and Rimsky et al. (2024), who compared activations for the same question paired with contrasting answers. We keep each question fixed and present either only correct or only incorrect options. This lets us compare activations under both conditions, even for questions the model initially answers correctly.
(3) Causal relevance beyond the localization prompts. Our paired prompts restrict the available answers during activation extraction. To test whether the blocks identified under this constraint also influence behavior in other settings, we applied TQA-selected interventions to CoLA (Figure 2) and evaluated free-form generation on TruthfulQA without candidate options (Appendix H, Table 6).
All three interventions improve GPT2-S CoLA accuracy and open-ended TQA truthfulness on both models. Freeze yields the largest truthfulness gains: +0.264 for Pythia-2.8B and +0.290 for LLaMA3-8B. These gains occur without predefined options, supporting the causal relevance of the localized blocks beyond the original prompts. We will discuss the limits of using restricted options for localization in the paper.
2. Cross-Task Transfer with Zero Ablation
(1) Zero ablation without source activations. We added a zero-ablation control on CoLA and ASDiv: we set the TQA-localized units to zero during evaluation, without using any TQA donor activations.
(2) Why this motivates FSI. Units associated with one behavior may also support computations needed by other tasks. In Dissecting Bias in LLMs (Table 3), intervening on bias-related edges reduced GPT2-S’s DSS1 bias score by 35.88%, while CoLA and CoNLL-2003 performance fell by 22.60% and 20.40%, respectively. Arditi et al. (Table 8) likewise found that removing a refusal-related direction reduced LLaMA3-8B’s TruthfulQA score from 51.7 to 48.3.
This motivates FSI: we use localized units to identify blocks and transfer the resulting structural configuration. In Figure 2, all three TQA-selected configurations improve GPT2-S CoLA accuracy over the 0.6600 baseline: 0.7140 for Freeze and 0.7040 for both Swap and Insert.
3. Target-Layer Selection and Figure 5
We select target layers on the source set and keep them fixed during evaluation and cross-benchmark transfer, without using evaluation labels for re-selection.
Figure 5 pools unit frequencies across TQA, CoLA, and ASDiv. We will correct the anomaly in the Pythia panel and clarify its description. Table 2 is TQA-specific: for Pythia-2.8B, Stage 3 selected Layer 10 on TQA, and the subsequent all-layer Freeze comparison showed the largest reduction in hallucinated-answer probability at that layer (0.4001 → 0.0838).
4. Auxiliary-Layer Selection
On the source set, Stage 3 identifies the target layer
l*. Freeze needs no auxiliary layer. For Swap and Insert separately, we keepl*fixed, test every other block, and select the one that minimizes hallucinated-answer probability on TQA or maximizes classification accuracy on CoLA or exact-match accuracy on ASDiv. We apply each configuration unchanged to the evaluation set; its scores are never used for selection.5. Unit-Type Normalization
Appendix J.2, Figure 6 reports the original shares. We present parameter-normalized shares over MLP and attention below: