acceptodds
Under review as a conference paper at ICLR 2027

From Units to Layers: Structural Intervention for LLM Hallucination

Abstract

Large language models (LLMs) are known to exhibit hallucination, producing fluent but factually incorrect outputs. Existing approaches largely treat hallucination as a black-box phenomenon, offering limited internal insight. Prior mechanistic studies of hallucinations have focused on individual computational units, whose ablation tends to suppress the targeted behavior at the cost of performance on unrelated tasks. We argue that this cost is structural. Individual units may participate in many functions at once, which removal almost necessarily perturbs unrelated computation. In this work, we propose a structure-level alternative. We localize hallucination-related units via attribution and causal validation, and map them onto the model's layer-wise structure. It reveals that hallucination concentrates in specific structural regions that are stable across tasks. Building on this insight, we introduce a set of structure-level intervention operators, including Freeze, Swap and Insert, to modify layer-wise computation while preserving the overall model structure. To our knowledge, this is the first systematic class of structure-level operations for hallucination mitigation in LLMs. We systematically evaluate these operators against unit-level ablation in both intra-task and cross-task settings, spanning benchmarks for knowledge-based question answering, grammatical acceptability, and reasoning. Our solution achieves comparable to or stronger performance than the state-of-the-art, indicating that hallucination is governed by higher-level computational organization rather than by isolated units. It highlights structure-level intervention is a more principled basis for behavioral control in LLMs.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Language Models, Hallucination, Causal Intervention, Activation Patching, Structural Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_73pFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presented a structure-level intervention framework to mitigate hallucinations in LLMs by localizing hallucination-related units via contrastive attribution and causal activation patching, aggregating them to the layer level, and modifying block-level execution trajectories. Experiments across several small-size models and three benchmarks were conducted in both intra-task and cross-task transfer settings to compare structure-level operations against unit-level interventions.

Critical strengths
  1. This paper investigated an important granularity question in mechanistic interpretability: whether layer-level interventions offer a better trade-off between behavioral control and general capability preservation than unit-level ablations.

  2. Experimental evaluation spans 5 models and includes out-of-distribution benchmarks as well as random perturbation controls to test cross-task robustness.

Critical weaknesses
  1. The primary TruthfulQA evaluation is conducted under a hallucination-only prompt where all candidate options are incorrect, so reducing the probability of the initial argmax hallucinated token only measures output distribution flattening or shifting between false answers rather than true hallucination mitigation.

  2. The cross-task transfer comparison in Figure 2 and Appendix K is unfair because unit-level transfer overwrites target-task, i.e., CoLA, activations with raw activation values recorded from unrelated source-task, i.e., TruthfulQA, prompts, whereas structure-level transfer only transfers the layer index without corrupting target activations.

  3. There are severe contradictions and potential oracle leaks in target layer selection: Appendix D.1 defined target layer selection per test sample using ground-truth contrastive prompts, and Section 6.3 claimed Stage 3 selects Layer 10 for Pythia-2.8B on TruthfulQA, whereas Appendix J.1 and Figure 5d state that Pythia-2.8B peaks in the deepest layers.

  4. Several critical implementation details were missing or inconsistent with the figures, including how the auxiliary layer is chosen for Swap and Insert, and how contrastive prompts are constructed for CoLA and ASDiv to achieve implausible intra-task accuracies, such as 99.25% on CoLA and 92.93% on ASDiv with GPT-2.

  5. The unit-type distribution analysis in Appendix J.2 didn’t normalize by the total number of units per component type, reflecting the fixed 4-to-1 architectural ratio of MLP neurons to residual stream dimensions rather than a mechanistic finding.

Questions for authors
  1. Under the hallucination-only prompt setting in TruthfulQA, how does the model perform on standard multiple-choice accuracy after applying Freeze, Swap, and Insert?

  2. How does unit-level cross-task transfer perform on CoLA and ASDiv when implemented via standard zero-ablation or mean-ablation rather than overwriting target activations with values from TruthfulQA prompts?

  3. Was the target layer selected globally on a held-out validation split or per test sample using oracle prompts, and what accounts for the discrepancy between selecting Layer 10 for Pythia-2.8B in Table 2 compared to the deepest-layer peak shown in Figure 5d?

  4. What exact rule is used to select the auxiliary layer for Swap and Insert, and how were the prompts and interventions constructed on CoLA and ASDiv without leaking ground-truth test labels?

  5. It would be helpful to normalize the unit-type distribution by the baseline parameter counts of each module type and evaluate open-ended generation fluency alongside truthfulness.

1 comment

We thank the reviewer for these comments and address each point below.


1. TruthfulQA: Accuracy and Hallucination Mitigation

(1) Rationale for separating correct and incorrect options. In our preliminary TruthfulQA experiments, we presented correct and incorrect options together. The model selected a correct answer on roughly half of the questions, leaving only the other half available for extracting activations associated with incorrect predictions. To retain every question for localization, we constructed two prompts per question: one with only correct options and one with only incorrect options. We extracted activations separately under each condition.

(2) Motivation for the controlled prompts. Our design is inspired by Li et al. (2023), who compared activations for truthful and false answers on TruthfulQA, and Rimsky et al. (2024), who compared activations for the same question paired with contrasting answers. We keep each question fixed and present either only correct or only incorrect options. This lets us compare activations under both conditions, even for questions the model initially answers correctly.

(3) Causal relevance beyond the localization prompts. Our paired prompts restrict the available answers during activation extraction. To test whether the blocks identified under this constraint also influence behavior in other settings, we applied TQA-selected interventions to CoLA (Figure 2) and evaluated free-form generation on TruthfulQA without candidate options (Appendix H, Table 6).

EvaluationModelBaselineFSI
TQA → CoLA: accuracy (Fig. 2)GPT2-S0.66000.71400.70400.7040
Open-ended TQA: truthfulness (Table 6)Pythia-2.8B0.0230.2870.1480.236
Open-ended TQA: truthfulness (Table 6)LLaMA3-8B0.0800.3700.2200.180

All three interventions improve GPT2-S CoLA accuracy and open-ended TQA truthfulness on both models. Freeze yields the largest truthfulness gains: +0.264 for Pythia-2.8B and +0.290 for LLaMA3-8B. These gains occur without predefined options, supporting the causal relevance of the localized blocks beyond the original prompts. We will discuss the limits of using restricted options for localization in the paper.

2. Cross-Task Transfer with Zero Ablation

(1) Zero ablation without source activations. We added a zero-ablation control on CoLA and ASDiv: we set the TQA-localized units to zero during evaluation, without using any TQA donor activations.

ModelChange in CoLA accuracyChange in ASDiv exact match
GPT2-S−0.1751−0.1018
LLaMA3-8B−0.0098−0.0015

(2) Why this motivates FSI. Units associated with one behavior may also support computations needed by other tasks. In Dissecting Bias in LLMs (Table 3), intervening on bias-related edges reduced GPT2-S’s DSS1 bias score by 35.88%, while CoLA and CoNLL-2003 performance fell by 22.60% and 20.40%, respectively. Arditi et al. (Table 8) likewise found that removing a refusal-related direction reduced LLaMA3-8B’s TruthfulQA score from 51.7 to 48.3.

This motivates FSI: we use localized units to identify blocks and transfer the resulting structural configuration. In Figure 2, all three TQA-selected configurations improve GPT2-S CoLA accuracy over the 0.6600 baseline: 0.7140 for Freeze and 0.7040 for both Swap and Insert.

3. Target-Layer Selection and Figure 5

We select target layers on the source set and keep them fixed during evaluation and cross-benchmark transfer, without using evaluation labels for re-selection.

Figure 5 pools unit frequencies across TQA, CoLA, and ASDiv. We will correct the anomaly in the Pythia panel and clarify its description. Table 2 is TQA-specific: for Pythia-2.8B, Stage 3 selected Layer 10 on TQA, and the subsequent all-layer Freeze comparison showed the largest reduction in hallucinated-answer probability at that layer (0.4001 → 0.0838).

4. Auxiliary-Layer Selection

On the source set, Stage 3 identifies the target layer l*. Freeze needs no auxiliary layer. For Swap and Insert separately, we keep l* fixed, test every other block, and select the one that minimizes hallucinated-answer probability on TQA or maximizes classification accuracy on CoLA or exact-match accuracy on ASDiv. We apply each configuration unchanged to the evaluation set; its scores are never used for selection.

5. Unit-Type Normalization

Appendix J.2, Figure 6 reports the original shares. We present parameter-normalized shares over MLP and attention below:

ModelMLP (original → normalized)Attention (original → normalized)
GPT2-S/M/L79.75% → 88.25%5.31% → 11.75%
Pythia-2.8B76.80% → 87.97%5.25% → 12.03%
LLaMA3-8B-Instruct77.64% → 85.36%3.17% → 14.64%
Reviewer_MuBbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors observed that many previous methods work on unit-level interventions which inevitably impact unrelated computations and thus impact the overall model capability. To address this problem, this paper proposes a structure-level intervention framework for mitigating hallucination in LLMs. It localizes the hallucination-related units by attribution-based localization and patch-based localization, projects them onto model’s layer-wise structure, and modifies the layer-wise computation by structure-level intervention operations including Freeze, Swap, and Insert. Extensive experiments across multiple models and three tasks show that these structural interventions can reduce hallucination while preserving performance on unrelated tasks.

Critical strengths
  1. This paper innovatively moves from unit-level interventions to a systematic class of structure-level operations for hallucination mitigation in LLMs.
  2. The paper localize the hallucination related units and shows that hallucination-related units concentrate in specific layer-wise regions, providing an empirical motivation for structure-level interventions.
  3. Table 1 supports the effectiveness of the proposed interventions through comprehensive experiments.
Critical weaknesses

1.The work would benefit from averaging the random-control results across multiple runs and reporting their variability. 2.Table 1 shows that unit-level intervention outperforms structure-level operations in several settings. For example, for GPT2-S, unit-level operation outperforms all structure-level operation on CoLA and ASDIV. The paper lacks an deeper analysis of when each intervention granularity is preferable and the trade-off between intra-task effectiveness and cross-task robustness

Questions for authors

What explains the cases where U outperforms structure-level interventions in Table 1, and how does this relate to cross-task robustness?

1 comment

We thank the reviewer for the constructive feedback. We address each concern separately below.


1.Response to the Concern About Random-Control Variability

We thank the reviewer for this suggestion. We extend the matched random controls to all five models and all four intervention families, reporting the mean over 20 independent random-intervention runs in TQA. In each run, we independently sample random unit sets for the same evaluation questions, matching the localized sets in both unit count and unit-type composition. Both sets receive the same activation-patching procedure using correct-condition activations from the corresponding question. For Freeze, Swap, and Insert, we apply the same layer-aggregation rule to the localized and random sets.

ModelInterventionBaselineLocalizedRandom mean
GPT2-SU0.64670.40000.6318
GPT2-SFreeze0.64670.42490.7043
GPT2-SSwap0.64670.32740.4046
GPT2-SInsert0.64670.43900.5589
GPT2-MU0.54930.28210.5786
GPT2-MFreeze0.54930.10210.5938
GPT2-MSwap0.54930.21450.5112
GPT2-MInsert0.54930.33450.5316
GPT2-LU0.49020.31080.4318
GPT2-LFreeze0.49020.13140.6536
GPT2-LSwap0.49020.20680.3637
GPT2-LInsert0.49020.31050.4271
Pythia-2.8BU0.40010.09300.4176
Pythia-2.8BFreeze0.40010.08380.3694
Pythia-2.8BSwap0.40010.15530.3658
Pythia-2.8BInsert0.40010.09590.5897
LLaMA3-8B-InstructU0.76240.13050.5896
LLaMA3-8B-InstructFreeze0.76240.32730.4862
LLaMA3-8B-InstructSwap0.76240.10630.6248
LLaMA3-8B-InstructInsert0.76240.11360.8349

2.Response to the Concern About Intra-Task Performance and Cross-Task Robustness

We thank the reviewer for highlighting this distinction. U’s strong intra-task performance supports the relevance of the identified units to the benchmark on which they were localized. However, effective task-specific intervention does not automatically imply effective cross-task transfer.

This distinction motivates FSI: we use localized causal evidence to identify structural configurations that can be transferred across benchmarks. The following GPT2-S results illustrate why both intra-task effectiveness and cross-task robustness matter:

Localization → EvaluationBaselineUFreezeSwapInsert
CoLA → CoLA, Table 10.66000.94340.83620.78530.8237
ASDIV → ASDIV, Table 10.24000.65470.48860.45700.6260
TQA → CoLA, Figure 20.66000.05300.71400.70400.7040

When localization and evaluation concern the same benchmark, U achieves larger gains in these cases. This advantage does not persist in the evaluated TQA-to-CoLA transfer: U falls below baseline, whereas all three FSI operators retain positive gains.

Related evidence appears in Chandna et al.’s Dissecting Bias in LLMs (2025, Table 3). For GPT-2 Small under DSS1, their intervention on bias-related edges reports a 35.88% reduction in bias, accompanied by performance drops of 22.60% on CoLA and 20.40% on CoNLL-2003 NER. These findings suggest that components involved in one behavior can also support other capabilities. Therefore, an intervention that improves one task may not preserve performance on others.

Figure 2 therefore complements Table 1 by testing transfer beyond the localization benchmark. Under the evaluated procedures, U transfers source-task activations, while FSI transfers structural configurations. The contribution of FSI is to extend the usefulness of source-task localization beyond its original task, achieving stronger cross-task performance in this comparison.

Reviewer_kBJyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes reducing LLM hallucinations by modifying the execution of Transformer layers. It compares internal activations and gradients using prompts with known correct or incorrect answer options. It then uses activation patching to identify components that influence incorrect predictions. These components help locate target layers. The method skips, swaps, or relocates selected blocks to improve model outputs. All model weights remain fixed, and no training is required.

Critical strengths
  1. The modification among multiple components brings better results than modifying a single component.
  2. The method is training-free.
  3. Experiments are meaningful.
Critical weaknesses
  1. The metric it uses cannot specifically prove the reduction in hallucinations.
  2. The paper leaves the auxiliary-layer selection procedure for Swap and Insert unclear. Also,
  3. It is uncertain whether answer texts in prompts can affect the selection process.
Questions for authors
  1. Can one intervention configuration improve unseen questions without access to their correct answers?
  2. How much does localization depend on answer wording and option labels? If you randomize option orders, paraphrase answers, or control answer lengths, do the same layers remain important?
1 comment

We thank the reviewer for the constructive feedback. We address each concern separately below.


1. Hallucination Metric

We includes open-ended TruthfulQA generation in Appendix H, Table 6, where the model generates free-form answers without predefined candidate options. We evaluate the truthfulness of these responses:

ModelBaselineFreezeSwapInsert
Pythia-2.8B0.0230.287 (+0.264)0.148 (+0.125)0.236 (+0.213)
LLaMA3-8B0.0800.370 (+0.290)0.220 (+0.140)0.180 (+0.100)

Results from Appendix H, Table 6. Higher scores indicate greater truthfulness; parentheses report absolute changes from baseline.

All three interventions improve truthfulness on both models, with Freeze producing the largest gains. These results provide evidence of benefits beyond reducing the probability of a selected incorrect answer in the controlled setting. We will highlight these existing results more clearly in the main text.


2. Auxiliary-Layer Selection

We distinguish four terms:

  • Source benchmark: used for target-layer localization and auxiliary-layer selection.
  • Evaluation benchmark: used to evaluate the selected intervention configuration.
  • Target layer: the Transformer block identified through Stage-3 localization.
  • Auxiliary layer: the second block paired with the target layer for Swap or Insert.

With the target layer fixed, we evaluate the eligible auxiliary layers on the source benchmark and select the best-performing pair separately for Swap and Insert. Freeze requires only the target layer.

Table 1 reports intra-benchmark results, with localization and configuration selection performed separately for each benchmark. Figure 2 reports TQA-to-CoLA transfer: the right panel applies TQA-selected structural configurations unchanged to CoLA, without re-selection or re-optimization; the left panel reports unit-level transfer.

Evaluation settingSource benchmarkEvaluation benchmark
Table 1: TQATQATQA
Table 1: CoLACoLACoLA
Table 1: ASDIVASDIVASDIV
Figure 2TQACoLA

The auxiliary-layer selection procedure is summarized below:

Input:
    Model M; source benchmark D_s
    Stage-3 target layer l*; eligible auxiliary layers A
    Source-benchmark score Q_s (higher is better)

For each operator O in {Swap, Insert}:
    For each a in A:
        score[a] = Q_s(O(M; l*, a), D_s)
    a*_O = argmax_a score[a]
    Save C_O = (O, l*, a*_O)

Return the selected configurations.

For TQA, QsQ_s is the negative hallucinated-answer probability, so maximizing it selects the largest reduction from baseline. Other benchmarks use their corresponding evaluation criteria. The target layer remains fixed throughout the search.


3. Fixed Configurations on Unseen Questions

The relevant evidence is provided by Figure 2 (right) and Appendix L, Table 7. We distinguish source-benchmark localization, which uses contrastive correct and incorrect conditions, from applying the resulting configuration. Once fixed, executing FSI does not require the target question's correct answer or correct-condition activations.

In Figure 2 (right), TQA-derived configurations are transferred to CoLA. For GPT2-S, the reported CoLA score increases from 0.6600 to 0.7140, 0.7040, and 0.7040 under Freeze, Swap, and Insert, respectively.

Appendix L evaluates reverse transfer: ASDIV-derived configurations are applied to TQA and CoLA without re-optimization. These results support the effectiveness of source-derived configurations on different benchmarks, although improvements vary across models, operators, and target tasks.


4. Answer Wording and Option Labels

Our motivation draws on distributed representations and superposition. Appendix E, Equation (27), schematically represents a feature as

f=∑iwiui,f = \sum_i w_i u_i,

where uiu_i denotes a direction in representation space. This motivates interventions beyond individual units, without assuming a one-to-one correspondence between units and behavioral features.

At the structural level, Equation (28) represents the model as a composition of Transformer blocks. Writing the model as M\mathcal{M},

M(x)=(FL∘FL−1∘⋯∘F1)(x).\mathcal{M}(x) = (F_L \circ F_{L-1} \circ \cdots \circ F_1)(x).

FSI modifies these complete block transformations: Freeze bypasses a selected block, Swap exchanges two blocks, and Insert relocates an auxiliary block. The learned parameters within the blocks remain unchanged.

To evaluate performance beyond predefined answer wording and option labels, we also test open-ended TruthfulQA generation in Appendix H, Table 6, reproduced in Response 1. All three interventions improve the reported truthfulness scores on both models. These results provide complementary behavioral evidence that FSI's benefits extend beyond the controlled candidate-answer format.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.