BRIDGE: Bi-subspace Residual Guidance for Spectral Representation Editing towards Truthful LLMs

Transformer-based Large Language Models (LLMs) sometimes produce "hallucinations" by generating content that deviates from facts seen during pretraining. Interpretability studies have shown that the representations within LLMs encode rich semantic information. Representation editing techniques can intervene in LLM representations at low cost, thereby alleviating hallucinations. Existing training-based representation editing methods freeze the base model and learn interventions for hidden representations specific to the task. However, these methods still suffer from heavy parameter optimization and high computational overhead. Furthermore, these methods often perform uncalibrated directional shifts without accounting for the underlying geometric structure of the base model's representation space. Direct additive interventions applied without global structural alignment risk displacing valid representation manifolds, leading to over-editing or degradation of non-target behaviors. To overcome these limitations, we propose Bi-subspace Residual Guidance for Spectral Representation Editing (BRIDGE) to enhance LLM truthfulness. BRIDGE first constructs a Target-Alignment and a De-biasing cross-covariance matrices to delineate the semantic evolutionary trajectories across the baseline, untruthful, and truthful domains. Through Singular Value Decomposition (SVD), it extracts the principal singular vectors mapping directly into the target truthful subspace. Subsequently, ultra-lightweight learnable representation editing vectors are introduced into the low-dimensional subspace spanned by these principal singular vectors. Extensive experiments on multiple models and benchmark datasets demonstrate that BRIDGE not only effectively enhances performance on various factuality and logical reasoning tasks, but also reduces the learnable parameters by 20x to 3000x compared to existing methods.