- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how to reduce factual hallucination in LLMs by editing their internal representations. The authors argue that existing representation editing methods use many parameters, are slow at inference, and ignore the distribution of the original representations. BRIDGE works as follows. It first extracts internal representations from three kinds of inputs: the question only, the question with a correct answer, and the question with a wrong answer. It then builds two cross-covariance matrices from these representations and applies SVD to obtain two low-dimensional subspaces. Finally, it learns a small number of editing vectors in these subspaces. At inference time, these operations reduce to a single residual addition, so the cost is very low. Experiments on several models and datasets show that BRIDGE improves truthfulness while using far fewer parameters than existing methods.
Critical strengths
-
The method is very lightweight. It uses very few learnable parameters, and inference only needs one residual addition, which adds almost no cost.
-
The results are fairly consistent. BRIDGE improves performance across several models and on multiple factuality and reasoning tasks.
Critical weaknesses
-
The method does not fully match the motivation. The paper criticizes existing methods for ignoring the distribution of the original representations at inference time. But at inference time, BRIDGE also adds a fixed vector that does not depend on the input, as the authors acknowledge in the introduction. So the claimed advantage of considering the original distribution is unclear. This is my main concern.
-
The need for two subspaces is not shown. Table 3 shows that using only one subspace with a larger dimension performs almost the same as BRIDGE. So the gain may simply come from using more dimensions.
-
The evaluation is limited. TruthfulQA is only evaluated in the multiple-choice format. There is no evaluation of generated answers, and no check of whether the method hurts the model's general abilities.
Questions for authors
-
BRIDGE builds two special subspaces. Would simpler choices work just as well? For example, a randomly chosen subspace, or simply learning one bias vector without any subspace. If these give similar results, the special construction may not be needed.
-
All TruthfulQA results use multiple-choice questions. Does BRIDGE still help when the model writes its own answers? Does it hurt the model's performance on other tasks?
I think using the structure of the base representations to guide editing is an interesting direction, and the method is very lightweight. I encourage the authors to further clarify the role of the two subspaces and to provide a more complete evaluation.
3 comments
Thank you for the detailed responses and additional experiments. The new controlled ablations, especially the comparisons between Top-k, Random-k, and Bottom-k selections, provide convincing evidence for the effectiveness of the proposed subspace construction. The additional open-ended generation results also address my concerns about the limited evaluation setting. Although some questions regarding general capability preservation remain, the authors have substantially addressed my main concerns. After considering the new evidence, I have decided to increase my rating to Weak Accept. I appreciate the authors' efforts in improving the paper. Congratulations!
We sincerely thank the reviewer for the thoughtful reassessment and for recognizing our efforts to strengthen the paper. We also appreciate your constructive suggestions. Thank you again for your time, valuable feedback, and encouraging evaluation.
We sincerely thank the reviewer for the constructive feedback. In response, we have added controlled ablations, geometric visualizations, and open-ended generation experiments.
1. Clarification of Distribution-aware Editing.
As explicitly clarified in the Introduction (p. 2), BRIDGE incorporates base-representation information during offline subspace construction, while retaining static residual injection at inference. However, the key distinction from distribution-blind editing lies in how the intervention directions are constructed, rather than whether the residual itself depends on each input. Although the learned editing vectors are input-independent, their directions are constrained by the underlying representation geometry captured from the contrastive distributions. We emphasize that BRIDGE provides offline distribution-aware geometric calibration rather than runtime distribution conditioning.
2. Necessity of the SVD-based Bi-subspace Construction.
To verify that BRIDGE's improvements arise from meaningful spectral directions, we added several complementary experiments.
First, we compare BRIDGE with a No-Subspace variant that directly optimizes editing vectors in the original representation space without SVD-based constraints. The newly added Table 4 (p. 9) reports:
BRIDGE consistently outperforms unconstrained full-space editing while requiring substantially fewer parameters. We also added optimization convergence comparisons in Figure 6 (Appendix, p. 16). BRIDGE converges substantially faster and exhibits smaller optimization fluctuations than No-Subspace.
Second, we added a controlled comparison of Top-k, Random-k, and Bottom-k selection under an identical dimension (k=50). As shown in the newly added Table 8 (Appendix, p. 17), Random-k and Bottom-k select alternative orthogonal singular directions, whereas BRIDGE retains the dominant spectral components.
Top-k consistently outperforms both alternatives.
Third, we added geometric visualizations to provide further evidence. Figure 7 (Appendix, p. 18) compares representations projected onto Top-2, Random-2, and Bottom-2 singular vectors. The Top-2 projections exhibit clear separation between positive and negative representations, whereas Random-2 and Bottom-2 show substantial overlap. Furthermore, Figure 3 (p. 8) demonstrates that injecting the learned editing vector moves negative representations toward the positive representation region. Together, these visualizations support the effectiveness of the learned spectral directions.
3. Complementarity of the Two Subspaces.
We extended the original ablation study to three additional backbones. The new results are reported in Table 3 (p. 8).
Under an identical parameter budget, joint editing consistently outperforms the best individual-subspace configuration:
Their joint optimization exploits both advantages without increasing the parameter budget. We clarify that the degree of complementarity depends on the underlying representation geometry of each backbone.
4. Open-ended Generation.
We added open-ended generation experiments on TruthfulQA (Appendix 7.5, Table 14, p. 20), together with generation comparisons (pp. 20–34). We evaluate truthfulness (True), informativeness (Info), and their composite score (True×Info).
Values are reported as Base→BRIDGE. Additionally, BRIDGE improves multi-hop and logical reasoning performance on MQuAKE and CLUTRR. The expanded cross-model results are presented in Appendix Tables 10 and 12 (pp. 17 and 19).