acceptodds
Under review as a conference paper at ICLR 2027

BRIDGE: Bi-subspace Residual Guidance for Spectral Representation Editing towards Truthful LLMs

Abstract

Transformer-based Large Language Models (LLMs) sometimes produce "hallucinations" by generating content that deviates from facts seen during pretraining. Interpretability studies have shown that the representations within LLMs encode rich semantic information. Representation editing techniques can intervene in LLM representations at low cost, thereby alleviating hallucinations. Existing training-based representation editing methods freeze the base model and learn interventions for hidden representations specific to the task. However, these methods still suffer from heavy parameter optimization and high computational overhead. Furthermore, these methods often perform uncalibrated directional shifts without accounting for the underlying geometric structure of the base model's representation space. Direct additive interventions applied without global structural alignment risk displacing valid representation manifolds, leading to over-editing or degradation of non-target behaviors. To overcome these limitations, we propose Bi-subspace Residual Guidance for Spectral Representation Editing (BRIDGE) to enhance LLM truthfulness. BRIDGE first constructs a Target-Alignment and a De-biasing cross-covariance matrices to delineate the semantic evolutionary trajectories across the baseline, untruthful, and truthful domains. Through Singular Value Decomposition (SVD), it extracts the principal singular vectors mapping directly into the target truthful subspace. Subsequently, ultra-lightweight learnable representation editing vectors are introduced into the low-dimensional subspace spanned by these principal singular vectors. Extensive experiments on multiple models and benchmark datasets demonstrate that BRIDGE not only effectively enhances performance on various factuality and logical reasoning tasks, but also reduces the learnable parameters by 20x to 3000x compared to existing methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Editing, Spectral Editing, Inference-Time Intervention, Alleviating Hallucination, Activation Steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wfa9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how to reduce factual hallucination in LLMs by editing their internal representations. The authors argue that existing representation editing methods use many parameters, are slow at inference, and ignore the distribution of the original representations. BRIDGE works as follows. It first extracts internal representations from three kinds of inputs: the question only, the question with a correct answer, and the question with a wrong answer. It then builds two cross-covariance matrices from these representations and applies SVD to obtain two low-dimensional subspaces. Finally, it learns a small number of editing vectors in these subspaces. At inference time, these operations reduce to a single residual addition, so the cost is very low. Experiments on several models and datasets show that BRIDGE improves truthfulness while using far fewer parameters than existing methods.

Critical strengths
  1. The method is very lightweight. It uses very few learnable parameters, and inference only needs one residual addition, which adds almost no cost.

  2. The results are fairly consistent. BRIDGE improves performance across several models and on multiple factuality and reasoning tasks.

Critical weaknesses
  1. The method does not fully match the motivation. The paper criticizes existing methods for ignoring the distribution of the original representations at inference time. But at inference time, BRIDGE also adds a fixed vector that does not depend on the input, as the authors acknowledge in the introduction. So the claimed advantage of considering the original distribution is unclear. This is my main concern.

  2. The need for two subspaces is not shown. Table 3 shows that using only one subspace with a larger dimension performs almost the same as BRIDGE. So the gain may simply come from using more dimensions.

  3. The evaluation is limited. TruthfulQA is only evaluated in the multiple-choice format. There is no evaluation of generated answers, and no check of whether the method hurts the model's general abilities.

Questions for authors
  1. BRIDGE builds two special subspaces. Would simpler choices work just as well? For example, a randomly chosen subspace, or simply learning one bias vector without any subspace. If these give similar results, the special construction may not be needed.

  2. All TruthfulQA results use multiple-choice questions. Does BRIDGE still help when the model writes its own answers? Does it hurt the model's performance on other tasks?

I think using the structure of the base representations to guide editing is an interesting direction, and the method is very lightweight. I encourage the authors to further clarify the role of the two subspaces and to provide a more complete evaluation.

3 comments

We sincerely thank the reviewer for the constructive feedback. In response, we have added controlled ablations, geometric visualizations, and open-ended generation experiments.

1. Clarification of Distribution-aware Editing.

As explicitly clarified in the Introduction (p. 2), BRIDGE incorporates base-representation information during offline subspace construction, while retaining static residual injection at inference. However, the key distinction from distribution-blind editing lies in how the intervention directions are constructed, rather than whether the residual itself depends on each input. Although the learned editing vectors are input-independent, their directions are constrained by the underlying representation geometry captured from the contrastive distributions. We emphasize that BRIDGE provides offline distribution-aware geometric calibration rather than runtime distribution conditioning.

2. Necessity of the SVD-based Bi-subspace Construction.

To verify that BRIDGE's improvements arise from meaningful spectral directions, we added several complementary experiments.

First, we compare BRIDGE with a No-Subspace variant that directly optimizes editing vectors in the original representation space without SVD-based constraints. The newly added Table 4 (p. 9) reports:

ModelMethodParamsMC1MC2MC3
Llama2-7BNo-Subspace77K52.2370.6247.81
BRIDGE7K62.4379.5152.64
Llama3-8B-InstNo-Subspace100.3K53.2472.9646.11
BRIDGE8.4K67.6982.6356.74
Mistral-v0.2No-Subspace100.3K66.1082.0959.90
BRIDGE5.6K74.0285.1263.01
Mistral-v0.3No-Subspace100.3K66.1082.7659.70
BRIDGE9.8K71.4886.5563.26

BRIDGE consistently outperforms unconstrained full-space editing while requiring substantially fewer parameters. We also added optimization convergence comparisons in Figure 6 (Appendix, p. 16). BRIDGE converges substantially faster and exhibits smaller optimization fluctuations than No-Subspace.

Second, we added a controlled comparison of Top-k, Random-k, and Bottom-k selection under an identical dimension (k=50). As shown in the newly added Table 8 (Appendix, p. 17), Random-k and Bottom-k select alternative orthogonal singular directions, whereas BRIDGE retains the dominant spectral components.

SelectionLlama2-7B MC1/MC2/MC3Llama3-8B MC1/MC2/MC3
Random-k47.98/67.31/39.5352.02/71.91/40.28
Bottom-k46.39/66.48/38.2950.92/72.07/40.85
Top-k (BRIDGE)51.41/70.61/42.4355.45/73.83/43.56

Top-k consistently outperforms both alternatives.

Third, we added geometric visualizations to provide further evidence. Figure 7 (Appendix, p. 18) compares representations projected onto Top-2, Random-2, and Bottom-2 singular vectors. The Top-2 projections exhibit clear separation between positive and negative representations, whereas Random-2 and Bottom-2 show substantial overlap. Furthermore, Figure 3 (p. 8) demonstrates that injecting the learned editing vector moves negative representations toward the positive representation region. Together, these visualizations support the effectiveness of the learned spectral directions.

3. Complementarity of the Two Subspaces.

We extended the original ablation study to three additional backbones. The new results are reported in Table 3 (p. 8).

Under an identical parameter budget, joint editing consistently outperforms the best individual-subspace configuration:

ModelBest Single (MC1/MC2/MC3)Joint (MC1/MC2/MC3)
Llama2-7B62.18/79.40/52.4462.43/79.51/52.64
Llama3-8B-Inst64.55/81.44/53.6566.34/82.17/55.54
Mistral-v0.367.32/81.94/56.9170.99/85.37/61.82
Mistral-v0.271.00/84.45/61.3274.02/85.12/63.01

Their joint optimization exploits both advantages without increasing the parameter budget. We clarify that the degree of complementarity depends on the underlying representation geometry of each backbone.

4. Open-ended Generation.

We added open-ended generation experiments on TruthfulQA (Appendix 7.5, Table 14, p. 20), together with generation comparisons (pp. 20–34). We evaluate truthfulness (True), informativeness (Info), and their composite score (True×Info).

ModelTrue (%)Info (%)True×Info (%)
Llama2-7B36.16→76.7790.73→87.9532.81→67.52
Llama3-8B-Inst58.59→81.8991.76→85.8253.76→70.28
Mistral-v0.271.56→82.6294.26→90.3867.17→74.67
Mistral-v0.363.57→84.8495.35→91.4460.61→77.58

Values are reported as Base→BRIDGE. Additionally, BRIDGE improves multi-hop and logical reasoning performance on MQuAKE and CLUTRR. The expanded cross-model results are presented in Appendix Tables 10 and 12 (pp. 17 and 19).

Thank you for the detailed responses and additional experiments. The new controlled ablations, especially the comparisons between Top-k, Random-k, and Bottom-k selections, provide convincing evidence for the effectiveness of the proposed subspace construction. The additional open-ended generation results also address my concerns about the limited evaluation setting. Although some questions regarding general capability preservation remain, the authors have substantially addressed my main concerns. After considering the new evidence, I have decided to increase my rating to Weak Accept. I appreciate the authors' efforts in improving the paper. Congratulations!

We sincerely thank the reviewer for the thoughtful reassessment and for recognizing our efforts to strengthen the paper. We also appreciate your constructive suggestions. Thank you again for your time, valuable feedback, and encouraging evaluation.

Reviewer_VnbdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses factuality hallucinations in LLMs through representation editing. The authors argue that existing training-based editing methods still need many learnable parameters, add inference overhead through projection and reconstruction, and apply directional shifts that ignore the geometry of the base model's representation space. To address this, they propose BRIDGE, which first extracts hidden representations for a question alone, the question with a truthful answer, and the question with an untruthful answer. From these, it builds two cross-covariance matrices, one capturing the shift from the base state to the truthful state and one capturing the shift from the untruthful to the truthful state. SVD is applied to each to obtain two low-dimensional subspaces, and small learnable editing vectors are trained within them while the model stays frozen. The authors show that this editing step simplifies to a single static residual added to the hidden representation, giving essentially no inference overhead. Experiments on TruthfulQA, FACTOR, MQuAKE and CLUTRR with several Llama and Mistral models show large improvements over the base models and competitive or better results than existing steering and parameter-efficient fine-tuning methods. The method uses roughly 20x to 3000x fewer trainable parameters, and further analyses cover ablations of the two subspaces, results on 10 base models, and sensitivity to the subspace dimension.

Critical strengths
  1. BRIDGE trains only 5K–17.6K parameters (Table 2) yet outperforms LoRA, ReFT, RED and LOFIT on TruthfulQA. For example, MC1 on Llama2-7B is 62.43 vs. 58.10 for LOFIT, and on Llama2-13B it is 63.53 vs. 57.00 (Table 6). The gains are consistent across 10 base models (Table 5) and reported over 5 seeds. If the gains hold under proper controls, this is a practically useful result.

  2. The final form (Eq. 11) is a single added vector per layer with no runtime projection. It is easy to implement and adds essentially no inference cost. The authors are also clear that the edit is static and not input-dependent (lines 104–106).

Critical weaknesses
  1. After simplification (Eq. 11), BRIDGE adds a fixed, input-independent bias that is restricted to the span of Vbp1:kV_{bp}^{1:k} and Vnp1:kV_{np}^{1:k}. The paper's main claims ("distribution-aware", avoiding "out-of-distribution" shifts, preserving the "orthogonal complement") therefore depend entirely on these subspaces being meaningful. However, several things suggest they may not be:

    • There is no comparison with a random orthonormal subspace of the same kk, or with an unconstrained bias vector trained the same way (DPO/SFT).
    • A single subspace with k=1000k=1000 gets 62.18 / 79.40 / 52.44 (ebpe_{bp}), almost identical to both subspaces with k=500k=500: 62.43 / 79.51 / 52.64.
    • The appendix argues directions are well defined only up to rank≤min⁡(N,dF)\mathrm{rank} \le \min(N, d_F). Yet MQuAKE has 134 training examples with kk = 600–800 (Table 10), and CLUTRR has 300 training examples with kk = 400–700 (Table 12). Unless more triplets were built for these tasks, most of the chosen directions are not determined by the data.
  2. The evaluation does not support the broader claims.

    • The abstract and conclusion claim BRIDGE avoids "degradation of non-target behaviors" and preserves "intrinsic language capacity". All evaluations are likelihood-based multiple choice or EM on the target task, with no general-capability (e.g., MMLU), fluency or open-ended generation results.
    • Hyperparameters vary per model and task (kk from 90 to 800, learning rates, and hand-picked layer lists for MQuAKE/CLUTRR), and the paper doesn't say whether they were chosen on the dev or test split.
    • On Mistral-v0.2, MC1 falls below the base model at large kk (about 48 vs. 55.20 at k=800k=800, Figure 5), which conflicts with the claim of robustness to kk.
    • Training-free methods (DoLa, ITI, PLI) are compared directly with a method trained with DPO on the same benchmark's training split.
Questions for authors
  1. Have you compared BRIDGE with (a) a random orthonormal subspace of the same kk and (b) a plain learned bias vector with no subspace constraint, trained with the same DPO/SFT setup?
  2. In Table 3, a single subspace with k=1000k=1000 performs almost the same as both subspaces with k=500k=500. How does this support the claimed synergy of the bi-subspace design, rather than simply more dimensions?
  3. For MQuAKE (134 training examples) and CLUTRR (300), how many triplets were used to build CbpC_{bp} and CnpC_{np}? With kk up to 800 and 700, are the extra singular directions arbitrary?
  4. How were kk, the learning rate and the edited layers selected for each model and task?
  5. Do you have any results on general capabilities (e.g., MMLU or perplexity) or on open-ended TruthfulQA generation, to support the claim that non-target behaviours are preserved?
2 comments

We sincerely thank the reviewer for the constructive feedback. In response, we have revised the manuscript by adding controlled ablations, spectral visualizations, optimization analyses, and open-ended generation experiments. Revisions are marked in red.

1. Effectiveness of the SVD-based Subspaces. We compare BRIDGE with a No-Subspace variant that directly learns an unconstrained editing vector in the original representation space. As shown in the newly added Table 4 (p. 9):

ModelMethodParamsMC1MC2MC3
Llama2-7BNo-Subspace77K52.2370.6247.81
BRIDGE7K62.4379.5152.64
Llama3-8B-InstNo-Subspace100.3K53.2472.9646.11
BRIDGE8.4K67.6982.6356.74
Mistral-7B-v0.2No-Subspace100.3K66.1082.0959.90
BRIDGE5.6K74.0285.1263.01
Mistral-7B-v0.3No-Subspace100.3K66.1082.7659.70
BRIDGE9.8K71.4886.5563.26

Second, we added a controlled spectral selection study (Appendix, Table 8, p. 17), comparing Top-k, Random-k, and Bottom-k selections under the same dimension (k=50).

SelectionLlama2-7B MC1/MC2/MC3Llama3-8B MC1/MC2/MC3
Random-k47.98/67.31/39.5352.02/71.91/40.28
Bottom-k46.39/66.48/38.2950.92/72.07/40.85
Top-k (BRIDGE)51.41/70.61/42.4355.45/73.83/43.56

We provide geometric evidence. Figure 3 (p. 8) visualizes positive, negative, and edited negative representations projected onto the top-2 singular vectors of the subspace across layers 9–15. The edited negative representations move toward the positive representation region. More importantly, the newly added Figure 7 (p. 18) compares Top-2, Random-2, and Bottom-2 projections. Clear distributional separation appears with Top-2, whereas substantial overlap remains with Random-2 and Bottom-2.

Figure 6 (Appendix, p. 16) compares the optimization trajectories of BRIDGE and No-Subspace. BRIDGE converges substantially faster, reaching a low training loss within approximately 100 steps, while No-Subspace exhibits larger fluctuations and slower convergence.

2. Complementarity of Bi-subspace Editing. We extended the original Llama2-7B ablation to three additional backbones (revised manuscript, Table 3, p. 8). The comparison uses an identical parameter budget: joint editing allocates k1 dimensions to each subspace (2k1 total), while each single-subspace setting uses k2=2k1 dimensions.

ModelSingle bp (k2)Single np (k2)Joint (k1)
Llama2-7B62.18/79.40/52.4460.71/78.70/52.2862.43/79.51/52.64
Llama3-8B-Inst64.55/81.44/53.4663.33/79.16/53.6566.34/82.17/55.54
Mistral-7B-v0.367.32/81.83/56.9166.71/81.94/56.6870.99/85.37/61.82
Mistral-7B-v0.271.00/83.14/61.3269.28/84.45/60.9074.02/85.12/63.01

Neither subspace consistently dominates the other across different backbones and evaluation metrics. Jointly optimizing the two editing vectors consistently achieves the best performance across all four backbones and all three metrics, even compared with individual-subspace configurations using the same total parameter budget. Although the gain is relatively marginal on Llama2-7B, this does not invalidate the complementary roles of the two subspaces.

3. Clarification of Training Samples and SVD Rank. The reported 134 MQuAKE and 300 CLUTRR training examples should not be confused with the samples used for subspace construction. These numbers correspond to the LOFIT-selected task-training subsets, which we adopted to align the downstream experimental settings with LOFIT. The complete training pools of both datasets contain additional training instances. We explicitly clarify the distinction between downstream task-training subsets and contrastive triplets used for spectral basis construction in Appendix Sections 7.3 and 7.4 (pp. 17–20).

4. Evaluation on Open-ended Generation. We have added a new set of open-ended generation experiments on TruthfulQA across four backbone models. These experiments are presented in the newly added Appendix Section 7.5 (Table 14, Page 20), together with qualitative comparisons of generated responses (Pages 20–34). The results are summarized below.

ModelTrue (%)Info (%)True × Info (%)
Llama2-7B36.16 → 76.7790.73 → 87.9532.81 → 67.52
Llama3-8B-Instruct58.59 → 81.8991.76 → 85.8253.76 → 70.28
Mistral-7B-Instruct-v0.271.56 → 82.6294.26 → 90.3867.17 → 74.67
Mistral-7B-Instruct-v0.363.57 → 84.8495.35 → 91.4460.61 → 77.58

All values are reported as Base → BRIDGE. Furthermore, we have added qualitative examples in Appendix Section 7.6 (Pages 20–34), illustrating how BRIDGE corrects factually misleading responses while retaining informative content.

Revision Overview

- Table 3: Expanded bi-subspace ablation from Llama2-7B to three additional backbones (Llama3-8B-Instruct and Mistral-v0.2/v0.3), demonstrating the complementary benefits of joint optimization under a matched parameter budget.

- Figure 3: Geometric visualization of positive, negative, and edited negative representations projected onto the top-2 singular vectors across layers 9–15, illustrating how editing steers negative representations toward the positive region.

- Table 4: Comparison between BRIDGE and No-Subspace editing across four backbones, demonstrating the effectiveness and parameter efficiency of SVD-based subspace constraints compared with unconstrained full-space editing.

- Figure 6: Training loss comparison between BRIDGE and No-Subspace across four backbones, showing faster convergence and more stable optimization with spectral subspace constraints.

- Table 8: Controlled comparison of Top-k, Random-k, and Bottom-k singular-vector selection under the same subspace dimension (k=50), demonstrating the advantage of selecting dominant spectral directions.

- Figure 7: Visualization of Top-2, Random-2, and Bottom-2 projections across layers 8–15, showing clear positive-negative representation separation in the dominant spectral directions compared with substantial overlap in random and bottom directions.

- Appendix Sections 7.3–7.4: Clarification of the distinction between LOFIT-aligned downstream training subsets (MQuAKE and CLUTRR) and the contrastive triplets used for spectral basis construction, including the use of 1,000 triplets for constructing the cross-covariance matrices.

- Appendix Section 7.6: Newly added TruthfulQA open-ended generation evaluation (Table 14), reporting truthfulness, informativeness, and their composite scores across four backbones, together with qualitative comparisons of generated responses.

Reviewer_86ziOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets factuality hallucinations. BRIDGE proceeds in three stages. First, from contrastive triplets it extracts FFN intermediate representations under three promptings, yielding base, positive, and negative representation matrices. Second, it forms two cross-covariance matrices:a Target-Alignment matrix and a De-biasing matrix (untruthful-to-truthful), and takes the top-k right singular vectors of each via SVD as projection bases. Third, it trains two k-dimensional editing vectors, one per subspace, added to the projected coordinates.Training adds an L1 penalty for sparsity.

Critical strengths
  • The subspace construction is distribution-aware in a meaningful sense.

  • The empirical evaluations are broad.

  • The framework is objective-agnostic.

Critical weaknesses
  • The method is named after the bi-subspace design, yet at k=1000 a single editing vector on the Target-Alignment subspace nearly reproduces the full method, while the joint bi-subspace configuration at the main setting (k=500) buys only a marginal improvement over the single-vector variant.

  • The evaluation never measures generation. Every headline metric is discriminative, i.e., multiple-choice likelihoods and exact match.

  • The stated advantage over prior editing is that calibrated, subspace-respecting interventions avoid the OOD displacement and non-target degradation caused by distribution-blind shifts. But the paper contains no general-capability evaluation.

  • No code and no Limitations section accompany the submission.

Questions for authors

See Weaknesses

2 comments

We thank the reviewer for the constructive feedback. In response, we added cross-model ablations, open-ended generation experiments, and methodological clarifications. Revisions are marked in red.

1. Complementarity of Bi-subspace Editing.

We extended the original Llama2-7B ablation to three additional backbones: Llama3-8B-Instruct and Mistral-7B-Instruct-v0.2/v0.3 (revised manuscript, Table 3, p. 8). The comparison uses an identical parameter budget: joint editing allocates k1 dimensions to each subspace (2k1 total), while each single-subspace setting uses k2=2k1 dimensions. Entries report MC1/MC2/MC3 (%).

ModelSingle bp (k2)Single np (k2)Joint (k1)
Llama2-7B62.18/79.40/52.4460.71/78.70/52.2862.43/79.51/52.64
Llama3-8B-Inst64.55/81.44/53.4663.33/79.16/53.6566.34/82.17/55.54
Mistral-7B-v0.367.32/81.83/56.9166.71/81.94/56.6870.99/85.37/61.82
Mistral-7B-v0.271.00/83.14/61.3269.28/84.45/60.9074.02/85.12/63.01

The results reveal two important observations. First, neither subspace consistently dominates the other across different backbones and evaluation metrics, indicating that they capture different aspects of truth-related information. Second, jointly optimizing the two editing vectors consistently achieves the best performance across all four backbones and all three metrics, even compared with individual-subspace configurations using the same total parameter budget. Notably, the improvements are particularly evident on Llama3 and both Mistral variants. Although the gain is relatively marginal on Llama2-7B, this does not invalidate the complementary roles of the two subspaces.

We provide geometric evidence. Figure 3 (p. 8) visualizes positive, negative, and edited negative representations projected onto the top-2 singular vectors of the subspace. The edited negative representations move toward the positive representation region.

2. Evaluation on Open-ended Generation.

We have added a new set of open-ended generation experiments on TruthfulQA across four backbone models. These experiments are presented in the newly added Appendix Section 7.5 (Table 14, Page 20), together with qualitative comparisons of generated responses (Pages 20–34). The results are summarized below.

ModelTrue (%)Info (%)True × Info (%)
Llama2-7B36.16 → 76.7790.73 → 87.9532.81 → 67.52
Llama3-8B-Instruct58.59 → 81.8991.76 → 85.8253.76 → 70.28
Mistral-7B-Instruct-v0.271.56 → 82.6294.26 → 90.3867.17 → 74.67
Mistral-7B-Instruct-v0.363.57 → 84.8495.35 → 91.4460.61 → 77.58
All values are reported as Base → BRIDGE.

Furthermore, we have added qualitative examples in Appendix Section 7.6 (Pages 20–34), illustrating how BRIDGE corrects factually misleading responses while retaining informative content.

These additional experiments provide direct evidence that BRIDGE improves factuality in open-ended generation, rather than merely influencing likelihood-based discriminative decisions. We have incorporated these results into the revised manuscript to strengthen the empirical validation of BRIDGE beyond multiple-choice evaluation.

3. Preservation of General Capabilities.

First, BRIDGE explicitly constrains representation modifications to the subspaces identified by SVD. As formally derived in Section 3.3, High-Dimensional Reconstruction and Subspace Substitution (Equations 9–11, Page 5), the original representation is decomposed into its edited subspace components and the corresponding orthogonal residual. By replacing only the selected subspace coordinates, the intervention is simplified to:

$\hat{h}=h+e_{bp}(V_{bp}^{1:k})^T +e_{np}(V_{np}^{1:k})^T.$

Consequently, the intervention introduces no direct modification to the orthogonal complement of the combined editing subspaces. This provides a structural mechanism for limiting unnecessary perturbations to the original representation, rather than applying unrestricted offsets in the full representation space.

We have the empirical evaluation with additional evidence on Logical and Multi-hop Reasoning Tasks. The main paper reports MQuAKE and CLUTRR results in Table 2 (Page 7), while the expanded cross-model results are provided in Appendix Sections 7.3 and 7.4, Tables 10 and 12 (Pages 17 and 19), respectively.

Moreover, the newly added TruthfulQA open-ended generation evaluation (Appendix Section 7.5, Table 14, Page 20) shows that BRIDGE substantially improves factuality while maintaining relatively high informativeness across all four backbones.

4. Reproducibility and Limitations.

We added a Reproducibility Statement (p. 11) and Limitations section, committing to release the code and datasets upon publication. Implementation details are given in Appendix 7.1 (p. 14), with hyperparameters in Tables 5, 9, 11, and 13 (pp. 15, 17, 18, and 20).

Revision Overview

- Table 3: Expanded bi-subspace ablation from Llama2-7B to three additional backbones (Llama3-8B-Instruct and Mistral-v0.2/v0.3), demonstrating the complementary benefits of joint optimization under a matched parameter budget.

- Figure 3: Geometric visualization of positive, negative, and edited negative representations projected onto the top-2 singular vectors across layers 9–15, illustrating how editing steers negative representations toward the positive region.

- Table 4: Comparison between BRIDGE and No-Subspace editing across four backbones, demonstrating the effectiveness and parameter efficiency of SVD-based subspace constraints compared with unconstrained full-space editing.

- Figure 6: Training loss comparison between BRIDGE and No-Subspace across four backbones, showing faster convergence and more stable optimization with spectral subspace constraints.

- Table 8: Controlled comparison of Top-k, Random-k, and Bottom-k singular-vector selection under the same subspace dimension (k=50), demonstrating the advantage of selecting dominant spectral directions.

- Figure 7: Visualization of Top-2, Random-2, and Bottom-2 projections across layers 8–15, showing clear positive-negative representation separation in the dominant spectral directions compared with substantial overlap in random and bottom directions.

- Appendix Sections 7.3–7.4: Clarification of the distinction between LOFIT-aligned downstream training subsets (MQuAKE and CLUTRR) and the contrastive triplets used for spectral basis construction, including the use of 1,000 triplets for constructing the cross-covariance matrices.

- Appendix Section 7.6: Newly added TruthfulQA open-ended generation evaluation (Table 14), reporting truthfulness, informativeness, and their composite scores across four backbones, together with qualitative comparisons of generated responses.

- Reproducibility Statement (p. 11); Appendix Section 7.1 (p. 14): Additional reproducibility details, including implementation settings, training procedures, and a commitment to release the complete source code and datasets upon publication.

- Limitations Section (p. 20): Explicit discussion of BRIDGE's reliance on offline contrastive representations, static inference-time residual injection, and heuristic-based layer selection, with more systematic and adaptive layer selection identified as an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.