- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on improving steering for hallucination mitigation. It focuses on an open-book setup where the correct answer is the context one. It compares against known mitigation methods such as DoLa and CAD. It shows its results on LLaMA-3-8B-Instruct and Mistral-7BInstruct-v0.3. The metric named DASH aims to improve contextual faithfulness by head selection that dynamically gates the heads and a nonlinear steering adaptor. It shows higher results compared to the common metric. Lastly, they provide an additional variation of their method named DASH++ that is a faster version of DASH.
Critical strengths
- The paper creates a novel metric for steering named DASH and DASH++ that uses a nonlinear steering adaptor and selects heads that gate which heads will be used.
- The paper shows that DASH has higher hallucination reduction compared to other known techniques.
- The paper shows an ablation of some of the construction setups, indicating the benefits of using all of them and mostly the nonlinear NSA.
Critical weaknesses
- The DASH metric (Section 3) of the paper lacks intuition and makes it hard to follow and understand the importance of each step in the steering construction and to reconstruct the DASH metric.
- Important steps in the steering metric, such as the gate (eq 2), lack intuition or an ablation for their specific calculation.
Questions for authors
- What is Lalign + Lentropy in line 72?
- Line 107: how did the authors extract y-? And did they evaluate different ways to do that? Also, what happened in questions where the parametric y- answer was the same as the contextual y+?
- Please add intuition to eq 1.
- Please add an explanation and intuition for each equation in Section 3.2. Also, why is this the gate score (eq 2)? Does it correspond to related work? Are there preliminary results that indicate that we should calculate the head score this way?
- Line 185 - “ We do not renormalise the attention”- can you run an ablation of how it will change the results if there were a normalization step?
- Line 65: missing citations (DeCoRe and CAD, and DoLa), again in line 315.
- Can the authors evaluate the impact of their steering on the model's generation coherence? E.g. perplexity.
- The paper shows results on older models LLaMA-3-8B-Instruct and Mistral-7BInstruct-v0.3 (from 2024), and can benefit from showing results on newer versions of the same or other model families.
4 comments
Authors' Response (2/2)
Response to Questions:
Q1
Thank you for pointing this out. They are the two gate regularisers defined in Section 3.5 (Eqs. 7–8). We can add this in the paper.
Q2
For NQ-Swap, is the dataset's original answer. For NQ and HotpotQA, GPT-4o is used to return a wrong answer as a proxy for parametric knowledge (Appendix C.3). This is done because there is no such for these two datasets and we wanted to keep the training data diverse. Following Reviewer Cum9's comments, we also tried a second method: is the base model's own answer to the question without context. It performs on par with DASH on all benchmarks (full ablation results in our response to Reviewer Cum9, please kindly refer to it).
If , Eq. 6 reduces to , a constant with zero gradient, so by design it has no effect during training.
Q3
estimates whether a head is carrying context information at step : measures how far the head's output is from the context, how much it agrees with it, and is a prior over which heads usually retrieve. We can add this after Eq. 1.
Q4
Please see W1 & W2. Please let us know if this needs to be added to the paper.
Q5
The gates are independent per-head controls, and our intuition was that normalising them would decrease the gap between the gate values of retrieval and non-retrieval heads; hence it is not done in the masked pass. Below is the ablation result (after retraining) when normalisation is enabled, we see a slight performance drop.
| Method | NQ-Swap sub_EM ↑ | NQ-Swap Para_Rate ↓ | HotpotQA EM ↑ | HotpotQA F1 ↑ | HotpotQA Ctr_Sub ↓ | NewsQA EM ↑ | NewsQA F1 ↑ |
|---|---|---|---|---|---|---|---|
| Base LM | 56.51 | 8.64 | 55.49 | 67.35 | 3.63 | 28.73 | 53.41 |
| DASH | 69.02 | 4.11 | 63.32 | 72.04 | 3.11 | 39.08 | 59.26 |
| DASH w/ normalised gates | 66.34 | 4.52 | 59.25 | 71.11 | 3.45 | 36.50 | 58.20 |
Q6
Thank you for pointing this out. We have cited each method on its first occurrence in the paper and left subsequent mentions uncited, this is to save space. We have kept the citations in the score tables for clarity as well. We have updated the PDF with your mentioned change, and other repeated citations.
Q7
We scored the answers of Base and DASH with an independent LM (Qwen2.5-7B-Instruct). The judge sees the same prompt as the generator (instruction, context and question) and we compute the perplexity of only the answer tokens.
| NQ-Swap | NQ | HotpotQA | |
|---|---|---|---|
| Base | 6.52 | 3.27 | 2.35 |
| DASH | 6.13 | 3.70 | 2.42 |
On NQ-Swap and HotpotQA the difference is not significant. On NQ, DASH is slightly higher, and most of this increase comes from questions DASH answers incorrectly. We can add this to the appendix.
Q8
Thank you for the suggestion. We used Llama and Mistral as those were the most common models used across these baselines and related works. We trained DASH on the newer Qwen3-8B (2025) with the same setting and evaluated it on NQ-Swap, HotpotQA and NQ. DASH improves over the base model on every metric, showing that it carries over to a newer model family. We can add this in appendix.
| Model | Swap Sub-EM↑ | Swap Para↓ | Hotpot EM↑ | Hotpot Sub-EM↑ | Hotpot F1↑ | Hotpot Ctr↓ | NQ EM↑ | NQ Sub-EM↑ | NQ F1↑ |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 63.75 | 3.48 | 47.67 | 61.04 | 59.62 | 3.42 | 52.88 | 59.52 | 65.33 |
| DASH (with Qwen3-8B) | 74.29 | 2.11 | 61.35 | 67.10 | 70.87 | 2.85 | 56.14 | 61.42 | 66.27 |
We thank the reviewer again. Your questions helped us confirm that normalising the gates lowers performance, which we had earlier taken as intuitive, let us measure the coherence of DASH's answers, and showed that DASH carries over to a newer model (Qwen3-8B). Please let us know which intuition parts you find necessary to include in the paper and if we missed anything. We hope our responses address your concerns, and if so, we kindly ask you to consider raising your score. We are happy to clarify anything further and welcome any recommendations for changes to the paper.
A Gentle Reminder to Check the Authors' Response
Dear Reviewer iDyi,
This is a gentle reminder to check the rebuttal we have provided above. We have addressed the concerns you raised in your review. We will be happy to discuss further on this.
Thanks and regards, Authors
I thank the authors for their answers, the additional intuition, especially the one in Section 3.2, and the perplexity, normalization, and new model analysis. These additions should be included in the paper.
In light of these new results, I increased my score.
Authors' Response (1/2)
We thank the reviewer for the detailed questions and for recognising the novelty of DASH. These questions were very helpful: they led to a revision of the paper, a few changes in the PDF (for Q1, Q6), and new ablations and experiments. We regret that the intuition behind Section 3 was not clear enough, and we give a more detailed explanation below.
W1 & W2: Intuition for Section 3 and Eq. 2
Below we give, for each subsection, the intuition that is not explicit in the paper based on our revision.
3.2 (Eqs. 1–2). For Eq. 1, please see Q3. The gate in Eq. 2 is a novel part of DASH: it depends on the head itself (leaf), on the other heads of its layer, which see the same input (branch), and on the token as a whole (root), since only tokens with a context–memory conflict need correcting. The tree is inspired by the hierarchical mixture-of-experts of Jordan & Jacobs (1994) and similar works; their gates combine expert outputs rather than scale heads, but this motivated our branch and root terms. Keeping only the leaf term drops performance, as evident in Abl 4 (Table 3).
3.3. Turning down the retrieval heads gives what the model would compute if it ignored the context, so Ht−Ht carries the context signal.
3.4 (Eqs. 3–4). DeCoRe adds a fixed multiple of this signal to the logits, which equals adding α(Ht−Ht) to the hidden state. The NSA instead learns a non-linear correction from Ht and Ht, so DeCoRe's linear steering is one special case it can represent (Theorem 1), and working in hidden space is cheaper (d≪∣V∣). Replacing the NSA with linear steering drops performance, as evident in Abl 1–2 (Table 3).
3.5 (Eqs. 5–9). LSFT and Lctr train both modules through the output, while Lalign and Lentropy shape only the gates: the first closes heads that read the context, the second stops the gate mass from collapsing onto one head (Theorem 2; Abl 3). Lctr is ablated in our response to Reviewer Cum9.
3.6. The entropy of the gates shows whether the HSC has singled out a few retrieval heads (low entropy, e.g. on an answer entity) or treats all heads alike (high entropy, e.g. on function words such as "the"). DASH++ runs the masked pass and NSA only in the first case, saving a forward pass on some tokens. Replacing the HSC with a fixed mask drops performance significantly (Abl 5, Table 3).
We can add some of these intuitions to Section 3 as the reviewer asks, for better understanding of the larger audience as well.