acceptodds
Under review as a conference paper at ICLR 2027

DASH: Mitigating LLM Hallucinations via Dynamic Attention Steering in Hidden-Space for Contextual Faithfulness

Abstract

LLMs frequently hallucinate when parametric memory overrides the provided context. Existing contrastive decoding methods rely on static, input-agnostic retrieval-head masks and linear logit-space corrections, both of which limit faithfulness. We introduce DASH, which learns contextual faithfulness atop a frozen base LLM via a hierarchical Head Selection Controller that dynamically gates retrieval heads from per-token geometric features and a Nonlinear Steering Adapter that emits counterfactual residual corrections natively in the model's hidden space. We show that these corrections strictly contain affine contrastive logit steering as a one-parameter linear special case, and that the controller's optimal gate distribution admits a closed-form softmax characterization. Across single and multi-hop reasoning benchmarks, DASH establishes state-of-the-art groundedness on exact-match, F1, and ROUGE while keeping its generations fluent. We further provide DASH++, an entropy-triggered early-exit inference mode of the same model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Faithfulness, Retrieval Attention Heads, Attention Steering, LLM Groundedness, Contrastive Decoding, Faithfulness, Hallucination Mitigation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iDyiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on improving steering for hallucination mitigation. It focuses on an open-book setup where the correct answer is the context one. It compares against known mitigation methods such as DoLa and CAD. It shows its results on LLaMA-3-8B-Instruct and Mistral-7BInstruct-v0.3. The metric named DASH aims to improve contextual faithfulness by head selection that dynamically gates the heads and a nonlinear steering adaptor. It shows higher results compared to the common metric. Lastly, they provide an additional variation of their method named DASH++ that is a faster version of DASH.

Critical strengths
  1. The paper creates a novel metric for steering named DASH and DASH++ that uses a nonlinear steering adaptor and selects heads that gate which heads will be used.
  2. The paper shows that DASH has higher hallucination reduction compared to other known techniques.
  3. The paper shows an ablation of some of the construction setups, indicating the benefits of using all of them and mostly the nonlinear NSA.
Critical weaknesses
  1. The DASH metric (Section 3) of the paper lacks intuition and makes it hard to follow and understand the importance of each step in the steering construction and to reconstruct the DASH metric.
  2. Important steps in the steering metric, such as the gate (eq 2), lack intuition or an ablation for their specific calculation.
Questions for authors
  1. What is Lalign + Lentropy in line 72?
  2. Line 107: how did the authors extract y-? And did they evaluate different ways to do that? Also, what happened in questions where the parametric y- answer was the same as the contextual y+?
  3. Please add intuition to eq 1.
  4. Please add an explanation and intuition for each equation in Section 3.2. Also, why is this the gate score (eq 2)? Does it correspond to related work? Are there preliminary results that indicate that we should calculate the head score this way?
  5. Line 185 - “ We do not renormalise the attention”- can you run an ablation of how it will change the results if there were a normalization step?
  6. Line 65: missing citations (DeCoRe and CAD, and DoLa), again in line 315.
  7. Can the authors evaluate the impact of their steering on the model's generation coherence? E.g. perplexity.
  8. The paper shows results on older models LLaMA-3-8B-Instruct and Mistral-7BInstruct-v0.3 (from 2024), and can benefit from showing results on newer versions of the same or other model families.
4 comments
AuthorsOpenReview ↗

Authors' Response (1/2)

We thank the reviewer for the detailed questions and for recognising the novelty of DASH. These questions were very helpful: they led to a revision of the paper, a few changes in the PDF (for Q1, Q6), and new ablations and experiments. We regret that the intuition behind Section 3 was not clear enough, and we give a more detailed explanation below.

W1 & W2: Intuition for Section 3 and Eq. 2

Below we give, for each subsection, the intuition that is not explicit in the paper based on our revision.

3.2 (Eqs. 1–2). For Eq. 1, please see Q3. The gate in Eq. 2 is a novel part of DASH: it depends on the head itself (leaf), on the other heads of its layer, which see the same input (branch), and on the token as a whole (root), since only tokens with a context–memory conflict need correcting. The tree is inspired by the hierarchical mixture-of-experts of Jordan & Jacobs (1994) and similar works; their gates combine expert outputs rather than scale heads, but this motivated our branch and root terms. Keeping only the leaf term drops performance, as evident in Abl 4 (Table 3).

3.3. Turning down the retrieval heads gives what the model would compute if it ignored the context, so Ht−H~tH _ {t} - \widetilde{H} _ {t} carries the context signal.

3.4 (Eqs. 3–4). DeCoRe adds a fixed multiple of this signal to the logits, which equals adding α(Ht−H~t)\alpha(H _ {t} - \widetilde{H} _ {t}) to the hidden state. The NSA instead learns a non-linear correction from HtH _ {t} and H~t\widetilde{H} _ {t}, so DeCoRe's linear steering is one special case it can represent (Theorem 1), and working in hidden space is cheaper (d≪∣V∣d \ll |V|). Replacing the NSA with linear steering drops performance, as evident in Abl 1–2 (Table 3).

3.5 (Eqs. 5–9). LSFT\mathcal{L} _ {\text{SFT}} and Lctr\mathcal{L} _ {\text{ctr}} train both modules through the output, while Lalign\mathcal{L} _ {\text{align}} and Lentropy\mathcal{L} _ {\text{entropy}} shape only the gates: the first closes heads that read the context, the second stops the gate mass from collapsing onto one head (Theorem 2; Abl 3). Lctr\mathcal{L} _ {\text{ctr}} is ablated in our response to Reviewer Cum9.

3.6. The entropy of the gates shows whether the HSC has singled out a few retrieval heads (low entropy, e.g. on an answer entity) or treats all heads alike (high entropy, e.g. on function words such as "the"). DASH++ runs the masked pass and NSA only in the first case, saving a forward pass on some tokens. Replacing the HSC with a fixed mask drops performance significantly (Abl 5, Table 3).

We can add some of these intuitions to Section 3 as the reviewer asks, for better understanding of the larger audience as well.

AuthorsOpenReview ↗

Authors' Response (2/2)

Response to Questions:

Q1

Thank you for pointing this out. They are the two gate regularisers defined in Section 3.5 (Eqs. 7–8). We can add this in the paper.

Q2

For NQ-Swap, y−y^- is the dataset's original answer. For NQ and HotpotQA, GPT-4o is used to return a wrong answer as a proxy for parametric knowledge (Appendix C.3). This is done because there is no such y−y^- for these two datasets and we wanted to keep the training data diverse. Following Reviewer Cum9's comments, we also tried a second method: y−y^- is the base model's own answer to the question without context. It performs on par with DASH on all benchmarks (full ablation results in our response to Reviewer Cum9, please kindly refer to it).

If y+=y−y^+ = y^-, Eq. 6 reduces to Lctr=γ\mathcal{L} _ {\text{ctr}} = \gamma, a constant with zero gradient, so by design it has no effect during training.

Q3

ϕl,h\phi _ {l,h} estimates whether a head is carrying context information at step tt: Ol,h−Hc′O _ {l,h} - H _ {c}' measures how far the head's output is from the context, Ol,h⊙Hc′O _ {l,h} \odot H _ {c}' how much it agrees with it, and E~l,h\tilde E _ {l,h} is a prior over which heads usually retrieve. We can add this after Eq. 1.

Q4

Please see W1 & W2. Please let us know if this needs to be added to the paper.

Q5

The gates are independent per-head controls, and our intuition was that normalising them would decrease the gap between the gate values of retrieval and non-retrieval heads; hence it is not done in the masked pass. Below is the ablation result (after retraining) when normalisation is enabled, we see a slight performance drop.

MethodNQ-Swap sub_EM ↑NQ-Swap Para_Rate ↓HotpotQA EM ↑HotpotQA F1 ↑HotpotQA Ctr_Sub ↓NewsQA EM ↑NewsQA F1 ↑
Base LM56.518.6455.4967.353.6328.7353.41
DASH69.024.1163.3272.043.1139.0859.26
DASH w/ normalised gates66.344.5259.2571.113.4536.5058.20

Q6

Thank you for pointing this out. We have cited each method on its first occurrence in the paper and left subsequent mentions uncited, this is to save space. We have kept the citations in the score tables for clarity as well. We have updated the PDF with your mentioned change, and other repeated citations.

Q7

We scored the answers of Base and DASH with an independent LM (Qwen2.5-7B-Instruct). The judge sees the same prompt as the generator (instruction, context and question) and we compute the perplexity of only the answer tokens.

NQ-SwapNQHotpotQA
Base6.523.272.35
DASH6.133.702.42

On NQ-Swap and HotpotQA the difference is not significant. On NQ, DASH is slightly higher, and most of this increase comes from questions DASH answers incorrectly. We can add this to the appendix.

Q8

Thank you for the suggestion. We used Llama and Mistral as those were the most common models used across these baselines and related works. We trained DASH on the newer Qwen3-8B (2025) with the same setting and evaluated it on NQ-Swap, HotpotQA and NQ. DASH improves over the base model on every metric, showing that it carries over to a newer model family. We can add this in appendix.

ModelSwap Sub-EM↑Swap Para↓Hotpot EM↑Hotpot Sub-EM↑Hotpot F1↑Hotpot Ctr↓NQ EM↑NQ Sub-EM↑NQ F1↑
Qwen3-8B63.753.4847.6761.0459.623.4252.8859.5265.33
DASH (with Qwen3-8B)74.292.1161.3567.1070.872.8556.1461.4266.27

We thank the reviewer again. Your questions helped us confirm that normalising the gates lowers performance, which we had earlier taken as intuitive, let us measure the coherence of DASH's answers, and showed that DASH carries over to a newer model (Qwen3-8B). Please let us know which intuition parts you find necessary to include in the paper and if we missed anything. We hope our responses address your concerns, and if so, we kindly ask you to consider raising your score. We are happy to clarify anything further and welcome any recommendations for changes to the paper.

AuthorsOpenReview ↗

A Gentle Reminder to Check the Authors' Response

Dear Reviewer iDyi,

This is a gentle reminder to check the rebuttal we have provided above. We have addressed the concerns you raised in your review. We will be happy to discuss further on this.

Thanks and regards, Authors

I thank the authors for their answers, the additional intuition, especially the one in Section 3.2, and the perplexity, normalization, and new model analysis. These additions should be included in the paper.

In light of these new results, I increased my score.

Reviewer_Cum9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DASH for mitigating LLM hallucinations. It includes a hierarchical Head Selection Controller that dynamically gates retrieval heads from per-token geometric features and a Nonlinear Steering Adapter that emits counterfactual residual corrections natively in the model’s hidden space. Experiments are conducted on NQ, NQ-Swap, and HotpotQA under in-distribution setting and on NewsQA, MuSiQue, and XSum under out-of-distribution setting.

Critical strengths
  1. Improving the faithfulness of LLMs' generated content through intervention rather than post-generation detection is a valuable task. The "Head Selection Controller" and "Nonlinear Steering Adapter" proposed by the authors are reasonable.
  2. The experiments cover diverse generation tasks, including extractive QA, single-hop QA, multi-hop reasoning, and abstractive summarization. Furthermore, DASH is evaluated in both in-distribution and out-of-distribution scenarios, demonstrating improvements over baselines.
Critical weaknesses
  1. During the training process, the authors incorporate unfaithful answers generated by GPT-4o as y−y^- for contrastive learning, using them to compute a contrastive faithfulness loss as part of the overall training objective. However, the ablation studies do not analyze the impact of this loss on the model's overall performance. If this generated data contributes significantly to the results, it raises concerns regarding the fairness of comparisons against baselines that do not utilize such data, as well as the accuracy of the claims made in the Introduction concerning the paper's core contributions.
  2. While the authors mention several existing methods in the Introduction and the baseline descriptions, the presentation is somewhat fragmented. There is a lack of a systematic overview of the existing methods, and the distinctions between DASH and various types of existing methods are not clearly described.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Authors' Response

We thank the reviewer for the careful reading, for appreciating the breadth of our evaluation, and for the constructive feedback. We carefully ran new experiments and made minor updates to the PDF where necessary; the PDF changes add the missing citations and an equation reference pointed out by Reviewer iDyi (Q1 and Q6).

W1: Role of Lctr\mathcal{L}_{\text{ctr}} and dependence on GPT-4o data

We ran two new experiments (Llama3-8B-Instruct, same test sets as before). Both needed retraining:

  • w/o Lctr\mathcal{L}_{\text{ctr}}: DASH trained without Lctr\mathcal{L}_{\text{ctr}} (λctr=0\lambda_{\text{ctr}}=0)
  • Closed-book y−y^-: every GPT-4o-generated y−y^- replaced by the frozen base model's own answer when given only the question, without context. Here Lctr\mathcal{L}_{\text{ctr}} is used, same as DASH.
MethodNQ-Swap sub_EM ↑NQ-Swap Para_Rate ↓HotpotQA EM ↑HotpotQA F1 ↑HotpotQA Ctr_Sub ↓NewsQA EM ↑NewsQA F1 ↑
Base LM56.518.6455.4967.353.6328.7353.41
DASH69.024.1163.3272.043.1139.0859.26
DASH w/o Lctr\mathcal{L}_{\text{ctr}}59.967.1757.8369.033.5235.1257.38
Closed-book y−y^- (no GPT-4o data)68.705.3263.0672.093.2138.4259.94

Lctr\mathcal{L}_{\text{ctr}} is important. Removing it noticeably lowers performance on every benchmark. LSFT\mathcal{L}_{\text{SFT}} teaches the model what to produce, while Lctr\mathcal{L}_{\text{ctr}} pushes it away from the competing parametric answer.

The gains do not depend on GPT-4o data. We thank the reviewer for this insightful comment, which inspired the second experiment: without context, the base model can only answer from its own knowledge, so its closed-book answer is a natural parametric y−y^- that needs no external data.

Would the reviewer like us to include these two results in the main text (e.g., in the ablation table, Table 3)? We would be happy to add them in the revision.

W2: Systematic overview of existing methods

We agree. Since ICLR does not require a related-work section, we had omitted it to save space and discussed prior methods in the Introduction and baselines, which left the positioning fragmented. Prior methods fall into four groups. Decoding-time methods (CD, DoLa, CAD, DeCoRe, AD) contrast or reweight next-token distributions with a fixed rule; for example, CAD contrasts the model with and without the context, and DeCoRe contrasts it with a copy whose retrieval heads are masked. DASH keeps this two-pass structure, but attenuates heads with soft per-token gates instead of a fixed mask and learns a nonlinear correction in hidden space instead of combining logits. Activation and attention steering (ITI, TruthX, PASTA) edits activations or attention toward a target behaviour, usually parametric truthfulness, often along a direction fixed once learned; DASH targets contextual faithfulness and recomputes its correction at every step from the base and head-masked hidden states. Head-level interventions (DeCoRe, ITI, PASTA) act on a subset of attention heads chosen once per corpus or task; DASH selects heads at every token. Finally, fine-tuning and detection methods such as Self-RAG fine-tune the LM, and FactCheckmate relies on a separate hallucination detector; DASH keeps the LM frozen and needs no detector.

Would the reviewer like us to add this overview to the paper as a Related Work section?


We thank the reviewer again. Your comments helped us confirm that the GPT-4o data does not influence DASH's gains, which we had earlier taken as intuitive, and the new experiment gave us a new way to generate the contrastive parametric answer y−y^- from the base model itself. We hope our responses address your concerns, and if so, we kindly ask you to consider raising your score. We are happy to clarify anything further and welcome any recommendations for changes to the paper.

AuthorsOpenReview ↗

A Gentle Reminder to Check the Authors' Response

Dear Reviewer Cum9,

This is a gentle reminder to check the rebuttal we have provided above. We have addressed the concerns you raised in your review. We will be happy to discuss further on this.

Thanks and regards, Authors

Thanks for your response and additional experimental results. I would like you to include these results and related work in the paper. Since most of my concerns have been addressed, I will raise my score to 3.

Reviewer_BSSuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DASH, a framework for reducing context-memory hallucinations in frozen LLMs. It comprises (i) a hierarchical Head Selection Controller that softly gates retrieval heads per decoding step using geometric per-head features and a tree-structured controller, (ii) a masked second forward pass producing counterfactual hidden states, and (iii) a Nonlinear Steering Adapter that emits hidden-space residual corrections before the LM head, trained with supervised, contrastive, alignment, and entropy losses. The authors prove that their correction family strictly contains affine contrastive logit steering (DeCoRe) as a one-parameter special case, and derive a closed-form softmax optimal gate distribution. Experiments on two 7–8B backbones across six benchmarks (three in-distribution, three OOD) show state-of-the-art EM/F1 groundedness over decoding-time baselines (CAD, DoLa, DeCoRe, ITI, AD), plus a DASH++ early-exit mode reducing forward passes toward 1.

Critical strengths

The strongest contribution is a coherent end-to-end inference-time architecture that replaces static retrieval-head masking with learned token-level soft gating and nonlinear hidden-space correction while keeping trainable parameters small and compatible with frozen open-weight LLMs; this is consequential because it directly targets input-agnostic masking and logit-space linearity, two real limitations of prior contrastive decoding/retrieval-head methods.

Critical weaknesses

The safety/faithfulness failure mode is not peripheral; it cuts against the paper’s central claim. DASH is trained only on concrete y+,y−y^+, y^- pairs with no abstention targets, and the authors report that roughly 25% of NQ-Swap errors are confident confabulations where the base model would abstain but DASH emits a plausible unsupported answer. For a paper framed as hallucination/groundedness mitigation, overriding correct refusal and producing confident wrong answers when the supplied context is faulty or domain-implausible is a material weakness. The ethics statement recommends trustworthy retrieval, but there is no abstention metric or training/evaluation mitigation, so deployability and the “groundedness” framing are weaker than claimed.

The theoretical and empirical claims are overclaimed relative to the evidence. Theorem 1 shows only that DeCoRe-style affine logit contrast is a linear special case; it does not establish that the actual low-rank NSA represents “any continuous δt\delta_t,” and that statement is not justified for a fixed two-layer bottleneck MLP. Empirically, “SOTA groundedness” is supported mainly by EM/F1 on two 7–8B backbones, while OOD Sub-EM can fall below the base (e.g., Llama NewsQA 57.63 vs 60.82, MuSiQue 50.42 vs 51.02; Mistral NewsQA 43.14 vs 47.82), and there are no paired significance tests against baselines.

Questions for authors
  1. In Table 1, DASH++ on Mistral exceeds DASH on NQ-Swap Sub-EM (72.81 vs 61.52), while DASH beats it elsewhere. Is this stability, or an artifact of a single τ and greedy decoding variance? Could you report variance across seeds for DASH vs DASH++?

  2. What fraction of DASH's NQ-Swap wins come from the abstention-overriding behavior (36.5% of wins per F.4.1)? Would the headline Sub-EM gains survive with an abstention-aware evaluation that credits base-model refusals?

  3. Please refer to the "weaknesses" section for other issues.

5 comments
AuthorsOpenReview ↗

Authors' Response (1/3)

We thank Reviewer BSSu for the detailed feedback and for recognizing the core contribution of replacing static retrieval-head masking and linear logit-space steering with learned token-level soft gating and nonlinear hidden-space correction on frozen LLMs. Below we respond to each concern and describe the specific changes we will make in the revised manuscript.

1. Confabulation on domain-implausible contexts and abstention framing (Weakness 1)

Our problem formulation (§3.1, p. 3; Ethics Statement, p. 10) targets context-memory conflict under trusted evidence, following the standard setup shared by CAD, DeCoRe, and DoLa where the provided passage is the ground-truth source and the decoder must overcome parametric bias. Within that scope:

  • Counterfactual extraction (Case S2) vs. confabulation (Case F2). Counterfactual grounding benchmarks such as NQ-Swap deliberately replace a known fact with an implausible entity (e.g., Shawn Michaels as the original host of What's My Line?, Case S2, p. 25) to test whether the model defers to the passage over parametric memory. Under NQ-Swap's task definition, extracting Shawn Michaels from the passage is the required grounded output, whereas the base model's refusal comes from parametric skepticism toward the entity swap. Case F2 (Paul Hogan substituted into a question asking for an NBA numeric record, where DASH outputs 841, p. 25) is a distinct boundary case where the substituted entity violates the question's expected answer type. Also, the ~25% figure for Case F2 (and ~16% for parametric override in Case F1) is a percentage of DASH's remaining NQ-Swap errors in our qualitative error analysis (Appendix F.4.2, pp. 25-26), not 25% of all NQ-Swap examples.
  • Clarifying the scope of our claims. By design, our training data (c,x,y+,y−)(c, x, y^+, y^-) teaches the controller and adapter to extract concrete context-supported answers over parametric foils (§3.5, p. 5), rather than acting as an unanswerability detector or refusal classifier (which we noted in §4.2, p. 8; §5, p. 9; and Appendix A, p. 15). To prevent any ambiguity between context adherence and refusal calibration on corrupted inputs, we will tighten the wording in the Abstract, Introduction, and §5 so that our claims explicitly specify contextual faithfulness when answering from trusted evidence rather than broad hallucination or safety mitigation, and we will clearly state in §4.2 and §5 that abstention under corrupted or unanswerable context is outside the scope of concrete-pair contrastive training.

2. Question 2: 36.5% abstention-overriding wins and abstention-aware evaluation on NQ-Swap

In our qualitative win analysis on NQ-Swap (Appendix F.4.1, p. 25, on Llama-3-8B-Instruct), we examined examples where DASH extracted the substituted context answer and the base model failed:

  • In 63.5% of DASH's wins, the base model failed by outputting the memorized parametric answer instead of the context entity (Case S1, e.g., outputting Muhammad Yunus instead of James Chadwick).
  • In the remaining 36.5% of wins, the base model refused to accept the swapped entity and abstained, whereas DASH extracted the entity from the passage (Case S2, e.g., Shawn Michaels).

The headline Sub-EM gains survive an abstention-aware perspective for two reasons:

  1. Valid counterfactual extraction vs. unanswerable context. In Case S2, the passage explicitly supplies a type-compatible answer to the question (a person's name for a "Who was the original host..." question); the base model refuses only because its parametric prior rejects the counterfactual swap. On a counterfactual grounding benchmark designed to test whether a model follows context over memory, refusing to answer when the passage explicitly states the entity is a grounding failure rather than a valid refusal on unanswerable context.
  2. Conservative lower bound removing all 36.5% refusal-override wins. Even under the most conservative calculation that credits every base-model refusal in that 36.5% category to the base model and retains only the 63.5% of wins where DASH overrides a direct parametric hallucination (Case S1), scaling DASH's net test-set Sub-EM gains in Table 1 (+12.51 pp on Llama-3-8B, 69.02 vs. 56.51; +27.79 pp on Mistral-7B, 61.52 vs. 33.73) by 0.635 still leaves a +7.94 pp gain over the base model on Llama-3-8B (64.45 vs. 56.51) and a +17.65 pp gain on Mistral-7B (51.38 vs. 33.73, which still exceeds CAD at 43.43 and DeCoRe at 36.91). We will include this discussion and clarify the exact scope of the Appendix F.4 breakdown in §4.2 and Appendix F.4.1.
AuthorsOpenReview ↗

Authors' Response (2/3)

3. Question 1: DASH++ vs. DASH on Mistral NQ-Swap (72.81 vs. 61.52), τ\tau, decoding variance, and seeds

  1. Zero decoding variance and shared checkpoint. All evaluations use deterministic greedy decoding (Reproducibility Statement, p. 10; Table 4, p. 16; Appendix C.4, p. 18), and DASH++ uses the exact same trained checkpoint as DASH (§3.6, pp. 5-6). There is therefore zero greedy decoding or checkpoint variance between DASH and DASH++; the 72.81 vs. 61.52 (+11.29 pp) difference on Mistral NQ-Swap comes deterministically from the entropy early-exit rule at τ=6.85\tau = 6.85.
  2. Behavior across τ\tau and metric trade-offs. Rather than tuning τ\tau per dataset, we fixed a single threshold τ=6.85\tau = 6.85 across all six benchmarks and both backbones (§3.6, p. 6), and reported the full sweep over τ∈{6.70,6.75,6.80,6.85,6.90}\tau \in \{6.70, 6.75, 6.80, 6.85, 6.90\} on NQ-Swap, HotpotQA, and NewsQA for both backbones in Appendix F.3 (Figure 3, p. 24). Because autoregressive decoding conditions each token on previously generated tokens, skipping the NSA correction on high-entropy steps changes the downstream trajectory and avoids over-steering on non-factual positions on Mistral NQ-Swap (§4.2, p. 8). At the same time, Table 1 (p. 8) shows the expected speed-accuracy trade-off on other metrics: DASH++ has a slightly higher Parametric Rate than full DASH on Mistral NQ-Swap (3.37 vs. 2.85) and Llama NQ-Swap (7.90 vs. 4.11), and lower EM on multi-hop HotpotQA (42.28 vs. 59.41 on Mistral).
  3. Variance across seeds. Because DASH and DASH++ are two deterministic inference modes of the same trained checkpoint, comparing DASH vs. DASH++ for a given checkpoint has zero inference variance. For training, we trained one checkpoint per backbone (seed 42 for Mistral-7B, seed 43 for Llama-3-8B; Table 4, p. 16; Appendix C.4, p. 18), and the ±\pm values in Tables 1-2 report 95% confidence intervals across test examples, which we will state explicitly in the Table 1-2 captions.

4. Theorem 1 and the fixed-width low-rank NSA (Weakness 2a)

Thank you for pointing this out. Our wording in §3.7 ("admits any continuous δt=fNSA(Ht,H~t,rt)\delta _ t = f _ {\mathrm{NSA}}(H _ t, \widetilde{H} _ t, r _ t)", p. 6) conflated the asymptotic universal-approximation motivation in Appendix E (p. 20) with the fixed-rank two-layer bottleneck MLP (db=256d _ b = 256, Eq. 4, p. 4) used in our implementation.

The core result of Theorem 1 is the exact algebraic equivalence: for full-column-rank WLMW _ {\mathrm{LM}}, any DeCoRe-style affine logit contrast Δz=α(zt−ztmask)\Delta z = \alpha(z _ t - z _ t^{\mathrm{mask}}) is realized in hidden space by the 1-parameter linear shift δt=α(Ht−H~t)∈Rd\delta _ t = \alpha(H _ t - \widetilde{H} _ t) \in \mathbb{R}^d within col(WLM)\mathrm{col}(W _ {\mathrm{LM}}) at dimension d≪∣V∣d \ll |V|, whereas the NSA parameterizes a learned rank dbd _ b nonlinear mapping conditioned on (Ht,H~t,rt)(H _ t, \widetilde{H} _ t, r _ t) (whose empirical advantage over both linear NSA and fixed-direction α(Ht−H~t)\alpha(H _ t - \widetilde{H} _ t) steering is validated in Table 3, p. 9).

To avoid any ambiguity in the revised manuscript, we will:

  1. Remove the phrase "admits any continuous δt\delta _ t" from Theorem 1 (§3.7, p. 6) and Appendix E (pp. 19-20), and explicitly specify the fixed bottleneck rank dbd _ b in the formal statement of FNSA\mathcal{F} _ {\mathrm{NSA}}.
  2. Clarify in the Abstract (p. 1), Introduction (p. 2), §3.7 (p. 6), and §5 (p. 9) that Theorem 1 establishes the exact hidden-space representation of 1-parameter affine logit contrast and motivates our low-rank nonlinear residual parameterization.
AuthorsOpenReview ↗

Authors' Response (3/3)

5. OOD Sub-EM regressions, scope of empirical claims, and paired significance testing (Weakness 2b)

  1. OOD Sub-EM vs. EM and F1. On the OOD benchmarks in Table 2 (p. 9) where Sub-EM is lower than the base model (Llama NewsQA 57.63 vs. 60.82, Llama MuSiQue 50.42 vs. 51.02 for DASH while DASH++ reaches 54.90, and Mistral NewsQA 43.14 vs. 47.82), DASH achieves large gains on strict EM and token-level F1: on NewsQA, EM improves from 28.73 to 39.08 (+10.35 pp) and F1 from 53.41 to 59.26 (+5.85 pp) on Llama, and EM improves from 19.73 to 29.23 (+9.50 pp) and F1 from 31.02 to 43.82 (+12.80 pp) on Mistral; on MuSiQue, EM improves from 41.21 to 47.53 (+6.32 pp) and F1 from 53.72 to 59.23 (+5.51 pp) on Llama, and EM improves from 20.63 to 42.04 (+21.41 pp) and F1 from 29.31 to 51.52 (+22.21 pp) on Mistral. As explained in §4.2 (pp. 8-9) and shown in Table 9 (p. 23), the base model generates much longer responses (averaging 11.7 tokens vs. DASH's 5.1 on NQ and 5.6 on MuSiQue for Mistral), which increases the chance of including the reference answer as a substring (Sub-EM) while hurting exact match (EM) and token precision (F1). Because LSFTL _ {\mathrm{SFT}} trains on concise gold answers, DASH produces concise outputs that lead all 16 answer-level EM and F1 columns across Tables 1-2, with a trade-off on lenient substring match on longer OOD spans.
  2. Scope of "SOTA groundedness" and statistical tests. In the Abstract (p. 1), Introduction (p. 2), and §5 (p. 9), we qualified our claim as "state-of-the-art groundedness at the answer level (EM and F1)." To make both the OOD Sub-EM trade-off and our experimental scope completely clear in the camera-ready version, we will refine this phrasing throughout to state consistent answer-level EM/F1 and summarization ROUGE gains over compared decoding-time baselines on Llama-3-8B-Instruct and Mistral-7B-Instruct, and we will explicitly note in the captions of Tables 1-2 and §4.2 that the ±\pm values report marginal 95% confidence intervals over test examples.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for the Reviews and Rebuttal Discussions

Dear Reviewers and Chairs of ICLR 2027,

We would like to thank all the reviewers for their time and effort in providing constructive feedback on our work. We have provided detailed point-by-point responses to all the major concerns raised in the reviews through our rebuttal. We appreciate that all the reviewers have engaged with our responses and increased their scores, with all current ratings at 3 (Weak Accept, out of 4).

We also want to inform the reviewers and chairs (AC, SAC, and PC) that all the manuscript changes described in our rebuttal have already been incorporated into the latest revised version uploaded to OpenReview. These are completed revisions, rather than proposed changes for a future version. We hope that the current version of the paper meets the technical quality expected of an ICLR 2027 research paper.

Thank you again for your time and consideration of our work.

Regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.