acceptodds
Under review as a conference paper at ICLR 2027

ATOMIC: Attention Reallocation to Mitigate Irrelevant Context in Small Language Models

Abstract

Small language models (SLMs) have recently gained increasing attention due to their strong cost efficiency, computational efficiency, and competitive performance across a wide range of reasoning tasks. However, when reasoning inputs contain irrelevant context, SLMs are substantially more vulnerable to distraction than large language models, leading to severe performance degradation. Through a fine grained analysis of inference time attention, we identify a severe underlying issue: SLMs exhibit a markedly weaker ability to distinguish relevant context (RC) from irrelevant context (IC), as evidenced by consistently lower AUROC scores when treating attention over RC and IC as a binary discrimination problem. As a result, SLMs often assign high attention to distracting tokens in irrelevant context and erroneously incorporate them as evidence during reasoning, which disrupts effective evidence aggregation. To address this limitation, we propose ATOMIC. Our approach first performs task conditioned context level filtering to suppress irrelevant information, then introduces an Evidence Likelihood Score to amplify attention on critical evidence tokens within relevant context, and finally reallocates attention during answer inference to further reinforce key evidence. Extensive experiments across multiple mathematical reasoning benchmarks demonstrate that our method achieves state of the art performance. Our code is available at: https://anonymous.4open.science/r/ATOMIC.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Small Language Models, Irrelevant Context

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bSzoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ATOMIC is a three-stage inference pipeline intended to improve small language models under distracting context. Stage I condenses question-relevant information, Stage II assigns evidence scores to short spans, and Stage III generates an answer while prioritizing highly scored evidence. The paper reports substantial improvements on several distractor-augmented reasoning benchmarks and analyzes the separation between relevant and irrelevant context using attention-based AUROC.

The practical problem is important and the empirical gains are often large. However, the default implementation is a multi-call prompting pipeline that asks the model to assign heuristic scores and prioritize textual spans. It does not directly modify attention scores as the central framing suggests. The theoretical analysis also does not provide a valid guarantee for the implemented procedure.

Critical strengths
  1. The paper addresses a practical problem: small language models often degrade sharply when useful evidence is mixed with plausible distractors. The reported gains are large across several models and datasets.

  2. The appendix provides the prompts, latency, token cost, hardware setup, and additional small-model results. These details make the operational behavior of the pipeline easier to understand.

Critical weaknesses
  1. The main results come from a textual prompting pipeline rather than an attention intervention. The model assigns heuristic scores to spans and is prompted to prioritize them. Describing this as attention reallocation overstates what the implementation demonstrates.

  2. The theoretical analysis is disconnected from the implemented scoring rule. It assumes unknown optimal weights w∗w^{*}, while the method simply sums four model-generated integer scores. The paper does not derive a relationship between these quantities, and one span-dependent likelihood term is treated as a constant without sufficient justification.

  3. Stage I is presented as finding a minimal sufficient subset, but the implementation is an unconstrained rewriting prompt. It can omit, paraphrase, or introduce information, and preserving the model’s previous answer does not imply preserving the correct evidence. The ablation also leaves the role of Stage III unclear because Stage I plus II appears to reproduce the full result.

Questions for authors
  1. Which headline results, if any, use direct attention-logit modification?
  2. What formal relationship connects the four integer evidence scores to the theoretical weights w∗w^{*}?
  3. Why does Stage I plus II appear to match the complete ATOMIC result?
Reviewer_nG9vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ATOMIC, a method to make SLM/LLMs more robust against distractions presented in the context. ATOMIC first rewrites the given context to get rid of distractors. The authors achieve this by finding the shortest context among those that lead to the same model output as the associated original context. Then, ATOMIC estimates token-level importance using a predefined function set and upweights more important tokens. The authors provide the theoretical analysis justifying the design of the scoring function they use to find important tokens. The experiments show that ATOMIC outperforms the other tested baselines, including CoT, LTM, and Program, in math reasoning tasks.

Critical strengths

Enabling language models to focus on relevant tokens while remaining robust to distracting tokens in the context is an important capability for solving real-world tasks. The motivation and goal of this work are well aligned with this direction.

Critical weaknesses
  1. The paper describes well what ATOMIC wants to do, but it does not provide sufficient details about its implementation.
  • How are p(s|RC) and p(s|IC) estimated? If they are obtained using a held-in dataset, do these estimates valid on held-out datasets?
  • How exactly is an input context rewritten to suppress irrelevant information?
  • How is the objective in Eq. 1 optimized in practice?
  • ELS is approximated as a weighted sum of "token-level features {ϕk\phi_k}" but the detailed implementation is not provided.
  1. We are interested in the situation where f(Q) is incorrect. However, Eq. 1 only makes sense when f(Q) is correct.
  2. Rather than math reasoning tasks, RAG tasks seem more directly relevant to problems involving distractors. Is the proposed method also effective on RAG tasks?
Questions for authors
  • How are p(s|RC) and p(s|IC) estimated? If they are obtained using a held-in dataset, do these estimates valid on held-out datasets?
  • How exactly is an input context rewritten to suppress irrelevant information?
  • How is the objective in Eq. 1 optimized in practice?
  • What are the token-level features {ϕk\phi_k}, and how their weights are assigned?
  • What if f(Q) is incorrect in Eq. 1?
  • Is the proposed method also effective on RAG tasks?
Reviewer_4sX2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why small language models are particularly vulnerable to irrelevant context during reasoning. Through an analysis of inference-time attention, the authors find that SLMs struggle to distinguish relevant from irrelevant information, causing them to incorporate distracting tokens as evidence. To address this, they propose ATOMIC, which combines task-conditioned context filtering, an Evidence Likelihood Score to emphasize critical evidence tokens, and attention reallocation during answer generation. The authors report state-of-the-art performance across multiple mathematical reasoning benchmarks.

Critical strengths
  • The proposed approach is well motivated and theoretically sound.
  • Thorough experiments demonstrate the effectiveness of the approach.
  • The method achieves substantial improvements over the baselines.
Critical weaknesses
  • The method appears to require formatting the input data using knowledge of the ground truth, which may limit its applicability to new tasks or domains where such information is unavailable.
  • The evaluation does not establish whether the approach generalizes beyond mathematical reasoning to other types of problems, leaving its broader applicability unclear.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.