- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ATOMIC is a three-stage inference pipeline intended to improve small language models under distracting context. Stage I condenses question-relevant information, Stage II assigns evidence scores to short spans, and Stage III generates an answer while prioritizing highly scored evidence. The paper reports substantial improvements on several distractor-augmented reasoning benchmarks and analyzes the separation between relevant and irrelevant context using attention-based AUROC.
The practical problem is important and the empirical gains are often large. However, the default implementation is a multi-call prompting pipeline that asks the model to assign heuristic scores and prioritize textual spans. It does not directly modify attention scores as the central framing suggests. The theoretical analysis also does not provide a valid guarantee for the implemented procedure.
Critical strengths
-
The paper addresses a practical problem: small language models often degrade sharply when useful evidence is mixed with plausible distractors. The reported gains are large across several models and datasets.
-
The appendix provides the prompts, latency, token cost, hardware setup, and additional small-model results. These details make the operational behavior of the pipeline easier to understand.
Critical weaknesses
-
The main results come from a textual prompting pipeline rather than an attention intervention. The model assigns heuristic scores to spans and is prompted to prioritize them. Describing this as attention reallocation overstates what the implementation demonstrates.
-
The theoretical analysis is disconnected from the implemented scoring rule. It assumes unknown optimal weights , while the method simply sums four model-generated integer scores. The paper does not derive a relationship between these quantities, and one span-dependent likelihood term is treated as a constant without sufficient justification.
-
Stage I is presented as finding a minimal sufficient subset, but the implementation is an unconstrained rewriting prompt. It can omit, paraphrase, or introduce information, and preserving the model’s previous answer does not imply preserving the correct evidence. The ablation also leaves the role of Stage III unclear because Stage I plus II appears to reproduce the full result.
Questions for authors
- Which headline results, if any, use direct attention-logit modification?
- What formal relationship connects the four integer evidence scores to the theoretical weights ?
- Why does Stage I plus II appear to match the complete ATOMIC result?