acceptodds
Withdrawn from ICLR 2027

SFWE: Mitigating Spurious Feature Reliance in LLM Reasoning via Weight Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Reasoning in large language models can be disrupted by simple perturbations (e.g., adding irrelevant distractors or rephrasing a problem's scenario), suggesting that their reasoning relies on spurious correlations. Identifying and suppressing the spurious features involved in extended chain-of-thought reasoning remains challenging. We introduce Spurious Feature Weight Editing (SFWE), a training-free method that extracts behavioral directions associated with spurious reliance by comparing activations on paired original and perturbed problems, localizes the contributing MLP neurons and attention heads along these directions, and suppresses their influence by removing the direction-aligned components from the selected output weights. Using two small datasets for extracting behavioral vectors and calibration, SFWE improves accuracy on perturbed inputs by 10.9 percentage points on average and accuracy on original inputs by 0.9 points across five open-weight reasoning models and five reasoning benchmarks, reducing the average relative gap between original and perturbed accuracy from 17.4% to 2.2%. Analyses identify two recurring forms of spurious reasoning—over-integrating problem conditions and overthinking—and show that SFWE improves reasoning robustness by suppressing the corresponding behavioral directions.

Primary area:
causal reasoning
Keywords:
shortcut learning, spurious correlations, llm reasoning, generalization

Related papers

Discussion (0)

Sign in to comment.