Counterfactual Error Scrambling: Measuring Error-Message Use in Tool-Using Language Agents

Tool-using language agents routinely receive execution errors, yet it remains unclear whether they act on the content of those messages or only on the fact that an error occurred. We introduce counterfactual error scrambling, a protocol that replaces each error message with a variant that preserves its length, line structure and exception class and keeps identifiers verbatim, while permuting the surrounding words to destroy compositional meaning; misleading and empty variants serve as controls. Applying it to nine models from 7B to frontier scale on MBPP-sanitized (3,600 trials; one serving stack; success scored by the execution harness), seven models recover significantly more often from real than from scrambled errors ($+13$ to $+30$ percentage points, Holm-corrected), including an 8B model without a reasoning mode, while two 7–8B models released in 2024 show no effect. Error-message use is therefore not determined by parameter count alone. An audit of the scrambler shows that on a quarter of the tasks it can move a mixed-case identifier out of its position in the message; restricted to tasks where identifiers stay in place, four effects remain significant ($+12$ to $+23$ points, including the 8B model) and three weaken. Decomposing repairs after NameErrors shows why: when the missing name is displaced, agents rarely adopt it (4%), and when it stays in place, but its frame is scrambled, adoption falls from 54% to 35%. What agents use is the position of the actionable token and the conventional frame around it. Within Qwen3-14B, suppressing reasoning leaves the model retrying without acting on the message at all. Sensitivity decays across successive errors, and on HumanEval, capable models rarely fail, so the contrast is undefined; we report both as scope conditions, and we release all trajectories and analysis code.