acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Error Scrambling: Measuring Error-Message Use in Tool-Using Language Agents

Abstract

Tool-using language agents routinely receive execution errors, yet it remains unclear whether they act on the content of those messages or only on the fact that an error occurred. We introduce counterfactual error scrambling, a protocol that replaces each error message with a variant that preserves its length, line structure and exception class and keeps identifiers verbatim, while permuting the surrounding words to destroy compositional meaning; misleading and empty variants serve as controls. Applying it to nine models from 7B to frontier scale on MBPP-sanitized (3,600 trials; one serving stack; success scored by the execution harness), seven models recover significantly more often from real than from scrambled errors ( to percentage points, Holm-corrected), including an 8B model without a reasoning mode, while two 7–8B models released in 2024 show no effect. Error-message use is therefore not determined by parameter count alone. An audit of the scrambler shows that on a quarter of the tasks it can move a mixed-case identifier out of its position in the message; restricted to tasks where identifiers stay in place, four effects remain significant ( to points, including the 8B model) and three weaken. Decomposing repairs after NameErrors shows why: when the missing name is displaced, agents rarely adopt it (4%), and when it stays in place, but its frame is scrambled, adoption falls from 54% to 35%. What agents use is the position of the actionable token and the conventional frame around it. Within Qwen3-14B, suppressing reasoning leaves the model retrying without acting on the message at all. Sensitivity decays across successive errors, and on HumanEval, capable models rarely fail, so the contrast is undefined; we report both as scope conditions, and we release all trajectories and analysis code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, tool-using agents, counterfactual error scrambling, execution feedback, self-repair, agentic AI, behavioral evaluation, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ejgcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether AI coding agents actually read error messages, or only notice that something went wrong. It shuffles the words in error messages while keeping the names and layout intact. Most strong models do noticeably worse with the shuffled messages, which shows they really are reading them.

Critical strengths

The test is cleanly designed. The authors were also honest enough to find a bug in their own shuffler.

Critical weaknesses

The main limitation is that almost all the errors are of one kind. The finding is therefore mostly “agents copy the missing name from the error,” not that they understand error messages in general. The main result also includes cases where the shuffling bug moved the name out of place, which inflates it. With those cases removed, four of the seven models still show a clear effect. I’d like to see a test with other kinds of errors.

Questions for authors
  1. Why was the function name, or the example tests that reveal it, left out of the task description?
  2. Does the finding that word order around the name matters hold if you count only the first error in each attempt and compare like with like? I would also suggest you repeat the test on tasks that produce a wider variety of errors.
3 comments

Thank you for the precise questions. We answer each one with data.

 

"The main result includes cases where the shuffling bug moved the name." Our audit confirmed that identifier displacement affected the original estimates more substantially than our pinned-task analysis indicated. We reran Real and Scrambled on the same 100 tasks with role-based pinning, under which no program-referring token can move (none of the 1,020 scrambled NameErrors loses its name). Recovery under Real stays within 5.1 pp of the original for six of the seven models (Gemini 2.5 Flash recovers more often in the rerun, 0.66 vs. 0.46), but recovery under Scrambled rises for every model. The Real–Scrambled difference falls from +13 to +30 pp to −2 to +10 pp, and only Ministral-3 8B remains significant after Holm correction (+10.1 pp [3.8, 16.7]). Pooled task success still favors Real (56 vs. 28 discordant task–model pairs, p = 0.003). The revision reports the rerun beside the original contrast (§4.1, Table 25, new App. R) and bases its conclusions on it.

 

Q2: first error only, like with like. For each model and task we paired the first tool error under Real and under Scrambled, keeping the pair only if the original error was identical. No message has been shown before it. Adoption of a missing name that stays in its slot:

TransformerPairsReal → ScrambledDiscordantp (exact McNemar)
Original (seven models)39371.5% → 53.9%84 / 158×10⁻¹³
Role-based pinning, rerun (six models)32272.0% → 65.2%37 / 150.003

The frame effect holds at the first error, but with no other program token displaced it is about 40% of the original estimate for the same six models (6.8 vs. 17.1 points; App. R). Names that the original transformer displaced were adopted in 7.0% of cases, against 50.0% under Real (38 vs. 1). New App. P.

Q3 and "other kinds of errors". We built an error-variety probe (new §4.7, App. Q) from failures that models actually produced. It uses 155 failing submissions that models wrote in the main experiment, from 53 tasks: 103 wrong outputs, 39 TypeErrors and 13 others. Every trial starts from the same failing submission and its first error, so all comparisons are first-error and like-with-like by construction. First-resubmission success, pooled over the six content-sensitive models:

Exception class (instances)RealScrambledClassOnlyEmpty
Wrong output, AssertionError (103)64.9%63.3%26.9%21.0%
TypeError (39)38.5%33.8%26.9%26.1%
Other classes (13)62.8%48.7%55.1%44.9%
All natural failures (155)58.1%54.6%29.2%24.3%
Control: defect visible in the code (27)84.6%84.0%81.5%82.7%

Removing only the detail after the exception class halves repair: Real exceeds ClassOnly by 28.8 pp [24.1, 34.0] (p < 10⁻⁵). The difference is significant after Holm correction in each of the seven models, including Llama-3.1-8B (12.9% vs. 3.9%). Scrambling the surrounding words costs 3.4 pp [1.2, 5.9]. It costs nothing detectable for wrong outputs, whose information is the failing assertion itself. So agents do use error content beyond a missing name: the failing test and its expected value, or the offending types. When the defect is visible in the code, the message is not needed.

Q1: why the prompt omits the function name. The user turn contains only the dataset's task description, and the tests serve only as the hidden grader. The entry-point name therefore first appears in an error message, which makes the first error mostly a NameError. The submission reports this (§4.2, §4.6) without discussing it; the revision states it and its consequences (§3.2, App. S). The probe states the function name, and all but one of its 155 submissions define it at module level (App. Q).

While re-auditing the harness for this question, we also found two sandbox restrictions: it blocks imports and class definitions, and it does not execute the dataset's test imports. As a result, five tasks cannot be solved in any condition. The restrictions are identical across conditions, and excluding these tasks changes no significance decision. The probe uses a corrected sandbox.

Dear Reviewer,

Since our previous response, we have conducted an additional experiment that addresses your concern about the dominance of NameErrors in MBPP and the need to evaluate other error types.

 

New BFCL v4 multi-turn evaluation (§4.7, Appendix T). We evaluated the protocol using BFCL's native multi-turn execution loop, stateful APIs, and evaluation checker, rather than our MBPP harness.

Across six models and 1,200 initial episodes, 559 episodes contained a tool error. We obtained 479 eligible first-error replay instances from 133 tasks, comprising 455 API error responses and 24 execution exceptions.

On these instances (brackets give 95% confidence intervals):

  • The error-containing turn succeeds under Real in 57.8% of cases, versus 29.6% under ClassOnly and 32.2% under Empty.
  • Real exceeds ClassOnly by 28.2 percentage points [20.8, 35.7], while scrambling the surrounding words reduces success by only 5.8 points [1.3, 10.3].
  • Final task success is 44.9% under Real versus 20.5% under Empty, a difference of 24.4 percentage points [18.0, 31.2].

 

Together with the previously reported 155-failure error-variety probe, these experiments show that the findings extend beyond missing-name recovery, MBPP, and a single code-submission tool. They also reinforce our central distinction between information-bearing error content and the arrangement of surrounding words.

The new results and experimental details are included in the revised manuscript, with changes highlighted in blue.

 

We would appreciate your assessment of whether this additional evidence addresses your concern about error diversity and the scope of the findings.

Dear Reviewer,

As the discussion period draws to a close, we would appreciate your assessment of whether our revised manuscript and additional experiments, including the new BFCL v4 multi-turn evaluation posted on November 17, address your original concerns.

We would be glad to provide any further clarification if there are remaining questions or issues that we have not fully addressed.

Reviewer_WH9pOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2(+1)
Presentation
3(+2)
Soundness
3(+2)
Read review
Paper summary

The paper asks whether tool-using LLM agents act on the content of execution error messages or only on the fact that an error occurred. It proposes counterfactual error scrambling: each error message is replaced by a variant that keeps its length, line structure, exception class and identifiers but permutes the remaining words, with misleading and empty messages as controls.

Critical strengths
  1. The topic is interesting and important. Understanding whether agents actually read error content gives a deeper view of agentic error handling, and intervening at the environment–agent boundary is a clean way to isolate it.
  2. Some of the analyses are interesting. For example, a displaced missing name is adopted only 3.8% of the time, versus 54% when it stays in its slot (Table 2). Also, Qwen3-14B with reasoning off resubmits byte-identical code on 69% of its re-attempts (Table 4).
  3. The paper offers useful feedback for future tool design for LLMs and agents. The finding that agents depend on the conventional error frame and slot position suggests that changing error formats could break the signals agents rely on (Sec. 5).
Critical weaknesses
  1. Missing related work on agent failure analysis and attribution, such as MAST (Cemri et al., 2025), Who&When (Zhang et al., 2025), AgentDebug (Zhu et al., 2025) and AgentRx (Barke et al., 2026). The related work section covers self-repair and input-perturbation studies, but does not position the paper relative to this line of work on why and where agents fail.
  2. The evaluation is essentially limited to one benchmark. All main results come from 100 MBPP-sanitized tasks with a single submit_code tool. HumanEval is used only as a scope check, where the contrast is undefined. Because MBPP tests call function names that the prompt never states, 72.3% of errors are NameErrors, and the semantic-error contrast is underpowered (the authors acknowledge this in Sec. 6). It is therefore unclear whether the conclusions generalize. Benchmarks with richer tool errors, such as ToolBench, τ-bench or BFCL, should be considered.
  3. The study covers only one failure type: error messages returned in tool outputs. Agentic systems fail in many other ways, for example in planning, memory, reasoning, or misreading environment state. It would be valuable to see how agents handle these other kinds of failures, or at least to discuss whether the findings are expected to transfer.
Questions for authors
  1. Could the analysis go deeper, for example into the latent space? Several of the evaluated models are open-weight (e.g., Llama-3.1-8B, Qwen2.5-7B, Qwen3-14B). Probing hidden states or attention to the error message under REAL / SCRAMBLED / MISLEADING, and with reasoning on vs. off, could show how each condition changes the model's internal handling of the error. It might also help explain why the 2024 models show no effect.
5 comments

Thank you for the detailed review and the pointers to related work.

 

W1: related work. We added a paragraph to §2. MAST, Who&When, AgentDebug, and AgentRx diagnose where and why agents fail from completed trajectories. They build failure taxonomies, attribute a failure to an agent and a decisive step, or localize the critical step. Our protocol is complementary. It intervenes on one input channel at the environment–agent boundary while holding the environment fixed, so it tests causally whether that channel shapes the next action. Attribution methods could use it to check a hypothesized cause such as "the agent ignored the error". We also cite work that trains recovery from execution errors (e.g., Fission-GRPO). That work presupposes that agents use error content; the protocol measures how much they do.

 

W2: one benchmark, 72% NameErrors. To assess whether our findings extend beyond NameErrors, we conducted two additional experiments:

  • An error-variety probe built from 155 failing submissions that models wrote: 103 wrong outputs, 39 TypeErrors and 13 others, only 2 of them NameErrors. The first error is prefilled and shown as Real, Scrambled, ClassOnly or Empty. With the real message, the six content-sensitive models fix 58.1% on the first resubmission; with only the exception class, 29.2%; with an empty message, 24.3%. On semantic errors, where the submission's Real–Scrambled contrast was underpowered, the detail of the message now shows a large effect: wrong outputs are fixed in 64.9% vs. 26.9% of cases (Real vs. ClassOnly), while scrambling the words around the failing assertion changes little (+1.6 pp, p = 0.29).
  • A rerun with role-based pinning, under which no program-referring token can move. It shows that displacement inflated the submitted Real–Scrambled estimates. With no program-referring token displaced, the difference falls to −2 to +10 pp, significant for Ministral-3 8B only. The tokens and their positions carry most of the effect; the arrangement of the other words adds little.

 

Both stay on MBPP and in the code domain. The protocol needs only a textual tool error at the environment boundary, so it applies directly to BFCL multi-turn (errors returned by its stateful APIs) and to τ-bench (database and policy errors returned as tool outputs). App. Q ("Application to function-calling benchmarks") specifies what to pin in each. We could not run these benchmarks to the same standard within the discussion period, and §6 lists them as the next step.

 

W3: other failure types. The protocol needs a textual observation to intervene on, so planning, memory and reasoning failures that produce no message are outside its direct scope. Misread environment state is within reach: the same counterfactual applies to any observation, for example scrambling a state description while pinning entity names. Our results suggest what to test there: whether agents use the tokens that carry information they lack and, where a conventional template exists, their position in it. The new Discussion paragraph "Beyond tool errors" (§5) states this scope and that any textual observation can be rendered counterfactually in the same way.

 

Q: latent space, and why the 2024 models show no effect. We agree that this is a valuable direction. Behavioral measures already show that the 2024 models process the message without acting on it in the main experiment (§4.5). Llama-3.1-8B re-attempts after 47% of Real errors but only 14% of Scrambled errors. Qwen2.5-7B and Qwen3-14B, without reasoning, resubmit identical code in 69–84% of re-attempts when the message has content, compared to 43–57% when it is empty. In the new probe, which starts after a well-formed tool call, Llama-3.1-8B does use the content (12.9% vs. 3.9% with the class alone). This suggests that its null result in the main experiment comes partly from failing to act at all rather than from ignoring the message. A teacher-forced probe on the open-weight models would test where this breaks down: the likelihood of the reference repair, and attention to the name token, under each condition. We list it as future work in §6.

I would like to raise my score from 2 to 2.5 which means neutral to the paper. Since there is no such option, I will keep the original score, but the AC can take into consideration.

The remaining suggestion would be to improve the overall presentation of the paper, to make it more clear and vivid with good diagrams and case analysis.

And the other limitation is I think the contribution is quite limited in a narrow scenario situation that I don't think it's the most serious issue of the agentic system right now.

And the evaluation is mainly based on MBPP, humaneval right now, which is quiet old. And I would also like too see more agentic harness taken into account.

Thank you for the follow-up and for reconsidering your assessment. We have conducted an additional evaluation on BFCL v4 multi-turn and improved the presentation to address your remaining points. New material in the revised PDF is highlighted in blue.

 

1. Presentation, diagrams, and case analysis.

  • A new Figure 1 illustrates the protocol using an actual model-generated failure. The submission returns the longest list, whereas the tests expect the pair (length, list), information absent from the task description.
    • All seven models successfully repair the submission after Real and Scrambled messages, where the failing assertion is preserved.
    • None succeeds after ClassOnly or Empty messages.
  • The figure also decomposes first-attempt repair across 155 natural failures: Empty yields 24.3% success; the exception information adds 4.9 percentage points, the information-bearing message content adds 25.4 points, and word order adds 3.4 points.
  • Figure 2 now directly compares the Real–Scrambled estimates before and after role-based pinning, per model.
  • Appendix M provides additional annotated cases, including the same task and model under both scrambling transformers.

2. Significance of the problem.

Error-message processing is a fundamental component of agentic execution and recovery, not merely an isolated debugging scenario. Tool errors expose failures that agents must interpret to complete their tasks, and self-repair methods rely on agents extracting actionable information from this feedback.

Our new BFCL evaluation provides direct evidence of its importance:

  • 47% of the 1,200 evaluated episodes contain a tool error.
  • On the instances built from these errors, changing the rendering from Real to Empty reduces final task success by 24.4 percentage points.

These results show that error-message interpretation has substantial consequences for task completion in multi-turn, stateful agent environments.

Beyond measuring recovery, our protocol identifies which components of feedback agents actually use: the information-bearing tokens and their conventional positions, much more than the surrounding wording. The transformer audit further demonstrates why controlling token position is essential for reliable counterfactual evaluation.

3. Evaluation on a recent benchmark and agentic harness.

We extended the experiments to BFCL v4 multi-turn (bfcl-eval 2026.3.23; §4.7 and Appendix T), comprising 200 tasks across stateful APIs for file systems, messaging, trading, travel, vehicle control, and other functions.

The experiments use BFCL's own multi-turn execution loop, environments, and evaluation checker rather than our MBPP harness.

Experimental design:

  • Each model initially executes all tasks with real error messages.
  • The first tool error in an episode whose earlier turns passed becomes a replay instance.
  • Each instance is reconstructed exactly up to the failing call, then evaluated under Real, Scrambled with role-based pinning, ClassOnly, and Empty.

 

Results across six models, 479 instances from 133 tasks:

  • The error-containing turn succeeds in 57.8% under Real, 52.0% under Scrambled, 29.6% under ClassOnly, and 32.2% under Empty.
  • Real exceeds ClassOnly by 28.2 percentage points [+20.8, +35.7], closely reproducing the 28.8-point effect in the error-variety probe.
  • Scrambling the surrounding words costs only 5.8 points [+1.3, +10.3], confirming that information-bearing content matters considerably more than word arrangement.
  • After an Empty message, the next call repeats the failing call in 31% of cases, compared with approximately 5% under Real or Scrambled.
  • The effect extends beyond immediate recovery: Real improves final task success over Empty by 24.4 percentage points [+18.0, +31.2].

Harness validation:

  • The ground-truth solutions pass all 200 tasks.
  • Real replay and the original continuation agree on 446 of 479 instances.

The MBPP findings also cannot be explained simply by familiarity with an older benchmark. Under Empty feedback, the hidden function name is adopted in only 14 of 1,777 post-NameError resubmissions (Appendix H).

Together with the error-variety probe, the new BFCL experiments show that our central findings extend beyond MBPP, NameErrors, and a single code-submission tool. They hold in a multi-turn function-calling benchmark using its native agentic harness and stateful environments.

Across both settings, agents rely primarily on the words of an error message that carry information the code or call does not, while the arrangement of the surrounding words contributes much less.

 

We hope the revision resolves the three remaining points, and we would be glad to clarify anything that is still open.

Reviewer_M95wOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

this paper introduces counterfactual error scrambling, a method for studying whether coding agents actually use the content of execution error messages when attempting to repair code. the authors compare real errors with scrambled, misleading, and empty error messages across nine models on mbpp-sanitized. they find that seven models recover more successfully when shown real errors, although this drops to four models when restricting the analysis to tasks where identifiers are preserved in their original positions. the authors also investigate how models use identifier positions, error-message templates, and reasoning modes when repairing code.

Critical strengths

i think the general question is interesting. understanding whether models actually use information from tool outputs, rather than just responding to the presence of an error, is relevant for studying agent behavior.

i appreciate that the authors audit their own scrambling method and report how the results change when identifiers are preserved. the behavioral analysis of how models respond to missing function names is also useful.

Critical weaknesses
  1. i'm not convinced the paper really measures semantic understanding of error messages.the main motivation is to determine whether models use the meaning of error messages or simply respond to the presence of an error. however, the majority of errors in the benchmark are nameerrors, where the relevant information is often just the name of a missing function. In these cases, the model doesn't necessarily need to understand the meaning of the error message in any sophisticated way. it can simply recognize a familiar traceback template and copy the missing identifier into its next code submission.The authors' own analysis suggests this is exactly what is happening. they find that models are particularly sensitive to the position of the identifier and the conventional wording around it.

i think this weakens the paper's main motivation.

  1. the scrambling intervention does not cleanly isolate the variable the authors want to study. one of my biggest concerns is that the scrambling procedure changes more than just the semantic content of the error. the authors acknowledge that for 25 of the 100 tasks, the missing identifier can be moved from its original position. this seems like a pretty significant problem given that the goal is to preserve actionable information while removing compositional meaning.
Questions for authors

the experiments are conducted primarily on 100 tasks from mbpp-sanitized, using a single tool that executes submitted code. do you think this would generalize? do you have any results outside to other domains?

2 comments

Thank you for the review and for noting the self-audit.

 

W1: does the paper measure semantic understanding? The paper does not aim to establish deep semantic understanding. Rather, it investigates how agents use different components of execution-error messages, and the additional experiments help distinguish these sources of information. The Introduction frames the question as a decomposition: does the next action depend on the error's presence, a token in it, the token's position, or the message's meaning? The revision adds that the protocol separates these sources rather than presupposing that agents read messages compositionally. Copying the name is part of the answer. Position matters too: a name that is present but displaced is adopted in only 3.8% of cases. In a rerun where no program-referring token can move, scrambling the words around a name in its slot still lowers adoption at the first error (72.0% → 65.2%, p = 0.003), but much less than with the original transformer. Beyond NameErrors, an error-variety probe on 155 failures that models actually produced (wrong outputs, type errors and others; new §4.7) shows that the detail after the exception class doubles first-attempt repair (58.1% vs. 29.2%). That detail is the failing test with its expected value, or the offending types. Scrambling the surrounding words costs only 3.4 points there. Agents use the information-bearing content of an error and its position more than its phrasing, and this is the claim the revision makes.

 

W2: the scrambler can move the name. The role-based pinning rerun confirms that identifier displacement affected the original estimates more substantially than our pinned-task analysis indicated. Under this stricter transformation, no program-referring token can move, and the Real–Scrambled difference falls from +13 to +30 pp to −2 to +10 pp. Only Ministral-3 8B remains significant after Holm correction, and pooled task success still favors Real (p = 0.003). The revision reports the rerun alongside the original contrast (§4.1, Table 25, App. R) and bases its conclusions on it.

 

Q: generalization. The new probe broadens the kinds of error, mostly wrong outputs and type errors in place of a missing name, but it stays on MBPP and in code execution. We do not claim more than that. The protocol needs only a textual tool error, so it carries over to function-calling benchmarks such as BFCL and τ-bench. App. Q ("Application to function-calling benchmarks") describes the mapping, and §6 lists these benchmarks as the next step.

Dear Reviewer,

Since our previous response, we have conducted an additional experiment that answers your question about generalization beyond MBPP and a single code-execution tool. Our earlier response noted that the error-variety probe remained within MBPP and identified BFCL as a next step. We have now completed that evaluation.

 

New BFCL v4 multi-turn evaluation (§4.7, Appendix T). We evaluated the protocol using BFCL's native multi-turn execution loop, stateful APIs, and evaluation checker, rather than our MBPP harness. The tasks are function calling over stateful APIs (file system, messaging, trading, travel, vehicle control), not code generation.

Across six models and 1,200 initial episodes, 559 episodes contained a tool error. We obtained 479 eligible first-error replay instances from 133 tasks, comprising 455 API error responses and 24 execution exceptions.

On these instances (brackets give 95% confidence intervals):

  • The error-containing turn succeeds under Real in 57.8% of cases, versus 29.6% under ClassOnly and 32.2% under Empty.
  • Real exceeds ClassOnly by 28.2 percentage points [20.8, 35.7], while scrambling the surrounding words reduces success by only 5.8 points [1.3, 10.3].
  • Final task success is 44.9% under Real versus 20.5% under Empty, a difference of 24.4 percentage points [18.0, 31.2].

 

Together with the previously reported 155-failure error-variety probe, these experiments show that the findings extend beyond missing-name recovery, MBPP, and a single code-submission tool. They also reinforce our central distinction between information-bearing error content and the arrangement of surrounding words.

The new results and experimental details are included in the revised manuscript, with changes highlighted in blue.

 

We would appreciate your assessment of whether this additional evidence answers your question about generalization.

Comments on the paper

2 comments
AuthorsOpenReview ↗

New evidence and changes in the revised PDF

We thank the reviewers for their careful reviews. All three raise one shared concern: the evidence comes mostly from NameErrors on one benchmark, so the paper may show only that agents copy a missing name. Two reviewers also ask whether identifier displacement inflates the main result. We ran two new experiments on the seven models still served by our gateway; Claude Sonnet 4 and Qwen2.5-7B have been withdrawn by their providers. The results confirm the second concern and answer the first. They change how we state the main result, and the revised PDF reflects this (changes in blue).

 

  1. Rerun with role-based pinning (App. R). We implemented the fix proposed in our Discussion: every quoted span, every name in the submitted code and every name() call is pinned, so no program-referring token can move. None of 1,020 scrambled NameErrors loses its name. Displacement inflated the submitted estimates by more than our pinned-task analysis indicated. Recovery under Real remains close to the original (within 5.1 pp for six of seven models), whereas recovery under Scrambled increases. The Real–Scrambled difference falls from +13 to +30 pp to −2 to +10 pp, and only Ministral-3 8B remains significant after Holm correction (+10.1 pp [3.8, 16.7]). Pooled task success still favors Real (56 vs. 28 discordant task–model pairs, p = 0.003). The new rule also leaves more words in place (35% of natural-language words move, against 51% before), so the rerun measures the effect of scrambling a smaller part of the frame (§6, App. R).
  2. Error-variety probe (new §4.7, App. Q). We use 155 failing submissions that models wrote in our experiments: 103 wrong outputs (AssertionErrors indicating the failing test), 39 TypeErrors, and 13 others, across 53 tasks. Each trial starts from the same failing submission and its first error, shown as Real, Scrambled, ClassOnly (final line cut to the exception class) or Empty. Pooled over the six content-sensitive models, the first resubmission passes in 58.1% of trials under Real, 54.6% under Scrambled, 29.2% under ClassOnly and 24.3% under Empty. Real exceeds ClassOnly by 28.8 pp [24.1, 34.0] (p < 10⁻⁵). A control of 27 bugs whose defect is visible in the code is repaired at the same rate in every condition (84.6% under Real, 82.7% under Empty).
  3. First error, like with like (§4.2, App. P and R). For a missing name that stays in its slot, adoption falls from 71.5% to 53.9% with the original transformer (393 pairs, p = 8×10⁻¹³). In the rerun, on the six of these models still served, it falls from 72.0% to 65.2% (322 pairs, p = 0.003).
  4. Related work on agent failure analysis and attribution (MAST, Who&When, AgentDebug, AgentRx) and on training recovery from errors.
  5. Harness details (App. S). We state that the prompt omits the function name and what follows from it, and we disclose two sandbox restrictions. Excluding the five tasks they make unsolvable moves every estimate slightly away from zero (by at most 1.8 pp) and changes no significance decision.

 

What the evidence now shows. The bare fact of an error is worth almost nothing: task success under Real vs. Empty differs on 413 vs. 1 task–model pairs. The information-bearing tokens of a message are worth a great deal: a missing name, a failing assertion with its expected value, the offending types. In the error-variety probe, the detail after the exception class adds 28.8 pp over the class alone. These tokens help only where conventions place them: a displaced name is adopted in 3.8% of cases. The arrangement of the remaining words is worth little: 3.4 pp in the probe, and a mean of 4.4 pp across models in the rerun. Agents act mainly on the payload of an error message and only a little on its prose. This tells tool designers which parts of a message carry the signal.

Since this summary, we have added a third experiment, a new Figure 1, a revised Figure 2 (each model's estimate before and after role-based pinning) and more annotated cases in Appendix M. The experiment is a first-error replay on BFCL v4 multi-turn, run in BFCL's own harness (§4.7, Appendix T). On 479 first errors from 133 tasks, Real exceeds ClassOnly by 28.2 pp [20.8, 35.7], against 28.8 pp in the error-variety probe, and scrambling the surrounding words costs 5.8 pp [1.3, 10.3]. BFCL therefore confirms the main conclusion above outside MBPP and code execution: agents use the information-bearing content of an error message much more than the arrangement of its words. Details are in our replies to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.