- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether AI coding agents actually read error messages, or only notice that something went wrong. It shuffles the words in error messages while keeping the names and layout intact. Most strong models do noticeably worse with the shuffled messages, which shows they really are reading them.
Critical strengths
The test is cleanly designed. The authors were also honest enough to find a bug in their own shuffler.
Critical weaknesses
The main limitation is that almost all the errors are of one kind. The finding is therefore mostly “agents copy the missing name from the error,” not that they understand error messages in general. The main result also includes cases where the shuffling bug moved the name out of place, which inflates it. With those cases removed, four of the seven models still show a clear effect. I’d like to see a test with other kinds of errors.
Questions for authors
- Why was the function name, or the example tests that reveal it, left out of the task description?
- Does the finding that word order around the name matters hold if you count only the first error in each attempt and compare like with like? I would also suggest you repeat the test on tasks that produce a wider variety of errors.
3 comments
Dear Reviewer,
Since our previous response, we have conducted an additional experiment that addresses your concern about the dominance of NameErrors in MBPP and the need to evaluate other error types.
New BFCL v4 multi-turn evaluation (§4.7, Appendix T). We evaluated the protocol using BFCL's native multi-turn execution loop, stateful APIs, and evaluation checker, rather than our MBPP harness.
Across six models and 1,200 initial episodes, 559 episodes contained a tool error. We obtained 479 eligible first-error replay instances from 133 tasks, comprising 455 API error responses and 24 execution exceptions.
On these instances (brackets give 95% confidence intervals):
- The error-containing turn succeeds under Real in 57.8% of cases, versus 29.6% under ClassOnly and 32.2% under Empty.
- Real exceeds ClassOnly by 28.2 percentage points [20.8, 35.7], while scrambling the surrounding words reduces success by only 5.8 points [1.3, 10.3].
- Final task success is 44.9% under Real versus 20.5% under Empty, a difference of 24.4 percentage points [18.0, 31.2].
Together with the previously reported 155-failure error-variety probe, these experiments show that the findings extend beyond missing-name recovery, MBPP, and a single code-submission tool. They also reinforce our central distinction between information-bearing error content and the arrangement of surrounding words.
The new results and experimental details are included in the revised manuscript, with changes highlighted in blue.
We would appreciate your assessment of whether this additional evidence addresses your concern about error diversity and the scope of the findings.
Dear Reviewer,
As the discussion period draws to a close, we would appreciate your assessment of whether our revised manuscript and additional experiments, including the new BFCL v4 multi-turn evaluation posted on November 17, address your original concerns.
We would be glad to provide any further clarification if there are remaining questions or issues that we have not fully addressed.
Thank you for the precise questions. We answer each one with data.
"The main result includes cases where the shuffling bug moved the name." Our audit confirmed that identifier displacement affected the original estimates more substantially than our pinned-task analysis indicated. We reran Real and Scrambled on the same 100 tasks with role-based pinning, under which no program-referring token can move (none of the 1,020 scrambled NameErrors loses its name). Recovery under Real stays within 5.1 pp of the original for six of the seven models (Gemini 2.5 Flash recovers more often in the rerun, 0.66 vs. 0.46), but recovery under Scrambled rises for every model. The Real–Scrambled difference falls from +13 to +30 pp to −2 to +10 pp, and only Ministral-3 8B remains significant after Holm correction (+10.1 pp [3.8, 16.7]). Pooled task success still favors Real (56 vs. 28 discordant task–model pairs, p = 0.003). The revision reports the rerun beside the original contrast (§4.1, Table 25, new App. R) and bases its conclusions on it.
Q2: first error only, like with like. For each model and task we paired the first tool error under Real and under Scrambled, keeping the pair only if the original error was identical. No message has been shown before it. Adoption of a missing name that stays in its slot:
The frame effect holds at the first error, but with no other program token displaced it is about 40% of the original estimate for the same six models (6.8 vs. 17.1 points; App. R). Names that the original transformer displaced were adopted in 7.0% of cases, against 50.0% under Real (38 vs. 1). New App. P.
Q3 and "other kinds of errors". We built an error-variety probe (new §4.7, App. Q) from failures that models actually produced. It uses 155 failing submissions that models wrote in the main experiment, from 53 tasks: 103 wrong outputs, 39 TypeErrors and 13 others. Every trial starts from the same failing submission and its first error, so all comparisons are first-error and like-with-like by construction. First-resubmission success, pooled over the six content-sensitive models:
Removing only the detail after the exception class halves repair: Real exceeds ClassOnly by 28.8 pp [24.1, 34.0] (p < 10⁻⁵). The difference is significant after Holm correction in each of the seven models, including Llama-3.1-8B (12.9% vs. 3.9%). Scrambling the surrounding words costs 3.4 pp [1.2, 5.9]. It costs nothing detectable for wrong outputs, whose information is the failing assertion itself. So agents do use error content beyond a missing name: the failing test and its expected value, or the offending types. When the defect is visible in the code, the message is not needed.
Q1: why the prompt omits the function name. The user turn contains only the dataset's task description, and the tests serve only as the hidden grader. The entry-point name therefore first appears in an error message, which makes the first error mostly a NameError. The submission reports this (§4.2, §4.6) without discussing it; the revision states it and its consequences (§3.2, App. S). The probe states the function name, and all but one of its 155 submissions define it at module level (App. Q).
While re-auditing the harness for this question, we also found two sandbox restrictions: it blocks imports and class definitions, and it does not execute the dataset's test imports. As a result, five tasks cannot be solved in any condition. The restrictions are identical across conditions, and excluding these tasks changes no significance decision. The probe uses a corrected sandbox.