acceptodds
Under review as a conference paper at ICLR 2027

Code Understanding is a Bottleneck for Coding Agents

Abstract

Repository benchmarks (e.g., SWE-bench) for coding agents often assume that edit volume can predict task difficulty, but their limited control over code and task types makes it hard to discern which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. Our framework designs tasks from scratch as call graph transformations and scales their size over four axes: function traversal, search, runtime analysis, and instruction following. We evaluate eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while on SWE-bench Verified these tool call counts predict agents' task accuracy better than edit volume—suggesting task difficulty for agents can lie in *understanding* code to edit, not solely in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this hypothesis, as agent accuracy finally falls. Overall, we argue for revisiting synthetic evaluation protocols like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond editing (e.g., understanding) that coding agents still find difficult—complementing repository-based tasks with controlled diagnosis.

Primary area:
datasets and benchmarks
Keywords:
coding agents, evaluation, agent analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ak3HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CABRA, which generates code editing tasks from directed acyclic call graphs and uses a task scale parameter to raise difficulty along four directions, function traversal, function search, runtime resolution, and instruction following. The authors run eight LLMs and six coding agents on 6840 tasks and report that LLM accuracy falls as scale increases, while agents are mostly able to stay near a perfect score and put more tool calls toward reading and analyzing code. The authors also compare the association of tool call metrics and edited line counts with accuracy on published SWE-bench Verified trajectories, and add a task that extracts equivalent logic across two classes, reporting that agent accuracy on this task falls as the number of code lines grows.

Critical strengths

The evaluation provides a fairly ample amount of information. The same task scale is used to report LLMs without tools, agents with tools, SWE-bench Verified trajectories, and the merge task together, which makes it possible to compare edit volume directly with reading and analysis tool calls.

Critical weaknesses

The code understanding bottleneck in the title is larger than the content the experiments can separate. On the first four scaling axes, reading and analysis tool calls increase with scale, while agent accuracy remains near a perfect score. In the merge task, behavior remains correct in more than 90% of cases, and the failures are mainly that the subclass retains too many lines of code. On SWE-bench Verified, the correlation between understanding call counts and accuracy is about -0.200 and the correlation for edited line counts is about -0.159, while in the appendix regression total tokens fit success and failure better than understanding call counts.

The programs used to represent code understanding are themselves very abstract. The base task is refactoring on random mathematical call graphs, the merge task is a calculator class with rewriting, and both lie rather far from code understanding in a repository. Readers therefore find it hard to judge which real programming ability the bottleneck in the title corresponds to.

Questions for authors
  1. Which ability that can be isolated on its own does the bottleneck in the title refer to in the paper? On the first four scaling axes, reading and analysis tool calls increase with scale, while agent accuracy remains near a perfect score. The drop in accuracy appears in the merge task, and this task requires both correct behavior and fewer lines retained in the subclass. Which result separates "understanding" from search, instruction following, and the minimal change constraint?
  2. Are the failures measured on the merge task a failure to understand the code, or a failure to complete the required extraction? The paper states that behavior remains correct in more than 90% of cases, and that the failures are mainly that the subclass keeps too many lines. Please supply how much of the drop in accuracy with the number of code lines remains when scoring only by whether behavior is preserved.
  3. After controlling for total tokens or trajectory length, how much predictive power do understanding call counts still have relative to edited line counts? The correlations in the main text are about -0.200 versus -0.159, and the fit of total tokens in the appendix is already stronger.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ak3H

Thank you for the thoughtful review! We address your points below:

Which result separates "understanding" from search, instruction following, and the minimal change constraint?

On L290, we define “code understanding” as reasoning about a program’s construction and behavior before editing, approximated using Read and Analyze tool calls. Our merge codebases task targets this understanding: as difficulty increases, Analyze calls rise sharply and accuracy drops, while Search, Edit, and Test calls and the number of instructions remain stable. This shows that understanding is the main source of difficulty and poor agent accuracy in this task.

Are the failures measured on the merge task a failure to understand the code, or a failure to complete the required extraction?

As discussed on L420, most errors in the merge codebases task stem from understanding which lines of code are semantically equivalent, rather than failing to complete the required extraction. Our task requires agents to (1) identify divergent lines of code between codebases and (2) move identical code into a parent class while preserving its behavior. Agents preserve the program’s behavior in >90% of cases, so they successfully extract something but don’t understand the code’s behavior well enough to extract the right thing. Further, accuracy improves when we tell agents that there is only a one-line difference (Figure A.20). Taken together, these suggest (1) is the source of difficulty

After controlling for total tokens or trajectory length, how much predictive power do understanding call counts still have relative to edited line counts?

Great question! To show the predictive power of understanding call counts, we re-run the regressions in Figure 4, adding each predictor separately to a model that includes total tokens. We report how much each predictor lowers AIC, with larger reductions indicating better model fit. After controlling for total tokens, only understanding-relevant tool call counts (Read, Analyze, Read + Analyze) improve model fit, shown via positive AIC reduction. In contrast, lines edited and edit tool calls provide no additional signal.

MetricAIC Reduction (↑\uparrow)
Analyze5.7
Understand (Read+Analyze)3.8
Read0.4
Test-0.3
Search-0.5
Files Edited-1.5
Other-1.6
All Tools-1.6
Edit-1.9
Lines Edited-2.0

Overall, this provides more evidence that understanding underlies task difficulty more than the number of lines edited. We’ll add this regression to Section 4.3 to strengthen our claims!

AuthorsOpenReview ↗

Reminder: Reviewer Response

Thank you again for your feedback! As the discussion period ends today, we would love to hear your thoughts on our response and whether it addresses your concerns

Reviewer_eGrcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CABRA: a controllable synthetic evaluation framework for coding agents in which tasks can be programmatically generated on four axes of task difficulty: function traversal, search, runtime resolution, and instruction following, as well as a fifth scalable refactoring task (section 5). This approach controllably discovers areas where LLMs and agents struggle: while scaling for all four axes hurts plain LLM performance, only instruction-following (increasing # of instructions to address) hurts agents. The equivalence refactoring task, designed to test agent understanding of increasingly complex code, further shows areas where agents struggle. The mechanistic task construction in CABRA provides a tool for researchers to carefully analyze properties of coding agent problems that explain their shortcomings.

Critical strengths
  1. I think this work is well motivated: controllable evaluation of coding agents is highly useful for more precisely understanding what makes certain tasks easy or hard for agents, and how their design and training of the underlying LLM should adapt.
  2. Likewise, I also think the controlled evaluation is carefully designed: each perturbation is well motivated and the scalable controls are well defined.
  3. Some of the controlled tasks do point to clear areas for future work on agents: the equivalence refactoring experiment in 5.2 suggests that current coding agents might struggle to safely refactor code. I think the paper could be even stronger if the connection could be shown with real-world refactoring or understanding tasks.
Critical weaknesses
  1. I think the key shortcoming of this current work is that very few of the perturbations resulted in tasks that were challenging for agents, and for those where failures arose, analysis was limited to quantifying these failures. While it was a bit surprising that agents performed well on most perturbations, I think more of the paper should be devoted to the cases where agents struggle: analyzing the cause of their failures, proposing solutions, or possibly evaluating further perturbations. For instance, agents struggle with the refactoring problem, but what explains their failures? Similarly, for instruction-following: can agents not track the number of instructions to follow or do they just run out of steps for implementation?
  2. Overall, I think the analysis in 4.3 isn’t very convincing. In particular, I think the interpretation of the point-biserial correlation values (Figure 4, right, L351-356) does not clearly state its findings. My understanding is the rr of −.2-.2 for UNDERSTANDnum\text{UNDERSTAND}_{\text{num}} indicates that agents which use more tool calls associated with reading or analyzing code perform worse on average. Further, higher or lower # of calls though would not define better or worse understanding. I think its plausible this is instead explained by # of understanding steps being a proxy for difficulty, rather than an indication that agents do or do not understand code. I think the text in L351-356 should clarify this correlation is negative, and possibly interpret it differently, as currently it states that code understanding plays a key role in evaluating coding agents, which is a little vague and I don’t think clearly follows from this result. Similarly, I don't fully agree with finding #2 in the abstract in its interpretation of 4.3.

To improve the paper: I think the paper offers a good explanation of why the tasks which were easy for agents were easy: tools support scalable search and code execution. I think the paper could be improved by providing analyses which explain why the tasks which are hard for agents are hard, and perhaps finding other tasks which are hard for agents (W1). Further, I think the paper could be improved with a better analysis connecting the synthetic tasks to those in the real-world (W2).

Questions for authors

Comments/Typos/Suggestions:

  1. VstartV_{start} and VinitV_{init} seem to both be used for the same variable.
  2. Iff is used for ‘if and only if’ but I think it would read more clearly spelled out
  3. SWE Refactor or similar benchmarks might be useful for repeating an analysis like the one in 4.3 in relation to the equivalence-based refactor perturbation in section 5.
  4. I think the LLM-only (no-tools) setting isn’t critical to the paper, as the paper’s main aims are to evaluate coding agents (with-tools). This part could be moved to an appendix.
2 comments
AuthorsOpenReview ↗

Response to Reviewer eGrc

Thank you for the detailed review! We address your points below

While it was a bit surprising that agents performed well on most perturbations, more of the paper should be devoted to cases where agents struggle

We highlight the surprisingly strong agent accuracy on most perturbations since it implicates how researchers design synthetic tasks (L483), but we agree that we could spend more time analyzing agent failures. We find the two questions you propose exciting, so we discuss them here:

  1. Agents struggle with the refactoring problem, but what explains their failures?

Section 5.2 shows agents fail to reason over multiple lines of code and over-rely on surface-level naming (Figure 6), and Appendix A.10 has model-specific quirks, like Gemini struggling with distributive multiplication. Agents preserve code behavior in >90% of cases, so failures stem from locating these differences versus making edits (L420–425). Figure A.20 further backs this: agent accuracy improves when they know there is just a one-line difference. Thus, it’s hard for agents to verify their solution is minimal without fully understanding the code (but behavior preservation is easily checked with tests on sampled inputs). We’ll reference these Appendix results to better explain why the merge task stresses code understanding!

  1. Can agents not track the number of instructions to follow or do they run out of steps?

No agent runs time out, so all failures stem from mistakes in the task. The table below digs deeper into instruction-following failures by disentangling the fraction of instances where the solution: (1) made a mistake in instruction following (“instr”), (2) made an error that affected code behavior (“behav”), or (3) gave code that didn’t run (“invalid code”). The last four columns break down “instr” failures (omitted a rule, misapplied a rule but with wrong parameters, or added a rule not asked for).

Setting# RunsAcc.Fail: instr.Fail: behav.Fail: invalid codeInstr. OmittedInstr. MisappliedInstr. Added (correct fn)Instr. Added (wrong fn)
Agents8,64082.815.91.30.013.43.52.71.0
LLMs11,52044.048.22.55.339.97.514.311.7

Failures are overwhelmingly due to instruction following errors. Most failures involve omitted rules, but LLMs also fairly often added extra rules. A per-model breakdown (omitted for brevity, but we can share it if needed) shows that only weaker agents fail on this task.

Thank you for the great suggestion! We'll add a paragraph to Section 4.1 titled “Diagnosing Instruction-Following” with these results.

The interpretation of the point-biserial correlation values does not clearly state its findings

We agree that L351–356 could be more precise. Your Figure 4 interpretation is right: agents that make more understanding tool calls tend to have lower accuracy, suggesting understanding effort is a proxy for task difficulty. We do not claim agents do/do not understand code from this result, but use it to motivate our merge codebases task (§5), where we scale understanding effort by design. As task size grows, understanding tool calls rise and accuracy drops, supporting understanding as an axis of difficulty.

We will note the negative correlation, rewrite L355–356 to claim “understanding effort is a proxy for task difficulty”, and separate the SWE-bench and CABRA results into separate findings in the abstract fore precision. Our response to Reviewer Ak3H has further regressions.

Refactoring benchmarks might be useful for repeating an analysis like the one in 4.3

Great idea! We repeat our analysis from §4.3 on SWE-Refactor-Bench, a recent benchmark of long-horizon refactoring tasks with public agent traces. We run our tool call classifier on 58k calls from Opus 5, GPT-5.6 Sol, and Kimi K3 agent runs, then compute point-biserial correlations between the count of each tool call type and accuracy like Figure 4b:

ToolOpus 5GPT-5.6Kimi K3
Read-0.381-0.157-0.356
Analyze-0.349-0.151-0.428
Search-0.3370.031-0.136
Edit-0.361-0.266-0.432
Test-0.397-0.123-0.187
Understand (Read + Analyze)-0.377-0.158-0.436

For 2/3 models, understanding tool call count (Read + Analyze) correlates more strongly with accuracy than Edit calls, further suggesting that understanding effort may be a better proxy for task difficulty than editing effort. Interestingly, Test calls best predict Opus's accuracy, suggesting verification a difficulty axis to scale in future coding agent benchmarks. We will add this discussion after Section 4.3!

Suggestions

We will standardize to VstartV_{start}, expand “iff”, and consider moving the LLM results if we need more space. We appreciate your attention to detail!

AuthorsOpenReview ↗

Reminder: Reviewer Response

Thank you again for your feedback! As the discussion period ends today, we would love to hear your thoughts on our response and whether it addresses your concerns

Reviewer_pkfdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CABRA, a generator of synthetic coding tasks. It builds programs from random call graphs and turns them into five refactoring tasks, each scaled along 4 axes: traversing functions, searching functions, runtime resolution and following the instructions. The authors ran 8 LLMs without tools and 6 agentic harness on ~7000 tasks. The author shows that LLM accuracy drops considerably as tasks grow, which was not a surprise, and agents stay near-perfect thanks to their ability to call tools and do agentic search and editing. They show that agents' read and analyze tool calls also grow as the size of the task grows. The most interesting part though for me is the new two class code merging task which the authors propose, that finally makes agents fail as the complexity grows. This failure of agents lead authors to conclude that code understanding and not mere editing is the real bottleneck.

Critical strengths

Authors have ensured that they are evaluating frontier class LLMs in this study. I also really liked author's handling of cross agent leakage openly with sandboxed reruns. The biggest strength of the paper though is its handling of the merge task and the corresponding ablations which yeild concrete failure modes.

Critical weaknesses

My main concern is that tools closing the gap on search and tracing tasks is expected. We know it. The paper already cites SWE-agent's agent-computer interface results which mentions this. My other big concern is that LLMs are being made to rewrite the entire codebase in their output while agents are doing edits in place. How fair is it to compare to LLMs esp when large codebases are involved ? The limited choice of a single harness further weakens the paper.

Questions for authors

I will go back to my concern about the choice of using LLMs to rewrite the entire codebase. How would LLMs score at large n with an edit or diff interface instead of full re writes ?

My second question is teasing apart understanding from merge failures. Can you tell understanding failures apart from minimality failures in the merge task ?

Lastly, are the result general enough and hold across other harnesses ?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pkfd

Thank you for the thoughtful review! We address your points below:

Tools closing the gap on search and tracing tasks are expected. SWE-agent mentions this

While SWE-agent shows that tools can improve agents’ code navigation and search, it does not examine their value as task difficulty scales. CABRA instead disentangles tasks where agents maintain accuracy at increasing difficulty with tools (e.g., searching, tracing) and tasks where agents struggle despite tool use (e.g., code understanding is still hard even with AST tools). We will clarify this distinction in the related work.

Can you separate understanding failures apart from minimality failures in the merge task?

We argue that detecting more differences than what actually exists (i.e., a minimality failure) is a failure of code understanding. If agents claim multiple lines differ when only one difference exists, then they have incorrectly reasoned about the behavior and construction of the code---our definition of code understanding (L290). We will clarify this in Section 5.

LLMs rewrite the entire codebase while agents edit in place

As shown on L195, Appendix A.3 compares accuracy when LLMs rewrite the entire codebase versus editing in place with diff patches like the coding agent. LLM accuracy was higher with full rewrites, so we adopt this setup in our evaluations. We use the comparison between LLMs and agents to expose how tool use can trivially solve synthetic tasks (L232), implicating how researchers design such tasks in the future (L483).

The limited choice of a single harness further weakens the paper.

Evaluations with the Copilot harness cost $24,869 (Table A.2), so runs with more harnesses are infeasible. Following benchmarks like SWE-bench Pro, we use a fixed harness to compare capabilities across LLMs and task types (Section 3). We also study the mini-swe-agent harness on SWE-bench Verified (Section 4.3) to offer insights into understanding errors beyond Copilot. We will acknowledge this limitation in the conclusion and highlight that our open-source code enables evaluations with other harnesses.

AuthorsOpenReview ↗

Reminder: Reviewer Response

Thank you again for your feedback! As the discussion period ends today, we would love to hear your thoughts on our response and whether it addresses your concerns

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.