- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CABRA, which generates code editing tasks from directed acyclic call graphs and uses a task scale parameter to raise difficulty along four directions, function traversal, function search, runtime resolution, and instruction following. The authors run eight LLMs and six coding agents on 6840 tasks and report that LLM accuracy falls as scale increases, while agents are mostly able to stay near a perfect score and put more tool calls toward reading and analyzing code. The authors also compare the association of tool call metrics and edited line counts with accuracy on published SWE-bench Verified trajectories, and add a task that extracts equivalent logic across two classes, reporting that agent accuracy on this task falls as the number of code lines grows.
Critical strengths
The evaluation provides a fairly ample amount of information. The same task scale is used to report LLMs without tools, agents with tools, SWE-bench Verified trajectories, and the merge task together, which makes it possible to compare edit volume directly with reading and analysis tool calls.
Critical weaknesses
The code understanding bottleneck in the title is larger than the content the experiments can separate. On the first four scaling axes, reading and analysis tool calls increase with scale, while agent accuracy remains near a perfect score. In the merge task, behavior remains correct in more than 90% of cases, and the failures are mainly that the subclass retains too many lines of code. On SWE-bench Verified, the correlation between understanding call counts and accuracy is about -0.200 and the correlation for edited line counts is about -0.159, while in the appendix regression total tokens fit success and failure better than understanding call counts.
The programs used to represent code understanding are themselves very abstract. The base task is refactoring on random mathematical call graphs, the merge task is a calculator class with rewriting, and both lie rather far from code understanding in a repository. Readers therefore find it hard to judge which real programming ability the bottleneck in the title corresponds to.
Questions for authors
- Which ability that can be isolated on its own does the bottleneck in the title refer to in the paper? On the first four scaling axes, reading and analysis tool calls increase with scale, while agent accuracy remains near a perfect score. The drop in accuracy appears in the merge task, and this task requires both correct behavior and fewer lines retained in the subclass. Which result separates "understanding" from search, instruction following, and the minimal change constraint?
- Are the failures measured on the merge task a failure to understand the code, or a failure to complete the required extraction? The paper states that behavior remains correct in more than 90% of cases, and that the failures are mainly that the subclass keeps too many lines. Please supply how much of the drop in accuracy with the number of code lines remains when scoring only by whether behavior is preserved.
- After controlling for total tokens or trajectory length, how much predictive power do understanding call counts still have relative to edited line counts? The correlations in the main text are about -0.200 versus -0.159, and the fit of total tokens in the appendix is already stronger.
2 comments
Reminder: Reviewer Response
Thank you again for your feedback! As the discussion period ends today, we would love to hear your thoughts on our response and whether it addresses your concerns
Response to Reviewer Ak3H
Thank you for the thoughtful review! We address your points below:
On L290, we define “code understanding” as reasoning about a program’s construction and behavior before editing, approximated using Read and Analyze tool calls. Our merge codebases task targets this understanding: as difficulty increases, Analyze calls rise sharply and accuracy drops, while Search, Edit, and Test calls and the number of instructions remain stable. This shows that understanding is the main source of difficulty and poor agent accuracy in this task.
As discussed on L420, most errors in the merge codebases task stem from understanding which lines of code are semantically equivalent, rather than failing to complete the required extraction. Our task requires agents to (1) identify divergent lines of code between codebases and (2) move identical code into a parent class while preserving its behavior. Agents preserve the program’s behavior in >90% of cases, so they successfully extract something but don’t understand the code’s behavior well enough to extract the right thing. Further, accuracy improves when we tell agents that there is only a one-line difference (Figure A.20). Taken together, these suggest (1) is the source of difficulty
Great question! To show the predictive power of understanding call counts, we re-run the regressions in Figure 4, adding each predictor separately to a model that includes total tokens. We report how much each predictor lowers AIC, with larger reductions indicating better model fit. After controlling for total tokens, only understanding-relevant tool call counts (Read, Analyze, Read + Analyze) improve model fit, shown via positive AIC reduction. In contrast, lines edited and edit tool calls provide no additional signal.
Overall, this provides more evidence that understanding underlies task difficulty more than the number of lines edited. We’ll add this regression to Section 4.3 to strengthen our claims!