acceptodds
Withdrawn from ICLR 2027

RepoTrace: Improving Requirement Tracing in Multi-Round Agent Interactions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Code agents have become increasingly capable of performing software engineering tasks through interaction with software environments. However, agents tend to degrade as the number of interaction rounds increases, making it difficult to maintain existing software during iterative modification. We analyze 2,656 agent trajectories in EvoCode-Bench and identify requirement focus misalignment as a recurring failure pattern, where agents struggle to distinguish the primary modification targets of a requirement from unrelated code. We propose Repotrace a requirement-driven repository understanding harness that combines repository structure with requirement semantics to construct a requirement-code correspondence and injects it into the agent context. This correspondence enables code agents to follow the repository architecture, trace requirements across multi-round interactions, and identify the core code elements corresponding to the current requirement. Experiments show that Repotrace improves the mean per-round reward (MPR) across different models (e.g., +19.99 on GLM-5.2) and agent harnesses (e.g., +10.22 on Claude Code). Across rounds, Repotrace increases the average per-round reward from 0.163 to 0.371 on Terminus-2 with DeepSeek-V4-Pro, while reducing average regression errors by 56.78%. We release the source code and agents' trajectories to support future research.

Primary area:
generative models
Keywords:
Code Agent, Requirements Tracing, Multi-round Interactions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NrTgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a requirement-code correspondence before each round of code modification. It uses repository dependencies and PageRank to describe the code structure, then asks an LLM to identify the target files and potentially affected prior components from the current and earlier requirements. On EvoCode-Bench, the method increases per-round reward from 44.65% to 54.87% with Claude Code running DeepSeek-V4-Pro.

Critical strengths

The method is being added to an existing coding agent, so this comparison gives some evidence that explicitly tracking requirements can help even when the agent already has tools to explore the repository. The results show that the average reward can increase 10% with Claude Code, which is useful.

Critical weaknesses
  1. The paper discusses related methods such as TraceDev, CodexGraph, and LocAgent, but does not compare with them. I would like to understand what is new in the requirement-code correspondence, beyond combining a repository graph with records of earlier requirements.
  2. The experiments compare the full method with the original agent, but do not test whether prior requirement records or an extra planning step would be enough.
Questions for authors
  1. Have you compared RepoTrace with the simpler baseline that gives the agent the same prior requirement records and asking it to plan before editing? I would like to know how much the repository structure analysis adds to that simpler setup.
  2. Which part of the requirement-code correspondence differs from the tracing and graph-based methods discussed in Section 3? Could you point to a comparison or ablation that shows why this difference matters?
Reviewer_gJjJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the causes of code agents degrading across multiple turns of software modification. Using 2,656 EvoCode-Bench trajectories, the authors find "requirement focus misalignment", and report that in failed rounds after Round 1, 55.16% of agent-modified source files are not within the files modified by the oracle. They propose RepoTrace, a prompt-level harness that runs before each round: (1) it constructs a file-level graph from imports and name-matched call sites, scores files with PageRank and fan-in, and diffs the graph against the previous round; (2) an LLM (with a keyword fallback) extracts target files and symbols, behavior constraints, and affected prior elements from the current requirement and a history of earlier requirement records; (3) the result is appended to the requirement as a “requirement-code correspondence” message. On EvoCode-Bench (26 tasks, 227 rounds), RepoTrace improves the mean per-round reward for DeepSeek-V4-Pro (14.14 to 36.39 on Terminus-2, 44.65 to 54.87 on Claude Code) and GLM-5.2 (16.19 to 36.18), with smaller or negative changes for MiniMax-M3 (+4.91) and Qwen3.7-Max (-0.51). The paper also reports per-round regression accuracy and includes a case study on the JobForge task.

Critical strengths
  1. The problem matters and the method is cheap to adopt. RepoTrace only adds context to the prompt, needs no training, and works with an existing harness. For DeepSeek-V4-Pro the gain holds on two harnesses, including a strong one (Claude Code, 44.65 to 54.87 MPR), while input tokens go down. If the gain is real and can be attributed to the method, this would be useful to practitioners.

  2. The appendix is unusually transparent: it gives the extraction prompt, the verbatim message injected in a real round, and the limits of the motivating statistic (parsing filter, path-level matching, a replay calibration). This let me check what the method actually puts in front of the agent, which is the basis of my main concern below.

Critical weaknesses
  1. The motivation does not establish a cause, and the paper's evidence does not support the claimed mechanism. (a) The off-oracle analysis (Appendix B) is computed only on failed rounds. Without the same ratio for successful rounds, it does not show that edits outside the oracle files are associated with failure, let alone that they cause the observed drift. The 55.16% figure is also hard to reconcile with the paper's own 30-trajectory calibration under the same setting, which gives 32.26% (static) and 34.29% (replay). Other plausible causes of drift, such as the agent losing track of earlier requirements as the session grows, are not examined. (b) In the JobForge case used to explain the gain, the injected message reports "13 source files, 0 dependency edges", so every file has fan-in 0 and fan-out 0, the impact scope is empty, and affected_prior is [] (Fig. 6, Fig. 8). The supplementary reconstruction of the same round shows 83 call edges (Fig. 9b), so the extractor missed dependencies that the code does have, and the structural part contributed nothing here. What the message does contain is the Round 6 requirement and its target files, the constraint "All existing subcommands unchanged", and a generic instruction to keep edits local and run a regression check. (c) Simple alternatives are not compared. Changes across rounds are already recorded by git, so the obvious baseline for the structural change summary and the requirement records is to give the agent the git log or diff of previous rounds together with the earlier requirements. Agent memory systems that store and retrieve interaction history (e.g., MemGPT, Mem0, A-MEM, Zep) address the same need and are neither discussed nor compared. EvoCode-Bench also keeps the same agent session across rounds (Sec. 5.1), so earlier requirements should already be in the agent's history unless they were compacted away, and it is unclear what re-injecting them adds. Without these baselines and an ablation of the message's parts, the gain cannot be attributed to requirement-code correspondence or to repository structure.

  2. The evaluation is narrow, and the metrics do not show where the gain comes from. (a) Only one benchmark is used, and it does not represent long-horizon maintenance of existing software well. Round 1 often initializes the repository (Appendix B), and the JobForge repository has 12 to 14 files in Rounds 6 to 8 (Appendix C). At that scale repository-structure analysis has little to work with. A benchmark built on larger real repositories with multi-step evolution (e.g., SWE-EVO or ChainSWE) would test the claim more directly. (b) MPR and FCR are EvoCode-Bench's own metrics, but they count a round as passed only when all cumulative tests pass. An unrepaired early regression therefore lowers every later round, and failures on the new requirement are mixed with regressions on old ones. The paper separates regression accuracy only for Terminus-2 (Fig. 3b). PTR and HTR move in steps of 3.85 pp (one task), so they carry little information at 26 tasks. (c) There is a single run per configuration, with no variance, confidence intervals, or paired tests. Large gains appear on 2 of 4 models; on MiniMax-M3, FCR drops from 23.08 to 15.38, and Qwen3.7-Max does not improve. The abstract also says regression errors drop by 56.78%, but 56.78% is the relative increase in regression test accuracy (57.06 to 89.46); the error rate itself goes from 42.94 to 10.54.

Questions for authors
  1. (Could change my rating) Please report, on at least the DeepSeek-V4-Pro / Terminus-2 setting: (a) the agent plus earlier requirement records and the "keep previous behavior / run regression checks" instruction, with no graph and no extracted entities; (b) the agent plus the git log or diff of previous rounds and the earlier requirements; (c) full RepoTrace with the structural sections (architecture snapshot, fan-in, impact scope) removed. If (a) or (b) recovers most of the gain, the contribution is different from the one claimed.

  2. (Could change my rating) A comparison with at least one agent memory baseline that stores and retrieves cross-round history.

  3. (Clarification) What is the off-oracle ratio in successful rounds under the same filtering? How were the 30 calibration trajectories in Appendix B selected, and why is their rate about 33% when the corpus estimate is 55.16%?

  4. (Clarification) Since the agent session persists across rounds, are earlier requirements still in the agent's context at round t, or has the harness compacted them? How often does compaction happen in your runs?

  5. (Clarification) How often is the extracted relation set E_t empty or nearly empty across tasks, by language? Why does the extractor find 0 edges in JobForge when the reconstruction finds 83 call edges?

  6. (Clarification) Can you report results on a benchmark with larger existing repositories, new-requirement and old-requirement test pass rates separately for every model and harness, and repeated runs or paired per-task tests for Tables 1 and 2?

  7. (Related work) Aider's repository map and RepoGraph (Ouyang et al., ICLR 2025) also inject graph-ranked repository context into coding agents. How does RepoTrace differ, and could either serve as a baseline?

Reviewer_t55AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies degradation in multi-round coding-agent tasks and identifies “requirement focus misalignment” as a recurring failure pattern. Its motivating analysis covers 1,846 failed, post-round-1 trajectories from EvoCode-Bench and reports that 55.16% of modified source-file instances fall outside human-oracle target files. RepoTrace combines a repository graph, LLM-extracted target elements and constraints, structural diffs, and prior requirements into a text message appended to each round's request. On EvoCode-Bench, it substantially improves mean per-round reward for some model–harness combinations and reduces regression failures, although results are mixed for other models. The intervention is practical, but the current experiments do not isolate which part of the bundled context causes the gains.

Critical strengths
  1. RepoTrace is a simple, harness-agnostic intervention: its output is plain text appended to the requirement. Some reported effects are large, especially the sustained accuracy on earlier-round tests and the gains with DeepSeek-V4-Pro and Claude Code. If these results survive controlled ablations, a lightweight pre-round context step that reduces regression in long-horizon coding would be useful in practice.

  2. The motivating trajectory analysis is more transparent than is typical for this area. The appendix reports parsing coverage, exclusion rules, the limitation that off-oracle edits are not necessarily incorrect, and a small replay-based calibration. This provides a concrete hypothesis about why multi-round agents fail, even though it does not yet establish causation.

Critical weaknesses
  1. The gains cannot currently be attributed to the proposed requirement–code correspondence. The injected message bundles prior requirements, explicit non-regression and testing instructions, LLM-extracted targets, and structural information, but the paper provides no component ablation or context-budget-matched repository-map/localization baseline. In the showcased JobForge case, the repository graph has zero dependency edges, so success there does not demonstrate the structural mechanism. Without these controls, the evidence is also consistent with a longer and more directive prompt being responsible for the improvement.

  2. The empirical support is limited to one 26-task benchmark with one run per configuration, without uncertainty estimates or significance tests. Results are mixed: Qwen3.7-Max's MPR decreases, while MiniMax-M3's FCR decreases and its input tokens nearly double. The motivating off-oracle ratio is measured only on failed trajectories, is not reported after applying RepoTrace, and differs substantially from the 30-trajectory replay calibration. These issues weaken both the causal motivation and the generality of the reported gains.

Questions for authors
  1. [Suggestion] Can the authors ablate requirement history and generic non-regression/testing instructions, LLM-extracted targets, and structural graph features separately, and compare against a repository-map or localization baseline under a matched injected-token budget? A shuffled-target control would also test whether the correspondence itself matters.

  2. [Clarification] Which model performs target extraction, how often does keyword fallback occur, and are all extraction tokens, latency, and cost included in the reported resource totals? Please also clarify whether Table 1 uses Terminus-2.

  3. [Suggestion] Can the authors report multiple runs with paired confidence intervals, measure whether RepoTrace reduces off-oracle edits and improves target-file precision/recall, and provide evidence on a second multi-round coding benchmark? These results would materially affect confidence in robustness and generality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.