acceptodds
Under review as a conference paper at ICLR 2027

ACCEED: Persistent Structured Context for Code Agents

Abstract

Code agents need to identify relevant modules and their interactions before modifying a repository. In large repositories, this can take extensive code exploration, which agents may repeat across tasks unless the knowledge is preserved. Documentation that preserves this knowledge for agents should state which module owns which code and how modules interact, let agents read only the detail a task needs, and stay current as code changes. Whether it meets these requirements depends on how it is organized: a tree gives each file one owner, but module interactions, such as shared dependencies, calls, and data flow, form a graph. We propose Artifact-Centric Code Exploration, Evolution & Drill-down for Hierarchical Knowledge Generation (ACCEED), built on a data structure that separates ownership from interaction, the Agent-oriented Documentation Intermediate Representation (AD-IR). Each AD-IR module owns a source scope, the files it is responsible for, nested in its parent's and disjoint from its siblings', and interactions are kept as typed relations and flows over the resulting tree. ACCEED uses these properties to build the tree recursively with one bounded scope per agent, give agents progressive access through doc-drill, and route code changes through source scopes to the nodes that own them. We also introduce ACCEED-Bench, 240 source-verified questions over six repositories, answered from documentation alone and scored by the weighted share of reference scoring points each answer covers. ACCEED's leaf pages alone outperform documentation from DeepWiki and Google Code Wiki on three shared repositories. AD-IR with doc-drill further improves coverage for every answering model, especially on cross-module and repository-wide questions. After 150 merged pull requests per repository, localized maintenance raises coverage on changed-code questions from \staleCov% with frozen documentation to \incrCov%, while changing about 1% of documentation lines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language models, code agents, repository understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2bZ8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies how to organize and maintain repository-level documentation so that coding agents can reuse codebase knowledge across tasks without repeating expensive code exploration. The authors propose ACCEED, built around an intermediate representation called AD-IR that separates source ownership from module interactions. Based on this structure, ACCEED constructs the documentation tree recursively with bounded agent scopes, provides progressive access via the doc-drill protocol, and routes code changes to candidate leaf pages for localized maintenance. To evaluate the approach, the authors introduce ACCEED-Bench, a 240-question benchmark across six repositories where agents answer questions from documentation alone, and show that AD-IR with doc-drill improves answer coverage over flat pages, external documentation wikis, and iterative BM25 source retrieval, while supporting localized updates across merged pull requests.

Critical strengths

First, the paper tackles a very important and practical problem for coding agents: how to help agents efficiently explore large-scale repositories and retain that understanding across tasks without starting from scratch every time. A structured approach like ACCEED proposed in the paper is intuitively promising for organizing codebase knowledge.

Second, I really appreciate that the authors seriously address the documentation staleness problem and validate their update mechanism on real code evolution. In practice, the biggest bottleneck for persistent context (such as AGENTS.md or DeepWiki) is that codebases change constantly and documentation quickly goes out of date. While most prior work only focuses on one-shot documentation generation, this paper explicitly designs localized maintenance via source scopes and tests it across 150 real merged pull requests per repository, restoring answer coverage on changed code from roughly 25% to 68% while modifying only about 1% of the documentation.

Critical weaknesses

My main concern is that, despite being motivated by repository-level issue resolution in the Intro, there is no experimental evidence showing that ACCEED can help agents address real downstream coding tasks (e.g., DeepSWE v1.1), now only experiments on simple QA tasks and trivial retrieval baselines.

The primary results come from the authors' self-constructed ACCEED-Bench, which only evaluates question answering from documentation. The questions are written by the authors, and GPT-5.6 Sol both generates the AD-IR documentation and acts as the sole judge scoring the answers, which introduces a risk of bias toward their own method. The additional SWE-QA experiment in Appendix is still a QA task and cannot answer whether ACCEED helps with actual code changes.

Moreover, from my perspective, the compared baselines are relatively trivial and not fair enough (e.g., BM25 search in RQ2). The experiments appear to prevent coding agents like Codex from using their native retrieval approach (i.e., directly with grep/find commands and file reads), as the Source Retrieval condition only exposes source code through a BM25 search tool. Even under this restricted baseline, the gain nearly vanishes for the strongest model, GPT-5.6 Sol (only +0.3). It is quite possible that modern agents are intelligent enough to quickly retrieve critical information and solve coding problems without needing such a structured retrieval approach.

Finally, the problem studied by the paper is very important for coding agents. I am happy to raise my score if the authors can address my concerns by adding experiments such as an evaluation on 30–50 tasks randomly sampled from DeepSWE v1.1, where the subset includes multiple tasks from the same repository so that the construction cost can be amortized (DeepSWE v1.1 is less prone to frontier-model overfitting than SWE-bench Verified or Pro and is used in the official GPT-5.6 report). Specifically, using the same Codex harness across two models (e.g., GPT-5.6 Sol and Luna), please compare native Codex (grep/find plus file reads) against Codex with ACCEED over 3 runs per condition, reporting the mean resolution rate and confidence intervals. Please also compare ACCEED's construction plus runtime cost against native Codex's cost, and estimate how many tasks a repository needs before ACCEED becomes cost-effective. Showing either a consistently higher resolution rate on both models, or a comparable resolution rate with a significantly lower marginal cost, would be sufficient.

Questions for authors
  1. In the Source Retrieval condition (RQ2), could the agent access the repository through shell commands (e.g., grep/find and reading files), or only through the BM25 search tool?
  2. Can the authors report the improvement by ACCEED on the downstream issue resolution task such as DeepSWE v1.1?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 2bZ8 (Part 1 of 3)

We sincerely thank the reviewer for recognizing the practical importance of persistent repository knowledge and our localized maintenance design across real pull requests, as well as for proposing a clear experimental protocol to evaluate downstream coding. We ran the DeepSWE v1.1 evaluation and shell-agent baselines as suggested, revised the manuscript accordingly, and provide our point-by-point responses below.

W1

Despite motivating the paper with repository-level issue resolution, the main evaluation relies only on QA rather than real downstream coding tasks such as DeepSWE v1.1, and does not compare construction plus runtime costs against native Codex on issue resolution.

Response to W1

Thank you for this constructive critique and concrete experimental design. We clarify the role of QA in our target workflow and report the new DeepSWE v1.1 coding evaluation below:

1. Why the main evaluation uses documentation-only QA. ACCEED targets cross-repository development. An agent modifying repository BB needs to understand an external dependency repository AA, whose source code may not be available in the current workspace. For example, application code may call a low-level operator library or runtime library. Our deployment in Section 5.1 serves this type of workflow.

Documentation-only QA tests whether prebuilt AD-IR provides the information about boundary behavior and module interactions needed to understand external dependencies. We have clarified this evaluation objective in Sections 1 and 4.2.

No suitable public benchmark currently evaluates cross-repository code editing. We discuss this limitation in Section 6 and add a DeepSWE v1.1 evaluation in Section 4.7 to examine AD-IR's role in actual code editing.

2. Downstream issue resolution on DeepSWE v1.1.

Following your suggested protocol, we included every repository in DeepSWE v1.1 with at least two tasks, covering all 38 eligible tasks across 16 repositories.

Both conditions use Codex CLI 0.166.0 at high reasoning effort with GPT-5.6 Luna and GPT-5.6 Sol. Each task is run independently three times with each model under each condition, yielding 456 trials in total, graded by the official test suites. The two conditions are:

  • Native Codex: source code at the task's base commit, with grep, find, shell commands, and tools for reading, writing, and editing files.
  • Codex + ACCEED: the same source code and tools, with read-only AD-IR available through doc-drill.
ModelConditionMean resolution rateTasks resolved in each of the three runsCost per trial
GPT-5.6 LunaNative Codex43.0%18, 14, 17\$1.03
GPT-5.6 LunaCodex + ACCEED52.6%19, 21, 20\$1.25
GPT-5.6 SolNative Codex61.4%23, 23, 24\$4.24
GPT-5.6 SolCodex + ACCEED65.8%26, 26, 23\$5.32

Task counts are listed in order for runs 1, 2, and 3, each covering all 38 tasks. Compared with Native Codex using the same model, adding ACCEED yields the following results:

  • Luna's mean resolution rate increases by 9.6 percentage points, with a 95% confidence interval of [+1.8, +18.4]. Performance improves on 13 tasks, declines on 5, and is unchanged on 20.

  • Sol's mean resolution rate increases by 4.4 percentage points, with a 95% confidence interval of [−4.4, +14.0]. Performance improves on 7 tasks, declines on 7, and is unchanged on 24.

  • Documentation helps resolve tasks that repeatedly failed, while it can also reduce repeatability on individual tasks. Examining the three runs for each task reveals where the benefit occurs more clearly than the average resolution rate alone. For example, GPT-5.6 Sol improves from 0/3 to 3/3 on kombu-virtual-queue-dead-lettering, and other tasks improve from 0/3 to 2/3 or 1/3. These changes mean that some previously unresolved tasks are solved at least once within the same three attempts. Most regressions are from 3/3 to 2/3 or from 2/3 to 1/3: these tasks remain solvable, but success is less repeatable.

    We interpret this pattern as documentation supplying module relationships and behavioral constraints that fill gaps in the agent's independent exploration, helping it find a viable modification that it previously missed. Additional context can also interfere with an otherwise effective judgment and cause occasional regressions. Documentation can therefore broaden the set of tasks resolved within a limited number of attempts, while its effect on the stability of individual runs must also be considered.

AuthorsOpenReview ↗

Response to Reviewer 2bZ8 (Part 2 of 3)

Response to W1 (continued)

  • Construction cost, runtime cost, and cost-effectiveness: we built 23 AD-IR instances for the different source versions used by tasks across the 16 repositories, at a total cost of \1,070,orapproximately1,070, or approximately \\47 per build.

    For the same model, adding AD-IR increases the cost per trial by 21%–25%, as shown in the table above. This experiment therefore offers no savings in runtime cost within the same model to offset documentation construction costs.

    Considering model choice, adding ACCEED raises Luna's resolution rate from 43.0% to 52.6%, closing more than half the gap to Native Sol's 61.4%. Its cost of \1.25pertrialisapproximately301.25 per trial is approximately 30% of Native Sol's \\4.24. Using Luna + ACCEED in place of Native Sol saves \2.99pertrial,offsettinga2.99 per trial, offsetting a \\47 documentation build in approximately 16 trials.


W2

The main results rely on the self-constructed ACCEED-Bench, where questions are written by the authors and GPT-5.6 Sol both builds AD-IR and acts as the sole judge, risking bias toward the proposed method.

Response to W2

To address the concern that the benchmark and its scoring may favor AD-IR, we added independent judges, manually audited scoring disagreements, and clarified the authoring process and downstream evaluation:

1. Multi-model, cross-family judge cross-check.

To assess the original judgments independently of GPT-5.6 Sol, which both built the documentation and answered questions, we re-scored the same 4,500 answers using GPT-6.1 Sol and Gemini 4 Argon under the same scoring criteria. Gemini 4 Argon belongs to a different model family and participated in neither documentation construction nor answering. We paired the judges' normalized total scores for each answer and calculated Pearson correlations. Each comparison includes all 4,500 answers, including those on which the judges agreed and disagreed.

Judge pairPearson correlation (r)
GPT-5.6 Sol and GPT-6.1 Sol0.84
GPT-5.6 Sol and Gemini 4 Argon0.76
GPT-6.1 Sol and Gemini 4 Argon0.71

Using GPT-6.1 Sol as the reference judge, we also found no additional preference by GPT-5.6 Sol for its own answers. GPT-5.6 Sol awards an average of 6.1 more points than GPT-6.1 Sol on its own answers (n=900n = 900), compared with 10.2, 6.8, 6.3, and 6.0 more points on answers from Haiku 4.5, Sonnet 5, Luna, and Terra, respectively. The gap on its own answers does not exceed the average gap across the other four answering models.

2. Exhaustive human check on all cross-judge disagreements.

We screened all 4,500 answers for disagreements among the three judges and manually checked each disputed judgment against the answer text, scoring points, and supporting source files. We found no clearly unreasonable judgments or systematic errors favoring AD-IR. The observed disagreements mainly concerned how much credit to award for partial explanations, reflecting differences in grading strictness: GPT-6.1 Sol is the strictest, GPT-5.6 Sol is intermediate, and Gemini 4 Argon is the most lenient.

3. Source-only authoring and judge-free test grading. All 180 questions, reference answers, and scoring points were written in isolated source-only sessions without consulting AD-IR. Moreover, the DeepSWE v1.1 evaluation in W1 grades patches strictly with official unit test suites, completely independent of our QA benchmark and any LLM judge.

AuthorsOpenReview ↗

Response to Reviewer 2bZ8 (Part 3 of 3)

W3

The Source Retrieval baseline in RQ2 restricts coding agents to BM25 search instead of native shell exploration (grep, find, and file reads), and even against BM25 the gain on GPT-5.6 Sol is only +0.3.

Response to W3

In the original submission, Source Retrieval exposed source code only through BM25 search to mirror retrieval-based repository baselines. To evaluate agents exploring source code directly, we added two experimental conditions:

  • Source + shell: the raw repository checkout at the benchmark commit and shell tools.
  • Source + AD-IR: the same source code and tools, with complete AD-IR additionally available through doc-drill.

Both conditions use the same CLI harness and allow repository exploration through ls, grep, find, and file reads, with no limit on tool calls.

The experiments cover all six repositories, 180 questions, and five answering models, yielding 900 answers per condition. We paired results by question and computed 95% confidence intervals using bootstrap resampling.

Answering modelSource + shell (%)AD-IR only (%)Source + AD-IR (%)(Source + AD-IR) − Source [95% CI]Cost: SourceCost: Source + AD-IR
Claude Haiku 4.560.366.664.4+4.1 [+1.6, +6.4]\$0.304\$0.228 (−24.9%)
Claude Sonnet 581.176.782.1+0.9 [−1.1, +3.0]\$0.510\$0.506 (−0.8%)
GPT-5.6 Luna83.479.085.1+1.7 [−0.2, +3.4]\$0.286\$0.238 (−16.8%)
GPT-5.6 Sol87.484.189.7+2.3 [+0.7, +3.9]\$1.271\$1.167 (−8.2%)
GPT-5.6 Terra84.777.984.8+0.1 [−1.6, +1.7]\$0.453\$0.443 (−2.2%)
Mean79.476.981.2+1.8 [+0.8, +2.8]\$0.565\$0.516 (−8.6%)

By scope, Source + shell / AD-IR only / Source + AD-IR achieve 83.6 / 78.7 / 85.5 on local questions, 79.3 / 78.2 / 80.7 on cross-module questions, and 75.3 / 73.6 / 77.4 on repository-wide questions.

Even with source code and unrestricted shell tools available, adding AD-IR raises overall coverage from 79.4% to 81.2% while reducing the cost per question from \0.565to0.565 to \\0.516. We believe this gain is related to AD-IR's organization of module relationships and interactions in advance. When exploring source code directly, agents must determine which modules participate in a behavior and then locate the relevant implementations. AD-IR provides these connections, helping agents identify what to inspect before following source anchors to verify the details. This can reduce the risk of overlooking relevant modules and the need to repeatedly locate code.

Haiku's results further illustrate the value of analyzing source code in advance. Its coverage is 60.3% with source code alone and 66.6% with AD-IR alone. AD-IR is built by the stronger GPT-5.6 Sol, which performs part of the module identification, dependency analysis, and explanation of behavior during construction. Haiku can then use these results to compose its answers, reducing the work needed to infer module relationships while answering. This offers an explanation for how models with weaker source analysis capabilities can benefit from the documentation.

AD-IR's value therefore lies in completing part of the repository analysis in advance so that agents can reuse it for subsequent questions. For models that can explore source code directly, the documentation provides search guidance and existing explanations of behavior, while source code supports verification of implementation details. The new experiments show that this help remains useful under unrestricted source exploration, with benefits extending beyond the original BM25 retrieval baseline.


Q1

In the Source Retrieval condition (RQ2), could the agent access the repository through shell commands (e.g., grep/find and reading files), or only through the BM25 search tool?

Response to Q1

In the original Source Retrieval condition, the agent could access source code only through the BM25 search tool. We have now added Source + shell and Source + AD-IR with unrestricted shell commands (ls, grep, find, and file reads) across all six repositories, 180 questions, and five models; please see the response to W3 for the full results.


Q2

Can the authors report the improvement by ACCEED on the downstream issue resolution task such as DeepSWE v1.1?

Response to Q2

We evaluated all 38 tasks across 16 repositories in DeepSWE v1.1, with three independent runs per condition. Adding ACCEED yields the following results:

  • GPT-5.6 Luna improves from 43.0% to 52.6% in mean resolution rate, a gain of 9.6 percentage points.
  • GPT-5.6 Sol improves from 61.4% to 65.8% in mean resolution rate, a gain of 4.4 percentage points.

See the response to W1 for the full experimental setup, confidence intervals, results from each run, and cost analysis.

Reviewer_84A7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes pre-generating structured documentation for a repository so that a coding agent can read it instead of exploring source. The representation is a tree of module nodes, each owning a set of source files nested inside its parent's and disjoint from its siblings, with typed links between sibling modules, cross-cutting flows that describe scenarios spanning branches, and detailed leaf pages anchored in source. A system builds this recursively with one bounded agent session per node, serves it through a protocol that lets the answering agent expand the tree and follow links on demand, and updates it after code changes by mapping the diff to the leaves whose files changed and revising upward. Evaluation uses 180 author-written questions over six repositories plus 60 questions about behaviour changed by 150 merged pull requests on two of them, scored as weighted coverage of reference points by an LLM judge. Agents answer better with the full structure than with the same leaf pages served flat (77 versus 70 averaged over five models), the generated pages beat exported DeepWiki and Google Code Wiki pages by a wide margin, the structure beats BM25 retrieval over source by six points overall but by nothing for the strongest model, and after 150 PRs the maintained documentation scores 68 versus 25 for a frozen copy, with under 1% of lines changed.

Critical strengths

The representation is well specified and its key property is actually used. Because scopes are nested and sibling-disjoint, each file has a unique deepest owner, and the nodes affected by a diff are exactly the ancestor paths of those owners. The update algorithm relies on this, and the statistics bear it out: under 1% of documentation lines change after 150 PRs.

Critical weaknesses
  • None of the headline comparisons is fair, so the central claim that persistent structured documentation beats on-demand exploration is not established. In the structure-versus-flat comparison the structured condition makes 20 tool calls per question against 12 for flat, so the 7-point gain comes with 69% more reads and no budget-matched control. In the comparison against source, the baseline can only search with BM25, the natural competitor, the same agent with a checkout and ls, grep and cat, which is how Codex and Claude Code actually explore a repository, is absent even though the paper says the documentation conditions already have shell tools, and against this weak baseline the strongest model gains 0.3 points with an interval spanning zero. The DeepWiki comparison does not report corpus sizes; the generated documentation runs to 16 to 23 megabytes per repository and cost over a hundred million tokens for one of them, so the 71-versus-41 result may be a size effect. The SWE-QA experiment pits a multi-turn agent against a single-turn retrieval baseline. I would want a shell-tool source baseline on all 180 questions, a budget-matched flat condition, corpus sizes with a size-matched subset, and a multi-turn retrieval arm.
  • The benchmark provenance and the update experiment are partly circular, and the judge is the builder. The authors wrote the 180 questions, and the repository-wide ones "cover every subsystem involved in a scenario, with scoring points assigned per module", which is the shape of one of the paper's flows: five of the six documentation sets were built before the benchmark was finalised, and the paper never says whether the question writers had the documentation in front of them. The 60 update questions were proposed by a model given the diffs and ask about the changed code's current behaviour, so documentation frozen before those diffs cannot answer them by construction, and 25 versus 68 shows that an update happened rather than that this update policy is good; there is no from-scratch rebuild and no trivial baseline such as appending the PR descriptions. GPT-5.6 Sol builds the documentation, is one of the five answerers, and is the only judge, with no agreement reported against a second judge.
Questions for authors
  • Were the 180 questions and their scoring points written before or after the documentation was built, and did the writers consult the pages, flows or module names? (Clarification)
  • Please run the source baseline with the same shell tools used elsewhere instead of BM25 only, for all five models. The title's claim depends on this comparison. (Suggestion)
  • How much of the 7-point structure-versus-flat gain survives when the structured condition is capped at the flat condition's tool-call budget? (Clarification)
  • What are the sizes of the DeepWiki and Google Code Wiki exports, and what does the generated documentation score when truncated to match? (Clarification)
  • What is the agreement between the current judge and one from a different model family, and does the ordering in Table 1 change? (Clarification)
7 comments
AuthorsOpenReview ↗

Response to Reviewer 84A7 (Part 1 of 4)

We sincerely thank the reviewer for recognizing the soundness of our representation and update mechanism, and for the detailed, constructive experimental suggestions. We have revised the manuscript accordingly and provide our point-by-point responses below.

W1

Fairness of the four main comparisons: (1) missing a shell-equipped source-agent baseline over a repository checkout across all five models, (2) 20 vs. 12 tool calls in Structured vs. Flat without tool-call budget analysis, (3) missing exported corpus sizes and size-effect controls in the DeepWiki / Google Code Wiki comparison, and (4) comparing a multi-turn agent against a single-turn retrieval baseline on SWE-QA.

Response to W1

Thank you for raising these four points. We address each comparison directly below:

1. Shell-equipped source-agent baseline across all 180 questions and all five models.

We added two experimental conditions:

  • Source + shell: the raw repository checkout at the benchmark commit and shell tools.
  • Source + AD-IR: the same source code and tools, with complete AD-IR additionally available through doc-drill.

Both conditions use the same CLI harness and allow repository exploration through ls, grep, find, and file reads, with no limit on tool calls.

The experiments cover all six repositories, 180 questions, and five answering models, yielding 900 answers per condition. We paired results by question and computed 95% confidence intervals using bootstrap resampling.

ConditionOverall (%)Local (%)Cross-module (%)Repo-wide (%)Cost / question
Source + shell79.483.679.375.3\$0.565
AD-IR only76.978.778.273.6\$0.527
Source + AD-IR81.285.580.777.4\$0.516

Per-model coverage and per-question cost (GPT-5.6 Sol judge):

ModelSource + shell (%)AD-IR only (%)Source + AD-IR (%)AD-IR − Source(Source + AD-IR) − Source [95% CI]Cost: SourceCost: Source + AD-IR
Claude Haiku 4.560.366.664.4+6.3+4.1 [+1.6, +6.4]\$0.304\$0.228 (−24.9%)
Claude Sonnet 581.176.782.1−4.4+0.9 [−1.1, +3.0]\$0.510\$0.506 (−0.8%)
GPT-5.6 Luna83.479.085.1−4.4+1.7 [−0.2, +3.4]\$0.286\$0.238 (−16.8%)
GPT-5.6 Sol87.484.189.7−3.3+2.3 [+0.7, +3.9]\$1.271\$1.167 (−8.2%)
GPT-5.6 Terra84.777.984.8−6.8+0.1 [−1.6, +1.7]\$0.453\$0.443 (−2.2%)
Mean79.476.981.2−2.5+1.8 [+0.8, +2.8]\$0.565\$0.516 (−8.6%)

Even with source code and unrestricted shell tools available, adding AD-IR raises overall coverage from 79.4% to 81.2% while reducing the cost per question from \0.565to0.565 to \\0.516. We believe this gain is related to AD-IR's organization of module relationships and interactions in advance. When exploring source code directly, agents must determine which modules participate in a behavior and then locate the relevant implementations. AD-IR provides these connections, helping agents identify what to inspect before following source anchors to verify the details. This can reduce the risk of overlooking relevant modules and the need to repeatedly locate code.

Haiku's results further illustrate the value of analyzing source code in advance. Its coverage is 60.3% with source code alone and 66.6% with AD-IR alone. AD-IR is built by the stronger GPT-5.6 Sol, which performs part of the module identification, dependency analysis, and explanation of behavior during construction. Haiku can then use these results to compose its answers, reducing the work needed to infer module relationships while answering. This offers an explanation for how models with weaker source analysis capabilities can benefit from the documentation.

AD-IR's value therefore lies in completing part of the repository analysis in advance so that agents can reuse it for subsequent questions. For models that can explore source code directly, the documentation provides search guidance and existing explanations of behavior, while source code supports verification of implementation details. The new experiments show that this help remains useful under unrestricted source exploration, with benefits extending beyond the original BM25 retrieval baseline.

AuthorsOpenReview ↗

Response to Reviewer 84A7 (Part 2 of 4)

Response to W1 (continued)

2. Tool-call count and budget analysis in Structured vs. Flat.

Neither condition limits the number of tool calls, and both use the same reasoning effort. Under Structured, doc-drill separates tree navigation, reading node descriptions, and inspecting typed relations into individual calls so that agents can retrieve content as needed.

In Table 1, Structured averages 20.1 tool calls per question, compared with 11.9 for Flat. Coverage increases by 6.9 percentage points, with a 95% confidence interval of [+5.9, +8.2]. To further examine the relationship between call count and coverage gains, we made the following comparisons:

  • Grouping by call count. Across all 900 question–model pairs, Structured makes no more calls than Flat in 229 pairs. Coverage still increases by 9.36 percentage points in this group, with a 95% confidence interval of [+7.36, +11.46]. The remaining 671 pairs show a gain of 6.06 percentage points, with a 95% confidence interval of [+4.86, +7.16].
  • Comparing ablation results. After removing edges and flows, Hierarchy-only averages 25.5 tool calls per question, exceeding the 20.1 calls made by full Structured, yet achieves lower coverage. Additional calls did not compensate for the loss of cross-module relations.
  • Comparing question scopes. In Table 1, the gain is 3.3 percentage points on local questions, 8.0 on cross-module questions, and 9.5 on repository-wide questions. Larger gains occur on questions that require understanding cross-module relationships.

3. Exported corpus sizes and per-call budget controls against DeepWiki and Google Code Wiki. The exported Markdown corpus sizes across the five shared repositories are:

RepositoryAD-IR (leaf text)DeepWikiGoogle Code Wiki
Codex15.6 MB1.46 MB0.26 MB
vLLM22.7 MB1.40 MB0.22 MB
Ollama5.3 MB0.85 MB0.28 MB
KataGo5.8 MB0.61 MB—
qBittorrent3.6 MB0.62 MB—

We did not truncate the corpora to equal sizes because truncation would change which modules the documentation covers. All systems use the same access interfaces; BM25 returns at most 10 chunks per call, each no longer than approximately 2,400 characters.

MetricACCEEDDeepWikiGoogle Code Wiki
GPT-5.6 Sol score with BM2581.256.052.9
Mean tool calls per questionAbout 24About 36—
Mean cost per questionAbout \$0.33About \$0.45—

Tool calls and costs are averaged across the five shared repositories and the Flat and RAG access modes.

ACCEED scores higher while requiring fewer tool calls and lower answering cost. We believe that documenting module behavior and interactions in advance helps reduce the search and information assembly needed during answering. When re-scored by Gemini 4 Argon, ACCEED still ranks first in all 20 experimental combinations.

4. SWE-QA multi-turn vs. single-turn baseline.

The Function-chunk RAG baseline follows SWE-QA's published protocol: retrieve the top 10 function or class chunks once, then answer in a single turn. Our agent can explore over multiple turns, so this experiment compares both documentation content and answering procedures and cannot isolate the effect of documentation content. We have clarified this scope in Appendix E of the revised paper.

To further compare performance under multi-turn exploration, we added the unrestricted Source + shell baseline and the Source + AD-IR condition across all 180 questions and five models. The results are reported in Point 1 above.

AuthorsOpenReview ↗

Response to Reviewer 84A7 (Part 3 of 4)

W2

Circularity concerns in benchmark provenance, the 60-question update experiment (missing a trivial PR-description baseline and a from-scratch rebuild reference), and using GPT-5.6 Sol as builder, answerer, and sole judge without cross-family or human validation.

Response to W2

We address all three concerns below with direct experimental controls and full cross-judge plus human validation:

1. Provenance and timeline of the 180 questions. Early AD-IR builds existed for five of the six repositories during system development before the 6-repository benchmark suite was finalised. However, every question, reference answer, and source-verified scoring point was authored in an isolated session with access only to the pinned source checkout, without viewing AD-IR pages, interaction flows, or module names. Repository-wide questions assign scoring points across the subsystems that participate in an end-to-end code path in source so that omitting a subsystem loses credit.

2. Trivial PR-description baseline and from-scratch rebuild for the 60 update questions. We agree that changed-code questions cannot be answered by frozen pre-change documentation by construction, and that evaluating the update policy requires both a trivial diff/PR baseline and a full rebuild reference. We ran both controls on the exact same 60 questions with the same answering model (Claude Sonnet 5):

Documentation stateCoverage (%)
Frozen at R(0)R^{(0)} (paper)24.7
Frozen at R(0)R^{(0)} + all 150 merged PR descriptions appended verbatim (new)46.7
Incrementally maintained across 150 PRs ($\approx$1% of documentation lines edited)67.5
Rebuilt from scratch at R(15)R^{(15)} (new)71.3
  • Updating existing documentation reduces the work needed to reconstruct current behavior during answering. PR descriptions record what changed and why. Even after finding the relevant PRs, the agent must combine them with the old documentation and subsequent changes to determine which statements still apply and what the current behavior is. Incremental maintenance incorporates changes into the affected modules' behavior descriptions, revises outdated content, and updates parent descriptions where needed, so later questions can query the current state directly. We view this integration as the main reason it outperforms appending PR descriptions verbatim. The appended-PR condition cites specific PRs on 42 of 60 questions yet still trails substantially, showing that finding change records alone is insufficient to answer questions about current code behavior.
  • Localized maintenance reuses repository understanding that remains valid. A full rebuild processes the entire repository again; localized maintenance retains unaffected documentation and concentrates edits on the scope affected by code changes. On these update questions, editing about 1% of documentation lines achieves 67.5 coverage, close to the rebuilt corpus's 71.3. This indicates that much of the existing documentation remains reusable and that integrating changes into the affected parts can recover much of the knowledge needed for answering.

3. Cross-family judge cross-check and full human validation on disagreements.

To assess the original judgments independently of GPT-5.6 Sol, which both built the documentation and answered questions, we re-scored the same 4,500 answers using GPT-6.1 Sol and Gemini 4 Argon under the same scoring criteria. Gemini 4 Argon belongs to a different model family and participated in neither documentation construction nor answering. We paired the judges' normalized total scores for each answer and calculated Pearson correlations. Each comparison includes all 4,500 answers, including those on which the judges agreed and disagreed.

Judge pairPearson correlation (r)
GPT-5.6 Sol and GPT-6.1 Sol0.84
GPT-5.6 Sol and Gemini 4 Argon0.76
GPT-6.1 Sol and Gemini 4 Argon0.71

Using GPT-6.1 Sol as the reference judge, we also found no additional preference by GPT-5.6 Sol for its own answers. GPT-5.6 Sol awards an average of 6.1 more points than GPT-6.1 Sol on its own answers (n=900n = 900), compared with 10.2, 6.8, 6.3, and 6.0 more points on answers from Haiku 4.5, Sonnet 5, Luna, and Terra, respectively. The gap on its own answers does not exceed the average gap across the other four answering models.

We screened all 4,500 answers for disagreements among the three judges and manually checked each disputed judgment against the answer text, scoring points, and supporting source files. We found no clearly unreasonable judgments or systematic errors favoring AD-IR. The observed disagreements mainly concerned how much credit to award for partial explanations, reflecting differences in grading strictness: GPT-6.1 Sol is the strictest, GPT-5.6 Sol is intermediate, and Gemini 4 Argon is the most lenient.

Reviewer_89H3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ACCEED, a pipeline that builds agent-facing documentation for a code repository once and keeps it in sync as the code changes. The documentation format, AD-IR, is a tree in which each node owns a set of source files, nested within its parent's and disjoint from its siblings', with cross-module interactions stored separately as typed relations (Sections 3.2 and 3.3). The authors introduce ACCEED-Bench, 240 questions over six repositories answered from documentation alone and scored by an LLM judge. They report that structured documentation beats the same pages in a flat folder, that AD-IR pages carry more knowledge than DeepWiki and Google Code Wiki, that answering from AD-IR beats source search at 18.6% lower per-question cost and that maintained documentation still answers questions about changed code after 150 merged pull requests per repository.

Critical strengths
  1. The core idea is tight: one ownership invariant (nested, sibling-disjoint source scopes) drives construction, reading and updating of the documentation. It is delivered as a working system with a benchmark and a maintenance experiment, not as a proposal.
  2. Table 1 is a proper controlled ablation: the same documentation pages with structure on versus off, paired confidence intervals, and a gain for every one of the five answering models.
Critical weaknesses
  1. The evaluation is closed loop. The authors wrote every benchmark question ("We wrote 30 questions per repository"), the question categories mirror the paper's own module tree, and the judge is the same model that built the documentation (GPT-5.6 Sol) and is also one of the scored answerers. There is no human validation of any part of this instrument, and every result in Sections 4.3 to 4.5 rests on it.

  2. The paper never compares against the thing it claims to replace: an actual tool-using code agent exploring the repository. The "source" baseline in Table 2 is BM25 retrieval only.

Questions for authors

Clarification requests:

  1. Were the benchmark questions written before or after AD-IR existed for each repository, and with access to it? Was the LLM judge checked against human scoring on any sample?
  2. Why is there no tool-using code agent baseline over the raw repository?
  3. Break-even: how many questions per repository are needed before the construction cost is recovered by the per-question saving?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 89H3 (Part 1 of 3)

We sincerely thank the reviewer for recognizing the value of our work and for the thoughtful and constructive feedback. We have carefully revised the manuscript accordingly and provide our point-by-point responses below.

W1

The evaluation is closed-loop: the benchmark questions are written by the authors, the same model (GPT-5.6 Sol) both builds the documentation and serves as the sole judge while also being one of the scored answering models, and there is no human validation of the evaluation pipeline.

Response to W1

Thank you for raising this concern. To validate the evaluation pipeline independently of GPT-5.6 Sol and add thorough human verification, we conducted a three-step check across all six repositories, 180 questions, and five answering models (Claude Haiku 4.5, Claude Sonnet 5, and GPT-5.6 Luna / Sol / Terra):

1. Multi-model, cross-family judge cross-check.

To assess the original judgments independently of GPT-5.6 Sol, which both built the documentation and answered questions, we re-scored the same 4,500 answers using GPT-6.1 Sol and Gemini 4 Argon under the same scoring criteria. Gemini 4 Argon belongs to a different model family and participated in neither documentation construction nor answering. We paired the judges' normalized total scores for each answer and calculated Pearson correlations. Each comparison includes all 4,500 answers, including those on which the judges agreed and disagreed.

Judge pairPearson correlation (r)
GPT-5.6 Sol and GPT-6.1 Sol0.84
GPT-5.6 Sol and Gemini 4 Argon0.76
GPT-6.1 Sol and Gemini 4 Argon0.71

Using GPT-6.1 Sol as the reference judge, we also found no additional preference by GPT-5.6 Sol for its own answers. GPT-5.6 Sol awards an average of 6.1 more points than GPT-6.1 Sol on its own answers (n=900n = 900), compared with 10.2, 6.8, 6.3, and 6.0 more points on answers from Haiku 4.5, Sonnet 5, Luna, and Terra, respectively. The gap on its own answers does not exceed the average gap across the other four answering models.

2. Exhaustive human check on all cross-judge disagreements.

We screened all 4,500 answers for disagreements among the three judges and manually checked each disputed judgment against the answer text, scoring points, and supporting source files. We found no clearly unreasonable judgments or systematic errors favoring AD-IR. The observed disagreements mainly concerned how much credit to award for partial explanations, reflecting differences in grading strictness: GPT-6.1 Sol is the strictest, GPT-5.6 Sol is intermediate, and Gemini 4 Argon is the most lenient.

3. Source-only question authoring and independent downstream evaluation.

Early AD-IR documentation existed for five of the six repositories before the benchmark was finalised. All 180 questions, reference answers, and scoring points were written in isolated sessions with access only to the pinned source checkout, without consulting any AD-IR pages, interaction flows, or module names. The questions and scoring criteria were therefore based on source code without using the generated documentation.

To further examine whether AD-IR's benefits depend on our own question set and LLM judges, we added a code repair evaluation on the external benchmark DeepSWE v1.1. The evaluation covers 38 tasks across 16 repositories. Both conditions use the same source code and tools, differing in whether AD-IR documentation is available. Each model was run independently three times on each task under each condition, with success determined by the official test suites.

Answering modelNative Codex resolution rateResolution rate with AD-IRGain in percentage points95% confidence interval for the gain
GPT-5.6 Luna43.0%52.6%+9.6[+1.8, +18.4]
GPT-5.6 Sol61.4%65.8%+4.4[−4.4, +14.0]

Adding AD-IR increases the mean resolution rate for both models. AD-IR's benefits therefore also appear in an independent code repair evaluation, providing evidence for its effectiveness beyond our own QA benchmark.

AuthorsOpenReview ↗

Response to Reviewer 89H3 (Part 2 of 3)

W2

The paper does not compare against an actual tool-using coding agent exploring the raw repository checkout, using only BM25 retrieval as the source baseline in Table 2.

Response to W2

We agree that a tool-using agent over the raw checkout is the natural baseline. We therefore added two experimental conditions:

  • Source + shell tools: the raw repository checkout at the benchmark commit and shell tools.
  • Source + AD-IR: the same source code and tools, with complete AD-IR additionally available through doc-drill.

Both conditions use the same CLI harness and allow repository exploration through ls, grep, find, and file reads, with no limit on tool calls.

The experiments cover all six repositories, 180 questions, and five answering models, yielding 900 answers per condition. We paired results by question and computed 95% confidence intervals using bootstrap resampling.

Overall and per-scope results (GPT-5.6 Sol judge):

ConditionOverall (%)Local (%)Cross-module (%)Repo-wide (%)Cost / question
Source + shell tools79.483.679.375.3$0.565
AD-IR only76.978.778.273.6$0.527
Source + AD-IR81.285.580.777.4$0.516

Per-model breakdown across all 180 questions:

Answering modelSource + shell (%)AD-IR only (%)Source + AD-IR (%)(Source + AD-IR) − Source [95% CI]Cost: SourceCost: Source + AD-IR
Claude Haiku 4.560.366.664.4+4.1 [+1.6, +6.4]$0.304$0.228 (−24.9%)
Claude Sonnet 581.176.782.1+0.9 [−1.1, +3.0]$0.510$0.506 (−0.8%)
GPT-5.6 Luna83.479.085.1+1.7 [−0.2, +3.4]$0.286$0.238 (−16.8%)
GPT-5.6 Sol87.484.189.7+2.3 [+0.7, +3.9]$1.271$1.167 (−8.2%)
GPT-5.6 Terra84.777.984.8+0.1 [−1.6, +1.7]$0.453$0.443 (−2.2%)
Mean79.476.981.2+1.8 [+0.8, +2.8]$0.565$0.516 (−8.6%)

Even with source code and unrestricted shell tools available, adding AD-IR raises overall coverage from 79.4% to 81.2% while reducing the cost per question from \0.565to0.565 to \\0.516. We believe this gain is related to AD-IR's organization of module relationships and interactions in advance. When exploring source code directly, agents must determine which modules participate in a behavior and then locate the relevant implementations. AD-IR provides these connections, helping agents identify what to inspect before following source anchors to verify the details. This can reduce the risk of overlooking relevant modules and the need to repeatedly locate code.

Haiku's results further illustrate the value of analyzing source code in advance. Its coverage is 60.3% with source code alone and 66.6% with AD-IR alone. AD-IR is built by the stronger GPT-5.6 Sol, which performs part of the module identification, dependency analysis, and explanation of behavior during construction. Haiku can then use these results to compose its answers, reducing the work needed to infer module relationships while answering. This offers an explanation for how models with weaker source analysis capabilities can benefit from the documentation.

AD-IR's value therefore lies in completing part of the repository analysis in advance so that agents can reuse it for subsequent questions. For models that can explore source code directly, the documentation provides search guidance and existing explanations of behavior, while source code supports verification of implementation details. The new experiments show that this help remains useful under unrestricted source exploration, with benefits extending beyond the original BM25 retrieval baseline.

AuthorsOpenReview ↗

Response to Reviewer 89H3 (Part 3 of 3)

Q1

Were the benchmark questions written before or after AD-IR existed for each repository, and with access to it? Was the LLM judge checked against human scoring on any sample?

Response to Q1

Early AD-IR builds existed for five repositories before the benchmark was finalised, but all 180 questions, reference answers, and scoring points were written in isolated source-only sessions without access to AD-IR. We re-scored the same 4,500 answers with two additional judges and manually checked each disagreement against the answer text, scoring points, and supporting source files, finding no clearly unreasonable judgments. Please see the response to W1 for the authoring process, cross-judge score correlations, and human audit results.


Q2

Why is there no tool-using code agent baseline over the raw repository?

Response to Q2

We have added the unrestricted shell-equipped code-agent baseline (Source + shell tools) and the combined condition (Source + AD-IR) across all six repositories, 180 questions, and five models. Please see the response to W2 for the complete overall, per-scope, and per-model results.


Q3

Break-even: how many questions per repository are needed before the construction cost is recovered by the per-question saving?

Response to Q3

Based on the average per-question savings across the five answering models, recovering the initial documentation construction cost of \10–10–\\200 per repository requires approximately 200–4,000 questions.

Across all 180 questions, adding AD-IR reduces the average cost per question from \0.565to0.565 to \\0.516, saving \0.049perquestion.Thetablebelowusestheaverageconstructioncostof0.049 per question. The table below uses the average construction cost of \\47 from the DeepSWE experiment to illustrate how many questions are needed to break even. That experiment involved 23 documentation builds for the different source versions used by tasks across 16 repositories, at a total cost of \$1,070.

Answering modelCost per question: source onlyCost per question: source + AD-IRSaving per questionQuestions to recover $47
Average across five models$0.565$0.516$0.049About 960
GPT-5.6 Sol$1.271$1.167$0.104About 450

Using AD-IR to support a lighter model in place of a more expensive model can shorten the time to break even. For example, GPT-5.6 Luna with source code and AD-IR achieves 85.1% coverage at $0.238 per question, while GPT-5.6 Sol with source code alone achieves 87.4% coverage at \1.271perquestion.Theformersavesapproximately1.271 per question. The former saves approximately \\1.03 per question, recovering the $47 construction cost in fewer than 50 questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 89H3,

If you have a moment before the discussion closes, we’d appreciate your thoughts on the scoring checks in Part 1 of our response. We rescored the answers with two additional judges and manually reviewed the cases where the judges disagreed. Please let us know if you have any questions so we can follow up promptly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Responses and Revisions

Dear Reviewers,

Thank you for your feedback, suggestions, and recognition of our work. We summarize our main responses and revisions below and have uploaded a revised manuscript.

  1. Source-agent baselines and AD-IR's role. We added an unrestricted shell baseline and a source-plus-AD-IR condition across all six repositories, 180 questions, and five models. Adding AD-IR improves average coverage and reduces answering cost even when agents can directly inspect source code. We further explain how organizing module relationships and behavior in advance complements source exploration. (Reviewer 89H3 W2, Q2; Reviewer 84A7 W1, Q2; Reviewer 2bZ8 W3, Q1)

  2. Judge validation and question construction. We re-scored all 4,500 answers with two additional judges, including the cross-family Gemini 4 Argon. Structured outperforms Flat for all five answering models under all three judges. We also manually audited scoring disagreements, checked for self-preference, and clarified that all 180 benchmark questions and scoring criteria were written in isolated sessions with access only to source code. (Reviewer 89H3 W1, Q1; Reviewer 84A7 W2, Q1, Q5; Reviewer 2bZ8 W2)

  3. Downstream code repair. We added a DeepSWE v1.1 evaluation comparing Native Codex and Codex + ACCEED under the same harness, covering 38 tasks across 16 repositories. Mean resolution rates improve for both models. (Reviewer 2bZ8 W1, Q2; Reviewer 89H3 W1)

  4. Comparison settings and evaluation protocols. We added an analysis grouped by observed tool-call counts and reported external Wiki corpus sizes and results under the same per-call retrieval budget. We also clarified the protocol differences between the single-turn retrieval baseline and our multi-turn agent on SWE-QA, and stated what each comparison evaluates. (Reviewer 84A7 W1, Q3, Q4)

  5. Controls for documentation updates. We added two controls: appending PR descriptions verbatim and rebuilding documentation from scratch. Localized maintenance achieves higher coverage than appending PR descriptions and approaches a full rebuild while editing about 1% of documentation lines. These results support the value of updating existing documentation and reusing prior analysis. (Reviewer 84A7 W2)

  6. Cost analysis and scope. We expanded the analysis of documentation construction, answering and runtime costs, and cost recovery, distinguishing changes within the same model from comparisons across models. We also clarified the purpose and scope of documentation-only QA, added cost-performance plots, and simplified the related tables and text. (Reviewer 89H3 Q3; Reviewer 2bZ8 W1)

Please review our responses and the revised manuscript and let us know if any concerns remain. If your main concerns have been adequately addressed, we would appreciate it if you could reconsider your rating.

Thank you again for your time and consideration.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.