- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how to organize and maintain repository-level documentation so that coding agents can reuse codebase knowledge across tasks without repeating expensive code exploration. The authors propose ACCEED, built around an intermediate representation called AD-IR that separates source ownership from module interactions. Based on this structure, ACCEED constructs the documentation tree recursively with bounded agent scopes, provides progressive access via the doc-drill protocol, and routes code changes to candidate leaf pages for localized maintenance. To evaluate the approach, the authors introduce ACCEED-Bench, a 240-question benchmark across six repositories where agents answer questions from documentation alone, and show that AD-IR with doc-drill improves answer coverage over flat pages, external documentation wikis, and iterative BM25 source retrieval, while supporting localized updates across merged pull requests.
Critical strengths
First, the paper tackles a very important and practical problem for coding agents: how to help agents efficiently explore large-scale repositories and retain that understanding across tasks without starting from scratch every time. A structured approach like ACCEED proposed in the paper is intuitively promising for organizing codebase knowledge.
Second, I really appreciate that the authors seriously address the documentation staleness problem and validate their update mechanism on real code evolution. In practice, the biggest bottleneck for persistent context (such as AGENTS.md or DeepWiki) is that codebases change constantly and documentation quickly goes out of date. While most prior work only focuses on one-shot documentation generation, this paper explicitly designs localized maintenance via source scopes and tests it across 150 real merged pull requests per repository, restoring answer coverage on changed code from roughly 25% to 68% while modifying only about 1% of the documentation.
Critical weaknesses
My main concern is that, despite being motivated by repository-level issue resolution in the Intro, there is no experimental evidence showing that ACCEED can help agents address real downstream coding tasks (e.g., DeepSWE v1.1), now only experiments on simple QA tasks and trivial retrieval baselines.
The primary results come from the authors' self-constructed ACCEED-Bench, which only evaluates question answering from documentation. The questions are written by the authors, and GPT-5.6 Sol both generates the AD-IR documentation and acts as the sole judge scoring the answers, which introduces a risk of bias toward their own method. The additional SWE-QA experiment in Appendix is still a QA task and cannot answer whether ACCEED helps with actual code changes.
Moreover, from my perspective, the compared baselines are relatively trivial and not fair enough (e.g., BM25 search in RQ2). The experiments appear to prevent coding agents like Codex from using their native retrieval approach (i.e., directly with grep/find commands and file reads), as the Source Retrieval condition only exposes source code through a BM25 search tool. Even under this restricted baseline, the gain nearly vanishes for the strongest model, GPT-5.6 Sol (only +0.3). It is quite possible that modern agents are intelligent enough to quickly retrieve critical information and solve coding problems without needing such a structured retrieval approach.
Finally, the problem studied by the paper is very important for coding agents. I am happy to raise my score if the authors can address my concerns by adding experiments such as an evaluation on 30–50 tasks randomly sampled from DeepSWE v1.1, where the subset includes multiple tasks from the same repository so that the construction cost can be amortized (DeepSWE v1.1 is less prone to frontier-model overfitting than SWE-bench Verified or Pro and is used in the official GPT-5.6 report). Specifically, using the same Codex harness across two models (e.g., GPT-5.6 Sol and Luna), please compare native Codex (grep/find plus file reads) against Codex with ACCEED over 3 runs per condition, reporting the mean resolution rate and confidence intervals. Please also compare ACCEED's construction plus runtime cost against native Codex's cost, and estimate how many tasks a repository needs before ACCEED becomes cost-effective. Showing either a consistently higher resolution rate on both models, or a comparable resolution rate with a significantly lower marginal cost, would be sufficient.
Questions for authors
- In the Source Retrieval condition (RQ2), could the agent access the repository through shell commands (e.g., grep/find and reading files), or only through the BM25 search tool?
- Can the authors report the improvement by ACCEED on the downstream issue resolution task such as DeepSWE v1.1?
9 comments
Response to Reviewer 2bZ8 (Part 2 of 3)
Response to W1 (continued)
-
Construction cost, runtime cost, and cost-effectiveness: we built 23 AD-IR instances for the different source versions used by tasks across the 16 repositories, at a total cost of \47 per build.
For the same model, adding AD-IR increases the cost per trial by 21%–25%, as shown in the table above. This experiment therefore offers no savings in runtime cost within the same model to offset documentation construction costs.
Considering model choice, adding ACCEED raises Luna's resolution rate from 43.0% to 52.6%, closing more than half the gap to Native Sol's 61.4%. Its cost of \4.24. Using Luna + ACCEED in place of Native Sol saves \47 documentation build in approximately 16 trials.
W2
The main results rely on the self-constructed ACCEED-Bench, where questions are written by the authors and GPT-5.6 Sol both builds AD-IR and acts as the sole judge, risking bias toward the proposed method.
Response to W2
To address the concern that the benchmark and its scoring may favor AD-IR, we added independent judges, manually audited scoring disagreements, and clarified the authoring process and downstream evaluation:
1. Multi-model, cross-family judge cross-check.
To assess the original judgments independently of GPT-5.6 Sol, which both built the documentation and answered questions, we re-scored the same 4,500 answers using GPT-6.1 Sol and Gemini 4 Argon under the same scoring criteria. Gemini 4 Argon belongs to a different model family and participated in neither documentation construction nor answering. We paired the judges' normalized total scores for each answer and calculated Pearson correlations. Each comparison includes all 4,500 answers, including those on which the judges agreed and disagreed.
| Judge pair | Pearson correlation (r) |
|---|---|
| GPT-5.6 Sol and GPT-6.1 Sol | 0.84 |
| GPT-5.6 Sol and Gemini 4 Argon | 0.76 |
| GPT-6.1 Sol and Gemini 4 Argon | 0.71 |
Using GPT-6.1 Sol as the reference judge, we also found no additional preference by GPT-5.6 Sol for its own answers. GPT-5.6 Sol awards an average of 6.1 more points than GPT-6.1 Sol on its own answers (), compared with 10.2, 6.8, 6.3, and 6.0 more points on answers from Haiku 4.5, Sonnet 5, Luna, and Terra, respectively. The gap on its own answers does not exceed the average gap across the other four answering models.
2. Exhaustive human check on all cross-judge disagreements.
We screened all 4,500 answers for disagreements among the three judges and manually checked each disputed judgment against the answer text, scoring points, and supporting source files. We found no clearly unreasonable judgments or systematic errors favoring AD-IR. The observed disagreements mainly concerned how much credit to award for partial explanations, reflecting differences in grading strictness: GPT-6.1 Sol is the strictest, GPT-5.6 Sol is intermediate, and Gemini 4 Argon is the most lenient.
3. Source-only authoring and judge-free test grading. All 180 questions, reference answers, and scoring points were written in isolated source-only sessions without consulting AD-IR. Moreover, the DeepSWE v1.1 evaluation in W1 grades patches strictly with official unit test suites, completely independent of our QA benchmark and any LLM judge.
Response to Reviewer 2bZ8 (Part 3 of 3)
W3
The Source Retrieval baseline in RQ2 restricts coding agents to BM25 search instead of native shell exploration (grep, find, and file reads), and even against BM25 the gain on GPT-5.6 Sol is only +0.3.
Response to W3
In the original submission, Source Retrieval exposed source code only through BM25 search to mirror retrieval-based repository baselines. To evaluate agents exploring source code directly, we added two experimental conditions:
- Source + shell: the raw repository checkout at the benchmark commit and shell tools.
- Source + AD-IR: the same source code and tools, with complete AD-IR additionally available through
doc-drill.
Both conditions use the same CLI harness and allow repository exploration through ls, grep, find, and file reads, with no limit on tool calls.
The experiments cover all six repositories, 180 questions, and five answering models, yielding 900 answers per condition. We paired results by question and computed 95% confidence intervals using bootstrap resampling.
| Answering model | Source + shell (%) | AD-IR only (%) | Source + AD-IR (%) | (Source + AD-IR) − Source [95% CI] | Cost: Source | Cost: Source + AD-IR |
|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | 60.3 | 66.6 | 64.4 | +4.1 [+1.6, +6.4] | \$0.304 | \$0.228 (−24.9%) |
| Claude Sonnet 5 | 81.1 | 76.7 | 82.1 | +0.9 [−1.1, +3.0] | \$0.510 | \$0.506 (−0.8%) |
| GPT-5.6 Luna | 83.4 | 79.0 | 85.1 | +1.7 [−0.2, +3.4] | \$0.286 | \$0.238 (−16.8%) |
| GPT-5.6 Sol | 87.4 | 84.1 | 89.7 | +2.3 [+0.7, +3.9] | \$1.271 | \$1.167 (−8.2%) |
| GPT-5.6 Terra | 84.7 | 77.9 | 84.8 | +0.1 [−1.6, +1.7] | \$0.453 | \$0.443 (−2.2%) |
| Mean | 79.4 | 76.9 | 81.2 | +1.8 [+0.8, +2.8] | \$0.565 | \$0.516 (−8.6%) |
By scope, Source + shell / AD-IR only / Source + AD-IR achieve 83.6 / 78.7 / 85.5 on local questions, 79.3 / 78.2 / 80.7 on cross-module questions, and 75.3 / 73.6 / 77.4 on repository-wide questions.
Even with source code and unrestricted shell tools available, adding AD-IR raises overall coverage from 79.4% to 81.2% while reducing the cost per question from \0.516. We believe this gain is related to AD-IR's organization of module relationships and interactions in advance. When exploring source code directly, agents must determine which modules participate in a behavior and then locate the relevant implementations. AD-IR provides these connections, helping agents identify what to inspect before following source anchors to verify the details. This can reduce the risk of overlooking relevant modules and the need to repeatedly locate code.
Haiku's results further illustrate the value of analyzing source code in advance. Its coverage is 60.3% with source code alone and 66.6% with AD-IR alone. AD-IR is built by the stronger GPT-5.6 Sol, which performs part of the module identification, dependency analysis, and explanation of behavior during construction. Haiku can then use these results to compose its answers, reducing the work needed to infer module relationships while answering. This offers an explanation for how models with weaker source analysis capabilities can benefit from the documentation.
AD-IR's value therefore lies in completing part of the repository analysis in advance so that agents can reuse it for subsequent questions. For models that can explore source code directly, the documentation provides search guidance and existing explanations of behavior, while source code supports verification of implementation details. The new experiments show that this help remains useful under unrestricted source exploration, with benefits extending beyond the original BM25 retrieval baseline.
Q1
In the Source Retrieval condition (RQ2), could the agent access the repository through shell commands (e.g., grep/find and reading files), or only through the BM25 search tool?
Response to Q1
In the original Source Retrieval condition, the agent could access source code only through the BM25 search tool. We have now added Source + shell and Source + AD-IR with unrestricted shell commands (ls, grep, find, and file reads) across all six repositories, 180 questions, and five models; please see the response to W3 for the full results.
Q2
Can the authors report the improvement by ACCEED on the downstream issue resolution task such as DeepSWE v1.1?
Response to Q2
We evaluated all 38 tasks across 16 repositories in DeepSWE v1.1, with three independent runs per condition. Adding ACCEED yields the following results:
GPT-5.6 Lunaimproves from 43.0% to 52.6% in mean resolution rate, a gain of 9.6 percentage points.GPT-5.6 Solimproves from 61.4% to 65.8% in mean resolution rate, a gain of 4.4 percentage points.
See the response to W1 for the full experimental setup, confidence intervals, results from each run, and cost analysis.
Response to Reviewer 2bZ8 (Part 1 of 3)
We sincerely thank the reviewer for recognizing the practical importance of persistent repository knowledge and our localized maintenance design across real pull requests, as well as for proposing a clear experimental protocol to evaluate downstream coding. We ran the DeepSWE v1.1 evaluation and shell-agent baselines as suggested, revised the manuscript accordingly, and provide our point-by-point responses below.
W1
Despite motivating the paper with repository-level issue resolution, the main evaluation relies only on QA rather than real downstream coding tasks such as DeepSWE v1.1, and does not compare construction plus runtime costs against native Codex on issue resolution.
Response to W1
Thank you for this constructive critique and concrete experimental design. We clarify the role of QA in our target workflow and report the new DeepSWE v1.1 coding evaluation below:
1. Why the main evaluation uses documentation-only QA. ACCEED targets cross-repository development. An agent modifying repository B needs to understand an external dependency repository A, whose source code may not be available in the current workspace. For example, application code may call a low-level operator library or runtime library. Our deployment in Section 5.1 serves this type of workflow.
Documentation-only QA tests whether prebuilt AD-IR provides the information about boundary behavior and module interactions needed to understand external dependencies. We have clarified this evaluation objective in Sections 1 and 4.2.
No suitable public benchmark currently evaluates cross-repository code editing. We discuss this limitation in Section 6 and add a DeepSWE v1.1 evaluation in Section 4.7 to examine AD-IR's role in actual code editing.
2. Downstream issue resolution on DeepSWE v1.1.
Following your suggested protocol, we included every repository in DeepSWE v1.1 with at least two tasks, covering all 38 eligible tasks across 16 repositories.
Both conditions use Codex CLI 0.166.0 at
highreasoning effort withGPT-5.6 LunaandGPT-5.6 Sol. Each task is run independently three times with each model under each condition, yielding 456 trials in total, graded by the official test suites. The two conditions are:grep,find, shell commands, and tools for reading, writing, and editing files.doc-drill.Task counts are listed in order for runs 1, 2, and 3, each covering all 38 tasks. Compared with Native Codex using the same model, adding ACCEED yields the following results:
Luna's mean resolution rate increases by 9.6 percentage points, with a 95% confidence interval of [+1.8, +18.4]. Performance improves on 13 tasks, declines on 5, and is unchanged on 20.
Sol's mean resolution rate increases by 4.4 percentage points, with a 95% confidence interval of [−4.4, +14.0]. Performance improves on 7 tasks, declines on 7, and is unchanged on 24.
Documentation helps resolve tasks that repeatedly failed, while it can also reduce repeatability on individual tasks. Examining the three runs for each task reveals where the benefit occurs more clearly than the average resolution rate alone. For example,
GPT-5.6 Solimproves from 0/3 to 3/3 onkombu-virtual-queue-dead-lettering, and other tasks improve from 0/3 to 2/3 or 1/3. These changes mean that some previously unresolved tasks are solved at least once within the same three attempts. Most regressions are from 3/3 to 2/3 or from 2/3 to 1/3: these tasks remain solvable, but success is less repeatable.We interpret this pattern as documentation supplying module relationships and behavioral constraints that fill gaps in the agent's independent exploration, helping it find a viable modification that it previously missed. Additional context can also interfere with an otherwise effective judgment and cause occasional regressions. Documentation can therefore broaden the set of tasks resolved within a limited number of attempts, while its effect on the stability of individual runs must also be considered.