- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces 1D-Wiki-Bench to evaluate task-independent project knowledge artifacts, including Wikis, graphs, and indexes, in a two-stage coding-agent workflow. For each fixed source group, a knowledge system builds an artifact before individual task requests are revealed. A clarification agent turns a vague request into a specification; a separate implementation agent then receives the request and specification, accesses the clean repository baseline, and produces a patch. Both agents retain raw-source access. The paper evaluates seven knowledge systems against a Raw-only baseline using artifact-quality scores, evidence-acquisition/application diagnostics, specification and patch rubrics, and token/time accounting.
The key empirical claim is that Wiki-producing methods often acquire more implementation evidence but yield lower end-to-end patch-rubric scores than Raw-only, while their construction costs are not recovered over the evaluated tasks. The paper therefore argues that knowledge production should be evaluated jointly with downstream task fulfillment and reuse cost. The benchmark uses accepted changes, tests, reviews, and project evidence to construct reference specifications, weighted criteria, and evidence needs. However, the main comparison scores candidate specifications and patches through a static LLM-judge rubric; it does not execute candidate patches with task-specific or regression tests.
Critical strengths
The paper addresses a relevant practical question. Repository-level Wikis, graphs, indexes, and other forms of agent context are increasingly proposed as reusable infrastructure, yet many evaluations focus only on documentation quality, retrieval quality, or a single issue-resolution outcome. The paper instead asks whether preparing such knowledge before a task is known yields incremental benefit over direct access to project sources, and whether that benefit could justify construction cost. This is a useful framing because a knowledge artifact should ultimately be judged by the coding work it enables, not solely by whether it appears coherent or well organized.
The use of a Raw-only baseline is also a sensible design choice. In realistic coding-agent settings, a Wiki usually complements rather than replaces repository search and file reading. Retaining raw-source access in both conditions avoids an artificial comparison in which a Wiki is forced to serve as the agent’s only source of information. The paper’s attempt to record resource consumption and to distinguish preparation cost from per-task consumption is similarly relevant to deployment decisions.
Finally, the diagnostic ambition is potentially valuable. Separating artifact quality, information access, output quality, and cost could help future work understand why a context mechanism succeeds or fails, rather than treating all improvements as a single end-to-end score. The reported discrepancy between evidence acquisition and patch-rubric performance is therefore worth investigating. However, these strengths are primarily strengths of the problem framing and research direction. They do not compensate for the lack of executable evaluation and the insufficient validation of the proposed diagnostic measurements.
Critical weaknesses
-
The central implementation-quality claims are not evaluated by executing candidate patches. The paper validates some historical reference patches using task-specific and available regression checks in Appendix C, and defines executable candidate-patch evaluation in Appendix D.4. However, Appendix D.4 and the Limitations section explicitly state that the reported main comparison and source-access ablation use static patch rubrics and that dynamic task/regression tests were not run. The main “E2E implementation rubric” is thus an LLM judge’s static assessment of candidate diffs against weighted criteria, as described in Appendix D.3 and Appendix J.3. Consequently, conclusions such as Raw-only producing better patches or Wiki methods harming implementation quality are supported only under this static LLM rubric measurement, not as claims about functional correctness, regressions, or issue resolution. This is a fundamental limitation for a benchmark paper whose central claim concerns downstream coding utility.
-
The evaluation relies excessively on LLM judging for almost every substantive metric. Section 5.1.2 states that GPT-5.5 evaluates Wiki quality and consumer outputs. Sections 4.1 and 4.2 and Appendix D show that Wiki quality, specification quality, patch quality, evidence acquisition, and evidence application are all evaluated through LLM semantic judgments. LLM judges can be useful, but the paper does not report judge stability, human validation, inter-rater agreement, calibration, or correlation with executable outcomes. Hence, it is unclear whether the method ranking reflects robust engineering utility or behavior specific to the chosen judge.
-
The benchmark stacks too many empirically designed intermediate constructs without establishing their necessity or validity. The pipeline includes reference specifications, weighted criteria, knowledge needs, evidence probes, acquisition judgments, application judgments, a two-stage specification handoff, and artifact-quality rubrics. The paper offers intuitive motivations, but does not demonstrate that this complicated pipeline is more reliable, more diagnostic, or more informative than a simpler evaluation centered on executable task success. Each extra layer introduces annotation choices, thresholds, weights, and additional LLM-judge dependence. For a benchmark paper, adding intermediate concepts is not itself evidence that the evaluation is better; the paper needs ablations or validation demonstrating the value of these constructs.
-
The construction of reference specifications, criteria, weights, and evidence probes is insufficiently transparent. Section 3.3 and Appendix C.1 only state that accepted changes, tests, reviews, and project evidence “inform” the reference specification and acceptance criteria. The paper does not clearly state who constructed these objects, whether LLMs assisted construction, what human review or adjudication occurred, how disagreements were handled, or how weights were chosen. For evidence acquisition, Appendix D.5 and Appendix J.2 require the judge to decide whether returned content in a long trace is sufficient for a pre-defined probe. This is coverage of an author-constructed schema, not an objective measure of all useful evidence obtained by the agent. The lack of released/validated task-level specifications, mappings, and alignment records undermines reproducibility.
-
The necessity of a new benchmark and the data-selection process are insufficiently justified. The paper does not explain why an executable benchmark such as SWE-bench could not be augmented with task-independent repository knowledge artifacts. Section 3.3 states that tasks are mined from feature requests and bug fixes associated with accepted changes after a fixed baseline, but does not report repository/task inclusion criteria, the number of candidate changes filtered out, filtering reasons, the proportion of issue-derived versus reconstructed vague requests, or the proportion of executable versus developer-reviewed tasks. Retrospectively reconstructing vague requests and reference specifications from known accepted changes also risks hindsight and selection bias.
-
The fixed, one-way clarification-to-implementation handoff is a strong synthetic assumption. Section 3.2 and Appendix B specify that the implementation stage starts in a fresh session and consumes a frozen specification. Real coding agents can iterate between clarification, source inspection, implementation, testing, and further clarification. The final patch score here mixes Wiki utility, clarification quality, handoff loss, and implementation ability. Since the implementation agent can also reread the raw sources, the paper cannot isolate the individual contribution of the generated specification or the Wiki. Direct single-stage, gold-specification, no-specification, or iterative-workflow controls are needed.
-
The paper is poorly written for a benchmark paper whose protocol is its main contribution. As noted above, the abstract does not establish the task and comparison conditions before using its key terms. The body also introduces numerous abstract concepts and abbreviations before giving a concise account of what is fixed, what the agent sees, how a task is constructed, and how each metric is calculated. This makes the protocol and the scope of its conclusions unnecessarily difficult to assess.
Questions for authors
-
For how many tasks can candidate patches be applied and evaluated with task-specific behavioral checks and regression checks? Why were these checks not run for candidate patches in the main comparison, given that the paper uses such checks to validate reference patches? Please report executable coverage, target-test pass rates, regression pass rates, and their relation to the static patch rubric.
-
Please precisely document how each task’s reference specification, weighted clarification/implementation criteria, knowledge needs, and evidence probes were constructed. Who constructed them? Was any LLM used? What human review, independent annotation, adjudication, or agreement analysis was performed? How were weights selected?
-
Please provide task-level examples connecting the vague request, reference specification/criteria, evidence probes and source locations, credited trace events, and final rubric decisions. How are semantically equivalent pieces of evidence not anticipated by a pre-defined probe handled?
-
Please report repository/task inclusion and exclusion criteria; the number of candidate changes considered and excluded; the fraction of issue-derived versus reconstructed requests; and the fraction of tasks validated by executable checks versus developer review only.
Suggestions for future work:
- Compare with direct single-stage agents, gold-specification and no-specification conditions, and iterative clarify/implement/test workflows to separate the effects of Wiki access, handoff, and implementation ability.
Author response on task construction and the validity of evidence-use metrics
Thank you for asking how we constructed the tasks and reference criteria, what the intermediate measures reveal, and whether the final patches implement the requested behavior. We have revised the main text and submitted an updated PDF of the manuscript. The revised passages are highlighted in red in the submitted PDF.
Workflow and task construction. Our domain experts synthesized a recurring pattern from current development projects. Teams clarify a business request against project rules, write an actionable specification, and hand it off for implementation. Our aim is to evaluate the practical utility of a project knowledge base in that workflow. We therefore compare agents that can consult a prepared knowledge base and the original sources with agents that can consult the same sources directly. Following each request through the handoff lets us measure what the knowledge base adds to the specification and the resulting patch, alongside the cost of building and reusing it. Section 3.3 and Appendix A describe the 62 atomic and 71 composite tasks. Requests come from issues or are reconstructed from accepted changes. Reference specifications, weighted criteria, and shared evidence probes draw on accepted changes, tests, reviews, and fixed sources. The revision describes this process and how weights enter the scores. The original candidate pool, exclusion counts, issue-derived fraction, and task-level records of model assistance, weight selection, and human adjudication remain incompletely documented.
What the executed checks establish. All 133 tasks have reference-validated acceptance checks, totaling 3,357 cases. Each task contributes its fraction of passing cases. OpenKB scores 44.18% and Raw-only 43.45%, and all seven paired 95% intervals include zero (Table 12). Across 132 tasks with cases that fail on the original code and pass with the reference patch, OpenWiki gains on 29 tasks, ties on 69, and declines on 34 (Table 14). The executable results capture variation in functional outcomes that a single average obscures. The revision does not provide separate regression rates, binary resolution rates, or candidate-level correlations between executable and static scores.
What the evidence measures add. On the Saleor composite task, OpenKB acquired evidence for 87.5% of the frozen knowledge needs, compared with 37.5% for Raw-only. Its patch addressed variant sorting but omitted customer metadata and external-authentication behavior. Its patch score was 40.00 versus 78.33 for Raw-only (Tables 10–11). The traces and criteria locate a specific gap between finding relevant information and implementing the requested behavior. Across 71 OpenWiki task pairs, 173 newly acquired probes mapped to 159 criteria. Those criteria gained weighted credit in aggregate, while other criteria in the same tasks lost more (Table 9). This is a descriptive comparison using pooled criterion weights. Equivalent evidence can satisfy a frozen need when the returned content supports it and the trace records access. The application measure also considers relevant knowledge extracted from the actual trace. A systematic ablation and a single worked record linking every construction and evaluation step remain open.
Measurement and workflow controls. Expert-reviewed and model-judged patch totals had a Spearman rank correlation of 0.699 on 16 candidates covering 196 criteria (Appendix C.6). This gives preliminary support for patch ranking. Artifact quality, reference specifications, and evidence judgments still require separate independent validation. On 20 tasks, OpenWiki’s mean static patch advantage was +4.43 points [+0.15, +9.16] across three independent generations. On those same tasks, giving both implementers the Raw-only specification from the preassigned first replicate yielded an OpenWiki difference of −0.99 [−7.10, +6.40] (Tables 17–18). The latter comparison addresses one handoff question. Single-stage, reference-specification, and iterative workflows remain open comparisons.
We prioritized executable evaluation on all 133 tasks. Given the revision period, the compute required for repeated runs, and the expert time required for independent checks, we ran the repeatability and shared-specification studies on 20 tasks and reviewed 16 candidate patches with experts. In future work, we will expand these controls across tasks and methods, document task selection and criterion construction more fully, and extend independent expert checks to reference specifications and evidence judgments.