acceptodds
Under review as a conference paper at ICLR 2027

1D-Wiki-Bench: Evaluating Knowledge Production and Consumption from Requirements to Code

Abstract

The value of reusable project knowledge depends on artifact quality and its utility across requirement clarification and code implementation. We introduce 1D-WIKI-BENCH, a dataset and evaluation framework that connects knowledge production, evidence use, and downstream development. It contains 133 tasks across 10 source groups spanning 12 repositories, with fixed sources, vague requests, reference specifications, and implementation criteria. Knowledge systems build artifacts before target requests are revealed. A clarification agent produces a specification for a separate implementation session; both stages reuse the artifacts while retaining raw-source access. We evaluate artifact quality, evidence acquisition and application, specification and patch rubrics, and stage-wise resource use. Raw-only achieves the highest specification and patch rubrics. On matched tasks, all six Wiki-producing methods improve implementation evidence acquisition by 2.25–10.95 percentage points over Raw-only, yet score 1.91–4.80 points lower on the patch rubric. In 71 OpenWiki task pairs, criteria linked to newly acquired implementation evidence gain credit overall, while other criteria lose more. Traces show direct evidence provision and navigation to original sources. OpenWiki and Graphify save 7.77M and 21.45M consumption tokens, respectively, but neither offsets its construction cost over the evaluated tasks. These results motivate evaluating knowledge production and consumption jointly through artifact quality, task fulfillment, and total reuse cost.

Primary area:
datasets and benchmarks
Keywords:
Software Development Agents, Knowledge Engineering, Benchmarking, Knowledge Construction and Utilization, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CqH6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces 1D-Wiki-Bench to evaluate task-independent project knowledge artifacts, including Wikis, graphs, and indexes, in a two-stage coding-agent workflow. For each fixed source group, a knowledge system builds an artifact before individual task requests are revealed. A clarification agent turns a vague request into a specification; a separate implementation agent then receives the request and specification, accesses the clean repository baseline, and produces a patch. Both agents retain raw-source access. The paper evaluates seven knowledge systems against a Raw-only baseline using artifact-quality scores, evidence-acquisition/application diagnostics, specification and patch rubrics, and token/time accounting.

The key empirical claim is that Wiki-producing methods often acquire more implementation evidence but yield lower end-to-end patch-rubric scores than Raw-only, while their construction costs are not recovered over the evaluated tasks. The paper therefore argues that knowledge production should be evaluated jointly with downstream task fulfillment and reuse cost. The benchmark uses accepted changes, tests, reviews, and project evidence to construct reference specifications, weighted criteria, and evidence needs. However, the main comparison scores candidate specifications and patches through a static LLM-judge rubric; it does not execute candidate patches with task-specific or regression tests.

Critical strengths

The paper addresses a relevant practical question. Repository-level Wikis, graphs, indexes, and other forms of agent context are increasingly proposed as reusable infrastructure, yet many evaluations focus only on documentation quality, retrieval quality, or a single issue-resolution outcome. The paper instead asks whether preparing such knowledge before a task is known yields incremental benefit over direct access to project sources, and whether that benefit could justify construction cost. This is a useful framing because a knowledge artifact should ultimately be judged by the coding work it enables, not solely by whether it appears coherent or well organized.

The use of a Raw-only baseline is also a sensible design choice. In realistic coding-agent settings, a Wiki usually complements rather than replaces repository search and file reading. Retaining raw-source access in both conditions avoids an artificial comparison in which a Wiki is forced to serve as the agent’s only source of information. The paper’s attempt to record resource consumption and to distinguish preparation cost from per-task consumption is similarly relevant to deployment decisions.

Finally, the diagnostic ambition is potentially valuable. Separating artifact quality, information access, output quality, and cost could help future work understand why a context mechanism succeeds or fails, rather than treating all improvements as a single end-to-end score. The reported discrepancy between evidence acquisition and patch-rubric performance is therefore worth investigating. However, these strengths are primarily strengths of the problem framing and research direction. They do not compensate for the lack of executable evaluation and the insufficient validation of the proposed diagnostic measurements.

Critical weaknesses
  1. The central implementation-quality claims are not evaluated by executing candidate patches. The paper validates some historical reference patches using task-specific and available regression checks in Appendix C, and defines executable candidate-patch evaluation in Appendix D.4. However, Appendix D.4 and the Limitations section explicitly state that the reported main comparison and source-access ablation use static patch rubrics and that dynamic task/regression tests were not run. The main “E2E implementation rubric” is thus an LLM judge’s static assessment of candidate diffs against weighted criteria, as described in Appendix D.3 and Appendix J.3. Consequently, conclusions such as Raw-only producing better patches or Wiki methods harming implementation quality are supported only under this static LLM rubric measurement, not as claims about functional correctness, regressions, or issue resolution. This is a fundamental limitation for a benchmark paper whose central claim concerns downstream coding utility.

  2. The evaluation relies excessively on LLM judging for almost every substantive metric. Section 5.1.2 states that GPT-5.5 evaluates Wiki quality and consumer outputs. Sections 4.1 and 4.2 and Appendix D show that Wiki quality, specification quality, patch quality, evidence acquisition, and evidence application are all evaluated through LLM semantic judgments. LLM judges can be useful, but the paper does not report judge stability, human validation, inter-rater agreement, calibration, or correlation with executable outcomes. Hence, it is unclear whether the method ranking reflects robust engineering utility or behavior specific to the chosen judge.

  3. The benchmark stacks too many empirically designed intermediate constructs without establishing their necessity or validity. The pipeline includes reference specifications, weighted criteria, knowledge needs, evidence probes, acquisition judgments, application judgments, a two-stage specification handoff, and artifact-quality rubrics. The paper offers intuitive motivations, but does not demonstrate that this complicated pipeline is more reliable, more diagnostic, or more informative than a simpler evaluation centered on executable task success. Each extra layer introduces annotation choices, thresholds, weights, and additional LLM-judge dependence. For a benchmark paper, adding intermediate concepts is not itself evidence that the evaluation is better; the paper needs ablations or validation demonstrating the value of these constructs.

  4. The construction of reference specifications, criteria, weights, and evidence probes is insufficiently transparent. Section 3.3 and Appendix C.1 only state that accepted changes, tests, reviews, and project evidence “inform” the reference specification and acceptance criteria. The paper does not clearly state who constructed these objects, whether LLMs assisted construction, what human review or adjudication occurred, how disagreements were handled, or how weights were chosen. For evidence acquisition, Appendix D.5 and Appendix J.2 require the judge to decide whether returned content in a long trace is sufficient for a pre-defined probe. This is coverage of an author-constructed schema, not an objective measure of all useful evidence obtained by the agent. The lack of released/validated task-level specifications, mappings, and alignment records undermines reproducibility.

  5. The necessity of a new benchmark and the data-selection process are insufficiently justified. The paper does not explain why an executable benchmark such as SWE-bench could not be augmented with task-independent repository knowledge artifacts. Section 3.3 states that tasks are mined from feature requests and bug fixes associated with accepted changes after a fixed baseline, but does not report repository/task inclusion criteria, the number of candidate changes filtered out, filtering reasons, the proportion of issue-derived versus reconstructed vague requests, or the proportion of executable versus developer-reviewed tasks. Retrospectively reconstructing vague requests and reference specifications from known accepted changes also risks hindsight and selection bias.

  6. The fixed, one-way clarification-to-implementation handoff is a strong synthetic assumption. Section 3.2 and Appendix B specify that the implementation stage starts in a fresh session and consumes a frozen specification. Real coding agents can iterate between clarification, source inspection, implementation, testing, and further clarification. The final patch score here mixes Wiki utility, clarification quality, handoff loss, and implementation ability. Since the implementation agent can also reread the raw sources, the paper cannot isolate the individual contribution of the generated specification or the Wiki. Direct single-stage, gold-specification, no-specification, or iterative-workflow controls are needed.

  7. The paper is poorly written for a benchmark paper whose protocol is its main contribution. As noted above, the abstract does not establish the task and comparison conditions before using its key terms. The body also introduces numerous abstract concepts and abbreviations before giving a concise account of what is fixed, what the agent sees, how a task is constructed, and how each metric is calculated. This makes the protocol and the scope of its conclusions unnecessarily difficult to assess.

Questions for authors
  1. For how many tasks can candidate patches be applied and evaluated with task-specific behavioral checks and regression checks? Why were these checks not run for candidate patches in the main comparison, given that the paper uses such checks to validate reference patches? Please report executable coverage, target-test pass rates, regression pass rates, and their relation to the static patch rubric.

  2. Please precisely document how each task’s reference specification, weighted clarification/implementation criteria, knowledge needs, and evidence probes were constructed. Who constructed them? Was any LLM used? What human review, independent annotation, adjudication, or agreement analysis was performed? How were weights selected?

  3. Please provide task-level examples connecting the vague request, reference specification/criteria, evidence probes and source locations, credited trace events, and final rubric decisions. How are semantically equivalent pieces of evidence not anticipated by a pre-defined probe handled?

  4. Please report repository/task inclusion and exclusion criteria; the number of candidate changes considered and excluded; the fraction of issue-derived versus reconstructed requests; and the fraction of tasks validated by executable checks versus developer review only.

Suggestions for future work:

  1. Compare with direct single-stage agents, gold-specification and no-specification conditions, and iterative clarify/implement/test workflows to separate the effects of Wiki access, handoff, and implementation ability.
1 comment
AuthorsOpenReview ↗

Author response on task construction and the validity of evidence-use metrics

Thank you for asking how we constructed the tasks and reference criteria, what the intermediate measures reveal, and whether the final patches implement the requested behavior. We have revised the main text and submitted an updated PDF of the manuscript. The revised passages are highlighted in red in the submitted PDF.

Workflow and task construction. Our domain experts synthesized a recurring pattern from current development projects. Teams clarify a business request against project rules, write an actionable specification, and hand it off for implementation. Our aim is to evaluate the practical utility of a project knowledge base in that workflow. We therefore compare agents that can consult a prepared knowledge base and the original sources with agents that can consult the same sources directly. Following each request through the handoff lets us measure what the knowledge base adds to the specification and the resulting patch, alongside the cost of building and reusing it. Section 3.3 and Appendix A describe the 62 atomic and 71 composite tasks. Requests come from issues or are reconstructed from accepted changes. Reference specifications, weighted criteria, and shared evidence probes draw on accepted changes, tests, reviews, and fixed sources. The revision describes this process and how weights enter the scores. The original candidate pool, exclusion counts, issue-derived fraction, and task-level records of model assistance, weight selection, and human adjudication remain incompletely documented.

What the executed checks establish. All 133 tasks have reference-validated acceptance checks, totaling 3,357 cases. Each task contributes its fraction of passing cases. OpenKB scores 44.18% and Raw-only 43.45%, and all seven paired 95% intervals include zero (Table 12). Across 132 tasks with cases that fail on the original code and pass with the reference patch, OpenWiki gains on 29 tasks, ties on 69, and declines on 34 (Table 14). The executable results capture variation in functional outcomes that a single average obscures. The revision does not provide separate regression rates, binary resolution rates, or candidate-level correlations between executable and static scores.

What the evidence measures add. On the Saleor composite task, OpenKB acquired evidence for 87.5% of the frozen knowledge needs, compared with 37.5% for Raw-only. Its patch addressed variant sorting but omitted customer metadata and external-authentication behavior. Its patch score was 40.00 versus 78.33 for Raw-only (Tables 10–11). The traces and criteria locate a specific gap between finding relevant information and implementing the requested behavior. Across 71 OpenWiki task pairs, 173 newly acquired probes mapped to 159 criteria. Those criteria gained weighted credit in aggregate, while other criteria in the same tasks lost more (Table 9). This is a descriptive comparison using pooled criterion weights. Equivalent evidence can satisfy a frozen need when the returned content supports it and the trace records access. The application measure also considers relevant knowledge extracted from the actual trace. A systematic ablation and a single worked record linking every construction and evaluation step remain open.

Measurement and workflow controls. Expert-reviewed and model-judged patch totals had a Spearman rank correlation of 0.699 on 16 candidates covering 196 criteria (Appendix C.6). This gives preliminary support for patch ranking. Artifact quality, reference specifications, and evidence judgments still require separate independent validation. On 20 tasks, OpenWiki’s mean static patch advantage was +4.43 points [+0.15, +9.16] across three independent generations. On those same tasks, giving both implementers the Raw-only specification from the preassigned first replicate yielded an OpenWiki difference of −0.99 [−7.10, +6.40] (Tables 17–18). The latter comparison addresses one handoff question. Single-stage, reference-specification, and iterative workflows remain open comparisons.

We prioritized executable evaluation on all 133 tasks. Given the revision period, the compute required for repeated runs, and the expert time required for independent checks, we ran the repeatability and shared-specification studies on 20 tasks and reviewed 16 candidate patches with experts. In future work, we will expand these controls across tasks and methods, document task selection and criterion construction more fully, and extend independent expert checks to reference specifications and evidence judgments.

Reviewer_MjzeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Wikis, graphs, and indexes are built from project sources for agent reuse, but their marginal value over direct source access across clarification and implementation, and whether reuse offsets construction cost, is unmeasured.

This paper evaluates prepared knowledge across the complete sequence of artifact construction, requirement clarification, and implementation. Its principal contribution is the linked benchmark and accounting framework. The benchmark contains 133 tasks across ten source groups and 12 repositories. Seven knowledge systems construct reusable artifacts before seeing target requests. A clarification agent produces a specification, which a separate implementation session consumes. All conditions retain access to original sources; Raw-only receives no prepared artifact. Evaluation measures wiki quality, evidence acquisition and application, specification and patch rubrics, and construction plus consumption resources.

On matched tasks, all six wiki-producing systems increase implementation evidence acquisition by 2.25–10.95 points, yet reduce patch-rubric scores by 1.91–4.80 points. A 71-task OpenWiki analysis finds gains on criteria linked to newly acquired evidence, offset by larger losses elsewhere. OpenWiki and Graphify save consumption tokens but do not recover construction costs over the evaluated tasks. Crucially, candidate patches are judged statically: the benchmark evaluation did not execute their task and regression tests.

Critical strengths

1.The question matters and the protocol answers it fairly: pre-disclosure construction, raw access in every condition, construction charged once per group.

  1. Diagnostics go beyond a leaderboard: provenance tracing, criterion-level decomposition showing local gains offset elsewhere, zero-score decomposition, retry-controlled timing.
Critical weaknesses
  1. The headline differences are small, LLM-judged, and static. The paper defines an executable-check extension and has validation records but never runs it. A claim that wikis reduce implementation quality needs executed tests or evidence the judge is reliable at 2–5 point resolution.

  2. The comparison mixes representation, retrieval interface, input scope, and tool-budget effects. Systems ingest different inputs and consume different numbers of the 30 calls. With one consumer model the result may be specific to this agent and budget.

Questions for authors
  1. Can you run the executable checks on tasks with fail-to-pass tests? Judge–human agreement on the patch rubric?

  2. On tasks with executable checks, does the Raw-only advantage persist under task and regression tests? Alternatively, what independent human validation supports the static patch ranking?

  3. How often did the 30-call "finish now" instruction fire per method? Provenance and relevance of the collected business documents? Run-to-run variance on a subset?

1 comment
AuthorsOpenReview ↗

Author response on patch-score reliability, executable tests, and tool budgets

Thank you for asking whether the small differences in static patch scores carry over to executable behavior and how the tool budget affects that comparison. We have revised the main text and submitted an updated PDF of the manuscript. The revised passages are highlighted in red in the submitted PDF.

Executable behavior. Raw-only had the highest mean static patch score on 133 tasks (56.57). OpenKB had the highest mean executable acceptance (44.18%, versus 43.45% for Raw-only). Acceptance averages the fraction of passing cases within each task. All seven paired 95% acceptance intervals include zero (Table 12). The task-level results help interpret those averages. On the 132 tasks with strict fail-to-pass cases, OpenWiki improved on 29, tied on 69, and declined on 34. The median gain on improved tasks was 23.53 percentage points and the median loss on declined tasks was 20.71 points. Gains and losses partly offset, yielding a mean difference of −2.09 points [−11.22, +4.82] (Tables 13–14). The executable evidence therefore leaves the overall ranking uncertain. Separate regression-suite comparisons remain open.

How much weight should a 2–5 point rubric difference carry? The expert pilot covered 16 candidate patches, two per method, and 196 criteria. Weighted candidate totals had a Spearman rank correlation of 0.699 with the expert-reviewed reference (Appendix C.6). This supports the rubric as a preliminary measure of relative patch quality. Its ability to resolve differences as small as two points remains uncertain. Under repeated judging of fixed outputs, the root-mean-square within-task standard deviations were 3.80 points for Raw-only and 5.49 for OpenWiki. On all 133 tasks OpenWiki’s static patch difference was −3.41 [−9.15, +2.43]. On the 20-task repeatability subset it was already +1.19 in the retained run and averaged +4.43 [+0.15, +9.16] across three independent runs (Tables 16–17). We have limited the interpretation of that positive result to the subset.

Interfaces and budget. All consumers had complete raw-source access, one consumer model, and the same limits of 70 turns, 7,200 seconds, and 30 non-writing tool calls per stage. Native retrieval interfaces differ in how much work one call performs (Section 5.1 and Appendix B). In the archived trace audit, the finish instruction fired in the following numbers of clarification and implementation traces, respectively. Denominators apply to each stage.

MethodClarificationImplementation
Raw-only120/13389/133
OpenWiki120/13395/133
Graphify91/13372/133
OpenKB100/13379/133
DeepWiki-Open125/13377/133
OpenDeepWiki114/13371/133
LLM-Wiki111/11782/117
SAG117/13285/132

These counts describe available archived traces. Their correspondence to the final acceptance candidates has yet to be verified. We interpret the main results as comparisons of complete systems under this budget. The effect of changing the budget remains open.

Workflow, sources, and cost. Our domain experts identified a recurring workflow in current projects. Teams turn a vague request into an actionable specification and use it to guide implementation. Our question is whether adding a project knowledge base to the sources already available helps teams deliver the requested change, and at what cost. The benchmark evaluates that question with agents performing the two stages. Appendix A reports 4,432 public business documents on concepts, product rules, and workflows alongside 68,072 engineering files. Document-level relevance screening still needs fuller documentation. OpenWiki and Graphify saved 7.77M and 21.45M consumption tokens, respectively, but neither recovered its construction cost within the evaluated tasks. The executable comparison covers all 133 tasks. Within the revision period, available compute and expert-review capacity limited repeated generation and the shared-specification control to 20 tasks spanning all ten groups, with expert review of 16 patches. In future work, we will repeat the comparison on more tasks and vary the tool-call budget and consumer model to test how sensitive the observed rankings are to those choices.

Reviewer_audpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper questions the usefulness of project knowledge (Wikis, graphs, indexes) by coding agents beyond direct access to raw source codes. If it does, does it pay back construction cost?

The paper introduces benchmark, containing 133 tasks in 10 source groups, covering 12 repositories (Appwrite, Outline, Saleor, Lago, Mattermost, Nextcloud and others). Each task has a vague request, a reference specification, an accepted patch and weighted acceptance criteria. Each system builds its artifact once per source group, before any task is revealed.

The paper's main finding is the Raw-only has the highest specification rubric and the highest patch rubric (56.57), questioning the utility of project knowledge.

Critical strengths
  1. The paper addresses a valued problem, whether prepared knowledge adds value over just reading the source code is a real question for code agents.

  2. The solution design is sound, artifacts are build before tasks are revealed, and implementation starts in a fresh session.

  3. Useful negative result. The finding that Raw-only wins, and that acquisition gains do not become patch gains, is a counterintuitive result worth knowing. It agrees with the AGENTS.md literature the paper cites.

Critical weaknesses

No uncertainty estimates for the main claims. The headline gaps are based from single runs, scored by one LLM judge. No confidence interval, bootstrap or significance test is given for Table 1 or Table 8. Agent are stochastic, and each task and stage uses a single generation. Run-to-run variance is never measured. Evaluation is an LLM judge with no validation, and no executed tests. Patches are judged statically. Appendix D.4 defines a test-based "Resolved" metric but never runs it, so "patch rubric" is a semantic score, not correctness.

Questions for authors

Please report confidence intervals for all paired differences in Tables 1 and 8, and repeat Raw-only (and at least one Wiki method) several times to estimate run-to-run variance.

How were requests, reference specifications and criteria produced and checked, and how many tasks have executable tests?

What does the comparison look like if every implementer receives the same specification (reference or Raw-only), so that implementation-stage effects are separated from clarification?

Can you run the executable checks defined in Appendix D.4 on the tasks that have fail-to-pass tests, and compare "Resolved" with the static rubric?

1 comment
AuthorsOpenReview ↗

Author response on repeatability and the shared-specification control

Thank you for asking whether the implementation results hold across repeated runs and when both implementers receive the same specification. We have revised the main text and submitted an updated PDF of the manuscript. The revised passages are highlighted in red in the submitted PDF.

Repeated runs. On all 133 tasks, OpenWiki scored 3.41 points below Raw-only on the static patch rubric (95% interval for the difference [−9.15, +2.43]). On the 20 tasks selected for repeated generation, OpenWiki already scored 1.19 points higher in the original run. Across three independent runs per method, its mean advantage on those same tasks was 4.43 points [+0.15, +9.16]. This is evidence of a local benefit. The subset comprised one atomic and one composite task from each of the ten source groups. Appendix G reports paired intervals from 20,000 source-group bootstrap resamples and distinguishes uncertainty across groups from variation across runs (Tables 16–17).

Holding the specification fixed. Our domain experts drew on current development projects to identify a recurring practice. Teams clarify an initially vague request, document the agreed requirements, and hand the specification to the implementer. We use this workflow to evaluate what a prepared project knowledge base adds to development when both stages already have access to the original sources. To examine the implementation stage separately, we gave OpenWiki the specification from Raw-only’s preassigned first replicate. On the same 20 tasks, OpenWiki scored 57.89 and Raw-only 58.88 using that specification, a difference of −0.99 points [−7.10, +6.40]. OpenWiki scored 62.80 with its own specification in that replicate (Table 18). The interval leaves the direction of the implementation-stage effect uncertain.

Task construction and executable checks. Requests come from issues or are reconstructed from accepted changes. We used accepted patches, tests, reviews, and fixed project sources to construct reference specifications and criteria (Section 3.3 and Appendix A). All 133 tasks have reference-validated acceptance checks. Across 3,357 cases, averaging the pass fraction within each task gives 43.45% for Raw-only and 44.18% for OpenKB. The strict subset contains 1,830 cases that fail on the original code and pass with the reference patch, covering 132 tasks (Appendix F). The revision reports partial task fulfillment through these fractions. Binary resolution rates and candidate-level correlations with the static rubric remain open.

Judge reliability and coverage. Expert-reviewed and model-judged totals for 16 candidate patches covering 196 criteria had a Spearman rank correlation of 0.699 (Appendix C.6). Rejudging fixed patches gave root-mean-square within-task standard deviations of 3.80 points for Raw-only and 5.49 for OpenWiki. Independent generation and scoring gave 12.31 and 12.83 points, respectively (Appendix G.1). Executable checks covered all 133 tasks. The time available for the revision, the compute required for repeated runs, and the availability of expert reviewers limited repeated generation and the shared-specification control to 20 tasks and expert review to 16 patches. In future work, we will extend both controls to more tasks and knowledge systems and broaden expert review of patch rankings, reference specifications, and criterion weights.

For clarity, the full 133-task comparison counts 16 tasks as zero after an LLM-Wiki build failure. Its evidence-acquisition and patch-score differences from Raw-only are +0.87 and −7.51 points there, versus +6.90 and −1.91 on the 117 tasks with a completed build. The other five Wiki methods use 133 tasks in both comparisons.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author summary of revisions to 1D-WIKI-BENCH

We thank all three reviewers for their detailed feedback and the Area Chair and Senior Area Chair for overseeing the discussion. We have revised the main text and submitted an updated PDF. The revised passages are highlighted in red.

For the Area Chair: Two changes affect how we interpret the main result. We now report executable acceptance alongside static patch scores, and we distinguish gains on particular tasks from an advantage across the full benchmark. The benchmark follows a workflow our domain experts identified in current development projects: teams clarify a business request, document the agreed requirements, and hand the specification to an implementer. Our question is what a reusable knowledge base adds to that workflow when agents can already consult the original documents and code.

Executable outcomes. Raw-only had the highest mean static patch score across 133 tasks. OpenKB had the highest executable acceptance, measured as the mean fraction of test cases passed per task: 44.18%, compared with 43.45% for Raw-only. All seven paired 95% acceptance intervals included zero. The task-level results show what these averages conceal. On the 132 tasks with strict fail-to-pass cases, OpenWiki improved on 29 tasks, tied on 69, and declined on 34, for a mean difference of −2.09 percentage points (Section 5.2.1 and Appendix F).

Repeatability and measurement. OpenWiki’s mean static patch score was 3.41 points below Raw-only across all 133 tasks. On the 20 tasks selected for repeated generation, OpenWiki was already 1.19 points ahead in the original run and averaged 4.43 points ahead across three independent runs (95% interval [+0.15, +9.16]). This supports a local benefit on that subset. When both implementers received the same Raw-only specification, the difference was −0.99 points [−7.10, +6.40], leaving the implementation-stage effect uncertain. An expert review of 16 patches covering 196 criteria yielded a Spearman rank correlation of 0.699 with the judge’s patch scores, providing preliminary support for patch ranking (Appendices C.6 and G).

The revised conclusion is that prepared knowledge can improve evidence acquisition, benefit particular tasks, and reduce consumption costs. OpenWiki and Graphify both saved consumption tokens, but neither recovered its construction cost over the evaluated tasks. Taken together, the results have yet to establish a large, consistent overall advantage over direct-source access under the evaluated model, budgets, and reuse horizon.

Within the revision period, we prioritized executable evaluation across all 133 tasks. Available compute and expert-review time limited repeated generation and the shared-specification control to 20 tasks spanning all ten source groups, and expert review to 16 patches. Future work will extend these controls to more tasks, methods, and consumer models and broaden independent validation of the scoring criteria and evidence judgments. Our responses under each review address the individual questions in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.