acceptodds
Under review as a conference paper at ICLR 2027

Repository-Local Decision Context for Reliable Coding Agents

Abstract

Large language model agents can generate plausible software quickly, but production engineering requires preserving product intent, architectural decisions, edge cases, and escalation boundaries. We investigate whether repository-local decision context improves coding-agent reliability and efficiency. This context includes structured product specifications, architecture decision records, task-local guidance, review policies, and human gates. We began with an industrial case. A multi-agent prototype ran approximately 250 agent sessions over 15 hours. It produced a running application and more than 20,000 lines of reviewable code. Hardening later exposed cyclic dependencies, unsuitable architecture, integration assumptions, and widespread test breakage. Narrower work units improved reviewability, yet technically valid changes remained functionally wrong when product decisions were implicit. A production workflow assigned humans responsibility for domain judgment and consequential decisions while agents performed bounded planning, implementation, testing, and review. An observational matched-product comparison reported roughly five times more output per engineer, but cannot isolate the bundled interventions. We propose a controlled, multi-repository benchmark comparing search-only access, global documentation, task-local decision context, and a complete context-review-escalation workflow. Hidden functional tests, machine-checkable architecture constraints, blinded review, and factorial ablations measure correctness, architectural violations, security, correction effort, token use, and abstention. The study tests which human decisions must be externalized for reliable agentic software development.

Primary area:
datasets and benchmarks
Keywords:
coding agents, repository-level software engineering, context engineering, architectural decision records, software engineering benchmarks, human-AI collaboration, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PuDvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether repository-local decision context, including product requirements, architectural decisions, and task-specific guidance, can improve the reliability of coding agents in repository-level software development tasks. The authors conduct an exploratory pilot study involving eight tasks from six repositories, evaluating a single coding model under four conditions: search-only access, global documentation, task-local context, and task-local context with independent agent review and revision. Across 32 completed runs, task-local context achieves a higher hidden-test pass rate than search-only access, although the improvement is concentrated in a single task among those shared by both conditions. The results also show no observed incremental benefit from independent agentic review, while limited sample size and incomplete experimental design prevent definitive conclusions about the effectiveness of these interventions.

Critical strengths
  1. Meaningful Research Topic. This paper investigates a meaningful and practically relevant research question: whether explicitly providing coding agents with product and architectural decisions, which may not be directly available from repository code or task instructions, can improve their performance on software development tasks.

  2. Comparative Experimental Design. The paper establishes four experimental settings to compare coding agents under different levels of contextual information and workflow support. It also evaluates the resulting code changes using hidden functional tests and deterministic rule-based checks covering behavioral, architectural, security, and scope constraints, providing multiple perspectives on agent performance.

Critical weaknesses
  1. Insufficient and incomplete experimental evidence. The evaluation covers only 8 tasks and 1 model, with just 32 of the planned 96 runs completed. Consequently, the experimental conditions contain different task mixtures and unequal repetitions, with some task–condition combinations entirely missing. The aggregate success-rate differences therefore conflate the context intervention with task composition and cannot establish a reliable comparative advantage. The overlap analysis provides limited additional support: LocalContext outperforms Search-only on only 1 of 4 shared tasks, and removing that task eliminates the observed advantage. Moreover, execution was terminated after interim outcomes had been inspected. Although the paper explicitly acknowledges these limitations and frames its findings as exploratory, the available evidence remains insufficient to establish the effectiveness or generality of the proposed intervention. A completed, balanced evaluation across substantially more tasks and multiple models is needed.

  2. Insufficient justification of how the evaluation captures the research objectives. The paper motivates its study through a distinction between functional correctness and adherence to product intent, architectural decisions, and scope constraints. However, the connection between these objectives and the actual evaluation requires substantially more explanation. The paper describes hidden behavioral tests and separate deterministic rule checks, but provides insufficient detail about how these tests and rules were derived, which sources establish the intended requirements, and how representative product edge cases and architectural constraints were translated into executable checks. Concrete examples linking a requirement or architectural decision to its evaluator would help assess this correspondence. The base/reference/faulty-patch validation is a useful sanity check, but does not establish adequate coverage or rule out false positives and false negatives on other plausible patches. Furthermore, the principal results concern functional test success, while architecture and scope outcomes are reported mainly as aggregate violation counts. More direct evidence is needed to demonstrate that the evaluation reliably captures the broader project requirements motivating the paper and that the intervention improves adherence to them.

  3. Limited analysis of the observed differences. The paper reports performance across conditions but provides little analysis of the behavioral differences underlying these results. It remains unclear how task-local context changes agents’ decisions or why additional review yields no observed benefit. Comparative case studies of agent trajectories and patches, particularly for the single task driving the LocalContext advantage, would strengthen the interpretation of the findings.

  4. Presentation and organization. The paper would benefit from clearer transitions and a more coherent narrative connecting the research questions, experimental design, and findings. The current presentation occasionally feels fragmented, making the overall argument harder to follow.

Questions for authors
  1. What are the sources and construction procedures for the product and architecture documentation in Global docs and the task-local context in LocalContext? Were these materials taken from existing project artifacts, adapted from them, or newly written by the authors or annotators? For adapted or newly authored materials, how were their accuracy and fidelity to the original project requirements verified, and how was the inclusion of hindsight from reference fixes or hidden evaluations prevented? These details are essential for interpreting the core experimental comparison.

  2. Did the authors audit whether the decision information supplied in the context conditions was already explicitly or implicitly available in repository documentation, comments, or code accessible to Search-only? How much overlap exists between these sources and the supplied context? Clarifying this would help distinguish the benefit of introducing otherwise unavailable decisions from that of consolidating existing information and making it easier to discover.

Reviewer_g2F5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the importance of providing coding agents with "extra" information that is not available in the code itself, so that the agent can write code that preserves product intent and architectural decisions. Examples of such information include architectural decision records, product intent, review policy, and escalation boundaries.

The authors argue that common benchmarks like SWE-Bench focus on verifying whether an agent can complete issue-resolution tasks. However, solving such issues may satisfy a local specification while violating a broader architectural constraint or missing an unstated product edge case. The authors therefore focus on what kind of data an agent needs to produce truly reliable software.

To study this, they build an eight-task pilot in which the model, harness, and resource budget are held constant, and the only variable is four different conditions: search-only (no authored decision-context artifacts, just repository search), global documentation (architecture documentation is provided in addition to the search tools), and LocalContext and FullWorkflow, where additional context is provided.

Critical strengths
  1. The paper positions itself in an interesting area. Common SWE-Bench-derived benchmarks only evaluate the ability of models to resolve limited-scope issues and do not account for the more complex needs of real-world/enterprise software. Studying how coding agents interact with large codebases while respecting high-level architectural decisions and product constraints, when resolving limited-scope issues, is of interest as software development is increasingly automated. However, other than surfacing this problem, the paper does not produce meaningful analysis.
Critical weaknesses
  1. The abstract does not help in understanding the core idea of the paper: experimental details and results are thrown at the reader without explanation. After repeated reading, the goal of the paper is still unclear. The introduction adds only marginal clarity. Overall, the paper is not easy to read.
  2. Experimental/benchmark design: insufficient depth. The benchmark consists of only 8 tasks. Of the 96 planned runs, only 32 were completed, and the main text never explains why the experiment stopped there. Any experimental conclusion is too weak given the size of the dataset and the lack of details. In particular, the headline improvement of local context over search-only comes from a single task; when that task is removed, the difference drops to zero. Likewise, the claim of no benefit from agent review rests on three shared tasks, all tied, and the FullWorkflow condition bundles several components, so it cannot be attributed to review. Different methods are compared (search, repository-level context, agent reviewer) without details on their implementation and design. Overall, it is not clear what kind of information is provided to the agent (which should be the real core of the paper). Hence it is difficult to assess the contribution, hence findings and insights are not actionable.
  3. The evaluation is unclear. How do the authors determine whether an agent generated the right solution? The tests (the most common validation tool in the SWE-Bench/coding-agent world) are not described. Moreover, it is not clear what "behavioral" means, for instance with respect to "functional" (line 209 talks about passing behavioral checks, but the tests seem to belong to another category, functional).
  4. Dataset construction lacks detail and depth. Some tasks are taken from well-known repositories while others come from internal pipelines, and no details are provided to certify the quality of the data used.
  5. Reproducibility is limited: the experiments rely on a single proprietary model, and half of the tasks are internal and private.
Questions for authors

Suggestions for future work:

  • Can you provide more details on the inputs to the agent? Concrete examples of "Product Intent, Architecture Decisions, Task-Local Guidance, and Escalation Policy" would help, as would a demonstration of why they would be relevant to solving a task compared to just the usual issue description.
  • Can you provide more details on how the output of the agent is evaluated?
Reviewer_Do25OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether giving a coding agent explicit "repository-local decision context" (product intent, architecture decision records, task-local guidance, review policy, escalation policy) improves functional correctness relative to repository search alone or repository-wide documentation, and whether adding an independent agent reviewer with bounded revision helps further.

The study is an eight-task pilot: four public Python issues (Django, scikit-learn, Sphinx, xarray) and four private TypeScript tasks from two internal services, one frozen model (gpt-5.4 through a private agent service), and four conditions (Table 1). A 96-run design (8 tasks x 4 conditions x 3 repetitions) was invalidated after seven runs by an evaluator/prompt mismatch, restarted with new run IDs, and then stopped by the author after 32 outcomes with interim results already visible (Sec. 6.3). Outcomes are hidden-test pass/fail plus deterministic architecture/security/scope rule counts.

Reported results are raw pass rates of 2/5 (search-only), 6/11 (global docs), 7/11 (local context) and 3/5 (full workflow). On the four tasks shared by search-only and local context, local context ties on three and wins on one, giving a task-macro difference of +25 points that drops to zero when that task is removed. The paper reports no inferential statistics, releases no tasks, evaluators or traces, and frames an industrial case with an observational ~5x productivity comparison as motivation only. The stated contributions are the pilot design, a model-agnostic context representation, the deviation/validity record, the 32-run descriptive evidence, and the industrial case.

Critical strengths

The evaluator design is reasonable and the deviation record is complete enough to show exactly what went wrong. Hidden behavioral tests are kept separate from deterministic architecture/security/scope rules, and every evaluator was validated against an unchanged base (must fail), the reference repair (must pass) and a deliberately faulty repair (must fail) before execution. If the planned 96-run design were completed and released, this could become a useful small benchmark for whether explicit decision context helps.

Critical weaknesses

The paper has no finding it can defend, and acknowledges as much. The main +23.6% local-context-vs-search-only difference rests on n=5 vs n=11 runs with different task mixes, one model with no temperature or seed control, repetitions clustered by task, and a stop decision taken after inspecting 26 outcomes. The Wilson intervals in Table 2 (40.0% [11.8, 76.9] vs 63.6% [35.4, 84.8]) overlap almost entirely, and the leave-one-task-out check in Sec. 7 shows the entire effect comes from one task. RQ2-RQ4 are not answered at all: tokens and cost were not measured, wall-clock differences are confounded by task mix, and the staleness/contradiction ablations were never run.

Four of the eight tasks are private, and prompts, evaluators, context artifacts, run summaries and traces live in an internal evidence package; "No clean-room reproduction of an anonymous package has yet been completed". The representation is five artifact categories with no example file, no token counts, and no matching of context volume across conditions, so representation and quantity are confounded. The paper also does not say how the task-local guidance was authored relative to the hidden tests and reference patch.

Sec. 8 spends nearly a page on an industrial case whose ~5x output comparison the authors disclaim as observational, bundled and non-randomized; it adds nothing to the scientific content.

The supplement material is just the latex repository used to generate the paper and provides no additional artifacts.

Questions for authors
  1. Please provide the full 32-run table (task, condition, pass/fail, violation counts, wall-clock). Which task produced the local-context gain, and what in its context artifacts explains it?
  2. How were the task-local guidance and product-intent artifacts written for each task?
  3. Why was execution stopped at 32 of 96 runs after viewing results (cost, time, something else)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.