acceptodds
Under review as a conference paper at ICLR 2027

Graph2Env: Stateful Construction of Repository Execution Environments

Abstract

Coding agents now increasingly rely on execution to validate their solutions, making the construction of reliable execution environments a critical enabling capability. However, repository environment construction is challenging because execution requirements are fragmented across repository artifacts and may only become apparent during execution. Existing agent-based approaches address this problem through iterative interaction, but information about the current construction state, including discovered requirements, satisfied and unresolved prerequisites, and their dependencies, can remain distributed across the interaction history. We present Graph2Env, an agent-based approach centered on DepGraph, a typed dependency graph that explicitly represents the environment requirements needed for repository execution, their dependency relations, and their states. Graph2Env uses DepGraph to guide environment construction and continuously refines it with execution feedback, while persisting successful repairs into a replayable construction procedure. The resulting artifacts are then applied in a fresh environment to verify that the constructed environment can be reproduced. We evaluate Graph2Env on a benchmark of 200 Python repositories drawn from RATBench and EnvBench, against a static dependency-inference baseline (pipreqs), three specialized environment-construction systems (Repo2Run, RAT, and SetupX), and two general-purpose coding agents (SWE-agent and Claude Code). Graph2Env achieves an 81.0% Environment Build Success Rate (EBSR) and a 59.3% Environment Setup Success Rate (ESSR), outperforming the strongest baseline by 9.5 and 9.0 percentage points, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, repository environment construction, software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mL2qOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Graph2Env, a method for automatically constructing executable environments for code repositories based on an explicit typed dependency graph (DepGraph), aiming to address the problem in existing LLM agent approaches (Repo2Run, RAT, SetupX) where implicit interaction-trajectory-based state maintenance causes early evidence to be "diluted" as trajectories grow, leading to repeated repair loops. DepGraph explicitly maintains typed nodes such as TARGET/IMPORT/PACKAGE/RUNTIME/TOOL/SYSTEMLIB/SERVICE/CONFIG along with their dependency relationships and satisfaction states. The paper also contributes a new 200-repository Python benchmark, designing a rigorous "Gold Test Count certification" protocol using Claude Opus 5 as an independent certification agent making 3 attempts to prevent methods with incomplete environments from scoring artificially high on smaller test sets. In the main experiments, all evaluated agent systems uniformly use DeepSeek-V4-Flash-0731 as the LLM backbone. Results show Graph2Env leads the strongest baseline by 9.5/9.0 percentage points with EBSR 81.0%/ESSR 59.3%.

Critical strengths

This method proposes a four-stage workflow is: initial graph construction (evidence collection + dependency resolution + lightweight probing) → graph-driven compilation into a replayable build script → graph-guided execution repair (ReAct-style, persisting successful fixes to the graph rather than to an ephemeral container) → mandatory clean replay verification. In addition, the Gold Test Count certification protocol is a rigorous, actionable methodological contribution with reference value for subsequent benchmark design in this domain. The difficulty-stratified results are highly consistent with the core hypothesis, providing more persuasive evidence than the overall leading numbers alone.

Critical weaknesses
  1. The paper needs to improve its ablation experiments. The existing experiments fail to break down the marginal contributions of each design choice—persistent graph refinement, candidate branch management, and clean replay for mandatory verification—weakening the experimental support for the core claim.

  2. The significant 21.7 percentage point gap between EBSR and ESSR exposes the limitations of the method in its core selling point of "potential requirement discovery." The paper's core motivation is that "some requirements (such as services and runtime configurations) only become apparent at runtime," but even in an environment successfully built with Graph2Env, 55.6% of test failures stemmed from OperationalError and 21.9% from RuntimeError. This indicates that the method's actual ability to handle such requirements remains limited.

Questions for authors
  1. Could a more direct ablation experiment be designed (e.g., freezing DepGraph after initial construction with no further runtime updates) to directly separate the respective contributions of "initial structured information" versus "continuous refinement mechanism"?

  2. The gold test count is certified by Claude Opus 5, a model far stronger than the systems being evaluated—does this set a systematically high, practically unattainable theoretical ceiling for ESSR across all methods? Could the distribution of each method's actual collected test count relative to the gold count be reported?

  3. Could specific data be provided on Graph2Env's improvement relative to other baselines on OperationalError/RuntimeError—these two categories of service/runtime configuration errors—to confirm that the method indeed handles the "latent requirements" problem defined by the paper itself better than the control group?

Reviewer_B2usOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Graph2Env, an LLM-agent system for building executable environments for Python repositories. Its core is DepGraph, a typed dependency graph whose nodes are requirements such as imports, packages, runtimes, tools, system libraries, services, and configs, each annotated with a satisfaction state. The graph is compiled into a build script, refined with execution feedback during a ReAct repair loop, and verified by a clean replay in a fresh container. On a new 200-repository benchmark, Graph2Env reports 81.0% EBSR and 59.3% ESSR, outperforming the strongest baseline, Claude Code, by 9.5 and 9.0 points.

Critical strengths

The evaluation is consistent and reproducibility-oriented. All agents share the same backbone and budget, and every method is scored by rebuilding its emitted Dockerfile in a fresh container rather than by self-reported success. Requiring a clean replay before finishing nearly eliminates non-reproducible Dockerfiles: Graph2Env has 1 image build failure, against 22 for Claude Code.

Critical weaknesses
  1. Limited novelty. Typed dependency graphs whose nodes carry execution-derived states such as install and import status already exist [1], and so does iterative environment repair driven by execution feedback [2]. Neither work is cited, and Sec. 2.1 describes earlier graph-based methods as computing requirements once without using execution feedback, which does not hold for [2]. What remains is attaching an LLM agent to such a graph, which I do not find a sufficient contribution.
  2. The graph’s contribution is not demonstrated. No ablation isolates DepGraph from clean replay, the persistent build script, and upfront uv resolution, and the available evidence points to these other components. With the graph in its prompt, SWE-agent reaches only 62.0% EBSR, while Claude Code reaches 71.5% without any graph. The authors also attribute Claude Code’s build failures to its lack of a clean replay. Finally, graph updates account for under 5% of actions in successful runs.

[1] W. Cheng, X. Zhu, W. Hu. Conflict-aware Inference of Python Compatible Runtime Environments with Domain Knowledge Graph. ICSE 2022.

[2] E. Horton, C. Parnin. V2: Fast Detection of Configuration Drift in Python. ASE 2019.

Questions for authors
  1. Can you report a control that keeps the same agent, budget, dependency resolution, persistent build script, and clean-replay-before-finish, but replaces DepGraph with a flat requirement list? This would directly test whether the graph structure matters.
  2. In the ESSR evaluation, is Graph2Env’s separately recorded runtime configuration, such as started services and environment variables, applied in the evaluation container, while baselines can only rely on their Dockerfile? If so, how does this affect the ESSR comparison?
Reviewer_cdhDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses automated repository environment construction for Python repositories. The authors argue that existing agent-based methods keep construction state implicit in the interaction trajectory, so agents lose track of what has been discovered, satisfied, or remains unresolved as trajectories grow. Graph2Env instead maintains an explicit Dependency Graph (DepGraph): typed nodes cover imports, packages, runtimes, tools, system libraries, services, and configuration, with prerequisite and mapping edges and per-node satisfaction states. The graph is built from repository evidence, package resolution, and lightweight probing, compiled into a dependency-ordered build script, refined during a repair loop using execution feedback, and finally verified by replay from a clean base. On a 200-repository benchmark drawn from RATBench and EnvBench, Graph2Env reaches 81.0 percent EBSR and 59.3 percent ESSR, versus 71.5 and 50.3 for Claude Code and 52.5 and 45.9 for SetupX. Giving the initial graph and script to SWE-agent raises its EBSR from 50.0 to 62.0.

Critical strengths
  1. The problem framing is clear and the motivating example concretely shows trajectory-based state loss.
  2. DepGraph is well designed. Separating imports from installable distributions and distinguishing prerequisite from mapping edges captures a real source of Python setup errors, and the typed taxonomy covers services and configuration that only fail at test execution.
  3. The compile-and-replay design is the paper's strongest contribution. Accepting an environment only after clean replay in a fresh container directly addresses the gap between a container that happens to work and one that is reproducible. This is supported by the failure analysis: one build failure for Graph2Env versus 22 for Claude Code.
  4. The benchmark construction is careful. Certifying gold test counts under integrity checks, requiring stable collection, and preventing evaluation against a smaller self-induced test set are all appropriate controls.
  5. The SWE-agent transfer experiment is a reasonable substitute for an internal ablation, since DepGraph drives every stage of Graph2Env, and the 12-point gain suggests the representation is informative on its own.
  6. The failure analysis is detailed and identifies runtime services and configuration as the dominant remaining failure mode.
Critical weaknesses
  1. Single backbone, single ecosystem. All arms use DeepSeek-V4-Flash and all repositories are Python. The claim that explicit state is more reliable may interact with backbone strength, and no second backbone is tested.
  2. The contribution of continuous graph refinement is not isolated. The SWE-agent experiment tests the initial graph and script, but not the refinement loop, which the authors identify as central. A frozen-graph variant would settle this.
  3. The benchmark is selectively constructed. Screening for repositories a strong agent cannot configure within ten steps focuses on hard cases, which is defensible, but the reported rates are not representative, and the relationship among screening agent, certification agent, and evaluated systems is unclear.
  4. No uncertainty estimates. With 200 repositories, several reported margins correspond to a handful of repositories, particularly in domain breakdowns where some domains have only four.
Questions for authors
  1. Can you add a second backbone, one weaker and one stronger, to test whether the benefit of explicit state depends on model strength?
  2. Do EBSR and ESSR use the same collection protocol? If not, what is the effect on the reported numbers?
  3. Can you report bootstrap confidence intervals for the main EBSR and ESSR figures and the domain breakdown?

Comments on the paper

1 comment
Area_Chair_izusOpenReview ↗

Please engage in the discussion

Dear Reviewers,

Thank you for your thoughtful reviews. The authors have posted their rebuttal, please read it and share any updated assessments.

As you know, we have only 3 reviews in total. Each piece of your opinions is valuable and may have crucial impact on the final decison. So please take it seriously.

Thanks again for your contribution!

Best Regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.