- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Graph2Env, a method for automatically constructing executable environments for code repositories based on an explicit typed dependency graph (DepGraph), aiming to address the problem in existing LLM agent approaches (Repo2Run, RAT, SetupX) where implicit interaction-trajectory-based state maintenance causes early evidence to be "diluted" as trajectories grow, leading to repeated repair loops. DepGraph explicitly maintains typed nodes such as TARGET/IMPORT/PACKAGE/RUNTIME/TOOL/SYSTEMLIB/SERVICE/CONFIG along with their dependency relationships and satisfaction states. The paper also contributes a new 200-repository Python benchmark, designing a rigorous "Gold Test Count certification" protocol using Claude Opus 5 as an independent certification agent making 3 attempts to prevent methods with incomplete environments from scoring artificially high on smaller test sets. In the main experiments, all evaluated agent systems uniformly use DeepSeek-V4-Flash-0731 as the LLM backbone. Results show Graph2Env leads the strongest baseline by 9.5/9.0 percentage points with EBSR 81.0%/ESSR 59.3%.
Critical strengths
This method proposes a four-stage workflow is: initial graph construction (evidence collection + dependency resolution + lightweight probing) → graph-driven compilation into a replayable build script → graph-guided execution repair (ReAct-style, persisting successful fixes to the graph rather than to an ephemeral container) → mandatory clean replay verification. In addition, the Gold Test Count certification protocol is a rigorous, actionable methodological contribution with reference value for subsequent benchmark design in this domain. The difficulty-stratified results are highly consistent with the core hypothesis, providing more persuasive evidence than the overall leading numbers alone.
Critical weaknesses
-
The paper needs to improve its ablation experiments. The existing experiments fail to break down the marginal contributions of each design choice—persistent graph refinement, candidate branch management, and clean replay for mandatory verification—weakening the experimental support for the core claim.
-
The significant 21.7 percentage point gap between EBSR and ESSR exposes the limitations of the method in its core selling point of "potential requirement discovery." The paper's core motivation is that "some requirements (such as services and runtime configurations) only become apparent at runtime," but even in an environment successfully built with Graph2Env, 55.6% of test failures stemmed from OperationalError and 21.9% from RuntimeError. This indicates that the method's actual ability to handle such requirements remains limited.
Questions for authors
-
Could a more direct ablation experiment be designed (e.g., freezing DepGraph after initial construction with no further runtime updates) to directly separate the respective contributions of "initial structured information" versus "continuous refinement mechanism"?
-
The gold test count is certified by Claude Opus 5, a model far stronger than the systems being evaluated—does this set a systematically high, practically unattainable theoretical ceiling for ESSR across all methods? Could the distribution of each method's actual collected test count relative to the gold count be reported?
-
Could specific data be provided on Graph2Env's improvement relative to other baselines on OperationalError/RuntimeError—these two categories of service/runtime configuration errors—to confirm that the method indeed handles the "latent requirements" problem defined by the paper itself better than the control group?
Please engage in the discussion
Dear Reviewers,
Thank you for your thoughtful reviews. The authors have posted their rebuttal, please read it and share any updated assessments.
As you know, we have only 3 reviews in total. Each piece of your opinions is valuable and may have crucial impact on the final decison. So please take it seriously.
Thanks again for your contribution!
Best Regards,
Your AC