- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ArchRSBench evaluates repository-level architectural planning as DAG generation, covering subtask semantics, target file mapping, execution order, and granularity. It contains 1,525 instances from 13 repositories, with ground truth DAGs reconstructed from historical pull requests. Validation includes deterministic metrics, LLM-as-a-Judge evaluation, five model baselines, a Tri-Agent pipeline, and context representation ablations.
Critical strengths
-
An interesting evaluation target. Explicitly evaluating architectural decomposition addresses a meaningful aspect of software development that final code correctness alone does not capture.
-
Useful data curation infrastructure. The collection of real pull requests across multiple ecosystems, together with documented curation prompts and evaluation schemas, provides a potentially valuable foundation for further research.
Critical weaknesses
- The reliability of the ground truth DAGs is insufficiently established.
- Historical commit boundaries do not necessarily correspond to appropriate architectural subtasks.
- Hunk overlap and historical co-change frequencies do not necessarily establish required execution dependencies. Assigning direction according to commit chronology does not resolve this ambiguity.
- The annotation unit and coverage of missing dependencies remain unclear in Appendix D.1, leaving the validity of the benchmark targets uncertain.
- The baseline evaluation does not adequately support the broader architectural reasoning claims.
- The Single-Call and Tri-Agent baselines use supplied candidate paths obtained through BM25 retrieval and golden file injection. They do not evaluate autonomous repository exploration, code inspection, or iterative tool use.
- The Full Content ablation does not specify whether it includes the same candidate set, potentially containing 150 or more files, or report input lengths, token budgets, and truncation policies.
So the evaluation does not establish performance under an interactive coding agent paradigm or isolate the effect of code content from context length and generation failures.
Questions for authors
-
Does the confusion matrix in Appendix D.1 count PR pruning decisions, commit pruning decisions, or individual dependency edges?
-
Did the human audit independently assess dependency necessity, direction, and missing edges?
-
Please clarify file counts, input token distributions, input and output limits, truncation policies, and how failed or incomplete outputs were scored. Actually, it needs coding agent baselines.