- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ORTG, a repository context interface for coding agents. ORTG stores semantic descriptions of individual files together with a bounded number of relation targets, organizes these entries by module, and provides either complete outlines or hierarchical access depending on the context budget. The editing model also updates the corresponding descriptions and relation fields after modifying code. Experiments on SWE-bench Verified, Deep-SWE, and FeatureBench report higher mean task success rates than the corresponding baseline agents. Additional experiments examine relation removal, randomized relation targets, different relation caps, and flat versus hierarchical delivery. Behavioral measurements show fewer interactions and file reads, accompanied by increased cumulative token usage.
Critical strengths
1.The reported improvements are practically meaningful and extend across several evaluation settings. Table 2 reports improvements from 90.5% to 95.6% on SWE-bench Verified, from 46.1% to 52.3% on Deep-SWE, and from 25.2% to 41.0% for Codex on FeatureBench. The comparison with Codex + RPG-Encoder, which achieves 29.8%, is particularly relevant. These results make the proposed interface worth investigating beyond a single illustrative task.
2.The component experiments and cost reporting provide useful diagnostic evidence.Table 3 separates the effects of removing relation fields, corrupting their targets, and changing delivery policies. The entities-only condition reaches 36.4%, compared with 25.2% for Codex and 41.0% for complete ORTG, suggesting that semantic context contributes substantially. The paper also explicitly reports the 26% increase in cumulative tokens and correctly avoids interpreting fewer interactions as demonstrated reductions in monetary cost or latency.
Critical weaknesses
1.All reported ablations retain the update workflow. Without comparing against the same initial index with updates disabled, the experiments cannot establish whether maintaining repository context during editing contributes beyond providing useful initial descriptions.
2.The flat delivery baseline uses fixed ordering and truncation, potentially excluding relevant entries. Its lower performance therefore does not isolate the benefit of hierarchical organization from differences in information coverage.
3.Appendix C acknowledges that outline accuracy, relation precision and recall, and reuse across successive tasks are not evaluated. Whether repeated updates preserve semantic correctness or accumulate errors consequently remains unclear.
4.Appendix B does not specify the exact Deep-SWE and FeatureBench versions, splits, instance counts, or numerical execution budgets. FeatureBench has multiple splits and revised task statements, making these omissions consequential for reproducibility. The RPG-Encoder integration also requires clearer documentation. GitHub
5.The confidence interval calculation is unspecified, and paired uncertainty estimates are absent. Smaller differences, such as 95.6% versus 94.1% on SWE-bench Verified, remain difficult to assess beyond their reported means.
6.Table 4 reports 26% more cumulative tokens despite fewer interactions. Although acknowledged, the absence of latency, billed cost, and separate initialization and update measurements leaves the practical balance between performance gains and resource requirements unresolved.
Questions for authors
-
Which exact dataset versions, splits, and instance counts underlie the Deep-SWE and FeatureBench results? Please identify the corresponding agent configurations, numerical execution budgets, evaluator versions, and configuration used to integrate RPG-Encoder.
-
Do you have a comparison between complete ORTG and the same initial outlines and relation fields with index updates disabled, while retaining equivalent source access and inspection instructions? Such a comparison would materially affect my assessment of the distinctive contribution.
-
How often does each benchmark use complete-outline delivery versus hierarchical access? Under the flat delivery baseline, what fraction of entries is truncated, and can the observed performance difference be separated from differences in accessible information?
-
How are the confidence intervals computed? Can you provide paired performance differences and their uncertainty for complete ORTG versus the closest graph baseline and the single-level delivery condition?
Dear reviewers,
The authors have submitted their rebuttals. Please take a look and interact with the authors if necessary.
Thanks a lot.
AC