- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper defines “epistemic debt” as the expected downstream correction cost arising from unverified assumptions in coding agents. It uses a dynamic assumption-dependency graph and paired counterfactual continuations to estimate verification value and select checks under a limited budget. Across 300 instances and three random seeds, the reported resolution rate increases from 31.9% to 48.8%, while reducing agent steps and API costs. Randomized controls matched on check count and cost support the effectiveness of selective verification, although the learned scorer’s additional gains primarily come from in-sample evaluation.
Critical strengths
1.The paper shifts verification priorities from general uncertainty to avoidable downstream rework, providing a practically meaningful and actionable principle for allocating coding-agent verification budgets.
2.It reports substantial resolution gains while reducing costs under a fixed scaffold and backbone. Controls matched on check count and expenditure strengthen the empirical evidence.
Critical weaknesses
1.The independent contribution of the core mechanism remains unclear. The randomized controls establish that check selection matters, but ablations removing the dependency graph or replacing exposure are missing. Consequently, the gains cannot be clearly attributed to the proposed graph-based mechanism. The learned scorer also does not significantly outperform the heuristic on unseen repositories.
2.The evaluation split requires clarification. The 300-instance set includes pydicom, a repository from the original SWE-bench development split. However, the source splits, sampling procedure, and overlap with scorer-training instances are insufficiently specified, making evaluation independence difficult to assess.
Questions for authors
1.With candidate checks and budgets held fixed, does removing the dependency graph or replacing exposure with trajectory length yield comparable gains?
2.Please clarify the source splits of the 300 evaluation instances and their overlap with instances used for the pilot, prompt development, and scorer training.
The supplementary files do not match the paper, including the stats and per-repo gain calculations. I will keep my rating until these issues are resolved.