acceptodds
Under review as a conference paper at ICLR 2027

Epistemic Debt in Coding Agents: Repaying Unsupported Assumptions in Repository-Level Software Engineering

Abstract

Repository-level coding agents act on provisional judgments while their evidence about a codebase is still incomplete. A wrong assumption about a call chain, a specification, or a test can silently become the premise of later edits, tests, and memory; every dependent action makes correcting it costlier. We call this accumulating exposure epistemic debt: the expected downstream correction cost of deferring verification of an action-dependent repository assumption. Existing agents verify reactively after a failure or indiscriminately before every write; neither selects which assumption is worth checking. We treat verification as a budgeted selection problem: a dynamic assumption graph links each falsifiable assumption to its evidence and dependent actions, and a budgeted controller runs the assumption–check pair whose expected avoided rework exceeds its cost. A paired counterfactual protocol measures that value by branching from the same checkpoint with and without a targeted check. On SWE-bench Lite (300 instances, three seeds), the debt-aware agent raises resolution from 31.9% to 48.8% with fewer agent steps and lower dollar cost. The gain replicates across three agent scaffolds and two backbones. The exposure-aware heuristic already provides a substantial improvement, while a learned scorer adds a further 4.5 percentage points on the full benchmark as an exploratory in-sample estimate; its held-out end-to-end transfer remains a limitation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, LLM agents, epistemic debt, repository-level software engineering, verification, value of information, budgeted decision-making, SWE-bench

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ybqYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper defines “epistemic debt” as the expected downstream correction cost arising from unverified assumptions in coding agents. It uses a dynamic assumption-dependency graph and paired counterfactual continuations to estimate verification value and select checks under a limited budget. Across 300 instances and three random seeds, the reported resolution rate increases from 31.9% to 48.8%, while reducing agent steps and API costs. Randomized controls matched on check count and cost support the effectiveness of selective verification, although the learned scorer’s additional gains primarily come from in-sample evaluation.

Critical strengths

1.The paper shifts verification priorities from general uncertainty to avoidable downstream rework, providing a practically meaningful and actionable principle for allocating coding-agent verification budgets.

2.It reports substantial resolution gains while reducing costs under a fixed scaffold and backbone. Controls matched on check count and expenditure strengthen the empirical evidence.

Critical weaknesses

1.The independent contribution of the core mechanism remains unclear. The randomized controls establish that check selection matters, but ablations removing the dependency graph or replacing exposure are missing. Consequently, the gains cannot be clearly attributed to the proposed graph-based mechanism. The learned scorer also does not significantly outperform the heuristic on unseen repositories.

2.The evaluation split requires clarification. The 300-instance set includes pydicom, a repository from the original SWE-bench development split. However, the source splits, sampling procedure, and overlap with scorer-training instances are insufficiently specified, making evaluation independence difficult to assess.

Questions for authors

1.With candidate checks and budgets held fixed, does removing the dependency graph or replacing exposure with trajectory length yield comparable gains?

2.Please clarify the source splits of the 300 evaluation instances and their overlap with instances used for the pilot, prompt development, and scorer training.

Reviewer_ZTFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines epistemic debt as the expected downstream correction cost incurred when a coding agent defers verification of a repository assumption. The system extracts falsifiable assumptions on which observable actions already depend, constructs an assumption–evidence–action graph, generates checks, and selects assumption–check pairs within a budget according to their expected value in avoiding rework. Value estimation uses both a fixed-factor heuristic and a gradient-boosted regressor trained on paired branch data.

Critical strengths

None

Critical weaknesses
  1. The system comparisons have not isolated the core dependency exposure mechanism, but comparisons with uncertainty ranking and the state-level gate simultaneously change settings.

  2. Inconsistent units in value scoring and budget accounting prevent exact reproduction of the main interpretable baseline and budget conclusions from the manuscript.

  3. The first Figure is AI-generated and needs to be revised.

Questions for authors

See above

Reviewer_2zMcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper treats verification in repository-level coding agents as a budgeted choice over assumption–check pairs. After each agent step, an LLM extractor updates a graph of assumptions, evidence, and dependent actions. A controller runs the check with the highest expected avoided rework. The value comes either from a fixed heuristic or from a gradient-boosted scorer trained on paired forks. On a 300-instance SWE-bench subset with mini-swe-agent and deepseek-chat, resolution rises from 31.9% (no verification) to 44.3% (heuristic) and 48.8% (learned scorer), with fewer steps and lower reported cost.

Critical strengths

The framing is clear and useful. It separates how uncertain an assumption is from how much work depends on it, and the decision unit says both what to check and how.

The controls are well chosen. Random checks matched on check count (35.6%) or on check spend (34.8%) stay well below the heuristic (44.3%). If the numbers hold, this rules out “more checks” as the explanation.

The paper is open about several limits: the in-sample scorer gain, the small fork audit, and the partly mechanical correlation in Fig. 8.

Critical weaknesses
  1. The benchmark cannot be identified, and its stated composition does not match SWE-bench. The paper does not say how the 300 instances were chosen and does not list them. The per-repository counts (Fig. 3a, Table 10) include 18 Flask and 17 pydicom instances. In the public SWE-bench release, the test split has only 11 Flask tasks, the dev and train splits have none, and pydicom appears only in the dev split. The Table 10 percentages confirm n = 18 for Flask (31.5% = 17/54). In addition, the PDF of Fig. 3a contains the text “SWE-bench Lite · pooled over three seeds” under a filled box that now reads “a 300-instance subset of SWE-bench”. SWE-bench Lite also has 300 instances from 12 repositories, but it has only 3 Flask tasks and no pydicom.

  2. Several results are hard to reconcile, and none can be checked now.

  • The debt-aware agent uses only 45% of B on average and stops when no pair has positive value. Yet Fig. 3b shows −5.2 pp at 0.5$×$ and +5.8 pp at 4$×$, while Table 4 calls the results “stable” across these budgets.
  • Table 17 adjusts the truncated fork labels, yet end-to-end resolution changes. That would require retraining and rerunning, which the paper does not describe.
  • App. D says code and data “are released with the camera-ready artifacts”, so reviewers have no access, even though the paper cites many specific files (e.g., checks.py, code/audit_numbers.py, full_run_data/meta.json) as provenance.
  1. The cost numbers look derived from step counts rather than measured.
  • Across the ten configurations in Table 1 and Fig. 7, cost per step is nearly constant at about $0.0026 (e.g., 0.072/27.2, 0.087/33.2, 0.059/22.4).
  • Table 11 implies about 0.00075 per check for every policy. This holds even though the policies use different check mixes, and the pilot forks report 0.006–0.007 per check.

4 The evidence for scorer transfer and for the mechanism is weak.

  • On held-out repositories the learned scorer improves Spearman correlation (0.066 → 0.328), but not top-10% realized value (0.028 vs. 0.035 for the heuristic) or end-to-end resolution.
  • The two repositories with the weakest gains, SymPy (+4.8 pp) and xarray (−3.9 pp), are exactly the two held out from scorer training. The paper instead attributes this to “symbolic reasoning” and points to App. C.12, which contains no such analysis.

Minor

  • Fig. 2 draws edges outside the paper’s own edge types (h₂→h* and a₅→e₃ as “contradicts”, plus edges between actions).
  • Fig. 5 shows a schematic curve (y-axis “none/high”) rather than measured data. Since H01 has one dependent action (Fig. 4), exposure under Eq. 3 would be a single step, not a compounding curve.
  • Fig. 7 and Table 11 omit the selection-only and equal-cost controls, and several labels in Fig. 7 overlap.
  • The check types in Fig. 8 (call chain, read impl, branch compare) differ from the three types defined in Sec. 3.2.
  • Broken references: “App. 4” (Sec. 4.1, App. C.4), and the “truncation rates of Sec. 4.1” cited by Table 17 are never given.
  • The extractor’s precision and recall (0.86 / 0.79) are based on single annotation of 28 trajectories. Missing one high-impact implicit assumption may matter more than these averages suggest.
  • Propositions 1–4 follow almost directly from the definitions.
Questions for authors

Please list the 300 instance IDs and the sampling procedure, and explain the Flask and pydicom counts and the “SWE-bench Lite” text in Fig. 3a.

Please give a per-instance cost breakdown: agent calls, extractor calls etc. Does “Mean cost” in Table 1 include the extractor?

What fraction of instances hit the budget cap at 1×? Why does more budget help so much when average use is 45%?

How does the policy handle an assumption that has low exposure now but is about to support extensive edits?

1 comment

The supplementary files do not match the paper, including the stats and per-repo gain calculations. I will keep my rating until these issues are resolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.