acceptodds
Under review as a conference paper at ICLR 2027

Metrics-in-the-Loop: Cheap, Timely Course Corrections Improve Coding Agents

Abstract

In spite of their impressive recent progress, coding agents still accumulate overly complex functions and untested code over long trajectories. Nothing flags these weaknesses as they form: the only signal is a failing test, and by then they likely have compounded. We introduce metrics-in-the-loop, an inference-time intervention that gives agents in-flight feedback: cheap, automatically computed code metrics injected into the agent's context during the trajectory. After each code-changing agent turn, we insert short feedback messages naming changed functions whose cyclomatic complexity exceeds a limit (complexity feedback) or changed lines that are skipped by tests (coverage feedback), giving the agent the opportunity to refactor code or strengthen tests before continuing. On two Python benchmarks, complexity feedback consistently leads agents to reduce the concentration of codebase complexity. It does so with no detrimental impact on functional correctness. Generic reminder messages, on the other hand, have almost no effect. Our experiments investigate a practical form of agent “re-railing”, where timely, actionable measurements can effectively redirect work before local weaknesses become embedded, suggesting practical paths for coding agent to work effectively on long-term-maintainable codebases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding agents, inference-time feedback, code generation, software quality, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jb58OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds an automatic check after each code edit and tells the agent which functions have cyclomatic complexity above 6 and which changed lines were not run by tests. It is tested with DeepSeek on SlopCodeBench, showing that complexity feedback lowers compared with reminders alone. No statistically significant gain in passing hidden tests is shown by using this method, so that whether the method is really helpful remains unknown.

Critical strengths

None

Critical weaknesses
  1. The main contribution looks just like adding a skill that asks the agent to check complexity and test coverage. The paper compares with reminders alone, but not with an agent loop with a skill that tells the agent to run the same tools, so I do not see enough novelty here.
  2. Asking the agent to lower function complexity makes it split functions. On Commit0, putting the single-use helper functions back together removes most of the score gain, and the paper shows no clear gain in correctness or in the cost of later edits.
Questions for authors
  1. Would adding a skill that tells the agent to run these same tools and fix the reported issues before finishing work to popular agent loops (claude code, codex, or opencode) just as well under the same budget?
  2. Does the resulting code make later changes cheaper or less likely to break things, or is the main benefit just a lower complexity score?
Reviewer_UJtQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MITL: after each code modification by a coding agent, SlapKit measures the cyclomatic complexity of the changed functions and test coverage of added or modified lines, then inserts a report identifying over-limit functions and uncovered lines into the next turn's context.

Critical strengths

None

Critical weaknesses
  1. The control does not isolate the value of measurement. C and CT receive an explicit complexity target of six, while R receives only general advice. Without a control given the same numerical target, it is unclear how much function-specific measurements add beyond a more concrete instruction.

  2. The maintenance benefit remains untested. Undoing single-use helper extraction removes almost all of the Commit0 erosion improvement. The code audits suggest that many splits are meaningful, but they do not show whether the resulting programs are easier to modify or less prone to regressions

  3. The benefit of timely feedback is not established separately. There is no comparison with the same reports delivered during a final review pass under a matched budget. The authors acknowledge this, but timing remains central to the paper’s motivation without a direct test of its value.

Questions for authors

See above

Reviewer_ZQZNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MITL, a harness plugin that measures the functions a coding agent changed after each turn and injects a short message into the agent's context before the next model call. It measures the changed functions and puts a short message in the agent's context before the next turn: which changed functions exceed a cyclomatic complexity limit of 6, and which changed lines the agent's own tests do not run. Five arms are compared: no message, a reminder with no measurement, complexity only, coverage only, and both. Two benchmarks are used, Commit0 (rebuild a Python library, 25 libraries, official tests withheld) and SlopCodeBench (extend a program over 3 to 8 checkpoints, 35 problems), with two open models. The pre-registered primary outcome is held-out pass rate; the secondary outcome is E10, the share of complexity mass in functions with complexity above 10. Complexity feedback lowers E10 on both benchmarks and both models, and the reminder alone does so only on SlopCodeBench. No arm changes held-out correctness detectably. Coverage feedback raises coverage on SlopCodeBench only and does not lower E10.

Critical strengths
  • The control arm isolates the measurement from the instruction. The reminder arm gets the same review text on the same schedule, so the complexity-minus-reminder contrast is the effect of the numbers alone. That contrast is large and consistent: E10 falls by 0.43 to 0.53 on SlopCodeBench and 0.12 on Commit0 against the reminder, while the reminder itself moves E10 only on SlopCodeBench and by less than half as much.

  • Measures the agent never saw move the same way. On SlopCodeBench the benchmark's own erosion score, its verbosity score, and the fraction of agent code the hidden tests execute all improve under complexity feedback with intervals clear of zero (Table 10). This is the evidence that the effect is not only the agent satisfying the number it was shown, and it is why the structural result deserves belief.

Critical weaknesses
  • Adding additional metrics as feedback to the agents is not new. Feeding linter or complexity output back into an agent loop is standard practice in production harnesses, and the paper acknowledges this ("our feedback intervention can be understood as a harness change", §6; "agent interfaces already give edit-time feedback such as syntax checks", §2). The paper's primary outcome, held-out correctness, is null everywhere (§5.4). Its headline outcome, E10, is the complexity the agent was instructed to reduce.

  • The claim of "no harm to correctness" is not supported. The SlopCodeBench interval for combined feedback under DeepSeek admits a six-point drop (Table 5); the Commit0 task variance is about nine times the estimated effect (Appendix J); and the coverage arms lost three to five times more cells than the others to a harness defect (Table 12). "Does not change detectably" is accurate. "No detrimental impact" is not. Pre-registration is asserted but no document is given, and seeds four and five were added to the two primary arms after a first reading.

Questions for authors
  • What outcome, other than the complexity the agent was told to reduce, shows the restructured code is better? If none was measured, what would you expect a maintenance experiment on frozen trees to show?
  • Where is the pre-registration document, and was E10 pre-specified as the headline, or chosen after the primary contrast came out null?
  • What drop in held-out pass rate would count as harm, and what is the largest drop the design rules out per benchmark and model?
  • How many code-changing turns went unmeasured because the code did not parse, and how many delivered an empty table, per arm? If harder code is less measurable, the missingness is not at random.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.