acceptodds
Under review as a conference paper at ICLR 2027

Monitoring Is Not Self-Improvement: An Evidence Ladder for Agent Harnesses

Abstract

Agent harnesses increasingly monitor execution traces, intervene during a run, and revise prompts, tools, or policies from experience. Evidence that any one of these mechanisms operates, however, does not establish that the agent has improved. We introduce an evidence ladder that separates three claims: measurement validity, runtime intervention efficacy, and persistent improvement on independent future tasks. Each level is expressed as an evidence contract with a distinct estimand, required observations, and invalid shortcuts. We instantiate the framework in HarnessEvolve through a task-paired study of a two-check monitoring package on all 89 Terminal-Bench 2.0 tasks, comprising 267 matched task-condition evaluations per arm. The monitored harness obtains 60/267 verified successes versus 61/267 for the static harness (difference -0.4 percentage points; 95% CI [-6.0,+5.2]), while the four-rule operational endpoint also remains near zero (29.5% versus 29.8%). Applying the ladder shows why these outcome estimates cannot be replaced by mechanism telemetry: the checks have unequal exposure, the dominant operational event is outside their target set, and deterministic trace rules remain proxies whose semantic interpretation is limited by human annotation. Exploratory update studies separately verify proposal, gating, and deployment without treating execution of that loop as evidence of future-task improvement. The result is an auditable protocol for matching self-improvement claims to the experiments needed to support them.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, agent harnesses, self-improving systems, failure detection, runtime monitoring, software engineering agents, empirical evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_S4WzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to evaluate self-improvement claims in LLM agent harnesses. The authors propose an evidence ladder that distinguishes three levels: measurement validity, runtime intervention efficacy, and persistent improvement on independent future tasks. The framework is instantiated in Harness Evolve, which introduces repeated-command warnings and verification gates to monitor agent execution.

Critical strengths
  1. This paper addresses an important problem in LLM agent evaluation by distinguishing monitoring, runtime intervention, and persistent self-improvement. The proposed evidence ladder provides a useful framework for assessing whether observed agent behaviors support claimed performance improvements.

  2. The paper employs task-paired comparisons, uncertainty estimation, and intervention-uptake analysis. The authors also report negative experimental results and explicitly distinguish mechanism execution from actual performance gains, providing useful insights into the limitations of current agent monitoring approaches.

Critical weaknesses
  1. Human annotation reveals poor agreement with the operational detectors, with the churn detector achieving only approximately 2% adjudicated precision in the annotated sample. This undermines the semantic interpretability of the primary operational metric.

  2. The monitoring mechanisms target only two of the four operational failure modes. Moreover, the repeat warning triggers in only 6/267 runs, raising concerns about intervention coverage and the interpretation of the null results.

  3. The persistent-update experiments involve small evaluation sets, while the main study uses only one base model and benchmark. These limitations weaken the evidence for generalizability and persistent improvement.

Questions for authors
  1. What methodological insights does the Evidence Ladder provide beyond existing evaluation principles?
  2. Can the authors provide component-level ablations and resource-matched controls to isolate the effects of each monitoring mechanism?
1 comment

Dear Reviewer S4Wz,

Thank you for your careful review and constructive suggestions. We appreciate your recognition of the distinction between monitoring, runtime intervention, and persistent improvement, as well as our task-paired comparisons and analysis of post-trigger behavior. We have added clarifications on measurement and controls to the working draft and plan to include them in the revised manuscript.


Our responses follow:

  1. W1: After human adjudication, 1/48 segments flagged for command-surface churn and 0/33 segments flagged for retry matched their intended failure categories. The revision will clarify that the average flag rate across the four rules measures the incidence of rule-defined events, not the rate of semantic failures.
  2. W2 & Q2: The repeat warning targets repeated command failures, a subset of what the retry rule flags, while the verification gate targets premature completion. After filtering, 6 of 267 monitored runs met the reconstructed warning condition; this is not a delivery count. Neither check directly targets overload or churn. The existing metric sensitivity analysis does not substitute for component ablations. The revision will describe the design of a four-arm comparison (static, warning only, gate only, and both) with matched tasks and budgets and separate reporting of actual token use (current means: 78.1K tokens for the monitored arm and 76.1K for the static arm).
  3. W3: The controlled comparison using glm-5.3-flash on Terminal-Bench 2.0 detected no gain in verified success: 60/267 for the monitored arm versus 61/267 for the static arm, a difference of −0.4 percentage points (95% CI [−6.0, +5.2]). Neither of the two small transfer evaluations of frozen updates showed an increase in successes on the same benchmark.
  4. Q1: The evidence ladder identifies distinct follow-up checks for the same near-zero result: low semantic precision motivates recalibrating the measure, whereas the 30 of 93 first completion declarations that fell outside the gate window motivate comparing window lengths and logging delivery. These follow-ups require different evidence, and neither alone explains the effect of the monitoring package.

Thank you again for your review and suggestions.

Reviewer_gGeDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evidence ladder for evaluating agent-harness self-improvement, separating measurement validity, runtime intervention efficacy, and persistent improvement on independent tasks. It instantiates the framework in HarnessEvolve, combining operational trace detectors, two runtime checks, and a regression-gated update loop. The main controlled study compares a static harness with one augmented by repeated-command warnings and a completion-verification gate across all 89 Terminal-Bench 2.0 tasks, totaling 534 runs. Verified success is 60/267 for the monitored harness versus 61/267 for the static harness, with no detected benefit. Additional analyses examine detector validity, intervention exposure, and post-trigger behavior. Separate exploratory update studies demonstrate execution of the update pipeline but do not establish persistent

Critical strengths
  1. The main controlled comparison covers all 89 Terminal-Bench 2.0 tasks and uses task-paired outcomes with task-clustered confidence intervals, accounting for repeated observations within tasks.

  2. The analysis reports intervention exposure and post-trigger behavior alongside verified success. This provides a useful account of whether the checks were delivered and how agents responded, while keeping these observations distinct from evidence of improved task outcomes.

  3. The authors report null results, analysis-plan deviations, and limitations in detector validity and transfer evidence candidly. They appropriately avoid interpreting the absence of a detected gain as equivalence or treating successful update deployment as proof of self-improvement.

Critical weaknesses
  1. Although the paper explains which events the repeat warning and verification gate target, the rationale for selecting these two checks and evaluating their combination is insufficiently developed.

  2. The evidence ladder largely restates standard requirements for measurement validity, controlled comparison, and independent testing. Its formulation as evidence contracts does not demonstrate a substantive methodological advance or additional diagnostic capability.

  3. The empirical finding is narrowly limited to no detected benefit from one monitoring package. The study provides little generalizable insight into when or why monitoring helps, limiting its contribution beyond a local engineering evaluation.

  4. The main controlled evaluation uses only Terminal-Bench 2.0 and one base model. The usefulness of the proposed general evaluation framework across different agent settings remains untested.

  5. The transfer experiments are underspecified (Section 4.5, Appendix F, Table 15). Although Table 15 reports disjoint evaluation tasks, task sources, selection procedures, and the separation of update proposal, regression selection, and final evaluation remain insufficiently detailed, limiting assessment of the independence and scope of the reported transfer evaluation.

Questions for authors
  1. Why were these two checks selected, and what makes their combined effect an informative research question beyond this implementation?

  2. What new analytical or diagnostic capability does the evidence ladder provide beyond established evaluation principles, and which result demonstrates that capability?

  3. What evidence supports the framework's usefulness beyond the evaluated Terminal-Bench/model configuration?

  4. Where do the 15 transfer tasks and the 30 edit/29 evaluation tasks come from? How do they relate to the 89 main-study tasks, and how were final evaluation tasks isolated from update proposal and regression selection?

1 comment

Dear Reviewer gGeD,

Thank you for reading our paper closely and raising specific questions. We appreciate your recognition of the paired comparison across all 89 tasks, the task-clustered confidence intervals, and the transparent reporting of post-trigger behavior, verified outcomes, and the limits of the evidence. We have updated the working draft in response to your concerns and plan to include these clarifications in the revised manuscript.


We address related comments together below:

  1. W1 & Q1: We chose two observable decision points where the agent can act: the repeat warning prompts diagnosis after repeated command failures, and the verification gate requests an objective check when the agent first declares completion within the eligibility window. Under identical resource rules, the comparison tests whether monitoring at both points improves verified success rates; it does not isolate the effect of each check.
  2. W2 & Q2: The evidence ladder separates measurement validity, runtime intervention, and transfer of frozen updates. It helps identify what to change next rather than introducing a new statistical method. Of 93 first completion declarations, 30 fell outside the eligibility window. This points to a change that can be tested separately: compare a longer window with the current one while logging whether the gate message was delivered.
  3. W3, W4 & Q3: We detected no gain in verified success from the combined checks. This finding is limited to Terminal-Bench 2.0, glm-5.3-flash, and the harness configuration studied here. The revision will distinguish the comparison design, which can inform evaluations in other settings, from the results for this single configuration.
  4. W5 & Q4: Both transfer evaluations drew from the original 89 tasks. We selected 15 difficult tasks outside the edit set based on prior static-harness performance and operational flags. These tasks were not used for rule proposals or regression screening in that round; two had been used in the earlier mechanism study. The other study used 30 tasks of mixed difficulty for editing and 29 evaluation tasks that did not overlap with the edit set. The archived runner selects regression probes only from the edit set, but per-round probe lists are unavailable. Of the 29 evaluation tasks, one had been used for editing in an earlier study and one had served as a probe. Appendix F now documents the split procedures and task reuse.

Thank you again for your careful review and suggestions.

Reviewer_vSp6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an evidence ladder separating measurement validity, runtime intervention efficacy, and improvement on independent future tasks. A paired Terminal-Bench study finds no detected benefit from a two-check monitoring package, while small update studies illustrate why executing an adaptation loop does not establish transfer.

Critical strengths
  1. The distinction between detecting an event, changing behavior, and improving future-task outcomes is useful. The evidence contracts make the corresponding claims and required comparisons explicit.

  2. The statistical reporting is careful: the analysis clusters by task, reports uncertainty, and correctly distinguishes an undetected benefit from equivalence or established non-inferiority.

Critical weaknesses
  1. The empirical instantiation offers limited support for the broader methodological contribution. The repeat warning fires in only 6/267 runs, the verification gate excludes 30 late declarations, and the dominant composite component is untargeted. The persistent studies are also small (e.g., 4/15 versus 4/15 on disjoint transfer). The result therefore teaches relatively little beyond the behavior of this particular, weakly exposed package; the incremental insight over general evaluation principles remains limited for ICLR.

  2. The main operational endpoint lacks convincing construct validity. Appendix C reports pre-adjudication precision of 0.08 for churn and 0.03 for retry, with sparse semantic positives. Calling these reproducible trace events appropriately narrows the claim, but does not make the composite informative about meaningful failures. With raw traces withheld, the detector and uptake analyses also cannot be independently regenerated.

Questions for authors
  1. Why does the gate stop accepting completion declarations at turn 15 when runs can last 25 turns? Do existing results show whether a better-exposed intervention changes the conclusion?

  2. Can the authors provide anonymized annotation examples or sufficient event records to audit the very low detector precision and reproduce the uptake counts?

  3. What does the ladder establish beyond existing harness-aware evaluation and disclosure frameworks, such as "Stop Comparing LLM Agents Without Disclosing the Harness" (2026)? A concrete distinction would clarify the contribution.

  4. Figures and writing can be largely improved.

1 comment

Dear Reviewer vSp6,

Thank you for your careful review and specific suggestions. We appreciate your recognition of our distinction between detecting events, changing behavior, and improving outcomes on future tasks, as well as our use of task-clustered uncertainty estimates. We have updated the working draft in response to your concerns and plan to incorporate these changes into the revised manuscript.


We address your points below:

  1. W1 & Q1: The verification gate intercepts a run's first completion declaration only on turns 1–14, leaving one turn for follow-up within a 15-turn horizon separate from the overall 25-turn run limit. Of 93 first declarations in the monitored arm, 30 fell outside the gate window. The 6/267 figure counts runs with reconstructed warning opportunities under a narrower diagnostic filter, not warnings delivered. Neither check directly targets churn, one of the most common flags. The current comparison used a fixed window; the revision will explain how to compare window lengths while logging delivery. In the small frozen-transfer comparison, each arm solved 4/15 tasks, which we do not take as evidence of persistent improvement.
  2. W2 & Q2: Using the union of the two annotators' initial labels as the reference, sample precision was 0.08 for churn and 0.03 for retry. After adjudication, 1/48 churn-flagged segments and 0/33 retry-flagged segments matched the intended semantic failures. The four-rule average therefore measures the incidence of rule-defined events, not semantic failures. The appendix explains how we reconstructed trigger opportunities and which statistics can be checked against the aggregate tables. We are not providing individual annotation examples or raw event sequences with this response, so the trace-level counts cannot be independently regenerated from the tables.
  3. Q3: We now discuss the study by Zhang et al. in Related Work. Their work focuses on harness disclosure and explaining differences across systems. We apply established evaluation principles to identify what to test next for a specific harness change. For example, the 30 late declarations motivate comparing gate windows and logging delivery rather than attributing the outcome directly to the prompt content.
  4. Q4: We have redrawn all four figures, distinguished rule flags from reconstructed trigger opportunities in Figure 2, and moved the main finding for each research question to the start of its results subsection.

Thank you again for your careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.