acceptodds
Under review as a conference paper at ICLR 2027

Beyond Failed Trajectories: Diagnosing and Repairing LLM Agent Harnesses Across Three Observability Levels

Abstract

Failure-driven debugging of LLM-agent harnesses implicitly assumes that the execution selected for diagnosis already contains evidence of the defect. We show that successful executions violate this assumption in two distinct ways. In L2, the task succeeds but the trajectory contains abnormal evidence; in L3, the ordinary execution remains healthy until another reachable condition exposes the defect. Across 9,058 harness-related issues and merged pull requests from 16 open-source agent projects, 25.1% of records with defined levels are L2 or L3, with high agreement in independent audits by two authors. A controlled study of 36 reconstructed historical defects further shows that passive source inspection can raise suspicion about latent defects without confirming their manifestation, and that source localization remains a separate bottleneck once an exposing execution is available. We introduce Proactive Invariant Harness Tomography (Proactive IHT), which audits all trajectories, routes successful runs with invariant violations to diagnosis, selectively probes healthy candidates for missing evidence, and verifies candidate mechanisms before scoped repair. Across four public benchmarks and four model backbones, Proactive IHT improves over HarnessFix in all 16 matched settings by 2.0–5.0 percentage points in task completion rate (TCR), with an average gain of 3.6 points. These results show that harness repair benefits from treating evidence acquisition as part of the debugging problem rather than using task failure as its only entry point.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, agent harnesses, harness debugging, harness repair, defect observability, runtime invariants, proactive testing, causal verification, agent reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ghfnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

< Note: Well I found the paper frustratingly difficult to follow: many concepts are insufficiently explained, and terms are used inconsistently. Within the limited time available for this review, I had to infer several intended meanings, so the summary below reflects my best interpretation of the work :) >

The paper aims to repair harness. The authors propose Proactive IHT, which probes selected apparently healthy executions to obtain additional evidence and verifies candidate mechanisms before repairing the harness. With expeirments, the authors report task completion rate improvements of 2.0–5.0% over HarnessFix across all 16 benchmark–backbone settings, averaging 3.6%.

Critical strengths
  • important problem
  • demonstrating some improvement via experiments
Critical weaknesses
  1. The contribution over the baseline (HarnessFix) looks a bit trivial

Table 3 leaves me uncertain about the contribution emphasized by the paper…The additional benefit from L2 and L3 therefore appears very small in this setting. The L1-only configuration already retains most of the observed improvement over HarnessFix.

I understand that this ablation uses one backbone and a limited activation budget. Table 19 in Appendix L.2 reports three independent test runs per variant, but I could not find confidence intervals or significance tests for these small increments. I therefore cannot tell whether the gains are statistically distinguishable from run-to-run variation… To help readers assess the contribution of L2 and L3, I would really appreciate results from independently repeated repair runs, together with uncertainty estimates and appropriate significance tests for these comparisons.

This makes the changes to diagnosis and verification particularly important. (by the way, “verification“ is typically a reserved word for formal verification in software enigneering/programming language domains) HarnessFix already has structured traces, runtime-to-source mappings, scoped repair specifications, and regression-aware acceptance. The paper acknowledges this foundation, but Section 5 does not make the changes to the diagnostic procedure sufficiently concrete. I can identify the additional evidence views and the set of competing mechanisms, but I still cannot reconstruct the changed diagnostic operations or connect them to the observed gain. I would like the claimed advance to be tied to specific changed operations and their measured effects.

  1. The evaluation does not sufficiently connect active probing to verified defects and repairs.

<Note: Again, I doubt the usage of term “verified“>

The historical-case experiments evaluate passive diagnosis or localization after a defect has already been exposed. The AppWorld analysis reports triggered anomalies, but does not clearly show how many correspond to confirmed harness defects and lead to accepted repairs. I would like to see these outcomes and one complete example from an initially healthy execution to a successful repair. A simple probing baseline under the same budget would also help establish whether the proposed risk ranking adds value.

  1. The decisions that determine routing and mechanism confirmation remain difficult to assess.

Appendix B.2 does not specify when the filter suppresses an invariant warning, although the reported L2 errors are attributed to this filter. I also could not follow how a probe distinguishes competing explanations for an anomaly. Making the symptom disappear could simply reflect a workaround. Please state the filtering rule and show what a confirmation probe changes, what it holds fixed, and how its outcome supports one mechanism over another.

  1. The opening argument conflates two different limitations.

The abstract and introduction state that L2 and L3 both violate the assumption that the execution selected for diagnosis already contains defect evidence. L2, as defined in the paper, does contain such evidence. It challenges the selection of only failed executions, whereas L3 concerns the absence of a behavioral witness in the current execution. HarnessFix explicitly studies failed trajectories; this scope does not imply that successful executions cannot contain defects. The paper has a reasonable extension to motivate, but the current argument does not establish it correctly. This issue is secondary to the evaluation concerns above, though it contributes to the uncertainty about the central claim.

Questions for authors
  1. is the improvement over baseline statistically representative?
7 comments
AuthorsOpenReview ↗

Proactive IHT goes beyond failure-driven debugging to uncover and repair harness defects missed by task-success metrics.

Thank you for the careful critique and for acknowledging both the importance of harness reliability and the consistent improvements over HarnessFix. Your questions helped us sharpen the distinction between improving diagnosis given a witness and obtaining a witness in the first place. We have revised the framing and added executable examples and focused validation.

W1 — What is the substantive advance over HarnessFix? Our contribution has three connected parts. First, an empirical study of 9,045 labeled harness development records across 16 projects identifies two ways failure-gated debugging misses evidence: L2 runs succeed but already contain an actionable trace anomaly (selection gap); L3 ordinary runs are healthy and require a changed execution condition to expose the defect (evidence gap). Second, the 36-case executable study separates obtaining behavioral evidence from locating the responsible source mechanism. Third, Proactive IHT brings these decisions into a common source-grounded, multi-view, intervention-supported repair loop. Across four benchmarks and four backbones, the complete method outperforms HarnessFix in 16/16 matched settings, by 2.0–5.0 TCR points (3.6 on average).

The Qwen ablation also helps identify where this benefit arises. L1-only IHT is not HarnessFix: it retains the new multi-view evidence and mechanism-assessment operations, so its 3.42-point gain over HF reflects a shared diagnostic contribution even before successful-run routing is enabled. L2/L3 then extend the set of executions and conditions from which this workflow can collect evidence. Their additional +0.86/+0.32 task-completion points under a two-candidate activation cap are modest; they should not be taken as a complete measure of latent harness defects, which may not change an ordinary benchmark outcome. Removing multi-view evidence or mechanism assessment from the full method reduces average TCR by 3.5 and 2.2 points, respectively (separate conditional ablations, not additive effects).

W2 — Where does probing connect to a real mechanism? We added two source-grounded contrasts. In a CAMEL L3 case, ordinary conversations complete, yet switching off the optional tracing integration makes the per-agent session accessor return None; two agents can then share the __default__ response-chain key. The repaired accessor preserves agent-a and agent-b. In an AppWorld ReAct L2 case, a successful code-producing response carries an orphan reasoning marker into the assembled assistant-visible content reused by the next model call. The repair removes the marker without changing the extracted executable code. The supplied case records include benign, triggering, and patched observations, source changes, and passing targeted checks. A broader AppWorld source-level audit also connects the two retained L2 and 38 L3 activation observations to nine distinct edits, with 27/27 targeted buggy/patched/merged checks passing; see Follow-up 1 and the new appendix table. These are specific harness obligations that task success alone does not test; Follow-up 1 gives the operation-level comparison and validation sequence.

W3 — Routing and mechanism confirmation. We now make the actual RQ2 six-rule filter explicit and distinguish that frozen filter from harness-identity gating of code-derived rules in the corrected RQ4 policy. The original AppWorld diagnostic audit inadvertently applied a Codex-specific keyword rule beyond its scope. Reviewing the 22 original L2-like positives removes 78 of 80 heuristic firings, leaving two assistant-message boundary candidates. More importantly, our method does not equate a warning with a repair: it proposes a mechanism, changes a specified condition, checks the predicted symptom, and tests the patched/benign executions. This is operational intervention-supported confirmation, not a claim of formal proof.

W4 — Motivation. We agree with your useful distinction: L2 already has trace evidence but is skipped by failure-only selection; L3 lacks a witness in the ordinary run. The revised abstract and introduction separate these gaps explicitly, without suggesting HarnessFix claims successful executions cannot contain defects.

The central point is that task completion and harness correctness are complementary evaluation targets. The revision makes the evidence-acquisition contribution concrete while preserving the original controlled TCR results.

HarnessFix already provides structured trajectories, source-aware diagnosis, scoped patches, and regression checks; we build directly on that foundation. Our added operations address a different decision point:

StepFailure-trajectory workflowProactive IHT addition
Select executionsEnter diagnosis from task failuresAudit successful runs for L2 contract evidence
Acquire missing evidenceUse the current failure witnessChoose a reachable condition for an apparently healthy L3 run
Compare explanationsSource-grounded hypothesesCombine trace, invariant, activation, and source views; test predictions under controlled changes
Accept a repairScoped change and regressionsAlso check the targeted manifestation and an unaffected benign condition

Concrete mechanism-level contrast. In the CAMEL example, the benign execution alone does not expose the shared session-key fallback. The changed condition (tracing integration disabled while two agent sessions are active) exposes a state-isolation violation. The source-level intervention fixes access to the ContextVar regardless of tracing availability, and the control preserves separate agent IDs. In the AppWorld example, the task code still executes successfully while the assembled assistant-message channel contains an internal-reasoning marker; the boundary repair removes the marker and leaves the code unchanged. These are meaningful harness correctness outcomes, not merely an alternative prompt. A broader AppWorld audit identifies nine separate source-level corrections linked to two retained L2 message-boundary observations and 38 activated L3 records: one ReAct message edit, one task-input edit, and seven trace-interface edits. We supply all nine unified diffs and a record-to-edit ledger. The 27/27 expected checks hold on targeted buggy, patched, and merged code, including selected benign controls. The counts refer to 40 evidence observations and nine distinct corrected source contracts, not 40 independent defect repairs. In particular, trace-interface fixes are identified as such rather than presented as changes to the main ReAct task policy.

Evidence selection under a controlled budget. The retrospective study enumerates 16 historical L3 cases × 14 trigger families. With the same selection budget of B=2 families per case, the recorded prior covers 8/16 cases, a fixed-seed random selection 4/16, and unchanged-condition replay 0/16. This directly isolates the value of which condition is tried. The exact conditional randomization tail is p=0.01576 on that archived family-outcome table; it is deliberately kept separate from the original four-benchmark TCR comparison and from patch acceptance.

This is why the L1-only and full-method results are not competing stories. One measures the shared improvement in diagnosis; the other adds access to evidence that failure-selected trajectories alone cannot supply.

Frozen RQ2 policy. The evaluator excludes these six signals from level routing (they may still be inspected for diagnosis): trace_evidence_sufficient, usage_cost_recorded, no_premature_hang_completion, required_checkpoint_not_skipped, cd_codex_secret_redaction, and cd_swe_clipped_observation_marker. In the corrected RQ4 routing configuration, the first four remain excluded; the Codex and SWE rules are instead enabled only for their source harnesses. The original ungated AppWorld characterization is retained as the original observation, not silently rewritten. We have added this complete matrix to Appendix B.

What is and is not an L2 witness? The eight-case RQ2 result remains the fixed submitted suite, but case-level analysis resolves a useful ambiguity. For oa_4224, the ordinary assistant trace repeats a purported {"status":"ok"} tool result without an actual function call: the evidence is present, and the Auditor missed it (L2). For langchain_39249, the ordinary assistant trajectories do not separate buggy and fixed revisions; only a verifier reports tool_invoked=false. That is more precisely a verifier-only observability boundary, not an ordinary-trace-visible L2 error. We use the original 8-case denominator for comparability while clarifying this boundary rather than retroactively improving reported recall.

Reliability of corpus labels. A separate, independently double-labeled 200-record sample gives 168/200 agreement (84%), Cohen's κ=0.755 (bootstrap 95% CI [0.676, 0.831]). The 36 executable cases provide the distinct runtime test of the L2/L3 definitions.

Variability. The Qwen fixed-variant TCR means (sample SD over three held-out tests) are HF 67.77±1.94, L1-only 71.19±0.82, L1+L2 72.05±1.81, full IHT 72.37±1.06. These are transparent descriptive increments, whereas the 16/16 benchmark/backbone improvements establish consistency across the matched settings. The experimental value of L3 is best assessed alongside defect-specific witnesses and correction of harness contracts, not by a single task-score increment.

We appreciate the opportunity to make the added operations and their evaluation meaning more precise.

Reviewer_W3xFOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors extend agent debugging beyond observable failures to uncover hidden failures, where the final outcome may appear successful despite silent failures in intermediate agent steps or bugs in the underlying harness. To systematically identify these issues, the paper introduces a three-layer taxonomy and finds that approximately 25% of failures stem from such hidden problems. These insights further enable the authors to identify and fix harness-level issues across different settings.

Critical strengths
  1. Important problem and novel formulation: The paper addresses an important problem in agent debugging and introduces a novel formulation that extends beyond observable failures and detects and diagnoses hidden failures.

  2. Substantial empirical evidence: The findings are supported by extensive empirical analysis of over 9,000 trajectories.

  3. Interesting findings: The discovery and characterization of hidden failures provide valuable insights into agent reliability and debugging.

  4. Well-designed methodology: The proposed approach is comprehensive and systematic, incorporating invariant auditing, proactive activation, source-grounded diagnosis, causal verification, and scoped repair.

Critical weaknesses
  1. Limited gains from proactive activation: While the overall improvements are consistent, the ablation results suggest that L1-only routing already accounts for approximately 74% of the gains over HarnessFix, while L3 activation contributes only an additional 0.3 points to the average TCR. This relatively modest gain may be due to the activation budget being limited to two candidates. It would be interesting to investigate whether allowing more activation probes could lead to larger improvements and better justify the additional complexity.

  2. Challenges in invariant-based detection: In RQ2, both the invariant Auditor and Passive IHT correctly identify only 1 out of 8 L2 cases, compared to 5 out of 8 for the raw LLM. The authors attribute this gap to conservative filtering, which may inadvertently discard genuine anomalies. However, given that detecting L2 defects is an important component of the proposed approach, these results raise questions about the effectiveness and generalizability of invariant-based detection. Further analysis of how these invariants perform across different harnesses, along with potential strategies to improve detection recall, would strengthen the paper.

  3. Computational overhead and scalability: The proposed approach introduces several additional steps, including trajectory auditing, activation probe generation, and causal verification of candidate defects. Although the paper reports some token costs, a more comprehensive analysis of the end-to-end runtime and computational overhead would be valuable. It also remains unclear how well the approach scales to larger or more complex agent systems, particularly when additional activation probes are required.

Questions for authors

Can you please shed lights on the three weaknesses I highlighted above?

3 comments
AuthorsOpenReview ↗

Proactive IHT consistently improves task performance while uncovering latent harness defects through targeted and efficient evidence acquisition.

Thank you for your encouraging assessment of the problem formulation, empirical study, and integrated methodology. Your three suggestions helped us strengthen the evaluation of what active evidence acquisition adds, how invariant checks are calibrated, and where the additional cost is incurred.

W1 — Why is the L3 task-score increment small? The goal of L3 is not simply to rerun an already successful task: it is to find a behavioral witness for a harness defect that the ordinary successful trajectory does not expose. This is a distinct capability from improving TCR under a fixed benchmark distribution. Proactive IHT improves over HarnessFix in all 16/16 benchmark–backbone settings, by 2.0–5.0 points (3.6 on average). The Qwen ablation distinguishes shared multi-view/intervention-supported diagnosis from additional observability coverage: L1-only already improves over HF by 3.42 points; L2 and L3 add 0.86 and 0.32 under the deliberately bounded two-candidate policy. The complementary mechanism-level question is: can we obtain useful evidence from a condition the original task did not exercise?

We added a focused 16-case, fixed-selection-budget analysis. At two archived trigger families per case, the recorded prior covers 8/16 latent cases, while fixed-seed random selection covers 4/16 and unchanged-condition replay 0/16. The exact upper-tail p=0.01576 is conditional on the archived trigger-family outcomes, rather than a full-pipeline significance test. This provides direct evidence that selecting an informative condition matters even when task success alone cannot expose the defect.

W2 — Invariant effectiveness and generalization. The frozen RQ2 Auditor's 1/8 result is not our preferred operating point, and we agree it requires explanation. The revision documents the full six-rule routing filter and the contract-applicability issue in the original AppWorld characterization. In the eight reconstructed L2 cases, a revised deterministic check flags 2/8 with 78/78 benign controls remaining quiet; a conservative model-assisted check flags 3/8 (71/72 quiet), while a permissive configuration flags 6/8 (12/72 quiet). These operating points make the recall–false-positive trade-off explicit. They also clarify that a useful warning requires a contract appropriate to the harness, not a keyword match from another framework. The frozen original AppWorld 22 positive records contain 80 warning instances; an applicability/evidence review reduces that set to two assistant-message boundary candidates.

W3 — Computational cost and scale. The deterministic audit is lightweight: in the 90-saved-trajectory microbenchmark, median latency is 0.79 ms per trajectory (p95 1.25 ms). The original GPT-5-mini offline optimization comparison is 49.0M tokens for IHT vs 44.3M for HF (+10.6%). Active probes and validation contribute separately to end-to-end cost; we therefore report the audit microbenchmark and the original offline repair-token count as different measurements. The two-candidate cap is a deliberate, bounded information-gathering policy rather than an assertion of negligible execution overhead.

We are grateful for the constructive suggestions. The revised paper makes task utility, defect-specific evidence, and resource accounting distinct, complementary evaluation axes.

Reconstructed L2 operating points:

CheckerCorrectly flagged casesBenign controls without warnings
Contract-aware deterministic checks2/878/78
Conservative model-assisted3/871/72
Permissive model-assisted6/812/72

Two cases previously grouped as “verifier-only” have different observability semantics. In oa_4224, the ordinary assistant trace explicitly echoes tool output: {"status":"ok"} without an actual function call. That is L2 evidence already visible to a trajectory-aware checker, and the original Auditor missed it. By contrast, for langchain_39249 the successful buggy/fixed assistant messages are indistinguishable, and only the external verifier reports tool_invoked=false. We now label the latter a verifier-only boundary, rather than forcing it into a trajectory-visible L2 definition. The original eight-case RQ2 table remains unchanged for transparent comparison with the submitted evaluation.

Original AppWorld warning origins: 22 metadata, 22 Codex-specific redaction, 12 reasoning-marker, eight sandbox, eight state/effect, and eight verification flags—80 total across the 22 original L2-like positives. After applying the appropriate harness contracts, only two reasoning-marker candidates remain. They have explicit </think> markers in assistant-visible content reused by the next model call; they are not Codex secret leaks. This recheck improves the specificity of what is sent to source-grounded diagnosis, instead of treating every cross-framework alarm as a repair request.

An invariant rule is useful precisely when its obligation, evidence channel, and applicability can be inspected. We now give the full RQ2 exclusion table and corrected RQ4 harness-identity gating in Appendix B.

A mechanism-level example beyond TCR. In CAMEL, successful agent conversations can still share a default response-chain key when an optional tracing integration is disabled, because the buggy session accessor hides an existing per-agent ID. Under the changed condition, a targeted source fix preserves session isolation while the benign single-agent control still passes. This is a harness state/correctness outcome rather than a change in task-score status. An AppWorld ReAct L2 example similarly keeps the executable completion code fixed but removes an internal reasoning marker from the assistant message boundary. The supplied benign, triggered and patched case traces make these differences inspectable. A broader AppWorld follow-up groups two retained L2 message-boundary observations and 38 L3 activation-positive records into nine concrete source edits (one ReAct message change, one input-construction change, seven trace-interface changes). All 27 targeted buggy/patched/merged checks satisfy their expected contract tests, with selected benign behavior preserved. These are source-level correctness outcomes, reported separately from held-out TCR.

A bounded follow-up on 16 reconstructed L3 cases: With a fixed budget of two selected archived trigger outcomes per case, repeating the healthy condition covers no latent defects, random selection covers four, and the recorded prior covers eight. This provides a targeted view of evidence acquisition under limited opportunity, distinct from the four-benchmark train-and-test repair loop. The original offline-token comparison and the auditor microbenchmark measure separate costs; the 0.79-ms figure is the checking stage, not end-to-end activation time.

Taken together, the new checks support a practical design trade-off: keep inexpensive invariant auditing broad, and spend the limited active-execution budget on conditions most likely to yield a discriminating witness. We appreciate your recommendation to make that trade-off measurable.

Reviewer_z7ZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks where the debugging of LLM agent harnesses should start. Existing methods diagnose and repair from failed trajectories, which assumes that the execution selected for diagnosis already contains evidence of the defect; the authors instead split executions into three observability levels: L1 where the task fails, L2 where the task succeeds but the trajectory contains abnormal evidence, and L3 where the task succeeds and the ordinary trajectory stays healthy. From 15,293 issues and merged pull requests across 16 open-source agent projects, a two-stage pipeline retains 9,058 harness-related records, 9,045 of which receive a level label, and L2 plus L3 account for 25.1%. The proposed Proactive IHT audits every trajectory, routes L1/L2 evidence into diagnosis by invariants, and applies bounded targeted activation to healthy/L3 candidates, after which all routes share source-grounded diagnosis, causal mechanism verification, and scoped repair. The evaluation covers a controlled study of 36 reconstructed historical defects and a matched closed-loop comparison against HarnessFix on GAIA, SWE-Bench Verified, AppWorld, and Terminal-Bench 2.0 across four backbones, where TCR improves in all 16 settings by 2.0–5.0 points, 3.6 on average.

Critical strengths
  1. Treating trajectories as the entry point for mining harness defects, and then asking what the default entry point of failed trajectories misses, is a valuable angle. L2 plus L3 account for 25.1% of the 9,045 labeled records (8.2%+16.9%, which I can reproduce), so restricting diagnosis to failures does skip a set of successful scenarios; the controlled study in Table 1 also separates evidence acquisition and source localization as two distinct bottlenecks. This is the main reason my rating stays at weak reject rather than a flat reject.
  2. The evidence chain is transparent, negative results are reported as they are, and I can reproduce the numbers. Appendices L.4 and M give per-trajectory routing and activation records for all 90 AppWorld training runs; Table 1 reports that the Auditor and Passive IHT classify only 1/8 L2 cases and that frozen IHT reaches a CR@3 of only 17/36, below HarnessFix's 18/36; the 16 per-setting deltas in Table 2 average to 3.6 points, which I recomputed. This makes me trust the reported results themselves, and my criticism concentrates on attribution and label validity.
Critical weaknesses
  1. The validity of the L2/L3 evidence itself is never established. The corpus labels are assigned from issue/PR text plus the fix diff (Section 3, Appendices E.3/E.4), not from replayed executions, and only 339 of 1,526 L3 records (22.2%) come with reproduction steps or a linked test/PR reproducer; this procedure cannot distinguish a trigger that never fired in an ordinary execution from one that fired but left no visible trace in the issue/PR text, and the two-author audit of 100 records per level (95/100 and 96/100 confirmed) checks against the same issue/PR text. On AppWorld, all 22 L2-like trajectories are triggered by the same cd_codex_secret_redaction rule (Codex secret redaction, Table 21 in Appendix L.4), while the harness is the official simplified ReAct code agent; the paper does not explain why this Codex rule applies there, and therefore what these 22 pieces of evidence actually establish.
  2. The title credits the gains to L2/L3 evidence, and the ablation in Table 3 does not support that. On Qwen3.6-Flash, L1-only averages 71.2, adding L2 routing gives 72.1 (+0.9), and adding the L3 branch gives 72.4 (+0.3); L1-only already retains 74% of the gain over HarnessFix (3.4 of 4.6 points). Removing multi-view diagnosis evidence gives 68.9, only 1.1 above HarnessFix's 67.8, the same magnitude as the fluctuation of a single evaluation. The gains therefore come mostly from the shared multi-view diagnosis and causal verification gate, not from the L2/L3 evidence acquisition the title advertises; reframing the contribution as "multi-view diagnosis plus a causal verification gate" would be self-consistent.
Questions for authors
  1. [Clarification] When corpus L3 labels rest only on issue/PR text, how do you distinguish a trigger that never fired in an ordinary execution from one that fired but is not visible in the report text?
  2. [Clarification] In Table 21 of Appendix L.4, all 22 L2-like AppWorld trajectories trigger cd_codex_secret_redaction. Why does this Codex-output rule apply to the AppWorld ReAct harness?
  3. [Clarification] Can you report the spread across the three held-out test runs, so readers can judge whether increments like +0.9 and +0.3 exceed noise?
  4. [Suggestion] Give HarnessFix an equal additional execution budget (the same number of activations/probes) before comparing.
  5. [Suggestion] Report how many of the finally promoted patches actually came from L2/L3 evidence, and run a blind relabeling of a sampled L3 subset with kappa reported.
7 comments
AuthorsOpenReview ↗

Independent validation and source-level repairs provide stronger evidence for the reliability and practical value of Proactive IHT.

We appreciate the thorough, artifact-level review, particularly your independent verification of the 25.1% corpus split and the 16 benchmark–backbone improvements. We share your emphasis on reproducible evidence. The revision now distinguishes report-level observability, an invariant warning, an exposed behavior, and an accepted repair, while providing concrete follow-up results for the main questions.

Q1 — How reliable are L3 labels? The original 9,045 L1/L2/L3 labels describe GitHub issue/PR reports and fix context, not a measured deployment prevalence. To test their reproducibility independently, two blinded annotators labeled 200 stratified records, agreeing on 168/200 (84.0%), with Cohen's κ=0.755 (95% bootstrap CI [0.676, 0.831]); 151 of their 168 agreements match the original category. The 36-case executable suite then tests a stronger, different claim using buggy/fixed versions and benign/exposing conditions. This combination lets us establish a repeatable report-level taxonomy and demonstrate its operational significance, without treating an unmentioned symptom as proof it never occurred.

Q2 — Why did a Codex rule fire in AppWorld? Your diagnosis is correct: the original AppWorld characterization applied cd_codex_secret_redaction outside its originating Harness, and the old substring matcher overreacted to ordinary code. We fixed the applicability policy and report the uncalibrated results transparently. The 22 originally positive AppWorld records contained 80 rule firings; after reviewing scope and evidence, 78 are removed and two assistant-message marker candidates remain. The Codex rule alone accounts for 22→0. Both remaining cases (3c13f5a_1, eb5ad85_3) contain </think> in assistant-visible content reused in the next model input, rather than a separate hidden-reasoning channel. A targeted ReAct source-level test shows that the marker can be removed without changing the executed task code. We keep the originally reported 23/22/45 screening distribution separately from the revised 23/2/65 classification ledger; they refer to different detector configurations, not 22 independently confirmed source defects.

Q3 — Are the small ablation differences beyond variation? We have added the actual three-test spread (Follow-up 1). In the Qwen fixed-variant comparison, HF averages 67.77±1.94, L1-only 71.19±0.82, L1+L2 72.05±1.81, and full IHT 72.37±1.06 (mean ± sample SD). The +0.86/+0.32 increments are descriptive, and we do not interpret the three tests as independent reoptimizations. Importantly, L1-only IHT still includes our multi-view source diagnosis and intervention-supported mechanism assessment; its strong gain over HF supports those shared changes, while L2/L3 broaden where useful evidence can originate.

Q4 — What does a fixed probe-selection budget show? In 16 reconstructed L3 cases with 14 archived trigger families each, selecting two families per case gives 8/16 confirmed-case coverage for the recorded prior versus 4/16 for a fixed-seed random selection and 0/16 for unchanged-condition replay (32 selected historical outcomes per arm). This shows that which condition is selected matters. The conditional uniform-random expectation is 3.81 cases, and the exact upper-tail probability of at least eight is p=0.01576. We keep this retrospective selection test separate from a prospective additional-execution study or the main four-benchmark repair comparison.

Q5 — Do L2/L3 findings reach source-level repairs? Two concrete before/after examples now make this link inspectable: a CAMEL session-state defect exposed by changing tracing availability, and an AppWorld ReAct assistant-message boundary violation despite task success. The case records include the benign/triggering/patched observations, source patches and successful targeted checks. We further include a nine-edit AppWorld patch ledger from two retained L2 and 38 activated L3 observations, with the tested code changes and 27/27 targeted regression-arm checks; Follow-up 2 separates agent-level changes from trace-interface corrections. We also distinguish the broader many-to-one mapping of warning/activation records to code edits from an independent count of defects or production-promoted patches. The method's value is not that every warning is a bug, but that it can find and act on evidence absent from failure-selected traces.

The two follow-ups supply the full detector accounting, variability table, and probe-budget details.

Blind annotation, independent of the original corpus labels. For 200 records (original strata: 50 L1, 60 L2, 90 L3), the annotation confusion matrix, rows=A and columns=B, is [[51,2,3],[6,43,14],[0,7,74]]. Agreement is 168/200, κ=0.755; 151/168 consensus labels agree with the prior corpus labels. This is direct evidence for the repeatability of interpreting issue/PR observability descriptions. Runtime meaning is evaluated separately by the reconstructed buggy/fixed cases.

Where the 80 AppWorld warnings came from (22 originally flagged successful records):

Rule familyOriginal firingsRetained after scope/evidence review
Trace metadata completeness220
Codex-specific secret-redaction rule220
Internal-reasoning marker122
Sandbox boundary80
Action/state correspondence80
Verification bypass / ignored result4 / 40 / 0
Total802

The originally reported AppWorld split (23 L1, 22 flagged L2-like, 45 healthy candidates) remains an authentic description of the original configuration. The revised classification ledger has 23 L1, two candidate L2 and 65 healthy. The latter is a routing reclassification; the two surviving markers have additional assistant-message boundary evidence. This separation prevents a Codex-only pattern match from becoming an AppWorld defect claim.

Three held-out tests of fixed Qwen variants (TCR, equally weighted across four benchmarks):

VariantTest 1Test 2Test 3Mean ± SD
HarnessFix67.4366.0169.8667.77±1.94
L1-only IHT72.0970.9770.5071.19±0.82
L1+L2 (without L3)73.9870.3971.7972.05±1.81
Full IHT71.6873.5971.8372.37±1.06

This makes the small L2/L3 increments easy to judge without overstating significance. The complete method's improvement across the original 16 matched settings is 2.0–5.0 points, +3.6 average.

Condition selection at equal retrospective budget. The archived study comprises 16 L3 cases, each with 14 candidate trigger families. All three selection policies inspect the same number of historical outcomes, B=2 per case (32 choices):

Selection policyArchived choicesCases with an exposing condition
Unchanged-condition replay320
Fixed-seed uniform random324
Recorded risk prior328

This contrast deliberately holds selection opportunity fixed. It is not the same as giving the unmodified HarnessFix an additional active-probing module or claiming equal wall-clock optimization budgets.

Conditional randomization test. Let k_i be the number of exposure-positive families for case i. A uniform selection of B distinct families has P_i(hit)=1−C(14−k_i,B)/C(14,B). At B=2, its expected coverage is 3.813 cases and the exact upper-tail probability of at least eight is 0.01576. This is conditional on the 14-family outcome table; we do not present it as a paired significance test of patch success. The selection table reports the per-case top-two prior; our evaluation keeps this conditional randomization analysis separate from an end-to-end repair-comparison claim.

From evidence to source changes. In CAMEL, the changed condition exposes state aliasing through the fallback session key; the source intervention restores access to the per-agent ContextVar and the benign case remains valid. In the AppWorld ReAct case, the ordinary successful response carries an internal marker into assistant-visible content; the repaired message-assembly boundary removes it while preserving apis.supervisor.complete_task(). Both cases include explicit before/after source patches and targeted benign/triggered checks. We keep these contract-level outcomes separate from the original four-benchmark TCR table. Patch accounting for AppWorld: 2 retained L2 observations map to one message-boundary edit; 38 L3 activation-positive records map to eight further edits (seven trace-interface functions, one task-input constructor). The nine unified diffs modify three files/nine functions and each has a buggy/patched/merged targeted regression; all 27 checks pass, including selected benign controls. The ledger explicitly separates the 40 evidence records from the nine unique code edits, and it labels trace-adapter changes separately from the task agent. These source-level checks address whether evidence can yield an inspectable correction even when ordinary task completion is unchanged.

Finally, the patch-provenance review is organized by the actual four RQ4 benchmarks (GAIA, SWE-Bench Verified, AppWorld, Terminal-Bench 2.0) and distinguishes prompt/context, local-behavior, mechanism/state, and trace-interface changes. This helps interpret why task-success gains and correctness of a harness execution contract need not move together. We appreciate the reviewer's insistence that every reported result have a clear measurement unit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.