- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
< Note: Well I found the paper frustratingly difficult to follow: many concepts are insufficiently explained, and terms are used inconsistently. Within the limited time available for this review, I had to infer several intended meanings, so the summary below reflects my best interpretation of the work :) >
The paper aims to repair harness. The authors propose Proactive IHT, which probes selected apparently healthy executions to obtain additional evidence and verifies candidate mechanisms before repairing the harness. With expeirments, the authors report task completion rate improvements of 2.0–5.0% over HarnessFix across all 16 benchmark–backbone settings, averaging 3.6%.
Critical strengths
- important problem
- demonstrating some improvement via experiments
Critical weaknesses
- The contribution over the baseline (HarnessFix) looks a bit trivial
Table 3 leaves me uncertain about the contribution emphasized by the paper…The additional benefit from L2 and L3 therefore appears very small in this setting. The L1-only configuration already retains most of the observed improvement over HarnessFix.
I understand that this ablation uses one backbone and a limited activation budget. Table 19 in Appendix L.2 reports three independent test runs per variant, but I could not find confidence intervals or significance tests for these small increments. I therefore cannot tell whether the gains are statistically distinguishable from run-to-run variation… To help readers assess the contribution of L2 and L3, I would really appreciate results from independently repeated repair runs, together with uncertainty estimates and appropriate significance tests for these comparisons.
This makes the changes to diagnosis and verification particularly important. (by the way, “verification“ is typically a reserved word for formal verification in software enigneering/programming language domains) HarnessFix already has structured traces, runtime-to-source mappings, scoped repair specifications, and regression-aware acceptance. The paper acknowledges this foundation, but Section 5 does not make the changes to the diagnostic procedure sufficiently concrete. I can identify the additional evidence views and the set of competing mechanisms, but I still cannot reconstruct the changed diagnostic operations or connect them to the observed gain. I would like the claimed advance to be tied to specific changed operations and their measured effects.
- The evaluation does not sufficiently connect active probing to verified defects and repairs.
<Note: Again, I doubt the usage of term “verified“>
The historical-case experiments evaluate passive diagnosis or localization after a defect has already been exposed. The AppWorld analysis reports triggered anomalies, but does not clearly show how many correspond to confirmed harness defects and lead to accepted repairs. I would like to see these outcomes and one complete example from an initially healthy execution to a successful repair. A simple probing baseline under the same budget would also help establish whether the proposed risk ranking adds value.
- The decisions that determine routing and mechanism confirmation remain difficult to assess.
Appendix B.2 does not specify when the filter suppresses an invariant warning, although the reported L2 errors are attributed to this filter. I also could not follow how a probe distinguishes competing explanations for an anomaly. Making the symptom disappear could simply reflect a workaround. Please state the filtering rule and show what a confirmation probe changes, what it holds fixed, and how its outcome supports one mechanism over another.
- The opening argument conflates two different limitations.
The abstract and introduction state that L2 and L3 both violate the assumption that the execution selected for diagnosis already contains defect evidence. L2, as defined in the paper, does contain such evidence. It challenges the selection of only failed executions, whereas L3 concerns the absence of a behavioral witness in the current execution. HarnessFix explicitly studies failed trajectories; this scope does not imply that successful executions cannot contain defects. The paper has a reasonable extension to motivate, but the current argument does not establish it correctly. This issue is secondary to the evaluation concerns above, though it contributes to the uncertainty about the central claim.
Questions for authors
- is the improvement over baseline statistically representative?
7 comments
HarnessFix already provides structured trajectories, source-aware diagnosis, scoped patches, and regression checks; we build directly on that foundation. Our added operations address a different decision point:
| Step | Failure-trajectory workflow | Proactive IHT addition |
|---|---|---|
| Select executions | Enter diagnosis from task failures | Audit successful runs for L2 contract evidence |
| Acquire missing evidence | Use the current failure witness | Choose a reachable condition for an apparently healthy L3 run |
| Compare explanations | Source-grounded hypotheses | Combine trace, invariant, activation, and source views; test predictions under controlled changes |
| Accept a repair | Scoped change and regressions | Also check the targeted manifestation and an unaffected benign condition |
Concrete mechanism-level contrast. In the CAMEL example, the benign execution alone does not expose the shared session-key fallback. The changed condition (tracing integration disabled while two agent sessions are active) exposes a state-isolation violation. The source-level intervention fixes access to the ContextVar regardless of tracing availability, and the control preserves separate agent IDs. In the AppWorld example, the task code still executes successfully while the assembled assistant-message channel contains an internal-reasoning marker; the boundary repair removes the marker and leaves the code unchanged. These are meaningful harness correctness outcomes, not merely an alternative prompt. A broader AppWorld audit identifies nine separate source-level corrections linked to two retained L2 message-boundary observations and 38 activated L3 records: one ReAct message edit, one task-input edit, and seven trace-interface edits. We supply all nine unified diffs and a record-to-edit ledger. The 27/27 expected checks hold on targeted buggy, patched, and merged code, including selected benign controls. The counts refer to 40 evidence observations and nine distinct corrected source contracts, not 40 independent defect repairs. In particular, trace-interface fixes are identified as such rather than presented as changes to the main ReAct task policy.
Evidence selection under a controlled budget. The retrospective study enumerates 16 historical L3 cases × 14 trigger families. With the same selection budget of B=2 families per case, the recorded prior covers 8/16 cases, a fixed-seed random selection 4/16, and unchanged-condition replay 0/16. This directly isolates the value of which condition is tried. The exact conditional randomization tail is p=0.01576 on that archived family-outcome table; it is deliberately kept separate from the original four-benchmark TCR comparison and from patch acceptance.
This is why the L1-only and full-method results are not competing stories. One measures the shared improvement in diagnosis; the other adds access to evidence that failure-selected trajectories alone cannot supply.
Frozen RQ2 policy. The evaluator excludes these six signals from level routing (they may still be inspected for diagnosis): trace_evidence_sufficient, usage_cost_recorded, no_premature_hang_completion, required_checkpoint_not_skipped, cd_codex_secret_redaction, and cd_swe_clipped_observation_marker. In the corrected RQ4 routing configuration, the first four remain excluded; the Codex and SWE rules are instead enabled only for their source harnesses. The original ungated AppWorld characterization is retained as the original observation, not silently rewritten. We have added this complete matrix to Appendix B.
What is and is not an L2 witness? The eight-case RQ2 result remains the fixed submitted suite, but case-level analysis resolves a useful ambiguity. For oa_4224, the ordinary assistant trace repeats a purported {"status":"ok"} tool result without an actual function call: the evidence is present, and the Auditor missed it (L2). For langchain_39249, the ordinary assistant trajectories do not separate buggy and fixed revisions; only a verifier reports tool_invoked=false. That is more precisely a verifier-only observability boundary, not an ordinary-trace-visible L2 error. We use the original 8-case denominator for comparability while clarifying this boundary rather than retroactively improving reported recall.
Reliability of corpus labels. A separate, independently double-labeled 200-record sample gives 168/200 agreement (84%), Cohen's κ=0.755 (bootstrap 95% CI [0.676, 0.831]). The 36 executable cases provide the distinct runtime test of the L2/L3 definitions.
Variability. The Qwen fixed-variant TCR means (sample SD over three held-out tests) are HF 67.77±1.94, L1-only 71.19±0.82, L1+L2 72.05±1.81, full IHT 72.37±1.06. These are transparent descriptive increments, whereas the 16/16 benchmark/backbone improvements establish consistency across the matched settings. The experimental value of L3 is best assessed alongside defect-specific witnesses and correction of harness contracts, not by a single task-score increment.
We appreciate the opportunity to make the added operations and their evaluation meaning more precise.
Proactive IHT goes beyond failure-driven debugging to uncover and repair harness defects missed by task-success metrics.
Thank you for the careful critique and for acknowledging both the importance of harness reliability and the consistent improvements over HarnessFix. Your questions helped us sharpen the distinction between improving diagnosis given a witness and obtaining a witness in the first place. We have revised the framing and added executable examples and focused validation.
W1 — What is the substantive advance over HarnessFix? Our contribution has three connected parts. First, an empirical study of 9,045 labeled harness development records across 16 projects identifies two ways failure-gated debugging misses evidence: L2 runs succeed but already contain an actionable trace anomaly (selection gap); L3 ordinary runs are healthy and require a changed execution condition to expose the defect (evidence gap). Second, the 36-case executable study separates obtaining behavioral evidence from locating the responsible source mechanism. Third, Proactive IHT brings these decisions into a common source-grounded, multi-view, intervention-supported repair loop. Across four benchmarks and four backbones, the complete method outperforms HarnessFix in 16/16 matched settings, by 2.0–5.0 TCR points (3.6 on average).
The Qwen ablation also helps identify where this benefit arises. L1-only IHT is not HarnessFix: it retains the new multi-view evidence and mechanism-assessment operations, so its 3.42-point gain over HF reflects a shared diagnostic contribution even before successful-run routing is enabled. L2/L3 then extend the set of executions and conditions from which this workflow can collect evidence. Their additional +0.86/+0.32 task-completion points under a two-candidate activation cap are modest; they should not be taken as a complete measure of latent harness defects, which may not change an ordinary benchmark outcome. Removing multi-view evidence or mechanism assessment from the full method reduces average TCR by 3.5 and 2.2 points, respectively (separate conditional ablations, not additive effects).
W2 — Where does probing connect to a real mechanism? We added two source-grounded contrasts. In a CAMEL L3 case, ordinary conversations complete, yet switching off the optional tracing integration makes the per-agent session accessor return
None; two agents can then share the__default__response-chain key. The repaired accessor preservesagent-aandagent-b. In an AppWorld ReAct L2 case, a successful code-producing response carries an orphan reasoning marker into the assembled assistant-visible content reused by the next model call. The repair removes the marker without changing the extracted executable code. The supplied case records include benign, triggering, and patched observations, source changes, and passing targeted checks. A broader AppWorld source-level audit also connects the two retained L2 and 38 L3 activation observations to nine distinct edits, with 27/27 targeted buggy/patched/merged checks passing; see Follow-up 1 and the new appendix table. These are specific harness obligations that task success alone does not test; Follow-up 1 gives the operation-level comparison and validation sequence.W3 — Routing and mechanism confirmation. We now make the actual RQ2 six-rule filter explicit and distinguish that frozen filter from harness-identity gating of code-derived rules in the corrected RQ4 policy. The original AppWorld diagnostic audit inadvertently applied a Codex-specific keyword rule beyond its scope. Reviewing the 22 original L2-like positives removes 78 of 80 heuristic firings, leaving two assistant-message boundary candidates. More importantly, our method does not equate a warning with a repair: it proposes a mechanism, changes a specified condition, checks the predicted symptom, and tests the patched/benign executions. This is operational intervention-supported confirmation, not a claim of formal proof.
W4 — Motivation. We agree with your useful distinction: L2 already has trace evidence but is skipped by failure-only selection; L3 lacks a witness in the ordinary run. The revised abstract and introduction separate these gaps explicitly, without suggesting HarnessFix claims successful executions cannot contain defects.
The central point is that task completion and harness correctness are complementary evaluation targets. The revision makes the evidence-acquisition contribution concrete while preserving the original controlled TCR results.