acceptodds
Under review as a conference paper at ICLR 2027

Causal Improvement Graph for Agentic Harness Optimization

Abstract

Agentic Harness is the runtime that constructs task context and controls execution flow, thereby shaping overall agent performance. Given a fixed model and external evaluation, automated Harness optimization seeks to improve this runtime through an iterative proposal–evaluation loop to better solve target tasks. Existing meta-harness methods mainly adopt *proposer-centric discovery*, in which an LLM-based proposer integrates accumulated experimental findings to determine subsequent Harness revisions. This places the burden of maintaining the evolving improvement state on the proposer as history expands and its underlying experimental logic becomes harder to discern. In this paper, we introduce the *Causal Improvement Graph* (CIG), a graph-governed meta-harness framework that externalizes the evolving improvement state in a persistent graph, allowing prior findings to directly govern subsequent Harness optimization through local proposer operations. CIG grows and links EVIDENCE, HYPOTHESIS, INTERVENTION, and OUTCOME nodes to represent what was observed, how it may be explained, how to test that explanation, and what the evaluation reveals. Their structural relations preserve how the improvement state changes across iterations, allowing local proposers to build directly on relations among prior findings rather than recover them from raw history. Across various agent tasks, CIG discovers stronger Harnesses than previous meta-harness baselines and remains robust to the choice of task solver and proposer. Structural ablations further support the design of an explicit improvement state with graph-governed evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent, Harness Evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FrQYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Causal Improvement Graph (CIG), a meta-harness optimization framework that represents search history as a persistent graph of Evidence, Hypothesis, Intervention, and Outcome nodes. CIG delegates observation, planning, plan selection, implementation, evaluation, and hypothesis revision to locally scoped proposer calls, and evaluates the resulting workflow on classification tasks and agent benchmarks including SWE-bench Verified, AppWorld, and Terminal-Bench.

Critical strengths
  • I find the paper addresses a practical limitation of proposer-centric harness search. As the history grows, the LLM proposer struggles to track past experiments, code, and results. I find the separation of persistent state from local operations useful and potentially practical.
  • I find the experiments fairly comprehensive. Table 1 shows improvements across three agent settings, with CIG achieving a macro average of 44.6 versus 40.1. Table 3 also shows gains for all proposer models on SWE-bench Verified. These results suggest that structured state can help different proposer models.
Critical weaknesses
  • The only concern that I have is that, at current state, the term “causal” is only operationally justified, not formally supported. Section 3.2 appropriately states that formal causal identification is outside the formulation. However, the title and repeated use of “causal” risk suggesting stronger claims than the method provides.
Questions for authors
  • What made you pick Evidence → Hypothesis → Intervention → Outcome as the schema? Did you try other node types, like a separate ‘Mechanism’ or ‘Failure mode’ node, and what happened?
  • How big does the graph get in a typical run, and do you ever need to summarize or prune it?
Reviewer_vTAuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces graphs into the harness optimization problem as an abstraction of ultra-long historical trajectories.

It creates a "Causal Improvement Graph" that links evidence, hypothesis, intervention, and outcome nodes to enable graph-governed discovery. This is an evolving graph as the agent continuous optimizes the harness and record the actions as nodes in the graph.

Experiments on (1) classification tasks on three Meta-Harness tasks, and LawBench criminal-charge classification, and (2) agent-harness evaluation on SWE-bench, AppWorld, Terminal-Bench show strong improvement of the proposed CIG compared with some baselines.

Critical strengths

Clear presentation and good motivation on using graphs to reduce the burden of maintaining the evolving improvement states.

A lot of evaluations have been conducted and the proposed CIG have the best performance overall.

Critical weaknesses

W1. The proposed causal improvement graph is more like a data structure of improvement history summary. For me, the proposed method seems not to use any benefit from the graph modeling. So why bother to use graphs for this summary, instead of just using summarized text? If you're still relying on the agent to do the observe, plan, implement, evaluate and update, text-based context is more friendly, isn't it?

W2. The experiment does not align with the motivation. In abstract and introduction the CIG is motivated by reducing the burden of long trajectory logs while continuously doing harness improvement. My interpretation is that this should be beneficial only in cases where the trajectory is too long to feed into agents. However, in the experiment, the trajectory seems not long enough (and there is no need for graphs or maybe even no need to summary). From line 325 and 1050, the study uses eight candidate evaluations, and full history and CIG only uses four waves of two candidates. Is the improvement context exceeding the context window for agents that we need to use summary?

W3. Since we're using nodes as a map of improvement history, and the observer is limited by compute budget, there has to be information loss. In other words, though all history are kept somewhere, some useful context might never be accessed by the agent using the graph modeling.

W4. The provided case study A.9 is only a part of a full run. The readers would be interested how the proposed method works on an harness optimization from, say, terminal bench, end-to-end. (I would be curious what's the difference between the evolved harness and initial harness, and how CIG creates a better improvement trajectory).

Questions for authors

Q1. Table 2 and Table 3 use two sets of models. One set is weaker, and one set is stronger, including models such as GPT-5.6 Sol and DeepSeek-V4-Pro. Is this an intentional design, and why?

Reviewer_cEPWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Causal Improvement Graph (CIG), a meta-harness optimization framework that maintains persistent Evidence-Hypothesis-Intervention-Outcome records and explicit relations between them. Locally scoped observation, planning, portfolio selection, implementation, evaluation, and revision operations use this state to guide subsequent harness changes. Experiments cover three classification tasks and three agent benchmarks, with solver/proposer studies and staged structural controls. The distinctive contribution is the use of linked, revisable hypothesis assessments to govern further experiments, not graph storage alone.

Critical strengths
  1. The matched structural controls provide meaningful evidence for the proposed organization.

  2. The method offers a concrete, inspectable way to separate measurements from evolving interpretations, with positive results across different task interfaces. The worked example preserves a locally consistent prediction despite a failed aggregate target, rather than treating a score change as proof of a mechanism. CIG also improves over the authors' Full History implementation on all three agent benchmarks and across all five tested proposers, under a stated validation-only selection protocol.

Critical weaknesses
  1. The empirical scope is shorter-horizon than the central growing-history motivation. Controlled studies use eight candidate evaluations, with two waves for Wide or four for Deep. This shows a benefit within small searches, but does not directly establish that the graph increasingly relieves reconstruction burden as history grows.

  2. End-to-end resource efficiency and reproducibility are less established than the candidate-budget results.

Questions for authors
  1. Clarification/evidence request: On SWE-bench Verified, what are total proposer usage, solver-evaluation cost, retries, and wall-clock time for CIG and the staged M0/B3 controls?

  2. Claim-scope clarification: Do existing longer-horizon or repeated checkpoint-intervention results support the specific claim that CIG alleviates growing-history reconstruction burden?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.