- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Causal Improvement Graph (CIG), a meta-harness optimization framework that represents search history as a persistent graph of Evidence, Hypothesis, Intervention, and Outcome nodes. CIG delegates observation, planning, plan selection, implementation, evaluation, and hypothesis revision to locally scoped proposer calls, and evaluates the resulting workflow on classification tasks and agent benchmarks including SWE-bench Verified, AppWorld, and Terminal-Bench.
Critical strengths
- I find the paper addresses a practical limitation of proposer-centric harness search. As the history grows, the LLM proposer struggles to track past experiments, code, and results. I find the separation of persistent state from local operations useful and potentially practical.
- I find the experiments fairly comprehensive. Table 1 shows improvements across three agent settings, with CIG achieving a macro average of 44.6 versus 40.1. Table 3 also shows gains for all proposer models on SWE-bench Verified. These results suggest that structured state can help different proposer models.
Critical weaknesses
- The only concern that I have is that, at current state, the term “causal” is only operationally justified, not formally supported. Section 3.2 appropriately states that formal causal identification is outside the formulation. However, the title and repeated use of “causal” risk suggesting stronger claims than the method provides.
Questions for authors
- What made you pick Evidence → Hypothesis → Intervention → Outcome as the schema? Did you try other node types, like a separate ‘Mechanism’ or ‘Failure mode’ node, and what happened?
- How big does the graph get in a typical run, and do you ever need to summarize or prune it?