- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how an agent harness decides whether to keep doing what the agent proposed or to adjust the workflow in long-horizon task (e.g., thinking more, gathering evidence, or wrapping up an answer). Current harnesses mostly make these choices with fixed rules, which the authors argue wastes compute and can sometimes hurt success. There, they propose CHILL-Harness that learns these choices instead. CHILL-Harness restores an execution from a saved checkpoint, runs both the original step and an alternative, and uses the difference in outcomes to train a model that estimates how much an alternative would help. These are all done offline. It also trains small predictors that decide how much extra deliberation to spend, whether to replace the current step, and when the task is done. At run time, the harness first decides how much deliberation is worth spending, then generates and scores alternatives, and only replaces the original step when the estimated benefit is clearly large enough. The method is evaluated on GAIA, SWE-bench Verified, and Terminal-Bench 2.0, where it reports lower token use and runtime than several public harnesses while keeping task success competitive.
Critical strengths
- The motivation makes sense. Harnesses often spend extra reasoning or verification even when it is not needed. Learning when such interventions are worth their cost is a useful direction for making long-horizon agents cheaper to run.
- Training signals come from comparing two branches restored from the same checkpoint, which is a reasonable way to estimate whether a change helps. At run time, the method falls back to the original step unless the evidence for a change is strong, and it only ends a task when an explicit check confirms completion. Deciding how much to deliberate before generating candidates also avoids paying for alternatives that are unlikely to be used.
- The experiments cover three quite different domains (information seeking, software repair, and terminal tasks) and compare against several well-known harnesses. Token counts include the extra model calls made by the method itself.
Critical weaknesses
- The writing obscures a fairly simple method. The abstract and introduction lean on vague phrases such as “advantage-guided workflow adaptation,” “confidence-weighted matched execution evidence,” and “causal–operational advantage,” and after reading them it is still unclear what the method actually does. The main text also never says which base harness CHILL-Harness is built on, which is basic information needed to interpret the results. Finally, five new acronyms (CIEL, ARCO, CDE, CRE, CAE), together with do-notation and potential-outcome notation, are used to describe what is essentially a comparison between two branches run from the same checkpoint. This could be stated in one sentence and would make the paper much easier to follow.
- It is unclear which part of the design does the work. The final decision combines a learned effect estimate, an intervention predictor, a route predictor, a replacement classifier, a hand-built operational score (progress, cost, risk, information gain, robustness, safety), and an authorization margin. The only ablation (Always-Full) does not separate these parts.
- The experiments do not test the paper’s main claim. The paper’s central claim is that CHILL-Harness reduces cost without hurting task success (Section 3.3, RQ1). The clean way to test this is to run the same base agent, with the same model, tools, and budget, both with and without CHILL-Harness, and compare the two. Section 4.1 says such matched reference runs were done, but their results are never reported. Instead, Table 1 compares CHILL-Harness with public systems that use different models and possibly different subsets of tasks, which the authors themselves describe as only “descriptive.” The lower token use and runtime could therefore come from these differences rather than from the method, and RQ1 remains unanswered.
Questions for authors
- Where are the matched reference results?
- Why does more deliberation cut GAIA success by more than half in the Always-Full ablation, and what does this say about the robustness of the replacement step?