- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work introduced an automated method that 1. use claude code with opus 4.8 to edit/modify a harness based on a student models rollout, 2. use Qwen3.5-35b-a3b to generate training data with the modified harness, 3. perform SFT to improve the student model, and repeat from step 1. The goal of this is to produce a model (the student) to be able to solve tasks without often hand-crafted heuristics that is baked in in many harness systems that comes with the benchmark. The authors then evaluated their method on benchmarks such as DISBench, AgentVisa, BrowseComp-VL, and MM-BrowseComp, and showed that their method outperforms equipping the base model with an oracle/hand-designed harness.
Critical strengths
-
This work introduced HarnessMind which trained agents that can use tools better than human designed harnesses across a diverse set of benchmarks including DISBench, AgentVisa, BrowseComp-VL, and MM-BrowseComp.
-
This work performed various analysis beyond performance, including performance breakdown in long-horizon tasks, ablation studies, as well as inference efficiency measurement.
Critical weaknesses
-
Method section is very hard to understand. I think one contributing factor is the excessive use of obscure terminology (especially in the method section), such as "component inventory," "control component," "branching action continuation," "search-to-process supervision," "context-supported failure reflections," "composable control search," and etc. (all from Section 3). Many of these terms are difficult to understand without further explanation, yet little to no explanation is provided.
-
If the goal is to train a model that can solve tasks without relying on "harness-specific control prompts" (L18-19), then I believe the proposed methods seems overly complicated without establishing an apple-to-apple comparison to simpler baselines in the main results. For example, from Section 3 the proposed method uses Claude Code (opus 4.8) to configure harnesses, trains on "additional synthesized tasks with difficult tasks selected" (L250-251), and uses a stronger model (Qwen3.5-35b-a3b) to generate trajectories for training smaller models such as 4B and 9B). Then baseline should include simply distilling rollouts from Opus 4.8 / Qwen3.5-35b-a3b with these prompts removed and/or also using the "additional synthesized tasks with difficult tasks selected. However, this result is absent.
-
Accordingly to ablation studies (Table 6), each of the introduced stage/component in the proposed methods is incremental and most of them only differs within 2 points overall. Is there any statistical significance?
Questions for authors
- Are the results in Table 1 statistically significant? Compared to the second best (usually w/ oracle harness) there is often <1 point improvement, and even comparing directly with the base model theres often only 2-3 points improvement.