acceptodds
Under review as a conference paper at ICLR 2027

Harness2Policy: Harness Internalization for Long-Horizon Vision-Language Agents

Abstract

Long-horizon vision-language tasks require agents to preserve visual evidence, manage memory, and coordinate tool use. External harnesses guide these decisions, but successful-trajectory distillation can omit the control supervision and failure experience needed for autonomous execution. We present Harness2Policy, a framework that converts harness-guided exploration into process supervision. We search task-adaptive control configurations to guide tree-based exploration, retaining successful paths with explicit planning, memory, verification, and context-supported failure reflections. Using these trajectories, we build HarnessMind agents through supervised learning without harness-specific control prompts, enabling control through their responses and tool calls. Repeating this process with the updated model enables recursive co-improvement of the harness and the model. Across four benchmarks, HarnessMind (4B, 9B, and 35B) outperforms its base models with and without benchmark-specific oracle harnesses. On tasks whose reference runs exceed 20 turns, the 9B model improves accuracy from 7.4% with successful-path distillation to 14.3%. It also raises aggregate accuracy from 24.4% to 25.7% over its oracle-harness baseline with 31% fewer inference tokens. This work paves the way for future research on scalable self-improvement through the co-evolution of agentic harnesses, training data, and foundation models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Horizon Agents, Agent Harness, Visual-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AynkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work introduced an automated method that 1. use claude code with opus 4.8 to edit/modify a harness based on a student models rollout, 2. use Qwen3.5-35b-a3b to generate training data with the modified harness, 3. perform SFT to improve the student model, and repeat from step 1. The goal of this is to produce a model (the student) to be able to solve tasks without often hand-crafted heuristics that is baked in in many harness systems that comes with the benchmark. The authors then evaluated their method on benchmarks such as DISBench, AgentVisa, BrowseComp-VL, and MM-BrowseComp, and showed that their method outperforms equipping the base model with an oracle/hand-designed harness.

Critical strengths
  • This work introduced HarnessMind which trained agents that can use tools better than human designed harnesses across a diverse set of benchmarks including DISBench, AgentVisa, BrowseComp-VL, and MM-BrowseComp.

  • This work performed various analysis beyond performance, including performance breakdown in long-horizon tasks, ablation studies, as well as inference efficiency measurement.

Critical weaknesses
  1. Method section is very hard to understand. I think one contributing factor is the excessive use of obscure terminology (especially in the method section), such as "component inventory," "control component," "branching action continuation," "search-to-process supervision," "context-supported failure reflections," "composable control search," and etc. (all from Section 3). Many of these terms are difficult to understand without further explanation, yet little to no explanation is provided.

  2. If the goal is to train a model that can solve tasks without relying on "harness-specific control prompts" (L18-19), then I believe the proposed methods seems overly complicated without establishing an apple-to-apple comparison to simpler baselines in the main results. For example, from Section 3 the proposed method uses Claude Code (opus 4.8) to configure harnesses, trains on "additional synthesized tasks with difficult tasks selected" (L250-251), and uses a stronger model (Qwen3.5-35b-a3b) to generate trajectories for training smaller models such as 4B and 9B). Then baseline should include simply distilling rollouts from Opus 4.8 / Qwen3.5-35b-a3b with these prompts removed and/or also using the "additional synthesized tasks with difficult tasks selected. However, this result is absent.

  3. Accordingly to ablation studies (Table 6), each of the introduced stage/component in the proposed methods is incremental and most of them only differs within 2 points overall. Is there any statistical significance?

Questions for authors
  • Are the results in Table 1 statistically significant? Compared to the second best (usually w/ oracle harness) there is often <1 point improvement, and even comparing directly with the base model theres often only 2-3 points improvement.
Reviewer_eGbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Harness2Policy, a framework for training vision-language agents to make control decisions without task-specific external guidance. The framework searches for task-adaptive harnesses, uses them to guide tree-based trajectory generation, and compiles successful paths into supervision containing planning, memory operations, verification, and context-supported reflections on failed alternatives. Students are trained with harness-specific control prompts removed. Experiments cover four benchmarks and three Qwen3.5 model sizes. The main results show improvements over successful-path distillation and corresponding base models with benchmark-specific harnesses, alongside reduced inference-token usage.

Critical strengths
  1. The process-supervision comparison provides meaningful evidence for the paper’s central motivation. The proposed method is compared with successful-path distillation using the same generation-policy search trees and matched numbers of training target tokens. On tasks whose reference runs exceed 20 turns, accuracy improves from 7.4% to 14.3%. The larger advantage on longer tasks supports the value of preserving control decisions and failure-related reasoning beyond imitating successful action sequences.

  2. The method improves performance while reducing dependence on task-specific inference guidance. All three student sizes outperform their corresponding oracle-harness baselines on the four primary benchmark metrics. For the 9B model, aggregate accuracy improves from 24.4% to 25.7% while mean inference-token usage decreases by approximately 31%. These results demonstrate a useful practical benefit under a shared tool and execution interface.

Critical weaknesses
  1. The central process-compilation mechanism is not sufficiently concrete. The paper describes retaining planning, memory, verification, and context-supported failure reflections, but does not show how these elements are extracted and assembled into a training example. In particular, it is unclear how the compiler determines whether a reflection is supported by the student’s effective context after removing harness prompts and folding memory. For example, a reflection could depend on an observation from a discarded branch even if its wording appears plausible. Since this filtering is central to the proposed supervision, an explicit algorithm and a worked example are needed to assess its reliability and distinguish it from ordinary trajectory augmentation.

  2. The evidence for control internalization remains indirect, and the uncertainty of the smaller gains is unclear. The declining benefit of external harnesses is consistent with internalization, but broader capability improvements could also reduce the usefulness of external guidance, as the authors acknowledge. The experiments do not directly measure whether the student makes better decisions about memory management, verification, or termination. In addition, each configuration is evaluated once per test task without reported uncertainty estimates. This limits confidence in the relatively small improvements over oracle harnesses, including the 1.3-percentage-point aggregate gain.

Questions for authors
  1. How does the compiler determine whether a failure reflection is supported by the effective student context? Is this checked through rules, model judgments, or manual inspection?

  2. What exactly is removed in the successful-path distillation baseline? Do successful paths retain their original planning and verification responses, or are these also excluded?

  3. Do you have measurements of actual control behavior, such as appropriate memory-fold timing, evidence verification, or termination decisions, that distinguish control internalization from general improvements in task-solving ability?

  4. How stable are the gains over oracle harnesses? Can you provide paired confidence intervals over test tasks or repeated evaluations for the primary 9B comparison, including the subset whose reference runs exceed 20 turns?

  5. How stable are the improvements across repeated runs of the iterative training pipeline? Since each round resamples development tasks and generates fresh trajectories, can you report variability across independent runs or training seeds to establish whether the observed performance gains and declining reliance on external guidance are reproducible?

Reviewer_EhRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Harness2Policy, which distills harness-guided agent trajectories into a vision-language model. The training data includes planning, memory management, verification, and failure reflections. Experiments on four benchmarks show improvements over standard trajectory distillation, particularly on long-horizon tasks.

Critical strengths
  • The approach reduces the need for task-specific harnesses at inference while improving accuracy and token efficiency.
  • The ablations are fairly convincing. Process compilation improves overall accuracy from 22.0% to 25.7%, with larger gains on tasks exceeding 20 turns.
Critical weaknesses
  • The methodological novelty is limited. The main contribution is a carefully designed data collection and SFT pipeline rather than a fundamentally new learning method.
  • The evidence for harness internalization is indirect. Reduced gains from external harnesses could also reflect general improvements in model capability. The iterative training still relies on an external teacher.
Questions for authors
  • Can the authors directly measure improvements in autonomous memory management, verification, and other control decisions?
  • Why does free-form harness search achieve higher accuracy (26.9%) than the proposed structured search (25.7%)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.