acceptodds
Under review as a conference paper at ICLR 2027

Harness for Inference-Time Alignment over Execution Trajectories

Abstract

Harness engineering has emerged as an important inference-time technique for large language model (LLM) agents, aiming to improve long-term performance through task decomposition and guided execution. However, more elaborate harnesses are not uniformly better: increasing decomposition or guidance can sometimes improve execution, but can also reduce final task success. We study harness design through the lens of inference-time trajectory alignment. This perspective separates harness into two mechanisms: task decomposition, which structures a task into sub-goals, and guided execution, which reshapes local action distributions during execution. This decomposition allows us to quantify how workflow granularity, retry budgets, and guidance-induced action reweighting shape the performance limits of harness design. It further reveals concrete failure modes, including over-decomposition, over-pruning, and hallucinated execution. We validate these predictions through controlled synthetic experiments and real terminal agent benchmarks. Inspired by the theory, we further show that effective harnesses can be partial: specifying only the initial steps and leaving the remaining execution to agent can achieve higher pass rate than fully structured workflows.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human-in-the-loop, Harness, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hAVzOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when inference-time scaffolding helps LLM agents, distinguishing workflow decomposition from guidance. It derives a finite-attempt granularity bound, an exact retention-gap identity, and a stopping rule for partial harnessing under homogeneous scaffold reliability and convex autonomous tail risk. Experiments show non-monotonic effects of adding structure, suggesting that scaffolding should match agent capabilities and available evidence.

Critical strengths
  1. The workflow/guidance distinction clarifies harness failures. The granularity analysis accounts for tolerance, cumulative action scale, and budget.

  2. The fixed-prefix retention-gap identity and stopping rule are clear and correct under their stated assumptions. The paper also explains when the unimodality conclusion can fail.

  3. Experiments across ten Terminal-Bench tasks favor shallower scaffolding, with 38% versus 30% success at sizes 2 and 10. Calibration, alternative judges, and human agreement strengthen the hallucination evidence.

Critical weaknesses
  1. The stated assumptions do not justify two general design claims. Increasing guidance strength need not preserve the sign of the retention gap: even exponential weighting can favor recoverable paths at low strength and harmful paths at high strength if the latter contain a rare, high-scoring trajectory. Likewise, lower pointwise autonomous tail risk does not guarantee earlier optimal stopping without an ordering assumption on marginal risk reductions. These issues limit the general interpretations while leaving the exact identity and conditional stopping theorem intact.

  2. The real-task evidence is difficult to assess. The main sweep uses one run per task/depth, and the expanded success comparison lacks explicit trial counts and uncertainty. Reported aligned hallucination rates differ substantially, while model labels and filtering comparisons need clarification. These differences may reflect separate configurations, which should be specified before combining the results as evidence for one mechanism.

Questions for authors
  1. What additional assumptions justify sign-preserving amplification and earlier stopping as autonomous capability improves? If these assumptions do not hold, will the general claims be narrowed?
Reviewer_KVmfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work formulates harness engineering as an alignment problem by modeling how harnesses shape the inference-time trajectories of agents. It introduces partial harnessing, a design strategy that specifies only the initial stages and leaves the rest to the agent. The authors empirically validate their theoretical predictions on harness design using synthetic data and Terminal-Bench.

Critical strengths
  • The empirical results are useful because it shows that in some cases, partial harnessing can outperform full specification. This is because an agent can sometimes benefit from the shorter structure and avoid getting stuck in a longer structured workflow.
Critical weaknesses
  • It is hard to follow some of the concepts. For example, Fig. 1 is not explicitly discussed in the text and the first two alignment principles (granularity and guidance) are not fully described until later in the paper (Sec. 4).
  • The paper would benefit from a discussion of limitations in the main text. For example, what is the vulnerability of the “slice rule” failing?
  • The "homogeneous progress slice" assumption should be discussed in the text, and especially the implications of treating each scaffolded stage as having an independent success prob.
Questions for authors
  • How are confidence intervals obtained in Fig. 2?
Reviewer_UjnvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper provides the grounding for partial harness engineering, where it models how the task decomposition and guidance affect an LLM agent's success. Misalignment in either decomposition or guidance turns scaffolding into a reliability bottleneck rather than an aid.

Critical strengths
  1. The problem setup is timely and important. Harness engineering can be very exhaustive and expensive. The paper provides insights on what to focus on during harness engineering.
  2. Both theoretical and empirical analyses make the paper more comprehensive and complete.
Critical weaknesses
  1. Hard to follow. I feel like the paper is overly complicating concepts such as recoverability. Maybe giving a more natural explanation in the main text could have been better for the readers.
  2. I am not sure why task decomposition is part of the "harness". My understanding is that the harness provides an interface for the model to interact with the environment, managing context, ... rather than task-related jobs like task decomposition. The task decomposition the authors are discussing is can be part of harness but it is smaller meaning such as agentic workflow [1] and a planning-and-execution-style workflow [2,3]. The experiment setting section explicitly mentions that the harness acts as a high-level planner, which I believe should be part of the model's job.
  3. Authors provide some theoretical explanation on the granularity and guidance alignment. But the empirical results and conclusion of the paper seem somewhat predictable. I think if the authors could provide any empirical results on the sweet spot on the granularity and guidance, it could become a stronger paper.

[1] Zhang et al, AFlow: Automating Agentic Workflow Generation, ICLR 2025
[2] Prasad et al, Adapt: As-needed decomposition and planning with language models, NAACL 2024 Findings
[3] Erdogan et al, Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks, ICML 2025

Questions for authors

Address above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.