acceptodds
Under review as a conference paper at ICLR 2027

AnchorOpt: Structured Runtime Optimization OF LLM Agent Harness

Abstract

Agent harnesses govern consequential runtime decisions around language models, including tool invocation, persistent-state modification, and error recovery. Optimizing these decisions jointly is challenging: intervention placement determines what can be observed and executed, signals and policies must be designed together, and local modifications interact through downstream trajectories. We introduce \anchoropt, a structured runtime optimization framework that decomposes this coupled harness-design problem into tractable local decision problems. First, backward localization identifies consequential decisions from residual failures, while runtime boundaries constrain observable signals and feasible interventions. Second, the optimizer searches for local policies under available signals and expands the signal representation when search stalls. Third, accepted controllers are installed sequentially, with residuals re-mined and candidates evaluated against the evolving incumbent by their downstream task value. The resulting controllers have explicit runtime activation and execution semantics. Experiments on BFCL Agent Memory and AppWorld demonstrate improved task performance and complementarity with global prompt optimization. Additional studies examine repeated-execution reliability and the trade-off between constraint mitigation and intervention cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent harnesses, harness optimization, runtime control, decision-centric AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PKLUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes AnchorOpt that aims to improve LLM agents, by adding learned runtime rules to a fixed agent harness. AnchorOpt is a non-parametric method where the model weights stay frozen. The agent loop is exposed through three hook points, including pre-generation, post-generation but pre-execution, and post-execution. An optimization LLM reads failed trajectories and attributes each failure to an earlier decision, a strategy called "backward localization." It then proposes “anchors” at those hook points. An anchor is a conditional rule (hook point, triggering condition, action). The action is drawn from a fixed vocabulary, such as no-op, reprompt, suppress, substitute, and transform. Each rule may only use information visible at its hook point and actions executable there. Candidate rules are evaluated by whole-corpus success against the current harness and accepted greedily one at a time, after which residual failures are re-mined. When no candidate is accepted, the set of checkable signals is expanded. The authors evaluate on BFCL Agent Memory and AppWorld against GEPA and Self-Harness, with additional studies on repeated-execution reliability.

Critical strengths
  1. The work is well motivated, as harness-level behavior is a real bottleneck for agents beyond the backbone model, and optimizing around a frozen model is practically useful. It is a timely work, especially given that the importance of harness has become a consensus.

  2. The core idea also makes sense. Instead of arbitrary source edits, AnchorOpt constrains LLM-proposed harness changes to strict, conditional hook rules. making it a sensible middle ground between global prompt optimization and open-ended code modification. Having the runtime evaluate the condition and dispatch the action, rather than relying on the host model to follow an instruction, is a meaningful design distinction.

  3. Experiment-wise, the authors have tested multiple baselines and backbones, both open-source and proprietary ones. And I appreciate the candid & explicit appendices.

  4. From a practical deployment angle, using a strong model offline to write rules that a small host then runs without that model (Opus --> Granite) is an appealing use case.

Critical weaknesses
  1. The readability is concerning and the method section is really hard to follow. For example, Sec. 3 introduces many symbols: boundary sets, observation sets, action sets, signal families, feasible controller spaces, portfolios, and marginal values. Most of them are never used to derive anything. In fact, no property or algorithmic step actually depends on them. Before we can clearly assess the value of this work, the clarity && readability issue has to be addressed.

  2. Following weakness 1, I found that the input and output of the pipeline are never stated clearly. It is more concerning than the excessive use of notation, as it fundamentally puts the rationality and clarity of the method at risk. As far as I can tell,

  • Input: a frozen agent, a task set, a human-written environment adapter, an optimization LLM, and a budget.
  • Output: an ordered list of conditional rules installed in that adapter.

The fact that the adapter, which defines the hook points, visible fields, and available actions, is written by hand for each environment is central to what the method can and cannot find. It is mentioned only briefly. The same concept appears under several names (anchor / controller / local controller; boundary / locus / decision point; intervention / operator / action).

  1. The scope is narrower than “harness optimization.” The method adds conditional patches to a fixed agent loop, at three predefined hook points, using five predefined operators and an adapter-defined set of observable fields. It cannot change the components that typically matter most in harness design, such as memory architecture, context management, agent orchestration, tool-use design, or the agent loop itself. Rules are also stateless and single-point. They cannot condition on patterns across steps. The framing as “harness optimization” should be reconciled with this scope.
Questions for authors
  • For each benchmark, who wrote the adapter? Roughly how much code is it, and which observable fields and actions does it expose? Were actions such as the “cross-container merge” and “capacity recovery” pre-existing in the benchmark, or added as part of the adapter?

  • Have you applied a portfolio learned on one host to another host, or one learned on one BFCL scenario to a different scenario? If portability is a claimed contribution, some such experiment seems necessary.

Reviewer_JLumOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalizes runtime optimization as a constrained search over executable local controllers, then combines three optimization strategies to manage the complexity. The strongest aspect is the distinction between: Where an intervention can occur. What information is available there. What action can be executed there. Whether the intervention improves end-to-end task performance. That is a useful formalization of agent-harness optimization.

Critical strengths

This paper formalizes runtime optimization as a constrained search over executable local controllers, then combines three optimization strategies to manage the complexity. The strongest aspect is the distinction between: Where an intervention can occur. What information is available there. What action can be executed there. Whether the intervention improves end-to-end task performance. That is a useful formalization of agent-harness optimization.

Critical weaknesses

Sequential greedy optimization can miss interactions. Equations 1–4 provide a useful formalization, but they do not establish convergence, optimality, or approximation guarantees. That is not necessarily a problem. This can be a systems and empirical optimization paper.

Questions for authors

Great paper formalizes runtime optimization as a constrained search over executable local controllers, then combines three optimization strategies to manage the complexity. one question is evaluate the portfolio is repeatedly modified based on downstream evaluation. This could potentially create a risk of overfitting to the development set through repeated candidate selection. detailed explanation can help, .e.g. A final untouched test set; Multiple random seeds; Confidence intervals; The number of candidate evaluations; Performance after many sequential controller installations.

Reviewer_5kUGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnchorOpt, a way to improve the runtime harness of LLM agents, without touching the model itself. The core idea is to treat harness optimization as a collection of local decisions, where each intervention is tied to a specific runtime boundary, an observable signal, and an action that can be executed.

The optimization procedure has three main parts: backward failure localization, alternating signal and policy search and sequential selection of interventions. The framework first identifies decision that may have contributed to earlier failures, then looks for corrective actions and evaluate how much they affect the downstream task performance. Successful interventions are added to the agent's execution loop as conditional rules, and the process repeats to address remaining failures.

The authors evaluate the framework on BFCL Agent Memory and App world. They use as baselines the native harness, GEPA and Self-Harness. They also include τ\tau- bench experiments on reliability under repeated execution as well as CCTU experiments to examine constraint violations and interventions costs. They also study how the method combines with GEPA by freezing GEPA's optimized prompt and by adding local controllers on top. The authors find them complementary when they target different failure mechanisms. They show gains in several settings, though performance is not uniform across benchmarks and models.

Critical strengths
  • Boundary aware formulation: tying interventions to explicit runtime boundaries rules out infeasible modifications and keeps policies easy to inspect. Installing executables controllers, rather than only editing prompts and separating offline optimisation from deployment are both practical design choices.
  • Insightful failure analysis: the paper shows that the point where a failure surfaces is not always the right place to intervene, as in the memory examples where failures stem from earlier write. The thin-content intervention (App C3) is a good highlight of that: the tool call succeeds but it also returns insufficient content and therefore AnchorOPT learns to trigger an extra merge which improves the adequacy of intermediate information rather than just action validity.
Critical weaknesses
  • Modest gains: AnchorOpt does not consistently beat the baselines (for instance GEPA is clearly better on BFCL with MiniMax: 58.7% vs 46.7%) and on AppWorld with MiniMax, AnchorOpt (60.6%) is below both GEPA (62.8%) and Self-Harness (61.7%). With Qwen, AnchorOpt retains the native harness on BFCL and the AppWorld held-out result is not measured, so one of the three hosts shows no gain on the main benchmarks. Comparisons differ in configurations, repetitions, and budgets. Furthermore, some held-out sets are small and statistical support for the smaller gains is limited.
  • No ablation on the components: for now, it is unclear to me how much backward localization, signal expansion or boundary conditioned search each contribution, and the signal-expansion result comes from a human-guide run. Greedy controller selection may also miss complementary controllers (for instance separate write and retrieval fixes that only help together). The author acknowledges this but testing pairs of individually rejected controllers would clarify its impact.
Questions for authors

1 - Can you compare against GEPA and Self-Harness under matched budgets and settings?

2 - How much does each component contribute to the final performance?

3 - How sensitive is the final portfolio to controller selection order?

4 - What is the full optimization cost relative to deployment gains?

1 comment
AuthorsOpenReview ↗

Response to Weakness 1: Modest gains and inconsistent comparisons, small held-out

We acknowledge that AnchorOpt does not unfiromly outperform baselines. Our contribution is a complementary optimization approach that learns targeted runtime interventions rather than global prompt modifications.

We conducted experiments on three additional hosts, i.e., Granite 4.2-8B, Granite 4.2-30B, and Gemma 4 12B, using the same evaluation split as Granite 4.1-8B, with each host serving as its own optimizer (updated Table 3 in the revised paper).

Table: Success rates (%) and improvements over H0.

ModelSplitH0GEPASelf-HarnessAnchorOpt
Granite 4.2-8BTRAIN+DEV38.9H0 returnedH0 retained49.8 (+10.9)
Held-out19.0H0 returnedH0 retained28.6 (+9.5)
Granite 4.2-30BTRAIN+DEV35.647.9 (+12.2)48.2 (+12.5)48.8 (+13.2)
Held-out25.035.7 (+10.7)38.1 (+13.1)34.5 (+9.5)
Gemma 4 12BTRAIN+DEV25.436.0 (+10.6)H0 retained45.7 (+20.3)
Held-out13.129.8 (+16.7)H0 retained32.1 (+19.0)

All three show statistically significant paired held-out improvements over their native harnesses (p ≤ 0.008), providing additional evidence that AnchorOpt can learn useful runtime controllers across model sizes and families.

Comparison with GEPA. GEPA outperforms AnchorOpt in some settings. We therefore emphasize complementarity. To investigate this, we expanded the composition study in the original paper to additional hosts, combining frozen GEPA prompts with independently learned AnchorOpt portfolios, without further optimization. On Granite 4.2-30B, the combination improves held-out success from 30/84 to 36/84; on Gemma, from 25/84 to 38/84. Together with controller-level studies showing both additional gains and overlapping effects, these findings suggest that complementarity depends on whether local interventions address residual failures not already resolved by the global prompt.

Comparison with Self-Harness. Across six BFCL host configurations (three original and three additional), Self-Harness improves performance in only two: Opus → Granite 4.1-8B and Granite 4.2-30B. This may reflect the difficulty of optimizing open-ended harness edits, whereas AnchorOpt searches a structured space of executable runtime interventions.

Evaluation consistency. In the original paper, MiniMax and Qwen used different BFCL evaluation splits from Granite 4.1-8B. However, for each host model, all methods were evaluated on the same split, ensuring fair within-host comparisons. Our three additional host experiments (including newer Granite 4.2 models) use the same split as Granite 4.1-8B (303 TRAIN+DEV and 84 held-out queries), further improving comparability across hosts.

Limitations. We acknowledge the relatively small held-out sets, variability across hosts, and evaluation protocols. We report paired gains and losses and statistical uncertainty where available, and do not claim that AnchorOpt uniformly outperforms the baselines. The revised paper incorporates these additional comparisons and composition studies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.