- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work proposes AnchorOpt that aims to improve LLM agents, by adding learned runtime rules to a fixed agent harness. AnchorOpt is a non-parametric method where the model weights stay frozen. The agent loop is exposed through three hook points, including pre-generation, post-generation but pre-execution, and post-execution. An optimization LLM reads failed trajectories and attributes each failure to an earlier decision, a strategy called "backward localization." It then proposes “anchors” at those hook points. An anchor is a conditional rule (hook point, triggering condition, action). The action is drawn from a fixed vocabulary, such as no-op, reprompt, suppress, substitute, and transform. Each rule may only use information visible at its hook point and actions executable there. Candidate rules are evaluated by whole-corpus success against the current harness and accepted greedily one at a time, after which residual failures are re-mined. When no candidate is accepted, the set of checkable signals is expanded. The authors evaluate on BFCL Agent Memory and AppWorld against GEPA and Self-Harness, with additional studies on repeated-execution reliability.
Critical strengths
-
The work is well motivated, as harness-level behavior is a real bottleneck for agents beyond the backbone model, and optimizing around a frozen model is practically useful. It is a timely work, especially given that the importance of harness has become a consensus.
-
The core idea also makes sense. Instead of arbitrary source edits, AnchorOpt constrains LLM-proposed harness changes to strict, conditional hook rules. making it a sensible middle ground between global prompt optimization and open-ended code modification. Having the runtime evaluate the condition and dispatch the action, rather than relying on the host model to follow an instruction, is a meaningful design distinction.
-
Experiment-wise, the authors have tested multiple baselines and backbones, both open-source and proprietary ones. And I appreciate the candid & explicit appendices.
-
From a practical deployment angle, using a strong model offline to write rules that a small host then runs without that model (Opus --> Granite) is an appealing use case.
Critical weaknesses
-
The readability is concerning and the method section is really hard to follow. For example, Sec. 3 introduces many symbols: boundary sets, observation sets, action sets, signal families, feasible controller spaces, portfolios, and marginal values. Most of them are never used to derive anything. In fact, no property or algorithmic step actually depends on them. Before we can clearly assess the value of this work, the clarity && readability issue has to be addressed.
-
Following weakness 1, I found that the input and output of the pipeline are never stated clearly. It is more concerning than the excessive use of notation, as it fundamentally puts the rationality and clarity of the method at risk. As far as I can tell,
- Input: a frozen agent, a task set, a human-written environment adapter, an optimization LLM, and a budget.
- Output: an ordered list of conditional rules installed in that adapter.
The fact that the adapter, which defines the hook points, visible fields, and available actions, is written by hand for each environment is central to what the method can and cannot find. It is mentioned only briefly. The same concept appears under several names (anchor / controller / local controller; boundary / locus / decision point; intervention / operator / action).
- The scope is narrower than “harness optimization.” The method adds conditional patches to a fixed agent loop, at three predefined hook points, using five predefined operators and an adapter-defined set of observable fields. It cannot change the components that typically matter most in harness design, such as memory architecture, context management, agent orchestration, tool-use design, or the agent loop itself. Rules are also stateless and single-point. They cannot condition on patterns across steps. The framing as “harness optimization” should be reconciled with this scope.
Questions for authors
-
For each benchmark, who wrote the adapter? Roughly how much code is it, and which observable fields and actions does it expose? Were actions such as the “cross-container merge” and “capacity recovery” pre-existing in the benchmark, or added as part of the adapter?
-
Have you applied a portfolio learned on one host to another host, or one learned on one BFCL scenario to a different scenario? If portability is a claimed contribution, some such experiment seems necessary.
Response to Weakness 1: Modest gains and inconsistent comparisons, small held-out
We acknowledge that AnchorOpt does not unfiromly outperform baselines. Our contribution is a complementary optimization approach that learns targeted runtime interventions rather than global prompt modifications.
We conducted experiments on three additional hosts, i.e., Granite 4.2-8B, Granite 4.2-30B, and Gemma 4 12B, using the same evaluation split as Granite 4.1-8B, with each host serving as its own optimizer (updated Table 3 in the revised paper).
Table: Success rates (%) and improvements over H0.
All three show statistically significant paired held-out improvements over their native harnesses (p ≤ 0.008), providing additional evidence that AnchorOpt can learn useful runtime controllers across model sizes and families.
Comparison with GEPA. GEPA outperforms AnchorOpt in some settings. We therefore emphasize complementarity. To investigate this, we expanded the composition study in the original paper to additional hosts, combining frozen GEPA prompts with independently learned AnchorOpt portfolios, without further optimization. On Granite 4.2-30B, the combination improves held-out success from 30/84 to 36/84; on Gemma, from 25/84 to 38/84. Together with controller-level studies showing both additional gains and overlapping effects, these findings suggest that complementarity depends on whether local interventions address residual failures not already resolved by the global prompt.
Comparison with Self-Harness. Across six BFCL host configurations (three original and three additional), Self-Harness improves performance in only two: Opus → Granite 4.1-8B and Granite 4.2-30B. This may reflect the difficulty of optimizing open-ended harness edits, whereas AnchorOpt searches a structured space of executable runtime interventions.
Evaluation consistency. In the original paper, MiniMax and Qwen used different BFCL evaluation splits from Granite 4.1-8B. However, for each host model, all methods were evaluated on the same split, ensuring fair within-host comparisons. Our three additional host experiments (including newer Granite 4.2 models) use the same split as Granite 4.1-8B (303 TRAIN+DEV and 84 held-out queries), further improving comparability across hosts.
Limitations. We acknowledge the relatively small held-out sets, variability across hosts, and evaluation protocols. We report paired gains and losses and statistical uncertainty where available, and do not claim that AnchorOpt uniformly outperforms the baselines. The revised paper incorporates these additional comparisons and composition studies.