acceptodds
Under review as a conference paper at ICLR 2027

Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable

Abstract

The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, execution environments, and application requirements change, the harness must be continually modified to add capabilities or adapt existing behaviors. Before a human developer or coding agent can make such a change, they must identify all code locations that implement the target behavior. This is difficult because production harnesses are often large, tightly coupled, and behaviorally distributed across files, functions, execution stages, and state transitions. Modification requests describe what the system should do, whereas repositories are organized by files, functions, and modules. They do not directly reveal the complete implementation path of a behavior. Existing approaches to code search, repository indexing, and long-context processing make code easier to inspect, but they still leave developers and coding agents to recover this mapping themselves. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase through static program analysis and LLM-assisted behavioral structuring. The Handbook organizes implementation knowledge around system behaviors and links each behavior to the corresponding source code. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides coding agents from high-level behavior descriptions to relevant implementation details and verifies candidate locations against the current source. We evaluate Harness Handbook on diverse modification requests from two open-source agent harnesses by comparing planning with and without Handbook access. Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens. The largest gains appear for changes involving scattered implementation sites, rarely executed code paths, and cross-module interactions. These findings indicate that evolving complex agentic systems depends not only on generating edits, but also on determining where those edits should be made.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Harnesses, Coding Agents, Behavior Localization, Repository Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eq9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies behavior localization: finding all distributed implementation sites relevant to a natural-language request to modify an agent harness. Harness Handbook combines static program facts with LLM-assisted behavioral organization to produce a three-level hierarchy, shared-state links, and source-backed locators. Behavior-Guided Progressive Disclosure navigates from requested behavior to candidate sites, expands through state/call relationships, and verifies them against current source. A broader workflow includes editing and handbook resynchronization. The experiments specifically evaluate read-only localization and planning on Terminus-2 and Codex, with 30 requests per repository and one planner model. The handbook improves model-judged plan quality and agreement with model-generated reference locations, and reduces planner tokens by 8.6% and 12.7%.

Critical strengths
  • The motivation identifies a concrete failure that the representation is designed to address. The completion-confirmation example requires coordinated changes to checking, initialization, and reset logic. Stage organization and shared-state links plausibly recover these nonadjacent responsibilities, and the source-grounded walkthrough demonstrates how such navigation can work. Requests include cross-file and search-hostile cases with no target-location hints, so the evaluation does exercise the problem described in the introduction (Sections 1, 3, and 4; Appendix E).
  • The experiments provide consistent initial evidence for planning assistance across two substantially different repositories. The same planner is compared on matched requests in a small Python harness and a large Rust repository. Three judges build answer keys before inspecting plans, and reference-location precision/recall complement subjective plan scores. The reported F1 improvements and reduced planner-token use support the usefulness of the complete handbook-assisted workflow. These are meaningful controls for a localization study, even though they do not isolate the behavior/state innovation.
Critical weaknesses
  • The experiments do not establish the specific premise that behavioral organization solves omissions left by structural repository representations. The motivation argues that file/symbol-oriented tools leave distributed behavioral responsibilities difficult to recover. However, the baseline only uses direct reading, in-file search, and directory listing, while the assisted arm gains a precomputed handbook and navigation instructions. A comparably constructed repository map or hierarchical summary would test that premise more directly. There is also no component ablation holding summaries fixed while removing shared-state links, stage organization, or call expansion. The walkthrough is a useful feasibility example, but does not show how often the proposed mechanism recovers otherwise missed initialization/reset/fallback sites. Consequently, the main results support the assistance package, while leaving the paper's distinctive explanation and innovation insufficiently tested.
  • The evaluation measures reference agreement more directly than complete coverage of a requested behavior. The two repositories and 60 requests offer useful scale and language variation, but the required sites and plan judgments are generated by models, without independent validation that the reference sets contain all necessary responsibilities or accept alternative valid scopes. Multiple judges reduce evaluator dependence but can still share omissions. Table 1 illustrates the unresolved completeness issue: for Codex against GPT-5.5 references, file recall changes from 49.4% to 49.8%, while symbol F1 improves from 43.8% to 51.2%. This establishes useful agreement gains, not that the motivating missing-site failure is solved. An independently checked subset with per-request complete-site coverage and omission categories would be a necessary direct test of that claim.
  • Writing quality: excessive defensive assurances about source grounding and evaluation scope. Sections 3.1–3.3 repeatedly stress that the repository is authoritative, static facts cannot be changed by the model, uncertain units are not guessed, and invalid locators are frozen. Section 4.1 adds a separate defense of evaluating plans rather than executed patches. Examples include “statically extracted facts remain authoritative” and “rather than being assigned guessed targets.” These explanations contain relevant design information, but their repetition makes the manuscript read like an anticipatory rebuttal instead of a concise account of the method. The repeated boundary-setting can give the impression of protesting too much and draw attention away from whether behavioral assignments and complete localization are actually validated. The authors should state the safeguards and scope once, retain only necessary local reminders, and devote more of the exposition to the evidence supporting the behavioral representation. This is the consequence of the author's excessive reliance on GPT writing, and the author should optimize these writing sections to make their paper statements more persuasive, thereby improving the quality of their writing.
Questions for authors
  • Does the advantage persist against a repository map or hierarchical summary built with comparable preprocessing effort and the same navigation guidance? This tests the claimed gap between behavioral and structural organization.
  • With the same source summaries and planner, what happens when state links, stage organization, or call expansion are removed individually? In particular, do state links measurably reduce omissions of initialization/reset or mirrored/fallback sites?
  • On an independently source-checked subset, how often does each method identify every necessary site for the request, and which kinds of responsibility remain missing? How are valid alternative localization scopes handled?
1 comment
AuthorsOpenReview ↗

To Reviewer eq9L

Q1: The experiments do not establish the specific premise that behavioral organization solves omissions left by structural repository representations.

A1: The Handbook makes explicit a relationship central to edit planning: which dispersed code sites jointly implement the behavior named in a change request. Our baseline uses the same planner and can read, search, and browse the repository. Adding the Handbook and BGPD improves both file- and symbol-level F1 on Terminus-2 and Codex while reducing planner tokens per request by 8.6% and 12.7%, respectively. These results establish the benefit of the combined workflow over direct repository exploration. We will distinguish this empirical finding from the design rationale for organizing code by behavior.

Q2: The evaluation measures reference agreement more directly than complete coverage of a requested behavior.

A2: The Codex/GPT-5.5 result illustrates where the gain occurs. File recall changes only from 49.4% to 49.8%, but precision rises from 53.0% to 62.2%. Thus, the planner covers a similar fraction of reference files while a larger share of its selected files falls within the reference scope. Symbol-level F1 also rises from 43.8% to 51.2%. Across both repositories, both independently generated references, and both granularities, F1 improves in all eight comparisons, by 5.0–18.8 percentage points. We will describe these results as improved agreement on edit scope and retain complete coverage as the task objective.

Q3: Writing quality: excessive defensive assurances about source grounding and evaluation scope.

A3: We will consolidate the construction-time validation rules in §3.2. Section 3.3 will focus on how the planner follows stage and state links to candidate sites and checks their roles in the source. Section 4.1 will state once that the evaluation scores edit plans. These edits will give more space to the central design question: once the planner finds the obvious edit site, how does it discover which other sites the change affects?

Q4: Does the advantage persist against a repository map or hierarchical summary built with comparable preprocessing effort and the same navigation guidance? This tests the claimed gap between behavioral and structural organization.

A4: The current comparison measures the benefit of the full workflow. Performance against a map or summary under those matched conditions remains unmeasured. The specific design choice at issue is that the Handbook organizes source units by execution stage and explicitly links sites that read or write the same state across stages. Once the planner reaches a behavior’s main entry point, those links give it additional sites to inspect. A matched comparison would isolate the value of that organization. The token savings we report apply to per-request planning cost.

Q5: With the same source summaries and planner, what happens when state links, stage organization, or call expansion are removed individually? In particular, do state links measurably reduce omissions of initialization/reset or mirrored/fallback sites?

A5: Appendix E records how a state link guided a concrete localization decision. The request required three consecutive confirmations before submission. The stage hierarchy led the planner to the completion gate, and the state register exposed initialization and per-run reset sites. After checking all seven occurrences of the state variable, the planner included resets both when a non-confirmation turn interrupts the sequence and when a new run begins. These sites explain why changing the submission threshold alone would leave the request incomplete. The trace makes the mechanism inspectable in this case. We evaluated the components jointly, so the results do not quantify their individual contributions or omission rates for mirrored and fallback paths.

Q6: On an independently source-checked subset, how often does each method identify every necessary site for the request, and which kinds of responsibility remain missing? How are valid alternative localization scopes handled?

A6: Table 1 provides a per-request measure of complete reference misses. On Codex, the share of requests with zero symbol overlap falls from 44.4% to 18.5% against Opus 4.8 and from 28.6% to 21.4% against GPT-5.5. Thus, with either reference, fewer plans miss every reference symbol. We did not independently adjudicate all necessary sites on a subset, so we cannot report the fraction of plans that found every site or the frequency of each omitted responsibility. For alternative implementations, different edits at the same files or symbols can receive the same localization score. A valid strategy that uses different sites may receive a lower score. We will clarify this distinction alongside Table 1.

Reviewer_vqP5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies behavior localization problem for agent-harness evolution, which requires agents to identify the distributed implementation sites relevant to a natural-language change request. Motivated by this research question, this work proposed Harness Handbook, which combines static program analysis and LLM-assisted organization to connect execution stages and shared states to source code. The BehaviorGuided Progressive Disclosure (BGPD) is also introduced to navigate and verify candidate locations. Across 60 modification requests on Terminus-2 and Codex, this work verify the effectiveness of proposed methods in generating modification plans.

Critical strengths
  1. This work focuses on a practical problem
  2. This work deducts expeiments on two repositories in different scales, verify the effectness and generability of proposed method.
Critical weaknesses
  1. The announced contribution is insufficiently isolated. The proposed workflow is only compared with vanilla repository exploration, lacking comparison between any other baselines, and the ablation studies on components are incompleted, which make the annonced contributions lack credibility.
  2. The evaluation methods both rely on reference plans and answer keys generated by the LLM, without reported expert or execution-based validation. The localization correctness comparing with these models is unreliable, especially when alternative valid implementations require different edit locations.
  3. The reported results need clarification. In section 4.2, the reported results of codex is different from Figure 5(a), which makes the conclusion unreliable.
Questions for authors
  1. Under matched models and budgets, how does Harness Handbook compare with hierarchical repository summaries or structural indexes? Which gains come from the execution-stage hierarchy, shared-state representation, and BGPD navigation?
  2. How can Codex’s overall win-rate improvement be reconciled with the larger improvements reported for every request type in Figure 5(a)? Please report the valid comparison counts, wins, losses, ties, and aggregation procedure.
  3. What's the difference between behavior localization and existed concept/feature location methods(see reference 1)? Which harness-specific challenges motivate the proposed formulation?

reference:

  1. Meghan Revelle and Denys Poshyvanyk. 2009. An Exploratory Study on Assessing Feature Location Techniques. In Proceedings of the 17th IEEE International Conference on Program Comprehension (ICPC), pp. 218–222.
2 comments
AuthorsOpenReview ↗

To Reviewer vqP5 (1/2)

Q1: The announced contribution is insufficiently isolated. The proposed workflow is only compared with vanilla repository exploration, lacking comparison between any other baselines, and the ablation studies on components are incompleted, which make the annonced contributions lack credibility.

A1: Our comparison measures the effect of giving the same coding agent access to the Handbook and its BGPD navigation instructions. Both arms use the same planner, requests, repository snapshots, read-only tools, and decoding settings (§4.1). Across both repositories, the combined workflow improves localization agreement with the reference plans while reducing planner token use by 12.7% on Codex and 8.6% on Terminus-2. We will clarify that these results measure the benefit of the combined workflow over direct repository exploration. Estimating each component’s contribution would require separate ablations.

Q2: The evaluation methods both rely on reference plans and answer keys generated by the LLM, without reported expert or execution-based validation. The localization correctness comparing with these models is unreliable, especially when alternative valid implementations require different edit locations.

A2: A valid implementation that uses different edit sites can receive a lower overlap score. To assess sensitivity to the choice of reference, we evaluate against plans generated independently by Opus 4.8 and GPT-5.5. Across both repositories, both references, and both localization granularities, F1 improves in all eight comparisons, with gains of 5.0–18.8 points (Table 1). For plan scoring, each judge builds its answer key from the request and source before seeing either plan. These results support improved agreement with reference edit scopes. Whether the resulting patches satisfy the requests without regressions remains untested.

Q3: The reported results need clarification. In section 4.2, the reported results of codex is different from Figure 5(a), which makes the conclusion unreliable.

A3: Figure 5(a) reports Codex gains of 26.7, 16.3, and 16.7 percentage points for Query, Cross-file, and Search-hostile requests, respectively. All three exceed the 10.0-point overall gain in Figure 3 and §4.2. The range stated in §4.2 also overlooks the 3.3-point gain shown for Terminus-2 Query requests. We will check both figures and the text against the original scores before drawing conclusions by request type. The file- and symbol-level F1 results in Table 1 are computed separately from these win rates.

AuthorsOpenReview ↗

To Reviewer vqP5 (2/2)

Q4: Under matched models and budgets, how does Harness Handbook compare with hierarchical repository summaries or structural indexes? Which gains come from the execution-stage hierarchy, shared-state representation, and BGPD navigation?

A4: We did not run a matched-budget comparison with hierarchical summaries or structural indexes. Appendix E illustrates each component’s role in a recorded run. For a request requiring three consecutive completion confirmations, the execution-stage hierarchy directs the planner to the completion gate. The shared-state register connects that gate to initialization and reset sites. BGPD guides the planner through those links and back to the source, where it checks all seven occurrences of the completion flag before proposing edits. The example shows how the components work together. The quantitative comparison measures their joint effect.

Q5: How can Codex’s overall win-rate improvement be reconciled with the larger improvements reported for every request type in Figure 5(a)? Please report the valid comparison counts, wins, losses, ties, and aggregation procedure.

A5: Each valid judge–request pair contributes one outcome. A score difference of at least 3 points on the 0–100 scale counts as a win for the higher-scoring arm. Smaller differences count as ties, and ties remain in the denominator. The overall win-rate gap must therefore be the average of the three request-type gaps, weighted by their valid comparison counts. No such weighting can produce a 10.0-point overall gap when the smallest subgroup gap is 16.3 points. We will reconcile the figure and text against the original scores and report the verified number of valid comparisons, wins, losses, and ties for each request type.

Q6: What's the difference between behavior localization and existed concept/feature location methods (see reference 1)? Which harness-specific challenges motivate the proposed formulation?

A6: We view behavior localization as a harness-specific application of feature location and change-impact analysis. The question is which sites must change together when a requested behavior spans several stages of an agent’s execution. For example, adding a per-command environment override in Codex requires tracing the new field through both shell-tool interfaces to process creation. Locating one tool definition gives the planner a starting point, but it can still miss the parallel interface or the code that passes the field to the new process. The Handbook makes these relationships explicit by organizing source locations around execution stages and shared state. We will clarify this connection to prior work in §2.

Reviewer_6qfNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how coding agents identify all implementation sites affected by a requested change to an agent harness. It introduces Harness Handbook, which combines static program analysis with LLM-generated descriptions to organize a repository into system, execution-stage, and implementation-level views. A complementary register view connects reads, writes, and resets of shared state across stages.

The authors propose Behavior-Guided Progressive Disclosure, a workflow that navigates this hierarchy, expands candidate locations through state and call relationships, and verifies them against current source before producing an edit plan. The framework also describes automatic handbook updates after repository changes.

Evaluation uses 60 modification requests across Terminus-2 and Codex, covering existing-behavior changes, cross-file additions, and difficult-to-search dependencies. A single DeepSeek-V4-Pro planner is compared with and without handbook guidance. Three models judge plan quality, while two models supply independent reference plans for file- and symbol-level localization. Reported localization F1 improves by 5.0–18.8 percentage points, and planner token usage decreases by 12.7% on Codex and 8.6% on Terminus-2.

The main contribution is a behavior-oriented repository representation coupled with a guided navigation workflow. The results provide preliminary evidence of improved planning efficiency and localization, while successful code modification and handbook reliability across evolving repository versions remain untested.

Critical strengths

The representation addresses a concrete and important failure mode in repository modification. I find the combination of execution-stage organization and shared-state tracking well motivated. A requested behavior change often affects initialization, updates, resets, and fallback paths beyond the most obvious implementation site. The handbook makes these relationships explicit and provides a systematic way to inspect them.

Critical weaknesses

The evaluation provides limited evidence that the identified edit scope is correct and complete. Both evaluation approaches rely on model-generated references: judges construct answer keys and score plans, while localization metrics measure agreement with plans produced by stronger models. These references may omit necessary sites or choose one of several valid implementation strategies. Consequently, higher agreement does not necessarily establish that the proposed changes would satisfy the request without regressions. The sensitivity to the reference model is noticeable: on Codex, file-level recall improves by 14.3 points against Opus but only 0.4 points against GPT-5.5. (Table 1)

Questions for authors

Can localization quality be validated independently of model-generated plans? Could experts adjudicate required sites, or could a subset of plans be executed and tested? How much do the two reference models agree, and how are alternative valid edit scopes handled?

1 comment
AuthorsOpenReview ↗

To Reviewer 6qfN

Q1: The evaluation provides limited evidence that the identified edit scope is correct and complete. Both evaluation approaches rely on model-generated references: judges construct answer keys and score plans, while localization metrics measure agreement with plans produced by stronger models. These references may omit necessary sites or choose one of several valid implementation strategies. Consequently, higher agreement does not necessarily establish that the proposed changes would satisfy the request without regressions. The sensitivity to the reference model is noticeable: on Codex, file-level recall improves by 14.3 points against Opus but only 0.4 points against GPT-5.5. (Table 1)

A1: We agree that the size of the gain depends on the reference. On Codex, file-level recall improves by 14.3 points against Opus 4.8 and only 0.4 points against GPT-5.5. Against GPT-5.5, however, precision improves by 9.2 points: the planner covers a similar share of reference files while selecting a higher proportion of files supported by the reference. Across both repositories, both granularities, and both reference models, F1 improves in all eight comparisons, by 5.0–18.8 points (Table 1). This supports improved localization agreement across references. Patch correctness and regressions require separate validation.

Q2: Can localization quality be validated independently of model-generated plans?

A2: Individual localization decisions can be checked directly against the source. In Appendix E, the request requires three consecutive completion confirmations. The planner traces all seven occurrences of _pending_completion and identifies where the state is initialized, updated, and reset. The reset sites explain why changing the submission condition alone could allow confirmations to carry over across interrupted sequences or separate runs. Readers can verify this reasoning directly in the code. The aggregate metrics remain reference-based.

Q3: Could experts adjudicate required sites, or could a subset of plans be executed and tested?

A3: Experts could check whether a plan covers the sites required by its chosen implementation strategy. Executing a patch and running behavioral and regression tests would assess the resulting behavior. Our evaluation focuses on the preceding planning stage, holding the planner, requests, repository snapshot, and tool permissions fixed across both arms (§4.1). We will clarify that our conclusions concern localization and edit-plan quality.

Q4: How much do the two reference models agree, and how are alternative valid edit scopes handled?

A4: We did not directly measure agreement between Opus 4.8 and GPT-5.5. Table 1 compares the planner with each reference separately, so their pairwise overlap cannot be inferred from it. The metrics compare file or symbol locations, allowing different edits at the same sites to receive the same localization score. A valid strategy that uses different sites may receive a lower score. We will clarify that Table 1 measures agreement with each reference scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.