- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies behavior localization: finding all distributed implementation sites relevant to a natural-language request to modify an agent harness. Harness Handbook combines static program facts with LLM-assisted behavioral organization to produce a three-level hierarchy, shared-state links, and source-backed locators. Behavior-Guided Progressive Disclosure navigates from requested behavior to candidate sites, expands through state/call relationships, and verifies them against current source. A broader workflow includes editing and handbook resynchronization. The experiments specifically evaluate read-only localization and planning on Terminus-2 and Codex, with 30 requests per repository and one planner model. The handbook improves model-judged plan quality and agreement with model-generated reference locations, and reduces planner tokens by 8.6% and 12.7%.
Critical strengths
- The motivation identifies a concrete failure that the representation is designed to address. The completion-confirmation example requires coordinated changes to checking, initialization, and reset logic. Stage organization and shared-state links plausibly recover these nonadjacent responsibilities, and the source-grounded walkthrough demonstrates how such navigation can work. Requests include cross-file and search-hostile cases with no target-location hints, so the evaluation does exercise the problem described in the introduction (Sections 1, 3, and 4; Appendix E).
- The experiments provide consistent initial evidence for planning assistance across two substantially different repositories. The same planner is compared on matched requests in a small Python harness and a large Rust repository. Three judges build answer keys before inspecting plans, and reference-location precision/recall complement subjective plan scores. The reported F1 improvements and reduced planner-token use support the usefulness of the complete handbook-assisted workflow. These are meaningful controls for a localization study, even though they do not isolate the behavior/state innovation.
Critical weaknesses
- The experiments do not establish the specific premise that behavioral organization solves omissions left by structural repository representations. The motivation argues that file/symbol-oriented tools leave distributed behavioral responsibilities difficult to recover. However, the baseline only uses direct reading, in-file search, and directory listing, while the assisted arm gains a precomputed handbook and navigation instructions. A comparably constructed repository map or hierarchical summary would test that premise more directly. There is also no component ablation holding summaries fixed while removing shared-state links, stage organization, or call expansion. The walkthrough is a useful feasibility example, but does not show how often the proposed mechanism recovers otherwise missed initialization/reset/fallback sites. Consequently, the main results support the assistance package, while leaving the paper's distinctive explanation and innovation insufficiently tested.
- The evaluation measures reference agreement more directly than complete coverage of a requested behavior. The two repositories and 60 requests offer useful scale and language variation, but the required sites and plan judgments are generated by models, without independent validation that the reference sets contain all necessary responsibilities or accept alternative valid scopes. Multiple judges reduce evaluator dependence but can still share omissions. Table 1 illustrates the unresolved completeness issue: for Codex against GPT-5.5 references, file recall changes from 49.4% to 49.8%, while symbol F1 improves from 43.8% to 51.2%. This establishes useful agreement gains, not that the motivating missing-site failure is solved. An independently checked subset with per-request complete-site coverage and omission categories would be a necessary direct test of that claim.
- Writing quality: excessive defensive assurances about source grounding and evaluation scope. Sections 3.1–3.3 repeatedly stress that the repository is authoritative, static facts cannot be changed by the model, uncertain units are not guessed, and invalid locators are frozen. Section 4.1 adds a separate defense of evaluating plans rather than executed patches. Examples include “statically extracted facts remain authoritative” and “rather than being assigned guessed targets.” These explanations contain relevant design information, but their repetition makes the manuscript read like an anticipatory rebuttal instead of a concise account of the method. The repeated boundary-setting can give the impression of protesting too much and draw attention away from whether behavioral assignments and complete localization are actually validated. The authors should state the safeguards and scope once, retain only necessary local reminders, and devote more of the exposition to the evidence supporting the behavioral representation. This is the consequence of the author's excessive reliance on GPT writing, and the author should optimize these writing sections to make their paper statements more persuasive, thereby improving the quality of their writing.
Questions for authors
- Does the advantage persist against a repository map or hierarchical summary built with comparable preprocessing effort and the same navigation guidance? This tests the claimed gap between behavioral and structural organization.
- With the same source summaries and planner, what happens when state links, stage organization, or call expansion are removed individually? In particular, do state links measurably reduce omissions of initialization/reset or mirrored/fallback sites?
- On an independently source-checked subset, how often does each method identify every necessary site for the request, and which kinds of responsibility remain missing? How are valid alternative localization scopes handled?
To Reviewer eq9L
Q1: The experiments do not establish the specific premise that behavioral organization solves omissions left by structural repository representations.
A1: The Handbook makes explicit a relationship central to edit planning: which dispersed code sites jointly implement the behavior named in a change request. Our baseline uses the same planner and can read, search, and browse the repository. Adding the Handbook and BGPD improves both file- and symbol-level F1 on Terminus-2 and Codex while reducing planner tokens per request by 8.6% and 12.7%, respectively. These results establish the benefit of the combined workflow over direct repository exploration. We will distinguish this empirical finding from the design rationale for organizing code by behavior.
Q2: The evaluation measures reference agreement more directly than complete coverage of a requested behavior.
A2: The Codex/GPT-5.5 result illustrates where the gain occurs. File recall changes only from 49.4% to 49.8%, but precision rises from 53.0% to 62.2%. Thus, the planner covers a similar fraction of reference files while a larger share of its selected files falls within the reference scope. Symbol-level F1 also rises from 43.8% to 51.2%. Across both repositories, both independently generated references, and both granularities, F1 improves in all eight comparisons, by 5.0–18.8 percentage points. We will describe these results as improved agreement on edit scope and retain complete coverage as the task objective.
Q3: Writing quality: excessive defensive assurances about source grounding and evaluation scope.
A3: We will consolidate the construction-time validation rules in §3.2. Section 3.3 will focus on how the planner follows stage and state links to candidate sites and checks their roles in the source. Section 4.1 will state once that the evaluation scores edit plans. These edits will give more space to the central design question: once the planner finds the obvious edit site, how does it discover which other sites the change affects?
Q4: Does the advantage persist against a repository map or hierarchical summary built with comparable preprocessing effort and the same navigation guidance? This tests the claimed gap between behavioral and structural organization.
A4: The current comparison measures the benefit of the full workflow. Performance against a map or summary under those matched conditions remains unmeasured. The specific design choice at issue is that the Handbook organizes source units by execution stage and explicitly links sites that read or write the same state across stages. Once the planner reaches a behavior’s main entry point, those links give it additional sites to inspect. A matched comparison would isolate the value of that organization. The token savings we report apply to per-request planning cost.
Q5: With the same source summaries and planner, what happens when state links, stage organization, or call expansion are removed individually? In particular, do state links measurably reduce omissions of initialization/reset or mirrored/fallback sites?
A5: Appendix E records how a state link guided a concrete localization decision. The request required three consecutive confirmations before submission. The stage hierarchy led the planner to the completion gate, and the state register exposed initialization and per-run reset sites. After checking all seven occurrences of the state variable, the planner included resets both when a non-confirmation turn interrupts the sequence and when a new run begins. These sites explain why changing the submission threshold alone would leave the request incomplete. The trace makes the mechanism inspectable in this case. We evaluated the components jointly, so the results do not quantify their individual contributions or omission rates for mirrored and fallback paths.
Q6: On an independently source-checked subset, how often does each method identify every necessary site for the request, and which kinds of responsibility remain missing? How are valid alternative localization scopes handled?
A6: Table 1 provides a per-request measure of complete reference misses. On Codex, the share of requests with zero symbol overlap falls from 44.4% to 18.5% against Opus 4.8 and from 28.6% to 21.4% against GPT-5.5. Thus, with either reference, fewer plans miss every reference symbol. We did not independently adjudicate all necessary sites on a subset, so we cannot report the fraction of plans that found every site or the frequency of each omitted responsibility. For alternative implementations, different edits at the same files or symbols can receive the same localization score. A valid strategy that uses different sites may receive a lower score. We will clarify this distinction alongside Table 1.