- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formalizes context injection for frozen LLM agents. A catalog of gated, hand-authored "factoids" emits short claims, which are embedded, weighted by Loom-style cosine reweighting, and passed to the agent. The contributions are threefold. Catalog utility is scored as , decomposed into coverage gap, redundancy (negative under synergy), and localization. A tight error bridge (Theorem 1) bounds the distance between averaged claim embeddings by , which then bounds catalog-omission error. The consensus weights are read through EM, Barber–Agakov, and m-projections, explicitly as analogies. On 50 SWE-bench Verified instances, the catalog significantly improves edit-file recall (+0.118), but resolve-given-submit stays near 40% across all arms. On 30 τ² airline tasks, refusals improve and writes get worse.
Critical strengths
-
A robust, pre-registered empirical finding. Across all slates, including the gold file list, the pass rate of submitted patches stays near 40%; better context raises how often the agent submits, not whether the patch is correct. The authors forecast the temperature-0 results before scoring them, and all four forecasts landed within a reasonable range. This matters because it cleanly separates where context injection helps (localization) from where it does not (correctness), which is a transferable insight made credible by the pre-registration.
-
Clear scoping of the theory. The paper clearly separates tight, proved results from analogies and states where no bounds exist. This matters because the claims it does make are trustworthy and auditable, even though their scope is limited.
Critical weaknesses
-
The theory is never tested in the experiments. The paper proposes several measures for judging a catalog (coverage, redundancy, and an error bound), but none of them are actually computed in the experiments. The consensus weighting, one of the main contributions, is never ablated, and the paper does not explain how the agent uses the weights. So we cannot tell whether the framework helps anyone build a better catalog. In addition, the theory only bounds errors from missing claims, while the experiments show the real bottleneck is elsewhere: the agent writes incorrect patches even with perfect context, and the theory says nothing about that stage. This is the main reason for my score.
-
Little theoretical novelty and a small evaluation. Most of the formal tools are standard results, and the main theorem follows directly from the triangle inequality. The observation that redundancy can be negative (synergy) is well known, and the relevant literature is not cited:
- McGill (1954), Multivariate information transmission, Psychometrika (interaction information).
- Bell (2003), The co-information lattice, ICA 2003.
- Williams & Beer (2010), Nonnegative decomposition of multivariate information, arXiv:1004.2515 (partial information decomposition, which separates redundancy from synergy, addressing the paper's own caveat that a near-zero value can hide both).
- Timme et al. (2014), Synergy, redundancy, and multivariate information measures: an experimentalist's perspective, J. Comput. Neurosci.
The experiments also lack comparisons to established code-localization methods, which are the natural baselines for the edit-recall result:
- Xia et al. (2024), Agentless: Demystifying LLM-based Software Engineering Agents, arXiv:2407.01489.
- Zhang et al. (2024), AutoCodeRover: Autonomous Program Improvement, ISSTA 2024.
- Chen et al. (2025), LocAgent: Graph-Guided LLM Agents for Code Localization, ACL 2025.
The evaluation itself is small: one model, 50 SWE-bench instances (mostly Django),
Questions for authors
- How does the agent actually use the consensus weights? Does an ablation against uniform weights or no reweighting show any effect?
- Was the catalog developed or tuned on any of the 50 test instances?
- Can you compute your own measures (coverage, redundancy, catalog recall) for a few catalog variants and show they predict edit recall?
- Does the catalog's advantage survive a length-matched baseline, and what is its edit precision?
We thank Reviewer
Z8y1for a sharp review, noting our pre-registered temperature-0 forecast and clear scoping of proved bounds vs. analogies, and pointing us to the synergy/PID and code-localization literature. You wrote: "The theory is never tested in the experiments... This is the main reason for my score." We computed all proposed measures in our experiments, showed how they improve catalogs, and added all 7 citations.1. Testing the Theory & Improving Catalogs (W1 & Q3)
In Sec. 7 Table 2, App. E.1 Tables 3-4, and App. F.1 Table 6, we compute Γ, Γ/H(Z), 1−RecallF, Red, I(Z;Si), Loc, Thm. 1 slack, and Prop. 1 across
Empty/BM25/Catalog/Goldand 3 versions (v1/v2/v3):Empty/BM25/Catalog/Gold), 1−RecallF (1.000/0.470/0.315/0.000) and Γ/H(Z) (1.000/0.175/0.000onEmpty/Catalog/Gold) drop, tracking edit recall (0.497/0.559/0.616/0.598).issue_traceback_files = 0.000bits, matchingsite-packagespaths) vs. strong rules (likely_edit_files = 3.228,identifier_def_sites = 2.846bits).Hull (ii):0.70→0.000):1.000) of 3,435 nonempty episodes (min slack = 0.000; exact0on all 500 SWE gold slates, 52% SWE patches, 67% airline writes).340/490SWE runs at RecallF=1 and73/200airline writes because prose/multi-item rules break Assumption (ii): 100% of matched atomic claims sit outside conv(D(x)) (0.60SWE,0.56airline).Hull (ii) = 0.70, 85% hold rate). Embedding each rule's short ID in D(x) while printing full text in the slate droppedHull (ii)to0.000, nearly halved hull error (0.57→0.30), and gave 100% Prop. 1 hold rate.v1/v2/v3, cutting Γ^/H^(Z) (0.23→0.21) lowered Thm. 1 error (0.160→0.145), raisingCompile@1(+5to+7 ppsingle-turn;+3.4to+8.9 ppon all 4 agents, beating the gold API-name oracle on all 4) andPass@1(+5to+6 ppsingle-turn; beating empty/BM25 on 3 of 4 agents, tying empty on the 4th; App. G). Stage 3 has no set-embedding bound without strong LLM assumptions; a 120B model + signature rules cuts realization errors.2. Literature & Theorem 1 (W2)
3. Questions 1, 2, and 4
[weight ROLE]in the slate. On SWE (4.90→4.14kept),0/500gold claims are lost; on 10-repeat airline (Table 2: writes0.911vs.0.897, official0.570vs.0.565), weighting drops65/76(85.5%) falsepolicy_decisionfirings on writes; on OpenRCA Bank (340.9claims/incident),Begleiter et al. (2026)show-5.88 ppstrict accuracy w/o reweighting (38.97%→33.09%).seed=42) and airlinebasesplit were dev slices (SWE-bench Pro / τ2 retail untouched; zero gold leakage). Our sole Django rule fired 0 times; on 19 non-Django tasks (9 repos)Cataloggets0.550vs.0.475BM25 /0.465Empty, and gains hold on >440 held-out tasks of the 500+-task benchmark.Catalogis #1 in precision:0.678vs.0.612Empty (+0.053, p=0.032),0.639BM25,0.671Gold (Recall|submit:0.825vs.0.763, p=0.019;0.783BM25). On slates ≤800 chars (561vs.580),Catalogleads (0.560vs.0.536/0.519); on the 500+-task benchmark it is<50%ofBM25(~1,040vs.~3,000chars) and wins on compile/pass.