acceptodds
Under review as a conference paper at ICLR 2027

Sparse Context Synthesis: A Formal Framework for Off-the-Shelf LLM Agents

Abstract

Context optimization improves off-the-shelf large language model (LLM) agents by dynamically injecting context into their prompts rather than fine-tuning model weights. We study a common paradigm of this approach: a catalog of gated factoids, which are small, independently authored functions that inspect an episode and emit short, actionable claims, such as relevant files or policy constraints. We formalize this process as sparse context synthesis: for a given episode, a sparse sub-catalog is activated, and its claims are embedded, weighted on the simplex, and synthesized by the agent into a free-text response. Within this framework, we make three primary contributions. First, we measure how well a catalog fits the evidence by the mutual information its outputs share with a latent target, formalizing three competing design pressures: coverage, redundancy, and localization. Notably, we demonstrate that negative redundancy signals structural synergy among factoids. Second, we establish a theoretical error bridge connecting catalog omissions to downstream agent error, proving that the distance between the embeddings of complete versus omitted claim sets is tightly bounded by their precision and recall deficits. We instantiate this for both observation claims and causal claim graphs to bound end-to-end omission errors. Third, we theoretically ground context importance weights by relating them to expectation-maximization (EM) responsibilities, the Barber–Agakov bound, and information projections, delineating formal proofs from structural analogies. Finally, numerical evaluations on SWE-bench Verified and -bench illustrate the pipeline's operational dynamics, revealing when injected claims successfully bind to guide agent behavior.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, harness optimization, context injection, sparse representations, information retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Z8y1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes context injection for frozen LLM agents. A catalog of gated, hand-authored "factoids" emits short claims, which are embedded, weighted by Loom-style cosine reweighting, and passed to the agent. The contributions are threefold. Catalog utility is scored as I(Z;SF(X))I(Z;S_\mathcal{F}(X)), decomposed into coverage gap, redundancy (negative under synergy), and localization. A tight error bridge (Theorem 1) bounds the distance between averaged claim embeddings by 2Bmax⁡(1−R,1−P)2B\max(1-R,1-P), which then bounds catalog-omission error. The consensus weights are read through EM, Barber–Agakov, and m-projections, explicitly as analogies. On 50 SWE-bench Verified instances, the catalog significantly improves edit-file recall (+0.118), but resolve-given-submit stays near 40% across all arms. On 30 τ² airline tasks, refusals improve and writes get worse.

Critical strengths
  1. A robust, pre-registered empirical finding. Across all slates, including the gold file list, the pass rate of submitted patches stays near 40%; better context raises how often the agent submits, not whether the patch is correct. The authors forecast the temperature-0 results before scoring them, and all four forecasts landed within a reasonable range. This matters because it cleanly separates where context injection helps (localization) from where it does not (correctness), which is a transferable insight made credible by the pre-registration.

  2. Clear scoping of the theory. The paper clearly separates tight, proved results from analogies and states where no bounds exist. This matters because the claims it does make are trustworthy and auditable, even though their scope is limited.

Critical weaknesses
  1. The theory is never tested in the experiments. The paper proposes several measures for judging a catalog (coverage, redundancy, and an error bound), but none of them are actually computed in the experiments. The consensus weighting, one of the main contributions, is never ablated, and the paper does not explain how the agent uses the weights. So we cannot tell whether the framework helps anyone build a better catalog. In addition, the theory only bounds errors from missing claims, while the experiments show the real bottleneck is elsewhere: the agent writes incorrect patches even with perfect context, and the theory says nothing about that stage. This is the main reason for my score.

  2. Little theoretical novelty and a small evaluation. Most of the formal tools are standard results, and the main theorem follows directly from the triangle inequality. The observation that redundancy can be negative (synergy) is well known, and the relevant literature is not cited:

    • McGill (1954), Multivariate information transmission, Psychometrika (interaction information).
    • Bell (2003), The co-information lattice, ICA 2003.
    • Williams & Beer (2010), Nonnegative decomposition of multivariate information, arXiv:1004.2515 (partial information decomposition, which separates redundancy from synergy, addressing the paper's own caveat that a near-zero value can hide both).
    • Timme et al. (2014), Synergy, redundancy, and multivariate information measures: an experimentalist's perspective, J. Comput. Neurosci.

    The experiments also lack comparisons to established code-localization methods, which are the natural baselines for the edit-recall result:

    • Xia et al. (2024), Agentless: Demystifying LLM-based Software Engineering Agents, arXiv:2407.01489.
    • Zhang et al. (2024), AutoCodeRover: Autonomous Program Improvement, ISSTA 2024.
    • Chen et al. (2025), LocAgent: Graph-Guided LLM Agents for Code Localization, ACL 2025.

    The evaluation itself is small: one model, 50 SWE-bench instances (mostly Django),

Questions for authors
  1. How does the agent actually use the consensus weights? Does an ablation against uniform weights or no reweighting show any effect?
  2. Was the catalog developed or tuned on any of the 50 test instances?
  3. Can you compute your own measures (coverage, redundancy, catalog recall) for a few catalog variants and show they predict edit recall?
  4. Does the catalog's advantage survive a length-matched baseline, and what is its edit precision?
1 comment

We thank Reviewer Z8y1 for a sharp review, noting our pre-registered temperature-0 forecast and clear scoping of proved bounds vs. analogies, and pointing us to the synergy/PID and code-localization literature. You wrote: "The theory is never tested in the experiments... This is the main reason for my score." We computed all proposed measures in our experiments, showed how they improve catalogs, and added all 7 citations.

1. Testing the Theory & Improving Catalogs (W1 & Q3)

In Sec. 7 Table 2, App. E.1 Tables 3-4, and App. F.1 Table 6, we compute Γ\Gamma, Γ/H(Z)\Gamma/H(Z), 1−RecallF1-\mathrm{Recall}_{\mathcal{F}}, Red\mathrm{Red}, I(Z;Si)I(Z;S_i), Loc\mathrm{Loc}, Thm. 1 slack, and Prop. 1 across Empty/BM25/Catalog/Gold and 3 versions (v1/v2/v3):

  1. SWE (N=50N=50) & τ2\tau^2 Airline (N=280N=280, 10 repeats):
    • On SWE (Empty/BM25/Catalog/Gold), 1−RecallF1-\mathrm{Recall}_{\mathcal{F}} (1.000/0.470/0.315/0.000) and Γ/H(Z)\Gamma/H(Z) (1.000/0.175/0.000 on Empty/Catalog/Gold) drop, tracking edit recall (0.497/0.559/0.616/0.598).
    • On airline writes (N=200N=200), Γ/H(Z)=0.009\Gamma/H(Z)=0.009 vs. 1−RecallF=0.4581-\mathrm{Recall}_{\mathcal{F}}=0.458: one reservation ID identifies which task's target set ZZ applies (Γ≈0\Gamma\approx 0) even when target flights in ZZ stay unnamed, showing Γ\Gamma and 1−RecallF1-\mathrm{Recall}_{\mathcal{F}} differ.
    • Per-rule I(Z;Si)I(Z;S_i) (Table 4) flags a broken SWE rule (issue_traceback_files = 0.000 bits, matching site-packages paths) vs. strong rules (likely_edit_files = 3.228, identifier_def_sites = 2.846 bits).
  2. Thm. 1 & Prop. 1 fix an embedding bug (Hull (ii): 0.70 →\to 0.000):
    • Thm. 1 holds on 100% (1.000) of 3,4353{,}435 nonempty episodes (min slack = 0.000; exact 0 on all 500 SWE gold slates, 52% SWE patches, 67% airline writes).
    • Prop. 1 fails on all 340/490 SWE runs at RecallF=1\mathrm{Recall}_{\mathcal{F}}=1 and 73/200 airline writes because prose/multi-item rules break Assumption (ii): 100% of matched atomic claims sit outside conv(D(x))\mathrm{conv}(D(x)) (0.60 SWE, 0.56 airline).
    • On a 500+500+-task systems code-generation benchmark (120B model), prose embeddings reproduced this (Hull (ii) = 0.70, 85% hold rate). Embedding each rule's short ID in D(x)D(x) while printing full text in the slate dropped Hull (ii) to 0.000, nearly halved hull error (0.57 →\to 0.30), and gave 100% Prop. 1 hold rate.
  3. Predicting gains & realization (Q3): Across v1/v2/v3, cutting Γ^/H^(Z)\hat{\Gamma}/\hat{H}(Z) (0.23 →\to 0.21) lowered Thm. 1 error (0.160 →\to 0.145), raising Compile@1 (+5 to +7 pp single-turn; +3.4 to +8.9 pp on all 4 agents, beating the gold API-name oracle on all 4) and Pass@1 (+5 to +6 pp single-turn; beating empty/BM25 on 3 of 4 agents, tying empty on the 4th; App. G). Stage 3 has no set-embedding bound without strong LLM assumptions; a 120B model + signature rules cuts realization errors.

2. Literature & Theorem 1 (W2)

  • Literature: Cited McGill (1954), Bell (2003), Williams & Beer (2010), Timme et al. (2014) on signed Red\mathrm{Red} and PID (Sec. 5.2), and Agentless (Xia et al., 2024), AutoCodeRover (Zhang et al., 2024), LocAgent (Chen et al., 2025) (App. A), contrasting SE agents with our zero-LLM-call front-end.
  • Thm. 1: Like Barber-Agakov, it gives the set-embedding form with tight 2B2B, ∣Recall−Precision∣|\mathrm{Recall}-\mathrm{Precision}| imbalance, and testable Assumption~(ii).

3. Questions 1, 2, and 4

  • Q1 (w(x)w(x) & ablations, Sec. 3 & App. G): w(x)w(x) drops duplicates/outliers, sorts claims high-to-low, and prints [weight ROLE] in the slate. On SWE (4.90 →\to 4.14 kept), 0/500 gold claims are lost; on 10-repeat airline (Table 2: writes 0.911 vs. 0.897, official 0.570 vs. 0.565), weighting drops 65/76 (85.5%) false policy_decision firings on writes; on OpenRCA Bank (340.9 claims/incident), Begleiter et al. (2026) show -5.88 pp strict accuracy w/o reweighting (38.97% →\to 33.09%).
  • Q2 (Tuning vs. transfer, App. G): The 50 SWE tasks (seed=42) and airline base split were dev slices (SWE-bench Pro / τ2\tau^2 retail untouched; zero gold leakage). Our sole Django rule fired 0 times; on 19 non-Django tasks (9 repos) Catalog gets 0.550 vs. 0.475 BM25 / 0.465 Empty, and gains hold on >440>440 held-out tasks of the 500+500+-task benchmark.
  • Q4 (Precision & length, Sec. 7 & App. G): On submitted SWE patches (T=1.0T=1.0), Catalog is #1 in precision: 0.678 vs. 0.612 Empty (+0.053+0.053, p=0.032p=0.032), 0.639 BM25, 0.671 Gold (Recall|submit: 0.825 vs. 0.763, p=0.019p=0.019; 0.783 BM25). On slates ≤800\le 800 chars (561 vs. 580), Catalog leads (0.560 vs. 0.536/0.519); on the 500+500+-task benchmark it is <50% of BM25 (~1,040 vs. ~3,000 chars) and wins on compile/pass.
Reviewer_SG56OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to construct sparse, task-relevant context for off-the-shelf agents without updating LLM parameters. The authors use a predefined catalog of gated functions to extract descriptive claims from the task and environment, then apply embedding-space consensus weighting and selection to construct context for the existing agent. The paper further uses mutual information to discuss catalog coverage, redundancy, and synergy, and establishes an error bound relating the precision and recall of claim sets to the distance between their mean embeddings. Experiments on a fixed subset of SWE-bench Verified and the airline domain of τ²-bench examine how context injection affects file coverage, submission, policy adherence, and final task success.

Critical strengths

Modular context construction has some practical value, and the paper provides useful analysis of intermediate agent behavior. Small gated functions turn common evidence-gathering practices into reusable rules that can be combined with off-the-shelf agents whose model parameters remain frozen. The experiments show improvements in some intermediate behaviors: edit-file recall on SWE-bench increases from 49.7% with empty context to 61.6%, and correct refusals on the airline tasks increase from 7/10 to 10/10. The authors also explicitly distinguish localization, submission, and final correctness, and report the performance decline on legitimate write tasks. These analyses help clarify when additional context can guide an agent and when it remains insufficient for task completion.

Critical weaknesses
  • W1: The added contribution and practical gains remain limited. Turning common search and evidence-gathering practices into rules that provide context to an agent is a natural approach, and the core pipeline builds on Loom. The theoretical analysis offers some interpretation, but does not yet make clear what important improvements the new design provides. The empirical gains mainly concern localization and refusal behavior, while final task success improves only modestly over BM25 and declines on legitimate write tasks. This leaves me with reservations about the method's added value.

  • W2: Validation on stronger models and ablations of core components are missing. The coding experiments use only one backbone and do not include stronger, state-of-the-art models, leaving it unclear whether these rules would still provide meaningful benefits to agents with stronger autonomous exploration capabilities. Consensus weighting and selection are also not evaluated in isolation: the current comparisons change both evidence acquisition and context content, so they cannot establish whether the gains come from the consensus mechanism or from richer information. Comparisons against simple selection or ranking methods using the same candidate claims and context budget are needed.

Questions for authors
  • Are results available for backbone models at different capability levels, particularly stronger, state-of-the-art models? With comparable catalogs, agent frameworks, and budgets, how do the marginal gains in file coverage, submission rate, and final task success vary with backbone capability? This would directly affect my assessment of the method's applicability and practical value.

  • Please add ablation studies to evaluate the contribution of consensus weighting and selection.

1 comment

We thank Reviewer SG56 for the constructive review and for noting the practical value of small gated rules for frozen agents, as well as our breakdown of localization, submission, and final task success. We address both of your main questions below and in the revised paper.

Q1 & W2: Stronger 120B Model Across Four Autonomous Multi-Turn Agents

On SWE-bench Verified with a smaller frozen model (nemotron-3.5-lightning), our 10-repeat study showed clear gains in file recall (0.616 vs. 0.497, Holm p=0.00073p=0.00073) and submitted-patch precision (0.678 vs. 0.612, p=0.032p=0.032), while conditional pass rate stayed near 40% even given Gold paths (0.399). That suggested turning better hints into more solved tasks requires a stronger code-writing model and/or compiler feedback.

To test this at scale (App. F.1 & App. G), we ran our framework with a 120B-parameter open-weights coding model across all 500+500+ tasks of a systems library code-generation benchmark (>440>440 strictly held-out tasks), across single-turn generation and four unmodified multi-turn coding agents with file/shell/compiler tools (5 repeats/arm on each multi-turn agent):

  1. How gains in API coverage, compilation (submission), and final pass rate scale:
    • Single-turn: Sparse slates raise compilation by +5 to +7 pp and functional Pass@1 by +5 to +6 pp (+16% to +19% relative) over empty, beating BM25 (k=5k=5) on API recall, API precision, compilation, and Pass@1, and edging conditional Pass@1 | compiled past even the gold API-name oracle.
    • Four multi-turn agents: Even though these agents explore headers and compiler errors on their own, sparse slates lead all non-oracle arms on solution API recall and precision, raise Compile@1 by +3.4 to +8.9 pp over empty (+5 to +13 pp on Compile@5), and beat even the gold API-name oracle on Compile@1 across all four agents (by up to +3.5 pp), while raising Pass@1 over both Empty and BM25 on three of four agents (tying Empty on the fourth) and cutting agent LLM turns by 3% to 12%.
  2. Why capable multi-turn agents still benefit: Without targeted hints, strong agents spend extra turns sifting through noisy retrieved chunks (BM25 injects >2×>2\times more characters, median ~3,000 vs. ~1,040, and induces more wrong-API calls) or guessing multi-argument signatures that bare API names (Oracle) omit. ~4 weighted signature/contract claims let the agent compile early rather than burning turns on trial and error.

W1: 10-Repeat τ2\tau^2 Airline & What Our Framework Adds Beyond Loom

  1. The airline write dip (9/20 vs. 11/20) was 1-run noise, gone over 10 repeats (Table 2 & App. G): Running all 30 τ2\tau^2 airline tasks 10 times/arm (900900 runs at T=1.0T=1.0, top-pp 0.950.95) shows writes do not drop: write-entity recall is 0.911 (catalog) vs. 0.897 (empty) (p=0.507p=0.507) and official pass1\mathrm{pass}^1 is 0.570 vs. 0.565 (p=0.949p=0.949), while refusal is unchanged (0.820 vs. 0.840, p=1.0p=1.0; single empty runs span [0.50, 1.00]). Hints help when narrowing a large code/API space, and do no harm when the base agent already finds ~90% of target entities from a short wiki.
  2. Beyond Begleiter et al. (2026) (Loom): Begleiter et al. (2026) used cosine-centroid reweighting for 1-call RCA summaries, where static docstring grouping hurt accuracy by -5.88 pp. We add (i) output-level Γ(F),Red(F),Loc(F)\Gamma(\mathcal{F}), \mathrm{Red}(\mathcal{F}), \mathrm{Loc}(\mathcal{F}) (Sec. 5.2, Table 2 & App. E.1), separating runtime duplication from synergy; (ii) Thm. 1 & Prop. 1 (App. F.1), whose hull check (Hull (ii) =0.60= 0.60 SWE, 0.56 airline, 0.70 prose systems rules) exposed why prose embeddings break omission bounds and led us to embed short atomic IDs (Hull (ii) = 0.000, 100% Prop. 1 hold rate, hull error 0.57 →\to 0.30); and (iii) multi-turn tool-using agents.

Q2 & W2: Ablating Consensus Weighting and Selection on Fixed Candidate Claims

Holding candidate claims fixed (Sec. 3, Sec. 7, App. G):

  1. Sparse firings (SWE, 4.90 →\to 4.14 kept): Keeps all fired claims on 30/50 tasks, trims low-agreement claims on 20/50 tasks (0/500 gold drops), and sorts claims high-to-low in the slate.
  2. Noisy firings (τ2\tau^2 airline writes, 200 runs): policy_decision fires mistakenly on 76/200 (38%) write runs; consensus weighting drops 65/76 (85.5%) of those false alarms (11/200 writes vs. 25/100 refuses).
  3. Dense firings (Begleiter et al., 2026, OpenRCA Bank, 136 incidents, 340.9 claims from 7.9 rules): With the same candidates and synthesizer, removing reweighting cuts strict accuracy by -5.88 pp (38.97% →\to 33.09%; -9.56 pp without static grouping, 44.85% →\to 35.29%).
Reviewer_jrZkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors study a form of context engineering for off-the-shelf LLM agents. The core idea is to use a fixed catalog of small, hand-authored “factoids”to produces context hints; these hints are weighted by an embedding-consensus procedure and injected into the agent’s prompt. Factoids are gated functions that inspect an episode and emit short claims such as candidate files or policy clauses.

Authors report experiments on a 50-instance slice of SWE-bench Verified and 30 τ²-airline tasks. Results show that factoid hints improve file-localization recall and correct refusals. However, they do not significantly improve resolve rate, and they reduce success on airline write tasks.

Critical strengths

The empirical observations are reported in detail and are somewhat interesting. Specifically, descriptive hints improve file localization over BM25 path lists and even over the gold path list, yet the pass rate is almost identical across different methods. This suggests that context injection mainly changes whether the agent commits to an action, not the quality of the action.

Critical weaknesses
  1. The mathematical framework does not contribute to the design of method in the paper. In practice, the method runs a set of hand-written regex/grep rules (“factoids”) and pastes their short text outputs into an LLM agent’s prompt. The paper describes these steps in formal terms but this formalism does not derive, change, or evaluate the method.

Specifically, I'm very confused on following: (a) In section 4.2, the weighting procedure is taken from Loom, and the paper states that the EM and m-projection views are only analogies? How does this help with the method? (b) In section 6.1, the embedding centroid the theory is built on is never given to the agent. The agent only reads text, and the paper says the centroid is “inserted for analysis”. (c) None of the proposed quantities (coverage gap, redundancy, residual) is computed in the experiments, and none is used to explain the main empirical finding

  1. The experiment is not sound and result does not support the claim. (a) The hint rules are hand-written separately for each benchmark (App. H), which conflicts with the paper’s claims of a domain-agnostic, optimizable framework. (b) The evaluation is really small. Only 50 SWE-bench instances with one model, and 30 airline tasks run once each. Picking the subset from a benchmark without justifying the rationale of why only choosing those is very concerning. (c) The hints help the agent edit the right files (edit recall goes from 0.50 to 0.62), but they do not robustly improve the number of tasks solved. It is only +0.01 over plain BM25 retrieval (d) On airline tasks that modify a booking, the catalog does slightly worse than no hints at all (9/20 vs. 11/20)

  2. The paper is very difficult to follow. It introduces a large amount of notation and terminology, much of it defined only informally (e.g., “template language functionals,” “natural language manifold,” “slate,” “gold-keep”). Much of the formal notation is never used later. For example, the generative model θ\theta is never instantiated, and the Barber–Agakov and EM interpretations play no role in the method or the experiments.

Questions for authors
  1. Can you show one case where the framework’s quantities (Γ^\hat\Gamma, Red^\widehat{\mathrm{Red}}, δ\delta, or the consensus weights) were computed on your catalogs and used to make a catalog decision that improved results?
  2. What is the contribution beyond Loom? Is the consensus reweighting new, and does it matter? Would be very helpful to see an ablation comparing it to injecting all fired factoids unweighted.
1 comment

We thank Reviewer jrZk for the candid feedback and for noting that descriptive file hints beat BM25 and gold path lists on localization while conditional pass rate stays constant. We revised our experiments and prose to address all points.

1. How the Math Evaluates, Explains, and Improves the Method (W1a-c & Q1)

  • 1(c) & Q1: Computing Γ,Red,δ‾\Gamma, \mathrm{Red}, \overline{\delta}, Thm. 1, and Prop. 1 (Sec. 7 Table 2; App. E.1 Tables 3-4; App. F.1 Table 6):
    1. Fixing an embedding flaw (Prop. 1 & δ‾\overline{\delta}): On SWE (n=490n=490) and τ2\tau^2 airline (n=200n=200), median hull residual δ‾\overline{\delta} is 0.63 and 0.55, and Prop. 1 failed at RecallF=1\mathrm{Recall}_{\mathcal{F}}=1 because prose/multi-path outputs break Assumption~(ii): 100% of matched atomic targets sit outside conv(D(x))\mathrm{conv}(D(x)) (0.60 SWE, 0.56 airline). Separating each rule's embedded short ID in D(x)D(x) from its printed rule text in the slate on a 500+500+-task systems code-generation benchmark (120B model) dropped hull distance from 0.70 to 0.000, nearly halved δ‾\overline{\delta} (0.57 →\to 0.30), and gave 100% Prop. 1 hold rate.
    2. Using Γ^/H^(Z)\hat{\Gamma}/\hat{H}(Z) & I(Z;Si)I(Z;S_i): Per-rule I(Z;Si)I(Z;S_i) (Table 4) flagged a broken SWE rule (issue_traceback_files = 0.000 bits, matching site-packages paths). Across 3 catalog versions on the 500+500+-task benchmark, Sec. 5.2's Design Rule cut Γ^/H^(Z)\hat{\Gamma}/\hat{H}(Z) (0.23 →\to 0.21) and Thm. 1 error (0.160 →\to 0.145), raising Compile@1 (+5 to +7 pp single-turn; +3.4 to +8.9 pp on all 4 multi-turn agents, beating the gold API-name oracle on all 4) and Pass@1 (+5 to +6 pp single-turn, beating BM25).
    3. Explaining findings: On SWE, Catalog cuts Stage 1 gap (Γ/H(Z):1.000→0.175\Gamma/H(Z): 1.000 \to 0.175; 1−RecallF:1.000→0.3151-\mathrm{Recall}_{\mathcal{F}}: 1.000 \to 0.315), raising localization and submission, while at Γ/H(Z)=0.000\Gamma/H(Z)=0.000 (Gold paths) pass-given-submit stays 0.399, isolating Stage 3 (the smaller frozen decoder). On airline writes (N=200N=200), Γ/H(Z)=0.009\Gamma/H(Z)=0.009 vs. 1−RecallF=0.4581-\mathrm{Recall}_{\mathcal{F}}=0.458: one reservation ID identifies which task's target set ZZ applies (Γ≈0\Gamma\approx 0) even when target flights in ZZ stay unnamed.
  • 1(b) vˉ∗\bar{v}^* & vˉlearned\bar{v}_{\mathrm{learned}} (Sec. 2.1, 3, 6.1): At runtime, the LLM reads only the text slate (filtered/sorted by w(x)w(x)). Offline, vˉ∗\bar{v}^* and vˉlearned\bar{v}_{\mathrm{learned}} isolate missing-rule error (∥emb(y∗)−vˉ∗∥2\|\mathrm{emb}(y^*)-\bar{v}^*\|_2) from weighting before rollouts.
  • 1(a) EM & mm-projections (Sec. 4, App. D): Viewing reweighting as β(J∣X)\beta(J\mid X) and an mm-projection explains when it works (65/76 (85.5%) false policy_decision firings dropped on airline writes) and when it breaks (duplicates double-counted unless w~\tilde{w} discounts Red\mathrm{Red}).

2. Beyond Loom & Unweighted Ablation (Q2)

  1. Beyond Begleiter et al. (2026): Centroid reweighting is from Begleiter et al. (2026) (1-call RCA, where static docstring grouping hurt accuracy by -5.88 pp). New here: (i) output-level Γ,Red,Loc\Gamma, \mathrm{Red}, \mathrm{Loc} (Sec. 5.2 + PID), explaining why static grouping fails; (ii) Thm. 1 & Prop. 1 (Sec. 6.3, 3,435+3{,}435+ episodes), fixing the prose hull gap; and (iii) multi-turn agents.
  2. Unweighted ablation: On SWE (4.90 →\to 4.14 kept), all fired claims are kept on 30/50 tasks (0/500 gold drops). Under noisy/dense firing, weighting drops 65/76 (85.5%) false policy_decision firings on airline writes, and on OpenRCA Bank (136 incidents, 340.9 claims/incident), Begleiter et al. (2026) report -5.88 pp strict accuracy w/o reweighting (38.97% →\to 33.09%; -9.56 pp w/o static grouping; App. G).

3. Soundness, Scale, and Clarity (W2a-d & W3)

  • 2(a): Like weak supervision (Ratner et al., 2016), weighting, slate formatting, and Γ/Red/Loc\Gamma/\mathrm{Red}/\mathrm{Loc} are domain-agnostic; each domain plugs in gated rules (~2 to 3 engineer-days).
  • 2(b)-(d): Locked first 50 SWE tasks (seed=42, 4,0004{,}000 agent + 4,0004{,}000 test runs, 2 temperatures); on submitted patches, Catalog raises recall (0.825 vs. 0.763, p=0.019p=0.019) and precision (0.678 vs. 0.612, p=0.032p=0.032), plus 0.550 vs. 0.475 BM25 / 0.465 empty on 19 non-Django tasks. Over 10 airline repeats (900900 runs, Table 2), writes do not drop (0.911 vs. 0.897 primary, p=0.507p=0.507; 0.570 vs. 0.565 official, p=0.949p=0.949; refusal 0.820 vs. 0.840, p=1.0p=1.0). On the 500+500+-task (>440>440 held-out), 120B study, slates raise Compile@1 on single-turn and all 4 agents, and Pass@1 over Empty/BM25 on single-turn and 3 of 4 agents (tying Empty on the 4th; App. G).
  • W3: Removed ornate terms, defined slate and gold-keep, and separated runtime vs. offline (θ,q\theta, q) in Sec. 2.1.
Reviewer_svzDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a way to help an existing language-model agent without changing the model itself. Handwritten rules inspect each task and produce short notes for the agent’s prompt, such as likely files to edit in a software bug report or a policy that applies to an airline booking. A similarity-based step selects among those notes. The authors then use information theory to describe what the rules capture and derive a bound on one kind of error caused by missing notes.

The experiments cover 50 software bug-fixing tasks and 30 airline customer-service tasks. In the software tasks, the notes help the agent touch more of the files used in the reference fixes, but they do not clearly increase the number of bugs it solves. In the airline tasks, correct refusals rise from 7/10 to 10/10, while successful booking changes fall from 11/20 to 9/20.

Critical strengths

The paper gives enough detail to understand what was tested. The appendices list the rules and experimental setup, and the authors report results where the added notes hurt or made little difference. They are also clear that only part of their error analysis has a proven bound.

I found the distinction between submitting a patch and submitting a correct patch useful. With the rule-based notes, the coding agent submits a patch in 74% of runs, compared with 65% when no notes are added. Yet only about 40% of submitted patches pass the tests across all four prompt settings, including the setting that supplies the correct file names. A second experiment at a different sampling temperature shows a similar pattern. For this agent, the added context seems more helpful in getting to a patch than in making that patch correct.

Critical weaknesses

The experiments do not show whether the proposed theory helps design better rules. The paper defines measures for information missed by the rules, repeated information, and the number of rules activated per task. It reports some simpler quantities, such as file recall, but does not test whether those theoretical measures predict what happens when rules are added or removed. The main bound concerns averages of claim embeddings and relies on an assumption about how those averages represent a full response. It does not cover the later steps where the notes are selected and the agent produces its answer.

It is also hard to isolate why the coding results improve. There is no comparison with passing along all applicable notes without the weighting step, or with another prompt of similar length. Runs with no submitted patch receive zero file recall, so an increase in submissions can also raise the file-recall score. The empty-prompt setting has more format failures than the settings with added text. These details matter because the rule-based notes solve 32% of tasks versus 31% with a simple file-search baseline; that difference is not clear evidence of better final performance.

Finally, the results say little about how well the rules transfer. The coding and airline rule sets were written by hand for their respective tasks. One airline rule directly encodes policy decisions, which may account for much of the refusal gain. The coding sample contains 31 Django tasks out of 50, the study uses one model, and each airline setting is run once. The paper does not say whether the evaluation tasks were seen while the rules were developed or how much work it took to write the rules.

Questions for authors

Were any evaluation tasks used while writing or tuning the rules? Could you test on held-out repositories or another airline-benchmark domain, and report the effort needed to build each rule set?

What happens if the agent receives every applicable note without the weighting step? Could you also compare with a prompt of similar length and report file recall separately for runs that submit a patch?

Do the proposed measures help identify useful rules in practice? A rule-removal study would help, particularly for the airline rule that makes policy decisions.

1 comment

We thank Reviewer svzD for the constructive review, and for noting our split between patch submission (74% vs. 65%) and conditional pass rate (~40% across slates and temperatures) and our clear scoping of proved bounds. Below we answer each question and weakness with measurements in the revised paper.

Q1 & W3: Development Tasks, Held-Out Transfer, 10-Repeat Airline, and Effort

  1. Were evaluation tasks seen? Yes, on the initial slices: we developed on the locked 50-task SWE-bench Verified slice (seed=42, first 50) and τ2\tau^2 airline base split while keeping SWE-bench Pro and τ2\tau^2 retail untouched as holdouts (App. G). Rules never saw gold patches, hidden tests, or write payloads, only issue/conversation text, the public wiki, and read-only repo/DB lookups.
  2. Transfer beyond Django (n=31n=31 vs. n=19n=19 across 9 repos): Only 1 of 14 SWE rules (error_code_sites) is Django-specific, and it fired 0 times on our 50 tasks (App. E.1 Table 4). On 50×1050\times 10 (T=1.0T=1.0) runs, BM25 drops near Empty outside Django, while Catalog keeps its lead (App. G):
    • Django (n=31n=31): Empty 0.517, BM25 0.610, Catalog 0.656 (+0.139 vs. empty), Gold 0.657.
    • Non-Django (n=19n=19, 9 repos): Empty 0.465, BM25 0.475, Catalog 0.550 (+0.085 vs. empty, +0.075 vs. BM25), Gold 0.502.
  3. 10 airline repeats (900900 runs) & held-out scale: Running all 30 τ2\tau^2 airline tasks 10 times/arm (T=1.0T=1.0, top-pp 0.950.95; Table 2 & App. G) shows single Empty refuse runs span [0.50, 1.00], so the 1-run jump (7/10 →\to 10/10) and write dip (11/20 →\to 9/20) were noise: 10-repeat writes do not drop (0.911 catalog vs. 0.897 empty recall, p=0.507p=0.507; 0.570 vs. 0.565 official pass1\mathrm{pass}^1, p=0.949p=0.949) and refusal is unchanged (0.820 vs. 0.840, p=1.0p=1.0). Rather than adding another ~90%-recall short-wiki domain, we tested holdout transfer on a 500+500+-task systems library code-generation benchmark (>440>440 held-out tasks, 120B model, single-turn + 4 unmodified multi-turn agents, 5 repeats/arm): slates raise Compile@1 (+3.4 to +8.9 pp over empty on all 4 agents) and Pass@1 on single-turn and 3 of 4 agents (tying empty on the 4th; App. G).
  4. Effort: Each catalog (14 SWE, 16 to 17 airline rules) took ~2 to 3 engineer-days (cf. ~1 engineer-week in Begleiter et al., 2026).

Q2 & W2: Submitted Patches, Prompt Length, and Unweighted Notes

  1. Submitted patches (Sec. 7 & App. G): On non-empty SWE patches (T=1.0T=1.0), Catalog raises both file recall (0.825 [n=373n=373] vs. 0.783 BM25 [n=357n=357] and 0.763 Empty [n=326n=326]; 0.835 Gold [n=358n=358]; Wilcoxon +0.046, p=0.019p=0.019) and edit precision (0.678, #1 overall, vs. 0.639 BM25, 0.612 Empty [+0.053, p=0.032p=0.032], 0.671 Gold). With Table 1 excluding format failures (0.673 vs. 0.601 BM25, p=0.046p=0.046), gains are not from extra submissions or extra files; resolve stays +0.010 over BM25 because the smaller model passes ~40% of submitted patches even given Gold paths.
  2. Length: On 14 SWE tasks with slates ≤800\le 800 chars (mean 561 vs. BM25 580), Catalog leads (0.560 vs. 0.536 BM25, 0.519 Empty). On the 500+500+-task benchmark, BM25 is >2×>2\times longer (~3,000 vs. ~1,040 chars) yet loses on compilation and pass rate.
  3. Unweighted notes: On SWE (4.90 →\to 4.14 kept), all fired notes are kept on 30/50 tasks (0/500 gold drops). Under noisy firing, weighting drops 65/76 (85.5%) mistaken policy_decision firings on airline writes, and removing reweighting across 340.9 claims/incident on OpenRCA Bank cuts strict accuracy by -5.88 pp (38.97% →\to 33.09%; Begleiter et al., 2026).

Q3 & W1: Rule Diagnostics, policy_decision, and Proposition 1

  1. Rule audit (App. E.1 Table 4 & App. G): Per-rule I(Z;Si)I(Z;S_i) caught a broken SWE rule (issue_traceback_files = 0.000 bits on 6 tasks, matching site-packages paths). On 10-repeat airline, consensus dropped 65/76 (85.5%) false policy_decision firings on writes, while on refuse it was kept 25/100 times (0.800 vs. 0.720 paired empty) and absent 75/100 times (0.827 refusal).
  2. Prop. 1 & Γ^\hat{\Gamma} (App. F.1 Table 6 & App. G): Assumption (i) residual is 0.00 on 430/490 single-file SWE runs (0.35 multi-file; 0.50 airline), while Assumption (ii) failed on 100% of matched claims (median hull distance 0.60 SWE, 0.56 airline). Embedding short IDs in D(x)D(x) while printing full rule text dropped hull distance from 0.70 to 0.000 (100% Prop. 1 hold rate) and nearly halved hull error (0.57 →\to 0.30), while cutting Γ^/H^(Z)\hat{\Gamma}/\hat{H}(Z) (0.23 →\to 0.21) lowered Thm. 1 error (0.160 →\to 0.145) and raised compilation and pass rates. (Stages 2-3 use Table 5 / App. G diagnostics.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.