acceptodds
Under review as a conference paper at ICLR 2027

Bayesian-Harness: Probabilistic Knowledge Control for LLM Agents

Abstract

An agent working on long-term tasks usually serialises its knowledge of, e.g., tools, interfaces, and codebases in its local environment. However, knowledge becomes stale and expires as the internal or external environment changes. Many existing harnesses update memory records after encountering an error, without explicitly modelling how knowledge expires over time. In this paper, we there- fore model knowledge staleness as a generative process from a Bayesian perspec- tive. We maintain two latent random variables for each memory, validity and quality. The validity is a binary latent state that fails at a Markov-modulated haz- ard and is restored only by a rewrite. We model quality in Bayesian form with a Beta-Bernoulli model over the memory’s success rate conditional on validity, with memories sharing statistical strength within each subsystem pool. We fur- ther build two datasets, MASKEDMIG and ACMESDK. MASKEDMIG extracts 157 real API changes from 18 Python libraries, each removing an old API and adding a named replacement in a release between April 2025 and August 2026. ACMESDK is a company Software Development Kit (SDK) benchmark. It is based on SOP- Bench and generated by Claude Code (Fable 5.1). Through extensive empirical investigation, from a 4B model to 27B and two closed-weight Gemini models, Bayesian-Harness is the best maintenance policy on every model of MASKEDMIG, 3.1–8.3 points above the strongest baseline of each, and in all 15 model–budget configurations of ACMESDK.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian, LLM, Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TvTYOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets maintenance of an LLM agent's external memory (notes about APIs/tools) as the environment drifts. It proposes Bayesian-Harness, which assigns each memory item two latent variables:

  • Validity: a binary absorbing state whose per-period hazard is modulated by a two-state continuous-time Markov regime (STABLE/CHANGING) shared across a subsystem.
  • Quality: the success probability while valid, modelled with a hierarchical Beta–Bernoulli prior pooled within the subsystem.

Observations (test outcomes, classified errors, active probes) update the joint posterior. A budgeted maintenance policy probes items ranked by usage-weighted predicted staleness and rewrites the ones verified as stale.

The paper introduces two benchmarks:

  • MASKEDMIG: 157 real API removals with named successors from 18 Python libraries (April 2025 – August 2026). Successor names are hashed to prevent recall from pre-training.
  • ACMESDK: a synthetic, LLM-generated company SDK with 12 subsystems and 36 monthly releases, with drift calibrated on real repositories.

Across Qwen3.5-4B/9B/27B and two Gemini models, the method beats None/Static/Random/TTL/Bayesian-Agent on every MASKEDMIG column (+3.1 to +8.3 over the strongest baseline) and in all 15 ACMESDK model–budget cells. Supporting experiments include component ablations, a validity/quality separability test, structural stress tests, and a command-line variant.

Critical strengths

1. A well-posed and useful problem decomposition, backed by a principled, interpretable model. Separating "the world changed" (validity) from "the note was never good" (quality) is a genuinely useful framing for long-lived agent memory. Coupling items through a shared subsystem regime is a clean way to let one failure raise suspicion about sibling items. The model is small, conjugate, and exactly updatable per item. Every constant is listed with its provenance (Table 5), and the authors state that no parameter was tuned on evaluation worlds. This matters because it makes the method deployable and auditable rather than another heuristic score.

2. Careful attention to contamination, plus unusually thorough empirical diagnostics. The paper shows explicitly that unmasked real-library evaluation is confounded: a no-memory agent repairs 60–78% of snippets. It fixes this by hashing successor names consistently across package, changelog and tests. The evaluation also includes:

  • per-component ablations on both corpora (Tables 2, 12);
  • a quality-freezing ablation (Table 11);
  • a targeted separability test with injected low-quality items (Table 3);
  • per-seed breakdowns (Table 10);
  • stress tests with misfiled groups, a hidden-axis drift, and altered rates/schedules (Table 4);
  • a second knowledge modality (ACMECTL, Table 13).
Critical weaknesses

1. The evaluation design favours the model's structural assumptions and withholds the most informative real-world signal. The baselines are correspondingly weak.

(a) Generator–model alignment on ACMESDK. The synthetic world is produced by a generator whose structure closely mirrors the inference model:

  • drift concentrated in subsystem-wide "heavy months" (rewriting 40–60% of one subsystem) vs. ρ = 0.5;
  • a ~1% background trickle vs. ε = 0.012;
  • heavy changes lasting exactly one release vs. µ = 1 per period.

The constants are said to be estimated from XGBoost/OpenCV/SciPy/vLLM, but the near-exact match with the generator is striking. The stress tests help but do not remove the concern. Even the "hidden-axis" world retains two subsystem heavy months and two package-wide heavy months, and every perturbed world still has bursty, grouped drift. A sensitivity sweep over ε, ρ, µ, or a world with genuinely unstructured drift, is missing.

(b) Withheld changelog and version signals. By design, a policy may not use the changelog to choose which item to maintain; it can read the changelog only after selecting an item. In practice, release notes and deprecation warnings are exactly how practitioners know what went stale, and package versions are always observable.

On MASKEDMIG this is especially consequential. Every library's items break in the quarter of that library's release and never recover. A trivial "probe items of any library whose installed version changed" baseline would therefore likely approach the oracle. The protocol turns an observable event into a latent-inference problem, which inflates the value of regime inference. The paper needs either a compelling deployment scenario where this information is unavailable, or a version/changelog-aware baseline.

(c) Baseline strength. Bayesian-Agent, the closest prior work, is reduced to the mean of a per-item Beta posterior because "our memory items carry no context features." Its ranking (1 − posterior mean) has no notion of time, so it cannot become suspicious of unused items. It is a structurally handicapped stand-in rather than a faithful reproduction.

There are also gaps on the non-Bayesian side:

  • There is no baseline that propagates a failure to sibling items in the same subsystem (a non-probabilistic analogue of the shared regime). The "two counters" baseline of §4.6 appears only in the separability study.
  • There is no LLM-driven prioritisation baseline.
  • None of the memory systems discussed in related work (MemoryBank decay, Mem0, A-Mem) is evaluated.

Together, (a)–(c) make it hard to know how much of the gain comes from the Bayesian machinery rather than from an evaluation that rewards it.

2. The headline contribution (validity–quality separation) has a small, noisy measured impact, and the gains on real data rest on limited statistical evidence.

(a) The quality variable is barely tested in the main results. §4.6 states that the main experiments test staleness only: every item starts verified, so there are no low-quality-but-valid items. The internal/external distinction motivating Figure 1(a) is therefore exercised only in the separability study, where low quality is injected artificially via "cheap repairs."

The direct evidence that estimating quality helps is also weak. In Table 11, estimated quality gives 46.0±6.5 / 67.0±3.4, compared with 36.9±11.0 / 63.8±3.6 when quality is frozen at 1. The overall gap is within one s.d. In Table 3, even the full model spends 62–69% of its probes on valid items.

(b) Statistical evidence on the real corpus is thin. On MASKEDMIG the 157 tests are identical across seeds; seeds differ only in when each break is met. The reported s.d. therefore understates uncertainty, and the effective sample is a single set of 157 items.

  • The Flash-Lite margin (12.7 vs. 9.6) corresponds to about five tests.
  • On ACMESDK, several "best in every cell" claims are within one s.d. of the runner-up, e.g., Qwen3.5-9B at budget 24: 52.8±7.7 vs. TTL 51.2±4.0.
  • Ablation drops on renamed drift (4.1–8.0) are comparable to the renamed s.d. (5.7).

Paired tests or bootstrap CIs over tests would be needed to support the "best everywhere" framing.

(c) The oracle is not a meaningful reference. The "oracle" is beaten by the method at budget 12 in three of five models (Table 6) and at 9B/budget 24 on renamed drift (Table 9). Its accuracy also declines over time (81.6 → 75.0 in Table 8) because it retries impossible repairs. As a result, the paper lacks a meaningful upper bound against which to judge how much headroom remains.

Questions for authors
  1. Please describe a concrete deployment where an agent cannot observe which package or subsystem released, or read release notes, before choosing what to probe. How does Bayesian-Harness compare with (i) a version-bump baseline on MASKEDMIG (probe all items of libraries whose version changed) and (ii) a changelog-aware baseline on ACMESDK? If the method still adds value on top of such signals (e.g., for silent semantic changes, about 20% of changes), that would substantially strengthen the paper.
  2. Please report the actual measured values of ε, ρ, µ from XGBoost/OpenCV/SciPy/vLLM before any rounding, and explain why they coincide so closely with the generator settings (1% trickle, 40–60% heavy-month coverage, one-release bursts). Please also provide a sensitivity sweep, e.g., ρ ∈ {0.1, 0.25, 0.5, 0.8}, µ ∈ {0.25, 1, 4}.
  3. Is the feature-free reduction faithful to Wu et al. (2026)? Did you consider giving it simple features (subsystem ID, time since verification) so that it is not structurally time-blind? Please also add a non-Bayesian "propagate failure to subsystem siblings" baseline to the main tables.
  4. For Table 1 and Table 6, can you report paired significance (e.g., McNemar or a paired bootstrap over tests) between Bayesian-Harness and the strongest baseline per column?
  5. Can you add a stronger reference, e.g., an oracle that knows staleness and skips repairs that previously failed, so that the gap to an achievable ceiling is interpretable?
  6. Is there any condition without injected cheap repairs where items are naturally low-quality while valid (e.g., notes written by a weaker model), so that the internal-vs-external distinction is tested without an artificial construction?
8 comments
AuthorsOpenReview ↗

Response to Reviewer TvTY (Part 1 / 6): W1, W1(a)–W1(b)

Thank you for your time and for the insightful suggestions. All new code and logs are in the branch reviewer_TvTY of our anonymous repository.

W1: Evaluation design

Code and logs


W1(a): Generator and model alignment

Short answer. We agree with this comment. This is why the main paper reports MaskedMig, which is built from real library changes. AcmeSDK is a complementary benchmark, and its results are mainly in the appendix.

  • The constants are held out. ε\varepsilon, ρ\rho and μ\mu were estimated on XGBoost, OpenCV, SciPy and vLLM. None of these libraries is in either benchmark.

  • The result does not depend on them. We swept ρ\rho, μ\mu, ε\varepsilon and κ\kappa over 81 settings on a held-out world.

Table R1: AcmeSDK, held-out world, 81 settings. Coverage is the share of exams whose note is valid.

Coverage (%)
Default (ρ=0.5,μ=1,ε=0.012,κ=5\rho = 0.5, \mu = 1, \varepsilon = 0.012, \kappa = 5)45.2
Best (ρ=0.7,μ=0.5,ε=0.012,κ=10\rho = 0.7, \mu = 0.5, \varepsilon = 0.012, \kappa = 10)46.0
Median of 8143.0
Worst of 8140.4
Bayesian-Agent31.3

Every setting stays above Bayesian-Agent. We have added this sweep to the appendix (Appendix A.12 in the revised paper).


W1(b): Changelog and version signals

Short answer. Thank you for this valuable point. Our paper studies the case where changelogs do not exist. We also ran the baseline you describe.

  • Our setting. Internal APIs, services and team conventions have no changelog and no version number [A, B]. So staleness must be inferred. We model it with two latent variables, Validity and Quality.

  • Your baseline. It reads the release signal for free. On MaskedMig it probes every note of a library whose version changed. On AcmeSDK it probes every note named in the release notes of the month.

Table R2: budget 24, Qwen3.5-27B, 3 seeds

MaskedMigAcmeSDK
Bayesian-Harness (ours)48.4 ± 2.874.6 ± 1.4
Oracle release-aware48.2 ± 1.681.3 ± 1.7

As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The oracles were run on this cluster, while our rows are taken from Tables 1 and 6.

  • MaskedMig. Our method is close to this baseline without any release signal.

  • AcmeSDK. The baseline is higher. This is expected because it reads every release note for free.

We will add both to the paper.

[A] Bi et al. An Empirical Study of Release Note Production and Usage in Practice. IEEE TSE, 2022.
[B] Raemaekers et al. Semantic Versioning and Impact of Breaking Changes in the Maven Repository. JSS, 2017.


AuthorsOpenReview ↗

Response to Reviewer TvTY (Part 2 / 6): W1(c)

W1(c): Baseline strength

Short answer. Thank you for this detailed comment. We ran five more baselines under the main protocol. All stay below our method at every model size.

Two baselines with time and group signals

  • Two counters. Bayesian-Agent with a time clock and a group feature. It is no longer time-blind.

  • Sibling propagation. A failure is passed to the other notes of the same subsystem. It is a rule with no probabilistic model.

Table R3: MaskedMig, budget 24, 3 seeds

Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Bayesian-Agent14.0 ± 4.818.5 ± 2.336.3 ± 7.8
TTL (MemoryBank decay)17.8 ± 3.922.9 ± 2.340.1 ± 7.6
Two counters16.3 ± 1.620.4 ± 2.338.0 ± 3.3
Sibling propagation rule14.9 ± 2.020.0 ± 2.039.3 ± 2.9
Bayesian-Harness (ours)25.5 ± 1.729.9 ± 2.248.4 ± 2.8

As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.

Both new rows stay below ours at every size. So the gain comes from the latent Validity and Quality variables and the shared regime. We will add both rows to the main tables.

Three LLM and memory baselines

  • LLM prioritisation. The LLM ranks which notes to check each period.

  • Mem0-style update. The LLM marks a note as outdated after each task.

  • A-Mem-style links. Related notes are linked, so one broken note flags the others.

LLM calls are free for all three.

Table R4: MaskedMig, budget 24, 3 seeds

Qwen3.5-4BQwen3.5-9BQwen3.5-27B
LLM prioritisation10.6 ± 0.718.0 ± 1.634.0 ± 5.5
Mem0-style update10.6 ± 2.415.3 ± 2.336.9 ± 1.9
A-Mem-style links15.1 ± 1.018.9 ± 2.037.6 ± 1.3
Bayesian-Harness (ours)25.5 ± 1.729.9 ± 2.248.4 ± 2.8

As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.

We will add them to the paper.

Code and logs are in reviewer_TvTY.


AuthorsOpenReview ↗

Response to Reviewer TvTY (Part 3 / 6): W2(a)–W2(b)

W2: Validity and quality separation


Code and logs

  • Paired tests, result files, the low-quality world and figures for W2 and Q4 to Q6 are in reviewer_TvTY.

  • The Oracle with back-off is in reviewer_zpd8.


W2(a): The quality variable

Short answer. Thank you for the careful reading. Table 11 is separate from the separability study, and a paired test shows a clear effect there. Your comment also led us to a new experiment with poor notes. The quality variable helps there too.

  • Why the means in Table 11 look close. Table 11 uses the main protocol and freezes only the quality variable. Its s.d. is taken across three worlds of different difficulty. A paired test removes this effect. On renamed drift, estimated quality wins 45 exams and loses 12 (p<0.001p < 0.001).

  • A new world with poor notes. 35% of the notes are valid but poor. They keep the API name and lose the usage example. No cheap repair is involved. A repair does not fix a poor note, so it is tested while valid, after a repair and as a context note in its subsystem.

Table R5: AcmeSDK, Qwen3.5-27B, budget 12, 3 seeds, 35% poor notes

all exams
Bayesian-Harness (ours)63.8 ± 3.8
Quality frozen at 1 (no quality variable)59.7 ± 3.8
Bayesian-Agent53.7 ± 3.5

In this world, estimated quality wins 73 exams and loses 31 (p<0.001p < 0.001).

  • Probes spent on valid notes. Freezing quality raises their share from 62.8% to 81.0% (Table 3). So the quality variable lowers this cost. The rest is the price of catching stale notes under a budget.

We will add the low-quality world to the appendix.

W2(b): Statistical evidence

Short answer. Thank you for this point. You are right that the s.d. alone is not enough, and paired tests are the right tool [C]. We now report paired McNemar tests on AcmeSDK and a conservative bound on MaskedMig. Every pooled comparison is significant.

  • The test. It counts the exams where exactly one of the two policies passes, with three seeds pooled.

  • Every AcmeSDK configuration. Bayesian-Harness wins more paired exams than the baseline at every model size and against both baselines.

  • Pooled. When we pool the three model sizes per corpus, every comparison is significant.

Table R6: Pooled over 4B / 9B / 27B, budget 24

wins / lossesp
Bayesian-Harness vs TTL, AcmeSDK375 / 194p<0.001p < 0.001
Bayesian-Harness vs TTL, MaskedMig489 / 381 (bound)p<0.001p < 0.001
Bayesian-Harness vs Bayesian-Agent, AcmeSDK491 / 162p<0.001p < 0.001
Bayesian-Harness vs Bayesian-Agent, MaskedMig489 / 324 (bound)p<0.001p < 0.001

The AcmeSDK rows are paired tests on the per-exam runs of Table 6. For MaskedMig we did not keep per-exam records, so the rows give a conservative bound from the accuracies of Table 1. They list the exams passed by each policy out of 1,413. We assume the two policies never pass the same exam, which gives the largest possible number of discordant exams. The true p-value can only be smaller.

The two largest ablation drops are significant too.

  • No recovery. It loses 71 exams and gains 24 (p<0.001p < 0.001, renamed 38/9).

  • No error channel, renamed drift. It loses 25 exams and gains 10 (p=0.017p = 0.017).

We will add the paired tests to the revision, and paired counts to every ablation row.

[C] Dietterich. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 1998.

Reviewer_zpd8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Bayesian-Harness maintains agent notes under environmental change by separating validity from usefulness and sharing change evidence across related notes. It allocates verification and repair budgets using these beliefs, with experiments on masked real-library migrations and a controlled SDK benchmark.

Critical strengths
  1. Separating outdated notes from valid but low-quality notes is useful. Table 3 tests this distinction directly and includes a simple two-counter comparison.

  2. The study combines 157 real-library migrations with controlled drift experiments. Table 1 reports gains of 3.1–8.3 percentage points over the strongest listed baseline across five models; Table 4 tests grouping and drift mismatch.

Critical weaknesses
  1. Section 4.3 reduces Bayesian-Agent [BA26] to a feature-free Beta posterior, so the comparison does not evaluate its full method. I would appreciate a main-setting comparison with the two-counter policy already used in Table 3. Please also distinguish persistent API changes from the probabilistic execution failures studied in ResiliBench [RB26]; this would clarify which failures the validity–quality model is intended to separate.

  2. Sections 4.3 and 4.4 leave failed repairs unclear: repairs count only after passing an assertion, yet staleness resets after an attempt. If the old note remains unchanged, why reset its stale status? This also affects the small-budget comparison with the retrying oracle.

I lean toward acceptance for the useful validity–quality distinction and supporting experiments, subject to clarifying repair semantics and comparison scope.

Questions for authors
  1. After a failed repair, what happens to the note, its staleness belief, and its quality counts? Is retry suppression separate from the belief update?

  2. Are main-setting two-counter results available, and which shared features could [BA26] use? Relative to [RB26], how does a probe distinguish a persistent API change from a transient execution failure? A discussion of the assumptions would suffice.

[BA26] Wu, X., Yang, C., Liu, H., Lin, X., Zhang, W., Shi, Z., Jiang, X., Xu, C., Li, J., and Guo, J. (2026). Bayesian-Agent: Posterior-Guided Skill Evolution Across LLM Agent Harnesses. arXiv:2606.08348. https://arxiv.org/abs/2606.08348.

[RB26] Ao, R., Min, Z., Zhu, T., Yin, W., and Wang, X. (2026). ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic Environments. International Conference on Learning Representations. https://openreview.net/forum?id=KTZ56LG7jZ.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zpd8 (Part 1 / 3): W1

We thank you for the comments that made the paper clearer. We have created a new branch, reviewer_zpd8, for you in response to your concerns.

W1: Bayesian-Agent, two counters and ResiliBench

Thank you for these suggestions.

Bayesian-Agent conditions its posterior on context features, which our method does not have. We therefore ran two-counter policy experiments by adding a group feature and a time clock. As you suggested, we ran it under the main-table protocol of MaskedMig.

Table R1: MaskedMig, budget 24, 3 seeds

Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Bayesian-Agent14.0 ± 4.818.5 ± 2.336.3 ± 7.8
TTL17.8 ± 3.922.9 ± 2.340.1 ± 7.6
Two counters16.3 ± 1.620.4 ± 2.338.0 ± 3.3
Bayesian-Harness (ours)25.5 ± 1.729.9 ± 2.248.4 ± 2.8

As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.

Two counters stays below our method at every size.

ResiliBench looks at random failures; we instead focus on lasting changes. Our method splits the cause of a failure into two factors, Validity and Quality.

Validity asks whether the API has changed.
Quality asks whether the note itself is good.

Random failures are not part of our setting.

Code, run logs and a README are in reviewer_zpd8.

AuthorsOpenReview ↗

Response to Reviewer zpd8 (Part 2 / 3): W2

W2: Failed repairs and the Oracle

Thank you for pointing this out. The paper is not clear here, and you are right about the small-budget comparison. The Oracle retries failed notes in every period, even when the model is not able to fix them. Our method can stop this meaningless retry, and this is also the main reason why we outperform the Oracle at some scales.

Based on your suggestion, we gave the Oracle the same kind of back-off, with no retry for 3 months after a failed repair. It then performs better than our method, as expected for an oracle.

Table R2: AcmeSDK, budget 12, 3 seeds

Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Oracle (retry every period)32.6 ± 4.240.4 ± 6.169.3 ± 3.0
Oracle with back-off46.1 ± 1.655.0 ± 4.573.8 ± 3.2
Bayesian-Harness (ours)35.1 ± 4.146.6 ± 4.270.9 ± 2.2

All rows are re-run on the same new stack so that the back-off is compared fairly. They differ slightly from Table 6, which stays unchanged.

We will add the Oracle with back-off to the paper and revise the claim.

Code, run logs and a README are in reviewer_zpd8.


AuthorsOpenReview ↗

Response to Reviewer zpd8 (Part 3 / 3): Q1–Q2

Q1: After a failed repair

Thank you for the question. After a failed repair:

  • First, the note stays the same and the agent will pass the old note in its exams.

  • Then, we set its staleness to zero, so we stop treating it as broken for now but put it at the end of the queue.

  • At the same time, its quality counts stay the same. A failed fix does not tell us whether the note is good or bad.

  • Finally, if the note is still broken, it fails more exams, its staleness goes up again, and we try to fix it later. We will make this clear in the paper.

Based on your suggestion, we ran an experiment that keeps it in the broken status and retries it every episode, since it still keeps the highest score compared with other notes. The results are shown in the following table:

(AcmeSDK, budget 12, 3 seeds)Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Reset after a failed repair (ours)35.1 ± 4.146.6 ± 4.270.9 ± 2.2
Keep as broken and retry25.9 ± 3.337.8 ± 3.168.5 ± 1.8

All rows are re-run on the same new stack so that the two rules are compared fairly. They differ slightly from Table 6, which stays unchanged.

From the table, we can see this mechanism hurts the performance at 4B and 9B as it wastes the budget.

The code, the 18 run logs and a README are in reviewer_zpd8.

Q2: Two counters and transient failures

Yes, the two-counter results are in Table R1. Two counters stays below our method at every model size. The features [BA26] could share in our setting are the time since the last check and the success rate of the library. The two-counter policy uses both.

On [RB26], our environment is deterministic. A probe runs the note's code against the library version of that month. So a failed probe means a lasting API change, and we update Validity. ResiliBench assumes that failures happen at random. In our setting the randomness comes from the agent, and Quality models it. We will state this assumption in the paper.

Two-counter logs are in reviewer_zpd8.


[BA26] arXiv:2606.08348
[RB26] https://openreview.net/forum?id=KTZ56LG7jZ


Thank you very much,
Authors

Reviewer_5XufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Bayesian-Harness, a probabilistic memory maintenance framework for LLM agents. The central idea is to distinguish between an intrinsically low-quality memory and a previously reliable memory that has become invalid due to environmental changes.

The method combines a Markov-modulated staleness model with hierarchical Bayesian quality estimation and a budget-constrained verification and rewriting policy. The authors introduce two benchmarks, MASKEDMIG and ACMESDK, and report consistent improvements over several maintenance baselines across five LLMs.

Critical strengths
  1. A meaningful research problem. The distinction between validity and quality is conceptually useful. Modelling correlated environmental changes through shared subsystem states also provides a reasonable motivation for propagating evidence across memory entries.
  2. Strong and comprehensive experiments. The paper evaluates 157 real API migrations, a controlled 36-month SDK environment, multiple model families and several maintenance budgets.
  3. Useful ablations and reproducibility efforts. The paper examines recovery dynamics, subsystem-level sharing, hierarchical quality pooling, and error-message evidence. It also investigates departures from the assumed drift process, provides detailed generation protocols, and releases code and datasets.
Critical weaknesses
  1. Comparison with baselines. The main experiments primarily compare against TTL and Bayesian-Agent, neither of which exploits shared subsystem information in the same way. The repository includes FactorRuleFilter, a simpler non-Bayesian method that propagates suspected invalidity across related memories, but it is absent from the main results.
  2. Potential inconsistency between the paper and implementation. Equation 7 and Appendix A.1 describe updating staleness period by period as the regime belief evolves. However, the released _touch_entry implementation appears to multiply the one-period hazard, evaluated using the final regime belief, by the number of elapsed periods.
  3. Generalizability of the evaluation setting. ACMESDK restricts documentation and introspection, and makes changelogs available only after a memory item has been selected for maintenance. While these choices enable controlled evaluation, they may favor the proposed approach over alternatives that directly consult available documentation. The real-library benchmark and command-line extension partially address this concern, but the results may not generalize to less restricted environments.
Questions for authors
  1. How does Bayesian-Harness compare with FactorRuleFilter and simpler subsystem-aware heuristics under the same budgets?
  2. Is the multi-period update in _touch_entry an intentional approximation? How does it affect the results compared with the update described in the paper?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5Xuf (Part 1 / 3): W1

We thank the reviewer for the careful reading of both the paper and the code. Your comments made the paper better. We have created a new branch reviewer_5Xuf in our anonymous GitHub repository, which includes all new experiments and logs.

W1: Comparison with baselines

Thank you for reading our code so closely. FactorRuleFilter is one step in the design of our method, not a full method, so we left it in the code instead of including a comparison in our paper. We have now run it at all three Qwen3.5 sizes under the main-table configuration.

Table R1: MaskedMig, budget 24, 3 seeds

Qwen3.5-4BQwen3.5-9BQwen3.5-27B
Oracle27.4 ± 1.938.9 ± 2.552.2 ± 1.3
None0.6 ± 0.60.6 ± 1.112.1 ± 0.6
Static0.6 ± 0.01.3 ± 2.28.9 ± 8.3
Bayesian-Agent14.0 ± 4.818.5 ± 2.336.3 ± 7.8
TTL17.8 ± 3.922.9 ± 2.340.1 ± 7.6
FactorRuleFilter (new)14.9 ± 2.020.0 ± 2.039.3 ± 2.9
Bayesian-Harness (ours)25.5 ± 1.729.9 ± 2.248.4 ± 2.8

As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.

FactorRuleFilter shares subsystem information and stays below our full method at every size. So the improvement comes from modelling validity and quality, not from sharing subsystem information alone, which can be seen from our ablation experiments as well (Table 2 in our paper).

Code and run logs are in reviewer_5Xuf.

AuthorsOpenReview ↗

Response to Reviewer 5Xuf (Part 2 / 3): W2-W3

W2: Paper vs. implementation

Thank you for this observation. You are right. Our code uses a first-order approximation of Eq. 7, and the paper did not explicitly describe it. We did the following four things to address this concern:

  1. We ran the exact update. We re-ran our method on MaskedMig with the period-by-period update as written in the paper (5 seeds).

    Table R2: MaskedMig, Qwen3.5-27B, budget 24, 5 seeds

    Accuracy (%)
    Approximation (used in the paper)48.5 ± 2.2
    Exact period-by-period update46.9 ± 4.0

    The two differ by 1.6 points, which is inside the seed-to-seed spread (about 4 points). Our conclusions do not change.

  2. We checked the scope AcmeSDK updates the filter every month so it is not affected. MaskedMig is somehow affected as its update spans three months.

  3. We will revise the paper. Eq. 7 and Appendix A.1 will describe the approximation we use, and the appendix will report this comparison.

  4. We release both versions. The code now has both exact and approximation version in the new branch.

Code and Logs are in reviewer_5Xuf.

W3: Evaluation setting

Thank you for raising this question. We agree with your core point that a simple changelog read-and-fix should work well. In this paper, we consider more scenarios where the changelog is inaccessible [A, B], such as company-internal system APIs, financial information, or conventions. Therefore, the notes are the only material that the agent can read and maintain. AcmeSDK is the benchmark that imitates this scenario.

We have further proposed a less restrictive benchmark in our paper, MaskedMig, where the real packages keep their introspection capabilities, so the agent can explore the sandbox to find the new API. Our method is still the best policy there. Accessible changelogs correspond to our Oracle, which reads every changelog for free. Without this signal, our method is close to it on MaskedMig (48.4 ± 2.8 against 48.2 ± 1.6 with Qwen3.5-27B, the oracle run on our rebuttal stack).


[A] Bi et al. An Empirical Study of Release Note Production and Usage in Practice. IEEE TSE, 2022.
[B] Raemaekers et al. Semantic Versioning and Impact of Breaking Changes in the Maven Repository. JSS, 2017.


AuthorsOpenReview ↗

Response to Reviewer 5Xuf (Part 3 / 3): Q1–Q2

Q1: FactorRuleFilter

FactorRuleFilter is one step in the design of our method, not a full method. We ran it under the same budgets, and the results are in Table R1. It stays below our method at every size.

Logs and tables are in reviewer_5Xuf/W1_FactorRuleFilter.

Q2: Multi-period update

Yes, it is intentional, and thank you for checking it. _touch_entry uses a first-order form to save computation. The exact update changes the result by 1.6 points, which is inside the seed spread of about 4 points (see Table R2). Our conclusions do not change.

Code of both update rules is in reviewer_5Xuf/W2_exact.


Thank you very much,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We thank you for the time and effort you spent reviewing our paper. We have answered every comment in the individual responses. Below we summarise our contributions, the new experiments and the new discussions.

For simplicity, we refer to Reviewers 5Xuf, zpd8 and TvTY as R1, R2 and R3 in this response.

We ran all the suggested experiments, and we will include them in the revised paper.


Core Contributions of Our Work

  1. A Bayesian model of knowledge staleness. We model the staleness of agent knowledge as a generative process with two latent variables, Validity and Quality. Each one has a hierarchical Bayesian structure that forms a Markov chain.

  2. Two datasets. We extracted 18 real Python libraries' version changes and formed them as the benchmark of MaskedMig. As some reviewers mentioned regarding changelog, we actually built a company internal SDK change, named AcmeSDK. We have released both in our anonymous github.

  3. Evaluation across model sizes. From 4B to 27B models and Gemini 3.1 Pro, Bayesian-Harness is around 3.1 to 8.3 points above the strongest baselines.


New Experiments In This Rebuttal

  1. Stronger baselines. We have added experiments of sibling propagation rule (R1, R3), the two-counter policy with a time clock and a group feature (R2, R3), and three LLM and memory baselines (R3).

  2. Stronger references. We added two release-aware oracles (R1, R3). Our method is close to the oracle on MaskedMig (48.4 against 48.2). We also added an Oracle with back-off (R2, R3).

  3. Checks of our method. As reviewer (R1) mentioned, we used a first-order approximation to save computational cost. We therefore added the exact period-by-period experiments based on Eq. 7. The difference is around 1.6 points, within the seed variance (R1).

  4. Statistics and robustness. Following R3's suggestion, we ran McNemar tests to compare with TTL and Bayesian-Agent. We also built a new world where 35% of the notes are still valid but of poor quality. Finally, we varied ρ\rho, μ\mu, ε\varepsilon and κ\kappa on held-out worlds (R3).


New Discussions during Rebuttal

  1. Scope. Our method targets the case where no changelog exists. A release signal can be added as evidence, so the two are complementary (R1, R3).

  2. Paper and code. Eq. 7 and Appendix A.1 will describe the first-order approximation used in the code (R1).

  3. Failed repairs and random failures. We explain what the method does after a failed repair. We will also state that our environment is deterministic, so a failed probe means a lasting change (R2).

  4. Baselines and the Oracle. We explain why our reduction of Bayesian-Agent is faithful to its maintenance rule (R3). We will revise the small-budget claim and use the Oracle with back-off as the reference (R2, R3).


All new code, logs and result files are in our anonymous repository, with one branch per reviewer.

We thank the reviewers again and are happy to answer further questions.


Thank you very much,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.