- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets maintenance of an LLM agent's external memory (notes about APIs/tools) as the environment drifts. It proposes Bayesian-Harness, which assigns each memory item two latent variables:
- Validity: a binary absorbing state whose per-period hazard is modulated by a two-state continuous-time Markov regime (STABLE/CHANGING) shared across a subsystem.
- Quality: the success probability while valid, modelled with a hierarchical Beta–Bernoulli prior pooled within the subsystem.
Observations (test outcomes, classified errors, active probes) update the joint posterior. A budgeted maintenance policy probes items ranked by usage-weighted predicted staleness and rewrites the ones verified as stale.
The paper introduces two benchmarks:
- MASKEDMIG: 157 real API removals with named successors from 18 Python libraries (April 2025 – August 2026). Successor names are hashed to prevent recall from pre-training.
- ACMESDK: a synthetic, LLM-generated company SDK with 12 subsystems and 36 monthly releases, with drift calibrated on real repositories.
Across Qwen3.5-4B/9B/27B and two Gemini models, the method beats None/Static/Random/TTL/Bayesian-Agent on every MASKEDMIG column (+3.1 to +8.3 over the strongest baseline) and in all 15 ACMESDK model–budget cells. Supporting experiments include component ablations, a validity/quality separability test, structural stress tests, and a command-line variant.
Critical strengths
1. A well-posed and useful problem decomposition, backed by a principled, interpretable model. Separating "the world changed" (validity) from "the note was never good" (quality) is a genuinely useful framing for long-lived agent memory. Coupling items through a shared subsystem regime is a clean way to let one failure raise suspicion about sibling items. The model is small, conjugate, and exactly updatable per item. Every constant is listed with its provenance (Table 5), and the authors state that no parameter was tuned on evaluation worlds. This matters because it makes the method deployable and auditable rather than another heuristic score.
2. Careful attention to contamination, plus unusually thorough empirical diagnostics. The paper shows explicitly that unmasked real-library evaluation is confounded: a no-memory agent repairs 60–78% of snippets. It fixes this by hashing successor names consistently across package, changelog and tests. The evaluation also includes:
- per-component ablations on both corpora (Tables 2, 12);
- a quality-freezing ablation (Table 11);
- a targeted separability test with injected low-quality items (Table 3);
- per-seed breakdowns (Table 10);
- stress tests with misfiled groups, a hidden-axis drift, and altered rates/schedules (Table 4);
- a second knowledge modality (ACMECTL, Table 13).
Critical weaknesses
1. The evaluation design favours the model's structural assumptions and withholds the most informative real-world signal. The baselines are correspondingly weak.
(a) Generator–model alignment on ACMESDK. The synthetic world is produced by a generator whose structure closely mirrors the inference model:
- drift concentrated in subsystem-wide "heavy months" (rewriting 40–60% of one subsystem) vs. ρ = 0.5;
- a ~1% background trickle vs. ε = 0.012;
- heavy changes lasting exactly one release vs. µ = 1 per period.
The constants are said to be estimated from XGBoost/OpenCV/SciPy/vLLM, but the near-exact match with the generator is striking. The stress tests help but do not remove the concern. Even the "hidden-axis" world retains two subsystem heavy months and two package-wide heavy months, and every perturbed world still has bursty, grouped drift. A sensitivity sweep over ε, ρ, µ, or a world with genuinely unstructured drift, is missing.
(b) Withheld changelog and version signals. By design, a policy may not use the changelog to choose which item to maintain; it can read the changelog only after selecting an item. In practice, release notes and deprecation warnings are exactly how practitioners know what went stale, and package versions are always observable.
On MASKEDMIG this is especially consequential. Every library's items break in the quarter of that library's release and never recover. A trivial "probe items of any library whose installed version changed" baseline would therefore likely approach the oracle. The protocol turns an observable event into a latent-inference problem, which inflates the value of regime inference. The paper needs either a compelling deployment scenario where this information is unavailable, or a version/changelog-aware baseline.
(c) Baseline strength. Bayesian-Agent, the closest prior work, is reduced to the mean of a per-item Beta posterior because "our memory items carry no context features." Its ranking (1 − posterior mean) has no notion of time, so it cannot become suspicious of unused items. It is a structurally handicapped stand-in rather than a faithful reproduction.
There are also gaps on the non-Bayesian side:
- There is no baseline that propagates a failure to sibling items in the same subsystem (a non-probabilistic analogue of the shared regime). The "two counters" baseline of §4.6 appears only in the separability study.
- There is no LLM-driven prioritisation baseline.
- None of the memory systems discussed in related work (MemoryBank decay, Mem0, A-Mem) is evaluated.
Together, (a)–(c) make it hard to know how much of the gain comes from the Bayesian machinery rather than from an evaluation that rewards it.
2. The headline contribution (validity–quality separation) has a small, noisy measured impact, and the gains on real data rest on limited statistical evidence.
(a) The quality variable is barely tested in the main results. §4.6 states that the main experiments test staleness only: every item starts verified, so there are no low-quality-but-valid items. The internal/external distinction motivating Figure 1(a) is therefore exercised only in the separability study, where low quality is injected artificially via "cheap repairs."
The direct evidence that estimating quality helps is also weak. In Table 11, estimated quality gives 46.0±6.5 / 67.0±3.4, compared with 36.9±11.0 / 63.8±3.6 when quality is frozen at 1. The overall gap is within one s.d. In Table 3, even the full model spends 62–69% of its probes on valid items.
(b) Statistical evidence on the real corpus is thin. On MASKEDMIG the 157 tests are identical across seeds; seeds differ only in when each break is met. The reported s.d. therefore understates uncertainty, and the effective sample is a single set of 157 items.
- The Flash-Lite margin (12.7 vs. 9.6) corresponds to about five tests.
- On ACMESDK, several "best in every cell" claims are within one s.d. of the runner-up, e.g., Qwen3.5-9B at budget 24: 52.8±7.7 vs. TTL 51.2±4.0.
- Ablation drops on renamed drift (4.1–8.0) are comparable to the renamed s.d. (5.7).
Paired tests or bootstrap CIs over tests would be needed to support the "best everywhere" framing.
(c) The oracle is not a meaningful reference. The "oracle" is beaten by the method at budget 12 in three of five models (Table 6) and at 9B/budget 24 on renamed drift (Table 9). Its accuracy also declines over time (81.6 → 75.0 in Table 8) because it retries impossible repairs. As a result, the paper lacks a meaningful upper bound against which to judge how much headroom remains.
Questions for authors
- Please describe a concrete deployment where an agent cannot observe which package or subsystem released, or read release notes, before choosing what to probe. How does Bayesian-Harness compare with (i) a version-bump baseline on MASKEDMIG (probe all items of libraries whose version changed) and (ii) a changelog-aware baseline on ACMESDK? If the method still adds value on top of such signals (e.g., for silent semantic changes, about 20% of changes), that would substantially strengthen the paper.
- Please report the actual measured values of ε, ρ, µ from XGBoost/OpenCV/SciPy/vLLM before any rounding, and explain why they coincide so closely with the generator settings (1% trickle, 40–60% heavy-month coverage, one-release bursts). Please also provide a sensitivity sweep, e.g., ρ ∈ {0.1, 0.25, 0.5, 0.8}, µ ∈ {0.25, 1, 4}.
- Is the feature-free reduction faithful to Wu et al. (2026)? Did you consider giving it simple features (subsystem ID, time since verification) so that it is not structurally time-blind? Please also add a non-Bayesian "propagate failure to subsystem siblings" baseline to the main tables.
- For Table 1 and Table 6, can you report paired significance (e.g., McNemar or a paired bootstrap over tests) between Bayesian-Harness and the strongest baseline per column?
- Can you add a stronger reference, e.g., an oracle that knows staleness and skips repairs that previously failed, so that the gap to an achievable ceiling is interpretable?
- Is there any condition without injected cheap repairs where items are naturally low-quality while valid (e.g., notes written by a weaker model), so that the internal-vs-external distinction is tested without an artificial construction?
8 comments
Response to Reviewer TvTY (Part 2 / 6): W1(c)
W1(c): Baseline strength
Short answer. Thank you for this detailed comment. We ran five more baselines under the main protocol. All stay below our method at every model size.
Two baselines with time and group signals
-
Two counters. Bayesian-Agent with a time clock and a group feature. It is no longer time-blind.
-
Sibling propagation. A failure is passed to the other notes of the same subsystem. It is a rule with no probabilistic model.
Table R3: MaskedMig, budget 24, 3 seeds
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | |
|---|---|---|---|
| Bayesian-Agent | 14.0 ± 4.8 | 18.5 ± 2.3 | 36.3 ± 7.8 |
| TTL (MemoryBank decay) | 17.8 ± 3.9 | 22.9 ± 2.3 | 40.1 ± 7.6 |
| Two counters | 16.3 ± 1.6 | 20.4 ± 2.3 | 38.0 ± 3.3 |
| Sibling propagation rule | 14.9 ± 2.0 | 20.0 ± 2.0 | 39.3 ± 2.9 |
| Bayesian-Harness (ours) | 25.5 ± 1.7 | 29.9 ± 2.2 | 48.4 ± 2.8 |
As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.
Both new rows stay below ours at every size. So the gain comes from the latent Validity and Quality variables and the shared regime. We will add both rows to the main tables.
Three LLM and memory baselines
-
LLM prioritisation. The LLM ranks which notes to check each period.
-
Mem0-style update. The LLM marks a note as outdated after each task.
-
A-Mem-style links. Related notes are linked, so one broken note flags the others.
LLM calls are free for all three.
Table R4: MaskedMig, budget 24, 3 seeds
| Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B | |
|---|---|---|---|
| LLM prioritisation | 10.6 ± 0.7 | 18.0 ± 1.6 | 34.0 ± 5.5 |
| Mem0-style update | 10.6 ± 2.4 | 15.3 ± 2.3 | 36.9 ± 1.9 |
| A-Mem-style links | 15.1 ± 1.0 | 18.9 ± 2.0 | 37.6 ± 1.3 |
| Bayesian-Harness (ours) | 25.5 ± 1.7 | 29.9 ± 2.2 | 48.4 ± 2.8 |
As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The new baselines were run on this cluster, while our row is taken from Table 1.
We will add them to the paper.
Code and logs are in reviewer_TvTY.
Response to Reviewer TvTY (Part 3 / 6): W2(a)–W2(b)
W2: Validity and quality separation
Code and logs
-
Paired tests, result files, the low-quality world and figures for W2 and Q4 to Q6 are in
reviewer_TvTY. -
The Oracle with back-off is in
reviewer_zpd8.
W2(a): The quality variable
Short answer. Thank you for the careful reading. Table 11 is separate from the separability study, and a paired test shows a clear effect there. Your comment also led us to a new experiment with poor notes. The quality variable helps there too.
-
Why the means in Table 11 look close. Table 11 uses the main protocol and freezes only the quality variable. Its s.d. is taken across three worlds of different difficulty. A paired test removes this effect. On renamed drift, estimated quality wins 45 exams and loses 12 ().
-
A new world with poor notes. 35% of the notes are valid but poor. They keep the API name and lose the usage example. No cheap repair is involved. A repair does not fix a poor note, so it is tested while valid, after a repair and as a context note in its subsystem.
Table R5: AcmeSDK, Qwen3.5-27B, budget 12, 3 seeds, 35% poor notes
| all exams | |
|---|---|
| Bayesian-Harness (ours) | 63.8 ± 3.8 |
| Quality frozen at 1 (no quality variable) | 59.7 ± 3.8 |
| Bayesian-Agent | 53.7 ± 3.5 |
In this world, estimated quality wins 73 exams and loses 31 ().
- Probes spent on valid notes. Freezing quality raises their share from 62.8% to 81.0% (Table 3). So the quality variable lowers this cost. The rest is the price of catching stale notes under a budget.
We will add the low-quality world to the appendix.
W2(b): Statistical evidence
Short answer. Thank you for this point. You are right that the s.d. alone is not enough, and paired tests are the right tool [C]. We now report paired McNemar tests on AcmeSDK and a conservative bound on MaskedMig. Every pooled comparison is significant.
-
The test. It counts the exams where exactly one of the two policies passes, with three seeds pooled.
-
Every AcmeSDK configuration. Bayesian-Harness wins more paired exams than the baseline at every model size and against both baselines.
-
Pooled. When we pool the three model sizes per corpus, every comparison is significant.
Table R6: Pooled over 4B / 9B / 27B, budget 24
| wins / losses | p | |
|---|---|---|
| Bayesian-Harness vs TTL, AcmeSDK | 375 / 194 | |
| Bayesian-Harness vs TTL, MaskedMig | 489 / 381 (bound) | |
| Bayesian-Harness vs Bayesian-Agent, AcmeSDK | 491 / 162 | |
| Bayesian-Harness vs Bayesian-Agent, MaskedMig | 489 / 324 (bound) |
The AcmeSDK rows are paired tests on the per-exam runs of Table 6. For MaskedMig we did not keep per-exam records, so the rows give a conservative bound from the accuracies of Table 1. They list the exams passed by each policy out of 1,413. We assume the two policies never pass the same exam, which gives the largest possible number of discordant exams. The true p-value can only be smaller.
The two largest ablation drops are significant too.
-
No recovery. It loses 71 exams and gains 24 (, renamed 38/9).
-
No error channel, renamed drift. It loses 25 exams and gains 10 ().
We will add the paired tests to the revision, and paired counts to every ablation row.
[C] Dietterich. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 1998.
Response to Reviewer TvTY (Part 1 / 6): W1, W1(a)–W1(b)
Thank you for your time and for the insightful suggestions. All new code and logs are in the branch
reviewer_TvTYof our anonymous repository.Code and logs
reviewer_TvTYfor W1 and Q1 to Q3reviewer_zpd8for the two countersreviewer_5Xuffor sibling propagationW1(a): Generator and model alignment
Short answer. We agree with this comment. This is why the main paper reports MaskedMig, which is built from real library changes. AcmeSDK is a complementary benchmark, and its results are mainly in the appendix.
The constants are held out. ε, ρ and μ were estimated on XGBoost, OpenCV, SciPy and vLLM. None of these libraries is in either benchmark.
The result does not depend on them. We swept ρ, μ, ε and κ over 81 settings on a held-out world.
Table R1: AcmeSDK, held-out world, 81 settings. Coverage is the share of exams whose note is valid.
40.431.3Every setting stays above Bayesian-Agent. We have added this sweep to the appendix (Appendix A.12 in the revised paper).
W1(b): Changelog and version signals
Short answer. Thank you for this valuable point. Our paper studies the case where
changelogs do not exist. We also ran the baseline you describe.Our setting. Internal APIs, services and team conventions have no changelog and no version number [A, B]. So staleness must be inferred. We model it with two latent variables, Validity and Quality.
Your baseline. It reads the release signal for free. On MaskedMig it probes every note of a library whose version changed. On AcmeSDK it probes every note named in the release notes of the month.
Table R2: budget 24, Qwen3.5-27B, 3 seeds
As there were more new experiments than we expected, we moved the code to an H200 cluster to get the results before the end of the rebuttal. The oracles were run on this cluster, while our rows are taken from Tables 1 and 6.
MaskedMig. Our method is close to this baseline without any release signal.
AcmeSDK. The baseline is higher. This is expected because it reads every release note for free.
We will add both to the paper.
[A] Bi et al. An Empirical Study of Release Note Production and Usage in Practice. IEEE TSE, 2022.
[B] Raemaekers et al. Semantic Versioning and Impact of Breaking Changes in the Maven Repository. JSS, 2017.