- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies state maintenance in longitudinal clinical memory. An earlier fact may stop describing the patient’s current state while remaining useful for understanding their treatment history. STAM separates ACTIVE and HISTORY memories, combines semantic retrieval with clinical relations to identify facts affected by new records, and updates their status. Historical information is then served selectively at query time. The evaluation covers four clinical benchmarks, state-maintenance diagnostics, and approximately context-matched comparisons.
Critical strengths
-
The clinical motivation is clear, and the workflow makes sense. Outdated medication doses and resolved symptoms should not be treated as current facts, but removing them would lose useful history. Separating state transitions from permanent deletion is a sensible design choice.
-
I appreciate the evaluation beyond final QA scores. At the same candidate budget, adding relation-based discovery improves supersession recall from 51.6 to 53.9 and reduces false archival from 12.3 to 7.6. The context-length controls and historical-access ablations also help explain where the method is useful.
Critical weaknesses
-
I am not yet convinced that the full state-maintenance design provides a sufficiently consistent downstream benefit. The gains over append-only memory are 3.24, 0.80, 1.46, and 0.95 points across the four benchmarks, without significance results for this central comparison. The matched-context advantages are also more convincing on MIMIC-QA than elsewhere. These results suggest that state maintenance can help, but do not yet establish the necessity of the full design.
-
The novelty needs closer comparison with related approaches. MedCache, StateMem, and A-TMA already address temporal validity, supersession, relational dependencies, or state-aware evidence serving. The paper discusses these methods but does not evaluate against them. Clinical relation-based impact discovery seems to be the clearest additional contribution, yet its value relative to these closer alternatives remains
Questions for authors
- Could you provide paired confidence intervals or significance tests for the append-only comparison, and confirm whether evidence-serving settings are matched?
- What is the most consequential methodological difference from the closest approaches? Are direct comparisons available under a common writer, reader, and evidence budget?
- How are the gold supersession pairs and false-archival labels constructed, and how are the metric denominators and annotation quality determined?
5 comments
Response to Reviewer 5PTM (Part 2/3): Write FT and Closest Baselines
Memory-side adaptation. Table 1 reports an MMB Write FT improvement from 37.4 to 39.3. Under the benchmark harness's reader prompt and decoding settings with a shared judge, STAM + Write FT improves 36.92 to 43.82: +6.90, Holm-adjusted over three benchmarks, p=.012. Only write modules are adapted in this MMB experiment; the Qwen3-4B reader and stock retriever remain frozen. Evaluation personas are disjoint from adapter-training data.
Methodological differences and closest baselines
Q2/W2. What is the most consequential methodological difference?
STAM addresses clinical impact discovery: identifying which earlier assertions require reconsideration when new clinical evidence arrives. For example, a treatment change can affect a related monitoring plan. Relations such as treatment-for and monitoring-for supply candidate paths alongside semantic retrieval; the updater then adjudicates each candidate's status.
Our controlled experiments use a B=48 updater candidate budget, which bounds the memories passed to the updater. Total write compute also includes extraction and relation linking. The fixed-budget controls in Part 3 measure the relation channel's contribution to discovery. Part 1 separately measures historical access.
State updates, dependency propagation, and history retention have important precedents. STAM's contribution is their clinical configuration, with relation-guided discovery of affected assertions and selective access to retained evidence.
StateMem: common-model comparisons
Our reimplementation follows StateMem's described whole-state ingestion procedure (Fan et al., Sections 5.1–5.3). With a 131,072-token encoder limit, it exceeded the context window on all five MMB personas and five of twelve MedLoCoMo patients.
The paper permits replacing whole-state ingestion with slice-based retrieval but does not report results for that variant. We therefore implemented bounded semantic top-K adaptations: K=48 on both benchmarks and K=200 on MMB. K denotes the encoder's write-time candidate count, not the reader's evidence-token budget.
For this new comparison, both STAM and StateMem use Qwen3-4B-Instruct-2507 as writer and reader, with three reader draws.
| StateMem configuration | STAM minus StateMem [95% CI] | StateMem/STAM write FLOPs |
|---|---|---|
| MMB, K=48 | +1.03 [−3.06, +4.95] | 0.92× |
| MMB, K=200 | −1.06 [−5.12, +3.30] | 3.8× |
| MedLoCoMo, K=48, quotes in both arms | +1.78 [−1.40, +4.85] | 2.07× |
MMB uses macro accuracy; MedLoCoMo uses type-macro Token-F1. The MedLoCoMo comparison approximately matches memory-content words, with medians of 650 and 637; both methods receive source quotes. These comparisons use common models and the stated serving configurations. We will report per-arm read budgets and rendered lengths explicitly.
Estimated write FLOPs show a configuration-dependent trade-off: StateMem costs 2.07× STAM on MedLoCoMo; on MMB, K=200 costs 3.8×, while K=48 costs 0.92×.
StateMem propagates dependency flags deterministically and excludes superseded units from its described answering stage, although it retains them for auditability. STAM uses clinical relations to discover candidates for adjudication and selectively serves historical evidence. The historical-access benefit in Part 1 is measured within STAM.
A-TMA: discovery and training requirements
A-TMA constructs candidates from host retrieval and existing state links, then applies a trained Sentry gate and a separately trained audit model (Shi et al., Section 4.3). It also preserves historical states and transition links.
STAM's base impact-discovery modules combine semantic retrieval with clinical-relation paths using prompted pretrained models; impact discovery requires no task-specific training. We could not locate the trained artifacts required for a faithful A-TMA evaluation, so this comparison is methodological.
Response to Reviewer 5PTM (Part 3/3): MedCache and State-Level Diagnostics
MedCache: LongHealth evaluation
On 400 LongHealth questions, STAM scores 63.5% accuracy [58.7, 68.1], compared with MedCache's reported 44.0%. We use STAM's published clinical write configuration and top-20 serving, MedCache's Qwen3.6-35B-A3B reader, greedy decoding, and the LongHealth prompt.
This uses a published result, not a MedCache rerun. Our Full Context calibration is 6.3 points above their reported value, indicating a setup difference. We therefore report this as an additional benchmark evaluation and published-reference comparison.
The revision will include these implementations and configurations, distinguishing the common-model StateMem results, the published MedCache reference, and the component controls within STAM.
Label construction, denominators, and state-level diagnostics
Q3. How are supersession pairs and false-archival labels constructed?
The benchmark provides annotated evidence points for state-update questions. The supersession pairs used in our diagnostics are silver labels derived from those annotations. A deterministic script orders each question's evidence points by time and pairs the latest point with earlier points. The source evidence is benchmark-annotated. We conducted a blinded model-judge audit of 71 silver predecessor pairs spanning all 20 personas. GPT-5 and Claude Opus 4.8 labelled the pairs, with Gemini 2.5 Pro resolving disagreements. Inter-judge agreement was 0.95. The judges classified 77% as describing the same evolving patient state and 61% as supersessions.
The diagnostics evaluate memory state when the session containing the later evidence becomes visible:
- Pair recall: among pairs whose older evidence matches a stored memory, whether that memory has been archived.
- False-archival proxy: whether the newer memory is already archived at that point.
- Impact ceiling: whether an older memory reaches the updater's candidate set by the time its supersession is observed.
- Cascade discovery: whether candidates at a dependent session contain a memory from its triggering session. This uses the benchmark's own
triggered_bydependency annotations.
Metric denominators
| Quantity | Unit and denominator |
|---|---|
| Candidate supersession pairs, personas 1–5 | 1,398 automatically derived pairs |
| Pair recall, Table 5(a) | 1,344 pairs whose older evidence matches a stored memory |
| False-archival proxy, Table 5(a) | 1,398 pairs, evaluated through the newer memory |
| Additional chain-level analyses, personas 1–5 | 50 supersession chains |
| Additional-persona chain analyses, personas 6–12 | 63 supersession chains |
| Cascade discovery | Evaluated dependent sessions with a triggered_by link |
Table 5 reports pair-level diagnostics; the new confirmatory tests resample supersession chains, a different aggregation unit. We will state the denominators explicitly, identify which metrics use automatically derived labels, and provide the construction and memory-matching procedures.
Replicated candidate-discovery results
The relation channel is evaluated at the same B=48 updater candidate budget. Impact ceiling is aggregated per silver supersession chain:
| Store set and comparison | Impact-ceiling gain [95% CI] | Holm p |
|---|---|---|
| Published MMB store versus no relation linking | +3.99 [+0.64, +7.01] | .034 |
| Second store set, personas 1–5, versus semantic-only discovery | +4.06 [+2.34, +5.76] | .0002 |
| Second store set, personas 6–12, versus semantic-only discovery | +3.50 [+1.32, +5.84] | .0016 |
On the published configuration, cascade discovery also improves by +4.60 points [+0.71, +8.45], Holm p=.034. A separate per-chain supersession-recall analysis on that configuration reports a +2.93-point improvement over no relation linking, p=.003.
These measures distinguish candidate availability, the updater's archival outcome, and downstream QA. The discovery benefit replicates across store sets and persona cohorts. The clinical relation vocabulary is configurable; the supported contribution is bringing affected memories into the update decision.
We will make the label provenance, metric definitions, denominators, and aggregation units explicit in the revision and release the diagnostic construction procedure.
Response to Reviewer 5PTM (Part 1/3): Paired comparisons and downstream contributions
Thank you for recognizing the clinical motivation and the value of evaluating state maintenance beyond final QA. We have added paired tests, repeated serving controls, and comparisons with closely related methods.
Q1/W1. Paired significance tests and evidence-serving settings
Table 4 reports 37.40 macro accuracy for STAM versus 34.16 for append-only memory on MMB, a +3.24-point gain. We complement the paper's results with additional paired analyses and repeated serving controls.
Additional paired analysis. For the other benchmarks, the following confidence intervals apply to the corresponding matched-run estimates:
The three additional matched-run point estimates favor STAM. We report these additional analyses separately from Table 4.
Repeated serving controls. We evaluate 694 MMB questions from personas 6–12 using a shared maintained memory store. STAM selects from ACTIVE and HISTORY separately; merged serving ranks these memories together. This tests serving organization on the same memories, rather than independently rebuilding an append-only write pipeline.
With the Qwen3-4B reader, both arms use the same retriever, graph read quota, 1,707-word evidence budget, and reader protocol. Served medians are 1,665 words for STAM and 1,684 for merged serving. Neither arm displays ACTIVE/HISTORY labels.
Across the broader grid of 13 prespecified settings, no difference survives max-T correction.
The GPT-5 arms use the same temporal prompt. STAM uses rank-ordered evidence, state tags, and successor checking. The merged arm's word budget targets STAM's median evidence-token volume: 4,049 versus 4,068.5 tokens.
STAM additionally maintains an explicit distinction between current and historical state. Part 3 evaluates the write-time contribution through fixed-budget discovery diagnostics.
Which components provide measurable downstream benefits?
Historical access. On the same 694 questions, we compare ACTIVE+HISTORY with ACTIVE-only serving, refilling ACTIVE-only packets with ACTIVE memories to match the evidence budget. The maintained store and reader are fixed. Each arm uses three draws.
Scores are macro accuracy. Paired question-level p-values are <.001 and .012, respectively. On temporal-localization and state-update questions, gains are +9.64 and +11.67 points, respectively. This establishes the value of historical access under the matched-budget control.
Evidence beyond MIMIC-QA. In a prespecified GPT-5 comparison on 469 MMB questions whose raw records exceed the input window, whole-memory serving scores 64.18% versus 57.14% for Full Context: +7.04 [+2.99, +11.09], Holm p=.002; persona-level p=.031, using 38% of Full Context's input tokens.
Both arms use high reasoning effort, the same temporal instructions, and one draw. Whole memory serves all visible extracted memories chronologically; Full Context supplies raw turns up to a 272K-token input allowance. This tests evidence retention under truncation. Part 2 adds LongHealth.