- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper stands on the observation that current "memory" harnesses badly solve "prospective memory" tasks, meant as tasks that need to be deferred and addressed at specific cues (time or triggers). The authors then propose a harness to address this gap, by mainly storing deferred tasks as typed records and not as textual notes. To demonstrate the validity of their proposed solution, they evaluate on PM-Bench against existing memory scaffolds (like A-Mem and Mem0) and report higher performance.
Critical strengths
Prospective memory is indeed a problem neglected by existing "memory" harnesses, I appreciate the effort in this direction. The paper hypothesizes that deferred intentions should be stored as explicit records (trigger, action, channels to monitor, lifecycle state) rather than as textual notes. If the empirical evidence solidly confirmed this hypothesis, that would be a strong contribution to the scaffolds literature.
Critical weaknesses
Critical Weaknesses* Please identify one or two critical weaknesses of the paper that support rejection. For each, explain briefly why it is consequential to your overall assessment. The goal is not to enumerate every negative point, but to identify and substantiate the considerations that most strongly determine your recommendation. Enter “None” if no material weakness can be identified.
The paper does not show that the gains come from the proposed store.
- In PM-Bench, most failures come from models not checking the clock or the hidden channels at the right steps (or not checking them enough at all), and ProMem forces a clock query whenever a timed task is pending, which in practice happens at almost every step. Similarly, channels not yet checked are injected in the prompt as "MUST query state". This alone could explain most of the gains, and indeed the paper reports that the gain is concentrated in time-triggered tasks. To support this, I ran the official PM-Bench code on GPT-5.6-luna with a harness that simply queries every channel at every step, without any intention store, and obtained 93.1±1.3 Set-F1 over 3 runs, above the 89.7 reported for ProMem on the same backbone. A comparison with such a polling baseline is needed to show whether the structure enforced by ProMem adds anything beyond polling alone.
- Apart from time-triggered tasks, the gains are not consistent. On the backbones with the strongest Baseline, ProMem does not improve on cross-day tasks and often does worse. False alarms increase on all backbones. On the 30-day set, ProMem misses more cross-day tasks than Baseline and produces many more false alarms. This goes against the claim that the gain also lies "in maintaining state across days and after revision", which the paper calls "the two core skills of prospective memory", and suggests that ProMem mostly helps the backbones that do not check the clock on their own.
- The paper presents the compact board, "rather than an ever-growing transcript", as a key advantage, but this does not apply to PM-Bench: the full week transcript is short and fits easily in the context window of current models, so the benchmark puts no pressure on context length. Even on their 30-day set, where the transcript is much longer, it still fits, and there ProMem misses more cross-day tasks than the full-context Baseline. No experiment isolates the effect of the compact context, while ProMem needs many more calls than Baseline and costs more.
This is important because the store is the main claim of the paper, and the current experiments do not allow clearly attributing the gains to it.
Anyways, the method is specialized to PM-Bench and the evaluation is weak.
- ProMem mirrors in code the benchmark's own definition of when a task is due, parses the daily plans with templates that match PM-Bench's phrasing, and even its stop-word list includes words such as "tucked" and "unlocked", which come from the fixed cue sentences of the PM-Bench generator. So it is hard to tell how much of it would transfer to a different environment, and the only other evaluation is a 30-day set built by the authors in the same format.
- I am afraid the re-run baselines might not be reliable: the todo-ledger collapses on most backbones, while it is one of the strongest scaffolds in the original PM-Bench paper.
This is important because, even setting aside the first point, the results do not show that ProMem works beyond the benchmark it was built on.
Questions for authors
My main concerns require new experiments and a substantial rewrite, so I do not expect a rebuttal to be enough to change my recommendation. Still, the following points could affect my assessment.
- Could you run the polling baseline on all seven backbones, i.e., the same backbone with the clock and every channel queried at every step and no intention store, and comment on how it compares with ProMem?
- Could you show that ProMem is not specialized to PM-Bench? For example, by evaluating it in an environment whose cues are not produced by the PM-Bench templates, and by explaining how the stop-word list, the parsing templates and the matching threshold were chosen.
- I am afraid some of my concerns, e.g., the compact context, are about the benchmark itself and not only about ProMem. In PM-Bench, a large share of the failures comes from models not checking the clock or the channels often enough. In a short text that simulates a human day, rather than an agentic task, I would not expect a model to check the clock at almost every step, so I am not convinced that these failures measure prospective memory, or any other useful agentic skill. Could you comment on this?