- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A good framing, and good question to ask. Relevance is present, but there are inconsistencies int he results.
Critical strengths
The question is useful: which part of an agent loop (query choice, memory, stopping) drives which outcome? The ACT/OBSERVE/UPDATE/STOP split is a clean way to ask it.
The state-fidelity analysis (B.5) is strong.
Critical weaknesses
There is no guarantee that the abstentin result is from calibration and not from abstention from OMS? There are multiple cases where OMS refuses more on quesitons not having answers. Additionally, a policy given no corpus at all scores 94.3 on LoCoMo's unanswerable stratum, because abstaining is the right answer there (B.11, p.24). The stratum rewards caution on its own. Abstention rate on unanswerable questions alone therefore cannot show calibration. The paper needs a threshold-free measure, such as selective accuracy or a utility that scores abstentions and errors on all questions, and it should compare arms at matched abstention rates.
Since the "ownership" effect is really an effect of the extractor prompt. The authors write: "We never ran the reverse condition, an OMS-style extractor prompt placed inside that call, so we do not claim that our particular instruction is what matters" (p.8, lines 417 to 418). So the paper never tests the claim in its title. A single control (the OMS extractor prompt placed inside the policy's own call) would separate "who writes the state" from "what the instruction says".
The Table 2 headline doesn't reproduce, the same rerun window is thin, and token cost swings widely.
OMS fails on both coding and multi-hop, the two settings closest to the paper's own motivation.
"The state term is a weak threshold knob and a decisive structural one" (B.8) is circular. Without OMS the growth condition is true by construction, so the gate becomes a repetition detector. That shows the gate needs some measure of state change, not that OMS is the right one.
Questions for authors
- Can you run the missing control, the OMS extractor prompt inside the policy's own call? Without it, how do you separate ownership from prompt content?
- Can you report a selective-prediction measure (accuracy vs coverage, or a utility over all questions) so that higher abstention is not counted as better calibration by default?
- Why does OMS not improve Iter-RetGen, the strongest native harness, in any domain? What is the gain over the best native system per domain?
- The abstract says rewriting no longer beats intact frontier harnesses, yet Table 19 shows +10.2 and +7.8 for gpt-5.6-terra on LoCoMo and MuSiQue. Which is right?