- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper formulates how the external-state checks required by stored intentions(clock, calendar, verifier queries, etc) should share a per-episode budget. It proposes two lightweight policies sharing a hard-cap executor: BudgetPM-Static, trained on replay-constructed labels of whether a check improves the current decision, and BudgetPM-Sequential, distilled from hindsight-DP full-episode schedules. Validation spans two benchmarks (PM-Bench, GoodAI) and three LLM backbones: end-to-end comparison against adapted Mem0/PMA, matched single-variable controls isolating the supervision label and scorer capacity, exact-cost structural analysis, and a fixed-budget load intervention. The central finding is a demand–capacity boundary: future-aware temporal allocation helps at B5/B10 but not at B15/B20, and the boundary moves with opportunity load.
Critical strengths
- Attribution methodology. Matched controls that change exactly one ingredient, paired bootstrap with prespecified power analysis, and exact-cost decomposition cleanly attribute gains to specific mechanisms. This makes the paper's sign-level claims credible even where effect sizes are small, and sets a good example for the field.
- Clear applicability boundary. The paper demonstrates that "future-aware supervision" only works under resource constraints and is useless at B15/B20. This gives practitioners a clear, actionable rule on when to avoid complex timing setups, raising the work above standard benchmark-chasing.
Critical weaknesses
- No proper comparison from the cited literature. When2Tool, BATS, CostBench, INTENT, PIS, and knapsack-bandit routers are cited as the closest lines of work but never run. In Table 1, Mem0/PMA run without an episode-level cap while their very low scores (0.201–0.513) plausibly reflect adapter/workflow mismatch rather than missing allocation, so "outperforms adapted memory-agent workflows" does not evidence the allocation contribution itself.
- The central finding rests on one benchmark, one generator, and unstated assumptions. The scarcity boundary is established only on PM-Bench (GoodAI is never tested at B5–B20); opportunities are near-uniform over episode time (193/197/204/207), a distribution favorable to temporal allocation; and supervision construction assumes queries do not change world state (Appendix A.3), a restriction never stated and violated whenever checking is itself an action. I think the assumptions are far from the real-world, so the paper's main contribution appears to reduce to a property of its own benchmark.
- Over-complicated framing. The method is essentially a budgeted tool-call gate that scores and executes queries under a hard cap; it does not manage memories itself. Terminology like "stored intentions" and "observation capacity" creates unnecessary confusion and mispositions the paper's novelty. The true novelty compared to budgeted tool-calling baselines (e.g., When2Tool) lies in its one-query-serves-many-intentions demand structure, not memory. The authors may frame the work as general budgeted allocation, treat PM as the application, and plainly emphasize this shared-resource structure.
Questions for authors
- Does the B5/B10 boundary persist under deadline-clustered or bursty opportunity processes? Your paired generator appears capable of injecting clustered opportunities.
- Is the method fundamentally restricted to read-only (non-state-changing) checks? If checking can alter the world (e.g., prompting a new supplier), how would the Outcome labels and DP teacher need to change?
- Given that policies are frozen after offline fitting, how would the framework handle distribution drift over long deployments — is periodic re-collection of trajectories the intended answer?
- The hindsight-DP teacher requires caching outcomes for every feasible query at development time and a DP over (TP, FP) states at each integer cost. Both are tractable at benchmark scale but may not scale to realistic settings. Please report the DP's runtime/memory scaling with T, B, and candidate-set size, and clarify the practical limits of this supervision pipeline outside the benchmark.
Thank you for the detailed response and your efforts to address my concerns. The added comparison with Pacing-LB and the matched-mean-cost analysis address most of my concerns, and I appreciate the more careful framing of the conclusions. However, I still have some reservations about the robustness and generality of the gains. I am therefore inclined to maintain my original rating.