- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper formulates selective observation refresh for tool-using agents operating in dynamic environments as a budgeted belief-repair problem. Rather than relying on recency heuristics or assuming independent per-source utility, the authors propose a set-conditioned critic trained via paired counterfactual rollouts from replayable checkpoints while keeping the language-model backbone frozen. However, the empirical evaluation relies entirely on a numerical synthetic simulator (using a hard-coded linear/pairwise formula rather than actual LLM execution), and the proposed set-conditioned selector underperforms a simpler Additive Value baseline across most main and multi-seed benchmark settings.
Critical strengths
-
Clear problem formulation: Articulates why observation staleness is not merely a function of age, highlighting how complementary and redundant information affect decision value under a budget.
-
Transparent reporting of negative results and reproducibility: The authors honestly report that the Additive baseline beats their method in aggregate and provide a thorough, multi-seed supplementary appendix (4,000 tasks across 20 seeds) with exact mathematical formulas.
Critical weaknesses
-
Absence of real LLM agent evaluation: The paper evaluates a purely numerical, 6-variable synthetic script requiring only Python and NumPy, with zero LLM agent trajectories or natural language tool outputs.
-
Empirical failure of the main hypothesis: Set-conditioned selection consistently underperforms the simpler, cheaper Additive baseline on in-distribution tasks across budgets and staleness scales, winning only in a single sub-family (Delivery Coordination) and OOD/long-horizon bar charts.
Questions for authors
-
What accounts for the discrepancy between Figure 2 and Figure 5? Why does the Set-conditioned method achieve 0.59 success at in Figure 2, while "Paired" supervision at in Figure 5 achieves 0.72 success?
-
How are unseen templates, unseen sources, and long-horizon drift (Figure 7) implemented? Given the fixed 6-source lookup formula in Appendix A.1, how does the selector generalize to unseen source combinations if relies on a dictionary lookup of sorted source identifiers?