- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
POLO proposes a new paradigm for LLM-based programming: rather than consisting of traditional code or one-shot Agent calls, software is built from a collection of persistent LLM objects, each with private, structured, persistent state that can be modified in natural language at runtime. The paper's central argument is that LLM agents should maintain structured, variable-like state, much like traditional programs, instead of relying primarily on full transcripts, summaries, or retrieved histories. Each event is processed using only the object definition, its current state, and the new message, after which the intermediate reasoning process is discarded. The authors also introduce FlowStateBench to evaluate state-dependent workflows spanning weeks or months.
Critical strengths
- The motivation is impressive and clear. The authors reframe “LLM memory” as “program state,” directly investigating whether LLMs can serve as software.
- FlowStateBench targets an important setting. The benchmark evaluates systems on scenarios involving interleaved events across multiple entities, distracting events, and mid-workflow rule changes, making the tasks more complex and realistic.
- The authors have conducted extensive experiments to demonstrate the effectiveness of their method.
Critical weaknesses
-
I think the experimental comparison is not entirely fair. The baselines use GPT-5.5 nano for compilation, while POLO uses GPT-5.4. According to the ablation study in Table 2, the choice of model has a substantial impact on performance, and the results show that GPT-5.4 performs significantly better than GPT-5.5 nano. In addition, tool access is not fully aligned. POLO can use tools to read the states of other objects, while the baseline methods do not have equivalent capabilities. Therefore, the experimental results are not very convincing to me.
-
The authors evaluate their method exclusively on a self-constructed benchmark, which may favor POLO by design. I would like to see additional comparisons on other public benchmarks to provide more convincing evidence of the method's generalizability and robustness.
Questions for authors
- What if POLO and other baselines utilize the same LLM for compilation ( and the same setting )?
- How would POLO perform against other baselines on some other public benchmarks? Overall, I would like to see a more fair comparison.
2 comments
I appreciate the authors' effort. I will maintain my positive score.
Response to reviewer dtz1
We thank the reviewer for the positive assessment, and for recognizing the reframing of LLM memory as program state and the design of FlowStateBench. We answer the reviewer's concerns about the fairness of the comparison below; the general comment summarizes the revisions.
1. The same model for compilation (Weakness 1, Question 1)
We tested all three ways of giving both systems the same models:
2. Tool access (Weakness 1)
read_shared_statewas left in the code by mistake from an earlier object graph, and we disclosed it (page 9, Appendix F). In the submitted runs, it was used in 6.2% of the event processing, and those executions had a lower pass rate (49.5% vs. 61.2% overall). The revised runtime removes the tool, and all current results, including the 85.5% above, are without it.3. Evaluation on a public benchmark (Weakness 2, Question 2)
The paper already includes an external benchmark. Section 6 reports results on Memora (Uddin et al., 2026), a public long-term memory benchmark. We extended this evaluation to include baseline comparisons on Memora.
POLO leads on every inspection time (weekly, monthly, and quarterly). All baselines use gpt-5-nano with the same task contract and scoring. Letta and OpenClaw rely on the agent to choose what to write using generic memory tools, and in our checks, they missed records, so they couldn't answer those questions correctly.
We hope these results address the reviewer's concerns about fairness, and we would be glad to answer any further questions.