acceptodds
Under review as a conference paper at ICLR 2027

POLO: Persistent LLM-Objects with Self-Authored State

Abstract

The dominant paradigm for building software with LLMs is agent-assisted code generation. We ask not whether LLMs can produce software, but whether they can be it. We propose Programmable Online LLM-objects (POLO): software as a set of persistent LLM-objects that own domain responsibilities, communicate through natural language messages, and are created and modified in natural language while running. The paradigm's core proposition is a new memory discipline: an LLM-object holds structured, private, and self-authored state with the semantics of program variables. The object derives its state schema from its natural language definition and can revise it when that definition changes. Each message handling rewrites the values its rules need, keeps dated entries only where a rule counts over a window, and discards the processing trace. Handling depends only on the object's definition, its state, and the incoming message, however long the system has been running. We build FlowStateBench, a benchmark of stateful, long-horizon workflows spanning weeks to months of simulated time, where correct handling of each event depends on accumulated data across the entire history of the workflow, and on rule changes arriving mid-stream. On FlowStateBench, POLO outperforms agent frameworks representing four memory strategies (LangGraph, Letta, CrewAI, and OpenClaw), and replacing its state with a transcript in the same runtime lowers state-dependent accuracy by 22.9 points. On Memora, an external long-term memory benchmark, a self-authored state schema performs comparably to one written by a developer, and both score above our single-agent baselines. The code, benchmark, and generator are released.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
memory, agent, state, program, software, text-to-code, long-horizon, long-running, workflow, json, schema, structured

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dtz1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

POLO proposes a new paradigm for LLM-based programming: rather than consisting of traditional code or one-shot Agent calls, software is built from a collection of persistent LLM objects, each with private, structured, persistent state that can be modified in natural language at runtime. The paper's central argument is that LLM agents should maintain structured, variable-like state, much like traditional programs, instead of relying primarily on full transcripts, summaries, or retrieved histories. Each event is processed using only the object definition, its current state, and the new message, after which the intermediate reasoning process is discarded. The authors also introduce FlowStateBench to evaluate state-dependent workflows spanning weeks or months.

Critical strengths
  1. The motivation is impressive and clear. The authors reframe “LLM memory” as “program state,” directly investigating whether LLMs can serve as software.
  2. FlowStateBench targets an important setting. The benchmark evaluates systems on scenarios involving interleaved events across multiple entities, distracting events, and mid-workflow rule changes, making the tasks more complex and realistic.
  3. The authors have conducted extensive experiments to demonstrate the effectiveness of their method.
Critical weaknesses
  1. I think the experimental comparison is not entirely fair. The baselines use GPT-5.5 nano for compilation, while POLO uses GPT-5.4. According to the ablation study in Table 2, the choice of model has a substantial impact on performance, and the results show that GPT-5.4 performs significantly better than GPT-5.5 nano. In addition, tool access is not fully aligned. POLO can use tools to read the states of other objects, while the baseline methods do not have equivalent capabilities. Therefore, the experimental results are not very convincing to me.

  2. The authors evaluate their method exclusively on a self-constructed benchmark, which may favor POLO by design. I would like to see additional comparisons on other public benchmarks to provide more convincing evidence of the method's generalizability and robustness.

Questions for authors
  1. What if POLO and other baselines utilize the same LLM for compilation ( and the same setting )?
  2. How would POLO perform against other baselines on some other public benchmarks? Overall, I would like to see a more fair comparison.
2 comments
AuthorsOpenReview ↗

Response to reviewer dtz1

We thank the reviewer for the positive assessment, and for recognizing the reframing of LLM memory as program state and the design of FlowStateBench. We answer the reviewer's concerns about the fairness of the comparison below; the general comment summarizes the revisions.


1. The same model for compilation (Weakness 1, Question 1)

"What if POLO and other baselines utilize the same LLM for compilation (and the same setting)?"

We tested all three ways of giving both systems the same models:

  • POLO: weak model everywhere. In the ablation studies we ran, POLO compiling with a weak model (gpt-5-nano) dropped the accuracy to 20.0%, below LangGraph with the weak model. The loss comes mainly from compilation itself: in 69% of the failures, the generated program was invalid or syntactically incorrect, and in 30% it contained logic errors. POLO therefore needs a capable model at compile time. This is a real requirement, and we now state it as a limitation. It is also a one-time cost: the strong model runs once per event kind and after changes to the instructions, while every event is handled by the small model.
  • The strong model everywhere. This is an upper reference with ten times the cost per event. On a stratified 20% of the samples, which matches LangGraph’s mean pass rate on the full benchmark, LangGraph reaches 82.1% overall and 78.2% on state-dependent events. Current POLO reaches comparable accuracy (85.5% and 81.2%) at $0.002 per event instead of $0.020.
  • Strong model compilation (as POLO). The strong model writes a stepwise NL plan once per event kind, and LangGraph using gpt-5-nano executes it. This matches the design of POLO as submitted. With the same models and the same type of plan, POLO as submitted reaches 61.2% overall and 54.2% on state-dependent events, and LangGraph reaches 31.5% and 9.6%. So the strong model does not explain the gain.
Systemgpt-5.4 useOverallState-dep.One-time / workflowPer event
LangGraph (nano)None36.3%16.0%$0$0.001
LangGraph (5.4 plan + nano)Once per kind31.5%9.6%$0.92$0.002
POLO as submittedOnce per kind61.2%54.2%$0.82$0.016
LangGraph (5.4 only)Every event82.1%78.2%$0$0.020
POLO (current)Once per kind85.5%81.2%$0.92$0.002

2. Tool access (Weakness 1)

"POLO can use tools to read the states of other objects, while the baseline methods do not have equivalent capabilities."

read_shared_state was left in the code by mistake from an earlier object graph, and we disclosed it (page 9, Appendix F). In the submitted runs, it was used in 6.2% of the event processing, and those executions had a lower pass rate (49.5% vs. 61.2% overall). The revised runtime removes the tool, and all current results, including the 85.5% above, are without it.


3. Evaluation on a public benchmark (Weakness 2, Question 2)

"I would like to see additional comparisons on other public benchmarks ..."

The paper already includes an external benchmark. Section 6 reports results on Memora (Uddin et al., 2026), a public long-term memory benchmark. We extended this evaluation to include baseline comparisons on Memora.

SystemWeeklyMonthlyQuarterly
POLO (current)72.551.226.2
POLO (submitted)56.122.110.0
CrewAI11.13.36.7
LangGraph2.23.30
Letta000
OpenClaw000

POLO leads on every inspection time (weekly, monthly, and quarterly). All baselines use gpt-5-nano with the same task contract and scoring. Letta and OpenClaw rely on the agent to choose what to write using generic memory tools, and in our checks, they missed records, so they couldn't answer those questions correctly.


We hope these results address the reviewer's concerns about fairness, and we would be glad to answer any further questions.

I appreciate the authors' effort. I will maintain my positive score.

Reviewer_ScVvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper attempts to address the issue of how AI agents are systematically measured and fairly compared in verifiable reasoning tasks. The authors propose the AlphaDiana evaluation system, which supports session isolation and controlled reuse, preserves harness-native execution, and links results to execution evidence, thereby supporting macro comparison of complete agents and micro intervention analysis of tools, skills, memory, and other components. Based on this, the experiments also reveal several counterintuitive empirical findings.

Critical strengths
  1. This paper proposes a natural language programming paradigm based on persistent LLM objects, which supports rule changes at runtime without losing state, providing a new implementation path for dynamic workflows.
  2. LLM objects independently derive structured state schemas, retaining only the information required for future decisions, and ablation experiments show a significant improvement in accuracy for long-term state-dependent tasks.
  3. FlowStateBench integrates cumulative counting, decoy events, and runtime rule changes, and verifies historical dependencies for selected events through counterfactual testing, providing a targeted benchmark for long-term state management.
Critical weaknesses
  1. POLO utilizes GPT-5.4 for task compilation, whereas all baselines rely solely on GPT-5-nano; the main experiments thus suffer from a significant confounding factor regarding model capabilities.
  2. In the ablation studies, the single-object version actually achieved a 4.2 percentage point increase in overall accuracy and an 11.4 percentage point increase in accuracy for state-dependent tasks, all while incurring lower costs.
  3. Although the paper emphasizes private state isolation and message-based communication between objects, the actual runtime implementation permits direct access to other objects' states via read_shared_state.
  4. FlowStateBench relies entirely on manually designed and generated synthetic workflows, lacking validation through real-world deployment.
Questions for authors
  1. Could the authors re-run key comparisons using a unified model configuration to clarify the extent to which the performance advantage stems from the POLO architecture itself, rather than differences in model capabilities?
  2. Could the authors further elaborate on the practical value of the multi-object architecture compared to the single-object design?
  3. The runtime allows direct access to the states of other objects via read_shared_state. How frequently was this mechanism invoked during the experiments? Would POLO’s performance be maintained if direct cross-object state reading were disabled?
  4. FlowStateBench comprises 100 workflows, yet the main experiments evaluated only about 40, and OpenClaw was tested on merely 10. Could the authors explain the rationale behind this selection? Would the performance gaps between the methods persist if the comparison were repeated using an identical subset of workflows?
1 comment
AuthorsOpenReview ↗

Response to reviewer ScVv

We thank the reviewer for the detailed review and for recognizing the NL programming paradigm, the self-authored state, and the long-horizon design of FlowStateBench. The general comment summarizes the revisions.


1. Does the stronger model explain the gain? (Weakness 1, Question 1)

"POLO utilizes GPT-5.4 for task compilation, whereas all baselines rely solely on GPT-5-nano ..."

POLO and the baselines handle events with gpt-5-nano. POLO also uses gpt-5.4 to compile the object's instructions into a program, once per event kind. In the revised runtime, the compiled program is code, and gpt-5-nano only extracts the fields from each NL message. On FlowStateBench, gpt-5.4 runs on 8.6% of the events: 65% of these calls occur in the first 10 events, and a further 23% follow instruction changes. We tested three setups:

  1. gpt-5-nano everywhere, including POLO's compiler. Accuracy drops to 20.0%. In 69% of the failures, the compiled program was invalid, and in 30% it had logic errors. POLO needs a capable compiler, and we now state this as a limitation.
  2. The same strong-model budget. gpt-5.4 writes a stepwise NL plan once per event kind, and gpt-5-nano executes it, as in POLO as submitted. With the same models and plan type, POLO reaches 61.2% (54.2% state-dependent) and LangGraph 31.5% (9.6%).
  3. gpt-5.4 on every event. On a stratified 20% of the samples, LangGraph reaches 82.1% (78.2%). Current POLO reaches comparable accuracy, 85.5% (81.2%), at \0.002pereventinsteadof0.002 per event instead of \\0.020.
Systemgpt-5.4 useOverallState-dep.One-time / workflowPer event
LangGraph (nano)None36.3%16.0%$0$0.001
LangGraph (5.4 plan + nano)Once per kind31.5%9.6%$0.92$0.002
POLO as submittedOnce per kind61.2%54.2%$0.82$0.016
LangGraph (5.4 only)Every event82.1%78.2%$0$0.020
POLO (current)Once per kind85.5%81.2%$0.92$0.002

2. Multi-object versus single-object design (Weakness 2, Question 2)

We agree that on FlowStateBench a single object scores at least as well, at a lower cost (Section 6.2). Each workflow is small, so one object suffices, and a single LLM-object is a valid program. Multiple objects add encapsulation: each has its own instructions and private state, reached only through messages. This helps in three cases. First, with similar but separate concepts, such as two budgets: in one object "the budget" is ambiguous, while separate objects give each concept one meaning and its own state schema. Second, when rules change at different rates: a change recompiles and migrates only the affected object. Third, when the instructions grow, separate objects keep each definition small. FlowStateBench does not test this case, and we list it as future work.


3. Cross-object state reads (Weakness 3, Question 3)

read_shared_state was left in the code by mistake from an earlier object graph, and we disclosed it (page 9, Appendix F). In the submitted runs, it was used in 6.2% of the event processing, and those executions had a lower pass rate (49.5% vs. 61.2% overall). The revised runtime removes the tool, and all current results, including the 85.5% above, are without it.


4. Synthetic workflows and benchmark validation (Weakness 4)

The core of each workflow is written by hand. Each is based on a public Zapier template, and the authors write its process, state-dependent rules, thresholds, time windows, instruction changes, expected tool calls, and object graph. Only the event stream is generated. Opus 5 expands each workflow into a full specification with tests. An LLM-written program then generates events one at a time and tracks state, so each outcome is known before it writes its text. An LLM writes each event in NL, and an independent solver that reads only the NL events must reproduce every outcome. We agree that FlowStateBench, like LoCoMo and LongMemEval, is not validated by a real deployment, and we state this limitation.


5. Evaluated subsets (Question 4)

The original full run recorded cached tokens incorrectly, so we re-ran POLO and OpenClaw on a stratified subset of 40 workflows. The count of 10 for OpenClaw in the paper is an editing error: OpenClaw also ran on these 40 workflows. The other baselines ran on the full benchmark. We will report all systems on the full benchmark in the revision.

We would be glad to answer any further questions.

Reviewer_fn6wOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores the possibility of a setup where software consists of a collection of objects with persistent states that are authored and maintained by LLMs. It shows that maintaining these states performs better than baselines where memory is stored in the context or memory blocks.

Critical strengths

This work applied in a small and limited scale could improve the recall quality of records for LLMs for long horizon tasks.

Critical weaknesses
  • The motivation of using an LLM itself as the object is unclear. The LLM object is orders of magnitude larger in cost, and almost all the benefits can be replicated using LLM code generation.
    • Specifically, the ability to alter code behaviour using natural language can already be done by rewriting code and migrating database schemas.
    • The cost of "code execution" (through LLM objects) is ongoing, larger than regular LLM function calls, and much much larger than code (whether human written or generated).
    • Code verification is impossible by design. The definitions, state, and messages "may vary" in outputs, and the "LLM checking" is shaky at best. There have also been examples where LLMs fail in basic arithmetics (e.g. comparing 9.9 and 9.11), whereas this is almost never the case with regular code.
  • The definition of "kind" has not been defined as part of the spec for the compiler.
    • Specifically, the router tries to reuse existing plans and create new ones if non fits. There is little control over granularity of the types, and specifically, there is no way to split/specialize when the behaviours diverge.
    • This also makes the issue of handling invalid inputs difficult, as there is no way to define which inputs are valid for what kind of message a priori.
Questions for authors
  • How does this model handle concurrent requests? What if a transaction has multiple parts that requires multiple updates? How is consistency maintained?
1 comment
AuthorsOpenReview ↗

Response to reviewer fn6w

We thank the reviewer for the detailed review. Since submission, we have revised the runtime to address concerns about cost and reliability.

R1. Why an LLM-object and not LLM-generated code

"... almost all the benefits can be replicated using LLM code generation ..."

LLM-objects offer three things that LLM-generated code does not:

  • Unexpected inputs. Generated code handles only the input kinds it was generated for. Other inputs can crash it or cause a silent bug. For example, changing an existing dress order from stripes to polka dots is neither a new order nor a cancellation. When an input matches no known kind, the runtime compiles a new program for it.
  • State migration. A database schema migration can affect every component that reads the changed tables. An LLM-object keeps its state private, so a change to one object does not affect the others.
  • Live change without redeployment. Platforms can replace stateless code without downtime, but a change that also migrates stored data is engineering work beyond code generation. In POLO, a change is a message to the running object. The object migrates its state and regenerates the affected programs. Messages that arrive meanwhile wait in its mailbox, and the other objects keep running.

R2. Cost per event

"The cost of 'code execution' ... is ongoing ..."

We agree that generated code is cheaper per event. In the submitted version, an LLM executed every plan step. Now, gpt-5.4 compiles the plan into code once per event kind, gpt-5-nano only extracts the fields from each NL message, and the runtime executes the program. The cost per event drops from \0.016to0.016 to \\0.002, plus about \0.92perworkflowforcompilation.Thecompilerrunson8.60.92 per workflow for compilation. The compiler runs on 8.6% of the events: 65% of these calls occur within the first 10 events, and a further 23% follow instruction changes. For comparison, LangGraph with gpt-5-nano costs \\0.001 per event at 36.3% accuracy. POLO reaches 85.5% at \$0.002.

R3. Code verification

"Code verification is impossible by design ... (e.g. comparing 9.9 and 9.11)"

POLO gives no formal guarantee, but the revised runtime limits where the LLM can vary. All arithmetic and comparisons run in code, so 9.9 vs. 9.11 is a Python comparison. The strong model compiles one program per kind, a verifier checks it, and the runtime reuses it for every input of that kind. Only the field extraction varies between events. The programs are ordinary code that can be read and logged. We agree that the verifier is an LLM check and can make mistakes.

R4. Kinds, granularity, and invalid inputs

"The definition of 'kind' has not been defined ... no way to split/specialize ..."

We add a precise definition to the revision. An event kind is like a method signature with its docstring: it states what the method does and which messages it accepts. The router matches a kind for each message. If none is found, the compiler generates a new method. Generated code cannot do this, since its methods are fixed at generation time.

Behaviors diverge only when the rules treat them differently. The router then opens a kind for each case, or the compiler writes a branch inside one method. When the instructions change, the affected kinds are recompiled, and one kind can become several (e.g., "purchase" splits once red and blue items need different handling). This assumes correct routing and detecting routing errors is future work.

POLO has no input schema, but a schema checks form, not meaning: a non-negative amount rule also rejects a refund of -$50 unless refunds were foreseen. An LLM-object reads the input against its rules and can tell a refund from an error, at the risk of accepting unusual inputs that a schema would reject.

Q1. Concurrency and consistency POLO follows the actor model: each object handles one message at a time, so no two handlings change the same state, and the code needs no locks. A request that updates several objects becomes several messages, and we trust the LLM to generate code that coordinates them correctly, as does any LLM-generated software. The runtime does not guarantee atomicity across objects --- we state this as a limitation.

We hope these answers address the reviewer's concerns and would be glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.