acceptodds
Under review as a conference paper at ICLR 2027

From External Orchestration to Intrinsic Continuous Agency

Abstract

Language-model agents increasingly preserve memory, stream perception, and schedule repeated inference, producing behavior that appears persistent even when the underlying model remains episodic. We use cognitive continuity to denote the stronger capability in which an agent's internal cognitive state itself keeps evolving through time, including without meaningful new external input, and this evolution can change what the agent remembers, infers, intends, and ultimately whether, when, and how it acts. Existing systems realize important parts of this capability through external orchestration and continuous autoregressive processing. We argue that cognitive continuity should also be pursued as an intrinsic learned model capability rather than primarily reconstructed around episodic inference or an advancing interaction history. Our central hypothesis is not one of expressive impossibility: strong harnesses and autoregressive models may reproduce the same surface behavior. Architectural value need not require an expressivity separation: architectures can realize overlapping functions while differing substantially in the inductive biases, coordination pathways, and optimization problems they expose to learning. Rather, the locus of continuity determines which representations, coordination pathways, and state transitions are jointly learned versus externally prescribed. As one concrete instantiation, we outline a Global Workspace Theory-inspired architecture organized around a persistent shared latent state whose recurrent dynamics evolve independently of discrete interaction turns. We conclude with comparative tests and a benchmark hierarchy separating floor awareness, reasons to manifest, and genuinely endogenous continuation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI, AGI, Position paper, Global Workspace Theory, Continuous agency, Autonomous agent, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YZcyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that cognitive continuity should be developed as an intrinsic capability of language-model agents rather than primarily implemented through external orchestration. The authors categorize existing approaches into external orchestration, continuous autoregressive processing, and persistent recurrent state. They further propose a shared recurrent latent workspace that continuously updates the agent's internal state, together with memory and output mechanisms. The paper is presented as a position paper and does not report new empirical results.

Critical strengths

Reasonable motivation. Enabling agents to maintain and update internal states across interaction boundaries is a relevant research direction. The discussion of different approaches to cognitive continuity provides some useful conceptual organization.

Critical weaknesses
  1. Limited novelty. The proposed architecture largely combines established ideas, including recurrent hidden-state computation, shared latent representations, persistent memory, and output gating. Although the paper emphasizes extending recurrence beyond individual inference episodes, it does not clearly establish a fundamentally new mechanism or sufficiently distinctive research insight compared with existing approaches.

  2. Insufficient technical substantiation. The paper provides neither an implemented and trained system nor theoretical evidence supporting its central architectural hypothesis. Important technical decisions, including the backbone architecture, training objectives, and learning procedure, remain unspecified. Consequently, the proposed architecture remains largely conceptual, and its feasibility and potential advantages are not demonstrated.

  3. Unconvincing justification for intrinsic continuity. External orchestration and recurrent internal computation operate at different levels of abstraction and can coexist within the same agent system. While the authors acknowledge this possibility, the paper does not convincingly explain why moving continuity into the model should provide meaningful advantages over strong existing orchestration-based approaches. The claimed benefits in coordination, generalization, and long-term adaptation remain speculative.

Questions for authors
  1. What is the key technical or conceptual contribution that distinguishes the proposed persistent recurrent workspace from existing recurrent architectures and memory-augmented agent systems?

  2. What concrete evidence or argument supports the hypothesis that intrinsic recurrent continuity offers advantages over learned external orchestration, particularly under comparable computational budgets and model capabilities?

5 comments

Thank you for the review. We respond to each point below.

W1/Q1 — What is novel beyond combining existing modules? The paper makes two contributions.

The first is defining and organizing a new class of systems. Always-on agents have quickly become a central direction of the field: xAI's Grok Bot, Meta's Muse, and OpenAI's Dots all launched this year, and OpenClaw became GitHub's most-starred software project. To our knowledge, no prior work defines the property these systems share or classifies them by where their continuity lives. The paper does both (Secs. 2–3, Table 5).

The second is the architecture. We do not claim to invent recurrence, shared latent states, memory, or a speak/silent gate. What is new is the role the shared state plays: a fixed-size state, rewritten at every step, that remains the agent's current cognitive situation after it acts, including during silence. Existing work uses recurrent state differently. Recurrent sequence models (LSTMs, state-space models) and recurrent reasoners (TRM, HRM-Text) carry state through one sequence or problem, and it ends there. Memory-augmented agents keep memories that survive between calls, but stored memory does not keep thinking. In our proposal, the gate that decides when to speak reads the live state, the decoder keeps reading it while speaking, and retrieved memories are written back into it, so one memory can bring up another.

W2 — Technical substantiation and position-paper scope. We agree that feasibility and superiority are not yet demonstrated. The paper presents them as hypotheses and states the results that would refute each one (Introduction, "Three claims"). ICLR has accepted position papers on this basis (e.g., Ngo et al., ICLR 2024, https://openreview.net/forum?id=fh8EYKFKns). Still, we have added the details you asked for. The new Appendix B.5 gives a reference implementation: a pretrained encoder–decoder in which the encoder updates the state at every step and the decoder reads only from it; a speaking gate trained to match reference timings as distributions; training through time over long trajectories; and a curriculum that adds one capability at a time. The new Appendix C.7 specifies how to test it against strong baselines.

W3/Q2 — Why move the carrier inside the model, and how is coexistence handled? We agree that orchestration and recurrence can coexist; our own system uses a runtime clock, an autoregressive decoder, and an external memory. The taxonomy does not classify which modules a system contains, but which state carries the agent's ongoing situation from one step to the next. In OpenClaw, that is the session state kept by the runtime, so it is external orchestration. In our architecture, the runtime only advances the clock and the situation lives in Z, so it is persistent recurrent state; the decoder and memory store are components. The revised Appendix G.4 turns this into a simple two-question test and works through borderline cases.

We do not claim that external systems cannot achieve continuity. Our hypothesis is that learning it inside the model works better in three situations: when what must persist is hard to write into a record (half-finished deliberation, how long a silence has lasted); when capabilities interact in orders not seen in training (further thought makes a memory relevant, retrieving it changes the conclusion, and a correction becomes warranted); and when perception, thinking, and action overlap. A learned controller improves routing, but each call still starts from the record, and what is written there is sampled text, so its long-term effects are learned only from sparse reward rather than from gradients through time. If the controller is instead given a latent state rich enough to hold the agent's situation, continuity has moved into a learned state, which is our proposal (Appendix G.4). A fixed-size state also keeps the cost of each step constant, however long the agent runs (Sec. 4.2).

The new Appendix C.7 tests this directly. A learned orchestrator, a bounded autoregressive model, shared Z, and partitioned Z share the backbone, training data, memory, and compute, and are tested on chains of reasoning, memory, perception, and timing in orders never seen in training. Baselines are also given 2× and 4× post-answer compute, so an advantage that disappears there is efficiency, not capability. If the orchestrator or the autoregressive model matches shared Z, the corresponding hypothesis weakens.

I still have concerns about the overall research contribution. Being a position paper does not mean that one can simply combine existing engineering components, introduce a new taxonomy, and outline hypothetical future experiments. A convincing position paper should articulate a clear scientific question and provide meaningful conceptual insights or arguments to support its position. The additional details in the rebuttal remain largely speculative, focusing on architectural descriptions and proposed experimental plans rather than concrete technical developments or evidence. Overall, this work reads more like an engineering assembly accompanied by an ambitious architectural blueprint than a well-formulated scientific research contribution. I therefore maintain my original rating.

Thank you for taking the time to read our revised manuscript. While we acknowledge that the paper does not provide experimental validation, we would like to emphasize that it does articulate a clear scientific question: whether the locus of cognitive continuity affects an agent's capabilities. Section 4 develops several conceptual arguments supporting our position. If the reviewer could clarify why these arguments remain unconvincing or insufficient as a scientific contribution, we would greatly appreciate the feedback.

Reviewer_ri21OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This position paper advocates cognitive continuity as an intrinsic capability of language-model agents, enabling internal cognitive states to evolve even without new external input. The authors distinguish three architectural approaches: external orchestration, continuous autoregressive processing, and persistent recurrent state. They further propose a shared latent workspace architecture and outline an evaluation framework for studying continuous cognition.

Critical strengths

S1. Meaningful research direction. The paper highlights an important distinction between behavioral persistence and continuous cognition, offering a relevant perspective on the development of long-running autonomous agents.

S2. Comprehensive literature synthesis. The paper systematically organizes existing approaches and acknowledges that different architectures can achieve similar behaviors, making its discussion relatively balanced.

S3. Concrete research agenda. The proposed recurrent workspace and evaluation framework provide actionable directions for investigating cognitive continuity, including causal interventions and comparisons across architectures.

Critical weaknesses

W1. Limited conceptual novelty. The distinction between cognitive continuity and existing paradigms such as continuous reasoning, self-reflection, and proactive agents remains insufficiently clear. The paper would benefit from identifying capabilities that are not adequately captured by these existing paradigms.

W2. Insufficient justification for intrinsic continuity. While the authors argue that intrinsic continuity may improve coordination and generalization, the advantages over sophisticated external orchestration remain largely speculative. More concrete arguments or representative scenarios are needed to establish why intrinsic continuity should be prioritized.

W3. Potential limitations of the evaluation framework. The proposed endogenous-continuation evaluation may conflate cognitive continuity with additional inference-time computation. It remains unclear whether successful delayed self-correction demonstrates persistent cognition rather than conventional iterative reasoning.

Questions for authors

Q1. What fundamentally distinguishes cognitive continuity from existing continuous reasoning or self-reflection mechanisms? Could the authors provide a concrete example illustrating this distinction?

Q2. Under what specific conditions would intrinsic continuity offer advantages over external orchestration with learned controllers and persistent memory?

Q3. How can the proposed evaluation distinguish genuine cognitive continuity from improvements resulting simply from additional test-time computation?

2 comments

Thank you for the review. We revised the paper to make these points concrete.

W1/Q1 — What is distinct from reflection, continuous reasoning, and proactivity? These are not alternatives to cognitive continuity. They are mechanisms, and a system that uses them may or may not have continuity. Our criterion is simple: after the agent has acted, with nothing new arriving, can its own evolving state give it a new reason to act, or to stay silent? Reflection or latent reasoning before the answer does not meet it; the same processes continuing on their own after the answer can. Reacting to a new visual event is proactive, but the trigger is external. And an always-on orchestrated agent can meet the criterion, since continuity is not tied to any one architecture.

A concrete example (new Figure 1a): a user says their flight is at six and asks if there is still time for coffee. The agent says yes, about 40 minutes. A minute later, with no new input, it says: "Wait, boarding closes at 5:40, not 6:00. I'd skip the coffee." Being reflective, a latent reasoner, or proactive does not guarantee this behavior; the criterion is what tests for it. Other capabilities these paradigms do not cover include reading a user's silence differently as it grows longer (Figure 1b) and deciding that staying silent is right; Sec. 6.2 turns them into task families. The revised Sec. 2.2 ("Cross-cutting properties", formerly Sec. 2.4) states these distinctions directly, and the system table (Table 3, now Table 5) adds recent always-on agents such as OpenAI's Dots.

W2/Q2 — Why should intrinsic continuity help over learned orchestration? This is a hypothesis, and we present it as one. A learned controller decides which call runs next and what to retrieve, but it does not change what carries the agent's situation between calls: a record that each call reads and writes. Persistent memory helps the agent remember facts, but not unfinished thinking. We expect an orchestrator with learned controllers and persistent memory to fall behind under three conditions:

  1. What must persist is hard to write down: half-finished deliberation, how long a silence has lasted, a subtle change in someone's expression. Whatever the record does not hold is lost when a call ends, however well calls are routed.
  2. The useful chain of steps is new. In the example of revised Sec. 4.1, further thought makes an old memory about the user's medication relevant, retrieving it changes the conclusion, and the agent corrects itself. A learned controller generalizes from routes seen in training; we expect a shared learned state to combine familiar steps in unfamiliar orders more easily. Appendix C.7 tests exactly this.
  3. Perception, thinking, and action overlap. A call in progress does not see new input, so revising an utterance midway needs extra synchronization.

Credit assignment adds to all three. What gets written into the record is sampled text, so its effect many steps later is learned only from sparse reward, while a learned state receives gradients through time.

Finally, one could close these gaps by giving the controller a latent state rich enough to hold the agent's ongoing situation. But then continuity has moved into a learned state, which is the approach we advocate (Appendix G.4 works through this case).

W3/Q3 — Continuity versus merely more test-time computation. The benchmark is not a reasoning-accuracy test, where extra compute would be an unfair advantage. It measures how well an agent behaves through time. The task families in Sec. 6.2 cover timing from conversation history, following timing instructions, interpreting another person's silence, temporal personality, speaking up because of something seen, self-correction with no new input, keeping what matters over minutes, and revising an utterance midway. Most cannot be solved by thinking harder: knowing what a 12-second pause means, or how long a timid persona should stay quiet, is not a reasoning problem.

These tasks compare systems that have continuity with each other. An ordinary model answers and stops, so it cannot attempt them. In the same way, OpenAI does not benchmark Dots against the model it runs on; the natural comparison is with other always-on agents such as Meta's Muse or xAI's Grok Bot. Within that class, computation after the agent acts is part of the capability, not a confound, and ordinary iterative reasoning that keeps going after an answer counts as continuity. What the benchmark excludes is behavior that only looks continuous: replies released by a timer or computed in advance. It caps compute before the first answer, pairs each case needing a correction with one needing silence, and includes timer and cached-reply baselines (Appendix C.6). When comparing architectures, compute is matched (Appendix C.7).

Area_Chair_1ZMbOpenReview ↗

Please follow up before the discussion period ends & AI use clarification

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Additionally, your review was flagged for AI use while the corresponding field in your review does not mention how it was used. Please clarify this for me.

Thank you for your contribution!

Best,

Your AC

Reviewer_26ENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This position paper argues that cognitive continuity should be studied as a capability in its own right: an agent's internal state should be able to keep changing after an interaction ends, even when no meaningful new input arrives. The authors distinguish three possible ways to support this—external orchestration, continuous autoregressive processing, and persistent recurrent state—and make a case for exploring a shared recurrent latent workspace. They also outline evaluation tasks and causal tests intended to separate ongoing cognition from delayed or precomputed responses. The paper presents a research agenda rather than a trained system or new experimental results.

Critical strengths

S1. I appreciate the distinction between an agent that simply produces another response after a pause and one whose intervening computation actually matters. The discussion of precomputed follow-ups in Appendix G, together with the proposed controls, makes this distinction more concrete. It is also important that the evaluation allows silence to be the correct outcome; otherwise, a system could appear more "continuous" simply by speaking more often.

S2. The paper does not claim that orchestration or autoregression is incapable of producing the same behavior. Section 4.3 and Appendix F separate the general case for studying cognitive continuity from the more specific hypotheses about intrinsic continuity and a recurrent workspace. I found this useful, particularly because the authors discuss what would count against their preferred design, including state drift, interference, and computational cost.

Critical weaknesses

W1. I am not fully convinced that the three-way taxonomy can be applied consistently to strong competing systems. For example, a learned orchestrator might maintain latent state and learn when to call different components, while an autoregressive agent might use learned compression and persistent memory. Appendix G.4 proposes carrier-based tests, and Equation 22 deliberately assigns cases with overlapping carrier predicates to UNKNOWN. That is a reasonable safeguard, but the paper does not work through a difficult borderline example to show how the boundary or state decomposition should be refined. As a result, I am unsure when improving a baseline would genuinely test the proposed inductive bias and when it would simply cause the baseline to be reclassified.

W2. The evaluation is clearer about how to detect useful post-response computation than about how to test the proposed advantage of a shared recurrent state. The deployment example in Appendix C.5 is a helpful minimal test: further processing may uncover a constraint and justify correcting an earlier answer. But the main architectural argument in Section 4.2 concerns less predictable interactions among reasoning, memory, perception, and action, especially over long periods. Appendix C identifies these as evaluation dimensions, but it does not give an equally concrete task distribution or held-out setting that would isolate this claimed benefit. Section 6.1 also leaves the training setup open. I would not expect a position paper to train a new system, but one worked comparison would make the stronger architectural hypothesis easier to assess.

Questions for authors

Q1. Could the authors walk through two borderline cases using the tests in Appendix G.4: a learned orchestrator with persistent latent state, and an autoregressive agent with learned compression? Which state would count as the continuity carrier in each case, and what change would move the system into a different category?

Q2. What would be a concrete experiment in which a shared recurrent workspace is expected to outperform a strong learned orchestrator or bounded autoregressive baseline, rather than simply benefiting from more inference-time computation? It would help to specify a task distribution, a held-out combination of cognitive functions, comparable training and compute budgets, and an outcome that would weaken the hypothesis.

Q3. Table 3 lists the deployed Sesame Maya/Miles system as passing the definition, whereas Appendix E.2 describes it as a behavioral positive with an unpublished control mechanism. Given the discussion of precomputation in Appendix G.3, is the "Yes" label meant to indicate behavioral compatibility with the definition, or evidence that the relevant internal computation actually continues? Separating those two claims might make the table more precise.

2 comments

Thank you for the constructive review.

W1/Q1 — What determines the locus, including for stronger baselines? A system can mix autoregression, recurrence, memory, and tools. Its label depends only on which state carries the agent's ongoing situation from one step to the next, not on which modules it contains. Our own architecture is REC even though it has an autoregressive decoder. The revised Sec. 3 states this rule up front and follows one example through all three routes.

The revised G.4 turns the rule into two questions. First, between steps, is the model restarted from a record that an outside program (hand-written or learned) keeps and feeds into each call? If so, the system is EXT. If not, does each step append to the model's state (AR) or rewrite a fixed-size state (REC)? Each question has one answer, so only one label can apply. The UNKNOWN branch you mention (Eq. 22, now Eq. 21) is therefore no longer about overlapping cases; it only covers closed systems, for which we report an inferred locus.

G.4 also works through your two cases:

  • Learned orchestrator. A controller with latent state h_t routes calls over a workspace S_t kept by the runtime. If keeping S_t and resetting h_t lets the agent resume, S_t is the carrier and the system is EXT, even though the controller is learned. It becomes REC once the situation lives only in a fixed-size h_t that is rewritten at every step.
  • AR agent with learned compression. If keeping the stream alone lets the agent resume, it is AR, and the compression and external memory are just components. It becomes REC once each step is computed from a fixed-size state that is rewritten rather than appended to.

This also answers when improving a baseline is a fair test. An improvement that keeps the baseline's carrier (better retrieval, compression, or learned control) gives a stronger baseline. One that moves the carrier into a learned fixed-size state is the treatment itself.

W2/Q2 — A discriminating architectural experiment. The revised Sec. 6.2 and Table 3 turn the evaluation dimensions you mention into eight task families, and most of them target a specific place where Sec. 4.2 predicts a recurrent state should do better:

  • whether elapsed time is part of the state: reading the same "No." after 0.4 s, after 12 s, or not at all (F3);
  • whether silence is used for thinking: correcting an earlier answer with no new input (F6);
  • whether retention is learned: recalling salient rather than ordinary details, including visual ones, minutes after they disappear (F7);
  • whether thinking continues while acting: an event or a new conclusion changes an utterance midway (F8);
  • whether perception, memory, and timing interact: something seen delays, triggers, or suppresses speech (F5).

Each family pairs similar situations that call for different later behavior, so a timer or cached reply cannot pass, and silence counts as correct when nothing warrants action.

The new Appendix C.7 uses these families to compare architectures fairly. It builds four versions of the agent: a learned orchestrator with its own latent state, a bounded AR model that keeps thinking silently and compresses old context, shared Z, and a partitioned Z split into fixed parts. All four get the same base model, training data, memory, and compute. They are trained on some combinations of skills, for example a memory brought up by something the agent saw, and tested on combinations they never saw, such as the same memory brought up by a conclusion the agent reached, and on longer chains and silences than in training. Appendix B.5 now also gives a reference training setup.

The hypothesis weakens if the orchestrator matches shared Z on unseen combinations (against intrinsic continuity), if the AR model does (against the recurrent state), or if partitioned Z does (against a single shared state). If the advantage disappears when the baselines get two or four times more compute after answering, it is efficiency, not capability.

Q3 — What does Sesame's "Yes" mean? It means the behavior meets the definition, not that we know the mechanism. Sesame's deployed assistant speaks again after the user falls silent, and does so adaptively: when and what it says depends on context, it follows requests about timing, and it can stay silent. A fixed timer cannot do that, and the definition now states this adaptivity requirement (Sec. 2.1). Behavior cannot show where that continuity lives, and Sesame's internals are unpublished, so we mark its locus (AR) as inferred. As you suggest, the system table (Table 3, now Table 5) separates these claims into three columns: whether a system exhibits continuity, what that judgment rests on, and its locus.

The table now also includes recent always-on agents (OpenAI's Dots, Meta's Muse, xAI's Grok Bot). They show how quickly such systems are spreading, which makes a shared definition and classification timely.

Area_Chair_1ZMbOpenReview ↗

Please follow up before the discussion period ends

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Thank you for your contribution!

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.