acceptodds
Under review as a conference paper at ICLR 2027

General Agents Must Learn Online

Abstract

It has been widely anticipated that scaling offline training could enable artificial general intelligence (AGI) in agents with ***frozen policies***. However, such agents rely on offline knowledge, restricting their capability in big worlds that exceed their capacity. We formalize this limitation by showing frozen agents require ***a model size*** that grows with environment complexity and the accuracy demanded across rewards. Moreover, frozen agents require substantial amounts of ***training data*** to acquire this capability offline. Finally, we establish an achievable tradeoff between frozen information and ***online experience***, showing that general agents are attainable with limited persistent capacity. These results motivate architectures that integrate offline training with online experience as an alternative route toward AGI.

Primary area:
reinforcement learning
Keywords:
general agent, reinforcement learning, online learning, information theory, big world hypothesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KAu6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that agents' behavior at deployment needs to be conditioned on the history of interaction, in order not to cause the number of parameters that the agent needs to become impractical. The argument is roughly that if the policies can't adapt, then they need to know the whole world model before deployment. The paper shows that in order to figure out the world model describing a given environement a fixed policy requires a lot of parameters. Then they show that in their theoretical framework, it is possible to trade-off the parameters for ingesting experience at inference.

Critical strengths

I believe that in reinforcement learning community many people still train agents that are not adaptable. This paper provides a strong signal for the community to move into the direction where adaptability is more important than just offline performance.

Critical weaknesses

I believe that the actual finding of the paper is interesting. However the introduction diverges from the true story, through using language models as an example of models that aren’t adaptive. Which they clearly are. For instance, according to definition 1, LLMs are not frozen as they are autoregressive. That makes Figure 1 confusing and the whole introduction to be as though it's from a different paper than the rest.

There is no real experimental section in this paper. I know that training RL agents in the offline to online regime tends to not be that easy, but an experimental result of this type, or alternatively one comparing an agent with and without history would make the paper much more complete.

Questions for authors
  • In practice, how do we adapt online without knowing the environment before the adaptation phase?
  • Could you spell out the connection to language models implied in the introduction?
  • How would you validate your theory experimentally?
  • Could the authors present even a very toy experiment supporting the theoretical findings?
  • If the real world is governed by patterns, that is, I expect that in each house, the fridge is somewhere close to the stove, do the theoretical results describe the real world accurately?
Reviewer_ifgBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether general agents under the Big World Hypothesis can rely on frozen, pre-deployment knowledge. It theoretically proves that the persistent capacity and offline data required by frozen agents must scale with increasing world complexity; in contrast, adaptive agents can effectively substitute persistent capacity with online interactions. Thus, the study contributes information-theoretic lower bounds and an existence-style "capacity–experience tradeoff", illustrated via small-scale tabular experiments.

Critical strengths

1.The paper elevates the qualitative “large-world hypothesis” into a quantitative framework, proving that the persistent capacity and offline data requirements of frozen generalist agents must scale with increasing world complexity.

2 Theorem 3 provides an existence proof showing that adaptive agents can use online interactions as a substitute for persistent memory. It demonstrates that continual online learning, rather than simply scaling up pretraining, is an essential and scalable path toward general intelligence in complex environments.

Critical weaknesses
  1. The zero-cost state-reset assumption overlooks the high stakes and expense of trial-and-error in real-world sequential interactions, underestimating the exploration challenge and the required number of online interactions K.
  2. The stationary environment assumption overlooks how the shifting policies of other agents or humans continually evolve the environment, potentially invalidating the theoretical bounds and rendering capacity–experience trade-offs unreliable under environmental drift.
Questions for authors

We highly appreciate the novel information-theoretic framing of the Big World Hypothesis. However, to bridge the gap between this elegant theory and real-world deep RL, we would like to explore some practical challenges:

Q1. The information-theoretic lower bounds (Theorem 1 & 2) scale with $∣S∣^2∣A∣ $ in tabular MDPs, which explodes in high-dimensional continuous state spaces. How do you think the representation learning and generalization capabilities of neural networks could be incorporated into your framework to reduce the practical "persistent capacity" wall in reality?

Q2.Theorem 3 assumes generative access, which eliminates the exploration cost. In reality, agents face sequential interactions and high risks of trial-and-error. How would relaxing this assumption to a standard sequential MDP without reset capabilities affect the capacity-experience tradeoff and the required K?.

Reviewer_4Mb2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission investigates the information-theoretic requirements of general agents in finite Markov decision processes. The authors establish a capacity lower bound for frozen reward-conditioned agents, and compare them to adaptive agents. They highlight a trade-off between an agent retaining information and its online experience. They also provide some numerical simulations to support the main theorems.

Critical strengths
  • The central question is well motivated, and Figure 1 provides a nice narrative for why scaling offline training alone may be insufficient, relative to learning online.
  • Separating the costs of acquiring and retaining environmental knowledge is useful, and the trade-off between capacity-experience provides a nice perspective on the role of online learning.
Critical weaknesses
  • The connection between the formal setting and the broader claims could be clearer. A general agent is one that is near optimal across all bounded rewards. Moreover, the problem formulation is a fixed, fully observable mdp. It is not obvious that this faithfully captures the big world setting. For instance, the numerical experiments presented are interesting, but ultimately don't appear to be the kinds of problems that we give our general agents, and which we expect would need to be solved online.
  • The contributions on both the technical and empirical side could be developed further. Theorem 1 and Figure 3 are helpful to illustrate the bounds, but would be better if the theorems could be connected to existing empirical settings so that they could provide insight to a practitioner. I suspect that the assumption of a fixed transition dynamic may be a blocker here, because nonstationary environments that are commonly used to evaluate continual reinforcement learning algorithms. Theorem 3's existence result is accompanied by Figure 4, but the explanation of their relationship is not clear even after checking their description in the appendix.
Questions for authors
  • Why should a general agent need to perform well for all bounded rewards? Can the lower bounds be applied to a more restricted set of rewards?
  • Under Definition 1 and Remark 1, what would an in-context learner with frozen weights be classified as, or how about a recurrent network with fixed weights? It would be interesting to know if these could count as general agents that learn online in your definition.
  • Could the trade-off or any of the quantities be measured in more representative environments? I would be particularly interested in how the analysis applies to nonstationary environments typical of continual reinforcemnt learning.
  • minor but: shouldn't the optimal value function be subscripted by r? Also: What is the protocol followed by an agent when it is reward-conditioned? How do you represent a reward if it is not encoded by a goal-state?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.