- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SCHEMA, an agentic framework that aims to explore, understand, and solve tasks in environments with unknown transitions and goals. It achieves this through a four-stage pipeline. First, it uses a Python program to ground the state into structural representations and hypothesize the environment transitions. Next, it checks whether the hypothesized transition functions match the recorded historical transitions and revises them if not. Then, it asks the agent to come up with a target to either explore the environment or solve the task, and use planning in this Python-form dynamic model to achieve it. Finally, it executes the plan in the real environment, where the agent will stop to replan whenever the environment transition conflicts with the Python one. It achieves superior performance on ARC-AGI-3, DiG-bench, and MazeBench, where the ablations show the necessity of each component in the pipeline.
Critical strengths
- The motivation for using Python programs as abstractions to hypothesize the unknown environments is well-motivated and discusses an interesting direction in Agentic problem solving.
- The paper is generally well-written and easy to follow.
- The experimental results demonstrate superior performance over baselines, especially on ARC-AGI-3, showcasing the usefulness of the proposed SCHEMA.
Critical weaknesses
- Some details of the proposed method are missing:
- The prompts or prompt design are missing. Since the paper aims to solve tasks in an unknown environment, it's important to ensure the prompts do not contain any human expert knowledge about the environment.
- The criteria for whether the transition function is ready to be planned are not provided. Since the authors report that the certification rate may not always be 100%, how does it ensure the certification process checks all the most critical transitions in the environment? (E.g., if the criteria is over 99%, and some transitions are rare in the history, ignoring them may still pass the certification stage)
- Details of the baselines are missing, e.g., what is the basic harness in ARC-AGI-3? Do they have a memory system? What's the difference between SCHEMA and these baselines? Including these details can make the paper more promising and self-contained.
- The paper misses a limitations section, where the current evaluation protocol focuses on text-based, discrete-action, deterministic environments. Including it would better help readers understand the scope of the current method.
Questions for authors
- What are the details of the prose model variant in Table 1? Does it still do some kind of planning, but in text form?
The following questions are out of the reviewer's curiosity and will not affect the score.
- What if the environment provides only a dense reward? E.g., the reward is given based on the distance to a possibly unknown target; how does the LLM encode it into a transition function without knowing the true nature of the reward?
- The current evaluation protocol follows an online setting; that is, it 'trains' the program and tests it in the same environment. It will be interesting to evaluate the program in other environments with the same transitions in a zero-shot manner to see if the program just overfit on the training environment.
- What if the environment is not deterministic? How do we evaluate the correctness of the transition function in this case?
7 comments
Response to Reviewer 7TCM [2/5]
Details of the baselines are missing, e.g., what is the basic harness in ARC-AGI-3? Do they have a memory system? What's the difference between SCHEMA and these baselines? Including these details can make the paper more promising and self-contained.
Thank you for the suggestion. At the end of Appendix B in the revision, we added full descriptions of both types of baselines (Table 3 summarizes how they differ from Schema), and added a brief description in Sec 4.1 of the main text. All harnesses use the same base model and the same action budget.
Basic harness. We use the official ARC-AGI-3 basic harness (the Standard harness that ARC Prize uses to evaluate frontier models). At each turn, the model sees the game state, the number of completed levels, the current frame (a numeric grid in text form, including animation frames), and the available actions. The system prompt asks it to write down in its reply any information it wants to keep and to give an action. The last action mentioned in the reply is executed in the next turn, and only one action is executed per turn. The harness provides no tools, so the model cannot run code or read or write files. Its memory is a rolling conversation window: each turn's observation and reply are appended to the conversation, and the oldest turns are dropped once the limit is exceeded (about 175K tokens, estimated from character count). If the model wants to keep a finding for the long term, it can only do so by writing it down again and again in later replies. There is no other memory system.
Coding harness (Claude Code / Codex). The basic introduction to the game in the system prompt, as well as the observation and action protocols, are essentially the same as in Schema. The agent keeps all the native coding tools of the CLI (shell, reading/writing/editing files, search). The model can freely keep persistent notes in its working directory, save its history, and write and run code. Its memory is therefore the conversation context plus the files it writes itself.
Differences from Schema.
| Basic harness | Coding harness | Schema | |
|---|---|---|---|
| World model | Context only | Notes or code | Executable program |
| Certification | None | Custom scripts | Full-history backtest |
| Planning | None | Custom search | BFS/A* |
| Execution | One action | Action sequences | Verified sequences |
Verified execution stops the sequence at the first prediction mismatch.
A coding harness can also write its own simulator, history replay script, and search program. Schema lets the same executable program directly support history backtesting, search-based planning, and step-by-step verified execution, while the harness maintains the interaction history automatically. The agent decides on its own when to revise the model, check it, plan, or keep exploring. The ablations in Table 1 keep the general-purpose tools and remove each of these mechanisms separately, which directly tests their contribution to performance.
The paper misses a limitations section, where the current evaluation protocol focuses on text-based, discrete-action, deterministic environments. Including it would better help readers understand the scope of the current method.
We thank the reviewer for the suggestion. We added a "Limitations and Future Work" section at the end of the main text in the revision. It discusses the challenges of extending to more complex environments, including learning mechanisms from raw real-world perception, handling stochastic events, and planning in continuous action spaces.
Response to Reviewer 7TCM [3/5]
What are the details of the prose model variant in
Table 1? Does it still do some kind of planning, but in text form?
Thank you for the question. We observe that the prose-model agent still checks its history, simulates its action plans, and compares execution results with what it expected, but it does all of these through natural-language reasoning.
This variant uses the same base model and interaction budget as Schema. It keeps the general-purpose tools and history access, and maintains its world model as written rules in notes.md. Since there is no executable prediction model, we removed the tools that depend on one: backtest, model_predict, and run_bfs. Action sequences can still be committed and executed, but execution cannot stop automatically when a prediction is wrong.
-
Certification: the agent uses
read_historyto pull up past transitions and checks them by working through the rules innotes.mditself. It does 185 such checks in total, covering about 1,600 transitions. On manual review, we found that some of these checks missed errors in the rules. Schema, in contrast, actually runs the program to produce predictions; the tool compares the predictions with the real observations one by one and returns the mismatched transitions along with the specific differences. -
Planning: the agent simulates candidate action sequences in natural-language reasoning, following the rules in
notes.md. Of the 2,832 action sequences it commits, about 50% contain predicted outcomes three or more steps ahead, and 58% state the expected outcome after the whole sequence. In simple cases, it can get correct routes of a dozen or so steps this way. -
Verified execution: the agent looks at the returned frames after the whole action sequence has run and compares them with what it expected. By committing shorter sequences, it gets environment feedback and checks its expectations more often, which approximates online verification during execution. In practice, it commits 3.7 steps per sequence on average, compared with 15.7 for Schema.
Response to Reviewer 7TCM [1/5]
We sincerely thank reviewer 7TCM for their detailed feedback. We would like to address their questions as follows:
We thank the reviewer for the suggestion. In
Appendix Bof the revision, we added the system prompt used by Schema, as well as the tool specifications.These prompts contain no game-specific rules, completion conditions, meanings of screen elements, or hints about mechanisms. They only contain general working principles, tool usage, and the necessary information about the environment interface. All environment-specific mechanisms are discovered by the agent itself through interaction with the environment.
We thank the reviewer for raising this question. Schema does not use a rule like "planning is allowed once history accuracy exceeds a certain threshold". The system prompt instructs the agent to have the program reproduce all recorded transitions before planning, but the harness does not make 100% agreement a mandatory gate for acting. Whether to keep revising the program or to first collect new evidence through interaction is decided by the agent, depending on the problem at hand. In practice, the agent usually plans after the history is fully reproduced, and it also keeps exploring when a few transitions cannot be explained for the moment.
Rare but critical transitions are not hidden by the overall accuracy.
backtestreports each mismatched historical transition individually, with its index, so that the agent can inspect the corresponding prediction and the real observation. Transitions that cannot be explained yet stay in the history, and as long as the mismatch is unresolved, it is reported again in later full backtests.In addition, the harness still verifies the program's predictions step by step when executing actions. As soon as the actual observation disagrees with the prediction, it immediately stops the remaining actions and returns that transition to the agent, which can then continue the investigation using both the new and the old evidence.
Explaining an existing mismatch sometimes requires new evidence from interaction. If all mismatches had to be eliminated before further interaction were allowed, the agent could lose the chance to get this evidence. Therefore, Schema allows the agent to leave unexplained transitions open for a while, keep running experiments to collect new evidence, and then revise the model accordingly and check it against the full history.
Here, we would like to share two interesting observations related to this.
A new mechanism can explain an old error in hindsight: in SU15 level 3, the agent could not yet explain a transition in which a block was shrunk and knocked back by a hunter. Clearing level 5 requires this mechanism, and through repeated experiments the agent found that it is triggered when the block blocks the hunter's movement. The updated program also correctly reproduces the transition left over from level 3 (
Appendix Iof the revision,Figure 24).An unresolved mismatch can serve as a clue: in KA59 level 5, the agent could not find a route over the wall, so it went back to the mismatch left over from level 4 and re-examined its assumption that "the timer only pushes pieces downward". It ran an experiment with a piece placed above the timer and found that the piece can be launched upward. It then corrected the direction rule and used this mechanism to plan how to clear the level (
Appendix Iof the revision,Figure 25).We have made the role of certification clearer in
Sec 3.2of the revision.