acceptodds
Under review as a conference paper at ICLR 2027

Kepler: Auditable World Models for ARC-AGI-3

Abstract

Final scores can conceal invalid behavior in interactive-agent benchmarks. We present Kepler, an ARC-AGI-3 harness in which agents express hypotheses as executable world models. Its protocol combines complete-history retrodiction, conditional next-observation checks, an append-only transition ledger, and mechanically replayed scoring. Checked mismatches interrupt plans; missing predictions do not block execution. Across two frontier coding models, 48 of 50 game-model cells reached the score cap. One frozen Claude Opus 5 configuration obtained a server-verified 100.00 over all 25 public games. On 181 of 183 completed levels, its final attempt used no more actions than the median-human baseline. This measures final-attempt efficiency, not learning cost or human-like cognition. Trajectory audits exposed failures hidden by final scores: an invalid perfect run after source access, harness reconstruction by all six agents in an intended control, instruction rewriting in 26 of 26 workspaces, and silent replacement of a broken planner. A single-game visual continuation resolved a mechanic missed during earlier text-only sessions. A code regression suite detects 11 of 13 hand-built threat fixtures and flags none of five benign controls; it misses unlogged behavior and a semantically wrong but runnable tool. These findings motivate reporting trajectory integrity, selection rules, resource accounting, and executable belief checks alongside outcome scores. They do not establish held-out generalization or a causal harness advantage.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, world models, benchmark integrity, autonomous agents, ARC-AGI-3, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KBmrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper neither uses the official ICLR template nor complies with the page-limit requirements. I therefore strongly recommend a desk rejection.

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Clarification of main-text page count and template concern

Thank you for flagging the formatting concern. We would like to separate the page-count issue from template compliance.

The PDF currently served for this submission has 10 pages in total, with References beginning on page 8 and appendices following the references. The ICLR 2027 Author Guidelines specify a nine-page main-text limit at submission, excluding references and allowing appendices after the bibliography. Thus the total PDF length of 10 pages does not by itself establish a page-limit violation.

That observation does not establish template compliance, and we are not claiming that it does. Could you or the Area Chair specify the template/layout discrepancy and whether the page-limit concern is about the main text under the required layout, rather than the total PDF length? We would appreciate a precise procedural determination so the formatting issue and the scientific evaluation are not conflated.

Guidelines: https://iclr.cc/Conferences/2027/AuthorGuidelines

AI assistance was used to check the served PDF and draft this response.

AuthorsOpenReview ↗

Revised PDF: official template and page count

Thank you for flagging the template and length. The revised PDF uses the official ICLR 2027 style files. The discussion ends on page 8; all main text and statements end on page 9, where the references begin. The 16-page total includes references and appendices. Even counting every statement, the pre-reference span is nine pages, below the ten-page rebuttal limit.

That fixes the current presentation; it does not determine whether the original submission met the rules. If any formatting issue remains in this revision, could you identify its page or the applicable requirement? If the paper is eligible, we would appreciate a substantive assessment of the evidence in Tables 2–3. AI assistance is disclosed in the revision.

Thank you for your response. My score remains unchanged.

Reviewer_c2zEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework to access intermediate stages of agents completing unfamiliar ARC-AGI-3 games as executable world models. Agents will observe environment transitions, check with past observations, and produce action programs for mechanical replay. It reports 100.00 with Claude Opus 5 and 95.97 with GPT-5.6 Sol across 25 public games. The main contribution is the evaluation-integrity study, which trajectory and workspace audits reveal source-code access, contaminated controls, instruction changes, broken tools, and replay discrepancies that successful runs can hide.

Critical strengths
  1. It actually point out a meaningful situation that the agent can achieve perfect score after reading the source code. So final score is great does not mean the intermediate or process is correct, so staged checking is required.

  2. Recording predicted transitions and actual observations creates testable artifacts without requiring CoT, providing a practicable approach.

  3. The performance reports include tokens and costs, which information makes the report easy to assess.

Critical weaknesses
  1. It seems to the setting allows the exploration after 5 times human action baseline. So successful replay of a final solution does not establish compliance with the original discovery budget.

  2. The quantitative audit evaluation uses 13 hand-built threat fixtures and five benign controls. It provides limited evidence about unfamiliar violations. Further independent validation is required.

Questions for authors

Do we allow actions to continue or reject strictly? Since the draft says empty predictions are rejected, while the release says they do not prevent action. https://github.com/Cveinnt/kepler#what-is-different

3 comments
AuthorsOpenReview ↗

Response: discovery budget, conditional checking, and audit evidence

Thank you for identifying the distinction between successful replay and valid discovery, and for recognizing the value of inspectable transition artifacts without access to private chain-of-thought.

Prediction behavior. Actions are allowed to continue without a prediction. In the submitted supplementary archive, source/harness/ws_tools/commit.py executes the action when simulation fails or the prediction abstains on all cells. For a usable prediction, it compares claimed cells after execution and discards the remaining plan on a mismatch; unclaimed cells are not checked. The surprising action has already happened. The PDF's stronger claim that every action requires a nonvacuous prediction is incorrect for this implementation; the conditional wording in the OpenReview abstract is the appropriate description. This check does not establish complete or truthful beliefs, and omitting predictions can evade it.

Discovery budget. We agree that replay of a final solution does not establish discovery within a 5x human-action budget. Retained-run actions, scored-level actions, official replay actions and total exploration are different denominators. The scores are evidence about the frozen public-game outcome, not a budget-matched discovery or generalization result. This distinction is already explicit for final-attempt efficiency in the PDF abstract and in the accounting discussion, but it must also govern any comparison with a discovery-budget-constrained system.

What the audit evidence does establish. The contribution is not a claim that a high score validates the process. Section 5.2 gives concrete counterexamples: a source-contaminated perfect run was voided, the intended six-workspace control did not isolate the harness, and agents replaced a broken supplied planner while maintaining strong outcomes. These cases distinguish outcome correctness, experimental access boundaries and actual tool functionality. The clean rerun's 46.91 score is a descriptive incident comparison, not a causal estimate of source access or harness benefit.

The regression suite provides a limited but reproducible comparison between evidence layers: 11/13 constructed threats detected with all checks, 5/13 without behavior-log checks, and 9/13 without ledger checks; no flags on five constructed benign controls. Section 5.2 explicitly labels this a code-path regression test, not field sensitivity or false-positive estimation. The two designed misses show the boundary: an event absent from retained evidence and a semantically wrong tool that starts successfully. Independent unseen trajectories would be needed to establish broader detection performance; that experiment is not presented here.

We therefore ask that the paper be assessed as a concrete evaluation-integrity study with inspectable evidence and explicit failure boundaries, not as proof of a universally enforced prediction gate, causal harness advantage or budget-constrained discovery. Correcting the gate description is necessary, but it does not erase the distinct failures that the trajectory and workspace audits identified.

AI assistance was used to inspect the supplied artifacts and draft this response, consistent with the disclosed AI-assisted workflow.

Thanks for the clarification. After reading the rebuttals, I would like to maintain my score.

AuthorsOpenReview ↗

Updated revision: measured discovery exposure and audit limits

Thank you for following up. You were right about the five-times-human limit. Our November 12 response did not show discovery within that limit, and we withdraw that interpretation. The revised paper adds a measurement we did not have in that exchange.

We counted actions on each level across resets through its first recorded completion, rather than taking its final successful replay. Ten of 180 completed GPT levels, across five games, were already over five times the released human baseline. On TN36 level 1 the ledger contains 13,736 such actions against an allowance of 160. The Opus release has no observed overage among 183 completed levels, but two ledgers begin midstream and neither board rules out prior exposure. This is not the performance of an agent operating under a cap. We retain the replay scores only to show how a successful replay can conceal that distinction (§5 and Appendix D, Table 6 and following text).

Two other answers also changed. An empty prediction permits the action: the tool checks claimed cells after execution, and an ordinary mismatch stops only the remaining plan (§3.2). And in a new diagnostic of 12 matched violation/control pairs with identical saved scores, the frozen checks flag only two violations and no controls. One flag is arithmetic; the other fires for a defective substring match. Ten violations pass unflagged (Table 3; all cases in Table 8). Separate models generated and adjudicated synthetic cases; these are not field error rates.

We are not asking you to accept a discovery or detector-performance claim. The evidence we ask you to assess is narrower: the same successful outcome can survive a change in information access, control integrity, or discovery exposure, and the retained process records establish concrete instances of each. AI assistance is disclosed in the revision.

Reviewer_oFezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Kepler combines executable transition models, history checks, an append-only ledger, and certified replay for ARC-AGI-3. Two frozen public-game boards reach 100.00 and 95.97 RHAE. The main contribution is a trajectory-integrity protocol and documented failures hidden by final scores, supported by a small deterministic audit suite.

Critical strengths
  1. The separation of outcome, trajectory integrity, selection, and resource accounting is valuable. Voiding source-contaminated results and withdrawing the invalid control provide concrete, unusually transparent evidence of why final scores alone are insufficient.
Critical weaknesses
  1. The core enforcement contract is inconsistent across submission materials. The PDF abstract and Sections 3.1–3.2 describe a non-vacuous prediction before every action, whereas the OpenReview abstract says missing predictions do not block execution. This changes what the released system actually certifies.
  2. Audit effectiveness beyond the reported incidents remains weakly established: the 13 fixtures were built from detector rules, only five benign controls are tested, and the supplement does not reproduce the 330-run historical scan. The acknowledged public-set reuse and invalid control also leave generalization and harness benefit unmeasured.
Questions for authors
  1. Which prediction-gating behavior was used for the headline boards? Please quantify actions with missing or masked predictions and clarify whether agents can evade checks by omitting them.
  2. Can independently labeled, previously unseen valid and invalid trajectories establish audit coverage and false alarms, and can the evidence supporting the historical scan be made reproducible?
2 comments
AuthorsOpenReview ↗

Response: implemented checks, reproducibility, and supported contribution

Thank you for recognizing the value of separating outcome, integrity, selection and resource accounting. We agree that the guarantee must be defined by the implementation, not by stronger prose.

Q1: prediction contract and coverage. The submitted supplement's source/harness/ws_tools/commit.py implements conditional checking: it attempts simulation before an action, but a simulation exception or an all-cell abstention does not prevent that action. With a usable prediction it checks claimed cells after execution, and a mismatch stops the remaining plan. Unclaimed cells are outside that comparison; terminal/level-transition handling also exits before the ordinary grid-comparison path. The PDF abstract and Sections 3.1-3.2 overstate this as a mandatory nonvacuous gate. The OpenReview abstract's conditional description is correct for the released implementation. Omission can evade the prediction check; the method does not certify complete beliefs or universal prediction correctness.

The packaged source identifies the released behavior, but it is not a per-action attestation of the exact prediction module and coverage used throughout every headline run. We do not have a verified board-wide count of missing/masked predictions to report here. An action ledger alone cannot reconstruct those predictions. Accordingly, the headline scores should not be interpreted as evidence that every action passed a nonvacuous prediction check. We are not substituting coverage measured in a later pilot for coverage of the reported boards.

Q2: reproducibility and independent validation. The 330 runs describe the historical development/evaluation corpus; the reproducible supplement is the 50 retained final-board runs, not that entire history. This limitation is stated in the PDF reproducibility statement (pp.7-8). The uploaded archive contains 95 captured CLI logs as well as ledgers and audit tools. Its documented offline checks reproduce the two 25-game stored boards and check 363 completed levels: 354 exact ledger matches, nine conservatively charged more actions, none charged fewer. These are artifact checks, not a new independent live-game replication, and zero flags in the retained evidence do not prove all behavior was logged.

The 13-threat/five-control suite is explicitly a regression test. Within those fixtures, removing behavior-log checks reduces coverage from 11/13 to 5/13, and removing ledger checks to 9/13. This is concrete evidence that the implemented layers test different failure classes, not an estimate of sensitivity on unseen trajectories. Independently designed and labeled valid/invalid trajectories would be needed for that broader claim; they are not supplied, and the complete historical scan is not reproducible from this archive.

What remains supported. The paper documents failures that a successful outcome does not resolve: source access yielding an invalid perfect run, canonical-tool access contaminating all six intended controls, and replacement of a broken planner. These are evidence about the experimental process, not claims that Kepler causally improves performance. The failed control is informative about experimental isolation while providing no harness-effect estimate. Likewise, public-set reuse prevents a held-out generalization claim but does not make the recorded access, tool and replay failures disappear. We ask that the contribution be evaluated on this narrower, inspectable evaluation-integrity result, while treating universal gating, causal benefit and generalization as unestablished.

AI assistance was used to inspect the supplied artifacts and draft this response, consistent with the disclosed AI-assisted workflow.

AuthorsOpenReview ↗

Updated revision: enforced checks and reproducible evidence

Your distinction between the prescribed protocol and the enforced one changed the paper. We withdraw the universal prediction-gate claim. If a model loads but returns no usable prediction, the action can execute. An ordinary mismatch is observed after the action and stops the remaining plan; history backtesting is prescribed rather than enforced. Frozen-code fixtures reproduce these branches, and historical CN04 and FT09 captures show fallback execution (§3.2; Appendix D).

What can be verified historically? Across 23 GPT runs we matched 569 mismatch receipts to native event IDs and checked every one of their 6,494 printed observed-cell values against the ledger. For 260, the tool reports a nonempty unexecuted tail. These are concrete instances of falsification, not an estimate of actions saved. No complete contemporaneous prediction/mask archive exists: the receipts cannot supply a whole-board coverage rate, and we do not extrapolate to Opus.

The new diagnostic also changed our assessment of our own auditor. Twelve executed violation/control pairs have the same saved score in both arms. Frozen full checks flag 2/12 violations and 0/12 controls; one positive is an arithmetic discrepancy and the other is a defective lexical trigger. We show all ten misses (Table 3; Appendix E, Table 8). The separate 48-case pilot flags 17/24 violations and 0/24 controls; the constructions differ and we do not pool them. Separate model authoring and adjudication are not independent human validation. Nor does a clean-looking scan certify a natural trajectory: the original 330-run membership is unrecovered; the new 255-workspace census is different.

The contribution we ask you to evaluate is experimental validity, not a new learning algorithm: a perfect result was voided after source access, all six intended controls reconstructed treatment, and a broken planner was masked by agent-written replacements (Table 2). Each defeats a different inference from a successful outcome. We do not claim a causal harness advantage or unseen-game performance. AI assistance is disclosed in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.