acceptodds
Under review as a conference paper at ICLR 2027

Compiled Worlds: What Adaptation Costs on Transformed ARC-AGI-3 Games

Abstract

Completion on familiar games does not show how an agent adapts when a game changes, or what the adaptation costs. We introduce Compiled Worlds, a protocol that edits official ARC-AGI-3 game programs into paired variants (colour permutations, action relabellings, resampled layouts and rule mutations), serves them through the unmodified toolkit, and attaches executable solution witnesses to layouts and mutations. We evaluate four harnesses with a common backbone, GLM-5.3, and selected configurations with three further backbones. Because every variant has an original, the protocol separates what a harness costs on a familiar level from what a change adds. On the original first levels of the five games with layout variants, Tycho with GLM-5.3 already uses nine times the calls of a compile-by-intervention harness (CbI) with the same backbone. Measured against its own originals, Tycho pays mainly for new layouts, 3.3 times its original first-level calls and more on all eight layouts it completes, whereas CbI's premium is at most 1.4 on every tier; each fails five of thirteen layouts. First-level rule mutations cost about as much as the original levels, and each model-building configuration evaluated on them with an API backbone solves at least four of five. Across 33 paired first runs, colour permutation does not change completion measurably. Finally, ten of CbI's twelve zero-completion first runs with GLM-5.3 reach full agreement with the logged transitions, so trace agreement does not ensure completion.

Primary area:
datasets and benchmarks
Keywords:
ARC-AGI-3, interactive reasoning, evaluation methodology, program transformation, adaptation cost, world models, LLM agents, procedural generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z5yFOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Compiled Worlds generates colour, action-label, layout, and rule variants of ARC-AGI-3 games and pairs them with their originals to measure completion and resource costs. It compares several agent harnesses and uses execution logs to examine transition-model fitting, goal inference, and task completion.

Critical strengths
  1. Pairing variants with originals helps distinguish baseline harness costs from the additional costs of task changes. Transformation audits and search-based solution witnesses improve interpretability.
  2. Reporting calls, tokens, time, and action efficiency reveals resource differences hidden by completion scores. The logs also provide useful failure diagnostics: agreement with observed transitions does not ensure completion.
Critical weaknesses
  1. The cost findings have limited statistical support. Layout and rule experiments cover only first levels from a small set of games. Premiums use a single original run as the denominator and include only successful variants, while repeated call counts can differ by 6.6×. These limitations are acknowledged, but the stability of the main cost comparisons remains untested.
  2. Exact reconstruction of the evaluated variants is incomplete. T2 sampling depends on unordered set iteration, so a fixed random seed does not ensure identical layouts across processes. The generator also skips seeds already recorded in the certificate file. The supplied T2/T4 records omit action witnesses, making the evaluated versions difficult to reconstruct and independently verify.
Questions for authors
  1. Could you repeat runs of both originals and variants, report uncertainty in the cost ratios, and test whether changing the evaluation budgets alters the relative findings for layout and rule variants?
  2. How can the evaluated T2/T4 versions be reconstructed exactly from a clean directory? Please address unordered sampling and certificate-based skipping, and provide the corresponding action witnesses.
3 comments
AuthorsOpenReview ↗

Response to Reviewer z5yF

Thank you. The revision adds exact reconstruction of the evaluated variants with stored solution witnesses, and matched repeats that report completion beside cost, with uncertainty intervals. The repeats do not establish a layout cost premium across games, so we withdraw the 3.3× figure. The strengths you listed are unchanged. We answer Q2 first; the budget part of Q1 is answered from stored runs only.

Q2. Exact reconstruction (Section 4.3, Appendix A). You were right on each point: the sprite draw iterated over a Python set, so a fixed seed did not fix the layout; the generator skipped seeds already recorded; and the witnesses were stored only inside the game files.

  • The evaluated layouts therefore cannot be re-drawn from their seeds. They are fixed by their recorded sprite positions, and the mutations by the registry. The supplement adds rebuild_versions.py, which rebuilds them from a directory holding only the official games: the 13 layouts, the 5 mutations and the ls20 layout.
  • It checks each program's SHA-256, the start frame and a fixed 40-action rollout, then replays the stored witness, which must complete the level at its last action and not before.
  • All 19 rebuild byte-identically (Python 3.12), including from the unpacked supplement. Witnesses and hashes are in worlds/witnesses.json.
  • The generator now draws in sorted order, and a --fresh mode starts from an empty record, so a clean-directory run depends only on the game files and the seeds.
  • The single-level originals and the sp80 layout of the new experiments come from these deterministic generators (hashes and witnesses: worlds/witnesses_ext.json).

Command, from research/compiled-worlds in the unpacked supplement: python worlds/rebuild_versions.py --orig <official_games_dir> --out <dir>

Q1a. Repeats and uncertainty (Section 6.3; Tables 2, 5, 6; Figure 2). We repeated a fixed subset: a single-level original and one layout on each of five games, and the five rule mutations (on four of them), on Tycho and CbI with GLM-5.3, three runs planned per version (2 to 5 retained; changes after seeing data: Appendix C), with 95% bootstrap intervals over runs and over games.

  • Tycho's layout premium on calls is 1.47, with an interval over games of [0.68, 3.83], not 3.3. We no longer claim a layout premium across games.
  • No cost increase is detected for rule mutations: Tycho 0.71 [0.38, 1.14], CbI 0.66 [0.28, 2.14]. The intervals are too wide to show equal cost.
  • Layouts are completed less often than mutations. On the four games with both tiers, Tycho completes 9 of 14 layout runs against 15 of 15 mutation runs, and CbI 7 of 11 against 14 of 15: an exact test stratified by harness and game gives two-sided p = 0.005 over both harnesses (post hoc; 0.015 after a Holm adjustment). On cost the contrast is directional: Tycho's layout premium exceeds its mutation premium in 95.1% of bootstrap draws.
  • With failed runs counted at what they spent (Table 2; defined after seeing the data), Tycho's layouts cost 1.59 [0.99, 2.72] times its originals and its mutations 0.57 [0.25, 1.05]; CbI's 1.22 [0.56, 3.09] and 0.65 [0.34, 1.41].
  • On two further games, Tycho's layouts cost 1.04 times its originals' calls (interval over runs [0.49, 2.77]).
  • Both problems you named moved the estimate. Only successful variants: Tycho's first runs completed three of the five repeated layouts (3.67×); with repeats it completes the other two as well, and the premium over all five is 1.91× on the same denominators. A single original run: repeated originals move 1.91× to 1.47×. One run per version, in every combination, gives 0.61 to 3.33 in 90% of combinations (post hoc).
  • Across the 18 repeated versions (matched repeats and extension), Tycho's largest call count between successful runs is a median of 2.6 times its smallest (up to 7.6 times).

Q1b. Budgets (Appendix C; Figure 2). We made no runs with other budgets. We truncated the matched-repeat runs at lower ones, counting a run as completed at a budget if it completed within it:

Tycho, call budget50100150200300
Originals completed (of 21)815171819
Layouts completed (of 18)5791212
Mutations completed (of 15)912151515
Cost ratio with failures counted, layouts1.111.271.461.501.59
Cost ratio with failures counted, mutations0.880.720.710.640.57

At each budget the layouts have the smallest share of completed runs (also for CbI, by LLM time). These describe the stored runs, without intervals. They do not show how an agent configured with a different cap would behave, and CbI's LLM time does not reconstruct what a different wall-time cap would have produced (Appendix C; Section 7).

First levels only. Bounded search solves one of the fourteen later levels we tried, so the suite has no later-level variants (Section 7).

I thank the authors for their thoughtful and detailed rebuttal. The response has satisfactorily addressed my main concerns and helped clarify several aspects of the work. I appreciate the authors' efforts and will maintain my original rating of 3 (Weak Accept) while increasing my confidence from 2 to 3.

I wish the authors the best of luck!

Thank you for your note and for your careful review. Your requests for exact reconstruction and for repeated runs with uncertainty made the paper more reliable, and we are grateful for them.

Reviewer_eUrWOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Compiled Worlds constructs paired variants of official ARC-AGI-3 programs via colour permutations, action relabelling, layout resampling, and rule edits, providing executable solution witnesses for the layout and rule variants. The paper evaluates several harness configurations across shared backbones, measuring task completion, resource usage, and cost relative to each harness’s original first-level run. With GLM-5.3, the reported Tycho layout premium is roughly 3.3 times the original call count, whereas rule-mutation costs remain close to the original baseline. Colour and action perturbations show no statistically significant effect on completion in the available runs. Additionally, CbI traces illustrate that consistency with observed transitions is not sufficient for task completion.

Critical strengths
  1. Pairing program transformations directly with their originals provides a clean and useful evaluation instrument. Separating rendering, action mapping, layout, and rule edits makes each intervention easy to interpret, while executable witnesses verify that accepted layout and rule variants remain solvable in the engine. The paper also properly distinguishes these witnesses from guarantees of globally optimal action counts, making the protocol a promising complement to evaluation on unseen games.

  2. The resource accounting is thorough and unusually transparent. Because the original first levels already cost Tycho 277 calls versus CbI’s 30, the paper rightly avoids attributing the entire cross-harness gap to adaptation. It also reports tokens, wall-clock time, failures, and conditional costs on successful runs separately: fewer CbI calls do not imply faster execution, and CbI is actually slower on 13 of the 18 layout and mutation variants. Candidly disclosing configuration changes, CbI’s development games, and uneven coverage further helps readers interpret what these comparisons establish.

Critical weaknesses
  1. The central adaptation-cost finding relies on a fragile estimate of the original-run baseline. Each cost premium is normalized by a single original first-level run, and multiple variants reuse that same stochastic denominator. Moreover, Tycho’s eight successful layouts come from only four source games, while the five failed layouts are excluded from this cost-to-completion statistic. Meanwhile, repeated runs of a single mutation require 126 versus 19 calls (Section 5.2), pointing to substantial run-to-run cost variability. While this does not imply the layout effect is spurious, the current design cannot quantify how stable the 3.26x premium or the layout-versus-rule contrast is across new runs. Although the authors appropriately label individual ratios as indicative and disclose the conditioning in Appendix A, the core empirical contribution would be much stronger with matched repeats of both originals and variants.

  2. The evaluated rule and layout suite is too narrow to establish broad benchmark utility. Thirteen layouts span five games and five rule mutations span four games, all restricted to the first level, with accepted mutation witnesses ranging from 2 to 21 actions (Section 3 and Table 11). Rule-mutation completion is already near saturation for the evaluated model-building configurations with API backbones, whereas direct play remains largely at the floor. Consequently, the more substantive transformations currently offer limited diversity and discriminative power beyond the small layout panel. The paper openly acknowledges the bounded-search selection and omitted click-only games, so this is a limitation of scope rather than an undisclosed flaw. Overall, I see a useful diagnostic prototype, but I would want stronger evidence of robustness and discriminative coverage for acceptance as a benchmark contribution. I currently lean toward weak rejection.

Questions for authors
  1. In the CbI trace analysis showing that agreement with observed transitions fails to yield task completion, what is the primary failure mode? Specifically, do failures occur because the agent under-explores the environment (leaving critical goal transitions unobserved so the synthesized model is only locally consistent), or because planning/search over the learned model fails even when the transition dynamics are adequately captured? Additionally, how do completion rates and costs compare between CbI’s development games and strictly held-out games?

  2. How sensitive are the adaptation results to the witness-filtering bottleneck? Within the accepted suite (2 to 21 reference actions), do variants with longer solution witnesses exhibit systematically higher failure rates or token/runtime overhead than very short variants (e.g., <5 actions), and what happens to candidate rule mutations that fail bounded-search verification: are they predominantly unsolvable, or simply beyond the search horizon?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eUrW

Thank you for the precise requests. We ran the experiment you asked for under a criterion fixed in advance and report all of it.

Q1. Matched repeats (Section 6.3; Tables 2, 5, 6; Figure 2; Appendix C). A fixed subset: a single-level original and one layout on each of five games, and the five rule mutations (on four of them), on Tycho and CbI with GLM-5.3, three runs planned per version. Premiums are ratios of calls, variant over repeated original.

Runs completedPremium over completed runs [95% interval over games]Expenditure ratio: failed runs counted at what they spent (post hoc)
Tycho, layouts12 of 181.47 [0.68, 3.83]1.59 [0.99, 2.72]
Tycho, mutations15 of 150.71 [0.38, 1.14]0.57 [0.25, 1.05]
CbI, layouts9 of 140.81 [0.57, 2.45]1.22 [0.56, 3.09]
CbI, mutations14 of 150.66 [0.28, 2.14]0.65 [0.34, 1.41]

Originals: Tycho completed 19 of 21 runs, CbI 12 of 15 (the last column counts their failures too). Variant runs pool September and November; originals are all new.

  • Is the 3.26× premium stable? No. It is 1.47, its interval over games includes 1, and we withdrew it. You identified the cause: "the five failed layouts are outside this cost-to-completion statistic". Two of those five (cd82, m0r0) are in the repeated subset. With repeats Tycho completes them (in one of three and three of four runs), at 0.93× and 0.44× their repeated originals' calls. On the September denominators, the repeated runs give 3.96 on the three layouts completed at first and 1.91 on all five; repeated originals move 1.91 to 1.47.
  • How do within-game premiums change across runs? Pairing completed Tycho layout runs with completed runs of their originals gives ratios all above 1 on cn04 (3.4–6.2) and ft09 (1.6–16.2), all below 1 on m0r0 (0.21–0.91), and mixed on cd82 and lp85 (Figure 2). An F-test on log calls gives evidence against a common ratio (p = 0.002; 0.03 with the two further games; post hoc), weaker when failed runs are counted (0.05 and 0.16). One run per version, in every combination, gives a five-game premium between 0.61 and 3.33 in 90% of combinations.
  • Does the layout-versus-rule contrast survive? On completion, yes; on cost, in direction only. On the four games with both tiers, Tycho completes 9 of 14 layout runs against 15 of 15 mutation runs, and CbI 7 of 11 against 14 of 15. Weighting the four games equally, the share of layout runs completed is 0.32 lower (95% interval over runs 0.17 to 0.47). An exact test stratified by harness and game gives two-sided p = 0.005 over both harnesses (post hoc; 0.015 after a Holm adjustment; Tycho alone 0.046, CbI alone 0.16; conditional on these four games). On cost, Tycho's premiums:
AnalysisLayoutsMutationsShare of bootstrap draws with layouts above mutations
Main: repeated single-level originals1.470.710.951
September single run of each original as denominator1.911.13none (one run per game)
Variant runs made in November only1.050.550.779

Across Table 5's resampled analyses the share is 0.78 to 0.96, and the interval of the ratio includes 1. CbI's cost contrast is weaker: 0.81 against 0.66 (share 0.759).

Departures from the plan: Section 5.2 and Appendix C.

Q2. Further games, later levels and differentiation (Sections 6.1, 6.4 and 7).

  • Two further certified games. Under a protocol fixed before any extension run or new layout generation, we added ls20 and sp80 (24 runs; one departure, on sp80's layout, is disclosed in Appendix D); vc33 yielded no certified layout. Tycho completed all 12 runs, with a layout-to-original call ratio of 1.04 [0.49, 2.77] (interval over runs). CbI completed 4 of 6 layout runs and 3 of 6 original runs. No layout premium is detected there; equivalence is not established.
  • Does the layout-versus-rule pattern persist there? Not tested. These games have no certified rule mutations.
  • Later levels. Search (240 s per level) solves one of the fourteen second and third levels of the seven games with layouts (cd82's second), so the paper has no later-level suite (Section 7).
  • Differentiation. We agree that the mutations saturate: model-building configurations with API backbones complete 4 or 5 of 5, and direct play 0 or 1 (Section 6.1). Layouts differentiate more: on first runs of the five planned layouts, the eight configurations complete 0, 0, 1, 2, 3, 4, 4 and 5 (one level is within run-to-run variation).

What the paper claims. The paired instrument you described as useful, with the 18 submitted variants now rebuilt exactly and checked against stored solution witnesses (Section 4.3); on the four games with both tiers, lower completion on layouts than on rule mutations (exploratory, and untested on the two further games); and a corrected cost claim with its uncertainty. Broad benchmark coverage is not established.

Thanks for running the extra experiemnts and being so honest about the results. However, the new numbers basically confirm my main concerns:

With matched repeats, the main 3.26x layout cost ratio drops to 1.47 [0.68, 3.83] and is no longer significant, so this headline claim had to be withdrawn. The completion rate gap between layouts and rules is interesting, but it is a post-hoc result on just 4 games.

On the two new games (ls20, sp80), Tycho finishes 12/12 runs with no extra cost (1.04x), and there are no rule mutations to test the layout-vs-rule pattern further. Also, search only solves 1 out of 14 later levels, showing that the current pipeline really struggles to scale beyond short first levels.

I really appreciate the authors' honesty in updating the claims, which fixes my soundness concern. Still, since the main cost result didn't hold up and the benchmark coverage remains very small, I will maintian my score of 2.

AuthorsOpenReview ↗

Reply to Reviewer eUrW's follow-up

Thank you for the follow-up and for confirming that the soundness concern is resolved. We agree with each point: the 3.26× premium is withdrawn; the completion contrast is post hoc, pooled over both harnesses, and rests on four games; on the two further games no layout premium is detected for Tycho and the contrast cannot be tested; and bounded search solves one of the fourteen second and third levels.

Two short clarifications of what the revised paper claims:

Cost. The paper no longer claims a layout premium across games. Its cost result is the correction itself. For Tycho with GLM-5.3, one run per version (the design behind the submitted 3.26×), drawn from the matched-repeat records in every combination, gives a layout premium between 0.61 and 3.33 in 90% of combinations, and game by game, among completed runs, the ratio ranges from 0.44 to 4.63 over the five games (both post hoc; Section 6.3, Figure 2). The 0.93 and 1.14 on ls20 and sp80 lie inside that per-game range.

Coverage. We agree that it is small. The paper presents a paired protocol on ten games: layouts on seven, rule mutations on four, first levels only. It does not claim broad benchmark coverage, and each finding is stated for the games it rests on (Sections 1 and 7). Whether layouts are completed less often than rule mutations beyond those four games is open: the extension's protocol, fixed before its runs, included no rule mutations (Sections 5.2 and 6.4; EXTENSION_PROTOCOL.md in the supplement).

Your request for matched repeats changed the paper's main result, and we are grateful for it.

Thank you for the clarification and additional experiments. I appreciate the your correction of the original cost claim. However, the remaining findings are limited by the small number of games and the exploratory nature of the completion comparison. My final rating remains 2.

Reviewer_ZDdbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Compiled Worlds, a protocol that rewrites official ARC-AGI-3 game programs into paired variants. The variants are colour permutations (T1), action relabellings (T1b), certified first-level layout resamplings (T2) and rule mutations (T4), and they are served through the unmodified toolkit with BFS solution witnesses. Four harnesses (the ARC Prize plain harness, Tycho, Twin and the authors' own CbI) are evaluated with a common GLM-5.3 backbone, plus selected configurations with DeepSeek V4.1 Flash, GPT-6 and Qwen3.8-27B. Because each variant has an original, the authors can separate a harness's baseline cost from its adaptation premium. Tycho already uses about nine times CbI's calls on the original first levels and pays mainly for new layouts (3.3 times), while CbI's premium stays at or below 1.4×; both fail 5 of 13 layouts. Rule mutations cost about as much as the originals, colour permutation shows no measurable completion change (sign test p = 0.33), and 10 of CbI's 12 zero-completion runs still reach full trace agreement, showing that trace agreement does not guarantee completion.

Critical strengths
  1. This paper proposes an interesting idea of how to evaluate the cost that agents take to transfer to transformed games.

  2. The transformation strategy for ARC-AGI-3 games can be useful to create more paired variations of the game for future research.

Critical weaknesses
  1. The scale of experiments is small. The number of games are small, and the types of games are limited in ARC-AGI-3 games. It is unknown whether these conclusions are generalized to more ARC-AGI-like games, or games outside the scope of ARC-AGI-3 games.

  2. The paper is very difficult to read. The main text is packed with numbers, mixes first-run and repeat-averaged statistics, and is hard to follow without constant reference to the appendices. There are some concepts that are mentioned before it is defined. For example, "premium" is mentioned in the abstract but it is defined (informally, and that definition sentence in Section 5.1 is difficult to read, consider using a math formula to present) in Section 5.1. Moreover, the paper assumed that the readers are familiar with ARC-AGI-3 games and harness such as Tycho and Twin. A brief introduction of them can help readers in a more broader area understand this work.

  3. Comparing the number of LLM calls alone between Tycho and CbI is not a convincing clue that CbI is more efficient because an agent can make several short calls or a long call, both of which may consume the same amount of time or number of tokens.

Questions for authors
  1. What is the "reference" and "agent" in the ROAE formula?

  2. I did not understand the "repeats" part in Section 4.2. Why do harnesses only repeat some specific games (and they are different across harnesses)? What does the mutation in this paragraph refer to?

  3. What is the meaning of the medium columns in Table 2?

  4. How does CbI work?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZDdb

Thank you for the detailed review. Each weakness led to changes: the paper is rewritten to follow one line of argument and to introduce premium, the benchmark and the harnesses early (your second point); cost is reported on calls and output tokens, with no efficiency ranking (your third); and the claims are narrowed to what this scale supports (your first).

Readability.

  • Premium defined before use. The abstract and introduction now define it in words: what a harness spends to complete a variant, divided by what it spends to complete the matched original. Section 5.2 (paragraph "Cost and premium") gives the formula, Πh(T)=∑vCˉh(v)/∑vCˉh(o(v))\Pi_h(T) = \sum_v \bar C_h(v) / \sum_v \bar C_h(o(v)), summing over the variants vv of tier TT with a completed run and a completed original, where Cˉh(v)\bar C_h(v) is harness hh's mean cost (calls or tokens) over its completed runs of vv, and o(v)o(v) is the matched original.
  • Background. A new Section 3 introduces the ARC-AGI-3 interface and the harnesses, and Box 1 there describes CbI step by step. Figure 1, one game with three kinds of variant, now sits in the introduction.
  • First-run and repeated statistics are labelled. Section 5.2 describes the runs as three stages (September matrix, matched repeats, two-game extension), and the results follow them: September completion in Section 6.1, matched repeats in 6.3, the extension in 6.4. Section 6.2 names the runs behind each sentence, and Appendix A (Table 3) lists the runs behind the headline counts.
  • Fewer trips to the appendix. The main text now holds the definitions, the formula, one table and one figure for the repeats (Table 2, Figure 2) and a table of first-run completions (Section 6.1). The appendices hold per-run detail and sensitivity analyses.
  • Structure. The abstract and introduction follow one question (what a change costs, and whether the changed game is still completed); the introduction lists four findings with their sections, and each results section is titled by its question.

Calls alone do not show efficiency. We agree. With GLM-5.3, on the repeated originals of seven games, Tycho makes more calls than CbI on all seven (4 to 26 times), but CbI generates more output tokens on six (2.2 to 7.8 times; a post hoc summary). In the September first runs CbI was also slower in wall time on 13 of 18 layout and mutation versions. The revision draws no efficiency ranking (Section 6.2) and reports the matched-repeat premiums on both calls and output tokens (Table 2).

Scale.

  • We added two games under a protocol fixed in advance and repeated a fixed subset at three planned runs per version.
  • The scale is still small, and the repeats confirmed your concern: on five games our headline did not hold (Tycho's layout premium is 1.47, with a 95% interval over games of [0.68, 3.83], not 3.3), and we withdrew it.
  • The revision keeps claims this scale can carry: a correction (a single-run estimate whose support across games did not survive matched repeats), an observation on seven games (which harness is cheaper depends on the metric), and a comparison within the repeated games (layouts are completed less often than rule mutations of the same four games; pooled exact test, p = 0.005, post hoc). We no longer claim a premium across games; each finding is stated for the games it rests on (Sections 1 and 7).

Questions.

  1. ROAE. ROAE = min(1, reference/agent)² per completed level. Reference is the length of the solution witness found by search; agent is the number of actions the agent took (Section 5.2).
  2. Repeats. In the September matrix only some cells were run a second time, and coverage differs across harnesses because of cost and usage limits (Section 5.1). Tycho repeated its five planned layouts and all five rule mutations (the T4 versions, which is what "mutation" meant there); the other repeats were chosen after inspecting first runs, mostly where levels were lost or harnesses disagreed, so they are not a random sample (Section 5.2; listed in Appendix A). The new experiment fixed its versions and run counts in advance.
  3. "Median" in the submitted Table 2 (now Table 4). It is the median, over the variants a harness completed, of the per-variant ratio of calls (variant over original). "Total" is the ratio of summed calls. The caption now defines every column.
  4. CbI (Box 1). (1) Probe: act and log each transition. (2) Compile: the model writes a Python function that predicts the next frame. (3) Trace test: replay every logged transition; failures go back for repair. (4) Propose goals as predicates over frames. (5) Plan by search over predicted frames. (6) Execute and verify: a mismatch returns to step 3.

To check quickly: premium in the abstract (p. 1), Figure 1 (p. 2), Section 3 and Box 1 (p. 3), the formula (p. 6), Table 2 (p. 8), Figure 2 (p. 9). If a passage is still hard to follow, please name it and we will revise it within the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.