- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Compiled Worlds generates colour, action-label, layout, and rule variants of ARC-AGI-3 games and pairs them with their originals to measure completion and resource costs. It compares several agent harnesses and uses execution logs to examine transition-model fitting, goal inference, and task completion.
Critical strengths
- Pairing variants with originals helps distinguish baseline harness costs from the additional costs of task changes. Transformation audits and search-based solution witnesses improve interpretability.
- Reporting calls, tokens, time, and action efficiency reveals resource differences hidden by completion scores. The logs also provide useful failure diagnostics: agreement with observed transitions does not ensure completion.
Critical weaknesses
- The cost findings have limited statistical support. Layout and rule experiments cover only first levels from a small set of games. Premiums use a single original run as the denominator and include only successful variants, while repeated call counts can differ by 6.6×. These limitations are acknowledged, but the stability of the main cost comparisons remains untested.
- Exact reconstruction of the evaluated variants is incomplete. T2 sampling depends on unordered set iteration, so a fixed random seed does not ensure identical layouts across processes. The generator also skips seeds already recorded in the certificate file. The supplied T2/T4 records omit action witnesses, making the evaluated versions difficult to reconstruct and independently verify.
Questions for authors
- Could you repeat runs of both originals and variants, report uncertainty in the cost ratios, and test whether changing the evaluation budgets alters the relative findings for layout and rule variants?
- How can the evaluated T2/T4 versions be reconstructed exactly from a clean directory? Please address unordered sampling and certificate-based skipping, and provide the corresponding action witnesses.
3 comments
I thank the authors for their thoughtful and detailed rebuttal. The response has satisfactorily addressed my main concerns and helped clarify several aspects of the work. I appreciate the authors' efforts and will maintain my original rating of 3 (Weak Accept) while increasing my confidence from 2 to 3.
I wish the authors the best of luck!
Thank you
Thank you for your note and for your careful review. Your requests for exact reconstruction and for repeated runs with uncertainty made the paper more reliable, and we are grateful for them.
Response to Reviewer z5yF
Thank you. The revision adds exact reconstruction of the evaluated variants with stored solution witnesses, and matched repeats that report completion beside cost, with uncertainty intervals. The repeats do not establish a layout cost premium across games, so we withdraw the 3.3× figure. The strengths you listed are unchanged. We answer Q2 first; the budget part of Q1 is answered from stored runs only.
Q2. Exact reconstruction (Section 4.3, Appendix A). You were right on each point: the sprite draw iterated over a Python set, so a fixed seed did not fix the layout; the generator skipped seeds already recorded; and the witnesses were stored only inside the game files.
rebuild_versions.py, which rebuilds them from a directory holding only the official games: the 13 layouts, the 5 mutations and the ls20 layout.worlds/witnesses.json.--freshmode starts from an empty record, so a clean-directory run depends only on the game files and the seeds.worlds/witnesses_ext.json).Command, from
research/compiled-worldsin the unpacked supplement:python worlds/rebuild_versions.py --orig <official_games_dir> --out <dir>Q1a. Repeats and uncertainty (Section 6.3; Tables 2, 5, 6; Figure 2). We repeated a fixed subset: a single-level original and one layout on each of five games, and the five rule mutations (on four of them), on Tycho and CbI with GLM-5.3, three runs planned per version (2 to 5 retained; changes after seeing data: Appendix C), with 95% bootstrap intervals over runs and over games.
Q1b. Budgets (Appendix C; Figure 2). We made no runs with other budgets. We truncated the matched-repeat runs at lower ones, counting a run as completed at a budget if it completed within it:
At each budget the layouts have the smallest share of completed runs (also for CbI, by LLM time). These describe the stored runs, without intervals. They do not show how an agent configured with a different cap would behave, and CbI's LLM time does not reconstruct what a different wall-time cap would have produced (Appendix C; Section 7).
First levels only. Bounded search solves one of the fourteen later levels we tried, so the suite has no later-level variants (Section 7).