- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CompressARC, a method for solving ARC-AGI puzzles without pretrained weights or cross-puzzle neural training. For each puzzle, it optimizes a small, puzzle-specific neural model using a minimum-description-length-inspired objective applied to the observed grids, while leaving the test output unknown. The method achieves 20% accuracy on the public ARC-AGI-1 evaluation set under the standard two-attempt protocol. Although this is below leading methods, the result is promising given the restrictive learning setting and the model’s small size. The architecture and heuristics are nevertheless informed by the ARC training puzzles.
Critical strengths
The paper demonstrates that a small neural network can achieve meaningful ARC-AGI-1 performance through puzzle-specific optimization alone, without pretraining. This is an interesting contribution to learning from limited data. The compression-based formulation provides a coherent motivation for the method, and the selected case studies offer useful insights into the structures encoded by its learned representations.
Critical weaknesses
1.Insufficient quantitative evaluation of design choices. Although the appendices discuss several architectural alternatives and development decisions, the paper does not provide a systematic quantitative ablation study of its central components or hyperparameters. Consequently, it is difficult to determine how much performance comes from the compression objective, the specialized architecture, or the optimization and output-selection heuristics. In particular, comparisons with reconstruction-only training and a simpler decoder would help establish the contributions of KL regularization and architectural engineering.
- Limited characterization of successful and unsuccessful puzzles. The objective of Section 5.1, “What Puzzles Can’t We Solve?”, is only partially fulfilled. The section and Appendix H identify observed limitations through selected examples, but do not provide a systematic quantitative (and statistical) characterization of the unsolved puzzles. In particular, they report neither the number of puzzles requiring each capability nor the success rate within each category. It would therefore be more appropriate to frame this discussion as a preliminary attempt to identify the method’s limitations. Equally important is the complementary question: “What puzzles can we solve?” Although Appendix H lists observed capabilities and the paper analyzes selected successful cases, these do not establish which properties characterize the solved evaluation puzzles. It remains unclear whether the reported accuracy primarily reflects a narrow set of relatively simple, architecture-supported patterns or extends to more complex combinations of operations. A broader quantitative analysis of both successes and failures, supplemented by controlled puzzle variations, would clarify the scope of the method’s generalization and help distinguish architectural limitations from optimization failures.
Questions for authors
1.Could the authors provide a quantitative analysis of solved and unsolved puzzles grouped by required operations or structural properties? Reporting category sizes and success rates would help clarify which patterns the method captures reliably and whether its successes are concentrated in a limited set of puzzle types. Controlled variations in factors such as translation distance or the number of repeated operations would also help identify capability boundaries.
2.Which methodological and architectural choices contribute most to the final accuracy? In particular, could the authors compare the full method with reconstruction-only training, vary the KL weight, and evaluate a simpler decoder or remove selected specialized operations? Reporting the resulting changes in evaluation accuracy across multiple random runs would help establish the importance and robustness of these choices.
3 comments
We have now completed the rest of the experiments as promised and we would like to present the results to you.
Firstly, we show the isolated effect of removing architectural components and heuristics in our method, while leaving the compression aspects intact. The additional results are in a new section of Table 1. We successively remove architectural components and re-run to see how the score differs as CompressARC gets successively simpler. The more the score decreases with removed components, the more responsible the architecture is for the performance as opposed to compression. The table:
| Modification | Evaluation pass@2 |
|---|---|
| CompressARC (original) | 21.33% +/- 0.94% |
| minus symmetrical weight tying (Appendix E.2) | 13.5% |
| minus directional communication layer (Appendix D.5) | 16.25% |
| minus shift, cummax, and softmax layers (Appendix D.4) | 7.75% |
| minus solution selection heuristics* | 7.25% |
*When removing solution selection heuristics, we score by selecting the two most frequent solutions gathered during the training run.
Unfortunately the sequential ablation did not have a monotonic effect on the score as we had hoped. Nevertheless, we hope the moderate but not absolute decrease in score serves as evidence that the architecture is not fully responsible for the score, and the compression indeed plays a major role. At the very least, the stacked architectural and heuristic ablations had much less of a detrimental effect on the score as did freezing or disabling the KL loss term, which were modifications designed to disable compression without damaging the architecture.
And secondly, we have also added a quantitative analysis of the abilities of CompressARC to perform various operations involved in solutions, by using Michael Hodel's domain-specific language solutions as a source of ground truth on how to solve each training set puzzle. The analysis is in Appendix H, and includes two large Tables 5 and 6, which can be found in the updated version of the paper. The analysis matches the listing of abilities and inabilities of CompressARC we had previously included in the original version of our paper, with the main exception of cropping, which the new analysis shows to be an inability. CompressARC is shown to be most capable of working with bounding regions, rendering on backgrounds, and constructing lines and rays, and is least capable of splitting and joining shapes, locating copies of a shape, building variable-sized output grids, and counting and arithmetic. These mentioned items each impose a greater than 14% differential effect on the ability of CompressARC to solve a training set puzzle, which is substantial compared to CompressARC's overall training set score of 34.75%.
Please let us know if you have more questions. We would like to hear your thoughts on the additional evidence we have given for CompressARC, and whether you would consider raising your score.
The quantitative analysis added in Appendix H is valuable, and I would encourage the authors to prioritize its main findings in the main text. Currently, a substantial portion of the main text is devoted to a detailed case study of a single puzzle. While this illustrates the model’s behavior, its representativeness is unclear, and the extensive space allocated to it provides less insight into the method’s general capabilities and limitations than a systematic analysis across puzzles would. I therefore suggest substantially shortening the case study and using the space to present which puzzle types are solved reliably and which remain challenging, supported by category-level success rates, sample sizes, and uncertainty, while clearly acknowledging the analysis’s limitations.
The additional experiments constitute a substantial effort and address several of my original concerns, particularly regarding the importance of KL regularization. However, the architectural ablations remove components successively and therefore do not isolate each component’s contribution to the full model. Moreover, the ablations beyond the default configuration use single seeds, leaving the robustness of the observed differences unclear. The capability analysis also remains limited to training puzzles, which informed the architecture’s design, and reports associations between operation presence and success rather than establishing isolated capabilities or explaining performance on complex combinations of operations. These limitations should be reflected more clearly in the interpretation of the results.
Addressing these concerns would benefit from further validation, including independent component ablations across multiple seeds and a capability analysis on the public evaluation set, together with a substantial revision of the main text to foreground the systematic findings. In my assessment, this work would benefit from further development before publication. I encourage the authors to build on these promising results and strengthen the evidence and presentation for a future submission.
We sincerely thank the reviewer for their constructive criticism of our paper. We address the concerns listed in individual responses below:
We agree that the attribution of performance to various components is unclear as presented - therefore we ran several experiments to clear this up. We have now included these ablations in Section 5 and uploaded an updated version of the paper, with changes to the paper marked in blue. We are also still running further analyses, whose results we will post here before including the changes to the paper marked in magenta. (We have shortened some Figures near the bottom to stay within the page limit.)
Firstly, we swept the relative contribution of the KL coefficient as opposed to reconstruction, resulting in the following:
We also tried freezing a random z value (sampled from N(0,1) i.i.d. at the beginning of training) while training purely on reconstruction to isolate the effect of the architecture alone without the compression. This resulted in a score of 1.0%, supporting that architecture alone without compression is not responsible for our performance.
We are currently running experiments which test the effect of architectural engineering and solution sampling heuristics by successively the most heavily engineered components and seeing how the score changes and what classes/types of puzzles are lost. The relative KL contribution is fixed to 0.1 and the output selection heuristic is our original one for these experiments. We will post the results here in a follow-up once this is done running.
Within the coming days, we plan to provide a correlative analysis relating whether or not CompressARC achieves the correct solution against what operations the solution consists of, based on the Hodel DSL solution set at https://github.com/michaelhodel/arc-dsl/blob/main/solvers.py. This is a publicly available code base which contains handwritten solutions to all 400 training puzzles, within a domain-specific language built for ARC-AGI.