acceptodds
Under review as a conference paper at ICLR 2027

ARC-AGI Without Pretraining

Abstract

Conventional wisdom in the age of LLMs dictates that solving IQ-test-like visual puzzles from the ARC-AGI-1 benchmark requires capabilities derived from massive pretraining. To counter this, we introduce *CompressARC*, a 76K parameter model without any pretraining that solves 20% of evaluation puzzles by minimizing the description length (MDL) of the target puzzle purely during inference time. The MDL endows CompressARC with extreme generalization abilities typically unheard of in deep learning. To our knowledge, CompressARC is the only deep learning method for ARC-AGI where training happens only on a fraction of one sample: the target inference puzzle itself, with the final solution information removed. Moreover, CompressARC does not train on the pre-provided ARC-AGI "training set". Under these extremely data-limited conditions, we do not ordinarily expect any puzzles to be solvable at all. Yet CompressARC still solves a diverse distribution of creative ARC-AGI puzzles, suggesting MDL to be an alternative, highly feasible way to produce intelligence, besides conventional massive pretraining.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
ARC-AGI, compression, test-time-training, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zR36OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CompressARC, a method for solving ARC-AGI puzzles without pretrained weights or cross-puzzle neural training. For each puzzle, it optimizes a small, puzzle-specific neural model using a minimum-description-length-inspired objective applied to the observed grids, while leaving the test output unknown. The method achieves 20% accuracy on the public ARC-AGI-1 evaluation set under the standard two-attempt protocol. Although this is below leading methods, the result is promising given the restrictive learning setting and the model’s small size. The architecture and heuristics are nevertheless informed by the ARC training puzzles.

Critical strengths

The paper demonstrates that a small neural network can achieve meaningful ARC-AGI-1 performance through puzzle-specific optimization alone, without pretraining. This is an interesting contribution to learning from limited data. The compression-based formulation provides a coherent motivation for the method, and the selected case studies offer useful insights into the structures encoded by its learned representations.

Critical weaknesses

1.Insufficient quantitative evaluation of design choices. Although the appendices discuss several architectural alternatives and development decisions, the paper does not provide a systematic quantitative ablation study of its central components or hyperparameters. Consequently, it is difficult to determine how much performance comes from the compression objective, the specialized architecture, or the optimization and output-selection heuristics. In particular, comparisons with reconstruction-only training and a simpler decoder would help establish the contributions of KL regularization and architectural engineering.

  1. Limited characterization of successful and unsuccessful puzzles. The objective of Section 5.1, “What Puzzles Can’t We Solve?”, is only partially fulfilled. The section and Appendix H identify observed limitations through selected examples, but do not provide a systematic quantitative (and statistical) characterization of the unsolved puzzles. In particular, they report neither the number of puzzles requiring each capability nor the success rate within each category. It would therefore be more appropriate to frame this discussion as a preliminary attempt to identify the method’s limitations. Equally important is the complementary question: “What puzzles can we solve?” Although Appendix H lists observed capabilities and the paper analyzes selected successful cases, these do not establish which properties characterize the solved evaluation puzzles. It remains unclear whether the reported accuracy primarily reflects a narrow set of relatively simple, architecture-supported patterns or extends to more complex combinations of operations. A broader quantitative analysis of both successes and failures, supplemented by controlled puzzle variations, would clarify the scope of the method’s generalization and help distinguish architectural limitations from optimization failures.
Questions for authors

1.Could the authors provide a quantitative analysis of solved and unsolved puzzles grouped by required operations or structural properties? Reporting category sizes and success rates would help clarify which patterns the method captures reliably and whether its successes are concentrated in a limited set of puzzle types. Controlled variations in factors such as translation distance or the number of repeated operations would also help identify capability boundaries.

2.Which methodological and architectural choices contribute most to the final accuracy? In particular, could the authors compare the full method with reconstruction-only training, vary the KL weight, and evaluate a simpler decoder or remove selected specialized operations? Reporting the resulting changes in evaluation accuracy across multiple random runs would help establish the importance and robustness of these choices.

3 comments

We sincerely thank the reviewer for their constructive criticism of our paper. We address the concerns listed in individual responses below:

1.Insufficient quantitative evaluation of design choices. Although the appendices discuss several architectural alternatives and development decisions, the paper does not provide a systematic quantitative ablation study of its central components or hyperparameters. Consequently, it is difficult to determine how much performance comes from the compression objective, the specialized architecture, or the optimization and output-selection heuristics. In particular, comparisons with reconstruction-only training and a simpler decoder would help establish the contributions of KL regularization and architectural engineering.

We agree that the attribution of performance to various components is unclear as presented - therefore we ran several experiments to clear this up. We have now included these ablations in Section 5 and uploaded an updated version of the paper, with changes to the paper marked in blue. We are also still running further analyses, whose results we will post here before including the changes to the paper marked in magenta. (We have shortened some Figures near the bottom to stay within the page limit.)

Firstly, we swept the relative contribution of the KL coefficient as opposed to reconstruction, resulting in the following:

KL/reconstruction loss coefficient ratioEvaluation score (pass@2)
00.375%
0.0117.5%
0.0320.25%
0.1 (CompressARC default)21.33% (+/- 0.94%)
0.223%
0.416.5%
0.616.38%
0.815.38%
1.011.88%
The sweep shows that the KL regularization is absolutely necessary in order to achieve our performance, and reconstruction-only does not suffice.

We also tried freezing a random z value (sampled from N(0,1) i.i.d. at the beginning of training) while training purely on reconstruction to isolate the effect of the architecture alone without the compression. This resulted in a score of 1.0%, supporting that architecture alone without compression is not responsible for our performance.

We are currently running experiments which test the effect of architectural engineering and solution sampling heuristics by successively the most heavily engineered components and seeing how the score changes and what classes/types of puzzles are lost. The relative KL contribution is fixed to 0.1 and the output selection heuristic is our original one for these experiments. We will post the results here in a follow-up once this is done running.

  1. Limited characterization of successful and unsuccessful puzzles. The objective of Section 5.1, “What Puzzles Can’t We Solve?”, is only partially fulfilled. The section and Appendix H identify observed limitations through selected examples, but do not provide a systematic quantitative (and statistical) characterization of the unsolved puzzles. In particular, they report neither the number of puzzles requiring each capability nor the success rate within each category. It would therefore be more appropriate to frame this discussion as a preliminary attempt to identify the method’s limitations. Equally important is the complementary question: “What puzzles can we solve?” Although Appendix H lists observed capabilities and the paper analyzes selected successful cases, these do not establish which properties characterize the solved evaluation puzzles. It remains unclear whether the reported accuracy primarily reflects a narrow set of relatively simple, architecture-supported patterns or extends to more complex combinations of operations. A broader quantitative analysis of both successes and failures, supplemented by controlled puzzle variations, would clarify the scope of the method’s generalization and help distinguish architectural limitations from optimization failures.

Within the coming days, we plan to provide a correlative analysis relating whether or not CompressARC achieves the correct solution against what operations the solution consists of, based on the Hodel DSL solution set at https://github.com/michaelhodel/arc-dsl/blob/main/solvers.py. This is a publicly available code base which contains handwritten solutions to all 400 training puzzles, within a domain-specific language built for ARC-AGI.

We have now completed the rest of the experiments as promised and we would like to present the results to you.

Firstly, we show the isolated effect of removing architectural components and heuristics in our method, while leaving the compression aspects intact. The additional results are in a new section of Table 1. We successively remove architectural components and re-run to see how the score differs as CompressARC gets successively simpler. The more the score decreases with removed components, the more responsible the architecture is for the performance as opposed to compression. The table:

ModificationEvaluation pass@2
CompressARC (original)21.33% +/- 0.94%
minus symmetrical weight tying (Appendix E.2)13.5%
minus directional communication layer (Appendix D.5)16.25%
minus shift, cummax, and softmax layers (Appendix D.4)7.75%
minus solution selection heuristics*7.25%

*When removing solution selection heuristics, we score by selecting the two most frequent solutions gathered during the training run.

Unfortunately the sequential ablation did not have a monotonic effect on the score as we had hoped. Nevertheless, we hope the moderate but not absolute decrease in score serves as evidence that the architecture is not fully responsible for the score, and the compression indeed plays a major role. At the very least, the stacked architectural and heuristic ablations had much less of a detrimental effect on the score as did freezing zz or disabling the KL loss term, which were modifications designed to disable compression without damaging the architecture.

And secondly, we have also added a quantitative analysis of the abilities of CompressARC to perform various operations involved in solutions, by using Michael Hodel's domain-specific language solutions as a source of ground truth on how to solve each training set puzzle. The analysis is in Appendix H, and includes two large Tables 5 and 6, which can be found in the updated version of the paper. The analysis matches the listing of abilities and inabilities of CompressARC we had previously included in the original version of our paper, with the main exception of cropping, which the new analysis shows to be an inability. CompressARC is shown to be most capable of working with bounding regions, rendering on backgrounds, and constructing lines and rays, and is least capable of splitting and joining shapes, locating copies of a shape, building variable-sized output grids, and counting and arithmetic. These mentioned items each impose a greater than 14% differential effect on the ability of CompressARC to solve a training set puzzle, which is substantial compared to CompressARC's overall training set score of 34.75%.

Please let us know if you have more questions. We would like to hear your thoughts on the additional evidence we have given for CompressARC, and whether you would consider raising your score.

The quantitative analysis added in Appendix H is valuable, and I would encourage the authors to prioritize its main findings in the main text. Currently, a substantial portion of the main text is devoted to a detailed case study of a single puzzle. While this illustrates the model’s behavior, its representativeness is unclear, and the extensive space allocated to it provides less insight into the method’s general capabilities and limitations than a systematic analysis across puzzles would. I therefore suggest substantially shortening the case study and using the space to present which puzzle types are solved reliably and which remain challenging, supported by category-level success rates, sample sizes, and uncertainty, while clearly acknowledging the analysis’s limitations.

The additional experiments constitute a substantial effort and address several of my original concerns, particularly regarding the importance of KL regularization. However, the architectural ablations remove components successively and therefore do not isolate each component’s contribution to the full model. Moreover, the ablations beyond the default configuration use single seeds, leaving the robustness of the observed differences unclear. The capability analysis also remains limited to training puzzles, which informed the architecture’s design, and reports associations between operation presence and success rather than establishing isolated capabilities or explaining performance on complex combinations of operations. These limitations should be reflected more clearly in the interpretation of the results.

Addressing these concerns would benefit from further validation, including independent component ablations across multiple seeds and a capability analysis on the public evaluation set, together with a substantial revision of the main text to foreground the systematic findings. In my assessment, this work would benefit from further development before publication. I encourage the authors to build on these promising results and strengthen the evidence and presentation for a future submission.

Reviewer_9muxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CompressARC, which solves ARC-AGI puzzles using a 76K-parameter heavily engineered multitensor network. This network is not pretrained and is trained from scratch only on a specific Puzzle. For this purpose, CompressARC samples a puzzle-dimension-sized random multitensor from a normal distribution (where the mean and variance are learnable), which is used as input to the decoder. The decoder uses this input and is (over)fitted to generate the specific puzzle as output using cross-entropy as the learning signal, while using Kullback–Leibler (KL) divergence on the learned normal distribution to the normal distribution with mean 0 and Variance 1. The influence of the test output on the loss is masked out and the prediction is whatever the decoder produces at the masked-out region. The KL part of the loss penalizes the information content of the input to the decoder. According to the authors, this is justified by Occam’s razor, according to which the shortest description of a puzzle should produce the most reasonable completion of the prediction. They show that CompressARC solves a non-trivial 20% (pass@2) of the eval puzzle. While below the 32% of other methods without prior training, this represents a new compression framework.

Critical strengths
  1. This work seems to use the insights of Hinton and Zemel (1994, "Autoencoders, Minimum Description Length and Helmholtz Free Energy") for ARC without training data, which seems to be novel. While this ignores \theta and therefore the Occam argument is not completely true, the non-trivial 20% performance raises the question of where the generalization is coming from.

  2. The CompressARC (as shown in Figures 5 and 6) provides a straightforward method to analyse the mechanism applied (Figures 5 and 6), which, together with Section 5.1, helps to understand which patterns CompressARC cannot solve.

Critical weaknesses
  1. The Decoder has around 76K parameters whose description length is not part of the optimized objective. Even at one nat per parameter, this dwarfs the ~100 nats a non-collapsed tensor (in the multitensor) has after training. This makes the question the non-trivial performance of this approach raises not less relevant. However, it makes the authors' Occam’s-razor argument non-applicable, or at least much less applicable. A partial explanation for the capability of CompressARC could come from the heavy engineering which, according to Ulyanov et al. (2020, "Deep Image Prior"), suffices to bias a generator network sufficiently to capture low-level image statistics. An indicator for this is what is discussed in 5.1, which shows that patterns the heavily engineered decoder is not built for cannot be solved. This could be a second dynamic making CompressARC work, and there are no ablations on this. The performance of a no-KL baseline with fixed z (like Ulyanov et al.) could explain much of the performance. The general statement of MDL being the determining factor of the performance is currently not supported.

  2. CompressARC is heavily engineered. This makes it unclear how well CompressARC will work on other tasks and, more relevantly, how well the observed dynamics transfer to other tasks/architectures. As discussed in point 1), this engineering could be the driving factor of the performance, in which case the paper would offer little insight beyond ARC and, as seen in 5.1, CompressARC fails on puzzles requiring abilities it was not built for. There seems to be no ablation on the Decoder architecture and input z structure. It is therefore unclear whether CompressARC dynamics only hold for this heavily engineered decoder (favoring the explanation from Ulyanov et al. discussed in point 1) or if less "specialized" Decoders have similar dynamics, making the insights a more general dynamic than just on this specific architecture and setting.

Questions for authors
  1. What is the pass@2 accuracy if we fix the z and use no KL loss on ARC-AGI-1? This would indicate how much of the performance is due to the architecture vs. the MDL motivation.

  2. What is the variance of pass@2 over multiple seeds on ARC-AGI-1, e.g., error bands on Figure 3?

  3. What kind of puzzles are still solvable when removing part of the multi-tensor (e.g., all tensors having direction as a dimension)? Did you try out other architectures that also had non-trivial results?

  4. How often does one of the 4 necessary tensors collapse?

5 comments

We sincerely thank the reviewer for their constructive criticism of our paper. As most of the concerns listed are centralized around the same theme, we address them all together with a suite of new experiments we ran for this discussion period and to include in an updated version of the paper. (To compensate, we have shortened some Figures near the bottom to stay within the page limit.)

We agree that the non-standard architecture is a large confounding factor when studying the effect of compression on the solution rate. While the quantity of model parameter information dwarfs the non-collapsed multitensor information, information in model parameters is only able to symmetrically affect the model behavior, meaning any asymmetry in the puzzle across any of the network equivariances is forced to come from the non-collapsed multitensor information. In this sense, we are only able to control and compress asymmetric information, but there are so many symmetries we engineered the architecture for that we reach our score despite this issue.

To address concerns about where to attribute performance to, we introduce a suite of ablations/analyses to study the relative contribution between compression and architecture to the solution. We have now included these ablations in Section 5 and uploaded an updated version of the paper, with changes to the paper marked in blue. Additionally, we have certain analyses still running, which we will post about here before applying further updates to the paper in magenta as well.

Firstly, we ran two extra seeds while logging solution rates and compressive losses, both resulting in pass@2 accuracy of 22%, so the headline score is now 21.33% with a standard deviation of 0.94% over the total of three seeds including the original. Across puzzles which were only solved by one of the two seeds, the KL loss was lower slightly more often for the successfully-solving seed than the failing one, with the effect most pronounced with pass@1 near 1500 steps of training. This supports that the compression-based loss is a good proxy for solution correctness. We have updated the paper with the following table showing the empirical frequency that correct runs attain better training metrics than incorrect runs.

IterationsPass@nKL is lowerReconstruction is lowerTotal loss is lower
400-500113/30 (43.3%)15/30 (50.0%)14/30 (46.7%)
400-500214/32 (43.8%)17/32 (53.1%)14/32 (43.8%)
900-1000118/30 (60.0%)12/30 (40.0%)16/30 (53.3%)
900-1000222/32 (68.8%)15/32 (46.9%)18/32 (56.2%)
1400-1500121/30 (70.0%)18/30 (60.0%)19/30 (63.3%)
1400-1500220/32 (62.5%)18/32 (56.2%)17/32 (53.1%)
1900-2000117/30 (56.7%)12/30 (40.0%)14/30 (46.7%)
1900-2000217/32 (53.1%)13/32 (40.6%)12/32 (37.5%)

We also updated all affected tables, figures, and mentioned numbers with results averaged over the three seeds.

Secondly, we ran a KL loss coefficient sweep to show that the role of compression on zz is necessary.

KL/reconstruction loss coefficient ratioEvaluation score (pass@2)
00.375%
0.0117.5%
0.0320.25%
0.1 (CompressARC default)21.33% (+/- 0.94%)
0.223%
0.416.5%
0.616.38%
0.815.38%
1.011.88%

We also performed a run with random fixed zz and observed that pass@2 accuracy worsens to 1.0%, which shows that the learnability of the latent information content is essential, and the architecture's engineered inductive biases alone are not enough.

We are currently running experiments which test the effect of architectural engineering by successively removing layers of our architecture and seeing how the score changes. The relative KL contribution is fixed to 0.1 and the output selection heuristic is our original one for these experiments. We will post the results here as well once this is done running. Changing out the multitensor structure is difficult in our setup, so we opted to remove layers and output selection heuristics as a similar mode of ablation, which we hope should give similarly informative results to address your concern.

If time permits, we will also re-run the "Color the Boxes" puzzle 100 times to observe the rate of tensor collapse.

Lastly, on architecture complexity: we confirm that our results on the effect of compression on solution quality hold specifically for this architecture and this dataset format. This compression-based approach cannot transfer to other domains without re-designing a new architecture for every new dataset. We will also refer you to our response to Reviewer gNLD where we argue it is quite standard for ARC-AGI methods to use benchmark-specific hand-engineered complex components, albeit this usually takes the form of data engineering (which we opt out of) rather than architecture engineering (which we do instead).

Thank you for the response and the additional experiments. They address some of my concerns, but others remain open.

(W1) I appreciate the symmetry argument. However, it does not hold completely: the paper notes that rotation equivariance holds only approximately (Appendix E.2), and layers such as cummax break geometric symmetries. A bigger discussion on this argument and its limits should be included in the paper.

(W2) I would suggest softening the broad claim, made in the abstract and the discussion, that MDL is an alternative way to produce intelligence. The symmetry argument above is closer to what is actually shown, and it is interesting in itself.

(Q1) Thank you for running this experiment. In a future revision, please also report the accuracy over training steps for the fixed-z run and the number of seeds used.

(Q2) Answered, thank you.

(Table 2) When comparing two seeds, a loss that carries no information about correctness would pick the correct run 50% of the time. I am therefore not convinced by the statement that the “compression-based loss is a good proxy for solution correctness”. In the final window (1900–2000 steps), the correct run has the lower total loss in only 14/30 and 12/32 cases, which is below chance. The strongest results are in the KL column, but even in the two best cells, an exact two-sided binomial test against 50% gives p ≈ 0.04 (21/30) and p ≈ 0.05 (22/32), before any correction for the 24 comparisons in the table. After such a correction, neither result is statistically significant.

(Table 1) Only the default configuration has a ± value, so I assume the other runs used a single seed. In a revised version, it would be helpful to report ± values for these as well.

The architecture ablations for (W2), (Q3) and (Q4) are still open. I look forward to the results the authors announced and will take them into account in my final assessment.

AuthorsOpenReview ↗

Further Ablations and Experiments (1/2)

Thank you for considering our ablations so far. We have now completed the remaining experiments requested and would like to share them with you. We will also address each comment in your reply as well.

On the writing:

(W1) I appreciate the symmetry argument. However, it does not hold completely: the paper notes that rotation equivariance holds only approximately (Appendix E.2), and layers such as cummax break geometric symmetries. A bigger discussion on this argument and its limits should be included in the paper.

We have decided to include a footnote where we first mention neglecting to count bits in the weights. We write:

To prevent the optimal strategy from being to move all necessary information from the seeds into the weights, we construct the architecture to be equivariant to as many puzzle transformations as possible. This ensures that the information in the weights can only affect symmetrical information, whereas any asymmetric information in the puzzle is forced to be stored in the seeds, and so is guaranteed to be captured properly despite ignoring weight information content.

On the breakage of symmetry, we tried to keep the breakage as minimal and reasonable as possible while we were engineering the architecture. For the cummax and shift layers, the breakage of translation symmetry is only due to the presence of the edge of the grid, otherwise the operation is equivariant to translations. It is no more equivariant than convolutions are in a CNN, which are sensitive to edges in the same way. If instead you are concerned about the rotational and flip equivariances, these equivariances are preserved in the cummax and shift layers, via interactions with the direction axis of the multitensor.

(W2) I would suggest softening the broad claim, made in the abstract and the discussion, that MDL is an alternative way to produce intelligence. The symmetry argument above is closer to what is actually shown, and it is interesting in itself.

We have taken your suggestion, and updated the abstract to say "feasible alternative way to solve IQ-test-like puzzles, besides conventional massive pretraining" rather than "feasible alternative way to produce intelligence, besides conventional massive pretraining". We have also removed the last paragraph of the discussion, which was too extravagant.

(Q1) Thank you for running this experiment. In a future revision, please also report the accuracy over training steps for the fixed-z run and the number of seeds used.

There was one seed used, and the final score of 1% amounts to 4 puzzles being solved, which makes the curve for accuracy over training steps incredibly noisy relative to it's own scale. We doubt that mentioning any of these mid-training accuracies can give any insight at all, so we felt we should not include it in the paper, though we are open to hearing any reason it may still be helpful to include.

(Table 2) When comparing two seeds,...

This is quite a fair assessment, we agree these results are not statistically significant.

(Table 1) Only the default configuration has a ±\pm value,...

Yes, the other runs used a single seed. We will report ±\pm values should we run more seeds.

Reviewer_gNLDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CompressARC, a pretraining-free neural solver that optimizes its weights and latent representations from scratch on each individual ARC puzzle. By pairing a compression-driven objective with an architecture hardwired for ARC symmetries and geometric operations, it achieves 20% pass@2 on the 400 public ARC-AGI-1 evaluation puzzles and offers insights into how the resulting latents represent puzzle structure.

Critical strengths
  • The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.
  • The paper explores an interesting alternative to pretrained neural solvers.
  • The architecture provides a concrete way to incorporate useful structural priors. The multitensor representation and symmetry-aware operations allow information to be shared across examples, colors, and spatial dimensions.
Critical weaknesses
  • The setting removes pretrained weights but retains substantial benchmark-specific knowledge. ARC training puzzles are used to design the architecture, choose coefficients, and tune answer selection. This is a valid setting, but it does not demonstrate learning from the target puzzle alone in the broader sense that the knowledge from other puzzles has already been encoded through manual design. The paper should distinguish without pretraining from without prior benchmark information.

  • Because CompressARC pairs its compression loss with a heavily tailored architecture and tuned post-processing heuristics, omitting a pure reconstruction-loss baseline makes it impossible to isolate the actual benefit of the MDL formulation. The authors should sweep the KL weight to show whether compression is genuinely essential or if the gains come primarily from the architectural inductive biases.

  • Although the method relies on stochastic latent sampling, the main evaluation only reports single-seed results. The authors should report the mean and standard deviation across multiple seeds to verify how reliably the method achieves the headline 20% score.

  • While the case studies suggest that lower KL aligns with better predictions, aggregate pass@2 actually drops from 21.5% at 1,500 steps to 20% at 2,000 steps. Instead of assuming more compression is always better, the authors should systematically test whether lower loss values actually correlate with correct solutions across checkpoints and random seeds.

Questions for authors
  • How well does the same architecture perform without the KL term? Does it still solve a substantial fraction of puzzles?
  • Does lower description-length loss reliably correspond to more accurate solutions, especially when comparing different runs of the same puzzle?
3 comments

We thank the Reviewer for providing their constructive criticism of our paper. We have addressed individual concerns about our paper below.

Because CompressARC pairs its compression loss with a heavily tailored architecture and tuned post-processing heuristics, omitting a pure reconstruction-loss baseline makes it impossible to isolate the actual benefit of the MDL formulation. The authors should sweep the KL weight to show whether compression is genuinely essential or if the gains come primarily from the architectural inductive biases.

Thank you for the suggestion. Here is a sweep of the KL weight.

KL/reconstruction loss coefficient ratioEvaluation score (pass@2)
00.375%
0.0117.5%
0.0320.25%
0.1 (CompressARC default)21.33% (+/- 0.94%)
0.223%
0.416.5%
0.616.38%
0.815.38%
1.011.88%

It shows that the KL loss component is necessary to solve puzzles. We have now included this ablation in Section 5 and uploaded an updated version of the paper, with changes to the paper marked in blue. (Further proposed changes are in magenta - we will discuss them here once we are done evaluating their experiments and before including them. We have shortened some Figures near the bottom due to the page limit.)

The setting removes pretrained weights but retains substantial benchmark-specific knowledge. ARC training puzzles are used to design the architecture, choose coefficients, and tune answer selection. This is a valid setting, but it does not demonstrate learning from the target puzzle alone in the broader sense that the knowledge from other puzzles has already been encoded through manual design. The paper should distinguish without pretraining from without prior benchmark information.

Indeed, we use a lot of prior benchmark information to design our network architecture, as a substitute for knowledge gained through pretraining. We have updated the top of the Architecture Section 4 (see edits in blue) to add: "In this way, we have engineered prior information about the benchmark into the structure of the architecture, as a substitute for learning the same information through pretraining or through engineered synthetic data and augmentations."

We should contextualize on this point that while we do not engineer the data, other methods in the Related Work Section 2 heavily hand-engineer the data. It is standard for ARC-AGI-1 methods to use hand-built synthetic puzzles, data augmentations picked on a per-puzzle basis, and even custom data generators using domain-specific languages extensively hand-fitted to the training set. Other methods can therefore hardly be considered to have little to no prior benchmark information either, and our prior information just shows up in the architecture engineering rather than the data engineering.

While the case studies suggest that lower KL aligns with better predictions, aggregate pass@2 actually drops from 21.5% at 1,500 steps to 20% at 2,000 steps. Instead of assuming more compression is always better, the authors should systematically test whether lower loss values actually correlate with correct solutions across checkpoints and random seeds.

By re-running with two extra seeds and filtering for puzzles that only get solved with one seed, we find that on average over many puzzles, the seed with the correct solution has a slightly higher empirical frequency of having a lower KL loss component. The effect is most pronounced with pass@1 evaluation near 1500 steps of training. We have updated the paper with the following table showing that correct runs generally attain better training KL than incorrect runs.

IterationsPass@nKL on correct run is lowerReconstruction on correct run is lowerTotal loss on correct run is lower
400-500113/30 (43.3%)15/30 (50.0%)14/30 (46.7%)
400-500214/32 (43.8%)17/32 (53.1%)14/32 (43.8%)
900-1000118/30 (60.0%)12/30 (40.0%)16/30 (53.3%)
900-1000222/32 (68.8%)15/32 (46.9%)18/32 (56.2%)
1400-1500121/30 (70.0%)18/30 (60.0%)19/30 (63.3%)
1400-1500220/32 (62.5%)18/32 (56.2%)17/32 (53.1%)
1900-2000117/30 (56.7%)12/30 (40.0%)14/30 (46.7%)
1900-2000217/32 (53.1%)13/32 (40.6%)12/32 (37.5%)

Although the method relies on stochastic latent sampling, the main evaluation only reports single-seed results. The authors should report the mean and standard deviation across multiple seeds to verify how reliably the method achieves the headline 20% score.

We have now updated all relevant tables and graphs in the paper with standard deviations using the two extra seeds mentioned above, which both attained a score of 22%. The headline 20% score is now updated to 21.33% with std 0.94%.

Answers to questions listed are already presented in the rest of our rebuttal above.

AuthorsOpenReview ↗

Further Ablations

Dear Reviewer,

We have completed some further ablations to show the isolated effect of removing architectural components and heuristics in our method, while keeping the compression aspects. The main additional results are in Table 1, where we successively remove architectural components and re-run to see how the score differs as CompressARC gets successively simpler. The more the score decreases with removed components, the more responsible the architecture is for the performance as opposed to compression. The table:

ModificationEvaluation pass@2
CompressARC (original)21.33% +/- 0.94%
minus symmetrical weight tying (Appendix E.2)13.5%
minus directional communication layer (Appendix D.5)16.25%
minus shift, cummax, and softmax layers (Appendix D.4)7.75%
minus solution selection heuristics*7.25%

*When removing solution selection heuristics, we score by selecting the two most frequent solutions gathered during the training run.

We have also added a quantitative analysis of the abilities of CompressARC to perform various operations involved in solutions, by using Michael Hodel's domain-specific language solutions as a source of ground truth on how to solve each training set puzzle. The analysis is in Appendix H, and includes two large Tables 5 and 6, which can be found in the updated version of the paper. The analysis matches the listing of abilities and inabilities of CompressARC we had previously included in the original version of our paper, with the main exception of cropping, which the new analysis shows to be an inability. CompressARC is shown to be most capable of working with bounding regions, rendering on backgrounds, and constructing lines and rays, and is least capable of splitting and joining shapes, locating copies of a shape, building variable-sized output grids, and counting and arithmetic.

Please let us know if you have more questions. We would like to hear your thoughts on the additional evidence we have given for CompressARC, and whether you would consider raising your score.

Thanks for your response. I would maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.