acceptodds
Under review as a conference paper at ICLR 2027

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

Abstract

The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether success reflects inference of the intended underlying rule or reliance on shortcuts. We address this gap by studying test-time task embeddings in Vision ARC (VARC), a model in which a pre-trained backbone is complemented by a trainable embedding representing the transformation rule. In VARC, test-time training (TTT) is jointly applied to the backbone and task embedding. Here we introduce a novel two-step TTT protocol: first finetune only the task embedding (Embed-TTT), then freeze it and finetune the backbone. Across ARC-AGI-1, ConceptARC, and two controlled datasets with known rules, Embed-TTT consistently yields improved task embeddings, ones that align better with underlying task rules, improve embedding-based retrieval, and enable accurate linear probing of known rules. Qualitatively, Embed-TTT identifies more semantically meaningful relations between test and train tasks. We also show that optimizing only task embeddings (less than 0.01% of model parameters) already solves a non-trivial fraction of ARC-AGI-1, ConceptARC and Mini-ARC tasks, while the full two-step pipeline improves final performance. Finally, we show that Embed-TTT recovers the underlying geometric structure of parametric rules and learns compositional capabilities that enable rule interpolation, but not extrapolation. These findings support a clearer separation between rule induction and rule execution in ARC-like evaluations, motivating benchmarks that better distinguish in-distribution from out-of-distribution rules.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
ARC tasks, VARC, rule induction, test-time training, fine-tuning, OOD generalization, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8BdtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers the Vision Abstraction and Reasoning Corpus (VARC) setting, where the goal is to learn the underlying rule from a few input-output demonstrations and then make predictions on test data. This work builds on a prior framework that uses a trainable task embedding and a neural network for prediction. In practical settings, the task embedding and the neural network weights are first fine-tuned based on the given demonstrations, and predictions are then made on the test data. The authors improve the training procedure for the overall network architecture. The basic idea is that the task embedding and the neural network parameters should be fine-tuned separately. The authors argue that this allows the task embedding to better capture the patterns in the demonstrations, thereby yielding better prediction results. The paper designs multiple experiments to show that the task embedding obtained in this way indeed better reflects the similarity among demonstration patterns than before, and that the overall fine-tuning performance is better than that of the previous training method.

Critical strengths

The motivation and proposed method of this paper are very clean and easy to understand. The training method is also fairly general: it can be applied to different backbone models and is easy to implement. Although training the task embedding and the backbone model separately is not new and has been used in many areas, the authors conduct a relatively extensive set of experiments to examine the relationship between the separately trained task embedding and the patterns in the demonstrations. Overall, this is an interesting piece of work.

Critical weaknesses
  • Regarding the writing, the paper is very clean before Section 4, but Section 4 is somewhat disorganized. There is no opening paragraph at the beginning of Section 4 to explain the overall organization, and some parts read awkwardly. For example, at the beginning of Section 4.1, after “We conduct the following experiments,” the text seems to introduce both the experiments and the datasets, which is somewhat confusing.

  • As I mentioned in the Strengths, the proposed method can be applied to different backbone models. Although the paper conducts many experiments, it does not test the performance of different backbone models under the proposed training strategy. The paper would be better if it tested different backbone models and showed that the current conclusions are consistent across them.

Questions for authors

Have you tested other backbone models? Does the Embed-Full method also achieve performance improvements over the original Full method?

2 comments
AuthorsOpenReview ↗

Answers to weaknesses and questions

We thank the reviewer for the overall positive and supporting review. We reply to weaknesses and questions hereafter.

Weakness:

  • Section 4 writing. We thank the reviewer for this remark and added a roadmap at the start of Section 4:

"We first evaluate how well test-time embeddings recover known rules in Section 4.1. We then examine the training tasks retrieved for ARC-1 test tasks to show the embeddings capture meaningful similarities in Section 4.2. Next, we compare performance across datasets in Section 4.3. Finally, in Section 4.4, we study the geometry of the embedding space to see where task generalization succeeds."

We also revised the opening of Section 4.1 to explain the retrieval and probing evaluations before introducing the datasets and their splits:

"Datasets and experimental details. We use two types of evaluation, according to Section 3.3. On ARC-1 and ConceptARC, we test whether an embedding fitted to a duplicate of a training task retrieves that task's original embedding. On ConceptARC, LUCD, and Moves, we use linear probes to measure whether the embeddings encode concept labels, rule labels, or displacement coordinates. The dataset descriptions below specify how we construct the training and evaluation sets."

  • Other architectures. We agree that verifying our results with various model architectures is valuable and we are currently running experiments with TRM, FPRM, LoopViT. Consequently, we provide an expanded version of Tables 1 and 2 in Appendix E and G, see Tables 3 and 4. Some results are still pending and will be updated as soon as we have them. Those expanded tables confirm the observations made about Tables 1 and 2 (modest but consistent improvements, especially with a matched finetuning budget for the backbone with the recursive models; non-negligible fraction of tasks of other datasets solved by Embed-TTT only; better rule induction metrics with Embed-TTT than with Full-TTT). We agree that modest gains from a single seed need careful interpretation, and we believe that we have been careful (see e.g. line 369 in the revised version, and the new paragraph lines 410-416 where we repeat that gains are modest). We underline that the consistency of our results throughout the paper (for VARC, and now across backbones) gives some degree of trust in their general validity. Let us also note that the gains in Table 2 were not our primary target but are mostly fortunate observations. Appendix A also details how these other models were adapted to enable our Embed-TTT process, which is not limited to VARC. We are happy to provide confidence intervals through additional runs after the rebuttal period, as we are currently dedicating our compute resources at the alternative backbone experiments.

Questions:

  • Other architectures. We provided our answer above, for the mentioned weakness.

Thanks to the authors for their response. I will maintain my initial positive rating.

Reviewer_enr7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a test-time training approach that first optimizes task embeddings (Embed-TTT) and then optionally fine-tunes the backbone (Embed-Full-TTT) for ARC-like tasks. Experiments show improved task representations and performance, along with an analysis of compositional generalization.

Critical strengths
  1. The paper addresses an interesting question by distinguishing rule induction from rule execution.

  2. The proposed approach is supported by experiments across multiple ARC-like datasets. The experiments also provide useful insights into the structure and compositional properties of the task embeddings.

Critical weaknesses
  1. The empirical validation of the proposed protocol is limited to VARC. Although the authors explain this choice in terms of computational cost, its applicability to other architectures and task-embedding designs remains unclear.
Questions for authors
  1. To what extent does Embed-TTT rely on the specific task embedding design of VARC? Could the approach be applied to other model architectures?

  2. The experimental results suggest that the embeddings encode rule-related information. To what extent do these findings support interpreting the embeddings as representations of human-interpretable abstract rules?

2 comments
AuthorsOpenReview ↗

Answers to weaknesses and questions

We thank the reviewer for the overall positive and supporting review. We reply to weaknesses and questions hereafter.

Weakness:

  • Other architectures. We agree that verifying our results with various model architectures is valuable and we are currently running experiments with TRM, FPRM and LoopViT. Consequently, we provide an expanded version of Tables 1 and 2 in Appendix E and G, see Tables 3 and 4. Some results are still pending and will be updated as soon as we have them. Those expanded tables confirm the observations made about Tables 1 and 2 (modest but consistent improvements, especially with a matched finetuning budget for the backbone with the recursive models; non-negligible fraction of tasks of other datasets solved by Embed-TTT only; better rule induction metrics with Embed-TTT than with Full-TTT). We agree that modest gains from a single seed need careful interpretation, and we believe that we have been careful (see e.g. line 369 in the revised version, and the new paragraph lines 410-416 where we repeat that gains are modest). We underline that the consistency of our results throughout the paper (for VARC, and now across backbones) gives some degree of trust in their general validity. Let us also note that the gains in Table 2 were not our primary target but are mostly fortunate observations. Appendix A also details how these other models were adapted to enable our Embed-TTT process, which is not limited to VARC.

Questions:

  • Other architectures. We provided our answer above, for the mentioned weakness.

  • Embeddings give human-interpretable abstract rules? This is indeed an interesting question. Our results do show that embeddings seem to encode information and group in ways that suggest at least part of the embeddings encode human-interpretable rule aspects. Another explicit visualization of this is Figure 12 of Hu et al.'s VARC paper. However, it is not straightforward how to perform the translation from embeddings into the natural language domain in an automated manner, without having access to textual descriptions of some training tasks. We believe this is a very interesting avenue for future work.

Thank you for the additional experiments and clarifications. I maintain my original rating.

Reviewer_xaMGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses an 18M vision transformer model called Vision ARC (VARC) that is trained on the ARC dataset. This model uses task-specific tokens to represent a task. In the default setting (Full-TTT), the entire model is trained end-to-end, optimizing the backbone and the task-specific tokens together. During test-time training, the tokens remain near their random initialization, meaning they are far from the task distribution learned during training. Therefore, the task tokens provide little information to the model, suggesting that the backbone has to carry this information instead. Beyond this observation, the authors provide alternative training strategies, focusing on making these task-specific tokens useful to the model during TTT. First, they train only the embeddings while the model remains frozen, ensuring the task-tokens get updated towards the representations that are most useful to solve the task during test-time. This method moves the test time task embeddings much closer to their training distribution. This model is called the Embed-TTT. Embed-TTT can optionally be extended by a subsequent step where the embeddings are frozen and the rest of the model gets fine-tuned (called Embed-Full-TTT).

These two new model versions are first analyzed to determine whether they produce more task-aligned task embeddings. This is done with linear probes and by checking whether a training task fitted at test time retrieves the original task embeddings. These checks confirm that the task embeddings are much more aligned with the task if they are trained separately with a frozen backbone (Embed-TTT) than if both the backbone and encodings are trained together (Full-TTT). However, while the task embeddings are more aligned with the task, the overall performance on ARC is considerably worse if only the embeddings are trained. If the backbone is then fine-tuned in a subsequent step (Embed-Full-TTT), the model trained in these two steps outperforms the baseline slightly on ARC-1 (e.g., pass@1 +0.7 p.p.) and even more clearly on ConceptARC and Mini-ARC.

Critical strengths

The paper has one very clear research focus, namely the task embeddings during TTT of a vision transformer. First, it is shown that the standard setting does not fully leverage such task embeddings and then a general solution is proposed: only train the embeddings first with the rest frozen, then freeze the embeddings and train the rest. Overall, the build up is very clear and intuitive, going from showing an existing problem to providing a solution. It is interesting that Embed-TTT with only the embeddings trained achieves ~13-17% on ARC-1, hinting at the importance of these embeddings and some overlap between training and testing tasks.

The authors further discuss the task embeddings with more intuitive control datasets. Specifically, they show that on the Moves datasets, the embeddings represent the grid of move-actions and that performance improves a lot if aligned moves are present but fail for composing moves if no compositions are seen during training. Appendix F also discusses composed tasks, showing that Embed-TTT retrieves the components correctly but cannot solve them while Full-TTT can solve them without retrieving the components. This suggests that identifying the rule (the task) and executing it may dissociate (if the alternative explanations below can be ruled out), which is very interesting and should even be discussed in the main part.

While I did not reproduce any results, the reported numbers look consistent and plausible.

Critical weaknesses

Novelty: Existing work already covers many of the ideas in this paper. For example, Macfarlane & Bonnet "Searching Latent Program Spaces" do gradient search during TTT on small task-like embeddings. There are many other related works such as Zintgraf et al. "Fast Context Adaptation via Meta-Learning" also updates only small context parameters via meta-learning, and Autodecoders such as DeepSDF by JJ Park et al. freeze the model during inference and only fine-tune a small task vector. Also, in other applications such as prompt tuning in general it is similar to freezing the model and fine-tuning only one task vector. The related work does not position the paper well in the context of this existing literature. It should be clearer what was already there before and what the novelty is, which is probably mainly the task-specific evaluation on Arc.

Experiments and Drawn Conclusions: While the numbers themselves look correct, the interpretation of them needs to be more careful. For instance, Table 1 shows that freezing the backbone and only fine-tuning the task embeddings makes those embeddings more aligned with the training distribution (which is to some extent expected). However, the used baseline is Full-TTT and this one trains both, the task embeddings and the backbone. Thus, the embeddings only carry meaning for its also fine-tuned backbone. Therefore, the comparison to the original training embeddings is expected to fail and yield the wrong measure, as an updated backbone likely shifts the distribution of the task embeddings. The authors mention this issue briefly and also report results where the probe is fitted on duplicated training tasks of the same Full-TTT protocol instead of the original training embeddings obtained after initial training. This comparison is more meaningful and should be shown in the main table (Table 1) and should be discussed. Focusing primarily on the first experiments as the main result is misleading, and statements like "loses any semantically meaningful information" are overclaimed in a general sense.

Also, it remains unclear how general these findings are since only a single model is used. Also, the claim that good ARC performance conflates rule induction and rule execution is only measured indirectly on ARC (Section 3.2 says it could be memorization) while the appendix provides much stronger evidence based on synthetic data.

Alternative Explanations: Some of the shown interpretations might be correct, but there are also alternative solutions that are not ruled out. Specifically, in Section 4.4, combined moves are compared with single-step movements (unit move). The comparison is done with cosine similarity, which only considers the direction from the origin, not the magnitude of these moves. To provide some intuition: the moves "up 1", "up 2", and "up 3" could be arranged on a perfect line with same distance between them but the line is not passing through the origin. That would mean that they represent distance well but cosine similarity could still be low. In such a case, R3-U2 is expected to be more similar to R3-U0 than to R1-U0 as it was observed in the paper. Thus, it cannot be concluded that the model cannot handle scaling.

Another example is the compositionality in Appendix F: The network is trained to move and change seperately and during inference it should compose these two actions. The network fails on this compositionality task, even though the embedding is close to the two relevant tasks. The authors interpret this phenomen that the correct task embedding was found but the model failed to execute it. A much simpler alternative explanation is that this is just a side effect of the loss and the embedding drifts kind of towards the middle of the two relevant embeddings without any understanding of it. A simple analogy just for understanding: We could have a color yellow and color blue learned during training and if both are done together, it drifts somewhere in the middle of these two embeddings representing green. But this is just the closest position in embedding space between the two, it does not mean we understood the concept of colors nor of green. These alternative explanations could simple be ruled out with control tasks (e.g. use two training tasks where the target overlaps pixel-wise but where it is not a composition of the rules -> if the embeddings are still between the two, they reflect an overlap not rule induction).

Presentation & General comments:

  • The structure is not optimal and methods, motivation, and results are blurred. E.g., the problem statement is defined inside related work, Section 3 refers to preliminary results as motivation but the details of it remain unclear
  • At some places, equations could make things a bit clearer, e.g., for the objective and the metrics
  • For concept transfer: Since it is up to 3 concepts, chance is up to 3/16=19% and not 1/16=6.3%
  • Moves should also have chance level reported. Why did you not do it for this method but all the others? It is very hard to interpret what an RMSE of 2.x means.
  • Table 1: It could be interesting to have a very simple baseline without TTT such as using a linear probe with the average input-output distance to identify the rule -> such a simple method could already work on the synthetic dataset and make the 100% probe accuracy less informative
  • The results seem to rest on a single model run once with a single seed. Since the gain is relatively small (about 3 tasks at pass@1), we need confidence intervals.
  • It is not clear whether the compute is matched between the runs or if the two-stage run is just trained longer
  • Description of training details, hyperparameters, etc. are missing and should be in the appendix
Questions for authors
  • Which model produced which result? I.e. it should be known what the training data size, training budget, whether it loaded from a checkpoint or from scratch, etc.
  • Is the Full-TTT baseline fully tuned? Can you report how you tuned the baselines? And combine it with confidence intervals and different seeds? Also, it reaches 69.4% on ConceptARC for retrieval and 6.5% on ARC-1 which is quite a gap. What causes this difference?
  • For a better interpretation of results, could you add another version of models where you do the same initialization as Embed-TTT and then fine-tune the entire model (embeddings + backbone)
  • What is the compute budget provided for each model? Is the model trained in two steps overall trained longer (more updates) than the model that is trained end-to-end?
  • For each model, how do the metrics change if the task embeddings are completely removed / set to 0?
6 comments
AuthorsOpenReview ↗

Answers to weaknesses 1 and 2

We thank the reviewer for the very informative review and for the suggestions that we believe greatly improved the quality of the manuscript. We provide our answers to the weaknesses, general comments, and questions hereafter.

Weakness 1 - Novelty

Thank you for pointing out these connections. As mentioned in our paper (originally lines 69, 475), our Embed-TTT protocol is novel for VARC in the context of ARC-like tasks. However, we acknowledge that our phrasing could be seen as misleading, as we did not invent the test-time adaptation protocol in general, so we have changed those lines accordingly. We also agree that our literature positioning was incomplete in that regard. In the revised version, we have explicitly acknowledged the established protocol ("Finetuning task embeddings" paragraph in the Related Work) and clarified that our main contribution concerns how test-time adaptation affects rule induction in VARC task embeddings, including properties like compositionality, task retrieval and recovering similarities.

Added paragraph:

"Finetuning task embeddings. Optimizing a latent conditioning vector while keeping a shared model fixed is an established approach, explored in context adaptation methods such as CAVIA (Zintgraf et al.) and auto-decoder models such as DeepSDF (Park et al.). Similarly, the field of prompt tuning (Lester et al.) shares this setup, where the embedding vector representing the natural language input is adjusted, while keeping a processing language model fixed. However, while soft-prompt tuning also focuses on fine-tuning of a small set of vectors (optionally at test-time), this approach does not necessarily result in meaningful representations of the corresponding natural language tasks. More directly related, LPN (MacFarlane and Bonnet) applies latent program search to ARC, primarily investigating its effectiveness for task solving while also examining latent structure and composition. Our work shares the idea of this adaptation principle but focuses on how different test-time training protocols preserve or disrupt induced rule information in VARC task embeddings. Through retrieval, linear probing of known rule labels and parameters, and controlled geometric and compositional experiments on ARC-like tasks, our novelties are the in-depth investigations of implicit rule induction, the task representation space and its relationship to successful rule execution."

We also rearranged the related work section to include a paragraph on other backbones (TRM, FPRM, LoopViT), which we now use in our experiments, as explained below.

Weakness 2 - Experiments and Drawn Conclusions

We agree that we compacted results too much and that we can reinforce our claims with our existing results. Regarding the more meaningful comparison (previously lines 248-249), having a task generator indeed makes our aligned setup more fair. Hence, we now show these results in Table 1 and state in the text that this is the default protocol for LUCD and Moves, while the previous results are mentioned in the text for informative purposes. We have expanded Table 1 in the Appendix E (Table 3) to include the other model backbones. So far, our observations about the large gap between Full-TTT and Embed-TTT hold for TRM, FPRM, LoopViT, with some results still pending (these need substantial model training on the datasets of interest, which takes several GPU days and more resources than the finetuning). We will update results as soon as we have them available. Similarly, our interesting results showing that VARC, despite being strong on ARC-1, does not learn simple LUCD or Moves embeddings (previously in Appendix D), and which thus teases some interpolation capabilities between known tasks rather than OOD generalization, are now in the main text in Section 4.3. We also reinforced this observation with other models, namely TRM and FPRM, while LoopViT results will be added upon completion.

AuthorsOpenReview ↗

Answer to weakness 3

Weakness 3 - Alternative Explanations

Thank you for pointing out these alternative explanations. We agree that cosine similarity alone cannot establish a lack of scaling capabilities, as movement magnitudes could be encoded linearly despite the observed similarities. Indeed, Figure 4 and previous Appendix D already suggest that the embeddings capture movement magnitudes and preserve the underlying geometric structure. Our observation is that composed moves are more closely aligned with their already-scaled atomic constituents than with unit moves, suggesting a preference for recombination of these atomic representations. Furthermore, as discussed at the end of Section 4.4 (and now in Appendix J), the model struggles to execute compositions of movements, such as combining +3 and +1 into +4. We have therefore slightly clarified the corresponding interpretation in Section 4.4 as follows:

"The cosine similarities with "unit" transforms (in the example case R1-U0 and R0-U1) are much weaker, suggesting that composed embeddings are more closely aligned with their already-scaled atomic constituents. However, this does not necessarily indicate a lack of scaling capabilities, as cosine similarity alone cannot establish whether movement magnitudes are encoded in the embedding space. Indeed, the geometric structure observed in Figure 4 suggests that the embeddings preserve information about movement magnitudes, even if the model struggles to compositionally execute these representations."

We also agree with the reviewer that proximity to the constituent embeddings does not necessarily establish successful rule induction. However, the fact that the correct color and movement are identified as the closest respective constituents in 100% and 99.5% of cases suggests that the embeddings capture more than arbitrary similarity. Nevertheless, we acknowledge that this could partly reflect similarities in the underlying input-output structure rather than an understanding of the underlying rules. Indeed, what exactly constitutes an "understanding" of a new rule, or of a new color in the reviewer's example, is difficult to establish. We generally argue that the fact that the model projects a task to a position from which the correct constituents can be directly identified suggests that rule induction is at least partially successful. Assuming a sufficiently capable backbone or "executor", the composition could, in principle, be realized by simply executing the two closest constituent rules. While we recognize the reviewer's alternative explanation, we believe that these findings support our interpretation of a dissociation between rule induction and rule execution. We have nevertheless qualified our interpretation in Appendix J (in the revised paper) to acknowledge this alternative explanation:

"While the observed proximity to the correct constituents could partly reflect similarities in the underlying input-output structure rather than rule induction alone, these results suggest that the bottleneck is primarily executability, not task-level rule identification."

AuthorsOpenReview ↗

Answers to general comments

  • Structure. We agree about the adjustments of the structure. In our updated manuscript we have moved the Problem Statement into the methodology setting and the motivation through preliminary results into the Related Work section.

  • Equations. We are happy to add more detailed formal definitions of our metrics, which are now present in Appendix B.

  • Concept transfer percentage. We thank the reviewer for pointing out this error. For the probability of random matches in Concepts (previously line 400), it is worth noting that the annotation of three is an upper boundary, where the average annotation count is 1.6 per task. After recalculation, concept retrieval by chance is about 9.8% on the test subset and 8.2% on the train subset, which we have adjusted in our manuscript and does not change our claim that our results are far above pure chance.

  • Chance level for Moves. Thank you for pointing out this omission. We have added an uninformed mean-prediction baseline for Moves: always predicting the mean training displacement, (0,0), yields an RMSE of 3.16.

  • Table 1: Very simple baseline. We are happy to add baselines that could be informative. However, unless we misunderstand the comment, we believe that having a dedicated architecture or baseline specifically engineered to solve Moves or LUCD seems to defeat the purpose of measuring capabilities of general ARC-task solving models. Hence, we would prefer refraining from doing so, unless such a baseline brings a significant value that we do not yet see.

  • Single model, single seed, small improvements, intervals. We agree that verifying our results with various model architectures is valuable and we are currently running experiments with TRM, FPRM, LoopViT. Consequently, we provide an expanded version of Tables 1 and 2 in Appendix E and G, see Tables 3 and 4. Some results are still pending and will be updated as soon as we have them. Those expanded tables confirm the observations made about Tables 1 and 2 (modest but consistent improvements, especially with a matched finetuning budget for the backbone with the recursive models; non-negligible fraction of tasks of other datasets solved by Embed-TTT only; better rule induction metrics with Embed-TTT than with Full-TTT). We agree that modest gains from a single seed need careful interpretation, and we believe that we have been careful (see e.g. line 369 in the revised version, and the new paragraph lines 410-416 where we repeat that gains are modest). We underline that the consistency of our results throughout the paper (for VARC, and now across backbones) gives some degree of trust in their validity, without yet having confidence intervals. Let us also note that the gains in Table 2 were not our primary target but are mostly fortunate observations. We are happy to perform the additional experiments after the rebuttal to include confidence intervals, as we are currently devoting our resources to replicating our results across other backbones.

  • Compute matched. To ensure that a longer training does not confound our results, we used the same number of epochs and thus of training samples in all cases. Hence, compute is matched in that regard. For the two-stage approach, half of the epochs served to finetune the task embedding, then the other half for the backbone. This is now clarified in Appendix A with the training details. Incidentally, the backbone finetuning, which drives most of the final performance, receives only half of the updates of the Full-TTT case, yet achieves slightly better results. However, since the backbone finetuning carries most of the performance gain at test time, we also ran new experiments with a doubled budget for Embed-Full-TTT, so that the backbone has the same training budget as Full-TTT. Interestingly, the performance of TRM and FPRM mostly improve, and tend to slightly outperform more often Full-TTT. We reported those extra results in the expanded version of Table 2 (see Table 4 in Appendix G).

  • Training details. Appendix A now documents the training data, checkpoints, hyperparameters, and adaptation budgets. We adapted TRM and FPRM to follow VARC's inductive protocol by restricting offline training to ARC training tasks and adapting each evaluation task independently using only its own demonstrations. We reduced the standard TTT budget for both models to approximately match VARC's number of demonstration-pair presentations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Expanded tables with more backbones

We agree with the reviewers that our results would be reinforced if confirmed by other backbone models. As said in our replies, we have been running our experiments with TRM, FPRM, and LoopViT. The expanded tables of results (Tables 3 and 4 in the Appendix E and G of the revised manuscript) are provided hereafter. We will complete them as more results become available. As also commented, our results mostly hold with these backbones.

Table 1 expanded. Quantitative implicit rule induction across backbones. We evaluate task embeddings from Full-TTT and Embed-Full-TTT using training-task retrieval, rule classification, and displacement regression. For the LUCD and Moves probes, we adapt the training and evaluation embeddings using the same TTT method.

Train datasetTrain/probeEvaluation and metricBackboneChanceFull-TTTEmbed-Full-TTT
ARC-1 train400/sameEmb. retrieval (train), top-5 match ↑VARC1.3%6.50%86.25%
TRM42.25%86.75%
FPRM14.00%40.50%
LoopViT9.00%73.25%
ConceptARC160/sameEmb. retrieval (train), top-5 match ↑VARC3.1%69.4%98.8%
TRM29.4%55.0%
FPRM43.1%66.9%
LoopViTPendingPending
ConceptARC160/sameLin. classif. (train), accuracy ↑VARC6.2%21.3%60.6%
TRM15%36.9%
FPRM27.5%50.6%
LoopViTPendingPending
Custom LUCD15/150Lin. classif. (test), accuracy ↑VARC6.7%39.2%100%
TRM12.67%35.33%
FPRM36.67%64.00%
LoopViT31.0%92.7%
Custom Moves121/1210Lin. regress. (test), RMSE ↓VARC3.161.270.44
TRM3.263.02
FPRM2.621.65
LoopViTPendingPending

Bold marks the better TTT score within each completed row. Pending indicates that results are not yet available.

 

 

Table 2 expanded. Cross-dataset results across backbones. We compare Full-TTT, Embed-TTT, and Embed-Full-TTT on ARC-1, ConceptARC, and Mini-ARC using ARC-trained checkpoints. Strict task pass@k requires every held-out pair in a task to be correct within the top k predictions. The ConceptARC pair-level block scores each held-out pair individually. Embed-Full-TTT improves over Full-TTT in several settings, with gains varying by backbone and dataset. All scores are percentages.

Strict task-level evaluation

Test datasetBackbonePass@1 FullPass@1 EmbedPass@1 E-FPass@1 E-F 2×│Pass@2 FullPass@2 EmbedPass@2 E-FPass@2 E-F 2×
ARC-1 testVARC (paper)48.813.049.550.5│53.817.255.255.5
TRM27.006.2523.5026.00│31.508.0027.7529.25
FPRM20.754.2518.7521.75│25.005.7521.5026.50
LoopViT58.2513.7558.00Pending│62.5017.7562.75Pending
ConceptARCVARC (paper)23.88.828.123.8│32.514.437.533.13
TRM15.004.3817.5017.50│22.5010.0021.2523.75
FPRM10.634.3811.2510.00│18.758.1316.8819.38
LoopViT23.757.5028.75Pending│33.1310.6336.25Pending
Mini-ARCVARC (paper)59.722.263.863.8│62.434.971.167.8
TRM51.6818.1252.3556.38│57.0526.1757.7261.07
FPRM46.3112.7548.3251.01│51.6817.4553.6956.38
LoopViT63.7613.4263.76Pending│69.1322.8269.80Pending

ConceptARC pair-level evaluation (480 held-out pairs)

BackbonePass@1 FullPass@1 EmbedPass@1 E-FPass@1 E-F 2×│Pass@2 FullPass@2 EmbedPass@2 E-FPass@2 E-F 2×
VARC (paper)47.124.049.848.1│53.833.357.156.7
TRM36.0418.9635.8337.08│44.3824.5841.4643.75
FPRM30.2113.5429.1729.58│39.1719.1739.5840.42
LoopViT48.7522.7153.13Pending│56.6729.5861.46Pending

VARC concept retrieval in ARC (Embed-TTT). Top-1 accuracy is 36.3% on test tasks (29/80) and 35.0% on training tasks (28/80).

E-F denotes Embed-Full-TTT. E-F 2× uses an extended budget with a backbone phase sized to the Full-TTT budget. The published VARC Mini-ARC E-F run already used a similar total number of native passes. Pending marks unfinished extended-budget runs.

AuthorsOpenReview ↗

Summary of rebuttal responses and manuscript revisions

Dear AC, dear Reviewers,

We thank the Reviewers for their feedback. Here is a summary of the improvements made during the rebuttal period.

Mainly, the three reviewers rightfully asked for additional experiments with other model backbones. We tested with 3 others (TRM, FPRM, LoopViT) and provided result tables during the discussion and in our appendix. The new results support the existing ones and strengthen the paper, as acknowledged explicitly by Reviewer xaMG.

Reviewer 8Bdt also had two comments about adding introductory paragraphs to Sect. 4 and 4.1, which we have done. Reviewer enr7 had an open question on embedding interpretability, which we answered. Both Reviewers acknowledged our answers and maintained their positive score.

Reviewer xaMG had many minor or fixable concerns (as Soundness was rated 2) about the paper. We addressed all of them in detail in the discussion and in the paper where appropriate, in addition to the results with extra backbones. Notably, they required a useful additional paragraph in the Related Work section, a more conservative interpretation of some results, a slightly different paper structure in sections 2 and 3, and clarifications about technical details. Reviewer xaMG then acknowledged that the changes improved the paper a lot and that some results were more convincing, while maintaining their score and pointing to other concerns.

We again addressed all of them with more conservative phrasings where relevant, by highlighting the consistency of our results throughout the paper and across the backbones despite single-seed experiments (while committing to run multiple seeds when our resources allow us to do so), answering the novelty question once more, and running the requested control experiment. At the close of the discussion period, we had received no further response.

We hope that the additional experiments, the detailed answers in the discussions and the changes in the revised manuscript will convince the AC and the Reviewers to consider our paper fit for publication at ICLR, and wish them a fruitful meta-reviewing period.

Have a nice day!

Best regards,

The Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.