acceptodds
Under review as a conference paper at ICLR 2027

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

Abstract

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks that measure such systems have several limitations: they generally focus on a single applied domain, they serve each instance only once, and most assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge and which the models under test solve only in part. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for the demonstrations or submit an answer, with retries permitted on an incorrect submission. Each instance is scored based on the share of this feedback budget (demonstrations and failed attempts) consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. We run three agent harnesses with frozen weights, each with and without persistence, comparing them with two prompting floors and a program-induction method. We also define a second regime in which one system learns from several users simultaneously. Across our experiments, we show that persistence in the harness does generally save on budget, but only modestly. None of them perform measurably better than a floor that simply caches each task's demonstrations. In contrast, a simple method that asks the LLM for one program per task and keeps following demo reproduction outperforms all harnesses by a wide margin. This opens the door to future harness design improvements, and to weight-based continual learning approaches that could teach models to effectively drive harnesses in new environments.

Primary area:
datasets and benchmarks
Keywords:
continual learning, lifelong learning, benchmark, ARC, ARC-AGI, abstraction and reasoning, skill acquisition, test-time training, reinforcement learning, agent harnesses, memory, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFmEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Currently, AI agents are being used in multiple scenarios, from small tasks that run in minutes to large tasks that require interaction between multiple agents and take hours or days to finish. To work, these AI agents use their internal knowledge (weights), but are complemented by their harness and skills, which are tools that extend their capabilities and allow them to perform procedures, like internet search or using programming languages. Even though these agents are everywhere, few benchmarks exist to test their ability to accumulate knowledge and reuse it to solve previous and new tasks. This paper presents a new benchmark based on the ARC benchmark to test the capability of knowledge retention and accumulation of AI agents. Experiments with different harnesses and approaches show that there is no clear advantage of using persistent memory when comparing budget and performance, which raises the question of how to better accommodate harnesses in continual scenarios.

Critical strengths

The paper provides a compelling rationale for a relevant and increasingly popular problem (AI agents). Identifying the limitations of existing benchmarks and how these can be mitigated by the new proposal could have a great impact on the community. This paper raises questions such as: When might a skill acquired in the past be relevant to future tasks? Which information is relevant to keep? These questions can help identify when a model is learning and/or reusing information, thereby helping to identify and mitigate forgetting. The work also raises questions about backward and forward transfer, highlighting that models can reuse information. This is undoubtedly a benchmark that could be of real value to the community.

Critical weaknesses

A benchmark is presented that promises to mitigate multiple shortcomings of previous benchmarks; however, these contributions are not clearly evident in the benchmark’s design or in the experiments conducted.

  • The paper discusses forward and backward transfer, but there is no analysis of how much these are evident in the experiments. What types of experiments reflect these values, and how can we mitigate the problem?
  • The paper discusses skill accumulation, but the experiments refer to observations. Is there a relationship between the two?
  • The tables and images from the experiments are not referenced in the text, which makes it difficult to establish a connection between the two.
  • The relationship between the three components of the benchmark (Sections 3.1, 3.2 and 3.3) and the experiments conducted is not very clear.
  • The experiments show the budget, but little is said about the ability to retain information or transfer information to future tasks.
Questions for authors
  • The paragraph beginning on line 155 discusses some parameters. Is this an example, or are these the hyperparameters of a task? Can they be modified to make the problem easier or harder? I would imagine that some agents would need more demonstrations to find the answer, but they might generalise better to different types of tasks.
  • How do the results change when the parameters described in Equations 2 and 3 are modified? When presenting a benchmark, it is important to identify its strengths and weaknesses when the problem is made easier or more difficult.
  • A visual representation of Sections 3.1, 3.2, and 3.3 would help better understand the proposal.
2 comments
AuthorsOpenReview ↗

Response to Reviewer oFmE (1/2)

Thank you for such a careful review, and for seeing the value this benchmark could have for the community if presented clearly. We agree that the contributions were clear in neither the design nor the experiments. We've restructured the paper around this feedback, which hopefully now makes for a smoother read. Some of your points overlap, so we've grouped those together below.

The paper discusses forward and backward transfer, but there is no analysis of how much these are evident in the experiments. What types of experiments reflect these values, and how can we mitigate the problem?

The experiments show the budget, but little is said about the ability to retain information or transfer information to future tasks.

We agree, and the revision now measures both directly. After the stream, every learner is frozen and plays two tests, set out in Section 2.3. The held-out test serves 100 tasks the learner never met, so it measures forward transfer; the seen test brings back every task of the stream, so it measures the backward side, what is left of earlier tasks after everything learned since, which is where forgetting would show. Section 2.4 scores each as the share of its reset's spend that the learner saves through learning, and Section 4 reports both of these for every learner in Table 1 and Figures 3 and 5. Figure 4 also follows the learners over their visits to a task during the stream. In short, the harnesses retained little unless given expert guidance, and the program library retained the most but transferred nothing measurable on the held-out test. Only training the weights transferred to new tasks. That is also where we see the most promise for mitigating the problem, which Section 6 discusses.

The paper discusses skill accumulation, but the experiments refer to observations. Is there a relationship between the two?

Again, thank you for pointing this out. Observations here are what a learner sees on each instance: the task id, the input grid, and whatever demonstrations and feedback it pays for. Accumulating skill means turning these into something that solves later instances with fewer observations, whether of the same task, which we call memorization, or of tasks never met, which we call generalization. The budget connects the two: every demonstration and wrong answer costs one, so skill shows up as observations a learner no longer needs to pay for. We therefore measure learning by the budget saved, irrespective of the form of the learning, whether notes in a harness, programs or weights. The demo-cache floor draws the line between the two: it simply keeps the demonstrations it observed and shows them again, so a learner that saves more than it must have learned something beyond the stored observations. Keeping a task's rule once deduced, for example, pays off on the seen test, while only abstract skills can carry over to new tasks, which is what the held-out test measures. Appendix E shows what each harness actually stored.

The tables and images from the experiments are not referenced in the text, which makes it difficult to establish a connection between the two.

Thank you for flagging it. Every table and figure in the revision is now cited where its result is discussed.

AuthorsOpenReview ↗

Response to Reviewer oFmE (2/2)

The relationship between the three components of the benchmark (Sections 3.1, 3.2 and 3.3) and the experiments conducted is not very clear.

A visual representation of Sections 3.1, 3.2, and 3.3 would help better understand the proposal.

Agreed. The benchmark section is now built around these three parts: one instance in Section 2.1, drawn in Figure 1; the stream in Section 2.2, with Figure 2 now drawing the whole ARC-AGI-1 stream; and the two tests after it in Section 2.3. Every result in Section 4 is then read on the stream and on both tests.

The paragraph beginning on line 155 discusses some parameters. Is this an example, or are these the hyperparameters of a task? Can they be modified to make the problem easier or harder? I would imagine that some agents would need more demonstrations to find the answer, but they might generalise better to different types of tasks.

To answer your question directly, those values are the benchmark's settings, the same for every learner so that results can be compared. The revision has updated some of them: an instance now allows up to eight demonstrations, and an unsolved instance is charged the whole budget of 16, as set out in Section 2.1. Since demonstrations are charged on the stream but generalization is scored separately on the held-out test, an agent that buys more demonstrations (costing some performance on the stream) but generalizes better (performs better on the held-out test) would show up as exactly that. Your question also prompted us to run the demonstration and attempt caps at 4 and 16 in place of 8, as part of the sweep below - thanks!

How do the results change when the parameters described in Equations 2 and 3 are modified? When presenting a benchmark, it is important to identify its strengths and weaknesses when the problem is made easier or more difficult.

We agree this is important to show. Appendix B now reports a first side experiment on the order, in which GPT-6 Luna's systems also played the same visits one task at a time and fully shuffled, and the trends held in every order. We are now also sweeping the stream's parameters, which are all in Equation 2 of the revision, on four of the methods: the number of tasks active at once, how often new tasks arrive and how long tasks rest, along with the order and the caps mentioned above. We will post the results here as they land, and add them to the appendix.

Hopefully these changes address some of your concerns, and we're very happy to discuss anything further! :)

Reviewer_rJZjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper introduces Continual-ARC, a benchmark for testing whether agents can learn and remember skills over time. tasks come back with new inputs, and agents are scored by how much feedback they need. experiments on three agent harnesses show that their memory systems provide limited benefits, while a simple program library works much better.

Critical strengths
  1. the benchmark design is interesting, especially having tasks return with new inputs instead of just repeating the same examples.
  2. the negative results on existing agent harnesses are useful. i also like the simple demo-cache baseline, which helps show whether agents are actually learning anything beyond storing examples.
Critical weaknesses
  1. a major motivation is to compare memory-based and weight-based continual learning under the same evaluation protocol, but no weight-based learning methods are actually tested. all models have frozen weights. the collective learning setting is also only proposed without experiments. so i dont think the paper fully validates some of its main claims about the benchmark.

  2. im not fully convinced the benchmark captures general skill acquisition. agents are given task IDs, and a large part of the evaluation is about remembering rules for previously seen tasks. the program library does this very well, but the paper finds no clear forward transfer to unseen tasks. this makes the connection to broader continual skill learning somewhat unclear.

Questions for authors
  1. can you include at least one weight-based continual learning baseline? i think this would be important given the motivation of the benchmark.
  2. how would you distinguish general skill acquisition from simply storing and reusing task-specific rules? can you better demonstrate what the benchmark tells us about transferable skills?

im open to raising this to a 3 if the authors can clearly address these concerns, especially the missing weight-based evaluation and the scope of skill acquisition being measured.

2 comments
AuthorsOpenReview ↗

Response to Reviewer rJZj

Thank you for a clear and constructive review, and for being open to reconsidering. Your two questions followed from your two concerns, so we've paired them together below, which should hopefully make it easy to follow.

a major motivation is to compare memory-based and weight-based continual learning under the same evaluation protocol, but no weight-based learning methods are actually tested. all models have frozen weights. the collective learning setting is also only proposed without experiments.

can you include at least one weight-based continual learning baseline?

Agreed, and both are now in the paper. The revision trains Qwen3.5-9B's weights with DAgger: between rounds, a LoRA adapter learns to imitate GLM-5.3 Flash's reply in each state the student met, in the same harness, under the same rules and on the same score as every other learner (Section 3 and Appendix H). DAgger also runs in the collective regime, with two and four users each bringing 25 tasks of their own (Section 2.5, and "More users" in Section 4). More users, and so more games, led to better generalization on the held-out test, though DAgger’s own gains seemingly levelled off at four users (Appendix H). We also tried a policy gradient on each game's outcome and self-distillation from the teacher's move, but neither learned reliably (Appendix H).

im not fully convinced the benchmark captures general skill acquisition. agents are given task IDs, and a large part of the evaluation is about remembering rules for previously seen tasks. the program library does this very well, but the paper finds no clear forward transfer to unseen tasks.

how would you distinguish general skill acquisition from simply storing and reusing task-specific rules? can you better demonstrate what the benchmark tells us about transferable skills?

Totally agree, and the revision now separates the two with two frozen tests after the stream (Section 2.3). The seen test brings back the same tasks seen in the stream, where storing a task's rule pays off, while the held-out test serves 100 unseen tasks. Each is scored on its own (SseenS_{\mathrm{seen}} and SheldS_{\mathrm{held}}, Section 2.4). Task ids remain, under an alias drawn for each run (Section 2.1), since a task's return is the setting being tested in the seen test, but on the held-out test they cannot help. The program library is a case in point: it saves the most on the seen test, but nothing measurable on the held-out test. Despite ARC-AGI-1's tasks all being built from one library of shared building blocks, no harness method generalized measurably. Training the weights did generalize: DAgger saved Sheld=0.29S_{\mathrm{held}} = 0.29 [0.23, 0.35] of its reset's spend at one user and 0.40 at two (Section 4), and trained once from the base model on four users' games, the student saved 0.46 [0.40, 0.53] (Appendix H).

Hopefully this goes some way towards addressing both of your concerns, and we're happy to discuss anything further if helpful! :)

AuthorsOpenReview ↗

An update for Reviewer rJZj

Just a short update on your two concerns, as our second revision adds to both since our reply above.

What the benchmark says about transferable skills. Appendix E now has a control that puts every file the teacher saved straight into the student's prompt. The student then memorizes about as well as the trained weights do (0.64 [0.55, 0.72] on returning tasks, against 0.57 [0.48, 0.67]), but still saves nothing measurable on unseen tasks (0.02 [−0.03, 0.07], against 0.29 [0.23, 0.35]). So with the same teacher's knowledge on both sides, the benchmark distinguishes between stored, task-specific rules and a skill that transfers.

The collective regime. Appendix H now trains two and four users' games the same way after the stream, and four users' games generalize further: 0.46 on unseen tasks against 0.39, a gap of 0.08 [0.03, 0.13].

Do let us know if either still falls short, we'd be very happy to add more :)

Reviewer_gNhTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a continual version of the ARC AGI benchmark for evaluating systems for continual learning whether they are weight-based or memory-based. In the experiments, self-updating harnesses generally fail to show significant value over prompt based approaches.

Critical strengths
  • The topic is very interesting and timely
  • It is cool to see agent harnesses assessed in this way
  • Adapting the ARC benchmark for continual learning is a sensible test-bed for these approaches
  • The results provide a sobering analysis of the kind of meta-learning happening within harnesses
Critical weaknesses
  • I am worried about the size of the contribution of this paper as ARC is an existing benchmark and the contribution of changing the order of task presentation to the agents is relatively minor.
  • There aren't big experimental conclusions to draw from this paper, most experiments are not statistically significant.
  • While it is somewhat interesting that memory based approaches do not do well in this setting, given that ARC is made to be resistant to the value of prior knowledge, it almost feels like an adversarial task for assessing these approaches. It is sobering about what is being learned from a meta-learning perspective, yet it still may be the case that these approaches.
  • In the abstract, the authors emphasize the ability for this benchmark to compare memory-based approaches to weight-based approaches, but the experiments do not deliver on this promise.
  • The authors admit that AI "drafted the abstract and every section of this paper from the author’s notes." This was extremely apparent. The writing quality of this paper is horrible by the standards of this conference. The organization of the narrative and ideas makes it very difficult for readers to digest. This is a topic I am actually very interested in and the writing made it a real struggle to read.
Questions for authors
  • Q1: Did you run any experiments with weight-based models? It is oddly really emphasized in the abstract and then dropped form the discussion later.
  • Q2: Did you consider running a larger number of trials to highlight statistical significance more clearly?
  • Q3: Do the authors believe that these harness-based approaches are not effective in general, or are these conclusions specific to the ARC tasks?
1 comment
AuthorsOpenReview ↗

Response to Reviewer gNhT

Thanks for the review, and for the points you've raised, much appreciated. We've taken them to heart, and since three of your questions follow from your concerns, we've paired those together below.

I am worried about the size of the contribution of this paper as ARC is an existing benchmark and the contribution of changing the order of task presentation to the agents is relatively minor.

That's fair: we haven't built a benchmark from scratch, or collected new data from a new domain. Continual-ARC is assembled from existing parts, ARC's tasks and Re-ARC's generators. Where we'd respectfully push back is on novelty being the measure of value. As far as we know, no other benchmark can expose this weak spot: harnesses built to learn on the job that gain nothing measurable from what they keep, scored on the same scale as a model learning in its weights (Section 5). ARC is what makes this possible. Its tasks need no domain knowledge, so any saving has to come from learning on the stream, and fresh task instances mean nothing can be cached. The revision also adds two frozen tests separating memorization from generalization, a private set of 50 new rules for verifying external results (which few benchmarks have), and two harder tracks on ARC-AGI-2 and ARC-AGI-3.

There aren't big experimental conclusions to draw from this paper, most experiments are not statistically significant.

Did you consider running a larger number of trials to highlight statistical significance more clearly?

We agree, and we did. Each test is now played four times by the learner and four times by its reset, with 95% intervals over task resamples, as Appendix A describes. Several conclusions clear them comfortably: the program library and the expert-guided harnesses memorize, with GLM-5.3 Flash's guided harness saving 0.74 [0.64, 0.84] of its reset's spend on the seen test, and only training the weights generalizes, saving 0.29 [0.23, 0.35] on tasks never seen. Where a method saves nothing, the intervals are now tight enough to say so too; the harness on Qwen3.5-9B saves −0.01 [−0.04, 0.03] on the seen test, for example. We are also playing four of the methods' streams twice more from scratch, so that each is learned three times rather than once, and will post the results here as they land!

While it is somewhat interesting that memory based approaches do not do well in this setting, given that ARC is made to be resistant to the value of prior knowledge, it almost feels like an adversarial task for assessing these approaches. [...]

Do the authors believe that these harness-based approaches are not effective in general, or are these conclusions specific to the ARC tasks?

Agreed, ARC does resist knowledge brought in from outside, but within the stream the same tasks keep returning, so what a learner stores along the way is exactly what pays off. Memory does pay here when it keeps the right thing: the program library, which keeps one verified program per task, saves the most on the seen test, and given simple expert guidance the same harnesses do also memorize, saving 0.62 to 0.72 of the reset's spend with GPT-6 Luna. As shipped, they failed in what they chose to store, even though every session's instructions told them that tasks recur, that each instance starts a new conversation and that every demonstration and wrong answer costs one. Appendix E traces what each harness stored. To answer your question directly, we think the default harness prompts were not well suited to tell the models we tested what to store in this domain; how they fare elsewhere remains open, and more powerful models may fare better, as Section 6 discusses.

In the abstract, the authors emphasize the ability for this benchmark to compare memory-based approaches to weight-based approaches, but the experiments do not deliver on this promise.

Did you run any experiments with weight-based models? It is oddly really emphasized in the abstract and then dropped form the discussion later.

Agreed, and thanks for pushing on this. The revision trains Qwen3.5-9B's weights with DAgger, imitating GLM-5.3 Flash in the same harness, at one, two and four users (Section 3 and Appendix H). It is the only learner that generalized measurably, and the comparison with memory now runs throughout the results and into the discussion in Section 6.

The authors admit that AI "drafted the abstract and every section of this paper from the author's notes." This was extremely apparent. [...]

Thank you for persevering with it... An AI assistant did help draft the submission, and we have since rewritten the main text line by line, in our own words, as the revised AI-use statement now says. If it still reads as badly, then the fault is now entirely ours, not the AI's! We have also rebuilt the narrative around the two questions the benchmark answers, memorization and generalization.

We hope this addresses your concerns, and we're happy to discuss anything further :)

Comments on the paper

8 comments
AuthorsOpenReview ↗

Summary of the revision

Thank you all for the careful and generous reviews, very much appreciated. There were two main points that came up across the reviews: the submission promised to compare memory and weights but trained no weights, and it did not show what the benchmark measured beyond remembering tasks it had previously seen. The revision should hopefully address both of these:

  • Weights. Qwen3.5-9B now learns in its weights with DAgger, imitating GLM-5.3 Flash in the same harness and under the same rules as every other learner, at one, two and four users (Section 3, Section 4 and Appendix H).
  • Memorization against generalization. After the stream every learner is frozen and plays two tests, the stream's own tasks and 100 tasks it never met, each scored as the share of its own reset's spend it no longer pays for (Sections 2.3 and 2.4). Only training the weights measurably generalizes to unseen tasks.
  • Statistics. Each test is played four times by the learner and four times by its reset, with 95% intervals over task resamples, simultaneous across learners in the figures (Appendix A).
  • Presentation. The paper is rewritten and restructured. Every table and figure is cited where its result is discussed. Figure 2 now draws the whole stream, and Figure 3 shows the headline results.

We are also running the further experiments Reviewers oFmE and gNhT asked for, a sweep of the benchmark’s parameters and more trials, and will post the results here as they land.

AuthorsOpenReview ↗

Summary of the second revision

We've uploaded a second revision, which covers the experiments we promised in our first responses. The main text is as it was in the first revision apart from the two changes under Reviewer gNhT below. The additions are in the appendix, and each answers a point from your reviews or follow-ups.

Reviewer oFmE, on how the stream's parameters and the caps move the results: Appendix B now has the sweep (Figure 7 and Table 4), over nine streams and four settings of the demonstration and attempt caps. Every method still saves on returning tasks in all thirteen, the harness as shipped least, and none gains consistently on unseen tasks.

Reviewer gNhT, on more trials: four methods' streams were played three times from scratch (Table 5 in Appendix B). Over the three runs the harness as shipped on GLM-5.3 Flash saved 0.25 [0.17, 0.33] on returning tasks and 0.10 [0.03, 0.17] on unseen ones. So Section 4 now gives it over three runs, and the abstract and introduction say that harness-based methods gain little or nothing on unseen tasks, where they said none gains. On separating knowledge from the harness: Appendix E adds two controls that put the teacher's knowledge straight into the student's prompt, as a letter and as every file the teacher saved.

Reviewer rJZj, on transferable skill and the collective regime: Appendix H now trains two and four users' games the same way after the stream, and four users' games generalize further (0.46 against 0.39). On the stream itself the two runs saved 0.40 and 0.39, as before.

For completeness, the other changes: Appendix F hands GLM-5.3 Flash the stream's library of primitives, which it rarely used; Appendix A cites a published result that programs in the same primitives solve evaluation tasks; Appendix K has a pilot of two more methods on the ARC-AGI-3 track; two rows of Appendix L's cost table are corrected; and the abstract and Section 2.6 each lose a few words.

Our replies to the follow-ups from Reviewers gNhT and oFmE are in their threads. Thanks again to all three of you for the time you've put into this, and do let us know if there's anything else you'd like us to add or change, we'd be very happy to :)

AuthorsOpenReview ↗

Response to Reviewer gNhT's follow-up (1/2)

Thanks for following up, and fair point. We didn't give good answers to those, so here's a more direct response.

"can you summarize the main experimental conclusions you would like to highlight"

There are three:

  1. Despite clear instructions to every harness that: tasks recur, each instance starts a new conversation, and every demonstration and wrong answer costs one, the harnesses as shipped still kept little that helped on returning tasks. A simple cache of each task's demonstrations saved much more.
  2. Told what to store, a sufficiently strong model can drive the harness to memorize nearly as well as the program library. However, Qwen3.5-9B needed notes written by GLM-5.3 Flash (its teacher) to memorize effectively, which is the third row below.
  3. Nothing held in context helped much on unseen tasks, whatever we put in there (the harness's own notes, expert guidance, the teacher's notes, a letter from the teacher, every file the teacher saved, cached demonstrations, verified programs, and the library of primitives the stream's tasks are built from). Training the weights on the teacher's replies did.

The table below has Qwen3.5-9B's side of this, as the harness driver. Each number is the share of its budget a learner no longer spends on the two tests (seen and unseen) after the stream, against itself wiped before every instance.

Qwen3.5-9Breturning tasksunseen tasks
harness as shipped (Q)−0.01 [−0.04, 0.03]−0.01 [−0.05, 0.03]
harness told what to store (QE)0.03 [−0.08, 0.13]−0.01 [−0.05, 0.03]
harness with its teacher's notes, both told what to store (QETE)0.34 [0.26, 0.42]−0.05 [−0.10, 0.00]
every file its teacher saved, in the prompt (Appendix E)0.64 [0.55, 0.72]0.02 [−0.03, 0.07]
cache of demonstrations (QC)0.37 [0.26, 0.51]−0.01 [−0.05, 0.03]
one verified program per task (QP)0.65 [0.46, 0.86]0.03 [−0.02, 0.09]
weights trained on its teacher's replies (QDE, DAgger)0.57 [0.48, 0.67]0.29 [0.23, 0.35]

The three runs from scratch we promised are also in, on GLM-5.3 Flash (Table 5 in Appendix B). Over them the harness as shipped saved 0.25 [0.17, 0.33] on returning tasks, half of the cache's 0.50, and 0.68 once told what to store, against the program library's 0.86. On unseen tasks the most any of the four saved was 0.10 [0.03, 0.17].

We'd see these as the case for the benchmark too, which is the paper's main contribution: notes and weights are rarely compared on one scale, and on its first track alone it separates them clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.