- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Currently, AI agents are being used in multiple scenarios, from small tasks that run in minutes to large tasks that require interaction between multiple agents and take hours or days to finish. To work, these AI agents use their internal knowledge (weights), but are complemented by their harness and skills, which are tools that extend their capabilities and allow them to perform procedures, like internet search or using programming languages. Even though these agents are everywhere, few benchmarks exist to test their ability to accumulate knowledge and reuse it to solve previous and new tasks. This paper presents a new benchmark based on the ARC benchmark to test the capability of knowledge retention and accumulation of AI agents. Experiments with different harnesses and approaches show that there is no clear advantage of using persistent memory when comparing budget and performance, which raises the question of how to better accommodate harnesses in continual scenarios.
Critical strengths
The paper provides a compelling rationale for a relevant and increasingly popular problem (AI agents). Identifying the limitations of existing benchmarks and how these can be mitigated by the new proposal could have a great impact on the community. This paper raises questions such as: When might a skill acquired in the past be relevant to future tasks? Which information is relevant to keep? These questions can help identify when a model is learning and/or reusing information, thereby helping to identify and mitigate forgetting. The work also raises questions about backward and forward transfer, highlighting that models can reuse information. This is undoubtedly a benchmark that could be of real value to the community.
Critical weaknesses
A benchmark is presented that promises to mitigate multiple shortcomings of previous benchmarks; however, these contributions are not clearly evident in the benchmark’s design or in the experiments conducted.
- The paper discusses forward and backward transfer, but there is no analysis of how much these are evident in the experiments. What types of experiments reflect these values, and how can we mitigate the problem?
- The paper discusses skill accumulation, but the experiments refer to observations. Is there a relationship between the two?
- The tables and images from the experiments are not referenced in the text, which makes it difficult to establish a connection between the two.
- The relationship between the three components of the benchmark (Sections 3.1, 3.2 and 3.3) and the experiments conducted is not very clear.
- The experiments show the budget, but little is said about the ability to retain information or transfer information to future tasks.
Questions for authors
- The paragraph beginning on line 155 discusses some parameters. Is this an example, or are these the hyperparameters of a task? Can they be modified to make the problem easier or harder? I would imagine that some agents would need more demonstrations to find the answer, but they might generalise better to different types of tasks.
- How do the results change when the parameters described in Equations 2 and 3 are modified? When presenting a benchmark, it is important to identify its strengths and weaknesses when the problem is made easier or more difficult.
- A visual representation of Sections 3.1, 3.2, and 3.3 would help better understand the proposal.
2 comments
Response to Reviewer oFmE (2/2)
The relationship between the three components of the benchmark (Sections 3.1, 3.2 and 3.3) and the experiments conducted is not very clear.
A visual representation of Sections 3.1, 3.2, and 3.3 would help better understand the proposal.
Agreed. The benchmark section is now built around these three parts: one instance in Section 2.1, drawn in Figure 1; the stream in Section 2.2, with Figure 2 now drawing the whole ARC-AGI-1 stream; and the two tests after it in Section 2.3. Every result in Section 4 is then read on the stream and on both tests.
The paragraph beginning on line 155 discusses some parameters. Is this an example, or are these the hyperparameters of a task? Can they be modified to make the problem easier or harder? I would imagine that some agents would need more demonstrations to find the answer, but they might generalise better to different types of tasks.
To answer your question directly, those values are the benchmark's settings, the same for every learner so that results can be compared. The revision has updated some of them: an instance now allows up to eight demonstrations, and an unsolved instance is charged the whole budget of 16, as set out in Section 2.1. Since demonstrations are charged on the stream but generalization is scored separately on the held-out test, an agent that buys more demonstrations (costing some performance on the stream) but generalizes better (performs better on the held-out test) would show up as exactly that. Your question also prompted us to run the demonstration and attempt caps at 4 and 16 in place of 8, as part of the sweep below - thanks!
How do the results change when the parameters described in Equations 2 and 3 are modified? When presenting a benchmark, it is important to identify its strengths and weaknesses when the problem is made easier or more difficult.
We agree this is important to show. Appendix B now reports a first side experiment on the order, in which GPT-6 Luna's systems also played the same visits one task at a time and fully shuffled, and the trends held in every order. We are now also sweeping the stream's parameters, which are all in Equation 2 of the revision, on four of the methods: the number of tasks active at once, how often new tasks arrive and how long tasks rest, along with the order and the caps mentioned above. We will post the results here as they land, and add them to the appendix.
Hopefully these changes address some of your concerns, and we're very happy to discuss anything further! :)
Response to Reviewer oFmE (1/2)
Thank you for such a careful review, and for seeing the value this benchmark could have for the community if presented clearly. We agree that the contributions were clear in neither the design nor the experiments. We've restructured the paper around this feedback, which hopefully now makes for a smoother read. Some of your points overlap, so we've grouped those together below.
The paper discusses forward and backward transfer, but there is no analysis of how much these are evident in the experiments. What types of experiments reflect these values, and how can we mitigate the problem?
The experiments show the budget, but little is said about the ability to retain information or transfer information to future tasks.
We agree, and the revision now measures both directly. After the stream, every learner is frozen and plays two tests, set out in Section 2.3. The held-out test serves 100 tasks the learner never met, so it measures forward transfer; the seen test brings back every task of the stream, so it measures the backward side, what is left of earlier tasks after everything learned since, which is where forgetting would show. Section 2.4 scores each as the share of its reset's spend that the learner saves through learning, and Section 4 reports both of these for every learner in Table 1 and Figures 3 and 5. Figure 4 also follows the learners over their visits to a task during the stream. In short, the harnesses retained little unless given expert guidance, and the program library retained the most but transferred nothing measurable on the held-out test. Only training the weights transferred to new tasks. That is also where we see the most promise for mitigating the problem, which Section 6 discusses.
The paper discusses skill accumulation, but the experiments refer to observations. Is there a relationship between the two?
Again, thank you for pointing this out. Observations here are what a learner sees on each instance: the task id, the input grid, and whatever demonstrations and feedback it pays for. Accumulating skill means turning these into something that solves later instances with fewer observations, whether of the same task, which we call memorization, or of tasks never met, which we call generalization. The budget connects the two: every demonstration and wrong answer costs one, so skill shows up as observations a learner no longer needs to pay for. We therefore measure learning by the budget saved, irrespective of the form of the learning, whether notes in a harness, programs or weights. The demo-cache floor draws the line between the two: it simply keeps the demonstrations it observed and shows them again, so a learner that saves more than it must have learned something beyond the stored observations. Keeping a task's rule once deduced, for example, pays off on the seen test, while only abstract skills can carry over to new tasks, which is what the held-out test measures. Appendix E shows what each harness actually stored.
The tables and images from the experiments are not referenced in the text, which makes it difficult to establish a connection between the two.
Thank you for flagging it. Every table and figure in the revision is now cited where its result is discussed.