- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AHABENCH, a benchmark evaluating whether LLM agents convert experience into reusable insights across three tasks: AHA-PUZZLE (lateral thinking questioning after solved episodes), AHA-EULER (computational transfer to new inputs after teaching), and AHA-VENDING (sustained business decisions in a simulator). The benchmark evaluates fixed-weight agents through context-accessible experience, introducing metrics for support withdrawal, procedure gaps, and temporal operational profiles. Results show that earlier episodes help some models explore puzzles (+12.57 points for Qwen 3.6 Plus), that code provision vastly exceeds answer-only teaching for computation (up to 100pp gap), and that terminal profit can mask prolonged losses in operational tasks.
Critical strengths
- The paper proposes three representative task types—AHA-EULER, AHA-VENDING, and AHA-PUZZLE—to evaluate whether LLM agents can transform prior experiences into reusable insights.
- The released tasks, verifiers, and simulators demonstrate good reproducibility.
Critical weaknesses
- The benchmark only evaluates in-context learning with fixed weights. Although positioned as a baseline, the lack of comparison with continuous learning methods involving weight updates limits the demonstration of its discriminative power for its intended purpose.
- The benchmark only measures the behavioral outcomes of experience, without providing deeper insights or concrete proposals for subsequent optimization.
Questions for authors
- The paper evaluates agents with fixed weights using context. Do you have any preliminary data suggesting that fine-tuning or memory-augmented architectures/harnesses could further improve model performance?
3 comments
Thank you for the detailed rebuttal. I appreciate the clarification regarding W2; I now recognize that Sections 4.2–4.4 and Appendix H do provide actionable optimization targets tied to measured behavioral gaps, and I consider this concern resolved.
Regarding W1 and Q1, while I understand that AHABENCH is designed as an open protocol where fixed-weight agents establish a baseline, my core concern remains: without at least one empirical demonstration comparing a weight-updating or memory-augmented method against this baseline, the benchmark's practical utility for discriminating among continual learning approaches is theoretically argued but empirically unproven. The qualitative suggestions in response to Q1 are insightful but do not substitute for preliminary data showing that such methods actually close the identified gaps.
Could the authors clarify whether any such comparative experiments are planned prior to publication, or if the benchmark relies entirely on future community adoption to validate its discriminative capacity?
Score Recommendation Rating: 2 — unchanged. Confidence: 3 — unchanged.
Follow-up response to Reviewer hLhN
Thank you for engaging with the rebuttal and for confirming that W2 is resolved. To answer your question directly: we do not plan to add a method run before publication, and the benchmark's discriminative capacity does not rest on future adoption. It is already measured, and what future methods add is the ranking the benchmark is built to produce. We explain our rationale as follows.
Discrimination is measured, not only argued. A learning method, whether it updates weights or keeps a memory, changes what the agent retains from experience and how it is presented later. The benchmark already separates such changes on the same targets, with paired intervals: raw history versus none for Qwen 3.6 Plus (+12.57, story interval [+7.42, +20.00]); full versus answer-only teaching for every model (32 to 90 points end to end, 95% lower bounds of 7.6 to 68.0); and answer-only teaching versus cold solving (GPT-5.4 from 56.5% to 73.9%, Gemini 3.1 Pro Preview from 37.5% to 62.5%, Section 4.3). A method that helps must move these same quantities, and Appendix C states in advance how large a move the tasks resolve: about 10 to 15 rubric points at the Puzzle endpoint and roughly 20 points in single-step Euler accuracy.
Few, expensive units are inherent to a long-horizon benchmark. Each unit of AhaBench is a whole collaboration: one t7 score needs a seven-story curriculum of up to 140 question-answer turns plus seven gradings, and each Aha-Vending run is a simulated year of decisions. One pass of the full suite costs about 50k USD in API fees, and every additional method multiplies that cost. A benchmark that measures long-horizon learning cannot be cheap per unit, and what it can do is make each unit count and make method comparisons affordable. That is why we built the replay protocol (Appendix E.1): a method reuses the recorded curricula and regenerates only the t7 episodes, 84 of the 672 per agent, one-eighth of the original Puzzle cost.
The benchmark is forward-looking by design. Weight-updating and memory-augmented agents are among the most active directions in the field, yet no method has become established: we are not aware of a standard recipe that a benchmark paper could take off the shelf and that the community would accept as representative. A demonstration with a method of our own design would mainly evaluate that method, not the benchmark. What these methods need first is a common test fixed before they mature: the same experience, the same access rules (reset units, content restrictions, reported parameters and extra calls), paired controls, and detectable effects stated in advance. AhaBench provides exactly this, and its fixed-weight results are the baseline such methods must beat. A continual-learning benchmark is most useful before strong methods exist, because it fixes the comparison before anyone can tune to it.
We hope this answers your question, and we would be grateful if the resolved W2 and the measured discrimination above were reflected in your final assessment.
Thank you for the review. Both weaknesses are answered by content the submission already contained: the first concerns the paper's stated scope, and the second is contradicted by its results sections.
W1. Only fixed-weight in-context learning is evaluated. The review itself describes the fixed-weight agents as "positioned as a baseline", and that is their role: AhaBench is built to compare learning methods, including those that update weights, and the baseline shows the gaps such methods must close. Discriminative power for learning methods means separating different ways of using the same experience, and the paired controls already do so: raw history versus none in Aha-Puzzle (Qwen 3.6 Plus +12.57, story interval [+7.42, +20.00]), and full versus answer-only teaching in Aha-Euler (32 to 90 points, all 95% lower bounds above zero). Appendix C states the smallest method difference each component can detect.
Weight-updating methods enter under the same rules (Appendix E.1): the same reset units (per curriculum, per task, per run), the same restrictions on content, and a reported count of updated parameters and extra calls. A parametric method restarts each Puzzle curriculum from the same initial weights, so its gain is paired against the same cold targets as every other method. Fixed-weight evaluation is also the common ground for today's frontier agents, most of which, such as GPT-5.4 and Gemini 3.1 Pro, are available only through APIs.
Question: preliminary evidence for fine-tuning or memory harnesses. Not from new runs, but the existing results already indicate where such methods should help.
W2. Only behavioral outcomes, no proposals for optimization. The submission contradicts this. Each of its results sections states concrete targets tied to a measured gap: revising hypotheses and checking the causal chain before stopping (Section 4.2), finding the right computation and implementing it within the budget (Section 4.3), and recovering sooner while keeping productive actions going (Section 4.4). These are the three targets summarized above; Appendix H ties each to recorded failures, and Appendix E adds an optimization signal for extractors: the gain of an extracted rule over a length-matched irrelevant rule on the same targets.
Summary. The second weakness names no specific missing analysis, while Sections 4.2 to 4.4 and Appendix H give measured, component-specific targets for learning methods, summarized above. We would appreciate it if you could point to an analysis you find missing and, if the additions above address your concerns, reconsider the score.