acceptodds
Under review as a conference paper at ICLR 2027

AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learning

Abstract

A solved example can help an agent complete a task without preparing it to solve the next one. AhaBench tests how agents use experience in later tasks and decisions. Aha-Puzzle tests questioning on a new mystery after solved episodes; Aha-Euler tests computation on new inputs after teaching; Aha-Vending follows decisions and feedback over a simulated business year. The benchmark leaves the learning mechanism open, and our evaluation is on fixed-weight agents. Earlier Aha-Puzzle solutions help some models, but scores remain far below solving with hints. Qwen 3.6 Plus gains +12.57 points over an attempt without earlier episodes. On single-step Aha-Euler tasks, it reaches 97.3% held-out accuracy with an explanation and code, versus 8.1% with the question and answer alone. In nominal Aha-Vending operation, Doubao Seed 2.0 Pro finishes at + $495, but its profit averaged over the year is -$10 because recovery arrives late. We release tasks, validators, a simulator, and evaluation records for studying how agents reuse experience on new tasks and sustain productive behavior over time.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, language agents, agent evaluation, long-horizon interaction, learning from experience, transfer, adaptive agents, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hLhNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AHABENCH, a benchmark evaluating whether LLM agents convert experience into reusable insights across three tasks: AHA-PUZZLE (lateral thinking questioning after solved episodes), AHA-EULER (computational transfer to new inputs after teaching), and AHA-VENDING (sustained business decisions in a simulator). The benchmark evaluates fixed-weight agents through context-accessible experience, introducing metrics for support withdrawal, procedure gaps, and temporal operational profiles. Results show that earlier episodes help some models explore puzzles (+12.57 points for Qwen 3.6 Plus), that code provision vastly exceeds answer-only teaching for computation (up to 100pp gap), and that terminal profit can mask prolonged losses in operational tasks.

Critical strengths
  1. The paper proposes three representative task types—AHA-EULER, AHA-VENDING, and AHA-PUZZLE—to evaluate whether LLM agents can transform prior experiences into reusable insights.
  2. The released tasks, verifiers, and simulators demonstrate good reproducibility.
Critical weaknesses
  1. The benchmark only evaluates in-context learning with fixed weights. Although positioned as a baseline, the lack of comparison with continuous learning methods involving weight updates limits the demonstration of its discriminative power for its intended purpose.
  2. The benchmark only measures the behavioral outcomes of experience, without providing deeper insights or concrete proposals for subsequent optimization.
Questions for authors
  1. The paper evaluates agents with fixed weights using context. Do you have any preliminary data suggesting that fine-tuning or memory-augmented architectures/harnesses could further improve model performance?
3 comments

Thank you for the review. Both weaknesses are answered by content the submission already contained: the first concerns the paper's stated scope, and the second is contradicted by its results sections.

W1. Only fixed-weight in-context learning is evaluated. The review itself describes the fixed-weight agents as "positioned as a baseline", and that is their role: AhaBench is built to compare learning methods, including those that update weights, and the baseline shows the gaps such methods must close. Discriminative power for learning methods means separating different ways of using the same experience, and the paired controls already do so: raw history versus none in Aha-Puzzle (Qwen 3.6 Plus +12.57, story interval [+7.42, +20.00]), and full versus answer-only teaching in Aha-Euler (32 to 90 points, all 95% lower bounds above zero). Appendix C states the smallest method difference each component can detect.

Weight-updating methods enter under the same rules (Appendix E.1): the same reset units (per curriculum, per task, per run), the same restrictions on content, and a reported count of updated parameters and extra calls. A parametric method restarts each Puzzle curriculum from the same initial weights, so its gain is paired against the same cold targets as every other method. Fixed-weight evaluation is also the common ground for today's frontier agents, most of which, such as GPT-5.4 and Gemini 3.1 Pro, are available only through APIs.

Question: preliminary evidence for fine-tuning or memory harnesses. Not from new runs, but the existing results already indicate where such methods should help.

  • Store procedures, not answers. In Aha-Euler, teaching with an explanation and code raises held-out accuracy over answer-only teaching by 32 to 90 points end to end (Table 4); the explanation and the code are not yet separated. A memory or skill library that keeps executable procedures, as in Voyager-style skills, targets this gap.
  • Store hypothesis revision, not story facts. In Aha-Puzzle, explanation correctness accounts for 72 to 86% of the score drop when hints disappear (Table 7). The failure traces (Appendix H) show repeated rejected hypotheses and stopping before the causal chain is checked, so a useful memory carries those strategies.
  • Store operating constraints and keep acting. In Aha-Vending, the recorded failures are slot-fit errors (Doubao Seed 2.0 Pro) and declining activity after profitable operation (Kimi K2.6), which call for rule memory plus a persistence check.

W2. Only behavioral outcomes, no proposals for optimization. The submission contradicts this. Each of its results sections states concrete targets tied to a measured gap: revising hypotheses and checking the causal chain before stopping (Section 4.2), finding the right computation and implementing it within the budget (Section 4.3), and recovering sooner while keeping productive actions going (Section 4.4). These are the three targets summarized above; Appendix H ties each to recorded failures, and Appendix E adds an optimization signal for extractors: the gain of an extracted rule over a length-matched irrelevant rule on the same targets.

Summary. The second weakness names no specific missing analysis, while Sections 4.2 to 4.4 and Appendix H give measured, component-specific targets for learning methods, summarized above. We would appreciate it if you could point to an analysis you find missing and, if the additions above address your concerns, reconsider the score.

Thank you for the detailed rebuttal. I appreciate the clarification regarding W2; I now recognize that Sections 4.2–4.4 and Appendix H do provide actionable optimization targets tied to measured behavioral gaps, and I consider this concern resolved.

Regarding W1 and Q1, while I understand that AHABENCH is designed as an open protocol where fixed-weight agents establish a baseline, my core concern remains: without at least one empirical demonstration comparing a weight-updating or memory-augmented method against this baseline, the benchmark's practical utility for discriminating among continual learning approaches is theoretically argued but empirically unproven. The qualitative suggestions in response to Q1 are insightful but do not substitute for preliminary data showing that such methods actually close the identified gaps.

Could the authors clarify whether any such comparative experiments are planned prior to publication, or if the benchmark relies entirely on future community adoption to validate its discriminative capacity?

Score Recommendation Rating: 2 — unchanged. Confidence: 3 — unchanged.

AuthorsOpenReview ↗

Follow-up response to Reviewer hLhN

Thank you for engaging with the rebuttal and for confirming that W2 is resolved. To answer your question directly: we do not plan to add a method run before publication, and the benchmark's discriminative capacity does not rest on future adoption. It is already measured, and what future methods add is the ranking the benchmark is built to produce. We explain our rationale as follows.

Discrimination is measured, not only argued. A learning method, whether it updates weights or keeps a memory, changes what the agent retains from experience and how it is presented later. The benchmark already separates such changes on the same targets, with paired intervals: raw history versus none for Qwen 3.6 Plus (+12.57, story interval [+7.42, +20.00]); full versus answer-only teaching for every model (32 to 90 points end to end, 95% lower bounds of 7.6 to 68.0); and answer-only teaching versus cold solving (GPT-5.4 from 56.5% to 73.9%, Gemini 3.1 Pro Preview from 37.5% to 62.5%, Section 4.3). A method that helps must move these same quantities, and Appendix C states in advance how large a move the tasks resolve: about 10 to 15 rubric points at the Puzzle endpoint and roughly 20 points in single-step Euler accuracy.

Few, expensive units are inherent to a long-horizon benchmark. Each unit of AhaBench is a whole collaboration: one t7 score needs a seven-story curriculum of up to 140 question-answer turns plus seven gradings, and each Aha-Vending run is a simulated year of decisions. One pass of the full suite costs about 50k USD in API fees, and every additional method multiplies that cost. A benchmark that measures long-horizon learning cannot be cheap per unit, and what it can do is make each unit count and make method comparisons affordable. That is why we built the replay protocol (Appendix E.1): a method reuses the recorded curricula and regenerates only the t7 episodes, 84 of the 672 per agent, one-eighth of the original Puzzle cost.

The benchmark is forward-looking by design. Weight-updating and memory-augmented agents are among the most active directions in the field, yet no method has become established: we are not aware of a standard recipe that a benchmark paper could take off the shelf and that the community would accept as representative. A demonstration with a method of our own design would mainly evaluate that method, not the benchmark. What these methods need first is a common test fixed before they mature: the same experience, the same access rules (reset units, content restrictions, reported parameters and extra calls), paired controls, and detectable effects stated in advance. AhaBench provides exactly this, and its fixed-weight results are the baseline such methods must beat. A continual-learning benchmark is most useful before strong methods exist, because it fixes the comparison before anyone can tune to it.

We hope this answers your question, and we would be grateful if the resolved W2 and the measured discrimination above were reflected in your final assessment.

Reviewer_XaHLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AhaBench studies whether agents use earlier experience to perform better on later tasks. It combines three settings: asking questions about new mysteries after seeing solved stories, solving computational problems after teaching examples, and running a simulated vending business over a year. The evaluated models keep their weights fixed and use experience through context and feedback. The results show some benefits from earlier examples, large improvements when reference code is supplied, and operational failures that final profit alone can hide.

I found the paper difficult to follow. The narrative moves between transfer from earlier examples, dependence on hints or supplied code, and business performance without clearly explaining how these results fit together. Individual findings are reported, but it is often unclear which claim they support and what conclusion we should draw across the three settings. I would suggest organizing each component around a specific question, the comparison that answers it, and the limits of that conclusion. The discussion should then explain what the findings collectively establish about learning from experience.

Critical strengths

The paper studies an important question: whether completing a task leaves an agent better prepared for the next one. I think benchmarks that distinguish solving individual tasks from accumulating useful knowledge are valuable and very relevant in this times where recursive self-improvement has become a big focus of attention.

Critical weaknesses

I am not convinced that the three components consistently measure whether agents learn reusable insights. Puzzle provides a useful endpoint comparison, but its curriculum increases prior experience while removing hints, so the trajectory does not isolate the effect of experience. Euler’s full-teaching condition supplies both an explanation and executable code, making it difficult to distinguish learning a method from adapting a supplied implementation. In Vending, changing profit can reflect inventory, cash, and demand rather than improved decisions from experience.

The evaluation and reporting make it difficult to judge how general and reliable the conclusions are. Puzzle contains only seven underlying stories; translations and repeats do not broaden the range of reasoning strategies tested. Euler’s main results exclude failures including budget exhaustion and malformed outputs (GPT-5.4’s answer-only accuracy falls from 73.9% to 34.0% when all requested evaluations are counted). Moving between these results and the different Vending settings also makes the overall argument difficult to follow.

Questions for authors

Do you have comparisons that vary the amount of prior puzzle experience while keeping the target and its hints fixed? This would help separate learning from the effect of removing direct guidance.

Do the Euler gains persist when teaching provides an explanation without reference code? This seems important for understanding whether agents acquire a reusable method or mainly adapt the supplied program.

Is there evidence that Vending agents make better decisions because of earlier experience when starting from the same business state? The matched-state test proposed in Section 5 seems particularly relevant here.

3 comments

Thank you for the structural suggestion, which we adopted (Table 1, Section 5). The review's validity points, however, do not hold against the submission itself: each targets a descriptive profile, a claim the paper does not make, or a limitation the paper already reported, in the Aha-Vending case together with the test that addresses it. None identifies an error in a reported result.

Aha-Puzzle: experience grows while hints are removed. The comparison your Question 1 asks for, varying prior experience with the target and its hints fixed, is the paper's headline comparison: cold and t7 use the same story, language and repeat, neither has hints, and only the six earlier reveal records differ. The review itself calls it "useful". Qwen 3.6 Plus gains +12.57 points (story interval [+7.42, +20.00]). The t1-to-t7 profile, where experience and hint withdrawal change together, is presented as descriptive (Section 3), so the objection to it does not reach the claim. Intermediate amounts of experience need no new design: the replay protocol in Appendix E.1 truncates the history to two or four reveal records, with the target and its hints fixed.

Aha-Euler: learning a method vs. adapting supplied code. This does not touch the paper's claim. The claim is that answer-only teaching, which withholds both the explanation and the code, leaves every model 32 to 90 points below full teaching end to end; it does not depend on which of the two drives the gap. Separating them is the next comparison the protocol supports, and Table 1 lists it.

Aha-Euler: excluded failures. The figure the review cites, GPT-5.4's answer-only accuracy falling from 73.9% to 34.0% when all requests are counted, comes from the submission, whose main text reported this end-to-end view and gave it for every model in Appendix B. Counting failures makes the result stronger: Table 4 now places both accountings side by side, and end to end the procedure gap is positive for every model, 32 to 90 points, with 95% lower bounds of 7.6 to 68.0. GPT-5.4's gap grows from 21.7 to 44.0 points, because its answer-only attempts fail to finish more often (27 vs. 10 of 50 requests, each time by exhausting the 7,200-second budget).

Aha-Vending: profit reflects inventory, cash and demand. This is the paper's point, not a counterargument. Aha-Vending does not stop at final profit precisely because profit reflects business state: time-averaged profit, drawdown and early and late earning rates on the same runs reveal what the endpoint hides (Doubao Seed 2.0 Pro ends at +495 USD but averages -10 USD over the year; Kimi K2.6's earning rate falls from 5.89 to 2.86 USD per day). The test the review recommends, restoring a saved business state and continuing with and without the agent's history under the same demand stream, is the matched-state test the submission proposed in Section 5, as the review itself notes.

Seven underlying stories. Each story is a whole simulated collaboration, not a single item: one t7 score requires a seven-story curriculum of up to 140 question-answer turns (77 on average) with a GPT-5.4 host, plus seven gradings, and one pass of the full suite costs about 50k USD in API fees. Translations and repeats replicate each story, and the analysis treats them exactly that way: intervals resample whole stories, Qwen 3.6 Plus's gain stays between +9.44 and +14.03 when any story is left out (Appendix A), and Appendix C states what seven stories resolve, about 10 to 15 rubric points at the story level.

Summary. Your main concerns were how the paper is organized and what each comparison establishes. We restructured it as you proposed (Table 1, Section 5), report the failure accounting you asked for, which strengthens the Euler result, and state the limits of each component explicitly. We would welcome a re-reading of the revised version and, if these concerns are resolved, a reconsideration of the rating.

Thank you for the clarifications and presentation changes. I acknowledge that the Puzzle endpoint comparison is controlled and that the end-to-end Euler results were already reported. However, my broader concern remains: the three settings provide limited evidence of learning reusable insights. Euler does not separate learning a method from adapting supplied code, Vending lacks the matched-state comparison, and Puzzle covers only seven underlying stories. The response clarifies the scope but does not sufficiently strengthen the contribution. I maintain my rejection.

AuthorsOpenReview ↗

Follow-up response to Reviewer XaHL

Thank you for re-reading the revision and for acknowledging that the Puzzle endpoint comparison is controlled and that the end-to-end Euler results were already reported. Each of the three remaining points asks for a further long-horizon evaluation rather than identifying a problem with a reported result, and we address them on that basis.

Why each request is a new long-horizon run, and why units are few. An explanation-only Euler condition, a matched-state Vending run and additional Puzzle stories are all new evaluations rather than analyses of the recorded data, and AhaBench's units are whole collaborations: one t7 score needs a seven-story curriculum of up to 140 question-answer turns plus seven gradings, and each Aha-Vending run is a simulated year of decisions. One pass of the full suite costs about 50k USD in API fees. Few, expensive units are therefore inherent to measuring learning over long horizons, not a shortcut, and the paper is designed around them: comparisons are paired on the same targets, intervals resample whole stories, leave-one-story-out checks bound Qwen 3.6 Plus's gain between +9.44 and +14.03 (Appendix A), Appendix C states the effects each component resolves, and the replay protocol of Appendix E.1 cuts a new Puzzle comparison to one-eighth of the original episodes. Each of the tests you ask for fits the released protocol and can be run under the same rules; the matched-state test is already specified in the paper.

Aha-Euler: learning a method versus adapting supplied code. The paper already measures learning without any supplied code: answer-only teaching gives neither an explanation nor code, and on tasks answered in all three conditions it raises GPT-5.4 from 56.5% cold to 73.9% and Gemini 3.1 Pro Preview from 37.5% to 62.5% (Section 4.3). That gain comes from the taught instance alone. What remains open, and Table 1 lists it, is which part of full teaching, the explanation or the code, adds the further gain; the finding that answer-only teaching leaves every model 32 to 90 points below full teaching end to end does not depend on it.

Aha-Vending: the matched-state comparison. Aha-Vending's claim is that final profit hides late recovery and declining activity on the same runs, not that experience causes better decisions; Table 1 states this scope, and the matched-state test is specified for the causal question.

Limited evidence of learning. We agree that today's agents show limited learning, and the paper reports this as a main finding. That is a property of the agents, not of the measurement: a long-horizon benchmark is needed precisely to show how far current agents fall short and to fix the controls before learning methods mature.

Thank you again for the structural suggestion, which made the paper clearer.

Reviewer_zU6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AhaBench asks whether an agent's experience on one task makes it better at the next, and proposed three tasks. Aha-Puzzle gives an agent seven lateral-thinking stories in sequence; the reference question–answer prefix for the current story shrinks to zero while reveal records of earlier stories accumulate, and the final hint-free story is compared with a cold attempt at the same story. Aha-Euler pairs a taught Project-Euler-style instance with a held-out instance of the same computation, under cold, answer-only, and full (answer, explanation, code) teaching. Aha-Vending runs a Vending-Bench-style simulator for a year and reports time-averaged profit and draw down alongside final balance. Seven to ten frontier models are evaluated as fixed-weight agents with raw history in context. The paper reports that earlier solved episodes help some models, that worked code transfers far better than answers alone, and that final profit hides late recovery.

Critical strengths
  • The protocol is well studied and tested. The benchmark decomposes “experience reuse” into three concrete, complementary axes—exploration after solved episodes, computation reconstruction, and sustained operation—each with its own protocol and metric. For The artifacts (tasks, rubrics, generators, exact validators, simulator, analysis tables) are complete enough that the protocol survives even if the present numbers change.

  • Aha-Euler separates two abilities that are usually conflated. Full teaching tests whether an agent can adapt code; answer-only teaching tests whether it can reconstruct a method. The gap is large (Qwen 3.6 Plus "97.3% with full teaching to 8.1% with partial teaching", §4.3) and the generated variants and hardened inputs make memorized Project Euler answers unlikely to explain it.

  • Long-horizon and continual-use evaluations are increasingly central; a benchmark that dissects experience reuse into measurable components and provides open tools is valuable.

Critical weaknesses
  • The benchmark is designed to compare learning methods, but none is tested. Every agent in the paper is a frozen model with earlier episodes pasted into its context. No method extracts a rule, stores a memory, or updates a skill, so the paper shows only that replaying history gives small gains; it does not show the benchmark can tell a good learning method from a bad one.

  • AHA-PUZZLE’s host and rubric are LLM-based (GPT-5.4/Pro), and GPT-5.4 is also among evaluated agents, raising fairness and alignment concerns. The human verification for the LLM-judge alignment (100 audited trajectories) is narrow relative to the full evaluation scale; inter-rater reliability and judge robustness under adversarial or off-distribution outputs are not reported.

  • Some protocol specifics critical for reproducibility or fairness are deferred to appendices (e.g., precise interpreter budgets/timeouts, multi-node DAG distributions), which makes it harder to assess comparability across models at a glance.

  • The closest long-horizon and exploration benchmarks are not discussed. UltraHorizon evaluates rule discovery through sustained exploration, which is the capability Aha-Puzzle targets. CoffeeBench, RetailBench and E-Commerce Bench run year-scale or thousand-day business operation with deterministic economics, which is the setting of Aha-Vending. The related work covers only Vending-Bench and the lateral-thinking lineage. A targeted comparison would sharpen AhaBench's contribution.

Questions for authors
  • For each model and table, how many valid pairs, complete curricula, or shared tasks underlie the reported mean? Is GPT-5.4's Aha-Puzzle row based on three pairs, and Kimi K2 Thinking's Aha-Euler row on four tasks?
  • Why did most Kimi K2 Thinking requests fail under full teaching, and are the four cold rates of exactly zero in Table 9 wrong answers or budget and format failures?
  • What may a persistent memory carry between curricula, between languages, and between the teaching and held-out phases of Aha-Euler? Is there a length or content budget for an extracted rule, and is memory reset per curriculum?
  • Would you run one learning method that extracts and reuses knowledge across tasks, so the benchmark's sensitivity to the thing it is meant to measure can be seen? The Section 5 protocol with length-matched irrelevant rules would be the natural form.
  • GPT-5.4 is both the rubric executor and an evaluated agent, and the human audit reports full agreement on 100 trajectories. Was the audit blind to the executor's grade, and what was criterion-level agreement?
  • What were the token budget, calculator timeout, and per-size task counts for Aha-Euler, and were they identical across models?
  • How sensitive are Aha-Puzzle scores to the host model? Have you run a second host, or restricted the allowed host response classes, to separate environment-induced variance from agent behavior?
4 comments
AuthorsOpenReview ↗

Response to Reviewer zU6p (Part 1 of 3: weaknesses)

Thank you for the detailed reading and the concrete questions. Your strengths section finds that the artifacts "are complete enough that the protocol survives even if the present numbers change" and that "a benchmark that dissects experience reuse into measurable components and provides open tools is valuable". Each weakness below is a scope the paper states, a reporting request the revision now fulfills, or a concern the recorded data answer; none contradicts that assessment.

W1. No learning method is tested. This is the paper's stated scope, not a defect. A benchmark must fix the experience, the access rules and the controls before methods can be compared on it; asking the paper that introduces it to also supply the method it will rank conflates two contributions. The substantive question is whether the benchmark can tell a good learning method from a bad one. A learning method decides what to keep from experience and how to present it later, and the benchmark already separates such choices on the same targets: raw history versus none for Qwen 3.6 Plus (+12.57, story interval [+7.42, +20.00]), and full versus answer-only teaching for every model (32 to 90 points end to end, all 95% lower bounds above zero). Appendix C states the smallest differences the tasks resolve (about 10 to 15 rubric points on the Puzzle endpoint), and Appendix E.1, summarized in Section 5, makes the benchmark ready for methods: access rules plus a replay protocol in which a method regenerates only the t7 episode, one-eighth of a model's original Puzzle episodes. The test you name in Q4, with length-matched irrelevant rules, is the one the submission itself specified in Section 5.

W2. LLM host and grader, and GPT-5.4 as an evaluated agent. The concern is testable, and the data answer it.

  • The grader agrees with blind human scores. Three annotators scored 100 random trajectories without seeing GPT-5.4's grade; their scores correlate with it at Pearson r = 0.987, with a mean absolute difference of 1.75 points (Q5). An audit's adequacy depends on how precisely it estimates agreement, not on its share of the evaluation pool: with 100 random trajectories, the 95% interval for this correlation is [0.981, 0.991].
  • Self-preference cancels in the gain. Cold and t7 for the same story are scored by the same grader, so a constant preference for one's own outputs cancels.
  • The host only executes rules. It is also GPT-5.4, but may only give human-checked minimal answers, yes/no or short replies that never volunteer unasked parts of the hidden explanation, and the audit confirmed that every host reply followed these rules (Q7).
  • No conclusion rests on GPT-5.4's Puzzle row. It has three pairs from one story (Q1); we mark it and do not interpret it.

W3. Protocol details in appendices. New Table 15 lists the protocol settings of the reported runs and new Table 14 the sample sizes; Section 4.1 points to both. The Aha-Euler time budget and calculator limits you ask about were the same for every model (Q6).

W4. Closest benchmarks. They are now discussed in Section 6, where they sharpen what AhaBench adds.

  • UltraHorizon measures hidden-rule discovery within a single long episode. Aha-Puzzle instead tests transfer across episodes: whether solved episodes of other stories improve exploration of a new hidden explanation, compared with a cold attempt on the same story.
  • CoffeeBench (a 90-day multi-agent supply chain), RetailBench (data-grounded supermarket operation designed for thousand-day runs) and E-Commerce Bench (a year of multi-store operation with deterministic demand and negotiation) score operation over the horizon; E-Commerce Bench also measures reuse of past negotiations.
  • Aha-Vending contributes temporal accounting: time-averaged profit and early and late earning rates expose late recovery and declining activity that final profit hides (Doubao Seed 2.0 Pro ends at +495 USD but averages -10 USD over the year). Table 1 states its scope, sustained operation rather than a causal effect of experience, and Appendix E specifies the matched-state test for the causal question.
AuthorsOpenReview ↗

Response to Reviewer zU6p (Part 2 of 3: questions)

Q1. Data behind each mean. All counts are now in Table 14.

ModelPuzzle pairs (of 84)Stories (of 7)Euler tasks answered in full and answer-onlyEuler tasks answered in all three
DeepSeek v3.2 Thinking8473837
Gemini 3.1 Pro Preview8072524
Qwen 3.6 Plus7773735
Doubao Seed 2.0 Pro7572220
Kimi K2 Thinking37743
Claude Opus 4.6144n/an/a
GPT-5.431 (Japanese only)2323
Hy3 Previewn/an/a86

GPT-5.4's Puzzle row rests on three pairs: the revision excludes three cold episodes that ended in an error, as for every other model, which leaves three Japanese pairs from one story; the row is marked and not interpreted. For every other model, complete curricula equal the pair counts. Kimi K2 Thinking's Euler row rests on four tasks only when unanswered attempts are dropped; end to end, the accounting behind every Euler claim, it rests on all 50 requests per condition, each unanswered attempt counted as incorrect (Q2).

A pair is lost when its cold episode or any episode of its curriculum ends without a scored answer; because a curriculum contains all seven stories, one such episode removes the whole curriculum. Most such episodes are blocked by safety refusals or content filters, since several stories contain violent or horror elements; the rest end in API or recording errors. We exclude them rather than score them as zero, so that they do not count as failed exploration (Appendix C). DeepSeek v3.2 Thinking loses no pairs, and every Vending row has five seeds. The headline result does not depend on the partial rows: Qwen 3.6 Plus covers all seven stories, and its gain has a story-level interval that excludes zero.

Q2. Kimi K2 Thinking failures and the zeros in Table 9. Kimi returned an answer for 18 of 50 full-teaching requests, and for 17 of 50 cold and 6 of 50 answer-only requests. Every unanswered request hit a time limit; none was a malformed response, a server error or a refusal. Of the 32 unanswered full-teaching requests, 20 ended in a request timeout at the API gateway (HTTP 504) during a teaching step, although the runner had already retried each of these tasks twice; the other 12 used up the 7,200-second budget per attempt. Cold (33 unanswered) and answer-only requests (44 unanswered) fail in the same two ways, so the failures are not specific to the longer full-teaching context. Even counting every failure as incorrect, Kimi's procedure gap is 36.0 points [13.0, 52.8] (Table 4, right). The four cold zeros in Table 9 (now Table 10) are wrong answers, not budget or format failures, because that table keeps only targets answered in all three conditions; they rest on 35, 37, 3 and 6 tasks (Qwen, DeepSeek, Kimi, Hy3), which the paper now states.

Q3. What persistent memory may carry. New Appendix E.1 answers this. Memory is reset per curriculum in Aha-Puzzle, because the same stories recur across schedules, languages and repeats and carried memory would leak the target; per task in Aha-Euler, where it may carry anything from the teaching exchange into the held-out question; and per seed in Aha-Vending. It may never be derived from the current target's solution or held-out answer. Methods report the tokens they carry or the parameters they update. Extracted rules default to a 512-token budget and are compared with length-matched irrelevant and shuffled rules.

Q4. One learning method. Not in this paper, for the reasons in W1. The test you describe is the one the submission specified in Section 5, and the replay protocol in Appendix E.1 now runs it on one-eighth of a model's original Puzzle episodes, with the detectable effects of Appendix C stated in advance.

AuthorsOpenReview ↗

Response to Reviewer zU6p (Part 3 of 3: questions, continued)

Q5. Audit blindness and criterion-level agreement. The audit was blind: three annotators scored 100 trajectories drawn at random from the evaluation pool, covering models, languages, and cold and curriculum episodes, without seeing GPT-5.4's grade. Their scores correlate with the grader's at Pearson r = 0.987, with a mean absolute difference of 1.75 points on the 100-point rubric, and every host reply followed the allowed-answer rules (Q7). The audit was run on total scores, the quantity the benchmark reports; criterion-level marks were not collected. Appendix D now reports these details.

Q6. Aha-Euler budgets. There is no token budget. Each attempt has one wall-clock budget of 7,200 seconds, shared by model calls and calculator runs; there is no cap on calculator calls and no separate calculator timeout, so a calculator run may use whatever remains of that budget. These limits were the same for every model: every attempt's prompt states them, and every budget-exhaustion record names 7,200 seconds. The graph bank has 40 tasks, 10 at each size of 2 to 5 nodes. The figure of 90 in Section 2.2 also counted the 50 single-step tasks, which the bank stores as one-node graphs; we corrected it. Every model is scored on all 50 single-step and all 40 graph tasks in every condition; a graph task the runner skips after earlier failures counts as incorrect. Table 15 now lists these settings.

Q7. Host sensitivity. The second option you suggest is how the protocol works: the host's responses are already restricted. It is an executor of fixed rules, not a judge: it knows the hidden explanation but may only give minimal yes/no or short answers and never volunteers unasked parts of it (Table 15). The human audit (Q5) confirmed that the host follows these rules: in 100 randomly drawn trajectories, covering cold and curriculum episodes, no host reply broke them. Because the host behaves the same way in both conditions, and each target's cold and t7 episodes face the same host, the host does not bias the gain. DeepSeek v3.2 Thinking's complete coverage also shows the host answered every story.

Summary. Your review rates the soundness concerns as minor or fixable, and each is addressed above: sample sizes and why pairs are lost, the GPT-5.4 row corrected to the three pairs you suspected, the audit described exactly, the closest benchmarks discussed, and the memory rules specified. If a remaining concern still supports a weak-reject rating, we would be grateful to know which one, so we can respond to it directly.

dear reviewer,

thanks for your support thus far ! It would be very helpful if you can react to the authors' comments.

many thanks, your area chair.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their efforts.

AhaBench asks whether agents turn experience into better later decisions, and answers it with paired controls in three settings of long human-AI collaboration. The reviews affirm both the question and the artifacts: zU6p finds that "the protocol is well studied and tested" and that Aha-Euler "separates two abilities that are usually conflated"; XaHL calls benchmarks that separate solving single tasks from accumulating useful knowledge "valuable and very relevant"; hLhN notes that the released tasks, verifiers and simulators "demonstrate good reproducibility".

What the revision adds

  • Structure. Section 2 presents the components as patterns of long human-AI collaboration; new Table 1 gives each component's question, controlled comparison, finding and scope; Section 5 opens with what the three establish together.
  • End-to-end Aha-Euler accounting (Table 4). Counting every unanswered attempt as incorrect, full teaching beats answer-only teaching by 32 to 90 points for every model, with 95% lower bounds of 7.6 to 68.0.
  • Full accounting and detectable effects (Appendix C). The pairs, tasks and seeds behind every mean (Table 14), every budget and limit (Table 15), and the effects the tasks resolve: about 10 to 15 rubric points on the Puzzle endpoint and a net change of roughly 20 points in single-step Euler accuracy, well below the observed teaching gaps. Puzzle pairs are lost mostly to safety refusals or content filters, since several stories contain violent or horror elements, and otherwise to API or recording errors. GPT-5.4's Puzzle row is corrected to three pairs from one story; it is marked, and no conclusion depends on it.
  • The blind human audit (Appendix D): three annotators, blind to the GPT-5.4 grade, scored 100 random trajectories; Pearson r = 0.987 with the grader, mean absolute difference 1.75 points.
  • Rules and a replay protocol for learning methods (Appendix E.1, summarized in Section 5), in which a method regenerates only the final episode, and related work on UltraHorizon, CoffeeBench, RetailBench and E-Commerce Bench.

Why each unit is a whole collaboration. One t7 score requires a seven-story curriculum of up to 140 question-answer turns (77 on average) with a GPT-5.4 host, plus seven gradings; each Aha-Vending run is a simulated year of decisions. One pass of the full suite costs about 50k USD in API fees, so AhaBench, like other long-horizon benchmarks, runs few units per model and makes each count: paired cold/t7 comparisons, story-level intervals, leave-one-story-out checks (Appendix A), and the detectable effects above.

Disclosed limitations are not grounds for rejection. The low scores rest mainly on limitations the submission itself reported: fixed-weight agents only, seven stories, profit that reflects business state, and language models as host and grader. The controls the reviews request are the ones the submission's Section 5 already specified: the matched-state test (XaHL) and length-matched irrelevant rules (zU6p). The 34.0% end-to-end accuracy XaHL uses against the Euler results was reported in the submission's main text. A benchmark that states its scope, its failure accounting and its detectable effects is easier to check, not weaker; penalizing that disclosure rewards papers that report only favorable numbers.

Where the reviews and the scores diverge. Beyond the reporting issues we have now fixed, no review identifies an error in the measurements, the paired designs or the released artifacts, and two reviews rate their soundness concerns as minor or fixable. The shared objection is that no learning method is evaluated. That is a property of today's agents, not a flaw of the benchmark: a continual-learning benchmark is most useful before strong methods exist, because it fixes the experience, the access rules and the controls in advance, and the recorded data already show that it resolves differences between experience interfaces with paired intervals. We respectfully ask the reviewers to judge the revised paper on what it measures and releases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.