acceptodds
Under review as a conference paper at ICLR 2027

Before You Rank Skill Memories, Measure What Your Benchmark Can Express

Abstract

Agent memory and skill systems are ranked by success rate on benchmarks whose capacity to express the gaps they report is never measured. We report that capacity for a controlled skill-form protocol, in which the same source experience is serialized four ways—trajectory trace, flat script, hierarchy, and a SKILL.md artifact—under a matched retriever, prompt, and injection budget. Four quantities govern what such a comparison can show: the error rate, the repeat-measurement floor (the rate at which per-task outcomes flip when one configuration is rerun), the discriminating fraction (the share of tasks whose outcome moves across arms for reasons a rerun does not explain), and whether the task slice represents the split. On an archived version of this protocol, which controlled variance by deterministic decoding, the floor is 14.7% of per-task outcomes on ALFWorld and 12.9% on WebShop, and only 3 of 50 WebShop tasks respond to skill injection at all. A rerun inflates the number of discordant pairs, never their direction, which motivates a practical reliability rule: a gap should exceed the control's own floor. On the published WebArena Shopping numbers of a recent budget-matched study of web-agent memory, none of its nine vanilla-versus-module gaps clears the vanilla actor's floor under any of its three executors. Rerun under a protocol that passes these checks (the full split, three replicates, a bank where 89 of 140 tasks retrieve a same-type source, length-matched fillers, two executor families), the serialization of the same experience, which bundles its actions, plan, and observations, decides whether it helps. A raw trajectory is indistinguishable from no injection on both executors and on every slice, while the hierarchical form of the same experience adds 15.7 and 19.3 points (p ≤ 0.001) and beats its own length-matched filler by 10.7 and 31.4 points. On one executor that gain needs a retrieved experience of the right task type; on the other it does not.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent memory, procedural memory, skill representation, evaluation methodology, measurement noise, benchmark design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fx1dOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that many existing benchmarks often lack sufficient statistical evidence to reliably rank agent memory or skill representations. In particular, it is often assumed that a benchmark can express a difference of the size being reported, but it is neither proven nor it universally holds. The benchmark is designed to take care of neglected but important problem.

Critical strengths
  1. The four checks (error rate, repeat-measurement floor, discriminating fraction, and slice representativeness) are natural criteria and indeed make sense for the specific case.

  2. The motivation is very important. Benchmark comparisons are expensive and nowadays many advantages or disadvantages are illustrated with a small amount of differences but they are used as evidence for methodological superiority/inferiority. Thus, soundness of this method shall indeed be scrutinized with caution.

Critical weaknesses

My main concern is the extent of empirical investigation. Only three families of models (Qwen, Gemini and Llama) are used, though medium-sized models are also tested. I would like to hear the authors' justification on this matter before making the final decision on the rating.

Questions for authors

Please see Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Fx1d

Thank you for the review.

Main change for your review: the form comparison now runs on twelve open-weight executors from seven families.

Model families. Study 1 used Qwen and Llama; Gemini appears only in the external web-agent study we reanalyse. We use open-weight executors because the checks need reruns of one configuration under greedy decoding with every episode released. The two closed models we piloted (gpt-6.1-sol and gpt-6.1-terra) solve 90–95% of tasks without injection, which leaves no room for any form to show an effect.

The revision adds ten executors: Qwen3.5-35B, Llama-3.1-70B and gpt-oss-120b (Study 2); gemma-4-31B (Google), Nemotron-3.5-Bolt (NVIDIA), Ministral-3-14B (Mistral) and the reasoning model Step-3.7-Flash (StepFun); and a Qwen size ladder at 4B, 8B and 27B. The first seven ran the full design of 17 arms (the forms, their pieces and their sources).

Of these seven, the hierarchical form beats no injection on six: +8.9, +8.0 and +14.4 pp on Study 2's executors, +8.2, +7.3 and +11.7 on gemma, Nemotron and Ministral. The exception is Step-3.7 (−3.3), whose no-injection reruns disagree on 21.9% of tasks against at most 3.6% on the other three; the paper reports this as a prediction that failed. On all seven the source actions beat the subgoal template. On the size ladder the gain is +19.2 and +25.4 pp at 4B and 8B, and +9.3 at 27B (seen rooms, one-sided p = 0.051). Together with the closed models' 90–95% baseline, this suggests the effect shrinks as executors get stronger; the paper states this as a limitation.

Reviewer_2sjSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines an important issue in agent memory and skill evaluation: whether a benchmark and its execution protocol can support the reported differences between methods. The authors propose four checks—error rate, repeat-measurement floor, discriminating fraction, and slice representativeness—and a practical reliability rule that compares a method’s gain with the control’s own repeat-measurement floor. In the empirical study, the authors serialize the same source experience into four forms, HIERARCHICAL, SCRIPT, ARTIFACT, and TRAJECTORY, and compare them under a fixed retriever, source bank, prompt, and interaction budget. Results on the full ALFWorld eval_id split show that HIERARCHICAL significantly outperforms NO-INJECTION on both executors, while TRAJECTORY does not show a significant net gain in success rate. The paper also analyzes how retrieval quality, injection length, task slices, and the action guard affect these conclusions.

Critical strengths
  1. The four forms share the same source experience and retrieval results: for each of the 140 tasks, all four forms retrieve the same source trajectory, and no injected item is truncated. The deterministic, rule-based serializers also avoid additional variation from model-generated serializations. NO-INJECTION and length-matched fillers provide useful controls, making the results easier to interpret than comparisons between complete memory systems.

  2. The authors not only report three replicates per arm and exact McNemar tests on majority-vote outcomes, but also provide Holm correction, replicate-level paired effects with confidence intervals, and an ablation with the action guard turned off. Therefore, the finding that HIERARCHICAL outperforms NO-INJECTION does not depend solely on a particular statistical aggregation method, nor can it be simply attributed to the action guard.

Critical weaknesses
  1. Section 3 identifies tasks whose majority outcomes differ across arms and then removes those whose outcomes flip across the three NO-INJECTION replicates. However, this procedure is insufficient to distinguish treatment effects from run-to-run variation: (1) A task’s instability under the control does not imply the absence of a genuine treatment effect; removing it may exclude a task that responds to the intervention. (2) A task’s outcomes remaining constant across three control replicates does not establish the absence of run-to-run variation. (3) Variation within the treatment arm may also produce differences in majority outcomes across arms. Therefore, the discriminating fraction is more appropriately interpreted as a descriptive measure under the current set of arms and limited replicates, rather than an estimate of the proportion of tasks with genuine treatment effects. Similarly, a task’s outcomes remaining constant under all tested arms does not establish that it cannot express a difference with more replicates, stronger methods, or other interventions.

  2. Table 3 shows that HIERARCHICAL includes subgoals and stop conditions, TRAJECTORY includes the source room’s observations, and ARTIFACT does not inject the concrete steps stored in the reference file. Each form also corresponds to only one deterministic serializer. Thus, the results support differences between four specific combinations of content and structure, but these differences cannot yet be independently attributed to hierarchical organization. Another key issue is that 79.3% of the lines in HIERARCHICAL are shared across bank items of the same task type, while Llama obtains similar gains from another type’s experience. This leaves open the possibility that the gains arise partly from generic procedural guidance rather than requiring the specific source experience.

  3. Table 5 divides tasks according to whether top-1 retrieval returns a same-type or other-type source, but the two groups differ in task-type composition and may also differ in difficulty and NO-INJECTION success rate. Therefore, Qwen’s larger gains in the same-type group and non-significant gains in the other-type group do not establish that same-type retrieval is necessary for a gain. A statistically significant effect in one subgroup and a non-significant effect in another do not, by themselves, establish a statistically significant difference between the subgroup effects. Similarly, Llama’s comparable gains across the two subgroups are consistent with the interpretation that this executor benefits from “structured procedural text as such,” but are insufficient to establish that mechanism.

Questions for authors

Please see Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2sjS

Thank you for the review.

Main changes for your review: the discriminating fraction is now a capacity check that no test uses (W1); Study 2 takes the hierarchical text apart and the attribution to hierarchical organisation is dropped (W2); the subgroup claim is removed after an interaction test (W3).

W1. The discriminating fraction. The revision withdraws the sentence that a constant task "cannot express an effect at any sample size". Section 3 now reads the count as a capacity check on a set of arms at a given number of replicates, and says it does not estimate how many tasks truly respond: an unstable task can respond, a stable one can still be noisy, and variation in the treatment arm also moves majorities. No test uses it. Every test is a paired difference of replicate means over all tasks, whose variance includes both arms' rerun noise (Section 3.1), so no task is removed.

W2. What carries the gain. The attribution to hierarchical organisation could not stand: the "plan" is one fixed template, the same for every task type. Study 2, designed after the reviews and frozen before its runs, uses exact subsets of the hierarchical text on three new executors (Section 6, Figure 3a). In the length-matched arms the template adds −0.1 pp over the header and the source actions +11.4 over the template; at natural length, +2.3 and +5.8 (adding filler alone lowers success by about 3 pp on two of the three executors). On generic guidance, Study 2 tests the shared text directly: the subgoal template, identical for every task, adds nothing measurable with the guard on. With the form fixed, a source of another task type loses 9.0 pp against the retrieved one, and against no injection it is −0.7 [−5.5, +3.6] on Qwen3.5 and −1.1 [−6.9, +4.9] on Llama-3.1, so the Llama pattern did not reproduce on its sibling; an oracle source gains 8.0. The content of the source matters. Two limits remain. The template adds +4.3 pp on ScienceWorld and +2.9 on ALFWorld with the guard off, so a small contribution of it is not ruled out. And the Spearman correlation between a task's action coverage and its gain is weak (about 0.2) and absent on ScienceWorld.

W3. The subgroup table. The interaction is not significant: +15.5 pp, 95% CI [−2.2, +32.8], p = 0.095. We agree that the subgroup results did not support the claim, and it is removed from the text and from the caption, which now labels the split exploratory (Section 5.3, Table 9).

The author's response solved my problem, I will revise the score.

Reviewer_XhCROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that researchers should first measure whether a benchmark can reliably detect the differences they want to study before using it to rank agent memory or skill systems.

The paper proposes several diagnostics, including the benchmark’s error rate, how often the same task changes outcome when rerun, how many tasks actually respond to the tested skill, and whether the evaluated task slice represents the full benchmark. The authors first analyze an archived protocol on ALFWorld and WebShop, where rerunning the same setup often changes outcomes and most tasks are unaffected by skill injection. They also reanalyze published aggregate results from a WebArena study.

The main new experiment is conducted on ALFWorld. The same source experience is serialized as a trajectory, script, hierarchy, or SKILL.md artifact, while the retriever, prompt, and token budget are kept fixed. Using two local executors and three replicates, the authors find that the hierarchical format performs best, while the raw trajectory is often indistinguishable from no memory. The paper also studies retrieval quality, filler controls, task-slice bias, and an action-guard ablation.

Critical strengths

The paper raises an important practical issue: a benchmark may be too noisy or too insensitive to support the rankings reported from it. The proposed checks for rerun instability, task responsiveness, and slice bias are easy to understand and could be useful for future agent evaluations.

The main ALFWorld experiment is carefully controlled. The authors use the same source experience, retriever, prompt, and token budget across forms, include length-matched fillers, run multiple replicates, and use paired statistical tests. The paper is also transparent about cases where the benchmark cannot support a strong conclusion.

Critical weaknesses

The experiments are careful, but they are too limited to support the paper’s broad claims. The main study uses only one benchmark (ALFWorld), one 140-task split, and two local models. The WebShop results come from an older experiment, and the WebArena results are taken from another paper rather than newly tested. This shows that the proposed checks are useful in this particular ALFWorld setup, but not that they work generally across different benchmarks, models, task sets, or memory systems. The main new contribution is therefore a useful checklist plus one case study, rather than a broadly validated evaluation framework.

The proposed “gap must be larger than the rerun floor” rule is a reasonable warning sign, but it is not yet a reliable statistical rule. It assumes that rerun errors are mostly random and that the control’s instability represents the uncertainty of the whole comparison. This may fail if the skill arm is more unstable, if the two arms are correlated, or if reruns have a systematic bias. The four skill formats also change several things at once: the actions they contain, their plans, their observations, their wording, and their length. The action guard further changes some actions, but the guard-off test covers only one form. As a result, the experiments show that these different skill packages behave differently, but they do not tell us which specific part caused the improvement.

Questions for authors
  1. Can the authors run the complete measurement protocol on a second benchmark, such as WebShop, rather than using WebShop only for archived repeatability analysis? If this is outside the paper’s scope, could the authors narrow the title and claims to a controlled ALFWorld case study?

  2. What assumptions justify comparing the treatment gap with the control arm’s rerun floor? Could the authors test the rule under simulated asymmetric and correlated rerun noise, and compare it with an uncertainty estimate that uses instability from both arms?

  3. Could the authors provide a factorial ablation that separately varies concrete actions, task-level plans, observations, and formatting? It would also be useful to rerun the complete form comparison with the action guard disabled for every arm.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XhCR

Thank you for the review.

Main changes for your review: a paired test, checked by the simulation you asked for, replaces the floor rule (Q2); Study 2 takes the forms apart, with the guard on and off (Q3); twelve executors and ScienceWorld widen the evidence, and the claims are narrowed to ALFWorld (Q1, W1).

Q1. A second benchmark or a narrower title. We ran ScienceWorld and narrowed the claims in the text; the title itself cannot change during the discussion. We ran the full protocol on ScienceWorld (117 tasks, 18 types; Section 6, Figure 3b). The hierarchical gain replicates when the two executors that pass the pre-specified inclusion rule are pooled (+11.8 pp [+6.2, +17.3]) and on Llama-3.1 alone; the per-executor test fails on gpt-oss (+3.8), on two later families the pooled gain is +7.3 and significant on neither, and action coverage predicts nothing. So the benefit of expert experience transfers in part and the ALFWorld mechanism does not. The abstract, introduction and limitations now confine the claims to ALFWorld and open-weight executors, and if the AC permits, we will add "a case study on ALFWorld" to the camera-ready title. We chose ScienceWorld over WebShop because the coverage test and the oracle source need gold paths, and in the archived WebShop runs 41 of 50 tasks never changed outcome across sixteen arms. The WebArena gaps are now bounded from the published aggregates rather than judged by the floor rule: eight of nine are undecided (Table 2).

W1. One split and two local models. Twelve executors from seven families now run the comparison, and every arm also ran on the 134 unseen-room tasks. Of the seven executors that ran the full 17-arm design, the hierarchical gain holds on six and the source actions beat the template on all seven (Figure 3a). Pooled over Study 2's executors, the gain over no injection is +15.0 pp in seen rooms and +5.7 in unseen rooms, where it is absent on Llama-3.1 (−1.5). The actions' advantage over the template holds in both (+12.3 and +10.4, positive on each executor; Appendix H).

Q2. Assumptions behind the rule. The rule assumed symmetric, independent rerun noise, and the revision retires it. Its replacement is a paired test on per-task replicate means whose variance includes both arms' rerun noise and the between-task spread (Section 3.1). Appendix I runs the simulation you suggest, including a rule that uses both arms' floors: run-level shocks inflate every task-level test and drift defeats every test, which is why each comparison runs all its arms interleaved in one batch on one server. We checked 606 pairs of replicate runs for run-wide shifts and found none. Asymmetry does occur: on SkillsBench the skills arm is 1.8 to 3 times less stable than the no-skills arm (Appendix J). Of the 24 pairwise comparisons the paired test called significant in Study 1, none reversed in Study 2; of the 41 that a single run called, 6 reversed (Table 21).

Q3. Factorial ablation and the guard. Study 2 separates actions, template and observations, and we reran all 17 arms with the guard off on all three executors (Appendix H; predictions frozen before the runs, all six hold).

ALFWorld, pooledguard onguard off
template over header−0.1+2.9 [90%: +1.1, +4.7]
source actions over template+11.4+12.8 (per executor +7.5, +14.4, +16.6)
actions among observations, against actions alone−10.9−8.4

So the guard does not produce the actions' advantage; it absorbs a small contribution of the template (+5.3 on Llama-3.1). We did not run a full factorial. The one interaction we can check, header plus actions against the full form at natural length, is within 0.6 pp. On wording and formatting, the same actions as a numbered procedure, a plain list or under the template differ by less than 1 pp. ScienceWorld has no guard.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers. The four checks and Study 1's data are as submitted; OpenReview's comparison with the submitted PDF shows every edit.

The revision in one sentence: a paired test replaces the floor rule, and a new study frozen before its runs finds that on ALFWorld the gain comes from the source task's concrete actions, not from hierarchical organisation.

1. A paired test replaces the floor rule (XhCR, 2sjS). The rerun floor is a pass-rate identity, F = pass@2 − pass^2, so it can be read off any paper that reports pass@k and pass^k. The revision tests the paired difference of per-task replicate means, whose variance includes both arms' rerun noise and the spread of the effect across tasks (Section 3.1); a simulation covers asymmetric noise, run-level shocks and drift (Appendix I). Both studies now use this one scale, so Study 1's headline gains read +15.0 and +20.0 pp instead of the majority-vote +15.7 and +19.3. No Study 1 conclusion changes.

2. The forms are taken apart (XhCR, 2sjS). Study 2 removes pieces of the hierarchical text one at a time on three new executors (Section 6, Figure 3, Table 5). Its predictions come from Study 1 and were frozen, with their analysis, in a timestamped file before any of its episodes ran; the file and its hash are in the supplement, and the guard-off amendment was also pushed to a public repository before its runs.

ALFWorld, three new executors pooledpp
subgoal template, over the header alone−0.1
source task's actions, over the template+11.4
same actions placed among the source room's observations, against the actions alone−10.9
form fixed, source of another task type−9.0

3. More executors and a second benchmark (XhCR, Fx1d). Twelve open-weight executors from seven families now run the comparison; seven of them ran the full 17-arm design. On all seven the source actions beat the template, and the hierarchical form beats no injection on six. On ScienceWorld the gain replicates when executors are pooled (+11.8 pp) but not on gpt-oss, and the action-coverage mechanism does not transfer, so the paper limits the mechanism to ALFWorld.

What the revision withdraws or limits.

  • The sentence that a constant task "cannot express an effect at any sample size"; the discriminating fraction is now a capacity check (2sjS W1).
  • "None of nine external gaps is supported": bounded from the published aggregates, eight are undecided and one is significant before correction (Table 2).
  • The attribution to hierarchical organisation: the "plan" is one fixed template, the same for every task type (Table 3).
  • The subgroup claim of the submission's Table 5, after an interaction test (2sjS W3).
  • Scope. The decomposition is stated for greedy decoding with the action guard on: at Qwen3.5's recommended sampling the actions' advantage shrinks to +3.6 pp (p = 0.058), and with the guard off the template adds +2.9 pp. The gain is smaller in unseen rooms (+5.7 against +15.0 pp). On an agent's own successful episodes, as we rendered them, no form helps.

The title is fixed during the discussion, so the narrowing is in the abstract, introduction and conclusion. The supplement has every episode, the frozen banks, the registration file and the analysis scripts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.