- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that many existing benchmarks often lack sufficient statistical evidence to reliably rank agent memory or skill representations. In particular, it is often assumed that a benchmark can express a difference of the size being reported, but it is neither proven nor it universally holds. The benchmark is designed to take care of neglected but important problem.
Critical strengths
-
The four checks (error rate, repeat-measurement floor, discriminating fraction, and slice representativeness) are natural criteria and indeed make sense for the specific case.
-
The motivation is very important. Benchmark comparisons are expensive and nowadays many advantages or disadvantages are illustrated with a small amount of differences but they are used as evidence for methodological superiority/inferiority. Thus, soundness of this method shall indeed be scrutinized with caution.
Critical weaknesses
My main concern is the extent of empirical investigation. Only three families of models (Qwen, Gemini and Llama) are used, though medium-sized models are also tested. I would like to hear the authors' justification on this matter before making the final decision on the rating.
Questions for authors
Please see Weaknesses.
Response to Reviewer Fx1d
Thank you for the review.
Model families. Study 1 used Qwen and Llama; Gemini appears only in the external web-agent study we reanalyse. We use open-weight executors because the checks need reruns of one configuration under greedy decoding with every episode released. The two closed models we piloted (gpt-6.1-sol and gpt-6.1-terra) solve 90–95% of tasks without injection, which leaves no room for any form to show an effect.
The revision adds ten executors: Qwen3.5-35B, Llama-3.1-70B and gpt-oss-120b (Study 2); gemma-4-31B (Google), Nemotron-3.5-Bolt (NVIDIA), Ministral-3-14B (Mistral) and the reasoning model Step-3.7-Flash (StepFun); and a Qwen size ladder at 4B, 8B and 27B. The first seven ran the full design of 17 arms (the forms, their pieces and their sources).
Of these seven, the hierarchical form beats no injection on six: +8.9, +8.0 and +14.4 pp on Study 2's executors, +8.2, +7.3 and +11.7 on gemma, Nemotron and Ministral. The exception is Step-3.7 (−3.3), whose no-injection reruns disagree on 21.9% of tasks against at most 3.6% on the other three; the paper reports this as a prediction that failed. On all seven the source actions beat the subgoal template. On the size ladder the gain is +19.2 and +25.4 pp at 4B and 8B, and +9.3 at 27B (seen rooms, one-sided p = 0.051). Together with the closed models' 90–95% baseline, this suggests the effect shrinks as executors get stronger; the paper states this as a limitation.