- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies memory extraction for assistants serving heterogeneous tasks. It builds the BEHEMOTH benchmark from 18 existing datasets and proposes CluE, which clusters training examples, analyses each cluster separately and merges the insights into a single extraction prompt.
Critical strengths
(1) Clear problem definition.
Scoring extraction by downstream utility makes different extraction prompts directly comparable.
(2) Empirical finding.
The static-prompt study shows that no single extraction prompt adapts all task types.
Critical weaknesses
(1) Limited novelty.
The central mechanism of CluE is an established prompt-optimisation technique applied to a new task. For example, UniPrompt (Juneja et al., Findings of ACL 2025) already clusters training data, collects feedback per cluster and merges it into a single prompt; the paper neither cites nor compares with it.
(2) Insufficient experiments.
The evaluation disregards the several important baselines, its advantage cannot stand out. For example, simply choosing the best existing static prompt for each task category already outperforms CluE (Table 1 vs. Table 2), yet no routing baseline is evaluated.
(3) No adaptation experiments.
Despite the “self-evolving” framing, the prompt is optimised once offline and then frozen, and its ability to adapt is never tested. For example, no experiment introduces a new task category or changes the task mix after evolution; even the continual experiment in section 6.3 keeps the prompt fixed.
(4) Small sample and a single model.
With few training examples and one backbone model, gains of a few points carry little reference value. For example, evolution uses only 330 training examples, and all main results rely on Qwen3-32B as extractor, optimiser and answerer at once.
(5) Syntax errors.
Table 3 omits the standard deviation for LongMemEval, although the stated protocol repeats runs for test sets of this size. Table 5 gives no citation for MathEquationBalancer or AIME. Fig. 4 and the AlfWorld panel of Fig. 5 use truncated y-axes, which visually exaggerate small differences. The case study in Fig. 8 contradicts the evolved prompt in Fig. 19: the former credits CluE with keeping a preference mentioned only in passing, while the latter requires each preference to be supported by at least two conversation threads.
Questions for authors
(1) How does CluE compare with the highly-similar baseline UniPrompt under the same optimisation budget, and what does it add beyond cluster-wise feedback?
(2) Does CluE outperform a simple router that selects among the existing static prompts for each conversation?
(3) When a new task category appears after evolution, can the evolved prompt be updated without degrading earlier categories?
(4) How large is the variance of the overall gain across independent evolution runs with different random seeds?
We thank the reviewers for their comments. Our work addresses memory extraction across heterogeneous tasks, and self-evolving optimization under heterogeneous training distributions, where existing frameworks tend to trade off performance across task types. We briefly respond to the main concerns below.
Regarding UniPrompt, while both methods group examples before analysis, UniPrompt refines a single-task prompt using one cluster at a time, whereas CluE combines insights from all clusters into each update. This distinction matters when one prompt must serve heterogeneous tasks and is optimized only indirectly through downstream utility. We will include UniPrompt in the comparison in future versions.
Routing-based methods are complementary to our focus on prompt optimization. A routed prompt can serve as the starting point for evolution (Table 4), and a single learned prompt remains useful when one conversation contains several kinds of memory.
On data splits, 173 of 2,291 test pairs (7.6%) share a target instance with the optimization set. All methods share identical splits and settings, so the comparison remains fair, and CluE also achieves the best result on LongMemEval, a held-out dataset never seen during evolution.
We also note that the evaluation is not limited to a single model. Evolution was repeated with Gemini-3-Flash and GPT-5.4-mini as extraction backends (Appendix B.3), and results on smaller test sets are averaged over three runs.
We thank the reviewers again for their time and effort in reviewing our paper.