- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes LIFT-Bench, a benchmark with two phases. In the first phase, an agent spends a fixed token budget choosing and studying learning resources (API documentation, worked code, theorem explanations, "local physical laws", etc.) and writing a persistent memory. In the second phase, it is evaluated with that memory frozen on 329 held-out tasks across eight domains. Every model call made during learning counts against the budget, including framework overhead.
The authors compare three conditions across three models, eight agent frameworks and several budgets:
- a selective policy (SIFT), in which the agent estimates value and a controller ranks resources;
- a fixed-order curriculum;
- no learning.
They report a domain-macro score and the score gain per learning token. The main findings are that SIFT's advantage depends on the model and framework, that learning can leave the agent worse off than not learning, and that studying a topic does not mean the agent can later use it.
Critical strengths
The question is interesting: what should an agent spend a limited learning budget on before it knows the tasks? As far as I know, it is not covered by existing memory or lifelong-learning benchmarks. LifelongAgentBench, MemoryArena and StreamBench all learn online, on the task stream itself. The protocol is also clean: memory is hashed and frozen before evaluation, and orchestration calls are charged to the budget.
For me, Section 6.4 is the most valuable part of the paper. In the theorem domain, SIFT covers 14 of 14 theorem units (versus 2 for SEQUENTIAL) and retrieval reaches 60/60, but solved tasks only go from 9 to 10. In another case, a correct teaching resource is compressed into a wrong memory (29,791 recorded as 7^5 instead of 31^3). Together these show nicely that acquiring knowledge is only one step toward using it.
Critical weaknesses
-
I don't think the benchmark actually poses the allocation problem it is built around. The learning pool mirrors the test set:
- All 65 families have test tasks.
- 245 of the 258 knowledge units are tested.
- The public cards show family hints and skill tags.
- In several domains there is exactly one resource per test unit (API schemas 30/30/30, local laws 50/50).
So there is little uncertainty about what will be asked, and almost nothing in the pool is not worth learning. In that setting, a good policy mostly needs to do two things: avoid studying duplicates, and prefer domains where a stored document directly answers a closed-book question. Neither requires much judgement about future value.
The results are consistent with this. SIFT's clearest win over SEQUENTIAL is coverage: at 650k, SEQUENTIAL studies 28 theorem resources but covers only 2 units, and it keeps redundant resources by design.
Without distractors, partially relevant resources, or some shift between the pool and the test distribution, I am not sure the benchmark measures much beyond deduplication and domain preference.
-
The main metrics mostly don't reflect the allocation decision. The final score mixes five things:
- which resources were chosen;
- how faithfully they were written into memory;
- whether the 3-item/600-token retrieval brings them back;
- whether the model can apply them;
- whether the framework adapter executes at all.
The paper's own analysis suggests the later steps dominate:
- theorem tasks go only 9→10/60 even at full coverage;
- DeepSeek with LangGraph produces no execution result on 209 of 329 tasks;
- about 24% of Qwen's budget goes to browsing and planning.
On top of this, every cell in Table 2 and every point in Figs. 3–4 comes from a single run. Meanwhile, the differences being discussed (e.g., −0.48 for Qwen; "16 of 48 runs below no-learning") correspond to one or two tasks in the macro average.
I would like to see allocation scored on its own, for example as the regret of the chosen set against an oracle that uses the support relations, with the learner and retriever held fixed. That should come with three references (random order, de-duplicated coverage, and learn-everything) and repeated runs.
Questions for authors
- How close does a simple non-agentic policy get to SIFT, e.g., de-duplicated coverage that prefers API/code domains? If it matches, what is left for the agent to decide?
- How much of SIFT's behavior comes from the hand-set ranking rule in Eq. 5 versus the agent's own estimates? For example, what happens if the controller runs with only its default values?
- Are the baselines too naive? StreamBench and Evo-Memory, and classic work on budgeted learning and curriculum selection (e.g., Graves et al., ICML 2017), seem relevant. They could also suggest stronger baselines than the fixed-order curriculum.