acceptodds
Under review as a conference paper at ICLR 2027

LIFT-Bench: Learning Investment for Future Transfer in LLM Agents

Abstract

Language agents can improve through interaction and persistent memory, but these gains incur reasoning and learning costs. As agents acquire reusable knowledge for diverse future tasks, they must decide what to learn under limited resources. Existing benchmarks typically emphasize task performance or per-task cost, providing limited support for jointly evaluating long-term learning gains and resource expenditure from a global perspective. We introduce LIFT-Bench (Learning Investment for Future Transfer) to evaluate allocation of a finite global token budget across competing learning opportunities. It contains 690 learning resources and 329 evaluation tasks across eight domains spanning code, tool use, mathematical reasoning, and local physical laws, organized by reusable knowledge units with explicit support relations and domain-specific verification. Inspection, study, practice, and memory construction share one learning budget; transfer is then evaluated with frozen memory and separate per-task inference allowances. A unified interface supports comparisons across three models and eight frameworks using domain-macro performance and return on learning-token investment. We evaluate three policies: Selective Investment for Future Transfer (SIFT), sequential learning, and no learning.Results distinguish the transfer value of learning expenditures and show that larger budgets need not improve future-task performance. LIFT-Bench measures what agents should learn, how they allocate learning budgets, and whether acquired knowledge benefits future tasks.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent benchmarks, lifelong learning, agent memory, knowledge transfer, resource allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qJFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LIFT-Bench, a benchmark with two phases. In the first phase, an agent spends a fixed token budget choosing and studying learning resources (API documentation, worked code, theorem explanations, "local physical laws", etc.) and writing a persistent memory. In the second phase, it is evaluated with that memory frozen on 329 held-out tasks across eight domains. Every model call made during learning counts against the budget, including framework overhead.

The authors compare three conditions across three models, eight agent frameworks and several budgets:

  • a selective policy (SIFT), in which the agent estimates value and a controller ranks resources;
  • a fixed-order curriculum;
  • no learning.

They report a domain-macro score and the score gain per learning token. The main findings are that SIFT's advantage depends on the model and framework, that learning can leave the agent worse off than not learning, and that studying a topic does not mean the agent can later use it.

Critical strengths

The question is interesting: what should an agent spend a limited learning budget on before it knows the tasks? As far as I know, it is not covered by existing memory or lifelong-learning benchmarks. LifelongAgentBench, MemoryArena and StreamBench all learn online, on the task stream itself. The protocol is also clean: memory is hashed and frozen before evaluation, and orchestration calls are charged to the budget.

For me, Section 6.4 is the most valuable part of the paper. In the theorem domain, SIFT covers 14 of 14 theorem units (versus 2 for SEQUENTIAL) and retrieval reaches 60/60, but solved tasks only go from 9 to 10. In another case, a correct teaching resource is compressed into a wrong memory (29,791 recorded as 7^5 instead of 31^3). Together these show nicely that acquiring knowledge is only one step toward using it.

Critical weaknesses
  1. I don't think the benchmark actually poses the allocation problem it is built around. The learning pool mirrors the test set:

    • All 65 families have test tasks.
    • 245 of the 258 knowledge units are tested.
    • The public cards show family hints and skill tags.
    • In several domains there is exactly one resource per test unit (API schemas 30/30/30, local laws 50/50).

    So there is little uncertainty about what will be asked, and almost nothing in the pool is not worth learning. In that setting, a good policy mostly needs to do two things: avoid studying duplicates, and prefer domains where a stored document directly answers a closed-book question. Neither requires much judgement about future value.

    The results are consistent with this. SIFT's clearest win over SEQUENTIAL is coverage: at 650k, SEQUENTIAL studies 28 theorem resources but covers only 2 units, and it keeps redundant resources by design.

    Without distractors, partially relevant resources, or some shift between the pool and the test distribution, I am not sure the benchmark measures much beyond deduplication and domain preference.

  2. The main metrics mostly don't reflect the allocation decision. The final score mixes five things:

    • which resources were chosen;
    • how faithfully they were written into memory;
    • whether the 3-item/600-token retrieval brings them back;
    • whether the model can apply them;
    • whether the framework adapter executes at all.

    The paper's own analysis suggests the later steps dominate:

    • theorem tasks go only 9→10/60 even at full coverage;
    • DeepSeek with LangGraph produces no execution result on 209 of 329 tasks;
    • about 24% of Qwen's budget goes to browsing and planning.

    On top of this, every cell in Table 2 and every point in Figs. 3–4 comes from a single run. Meanwhile, the differences being discussed (e.g., −0.48 for Qwen; "16 of 48 runs below no-learning") correspond to one or two tasks in the macro average.

    I would like to see allocation scored on its own, for example as the regret of the chosen set against an oracle that uses the support relations, with the learner and retriever held fixed. That should come with three references (random order, de-duplicated coverage, and learn-everything) and repeated runs.

Questions for authors
  1. How close does a simple non-agentic policy get to SIFT, e.g., de-duplicated coverage that prefers API/code domains? If it matches, what is left for the agent to decide?
  2. How much of SIFT's behavior comes from the hand-set ranking rule in Eq. 5 versus the agent's own estimates? For example, what happens if the controller runs with only its default values?
  3. Are the baselines too naive? StreamBench and Evo-Memory, and classic work on budgeted learning and curriculum selection (e.g., Graves et al., ICML 2017), seem relevant. They could also suggest stronger baselines than the fixed-order curriculum.
Reviewer_8eUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a worthwhile problem: what should an agent learn before its future tasks are revealed? Given an agent with limited learning budget, the authors turn its potential decisions into an executable environment, LIFT-Bench, which contains 690 learning resources and 329 evaluation tasks across eight domains. The paper evaluates the agent on three conditions: no learning, sequential learning, and SIFT. The experiments indicate that the value of learning depends on selecting useful knowledge, retaining it correctly, and applying it successfully. Meanwhile, spending more tokens or covering more resources does not necessarily improve future-task performance.

Critical strengths
  1. This paper provides a reasonable and distinct problem formulation. Existing benchmakrs already study experience accumulation, memory use across tasks, and how to allocate memory and inference resources after an incoming task is available. While the proposed LIFT-Bench moves the allocation decision earlier: heterogeneous learning opportunities compete for a shared budget before future tasks are revealed.
  2. The paper construct a relatively careful separation between learning and evaluation. The evaluation uses frozen memory and fresh sessions, and each model-framework pair has a matched no-learning baseline. To some extent, these choices address several straightforward sources of unfair comparison.
Critical weaknesses
  1. It is unclear how much of SIFT’s advantage comes simply from avoiding redundant learning. Sequential learning uses a fixed seed and studies related resources consecutively. This design can make the baseline particularly inefficient combined with repeated resources in the opportunity pool. Therefore, does SIFT benefit from accurately estimating future learning value or simply prevents repeatedly studying closely related material?
  2. According to the experiment results, the final score reflects resource selection, memory quality, reasoning, and framework execution. It is difficult to assess whether it measures learning-allocation quality reliably.
  3. The experiments use a single ordering seed, and there is no confidence intervals. Moreover, the experiments use independent budget initialization. This cannot establish that continued learning leads to performance deterioration along the same trajectory.
  4. In addition, the framework execution can become a major bottleneck. For example, in the Deepseek-LangGraph 300k SIFT configuration, 209/329 tasks have no execution results compared with 16/329 for GLM under the same graph structure.
Questions for authors

Described in weaknesses above.

Reviewer_FoWSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LIFT-Bench, which quantifies an Agent's "learning investment" as an investment decision under a limited global token budget: before facing unknown future test tasks, it measures the agent's ability to select topics from a heterogeneous resource pool, learn, practice, write into long-term memory, and measure future transfer. The evaluation covers three models, three strategies, eight frameworks, and three budgets, using domain macro-average score, RoTI, and budget average score to measure the model's "learning investment ability".

Critical strengths

1.The issues raised in this article have a strong degree of distinction from existing works, possess certain practicality and novelty, and have considerable value in the long-term learning scenarios of agents. The paper differs from previous Benchmark works by treating "learning investment" as an independent research indicator. It uses a relatively complete and comprehensive engineering implementation to evaluate how the model selects learning resources and measures future transfer under a limited global learning budget.

2.The experiment fully exposed the key phenomenon that "learning does not guarantee positive transfer". The paper designed three strategies: SIFT, sequential learning, and no learning. Through multiple experiments, the core argument was verified. The experimental analysis perspectives were diverse, and all could be centered around the same core argument. Among them, further positioning the failure of learning to the application stage after memory retention made the experimental conclusion more diagnostic.

Critical weaknesses

1.The design concept of the LIFT-Bench is insufficient to prove that it can assess the learning investment of large models. The paper's definition and analysis of the concept of "learning investment ability" are inadequate, and it specifically focuses on "scores under the protocol", but does not define what observable components constitute it, what it should be related to, and what it should not be related to. In other words, this Bench measures the outcome of the investment, but does not involve the quality of the investment judgment.

2.Defects of the Experiment. Table 2 reports the scores of the bench, but does not provide further analysis. For instance, when the model is Qwen3.8-27B and the budget is 300k, the SIFT scores are all lower than Seq., and even some scores are lower than No learning. The paper attributes this to "Larger budgets can amplify allocation errors." However, for other models, the SIFT scores generally increase significantly at a 300k budget, and the paper also attributes this to "Selective learning benefits from addressable information gaps." Even though these are two different explanations, they do not reconcile with each other, raising doubts: Why do different models arrive at different conclusions under the same budget and configuration? Further, it is questionable whether it is the characteristics of the model/harness that cause this, or whether the design of LIFT-Bench itself may have problems?

Questions for authors

1.There is doubt about the measurement indicator of "learning investment ability". It seems that there is no relevant theory or experimental evidence to prove that the score of this LIFT-Bench reflects this "learning investment ability" straightly. The paper merely provided a general definition and then proceeded to quantify the indicators, but it did not explain why this was done. This does not convince me. It is suggested to introduce relevant analysis and provide specific explanations.

2.Should the experiment be further analyzed, and supplementary ablation experiments or other experiments be conducted to support the previous methodological conclusions? (For example, for the construction of SIFT, ablation experiments could be carried out.) And should the design structure of LIFT-Bench be examined to determine if there are any problems or flaws?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.