acceptodds
Under review as a conference paper at ICLR 2027

TenureBench: Benchmarking lifelong agent capability over a decade of evolving workplaces

Abstract

LLM agents are increasingly expected to work for long periods and to improve from experience over their lifetime. In real deployments, however, an agent works on its own without learning whether its answers were right, while the records it relies on keep changing. We introduce TenureBench to measure the lifelong capability of agents, that is, how an agent's capability changes as it keeps working. TenureBench replays 13,407 official documents from ten years of finance, medicine, and law to one persistent agent per field in publication order, keeps old versions next to new ones, and never shows the agent the correct answers. On a 300-task sample of its 920 tasks, the best of five frontier models solves 52.7%, and every agent is weakest overall at tracking what changed between two dates. Instead of improving over its lifetime, each agent does worse on later tasks. A long working history makes an agent more efficient, halving its search, but it does not make the agent more accurate. Given the source documents and the grading criteria without their answers, the same agent solves 93%. The agent has the ability to solve these tasks, and what it lacks is the professional sense, built over a career, of what a complete answer must cover. The time-driven protocol of TenureBench extends to any evolving archive to measure the lifelong capability of agents.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Lifelong Agents, Evolving Knowledge, Temporal Reasoning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XMsnOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TenureBench uses 13407 dated official documents from finance, decision, and law (2016-2026) to persist agent-per-field evaluation. The agent is never reset and never sees the answers. There are 920 tasks with 13318 binary rubric criteria on a 300-task sample; the best of five models solves 52.7%. Success falls over the lifetime, and as the context fills and the long history halves, tool calls without improving accuracy. This is a very high-quality benchmark with comprehensive evaluation, which makes it an important step towards real-world agents.

Critical strengths

1- A new an realsitic protocol for evaluation is presented in this paper. Old versions stay on file, every task needs several dates, and the agent persists without feedback. Table 1 positions the work clearly against prior work. 2- Real records with revision-heavy entries chosen by explicit rules, a construction funnel documented in detail, and an expert reading of every released task. 3- The analysis goes past headline numbers: per-domain intervals, same-task context fill comparisons, gold-evidence-opened mechanisms, and a controlled ablation of what a long history contributes.

Critical weaknesses

1- The human-expert comparison is not measured. In figure 2 and the professional intuitions claims imply that experts don't degrade as context grows, with no baseline. Either measure it or reframe. 2- Entity relationships and multi-date necessity are decided by LLM workers. Expert review lacks qualifications, agreement numbers, and rejection categories. The paper never measures how much of the archive is relevant per task, so the effect of a growing archive is confounded with a growing share of distractors. 3- The judge is not validated and only one single judge is judged. 4- Later tasks may be harder in themselves (more dates, more entities), and context fill varies with each model's summarization pace. A control would help, such as the same task asked at different points in the lifetime, or Fresh vs. Lifelong compared across the lifetime sixths. 5- Grading might be too strict: A task succeeds only if every criterion passes (median 9). Rubric scores (76–91%) and task success (23–53%) diverge sharply, and a failure over 54.57 vs 54.58 suggests the tolerance rules need stating.

Questions for authors

1- How well does the judge agree with human expert labels, and do rankings hold under a judge from another model family? 2- Does the lifetime decline survive asking the same tasks at different lifetime positions? 3- What are the expert reviewers' qualifications, and what did the 57 rejections share? 4- What fraction of each task's archive is relevant, and does the finance drop persist when you control for distractors?

3 comments
AuthorsOpenReview ↗

Response 1 to Reviewer XMsn

Thank you very much for your encouraging assessment and valuable suggestions. We have addressed each of the concerns you raised accordingly. Changes in the revised paper are marked in blue.

W1: Is the human-expert comparison measured?

We measured the human baseline. The experts' task success stays stable across the lifetime, at 82.4% in the first sixth and 85.7% in the last. One expert with doctoral training per domain answered the evaluated tasks of that domain with the archive as of each question date and without the reference answers, the gold evidence, or the criteria, and the same rubrics graded the answers. Overall the experts solve 84.0% of the tasks and meet 98.2% of the criteria (new row of Table 3). The table below reports their task success by sixth of the lifetime (Table E.1).

DomainSixth 123456Overall
Finance expert76.5%82.4%76.5%76.5%82.4%76.5%78.4%
Medical expert85.0%90.0%85.0%85.0%85.0%89.5%86.6%
Legal expert85.7%84.6%92.3%84.6%84.6%92.3%87.3%
All experts82.4%86.0%84.0%82.0%84.0%85.7%84.0%

W2: Are entity relationships and multi-date necessity decided by LLM workers alone?

Two models other than the writing model verify that every named entity and at least two dated states are needed, and a fixed rule rechecks the dates. The first worker only proposes a group of related entities and the dated states that connect them. Two verification workers that run other models (GPT-5.6 Luna and Grok 4.6) then rebuild the answer from the archive as of the question date. They pass a candidate only if every named entity is needed and the task needs at least two dated states. No single document, single date, later summary, or latest state may give the answer. The single-source check then recomputes the multi-date test from the evidence dates by a fixed rule (Section 4 and Appendix C).

Q3 / W2: What are the expert reviewers' qualifications and agreement, and what do the rejections share?

Six domain experts with doctoral training reviewed the candidates, and a second expert reached the same decision as the first on 95.6% of the tasks reviewed twice. Two experts each in Finance, Medical, and Legal reviewed the 977 candidates of their own field and rejected 57. A review took a median of 12 minutes (interquartile range 8 to 18). A second expert of the same field, who did not see the first decision, reviewed 114 tasks (the 57 rejected candidates and 57 matched released tasks) and made the same decision on 109 (95.6%, Cohen's κ = 0.93). A further review of the 57 released tasks with the complete source documents changed no decision (Appendix C).

The rejected candidates fall into four types. In the first, the question only asks the agent to locate and report facts, or it leaves nothing to decide beyond a lookup and arithmetic. In the second, the requested change is zero or the two dated states are identical, so one state gives the whole answer. In the third, the task repeats another task on the same entities and method. In the fourth, the criteria and the question do not match.

Q4 / W2: What fraction of each task's archive is relevant, and does the finance drop persist when distractors are controlled?

In finance, the relevant share of the archive stays between 2.9% and 4.3% while the archive nearly triples. The finance drop therefore occurs at a nearly constant share of distractors. For each evaluated task we count the documents of the entities the task names, published by its question date, over the documents in the archive at that date. The table below reports this share by sixth of the lifetime (Table B.2).

DomainSixth 123456
Finance2.9%3.9%4.0%4.0%3.3%4.3%
Medical4.4%4.8%4.4%3.7%4.6%4.3%
Legal7.2%6.1%4.5%5.5%5.0%4.3%

In finance the archive grows from a median of 3,466 documents in the first sixth to 9,415 in the last. The share of other documents therefore stays between 95.7% and 97.1%, while task success in finance falls from 56.5% in the first sixth to 31.1% in the later sixths.

AuthorsOpenReview ↗

Response 2 to Reviewer XMsn

Q1 / W3: How well does the judge agree with human experts, and do the rankings hold under a judge from another model family?

On 105 answers, the GPT-5.6 Sol judge agrees with domain experts on 94.3% of the task-success decisions (κ = 0.88). On 420 answers, it agrees with judges from two other model families on 94.8% and 93.6%, and their system rankings correlate with its own at Spearman ρ = 0.96 and ρ = 0.93. Section 5.1 and Appendix D report the study.

We rescored 420 of the 2,100 answers, 20 for each of the seven systems in each domain, with Claude Opus 4.6 and DeepSeek Flash as judges from two other model families. Each judge ran in a fresh session on the same input and did not know which system wrote an answer. The table below reports their agreement on task success.

Judge pairTask-success agreementCohen's κ
GPT-5.6 Sol vs. Claude Opus 4.694.8%0.87
GPT-5.6 Sol vs. DeepSeek Flash93.6%0.83
Claude Opus 4.6 vs. DeepSeek Flash93.1%0.81

For the comparison with domain experts, two experts per domain graded 105 of the 420 answers, five per system and domain, without seeing the system's name or any judge's verdict. The first expert graded every criterion, and the second reviewed each grade and changed 2.6% of them. Against the experts' task-level decisions, GPT-5.6 Sol agrees on 94.3% (κ = 0.88), Claude Opus 4.6 on 92.4% (κ = 0.84), and DeepSeek Flash on 91.4% (κ = 0.82). The results show that The LLM judge agrees closely with both human experts and other models, supporting the reliability of the automated evaluation.

Q2 / W4: Does the lifetime decline survive a control for later tasks being harder, and does context fill depend on each model's summarization pace?

The same task cannot be moved along the lifetime, because its question date fixes the archive it is asked on. What can change is the history the agent brings to it. Therefore, we ran the Fresh agent on all 300 evaluated tasks for the five systems. The table below reports task success by sixth of the lifetime, pooled over the five systems (the dashed blue line of Figure 5a, with each system in Figure 6).

AgentSixth 123456
Lifelong54.1%41.2%40.0%44.4%32.8%42.9%
Fresh49.8%45.6%48.8%52.4%48.8%48.6%

The Fresh agent stays between 45.6% and 52.4% across the six sixths, on the same tasks and archives as the Lifelong agent. Later tasks do span more dates and entities (Figure 5b), yet an agent without the history solves them as often as the early ones. The decline is therefore tied to the accumulated history.

For the summarization pace, Appendix A.4 uses only each system's first context window, before any summary replaces the conversation. Pooled over the five systems, success is 49.0% for tasks that arrive with three to four fifths of the window filled and 30.4% above four fifths. Section 5.4 estimates the effect of context load with task and system fixed effects, which compares the same task arriving at different loads.

AuthorsOpenReview ↗

Response 3 to Reviewer XMsn

W5: Is the all-criteria grading too strict, and what is the tolerance rule for numbers?

The rubric criteria reflect real-world requirements in finance, medicine, and law rather than being intentionally stringent. Each task is divided into multiple criteria to assess, from different perspectives, whether the answer meets professional standards. The tolerance rule is stated in the judge template.

The rubric score is the share of criteria met, while task success asks whether the answer is complete enough to be delivered. Of the 861 failed answers, 98.5% miss at least one correctness or reasoning criterion, that is, a wrong or missing fact, value, conclusion, or derivation step. A high rubric score indicates that the model satisfies most individual requirements, but does not mean that it can consistently produce a complete final answer with accurate facts, sound reasoning, and adequate citations.

Significant gaps between ask success rates and partial-credit scores are also common in complex agent benchmarks. For example, the gap exceeds 30 pp+ in Agents' Last Exam [1] and 50 pp+ in CL-Bench [2]. Therefore, the gap between overall task success and rubric scores should not be attributed solely to overly strict evaluation.

The judge template (Figure H.22) defines the two verdicts. MET means fully and correctly satisfied, NOT_MET covers an omitted, incomplete, imprecise, unsupported, or contradicted answer, and there is no partial credit. A criterion that allows a tolerance says so in its wording, and a criterion that states a number is met only by that number.

In Appendix Figure I.5, the expected share of 2025 revenue from Filtration is 54.58%, whereas Claude Sonnet 5 reports 54.57%. The paper therefore marks Claude Sonnet 5 as failing this task. In this case, total 2025 product revenue is approximately 738 million dollars, so a difference of 0.01 percentage points corresponds to about 73,800 dollars. Although small relative to total revenue, such a discrepancy can create downstream issues in financial audits, regulatory filings, and related processes. Finance, medicine, and law all demand high accuracy, although the risks and material consequences of numerical errors vary by context. Our strict criteria are grounded in real-world requirements. In finance, monetary amounts, ratios, and financial statements must be reconcilable, reproducible, and internally consistent. In medicine, values such as dosage, concentration, administration frequency, and contraindication thresholds can directly affect safety. In law, effective dates, deadlines, eligibility thresholds, and statutory amounts often require exact precision; a one-day difference or crossing a threshold may change the legal conclusion.

We hope these clarifications and explanations address your concerns. If you have any further questions, we would be happy to discuss them with you.

References

[1] Sun et al. Agents' Last Exam. arXiv:2606.05405, 2026.

[2] Dou et al. CL-bench: A Benchmark for Context Learning. arXiv:2602.03587, 2026.

Reviewer_eczNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets to solve the question on does an agent improve as it accumulates experience over a long operational lifetime? It introduces TenureBench, a benchmark for evaluating the lifelong capabilities of LLM agents operating in evolving information environments. TenureBench maintains persistent agent histories while introducing documents chronologically, preserving historical versions.

The benchmark replays 13,407 official documents from ten years in the domains of finance, medicine, and law to one persistent agent per field in publication order. The authors evaluate five frontier models using a shared agent framework on a stratified sample of 300 tasks. The best-performing model achieves 52.7% task success, with change tracking emerging as the weakest capability. Performance generally declines over longer operational histories as document archives and context windows grow.

Critical strengths
  1. Novel and realistic lifelong agent evaluation setting. The paper introduces a benchmark that combines agent histories, chronological documents, historical versions and multi-state reasoning. This setting reflects agent deployments in real world with evolving enviroments better compared to conventional benchmarks that reset agents between task to task.
  2. Benchmark construction is comprehensive. The benchmark includes 13407 official documents spanning 10 years across different domains. The use of real-world, evolving records and the multi-stage task verification process strengthens the benchmark’s practical relevance.
Critical weaknesses
  1. Reduced tool usage does not necessarily indicate improved efficiency. The paper reports that lifelong agents require approximately half as many tool calls as fresh agents and interprets this as an efficiency improvement. However, several factors complicate this conclusion. 1 tool-call counts alone do not capture overall computational cost. Lifelong agents maintain longer conversation histories, which potentially increasing input-token consumption and inference latency. Although the framework records token usage, the paper does not provide a direct token cost or latency comparison between fresh and lifelong agents. 2. fewer tool calls may reflect less evidence retrieval rather than more efficient information reuse. Lifelong agents open a smaller proportion of gold evidence documents (38.8% versus 55.9%) and more frequently cite documents that were not opened during the current task (8.1% versus 1.1%). While some citations may rely on information retrieved in earlier tasks, the results do not establish whether this behavior represents effective reuse or insufficient evidence verification.
Questions for authors

Doe the decline over the lifetime survive regression that controls the number of evidence datas and domain? Does persistent interaction history provide measurable benefits beyond retrieving relevant historical documents on demand?

1 comment
AuthorsOpenReview ↗

Response to Reviewer eczN

Thank you very much for your encouraging assessment and valuable suggestions. We have addressed each of the concerns you raised accordingly. Changes in the revised paper are marked in blue.

W1: Do fewer tool calls mean a lower computational cost?

A long history lowers tool calls and raises the tokens read from cache, by 2.48 to 13.54 times across the seven system configurations. The revised paper therefore reports the result as a reduction in tool calls. The table below compares the Lifelong and the Fresh agent on uncached input, cache-read (the earlier conversation read back from the cache), reasoning, and output tokens, in millions. We compare tokens rather than latency because wall-clock time in these runs depends on network conditions and on the load of each provider's service. Both varied from call to call, so latency would not give a fair comparison.

SystemConditionUncached inputCache readReasoningOutput
GPT-5.6 SolLifelong0.006814.880.2851.138
Fresh0.007113.460.3521.073
Claude Sonnet 5Lifelong0.0061571.841.2732.425
Fresh0.013316.431.3861.885
Grok 4.6Lifelong8.74453.020.4471.141
Fresh25.02118.331.2612.374
DeepSeek FlashLifelong102.241224.262.7203.939
Fresh33.59492.864.0305.893
GLM-5.3 lowLifelong21.181367.780.9781.637
Fresh13.10101.010.6121.203
GLM-5.3 highLifelong31.011782.212.3773.155
Fresh17.86192.441.9712.856
GLM-5.3 maxLifelong30.211800.152.6053.482
Fresh18.07196.651.9642.843

The Lifelong agent reads more from cache on every configuration, and the other three token counts move in both directions. The Abstract, Introduction, Section 5.5, and Conclusion now state the measured result, that a long history reduces tool calls.

W2: Do fewer tool calls reflect reuse of earlier reading or less verification?

The Lifelong agent reuses what it read. Of the documents it cites, 18.0% were opened only in an earlier task, and its answers meet as many criteria as the Fresh agent's, 89.0% against 89.5% (Figure 8a).

On the same 60 tasks, GLM-5.3 solves 33 as the Lifelong agent and 31 as the Fresh agent (exact McNemar test, p = 0.82), with a median of 13.5 tool calls against 27. Three models with three runs each repeat the pattern. The Lifelong agent opens 38.8% of the gold evidence within the current task against 55.9% for the Fresh agent, which is consistent with having read those documents in earlier tasks. Figure 8(c) gives the origin of every cited document, including the 8.1% for the Lifelong agent and 1.1% for the Fresh agent that were neither opened nor matched by a search in the session. Reuse saves search without raising accuracy in this comparison, and pooled over the five systems the Lifelong agent falls behind the Fresh agent after the first sixth (see Q1).

Q1: Does the lifetime decline survive a regression that controls the number of evidence dates and the domain?

The decline is tied to the history the agent carries. Fresh agents asked the same tasks do not decline, and in the regression, lifetime position and the number of evidence dates rise together and cannot be fully separated. We fit a logistic regression of task success on lifetime position with domain and system indicators, over the five systems and the 300 tasks, with 95% intervals from a bootstrap over tasks.

ModelLifetime positionEvidence dates (per date)
Domain and system only−0.62 [−1.23, −0.11]
With the number of evidence dates−0.27 [−0.90, 0.27]−0.20 [−0.31, −0.11]

Tasks late in a lifetime span more dates, 3.6 on average in the first sixth and 6.0 in the last, and once the number of dates is controlled, the 95% interval for lifetime position includes zero. A direct control separates the two. Pooled over the five systems, Fresh agents asked the same 300 tasks, with the same dates to reconcile, solve 49.8% in the first sixth and 48.8% in the later sixths. The Lifelong agents fall from 54.1% to 40.2%.

Q2: Does a persistent history bring a measurable benefit beyond retrieving documents on demand?

The measurable benefit of the persistent history is fewer tool calls, and for one system higher task success. The Fresh agent is the on-demand condition, since it has the same archive and retrieves what each task needs without any history. Over three models and three runs each, the Lifelong agent makes 49.4% fewer tool calls on average (Table A.5). On the 300 evaluated tasks, GPT-5.6 Sol also solves more tasks with its history than without it, 52.7% against 45.0%, the only one of the five systems to do so (Figure 6).

We hope these clarifications and explanations address your concerns. If you have any further questions, we would be happy to discuss them with you.

Reviewer_DAs7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TenureBench, a benchmark for evaluating LLM agents in long-term deployments where they ingest information over time and solve tasks using the full archive of information over time. The goal is to evaluate if agents are able to learn from experience and deal with changing information over time. The benchmark consists of tasks from finance, medicine, and law. Each full episode is based on a corpus of dated documents relevant to the current field, and each document is fed to the agent according to the date in the trajectory. Tasks are generated by LLM agents and inserted at points within the episode. Solving a task requires retrieving a set of documents from the current document archive, and the agent only has access to the documents up to that point. Each task is designed with a rubric (not visible to the agent) that is used to grade task performance by an LLM judge. The benchmark is evaluated on five frontier models, all run within a custom harness that provides basic file search and read/write capabilities. The evaluation demonstrates that even these frontier models have significant gaps in being able to fully solve this benchmark.

Critical strengths

The benchmark’s setting is well-defined and a valuable area of study, considering the increasing adoption of long-term / lifelong LLM agents. The benchmark’s scale is substantial, so the claim that the benchmark evaluates lifelong capabilities is well-supported.

Critical weaknesses
  • The paper’s main lifelong claims are not well supported. The finding from Figure 1 that a long history improves efficiency but not accuracy is based on just a single run with one model. The decline over the lifetime is also not properly isolated from task difficulty, since later tasks could likely be more difficult due to having to find evidence from a larger archive. Could the fresh and lifelong agents be compared on early and late tasks separately? If the fresh and lifelong agents have similar levels of decline for later tasks, it would mean the benchmark is not really measuring lifelong capability.
  • A task only counts as solved if every rubric criterion is met. Tasks that require more evidence likely also have more criteria, so they would be much easier to fail if any single criterion is failed, even for minor mistakes. Could success be reported against the number of criteria, and the lifetime trends be shown using the rubric score?
  • Agents are never properly compared against human experts, and the Fresh+sources+criteria agent cannot serve as the sole point of reference since The 93% success when the agent ‘s 93% performance (Figure 1) could suggest that the benchmark contains tasks that are underspecified or not possible to solve. Could an expert baseline be provided on a subset of tasks?
  • The benchmark draws from public corpuses, so the evaluated models may already have been trained on these corpuses. Could results be shown on some corpus published after the model cutoff dates, or could the existing corpuses be trimmed to only contain content after the cutoff dates?
  • The paper claims that domain experts review every task but provides no details beyond that experts accept/reject each task. How many experts were recruited, what is their general background (without breaking anonymity), and how long does each review take? Each task is also read by a single expert who does not see the source documents, so could an independent second review of a random sample be provided to estimate the error rate, and can another review be done with access to the source documents?
  • Many details are still missing regarding the LLM workers. The paper specifies only the model families and not which models and settings were used at each step or how many tasks each produced. Are any measures taken to prevent bias? Could results be reported separately for tasks generated by each worker family?
  • Evaluation is performed solely using a GPT-5.6 Sol LLM judge, and there is no deterministic evaluation of the task outputs. The judge is only validated on six case studies. Can a study be provided on the accuracy of the judge on a larger subset of tasks, compared to domain experts and to a judge from a different model family?
  • The evaluation is conducted using a single harness adapted from an existing coding agent. It is not clear if results reflect model performance or each model’s fit with the specific harness. Could an ablation study over the model harness be provided, especially with more traditional harnesses like Claude Code?
  • Table 1 does not seem to represent the related work accurately. I believe DialSim meets the paper’s own definition of persistence, and it includes dated multi-hop questions. StreamingQA keeps updating the same models as new articles arrive, rather than starting fresh for each question. Fin-RATE tracks companies across years of SEC filings but is not included in the table.
Questions for authors

N/A

7 comments
AuthorsOpenReview ↗

Response 1 to Reviewer DAs7

Thank you very much for your valuable suggestion. We have addressed each of the concerns you raised accordingly. Changes in the revised paper are marked in blue.

W1: Does the Figure 1 finding hold beyond one run of one model, and do the Fresh and Lifelong agents differ on early and late tasks?

Three models with three runs each reproduce the Figure 1 result. In addition, a Fresh agent asked the same 300 tasks does not decline over the lifetime, whereas the Lifelong agent does.

We ran GLM-5.3, GPT-5.6 Sol, and Grok 4.6 at high reasoning effort as the Lifelong and the Fresh agent on the same 60 tasks, three runs per model and condition. The table below reports the mean and standard deviation over the three runs (Table A.5). Tool calls are the mean number of calls per task.

ModelFresh SuccessLifelong SuccessΔ Success (pp)Fresh Tool CallsLifelong Tool CallsΔ Tool Calls
GLM-5.352.2% ± 1.0%54.4% ± 1.0%+2.227.1 ± 0.713.6 ± 0.2−49.9%
GPT-5.6 Sol55.6% ± 1.0%58.89% ± 2.1%+3.2925.4 ± 0.614.7 ± 0.5−42.1%
Grok 4.650.6% ± 1.0%51.7% ± 1.7%+1.128.8 ± 0.612.9 ± 0.4−55.2%
Average52.8%55.0%+2.227.113.7−49.4%

On every model, the Lifelong agent makes 42.1% to 55.2% fewer tool calls, while its task success is 1.1 to 3.29 percentage points higher than the Fresh agent's.

To compare early and late tasks, we ran the Fresh agent on all 300 evaluated tasks for the five systems. The table below reports task success (%) in the first sixth of the lifetime and in the later sixths (Table A.7, with the curves in Figures 5a and 6).

SystemLifelong, first sixthLifelong, laterFresh, first sixthFresh, later
GPT-5.6 Sol62.750.645.145.0
GLM-5.356.947.845.150.2
Grok 4.668.641.464.754.6
DeepSeek Flash51.040.660.857.0
Claude Sonnet 531.420.933.337.3
Five systems pooled54.140.249.848.8

The two agents do not decline alike. Pooled over the five systems, the Fresh agent solves 49.8% of the tasks in the first sixth and 48.8% in the later sixths, whereas the Lifelong agent falls from 54.1% to 40.2%. On every system, the Lifelong agent drops more than the Fresh agent (between −10.1 and +5.1). The decline therefore comes with the history the agent carries, which is what TenureBench is designed to measure.

W2: Does the all-criteria rule make tasks with more criteria fail more often, and does the lifetime trend hold under the rubric score?

The all-criteria rule does not produce the lifetime trend. Every task is graded by the same rule, and the number of criteria a task carries is unrelated to its position in the lifetime (Spearman ρ = 0.01 over the 300 evaluated tasks).

The table below reports task success against the number of criteria, pooled over the five systems (Table A.6).

CriteriaTasksTask successIf criteria failed independently
4 to 66154.8%44.0%
7 to 88446.7%33.3%
9 to 125137.6%23.8%
13 to 203941.0%11.1%
21 to 686530.8%2.5%

Tasks with more criteria are solved less often, but far more often than they would be if each criterion failed independently.

The next table reports the lifetime trend under both measures.

Lifetime sixth123456
Task success54.1%41.2%40.0%44.4%32.8%42.9%
Rubric score86.2%88.1%87.8%85.8%79.6%83.3%

The pooled rubric score changes little because the 133 tasks with more than nine criteria account for 73% of all criteria, and their rubric score is 87.2% both in the first sixth and later. Among the 167 tasks with at most nine criteria, both measures fall from the first sixth to the later sixths, with task success falling from 61.3% to 44.6% and the rubric score from 84.8% to 78.9%.

W3: Can a human expert baseline show that the tasks are clear and solvable?

Human experts solve 84.0% of the evaluated tasks and meet 98.2% of the criteria, against 52.7% and 90.6% for the best agent. One expert with doctoral training per domain answered the evaluated tasks of that domain with the archive as of each question date and without the reference answers, the gold evidence, or the criteria. The same rubrics and judge graded their answers (the new Human experts row of Table 3).

SystemTask success: AllFinanceMedicalLegalRubric score: AllCorr.Reas.Cit.
Human experts84.078.486.687.398.298.097.398.7

The experts score above every system in all eight columns, which indicates that the task statements and criteria are clear enough for a specialist to satisfy.

AuthorsOpenReview ↗

Response 2 to Reviewer DAs7

W4: Can results be shown on a corpus published after the model cutoff dates?

A corpus restricted to documents published after the model cutoffs would be too short for this benchmark, because every task compares dated states that lie years apart and each history spans ten years. In addition, we evaluate every model on the same fixed test set for three reasons.

First, a fixed test set is the standard model-independent protocol of benchmarks such as MMLU [1], LongBench [2], L-Eval [3], MMMU [4], and MathVista [5]. Providers of closed models do not disclose a common definition of the cutoff, and a filter by cutoff would evaluate different models on different tasks. Moreover, every TenureBench task is newly written and needs at least two dated states from several official records. Exposure to the raw records alone does not supply the answer.

Second, studies of contamination find that overlap with pretraining data rarely changes the ranking of models. Li et al. [6] find minimal effects on MMLU across more than 15 models, Sánchez Salido et al. [7] find a Pearson correlation of 0.98 between model rankings on unpublished exams and on public MMLU, and the GPT-4 Technical Report [8] finds small differences of inconsistent sign after removing possibly contaminated items.

Third, a split by cutoff date is not a clean control. Zhang et al. [9] analyze eight models over 26 months and find no systematic drop after the cutoff on questions constructed by language models, while changing only the wording of a question makes the gap appear, shrink, or vanish.

W5: Who reviewed the tasks, and do a second review and a review with the source documents confirm the decisions?

Six domain experts with doctoral training reviewed the 977 candidates. A second expert, blind to the first decision, reached the same decision on 109 of 114 tasks (95.6%, Cohen's κ = 0.93), and a further review with the complete source documents changed none of the 57 decisions (Section 4 and Appendix C).

Two experts each in Finance, Medical, and Legal reviewed only the tasks of their own field. For each candidate, one expert read the task statement, the reference answer, the evidence with its quoted source passages, the criteria, and the rubric-audit verdict, and labeled the task agree, disagree, or unsure. The experts rejected 57 candidates and kept the 920 released tasks. A review took a median of 12 minutes (interquartile range 8 to 18).

For the second review, an expert of the same field who had not seen the task or the first decision reviewed 114 tasks with the same materials. These are the 57 rejected candidates and 57 released tasks matched to them by domain, question date, and construction batch. Of the 57 released tasks, the second expert agreed on 55, disagreed on one, and was unsure on one, and the authors and both experts found no error in either task.

For the review with the source documents, the second expert then received the complete source documents of the 57 released tasks and reviewed them again. No decision changed.

W6: Which model and settings ran each construction step, and how is bias from the generating model prevented?

GPT-5.6 Sol writes every task and its rubric, while GPT-5.6 Luna and Grok 4.6 each check every candidate. Each task is thus checked by two models other than the one that wrote it (Table C.1).

StepWorkerReasoning effort
GenerationGPT-5.6 Solhigh
VerificationGPT-5.6 Luna, Grok 4.6max, xhigh
Rubric writingGPT-5.6 Solhigh
Rubric auditGPT-5.6 Luna, Grok 4.6max, xhigh
Answer-leak checkGPT-5.6 Luna, Grok 4.6max, xhigh

For every worker we set the model and its reasoning effort and leave the other sampling parameters at the provider defaults. Each worker starts in a fresh session, reads only the files on its allowlist, and has no editing, shell, or web tools. GPT-5.6 Sol generated every candidate, and both checking models ran on every candidate that reached their step. Table C.2 gives the count at each step, from 2,118 candidates to the 920 released tasks.

To limit bias from the writing model, verification rebuilds the answer from the archive as of the question date, the rubric audit checks every criterion against the full archive, and the answer-leak check sees only the task statement. Deduplication and the single-source check are fixed rules, and a domain expert reviews every remaining task.

Results cannot be split by worker family, because one model writes all 920 tasks and the two other models check all of them.

AuthorsOpenReview ↗

Response 3 to Reviewer DAs7

W7: How accurate is the judge against domain experts and against judges from other model families?

On 420 answers the GPT-5.6 Sol judge agrees with judges from two other model families on 94.8% and 93.6% of the task-success decisions, and on 105 answers it agrees with domain experts on 94.3% (κ = 0.88). Section 5.1 and Appendix D report the study.

We rescored 420 of the 2,100 answers, 20 for each of the seven systems in each domain, with Claude Opus 4.6 and DeepSeek Flash as judges from two other model families. Each judge ran in a fresh session on the same input and did not know which system wrote an answer. The table below reports their agreement on task success.

Judge pairTask-success agreementCohen's κ
GPT-5.6 Sol vs. Claude Opus 4.694.8%0.87
GPT-5.6 Sol vs. DeepSeek Flash93.6%0.83
Claude Opus 4.6 vs. DeepSeek Flash93.1%0.81

The system rankings under Claude Opus 4.6 and under DeepSeek Flash correlate with the ranking under GPT-5.6 Sol at Spearman ρ = 0.96 and ρ = 0.93.

For the comparison with domain experts, two experts per domain graded 105 of the 420 answers, five per system and domain, without seeing the system's name or any judge's verdict. The first expert graded every criterion, and the second reviewed each grade and changed 2.6% of them. Against the experts' task-level decisions, GPT-5.6 Sol agrees on 94.3% (κ = 0.88), Claude Opus 4.6 on 92.4% (κ = 0.84), and DeepSeek Flash on 91.4% (κ = 0.82).

W8: Do the results reflect the models or their fit to the one harness?

The findings reflect differences in model capabilities rather than how well a particular model is adapted to a specific Harness. All models operate under the same fixed Harness, controlling factors such as system prompts and context management strategies to ensure comparability across models. In contrast, mature Agent products such as Claude Code incorporate their own system prompts, tool selection mechanisms, and other internal designs that are not fully disclosed. This makes it difficult for researchers to ensure consistency in experimental variables.

The harness is built on the open-source pi agent [10] at a pinned release. It gives every model the same six file tools, the same system prompt, and the same compaction rule, with no shell, no internet, and no subagents. Section 3 now states: "The prompt, the tools, the replay protocol, and the compaction rule are held fixed, so a difference between two systems comes from the model."

Three results support this.

  1. The five models come from five providers. All five score lower in the last sixth of the lifetime than in the first, and change tracking is the weakest capability of every one.
  2. Section 5.5 compares the Lifelong and the Fresh agent with the model and the harness both fixed, and the Lifelong agent makes about half the tool calls on all three models tested.
  3. Appendix A.4 uses only each system's first context window, before any compaction. Pooled over the five systems, success is 49.0% for tasks that arrive with three to four fifths of the window filled and 30.4% above four fifths. The drop with context fill therefore does not depend on the harness's compaction.

W9: Does Table 1 represent DialSim, StreamingQA, and Fin-RATE accurately?

Table 1 now marks DialSim as fully Persistent, and Section 2 states how StreamingQA and Fin-RATE differ from TenureBench.

DialSim [11] meets our definition of Persistent, because it keeps the agent's memory across dialogue sessions and does not show it the correct answers. Its Multi-state entry stays partial, because questions that span several dated sessions are one of its question types and not a requirement on every task.

StreamingQA [12] updates the model or its retrieval index as articles arrive, and it asks each question on its own. Section 2 now reads: "StreamingQA updates the model or its index as articles arrive, but it asks independent questions and keeps no agent state from one question to the next." The Introduction now says that QA benchmarks treat each question independently.

Fin-RATE [13] is cited in Section 2 with FinanceBench [14] and LegalBench [15] among the professional benchmarks over a fixed set of documents. Its Longitudinal Tracking questions draw on SEC filings from several reporting periods, which is Multi-state in part. However, it does not disclose documents in publication order to one persistent agent or keep superseded versions beside newer ones, and it evaluates each question independently. We therefore discuss it with the domain benchmarks in the text.

We hope these clarifications and explanations address your concerns. If you have any further questions, we would be happy to discuss them with you.

Comments on the paper

2 comments
Area_Chair_72aBOpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, AC, and SAC,

As the discussion period closes, we thank all reviewers for their careful and constructive reviews. We are pleased to see that reviewers describe our setting as “well-defined and a valuable area of study” (DAs7) and “Novel and realistic” (eczN), and TenureBench as “a very high-quality benchmark” (XMsn). We thank the reviewers for recognizing the value and potential of this work.

The discussion examined four aspects of the benchmark, namely its motivation, its main finding, its evaluation, and its task quality. The evidence on each confirms the paper's design and findings. Details are in our replies.

  1. The tasks are solvable, and agents fall well short of human experts. (DAs7, XMsn) Domain experts solve 84.0% of the tasks without the reference answers or the criteria, against 52.7% for the best agent (Table 3). Their success holds steady across the lifetime (Table E.1), while each agent does worse on later tasks.
  2. The lifetime decline is tied to the history the agent carries. (DAs7, XMsn, eczN) On the same tasks, pooled over the five systems, a Fresh agent with no history stays level (49.8% in the first sixth of the lifetime, 48.8% later), while the Lifelong agent falls from 54.1% to 40.2% (Table A.7).
  3. The LLM judge agrees with domain experts. (DAs7, XMsn) On task success, the agreement is 94.3% with domain experts and similar with judges from two other model families (Appendix D).
  4. The task review is consistent between experts. (DAs7, XMsn) A second expert, blind to the first decision, reached the same decision on 109 of 114 tasks, and a further review with the complete source documents changed no decision (Appendix C).

TenureBench makes three contributions.

  1. A professional dataset spanning ten years. From 13,407 official documents on 150 entities in finance, medicine, and law, we build 920 tasks with 13,318 rubric criteria through multi-stage verification and expert review.
  2. A lifelong evaluation protocol. One agent keeps working as documents arrive in publication order, old versions stay on file, and it gets no feedback on its answers. The protocol measures how its capability changes as its history accumulates.
  3. A systematic analysis of the bottlenecks of lifelong agents. We evaluate five frontier models on 300 tasks and analyze success over the lifetime, as the context fills, by the evidence the agent opens, and with and without its history. There are four findings. (i) Unlike human experts, agents are not stable over time. (ii) Agents that carry their history do worse later, not better. (iii) History clearly reduces retrieval but barely helps accuracy. (iv) Every agent is weakest at tracking what changed between two dates.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.