- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TenureBench uses 13407 dated official documents from finance, decision, and law (2016-2026) to persist agent-per-field evaluation. The agent is never reset and never sees the answers. There are 920 tasks with 13318 binary rubric criteria on a 300-task sample; the best of five models solves 52.7%. Success falls over the lifetime, and as the context fills and the long history halves, tool calls without improving accuracy. This is a very high-quality benchmark with comprehensive evaluation, which makes it an important step towards real-world agents.
Critical strengths
1- A new an realsitic protocol for evaluation is presented in this paper. Old versions stay on file, every task needs several dates, and the agent persists without feedback. Table 1 positions the work clearly against prior work. 2- Real records with revision-heavy entries chosen by explicit rules, a construction funnel documented in detail, and an expert reading of every released task. 3- The analysis goes past headline numbers: per-domain intervals, same-task context fill comparisons, gold-evidence-opened mechanisms, and a controlled ablation of what a long history contributes.
Critical weaknesses
1- The human-expert comparison is not measured. In figure 2 and the professional intuitions claims imply that experts don't degrade as context grows, with no baseline. Either measure it or reframe. 2- Entity relationships and multi-date necessity are decided by LLM workers. Expert review lacks qualifications, agreement numbers, and rejection categories. The paper never measures how much of the archive is relevant per task, so the effect of a growing archive is confounded with a growing share of distractors. 3- The judge is not validated and only one single judge is judged. 4- Later tasks may be harder in themselves (more dates, more entities), and context fill varies with each model's summarization pace. A control would help, such as the same task asked at different points in the lifetime, or Fresh vs. Lifelong compared across the lifetime sixths. 5- Grading might be too strict: A task succeeds only if every criterion passes (median 9). Rubric scores (76–91%) and task success (23–53%) diverge sharply, and a failure over 54.57 vs 54.58 suggests the tolerance rules need stating.
Questions for authors
1- How well does the judge agree with human expert labels, and do rankings hold under a judge from another model family? 2- Does the lifetime decline survive asking the same tasks at different lifetime positions? 3- What are the expert reviewers' qualifications, and what did the 57 rejections share? 4- What fraction of each task's archive is relevant, and does the finance drop persist when you control for distractors?
3 comments
Response 2 to Reviewer XMsn
Q1 / W3: How well does the judge agree with human experts, and do the rankings hold under a judge from another model family?
On 105 answers, the GPT-5.6 Sol judge agrees with domain experts on 94.3% of the task-success decisions (κ = 0.88). On 420 answers, it agrees with judges from two other model families on 94.8% and 93.6%, and their system rankings correlate with its own at Spearman ρ = 0.96 and ρ = 0.93. Section 5.1 and Appendix D report the study.
We rescored 420 of the 2,100 answers, 20 for each of the seven systems in each domain, with Claude Opus 4.6 and DeepSeek Flash as judges from two other model families. Each judge ran in a fresh session on the same input and did not know which system wrote an answer. The table below reports their agreement on task success.
| Judge pair | Task-success agreement | Cohen's κ |
|---|---|---|
| GPT-5.6 Sol vs. Claude Opus 4.6 | 94.8% | 0.87 |
| GPT-5.6 Sol vs. DeepSeek Flash | 93.6% | 0.83 |
| Claude Opus 4.6 vs. DeepSeek Flash | 93.1% | 0.81 |
For the comparison with domain experts, two experts per domain graded 105 of the 420 answers, five per system and domain, without seeing the system's name or any judge's verdict. The first expert graded every criterion, and the second reviewed each grade and changed 2.6% of them. Against the experts' task-level decisions, GPT-5.6 Sol agrees on 94.3% (κ = 0.88), Claude Opus 4.6 on 92.4% (κ = 0.84), and DeepSeek Flash on 91.4% (κ = 0.82). The results show that The LLM judge agrees closely with both human experts and other models, supporting the reliability of the automated evaluation.
Q2 / W4: Does the lifetime decline survive a control for later tasks being harder, and does context fill depend on each model's summarization pace?
The same task cannot be moved along the lifetime, because its question date fixes the archive it is asked on. What can change is the history the agent brings to it. Therefore, we ran the Fresh agent on all 300 evaluated tasks for the five systems. The table below reports task success by sixth of the lifetime, pooled over the five systems (the dashed blue line of Figure 5a, with each system in Figure 6).
| Agent | Sixth 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Lifelong | 54.1% | 41.2% | 40.0% | 44.4% | 32.8% | 42.9% |
| Fresh | 49.8% | 45.6% | 48.8% | 52.4% | 48.8% | 48.6% |
The Fresh agent stays between 45.6% and 52.4% across the six sixths, on the same tasks and archives as the Lifelong agent. Later tasks do span more dates and entities (Figure 5b), yet an agent without the history solves them as often as the early ones. The decline is therefore tied to the accumulated history.
For the summarization pace, Appendix A.4 uses only each system's first context window, before any summary replaces the conversation. Pooled over the five systems, success is 49.0% for tasks that arrive with three to four fifths of the window filled and 30.4% above four fifths. Section 5.4 estimates the effect of context load with task and system fixed effects, which compares the same task arriving at different loads.
Response 3 to Reviewer XMsn
W5: Is the all-criteria grading too strict, and what is the tolerance rule for numbers?
The rubric criteria reflect real-world requirements in finance, medicine, and law rather than being intentionally stringent. Each task is divided into multiple criteria to assess, from different perspectives, whether the answer meets professional standards. The tolerance rule is stated in the judge template.
The rubric score is the share of criteria met, while task success asks whether the answer is complete enough to be delivered. Of the 861 failed answers, 98.5% miss at least one correctness or reasoning criterion, that is, a wrong or missing fact, value, conclusion, or derivation step. A high rubric score indicates that the model satisfies most individual requirements, but does not mean that it can consistently produce a complete final answer with accurate facts, sound reasoning, and adequate citations.
Significant gaps between ask success rates and partial-credit scores are also common in complex agent benchmarks. For example, the gap exceeds 30 pp+ in Agents' Last Exam [1] and 50 pp+ in CL-Bench [2]. Therefore, the gap between overall task success and rubric scores should not be attributed solely to overly strict evaluation.
The judge template (Figure H.22) defines the two verdicts. MET means fully and correctly satisfied, NOT_MET covers an omitted, incomplete, imprecise, unsupported, or contradicted answer, and there is no partial credit. A criterion that allows a tolerance says so in its wording, and a criterion that states a number is met only by that number.
In Appendix Figure I.5, the expected share of 2025 revenue from Filtration is 54.58%, whereas Claude Sonnet 5 reports 54.57%. The paper therefore marks Claude Sonnet 5 as failing this task. In this case, total 2025 product revenue is approximately 738 million dollars, so a difference of 0.01 percentage points corresponds to about 73,800 dollars. Although small relative to total revenue, such a discrepancy can create downstream issues in financial audits, regulatory filings, and related processes. Finance, medicine, and law all demand high accuracy, although the risks and material consequences of numerical errors vary by context. Our strict criteria are grounded in real-world requirements. In finance, monetary amounts, ratios, and financial statements must be reconcilable, reproducible, and internally consistent. In medicine, values such as dosage, concentration, administration frequency, and contraindication thresholds can directly affect safety. In law, effective dates, deadlines, eligibility thresholds, and statutory amounts often require exact precision; a one-day difference or crossing a threshold may change the legal conclusion.
We hope these clarifications and explanations address your concerns. If you have any further questions, we would be happy to discuss them with you.
References
[1] Sun et al. Agents' Last Exam. arXiv:2606.05405, 2026.
[2] Dou et al. CL-bench: A Benchmark for Context Learning. arXiv:2602.03587, 2026.
Response 1 to Reviewer XMsn
Thank you very much for your encouraging assessment and valuable suggestions. We have addressed each of the concerns you raised accordingly. Changes in the revised paper are marked in blue.
W1: Is the human-expert comparison measured?
We measured the human baseline. The experts' task success stays stable across the lifetime, at 82.4% in the first sixth and 85.7% in the last. One expert with doctoral training per domain answered the evaluated tasks of that domain with the archive as of each question date and without the reference answers, the gold evidence, or the criteria, and the same rubrics graded the answers. Overall the experts solve 84.0% of the tasks and meet 98.2% of the criteria (new row of Table 3). The table below reports their task success by sixth of the lifetime (Table E.1).
W2: Are entity relationships and multi-date necessity decided by LLM workers alone?
Two models other than the writing model verify that every named entity and at least two dated states are needed, and a fixed rule rechecks the dates. The first worker only proposes a group of related entities and the dated states that connect them. Two verification workers that run other models (GPT-5.6 Luna and Grok 4.6) then rebuild the answer from the archive as of the question date. They pass a candidate only if every named entity is needed and the task needs at least two dated states. No single document, single date, later summary, or latest state may give the answer. The single-source check then recomputes the multi-date test from the evidence dates by a fixed rule (Section 4 and Appendix C).
Q3 / W2: What are the expert reviewers' qualifications and agreement, and what do the rejections share?
Six domain experts with doctoral training reviewed the candidates, and a second expert reached the same decision as the first on 95.6% of the tasks reviewed twice. Two experts each in Finance, Medical, and Legal reviewed the 977 candidates of their own field and rejected 57. A review took a median of 12 minutes (interquartile range 8 to 18). A second expert of the same field, who did not see the first decision, reviewed 114 tasks (the 57 rejected candidates and 57 matched released tasks) and made the same decision on 109 (95.6%, Cohen's κ = 0.93). A further review of the 57 released tasks with the complete source documents changed no decision (Appendix C).
The rejected candidates fall into four types. In the first, the question only asks the agent to locate and report facts, or it leaves nothing to decide beyond a lookup and arithmetic. In the second, the requested change is zero or the two dated states are identical, so one state gives the whole answer. In the third, the task repeats another task on the same entities and method. In the fourth, the criteria and the question do not match.
Q4 / W2: What fraction of each task's archive is relevant, and does the finance drop persist when distractors are controlled?
In finance, the relevant share of the archive stays between 2.9% and 4.3% while the archive nearly triples. The finance drop therefore occurs at a nearly constant share of distractors. For each evaluated task we count the documents of the entities the task names, published by its question date, over the documents in the archive at that date. The table below reports this share by sixth of the lifetime (Table B.2).
In finance the archive grows from a median of 3,466 documents in the first sixth to 9,415 in the last. The share of other documents therefore stays between 95.7% and 97.1%, while task success in finance falls from 56.5% in the first sixth to 31.1% in the later sixths.