acceptodds
Under review as a conference paper at ICLR 2027

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows

Abstract

Agents used over time encounter recurring professional work: each case requires different evidence and judgment, while the underlying workflow can be reused. Benchmarks built from independent tasks cannot reveal whether an agent turns earlier experience into better procedures for later cases. We introduce FinEvo-Bench, a longitudinal benchmark designed around this structure. It contains 120 open-ended tasks drawn from real cases across 20 business scenes in six financial domains. Each scene contains six substantively different cases that share a professional workflow and an expert-authored rubric for task quality and financial compliance. Constructing and validating the benchmark required approximately 1,200 person-hours. Finance provides a natural test bed because recurring analyses apply shared professional and compliance requirements to heterogeneous inputs, producing case-specific analyses and conclusions. We evaluate four self-evolving agent scaffolds with Qwen3.7-Max on three independently shuffled, globally interleaved task streams. A Claude Code rubric judge backed by Claude Opus 4.6 evaluates all outputs, and paired state-reset controls estimate each scaffold’s gain from retained experience. Evolving runs score 9.33–19.37 points higher and trigger 0.12–0.44 fewer compliance issues per task than their paired controls. Paired score gains at within-scene ranks 4–6 exceed those at ranks 1–3 by 6.10–8.70 points. FinEvo-Bench measures whether retained experience improves later professional work under continued use. The dataset is available at https://anonymous.4open.science/r/finevobench-710A/.

Primary area:
datasets and benchmarks
Keywords:
self-evolving agents, agent benchmarks, longitudinal evaluation, professional workflows, financial agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FNikOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces FinEvo-Bench, a benchmark of 120 financial tasks across 20 recurring workflows. It studies whether agents can use feedback and persistent experience to improve on later cases with different inputs and conclusions.

Critical strengths
  • The focus on recurring professional work is useful. Cases share a workflow but require fresh analysis, giving experience reuse a practical test.

  • The dataset shows substantial curation effort. The multi-file tasks, expert-built rubrics, and human validation of the judge give the evaluation a reasonable foundation.

  • The state-reset controls and fixed expert-skill baseline make the gains easier to interpret. The memory/skill, feedback, and scene-isolation comparisons are also useful.

  • The paper is clearly written and easy to follow. The appendix provides concrete examples of the tasks, scoring rules, and changes made to skills after feedback.

Critical weaknesses
  • How does rewriting feedback into skills compare with simply storing and retrieving the judge’s comments?
  • How much do the reported gains and framework rankings vary across the three runs?
  • Which issues count as compliance violations, and how closely do the judge’s labels agree with expert judgments for each type?
  • Do the framework rankings and benefits of skill updates hold with a backbone other than Qwen3.7-Max?
Questions for authors

See in weakness↑

Reviewer_ABvkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FinEvo-Bench is a longitudinal benchmark that evaluates whether agents improve across repeated professional financial workflows. It contains 120 open-ended tasks organized into 20 business scenes across six financial domains; each scene contains six substantively different cases governed by a shared professional procedure. The benchmark includes 775 input files, and scene-specific rubrics evaluate evidence use, analysis, conclusions, report quality, and financial compliance.

The benchmark is constructed from institutional scene descriptions, professionally validated procedures, and institutional or publicly documented real cases. Domain experts construct and cross-review tasks, reference answers, and rubrics, with approximately 1200 person-hours reported for construction and validation. Each scene uses a shared 100-point rubric, and two experts participate in rubric construction and applicability checking.

For longitudinal evaluation, the authors shuffle all 120 tasks into three globally interleaved streams and evaluate agent harnesses including Claude Code, Codex, Letta, and GenericAgent using the same Qwen3.7-Max backbone. Each task is executed, scored by a fixed Claude Opus 4.6 judge, followed by diagnostic feedback, reflection, and persistent-state consolidation. Every evolving run is paired with a state-reset condition that shares the same task order and inference configuration. The automated judge is validated against one financial expert on 120 outputs.

Experiments show that retaining experience improves task performance over state-reset controls, with larger average gains on later cases within the same scene. Additional analyses compare experience carriers, diagnostic feedback against reference-answer feedback, capability dimensions, retrieval activation, and scene-isolated versus globally interleaved evolution.

Critical strengths
  • The benchmark captures recurring professional workflows while requiring substantive adaptation to individual cases. Recurring professional procedures link cases whose input files, analytical requirements, and appropriate conclusions differ. This structure makes accumulated experience useful while requiring adaptation to each case. Expert review and rubrics covering evidence use, substantive analysis, and financial compliance tie task success to professional standards.
  • The controlled comparisons help distinguish the effects of retained experience from those of cross-scene interaction. Matched evolving and state-reset runs share task order, backbone, inference settings, and scoring, isolating the contribution of retained state. Scene-isolated runs preserve the within-scene task order while removing experience from intervening scenes. The resulting comparison reveals a practical tradeoff: isolation improves task-specific quality, whereas interleaving transfers compliance practices across scenes.
Critical weaknesses
  • A shared scene-level rubric may obscure differences in the analytical priorities of individual cases. For example, A healthy borrower, a financially distressed company, and a company with incomplete disclosures require different evidence to guide the assessment. The rubric rewards completion of common analyses, correct calculations, and supported conclusions, which may favor increasingly comprehensive procedures. This makes it difficult to distinguish better workflow coverage from better judgment about which issues should drive a particular case’s assessment.
  • The external scoring agent performs much of the diagnosis that drives subsequent improvement, which is converted by the evaluated agent into persistent memories or skills. The reported gains therefore reflect both externally supplied professional guidance and the agent’s ability to consolidate and reuse it, making the contribution of the agent’s own diagnosis and learning difficult to separate.
Questions for authors
  • How does the shared rubric distinguish comprehensive procedure execution from the ability to identify and prioritize the analyses most relevant to each case?
  • Which parts of the improvement process depend on the evaluated agent’s own diagnosis, and which are supplied by the scoring agent? How should this division of responsibility shape the interpretation of the reported self-evolution gains?
Reviewer_Qs7AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a longitudinal benchmark of 120 financial workflow tasks across multiple scenes and uses persistent-state versus reset comparisons to study whether agents improve by retaining experience. It also compares memory- and skill-based configurations and reports expert agreement for its automated rubric judge. The key insight—that experience should be converted into reusable procedures rather than merely stored—is somewhat useful. However, the work mainly evaluates external memory/skill mechanisms in a bounded financial setting rather than proposing a new self-evolution algorithm.

Critical strengths

The longitudinal setup and state-reset controls are more informative than evaluating each task independently, because they attempt to measure whether experience changes later performance.

Critical weaknesses
  1. Unclear central contribution. The paper’s contribution is spread across benchmark design, task sequencing, memory/skill ablations, feedback analysis, and compliance evaluation. The central methodological contribution is difficult to distinguish from a collection of evaluation choices, and the overall novelty is not sufficiently clear.

  2. Limited scope and generalizability. The benchmark contains only 120 tasks concentrated in financial workflows, and all experiments use a single backbone model. While the results demonstrate the benefits of experience reuse in this setting, they do not establish broader self-evolution capabilities or reliable transfer to unseen domains and model families. Moreover, the use of model-generated diagnostic feedback raises concerns about how realistic the evaluation protocol is compared to actual deployments.

  3. Writing and presentation need improvement. The paper's narrative is not always well organized, and some arguments are difficult to follow. For example, the abstract introduces FinEvo-Bench before clearly motivating the choice of the financial domain. It also reports ICC(A, 1) = 0.95 without explaining what this metric represents. More generally, the paper presents many numerical results without sufficiently highlighting the key findings and their implications, making the main takeaways difficult to identify.

Questions for authors

see weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Qs7A (Part1)

Thank you for recognizing the value of the longitudinal setup and state-reset controls. Below, we explain the paper's central contribution, the additional findings from cross-backbone and feedback comparisons, and the motivation in the abstract and the main results.

W1: Central contribution and novelty

Thank you for your constructive feedback.

The central contribution of this paper is FinEvo-Bench, a dataset for testing agents' self-evolution capabilities, targeting scenarios in which users repeatedly ask agents to handle similar tasks over prolonged use. These tasks share procedures and delivery requirements while differing in their materials, analytical priorities, and appropriate conclusions. We aim to test whether agents can accumulate useful experience from earlier tasks and apply it to subsequent cases.

To make the distinction from recent benchmarks (2025–2026) more concrete, we reproduce the comparison in Table 1 of the paper below.

BenchmarkCross-task EvolutionDomain WorkflowOpen-ended ArtifactMulti-aspect Evaluation
LifelongAgentBench (2025) [1]✓
SEA-Eval (2026) [2]✓✓
Evo-Memory (2025) [3]✓✓
StuLife (2025) [4]✓✓✓
Frontier-Eng (2026) [5]✓✓✓
FinEvo-Bench (Ours)✓✓✓✓

FinEvo-Bench brings these dimensions together to evaluate whether experience accumulated from earlier cases improves subsequent work within recurring professional workflows. A shared procedure provides opportunities to reuse experience, while substantive differences between cases require agents to adapt their analyses and judgments. Open-ended deliverables and multidimensional evaluation allow us to assess how the quality of this professional work changes as experience accumulates.

For tasks such as database operations and coding, completion can generally be checked through target states, query results, or test cases, and existing benchmarks already cover such tasks. We focus on semi-open-ended tasks whose deliverables are analytical reports, assessment opinions, and the like: they have clear professional requirements but may admit multiple reasonable analytical approaches and outcomes, requiring quality assessment against a rubric in conjunction with case-specific evidence.

Financial business provides a suitable source of tasks for this evaluation. Taking financial statement analysis as an example, different companies all require analyses of profitability, cash flow, and solvency; however, financially healthy companies, companies in financial distress, and companies with insufficient disclosures may require different evidence, analytical priorities, and final recommendations. On this basis, we constructed 120 tasks covering 20 financial business scenes. The six real cases in each scene share a professional procedure while retaining substantive differences between individual cases. Dataset construction required approximately 1,200 person-hours in total.

The main experiments and analytical experiments serve different roles around this dataset. The main experiments evaluate four common frameworks, including Claude Code, measuring their self-evolution performance through evolving/state-reset comparisons under the same task order. The analytical experiments further examine specific questions in agent self-evolution: whether expressing experience as memory or skills is more effective, how targeted diagnosis and reference answers affect updates, which capabilities are easier to improve, and so on, helping us understand how agents improve through accumulated experience in these tasks.

We will incorporate this explanation into the introduction.

References

[1] Junhao Zheng, Xidi Cai, Qiuke Li, et al. LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners. arXiv:2505.11942, 2025.

[2] Sihang Jiang, Lipeng Ma, Zhonghua Hong, et al. SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment. arXiv:2604.08988, 2026.

[3] Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, et al. Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory. arXiv:2511.20857, 2025.

[4] Yuxuan Cai, Yipeng Hao, Jie Zhou, et al. Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark. arXiv:2508.19005, 2025.

[5] Yizhe Chi, Deyao Hong, Dapeng Jiang, et al. Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization. arXiv:2604.12290, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.