- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces FinEvo-Bench, a benchmark of 120 financial tasks across 20 recurring workflows. It studies whether agents can use feedback and persistent experience to improve on later cases with different inputs and conclusions.
Critical strengths
-
The focus on recurring professional work is useful. Cases share a workflow but require fresh analysis, giving experience reuse a practical test.
-
The dataset shows substantial curation effort. The multi-file tasks, expert-built rubrics, and human validation of the judge give the evaluation a reasonable foundation.
-
The state-reset controls and fixed expert-skill baseline make the gains easier to interpret. The memory/skill, feedback, and scene-isolation comparisons are also useful.
-
The paper is clearly written and easy to follow. The appendix provides concrete examples of the tasks, scoring rules, and changes made to skills after feedback.
Critical weaknesses
- How does rewriting feedback into skills compare with simply storing and retrieving the judge’s comments?
- How much do the reported gains and framework rankings vary across the three runs?
- Which issues count as compliance violations, and how closely do the judge’s labels agree with expert judgments for each type?
- Do the framework rankings and benefits of skill updates hold with a backbone other than Qwen3.7-Max?
Questions for authors
See in weakness↑
Response to Reviewer Qs7A (Part1)
Thank you for recognizing the value of the longitudinal setup and state-reset controls. Below, we explain the paper's central contribution, the additional findings from cross-backbone and feedback comparisons, and the motivation in the abstract and the main results.
W1: Central contribution and novelty
Thank you for your constructive feedback.
The central contribution of this paper is FinEvo-Bench, a dataset for testing agents' self-evolution capabilities, targeting scenarios in which users repeatedly ask agents to handle similar tasks over prolonged use. These tasks share procedures and delivery requirements while differing in their materials, analytical priorities, and appropriate conclusions. We aim to test whether agents can accumulate useful experience from earlier tasks and apply it to subsequent cases.
To make the distinction from recent benchmarks (2025–2026) more concrete, we reproduce the comparison in Table 1 of the paper below.
FinEvo-Bench brings these dimensions together to evaluate whether experience accumulated from earlier cases improves subsequent work within recurring professional workflows. A shared procedure provides opportunities to reuse experience, while substantive differences between cases require agents to adapt their analyses and judgments. Open-ended deliverables and multidimensional evaluation allow us to assess how the quality of this professional work changes as experience accumulates.
For tasks such as database operations and coding, completion can generally be checked through target states, query results, or test cases, and existing benchmarks already cover such tasks. We focus on semi-open-ended tasks whose deliverables are analytical reports, assessment opinions, and the like: they have clear professional requirements but may admit multiple reasonable analytical approaches and outcomes, requiring quality assessment against a rubric in conjunction with case-specific evidence.
Financial business provides a suitable source of tasks for this evaluation. Taking financial statement analysis as an example, different companies all require analyses of profitability, cash flow, and solvency; however, financially healthy companies, companies in financial distress, and companies with insufficient disclosures may require different evidence, analytical priorities, and final recommendations. On this basis, we constructed 120 tasks covering 20 financial business scenes. The six real cases in each scene share a professional procedure while retaining substantive differences between individual cases. Dataset construction required approximately 1,200 person-hours in total.
The main experiments and analytical experiments serve different roles around this dataset. The main experiments evaluate four common frameworks, including Claude Code, measuring their self-evolution performance through evolving/state-reset comparisons under the same task order. The analytical experiments further examine specific questions in agent self-evolution: whether expressing experience as memory or skills is more effective, how targeted diagnosis and reference answers affect updates, which capabilities are easier to improve, and so on, helping us understand how agents improve through accumulated experience in these tasks.
We will incorporate this explanation into the introduction.
References
[1] Junhao Zheng, Xidi Cai, Qiuke Li, et al. LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners. arXiv:2505.11942, 2025.
[2] Sihang Jiang, Lipeng Ma, Zhonghua Hong, et al. SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment. arXiv:2604.08988, 2026.
[3] Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, et al. Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory. arXiv:2511.20857, 2025.
[4] Yuxuan Cai, Yipeng Hao, Jie Zhou, et al. Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark. arXiv:2508.19005, 2025.
[5] Yizhe Chi, Deyao Hong, Dapeng Jiang, et al. Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization. arXiv:2604.12290, 2026.