- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FinEvolveBench, a chronological benchmark for evaluating experience-based adaptation in LLM agents without updating model parameters. Agents predict industry-level market responses from financial news and market observations, and receive delayed feedback at different trading-day horizons. The paper characterizes evaluation scenarios along four dimensions: outcome observability, credit assignment complexity, environmental dynamics, and experience reusability. Experiments compare five experience mechanisms against an experience-free baseline using two LLM backbones and time-series and cross-sectional information coefficients. The results show inconsistent gains and frequent degradation, motivating further investigation of experience validity in changing environments.
Critical strengths
- The benchmark addresses a useful evaluation gap: whether external experience remains beneficial when feedback is delayed and noisy, and historical patterns may become outdated. The chronological replay protocol makes feedback availability explicit and supports controlled comparisons under a shared information stream.
- The evaluation includes multiple experience mechanisms, two backbones, and three prediction horizons. Reporting both time-series and cross-sectional predictive association reveals variation that a single aggregate metric would obscure. The negative results are informative and challenge the assumption that experience augmentation reliably improves agent performance.
Critical weaknesses
- The results do not sufficiently separate limitations of the experience mechanisms from effects of their benchmark-specific adaptations. Appendix B describes potentially consequential choices, including retrieval of the latest ten insights for ExpeL and a shared MemRL utility value updated by feedback from three different horizons. The paper should clarify which components of the original methods are retained and validate these adaptations through targeted ablations. Otherwise, the evidence primarily supports conclusions about the evaluated implementations.
- The four scenario dimensions provide a useful framing, but their role in explaining the observed degradation remains insufficiently established. A single financial setting combines all four challenges, and the case study illustrates a plausible failure mode without isolating its cause. Controlled comparisons or quantitative analyses of experience reuse and temporal validity would strengthen the explanation. Alternatively, the conclusions should be narrowed to the observed instability within this benchmark.
Questions for authors
-
How do the adapted baselines preserve the core experience extraction, refinement, and retrieval mechanisms of their original implementations? In particular, would maintaining separate MemRL utility values for each prediction horizon change the results?
-
Were the experiments repeated across multiple runs, and how stable are the reported improvements and degradations? Could uncertainty estimates account for temporal dependence arising from overlapping prediction horizons, for example through a block bootstrap over dates?
-
What are the exact backbone versions, access dates, and documented knowledge cutoffs? The replay protocol controls information supplied during evaluation, but how is possible pretraining exposure to historical market outcomes addressed?
-
What evidence supports the characterization of this dataset as having low experience reusability and substantial non-stationarity? Could performance be analyzed by historical-case similarity or temporal regime to test these explanations?
-
Figure 2 overlaps with the preceding paragraph around lines 251–253, obscuring part of the text. Please adjust the figure placement and spacing.
-
OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents is quite related to your work, could you discuss it in the related work?
2 comments
Thank the author's response and it answered all my questions. I have updated my score to 3.
Response to Reviewer qpFN
We thank Reviewer
qpFNfor the constructive feedback acrossW1–W2andQ1–Q6.1. Paradigm Limits vs. Adaptations & Horizon-Separated Experiments (
W1&Q1)Thank you for raising this question separating adaptations from paradigm limits:
SUCCESS/FAILURErollouts offline; since replay gives one continuous trajectory per(date, sector)without resets, our adapter re-distills rules online over the 10 latest reflections, scoping claims to online single-trajectory rule abstraction.Q1): In all Table 3 runs (memrl/experiment.py), each memory keeps strictly one shared scalar Q, updated via Q←Q+0.3(uh−Q) as horizons mature under our 1-call-per-sector-day protocol (Fig. 1). Short-horizon gains can thus conflate long-horizon retrieval (05-29's+.73%1d return raisedscoreto0.85despite-8.12%at 40d), while separate Qh avoids cross-horizon conflation (though scalar utility without semantic attribution still faces noiseO). Inspired by your question, we are running horizon-separated experiments across all methods (still compiling under tight server capacity). From our prior experience, LLM methods (Mem0,Reflexion,ExpeL) show minimal difference between separate and merged horizon calls; we will add full results and analysis in the Appendix of the final version.2. Controlled Validation of 4D Taxonomy vs.
Pipe(W2&Q4)Stratifying all 12 runs (
299d) vs.Pipe(MBB B=2,000, L=10):D— 10d CSI 1000 Vol:LowVol150d vs.HighVol149d):LowVolbeatsPipe:DS +Reflexion 20dtsIC=+.1472(p<.001,+.0782vs.Pipe, p=.014);DS +ReasBank 40d+.0417/+.0393(p=.011/.022);Qwen +ReasBank20d+.0376/+.0461(p=.019/.014) &40d+.1165(p<.001,+.0606/+.0587vs.Pipe, p=.005/.023).HighVol40d reflections/rules flip to negative transfer vs.Pipe(DS ΔcsIC:-.0454/-.0497, p=.022/.023;Qwen:-.0875/-.0510, p=.003/.025), while+ReasBank 40dholds (DS:-.0072/-.0006;Qwen:+.0272/+.0183).C— Coldd1..40vs. Matured41..299,40d):+ReasBanktrailsPipeind1..40(-.0644DS;-.0822Qwen) before 40d outcomes mature, then beatsPipeind41..299(DS:+.0289/+.0320, p=.047/.023;Qwen:+.0585/+.0567, p<.001, standalone+.1225;Qwen +MemRL:-.0602, p=.013).R— Fig. 3(b) News Split): Median-splitting all 31 sectors by uniqueimportance>=8news Mi∈[67,9795] (1,344cutoff) gives 15 sparse (LowRep, mean627) vs. 16 dense (HighRep, mean4,918, 7.8×) sectors. Reflection excels inHighRep(Qwen 20d csIC +.1672,+.1109vs.Pipe, p=.001) but shows 40d negative transfer inLowRep(DS +Reflexion ΔcsIC=-.0840, p=.024;DS +ExpeL ΔtsIC=-.0287, p=.033), while+ReasBank 40dbeatsPipe(DS +.0185, p=.031;Qwen +.0724, p=.065).3. Stability & Block Bootstrap (
Q2)Over
8,818–9,269forecasts/run (0parse failures;temp=0.5,sglang), Paired MBB (B=5,000, L=h) confirms negative transfer inMem0(DS 20d:-.0192, p=.027;Qwenavg:-.0143, p=.041; full 20d:-.0340, p=.006) andMemRL(Qwen 40d:-.0542, p=.010; avg:-.0267, p=.046), and 40d gains inReasBank(Qwen 40d:+.0448, p=.058; mature p<.001).4. Backbones & Contamination Analysis (
Q3)We evaluate frozen offline models (
sglang, zero web)—Qwen/Qwen3.6-35B-A3B(2026-04-16, Blog) anddeepseek-ai/DeepSeek-V4-Flash(2026-04-24, Release)—over2025-01-01to2026-03-31(299trading days). Although this window precedes April 2026 open-sourcing, data leakage is unlikely: (1) Offline Prompting Verification: direct prompt-engineering probes under offline deployment cannot elicit Shenwan L1 index prices or forward trends (Pipegets only0.018–0.060tsIC, andMethod - Pipecancels static priors); and (2) Pretraining Pipeline Dynamics: base-model data curation and training span long periodic cycles pre-release, where financial corpora form a tiny share and raw high-noise market series rarely enter backbone training pipelines without dedicated processing.5. Layout (
Q5) & OpenPM (Q6)We fixed Fig. 2 overlapping lines 251–253 (p. 5) and lines 139/170 typos, and contrasted
OpenPM(arXiv:2608.09988) (OpenPM-Bench)—44d S&P 500 auditing—with our 300d self-evolution testbed.