acceptodds
Under review as a conference paper at ICLR 2027

FinEvolveBench: A Benchmark for Self-Evolving Agents under Challenging Experience Regimes

Abstract

Large language model (LLM) agents increasingly rely on external experience to continually adapt to changing environments without modifying their underlying models. Recent experience mechanisms have demonstrated promising results across diverse tasks. However, their effectiveness is typically evaluated within individual benchmark settings, and how experience mechanisms generalize across different scenarios remains insufficiently explored. In this work, we present a scenario-oriented analysis of experience mechanisms for LLM agents. We characterize existing evaluation scenarios along four dimensions: outcome observability, credit assignment complexity, environmental dynamics, and experience reusability. Our analysis shows that existing benchmarks often evaluate experience mechanisms under scenarios where at least one dimension is comparatively favorable, leaving more challenging combinations of scenario properties underexplored. To address this gap, we introduce FinEvolveBench, a reproducible benchmark built on a chronological stream of rich financial news and market data that enables systematic evaluation of experience-based self-evolution under challenging experience regimes characterized by noisy feedback, ambiguous credit assignment, environmental non-stationarity, and limited experience reusability. Experiments show that existing approaches exhibit substantially reduced or inconsistent gains in this setting, highlighting the scenario-dependent nature of experience mechanisms and the challenge of maintaining valid experience under changing environments.

Primary area:
datasets and benchmarks
Keywords:
Self-Evolving Agents, Experience, Experience Learning, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qpFNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinEvolveBench, a chronological benchmark for evaluating experience-based adaptation in LLM agents without updating model parameters. Agents predict industry-level market responses from financial news and market observations, and receive delayed feedback at different trading-day horizons. The paper characterizes evaluation scenarios along four dimensions: outcome observability, credit assignment complexity, environmental dynamics, and experience reusability. Experiments compare five experience mechanisms against an experience-free baseline using two LLM backbones and time-series and cross-sectional information coefficients. The results show inconsistent gains and frequent degradation, motivating further investigation of experience validity in changing environments.

Critical strengths
  1. The benchmark addresses a useful evaluation gap: whether external experience remains beneficial when feedback is delayed and noisy, and historical patterns may become outdated. The chronological replay protocol makes feedback availability explicit and supports controlled comparisons under a shared information stream.
  2. The evaluation includes multiple experience mechanisms, two backbones, and three prediction horizons. Reporting both time-series and cross-sectional predictive association reveals variation that a single aggregate metric would obscure. The negative results are informative and challenge the assumption that experience augmentation reliably improves agent performance.
Critical weaknesses
  1. The results do not sufficiently separate limitations of the experience mechanisms from effects of their benchmark-specific adaptations. Appendix B describes potentially consequential choices, including retrieval of the latest ten insights for ExpeL and a shared MemRL utility value updated by feedback from three different horizons. The paper should clarify which components of the original methods are retained and validate these adaptations through targeted ablations. Otherwise, the evidence primarily supports conclusions about the evaluated implementations.
  2. The four scenario dimensions provide a useful framing, but their role in explaining the observed degradation remains insufficiently established. A single financial setting combines all four challenges, and the case study illustrates a plausible failure mode without isolating its cause. Controlled comparisons or quantitative analyses of experience reuse and temporal validity would strengthen the explanation. Alternatively, the conclusions should be narrowed to the observed instability within this benchmark.
Questions for authors
  1. How do the adapted baselines preserve the core experience extraction, refinement, and retrieval mechanisms of their original implementations? In particular, would maintaining separate MemRL utility values for each prediction horizon change the results?

  2. Were the experiments repeated across multiple runs, and how stable are the reported improvements and degradations? Could uncertainty estimates account for temporal dependence arising from overlapping prediction horizons, for example through a block bootstrap over dates?

  3. What are the exact backbone versions, access dates, and documented knowledge cutoffs? The replay protocol controls information supplied during evaluation, but how is possible pretraining exposure to historical market outcomes addressed?

  4. What evidence supports the characterization of this dataset as having low experience reusability and substantial non-stationarity? Could performance be analyzed by historical-case similarity or temporal regime to test these explanations?

  5. Figure 2 overlaps with the preceding paragraph around lines 251–253, obscuring part of the text. Please adjust the figure placement and spacing.

  6. OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents is quite related to your work, could you discuss it in the related work?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qpFN

We thank Reviewer qpFN for the constructive feedback across W1–W2 and Q1–Q6.

1. Paradigm Limits vs. Adaptations & Horizon-Separated Experiments (W1 & Q1)

Thank you for raising this question separating adaptations from paradigm limits:

  • ExpeL: Standard ExpeL contrasts binary SUCCESS/FAILURE rollouts offline; since replay gives one continuous trajectory per (date, sector) without resets, our adapter re-distills rules online over the 10 latest reflections, scoping claims to online single-trajectory rule abstraction.
  • MemRL Shared QQ & Ongoing Horizon-Separated Ablation (Q1): In all Table 3 runs (memrl/experiment.py), each memory keeps strictly one shared scalar QQ, updated via Q←Q+0.3(uh−Q)Q \leftarrow Q + 0.3(u_h - Q) as horizons mature under our 1-call-per-sector-day protocol (Fig. 1). Short-horizon gains can thus conflate long-horizon retrieval (05-29's +.73% 1d return raised score to 0.85 despite -8.12% at 40d), while separate QhQ_h avoids cross-horizon conflation (though scalar utility without semantic attribution still faces noise O). Inspired by your question, we are running horizon-separated experiments across all methods (still compiling under tight server capacity). From our prior experience, LLM methods (Mem0, Reflexion, ExpeL) show minimal difference between separate and merged horizon calls; we will add full results and analysis in the Appendix of the final version.

2. Controlled Validation of 4D Taxonomy vs. Pipe (W2 & Q4)

Stratifying all 12 runs (299d) vs. Pipe (MBB B=2,000, L=10):

  1. Drift (D — 10d CSI 1000 Vol: LowVol 150d vs. HighVol 149d): LowVol beats Pipe: DS +Reflexion 20d tsIC=+.1472 (p<.001p<.001, +.0782 vs. Pipe, p=.014p=.014); DS +ReasBank 40d +.0417/+.0393 (p=.011/.022p=.011/.022); Qwen +ReasBank 20d +.0376/+.0461 (p=.019/.014p=.019/.014) & 40d +.1165 (p<.001p<.001, +.0606/+.0587 vs. Pipe, p=.005/.023p=.005/.023). HighVol 40d reflections/rules flip to negative transfer vs. Pipe (DS ΔcsIC: -.0454/-.0497, p=.022/.023p=.022/.023; Qwen: -.0875/-.0510, p=.003/.025p=.003/.025), while +ReasBank 40d holds (DS: -.0072/-.0006; Qwen: +.0272/+.0183).
  2. Delay (C — Cold d1..40 vs. Mature d41..299, 40d): +ReasBank trails Pipe in d1..40 (-.0644 DS; -.0822 Qwen) before 40d outcomes mature, then beats Pipe in d41..299 (DS: +.0289/+.0320, p=.047/.023p=.047/.023; Qwen: +.0585/+.0567, p<.001p<.001, standalone +.1225; Qwen +MemRL: -.0602, p=.013p=.013).
  3. Recurrence (R — Fig. 3(b) News Split): Median-splitting all 31 sectors by unique importance>=8 news Mi∈[67,9795]M_i \in [67, 9795] (1,344 cutoff) gives 15 sparse (LowRep, mean 627) vs. 16 dense (HighRep, mean 4,918, 7.8×7.8\times) sectors. Reflection excels in HighRep (Qwen 20d csIC +.1672, +.1109 vs. Pipe, p=.001p=.001) but shows 40d negative transfer in LowRep (DS +Reflexion ΔcsIC=-.0840, p=.024; DS +ExpeL ΔtsIC=-.0287, p=.033), while +ReasBank 40d beats Pipe (DS +.0185, p=.031; Qwen +.0724, p=.065).

3. Stability & Block Bootstrap (Q2)

Over 8,818–9,269 forecasts/run (0 parse failures; temp=0.5, sglang), Paired MBB (B=5,000, L=h) confirms negative transfer in Mem0 (DS 20d: -.0192, p=.027; Qwen avg: -.0143, p=.041; full 20d: -.0340, p=.006) and MemRL (Qwen 40d: -.0542, p=.010; avg: -.0267, p=.046), and 40d gains in ReasBank (Qwen 40d: +.0448, p=.058; mature p<.001p<.001).

4. Backbones & Contamination Analysis (Q3)

We evaluate frozen offline models (sglang, zero web)—Qwen/Qwen3.6-35B-A3B (2026-04-16, Blog) and deepseek-ai/DeepSeek-V4-Flash (2026-04-24, Release)—over 2025-01-01 to 2026-03-31 (299 trading days). Although this window precedes April 2026 open-sourcing, data leakage is unlikely: (1) Offline Prompting Verification: direct prompt-engineering probes under offline deployment cannot elicit Shenwan L1 index prices or forward trends (Pipe gets only 0.018–0.060 tsIC, and Method - Pipe cancels static priors); and (2) Pretraining Pipeline Dynamics: base-model data curation and training span long periodic cycles pre-release, where financial corpora form a tiny share and raw high-noise market series rarely enter backbone training pipelines without dedicated processing.

5. Layout (Q5) & OpenPM (Q6)

We fixed Fig. 2 overlapping lines 251–253 (p. 5) and lines 139/170 typos, and contrasted OpenPM (arXiv:2608.09988) (OpenPM-Bench)—44d S&P 500 auditing—with our 300d self-evolution testbed.

Thank the author's response and it answered all my questions. I have updated my score to 3.

Reviewer_B1MzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FinEvolveBench evaluates whether agents with frozen model parameters benefit from accumulating external experience in a chronological financial prediction environment. Agents predict industry excess returns over 10, 20, and 40 trading days from news, and experience is updated only when the corresponding feedback matures. The benchmark covers 31 industries and 300 prediction days, with two backbones and five experience mechanisms. The paper uses this setting to study experience under delayed outcomes, difficult attribution, environmental change, and limited reuse, reporting inconsistent gains over a pipeline without experience.

Critical strengths
  1. A worthwhile evaluation problem. Testing experience under delayed, noisy outcomes can reveal limitations missed by immediate-feedback tasks. The intended replay protocol specifies when feedback matures, predictions freeze, and new memories become retrievable, providing a useful basis for reproducible experience-learning evaluation.

  2. The results expose informative heterogeneity. Cross-sectional and time-series correlations across horizons and two backbones show that experience does not consistently improve every setting. Positive cases, including Qwen ReasoningBank at 40 days, prevent interpreting this as universal failure. Such negative and mixed results can benefit memory research without requiring a new algorithm or a profitable trading strategy.

Critical weaknesses
  1. The protocol producing the reported results cannot yet be confidently reconstructed. Section 3.1 removes market-wide news, but Mem0/MemRL examples contain All Sectors items (pp. 25, 28). Equation 2 defines market-adjusted return feedback, while Reflexion examples reason about 91.20, 101.42, and 34.16 index-point gains and compare them with bounded prediction scores (pp. 20–23). The May 30 case also describes accumulated utility .85 for a May 29 memory before its shortest outcome horizon could mature (pp. 28–29). Initialization or examples from another version may explain these observations; they are not proof of leakage. Their connection to Table 3 is nonetheless essential to benchmark validity.

  2. Method-level conclusions are not sufficiently separated from adaptation choices and sampling variation. Appendix A's horizon-specific utility language needs reconciliation with B.7's shared MemRL Q value across three horizons. ExpeL's single-trajectory insights and recent-K retrieval also define a particular adaptation; context allocation must establish whether memory displaces current news. Meanwhile, overlapping return windows and correlated industries require dependence-aware uncertainty for paired differences, particularly for gains such as .0001 average csIC. These factors limit attributing the observed differences to experience mechanisms or to the proposed four-dimension taxonomy.

Questions for authors
  1. Could you provide the May 30 case's raw input, feedback payload with units, and memory creation/update timestamps, and connect them to the Table 3 run version? What produced the .85 score before the memory's first outcome could mature?

  2. Does MemRL use shared or separate utility per horizon? Please explain the Appendix A/B wording and specify each method's adaptation and context-allocation rules, including whether experience can truncate current news.

  3. Can you report paired uncertainty that preserves temporal dependence and the parsing-failure/intersection counts used for comparison? These would clarify which reported differences are stable on comparable observations.

2 comments
AuthorsOpenReview ↗

Response to Reviewer B1Mz

We thank Reviewer B1Mz for the forensic review of our text, appendices, and logs, and for noting that FinEvolveBench needs neither a new algorithm nor a profitable trading strategy once the benchmark contract is clear and paired uncertainty is quantified. Every discrepancy traced to legacy adapter details audited below.

1. Reconciling Appendix C Logs, Protocol Documentation, and Table 3 (W1 & Q1)

Re-running eval/evaluator.py across all 12 runs ([deepseek-ai/DeepSeek-V4-Flash] and [Qwen/Qwen3.6-35B-A3B]×\times 6 methods) reproduces 70 of 72 Table 3 cells to 4 decimals (0.0000 diff). We will conduct additional independent runs and report more stable mean results with uncertainty estimates in the camera-ready version. Appendix C logs come directly from these main runs:

  1. 2025-05-29 Memory Showing Memrl_score: 0.85 on 2025-05-30 (Zero Future Leakage): New memories in memrl/memrl.py start at Q0=0.0Q_0 = 0.0 (score = 0.5 + 0.5\tanh(Q/0.05) = 0.50). Our Pipe, Mem0, and MemRL runs predicted five horizons (1, 5, 10, 20, 40d; visible in 1d/, 5d/ folders and App. C lines 1397 & 1599: "short term (1–5 days)"; 1d, 5d omitted from Table 3 for space). On 05-29, DS +MemRL predicted 1d = +0.70 on Transportation (801170.SI). After 05-30 close, the realized 1d excess return (05-29 to 05-30) was +0.73% vs. CSI 1000 (+0.48% CS-demeaned). Under our causal order (post-close maturity -> memory update -> next prediction), this positive 1d utility updated Q←Q+0.3(u1d−Q)Q \leftarrow Q + 0.3(u_{1\text{d}} - Q), raising Memrl_score to 0.85 with zero leakage of future 10/20/40d outcomes (-0.96% / -4.41% / -8.12%).
  2. All Sectors News (id=41014) vs. Section 3.1: Section 3.1 describes our 31-sector prediction universe and the strict 31-sector filter in Reflexion, ExpeL, and ReasoningBank (8,818 active sector-days). Pipe, Mem0, and MemRL (max_broad_news=10) prepended up to 10 importance>=8 macro articles (id=41014), also predicting on 451 sector-days lacking sector news (9,269 pairs). Evaluating on the 8,818 common intersection (Part 2) removes any asymmetry; we will document this in Sec. 3.1 and App. B.
  3. Index Points (+91.20) vs. Excess Returns: Legacy Reflexion/ExpeL reflection prompts formatted historical outcomes as Shenwan L1 point diffs ΔP=Pt+h−Pt\Delta P = P_{t+h} - P_t (README.md line 127), whereas MemRL, ReasoningBank, and refactored Reflexion/workflow.py pass excess returns αi,t,h\alpha_{i,t,h}. Final evaluation in eval/evaluator.py is 100% identical across all 12 runs, using CS-demeaned close-to-close excess returns and Pooled Pearson tsIC (we will align Eq. 2 & App. A.4 Eq. 6 verbatim with eval/metrics.py; for csIC, CS-demeaning and subtracting CSI 1000 are identical).

Audit Record for 2025-05-30 Transportation (801170.SI) (Q1):

  • Raw News ([05-26, 05-30], importance>=8): 25 sector items (9,054 chars, 100% preserved): US-China container rush (>40% freight spike), -19.8% MoM April US import drop, Red Sea rerouting, autonomous logistics.
  • Ground Truth (2236.52 close on 05-30): 10d (06-13) 2196.46 (-40.06 pts, CSI 1000 -2.37%, CS -1.78%); 20d (06-27) 2196.71 (-39.81 pts, -6.58% / -4.17%); 40d (07-25) 2289.20 (+52.68 pts, -10.75% / -8.30%).
Method (DS)(10d,20d,40d)Memory on 05-30Post-Close Maturity
Pipe(-.10,-.20,-.30)None (all 3 signs right)None (stateless)
+Mem0(+.10,+.30,+.30)Top-5 shipping episodesAppended 05-30
+MemRL(+.50,+.40,+.30)05-29 (0.85 via +.73% 1d)QQ updated 05-30..07-25
+Reflexion(+.70,+.50,+.30)Reflection (13,262 chars)06-13/06-27/07-25 (pts)
+ExpeL(+.70,+.40,+.20)Rules (10,026 chars, top-10)06-13..07-25 rule update
+ReasBank(+.70,+.30,-.20)Top-3 principles (-.20 40d)06-16/06-30/07-28 (α\alpha)
AuthorsOpenReview ↗

Response to Reviewer B1Mz About Implementation Fidelity & Paired Bootstrap

2. Implementation Fidelity, MemRL Shared QQ & Zero Truncation (W2 & Q2)

  • MemRL Single Shared QQ (App A.3 vs. B.7) & Ongoing Horizon-Separated Ablation: Thank you for catching this mismatch and raising horizon decoupling. App. B.7 matches memrl/experiment.py across all runs: each memory keeps strictly one shared scalar record["q"] (under our 1-call-per-sector-day protocol), updated via Q←Q+0.3(uh−Q)Q \leftarrow Q + 0.3(u_h - Q) as horizons mature; App. A.3's "horizon-specific utility" meant horizon-matured reward, which we will fix. Inspired by your question, we are running horizon-separated experiments across all methods (still compiling under tight server capacity; our prior empirical experience shows minimal difference between separate vs. merged calls for LLM methods like Mem0/Reflexion/ExpeL), and we will add full results and analysis in the Appendix of the final version.
  • ExpeL Adaptation: With one continuous trajectory per (date, sector) and no resets, Expel/exp_memory.py re-abstracts rules online over the 10 latest reflections (maxReflections=10, cap 4,000 chars).
  • Zero News Truncation (0.00%): Experience blocks are capped upstream (4,000 chars in Reflexion/ExpeL; top-5 ≤800\le 800 chars in Mem0/MemRL; top-3 principles in ReasBank; reflectMaxNewsChars=3000 is only in _buildReflectContext, never predict()). Prompt lengths (min/med/P99/max, 8,818 calls): 147/6,729/39,174/58,767 (Reflexion), 147/6,120/36,661/54,514 (ExpeL), 998/6,161/36,104/53,945 (ReasBank), inside 65536 (max ~34k tokens; 0.00% truncation).

3. Valid Counts & Paired (Method - Pipe) Block Bootstrap (W2 & Q3)

  • Zero Parse Failures: Across all 12 runs and horizons, parse failures = 0 (0.00%). Common intersections (Method ∩\cap Pipe) on DS: 9,269/9,269/9,268 (Mem0), 9,269/9,269/9,267 (MemRL), 8,818 (Reflexion/ExpeL/ReasBank); on Qwen (Pipe: 270d at 10/20d, 299d at 40d): 8,269/8,269/9,257, 8,220/8,220/9,246, 7,952/7,952/8,806.
  • Paired MBB (B=5,000, L=hL=h day-blocks, ΔtsIC/ΔcsIC (p)): Cancels macro shocks, cutting SEs by 45%–70%:
Method10d Δts/Δcs20d Δts/Δcs40d Δts/ΔcsAvg Δts/Δcs
DS Mem0-.0040/-.0028 (.75/.87)-.0192/-.0168 (.027/.047*)-.0052/+.0093 (.62/.42)-.0095/-.0035 (.18/.68)
DS MemRL-.0068/-.0030 (.68/.94)-.0335/-.0349 (.074/.10)-.0061/-.0058 (.71/.69)-.0155/-.0146 (.22/.31)
DS Reflex+.0001/+.0114 (.86/.50)+.0179/+.0203 (.36/.24)+.0107/-.0063 (.71/.90)+.0096/+.0085 (.54/.60)
DS ExpeL+.0102/+.0221 (.50/.26)-.0002/-.0041 (.73/.84)+.0006/-.0131 (.94/.77)+.0035/+.0016 (.81/.91)
DS ReasBk-.0070/-.0018 (.80/.91)-.0034/-.0036 (.80/.81)+.0183/+.0194 (.25/.19)+.0026/+.0047 (.81/.69)
Qw Mem0-.0054/-.0044 (.57/.68*)-.0154/-.0046 (.057/.65*)-.0213/-.0214 (.25/.39)-.0143/-.0110 (.041/.25)
Qw MemRL-.0138/-.0164 (.32/.24)-.0102/+.0087 (.41/.97)-.0542/-.0523 (.010/.012)-.0267/-.0217 (.046/.14)
Qw Reflex-.0020/+.0159 (.97/.48)+.0045/+.0195 (.73/.49)-.0138/-.0261 (.78/.65)-.0025/+.0050 (.89/.81)
Qw ExpeL-.0040/+.0110 (.84/.70)+.0226/+.0432 (.29/.11)-.0152/-.0292 (.66/.53)+.0014/+.0090 (.93/.67)
Qw ReasBk-.0082/-.0073 (.71/.73)+.0150/+.0175 (.21/.21)+.0448/+.0386 (.058/.15)+.0211/+.0224 (.060/.11)

*95% MBB CIs: DS Mem0 20d [-.0334,-.0021] (NW Δcs\Delta\text{cs} p=.047p=.047); Qw Mem0 avg [-.0304,-.0009] (full 10/20d -.0217/-.0340, p=.020/.006p=.020/.006); Qw MemRL 40d [-.1108,-.0087], avg [-.0640,-.0005]; Qw ReasBk 40d [-.0019,+.0911], DS 40d Δcs\Delta\text{cs} [-.0115,+.0577].

  • Opposing Regimes: Full-sample averages pool cold-start/drift with mature/stationary phases: (i) in Mature 40d (d41..299), +ReasBank beats Pipe on both backbones (DS: +.0289/+.0320, p=.047/.023p=.047/.023; Qwen: +.0585/+.0567, p<.001p<.001); (ii) in LowVol, DS +Reflexion 20d beats Pipe by +.0782 (p=.014p=.014) and Qwen +ReasBank 40d by +.0606 (p=.005p=.005), while in HighVol, Reflexion/ExpeL show 40d negative transfer vs. Pipe (DS: -.0454/-.0497, p=.022/.023p=.022/.023; Qwen: -.0875/-.0510, p=.003/.025p=.003/.025).
Reviewer_kqjTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FinEvolveBench, a benchmark for evaluating experience-based self-evolving LLM agents under challenging real-world conditions. The authors characterize experience-learning scenarios along four dimensions: outcome observability, credit assignment complexity, environmental dynamics, and experience reusability. FinEvolveBench uses chronological financial news and market data covering 31 Chinese A-share industries, with delayed feedback and a causal replay protocol to prevent future information leakage. Experiments with five representative experience mechanisms across two LLM backbones and three prediction horizons show that existing methods often fail to consistently outperform an experience-free baseline, highlighting the difficulty of reliable experience learning in non-stationary environments

Critical strengths
  1. The paper identifies an important limitation of existing self-evolving agent benchmarks, where experience is often evaluated under relatively favorable conditions. The proposed four-dimensional taxonomy provides a useful perspective for understanding these limitations.

  2. The benchmark includes a relatively large collection of real-world financial news and market data. The chronological replay protocol with delayed feedback is well motivated and helps prevent future information leakage.

Critical weaknesses
  1. The benchmark combines all four proposed challenges in a single financial prediction setting, making it difficult to understand which factors actually cause the performance degradation. The paper lacks controlled experiments or ablations to isolate the effects of noisy feedback, credit assignment, non-stationarity, and experience reusability.

  2. The experimental results do not sufficiently support the main conclusions. Most reported IC values are relatively small, and the paper does not provide statistical significance tests or uncertainty estimates. It is therefore unclear whether the differences between methods are meaningful or simply reflect noise in financial markets.

Questions for authors

Could the authors provide controlled experiments to isolate the contribution of each of the four proposed challenges? This would help establish whether the observed failures are caused by the intended benchmark properties.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kqjT

We thank Reviewer kqjT for recognizing FinEvolveBench's motivation, our 4D taxonomy (O, C, D, R), and our causal replay design. Across all 12 runs ([deepseek-ai/DeepSeek-V4-Flash]and [Qwen/Qwen3.6-35B-A3B]×\times 6 paradigms over 299 days and 8,818 active forecasts/horizon), we resolve both concerns: (1) controlled isolation of O, C, D, R (W1/Q1), and (2) statistical significance of 0.01–0.10 ICs (W2).

1. Controlled Empirical Isolation of Individual Dimensions (O, C, D, R) (W1 & Q1)

Paired differencing against stateless Pipe (Method - Pipe) on identical trading days removes baseline market predictability, cleanly isolating each axis (O, C, D, R) via Moving Block Bootstrap (MBB):

  1. O (Outcome Noise — Horizon SNR Gradient: 10d noise →\to 20d →\to 40d fundamental drift):
  • +ReasoningBank scales monotonically as horizon SNR rises: on DS, 10d tsIC +.0124 (vs. Pipe Δ=−.0070\Delta=-.0070) →\to 20d +.0419 →\to 40d +.0699 (Δ=+.0183\Delta=+.0183); on Qwen, 10d +.0101 (Δ=−.0082\Delta=-.0082) →\to 20d +.0465 →\to 40d +.1033 (vs. Pipe Δ=+.0448,p=.058\Delta=+.0448, p=.058). High 10d noise corrupts credit assignment, whereas higher 40d SNR unlocks positive transfer in principle distillation.
  1. C (Delayed Credit Assignment — Cold-Start d1..40 [zero 40d feedback] vs. Mature d41..299 [259d]):
  • Before any 40d outcome matures (d1..40), +ReasoningBank 40d trails Pipe (-.0644 DS; -.0822 Qwen). Once 40d feedback matures (d41..299), +ReasoningBank significantly beats Pipe on both backbones: DS +.0289/+.0320 (p=.047/.023p=.047/.023) (standalone +.0915); Qwen +.0585/+.0567 (p<.001p<.001) (standalone +.1225), while Qwen +MemRL drops to -.0602 (p=.013p=.013)—isolating 40d delay cost (C).
  1. D (Non-Stationary Drift — Causal Volatility Split: 10d CSI 1000 σt−9:t\sigma_{t-9:t} into LowVol 150d vs. HighVol 149d):
  • In stationary LowVol, agents beat Pipe: DS +Reflexion 20d hits tsIC = +.1472 (p<.001p<.001, +.0782 vs. Pipe, p=.014p=.014); Qwen +ReasoningBank 40d hits +.1165 (p<.001p<.001, +.0606 vs. Pipe, p=.005p=.005). In HighVol shifts, uncurated 40d reflections/rules flip to negative transfer vs. Pipe (DS ΔcsIC = -.0454, p=.022; Qwen = -.0875, p=.003), while +ReasoningBank 40d avoids collapse (Qwen: +.0272 over Pipe). Thus, agents are accurate (IC = 0.11–0.15, p<.001p<.001) in stationary regimes; negative transfer stems from drift (D).
  1. R (Low Repetition — Figure 3(b) Unique High-Salience News Median Split):
    • Ranking all 31 Shenwan L1 industries by unique importance >= 8 news count Mi∈[67,9795]M_i \in [67, 9795] and splitting at the median (1,344 cutoff, zero ties) defines 15 sparse long-tail (LowRep, mean 627 articles / 2.1/d, N=4,034N=4,034) vs. 16 dense head (HighRep, mean 4,918 / 16.4/d — 7.8×7.8\times density, N=4,784N=4,784) sectors. Verbal reflection thrives in HighRep (Qwen 20d csIC +.1672, +.1109 vs. Pipe, p=.001p=.001; DS 40d +.0822/+.0497), but flips to 40d negative transfer in LowRep (DS +Reflexion: -.0477/-.0991, vs. Pipe ΔcsIC=−.0840,p=.024\Delta\text{csIC}=-.0840, p=.024; DS +ExpeL: ΔtsIC=−.0287,p=.033\Delta\text{tsIC}=-.0287, p=.033), whereas +ReasoningBank 40d beats Pipe in LowRep (DS: +.0185, p=.031p=.031; Qwen: +.0724, p=.065p=.065). We will add this four-axis isolation analysis as a dedicated subsection in Section 5.

2. Statistical Significance & Why Full-Sample ICs Are 0.01–0.10 (W2)

  1. Dependence-Aware Significance (Paired MBB, B = 5,000, L = h): Resampling hh-day 31-sector blocks confirms significant transfer in both directions:
  • Negative Transfer: +Mem0 (DS 20d: -.0192, p=.027, NW-HAC p=.047p=.047; Qwen avg: -.0143, p=.041p=.041; full 20d: -.0340, p=.006p=.006), +MemRL (Qwen 40d: -.0542, p=.010$, $\Delta\text{csIC} = -.0523, p=.012$; DS 20d HighVol: -.0425, p=.010), and 40d verbal reflections/rules in HighVol($p=.003\text{–}.025$) andLowRep` (p=.020–.037p=.020\text{–}.037).
  • Positive Transfer: Stationary LowVol ICs reach +0.1472 ([+.0773, +.2160], p<.001p<.001) on DS and +0.1331 (p<.001p<.001) on Qwen, while mature (d41..299) +ReasoningBank 40d hits +0.1225 (p<.001p<.001, +.0585 over Pipe, p<.001p<.001) on Qwen and +0.0915 (+.0320 csIC over Pipe, p=.023p=.023) on DS. Full-sample ICs (0.02–0.10) reflect positive stationary/mature transfer (+0.10–0.15) cancelling against cold-start/drift/sparse negative transfer (-0.02 to -0.10).
  1. Signal Scale: Exogenous shocks bound equity out-of-sample R2=IC2R^2 = \text{IC}^2 near 0.1%–1.5% (∣IC∣≈0.03–0.12|\text{IC}| \approx 0.03\text{–}0.12). Across 3131 sectors over 299299 days (N=8,818N=8,818), csIC of 0.04–0.10 yields daily breadth csIC31≈0.22–0.56\text{csIC}\sqrt{31} \approx 0.22\text{–}0.56, while a -0.025 to -0.054 loss (Mem0/MemRL) erases most signal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.