- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors identify three main sources of hallucinations that limit effectiveness of financial agents: evidence misalignment, context noise accumulation, and unreliable experience reuse. FinHarness is the proposed agent framework to address the three limitations using (a) an adaptive evidence retriever that allocates agents and tools considering task-specific evidence requirements, (b) a dynamic context orchestrator for controlling which outputs are exposed, and (c) a skill-based experience manager that compiles verified trajectories into parameterized skills. Empirical results on ProFinR demonstrate that FinHarness raises the overall score over ProFinAgent across four backbones with reduced hallucination rates. In a 6-month, 5-stock backtest, FinHardness with a 4B model shows notable performance improvement over AlphaQuanter models.
Critical strengths
-
Strong empirical results with ProFinR. Across a series of both open and proprietary models, FinHarness shows notable gains in average accuracy with largely consistent gains across task types (Table 1). In terms of error and hallucination rates, FinHarness also shows substantial improvements, supporting the core claim that the framework is designed to address sources hallucinations in financial agents (Table 2).
-
Favorable comparison against training-based methods. In a 6-month, 5-stock backtest, the authors compare FinHarness against multiple variants of AlphaQuanter to show that the framework compares favorably against backbones of larger scale as well as RL-trained trading policy.
Critical weaknesses
-
Some comparisons do not adequately isolate the respective components. First, the claim that skills "outperform the strongest experience baseline by 20.07%" is based on comparing FinHarnessSkill (52.08%) with ACE (32.01%). However, ACE, DC, and ICL are implemented with the fixed-role workflow, whereas the skills are integrated into FinHarness. Thus, a substantial portion of the improvement may be due to the framework rather than the memory component. Second, removing the dynamic context orchestrator reduces performance significantly, well below all tool-using baselines. That suggests the ablation breaks the pipeline rather than isolating the value of filtering.
-
While the backtest results show useful comparison, several concerns exist. One is that the main backbone (Qwen3-4B) was released after the test window, so some data contamination is potentially possible. The test covers only 5 large-cap US stocks over a rather short period of time, with no variance analysis. Generalization claims based on the results therefore seem to require additional support.
Questions for authors
Q. For trading, how can you rule out potential data contamination from backbones released after the test window? Can we evaluate other backbones released earlier (or, evaluate the same models on a more recent window)?
5 comments
W2 & Q1: Clarification and Supplementary Experiments on Data Leakage and Generalization.
We sincerely appreciate the reviewer's careful observations and constructive suggestions. We have incorporated your feedback by conducting extensive supplementary experiments and providing detailed clarifications to address your concerns regarding potential data contamination and the backtesting window, and we hope these additions will successfully resolve your doubts and earn your approval.
Table 4: Performance comparison of various models and frameworks in trading simulation (2026-01-02 to 2026-06-30, 123 trading days).
| Model/Framework | GOOGL | META | MSFT | NVDA | TSLA | Average | ||
|---|---|---|---|---|---|---|---|---|
| ARR (↑) | ARR (↑) | ARR (↑) | ARR (↑) | ARR (↑) | ARR (↑) | SR (↑) | MDD (↓) | |
| Market | ||||||||
| B&H | 27.86% | -28.28% | -41.08% | 11.31% | -16.20% | -9.28% | -0.038 | 23.60% |
| TradingAgents | ||||||||
| Qwen3-4B-Instruct-2507 | 25.09% | -14.89% | -18.57% | -2.70% | -41.08% | -10.43% | -0.255 | 14.55% |
| GPT-4o-mini | 27.24% | -24.04% | -30.71% | 7.06% | -16.19% | -7.33% | -0.049 | 23.93% |
| GPT-4o | 39.28% | -24.07% | -6.54% | -20.45% | -15.67% | -5.49% | -0.124 | 13.53% |
| AlphaQuanter w/o RL | ||||||||
| Qwen3-4B-Instruct-2507 | 37.22% | -42.15% | -12.99% | -9.71% | -28.52% | -11.23% | -0.024 | 20.98% |
| Ours | ||||||||
| FinHarness | 20.64% | -43.00% | -15.91% | -22.02% | -15.08% | -15.07% | -0.428 | 20.10% |
| FinHarnessSkill | 30.38% | -14.27% | 7.88% | 9.21% | -6.89% | 5.26% | 0.239 | 12.74% |
Table 5: Detailed Sharpe Ratio across various models and frameworks.
| Model / Framework | GOOGL | META | MSFT | NVDA | TSLA |
|---|---|---|---|---|---|
| Market | |||||
| B&H | 0.689 | -0.371 | -0.869 | 0.356 | 0.006 |
| TradingAgents | |||||
| Qwen3-4B-Instruct-2507 | 1.019 | -0.613 | -0.490 | -0.027 | -1.165 |
| GPT-4o-mini | 0.646 | -0.347 | -0.649 | 0.259 | -0.153 |
| GPT-4o | 1.570 | -0.759 | -0.271 | -0.547 | -0.613 |
| AlphaQuanter w/o RL | |||||
| Qwen3-4B-Instruct-2507 | 0.085 | -0.102 | -0.025 | -0.013 | -0.065 |
| Ours | |||||
| FinHarness | 0.550 | -1.086 | -0.347 | -0.534 | -0.723 |
| FinHarnessSkill | 1.002 | -0.197 | 0.388 | 0.316 | -0.316 |
Table 6: Detailed Maximum Drawdown across various models and frameworks.
| Model / Framework | GOOGL | META | MSFT | NVDA | TSLA |
|---|---|---|---|---|---|
| Market | |||||
| B&H | 20.36% | 28.72% | 26.69% | 18.23% | 24.00% |
| TradingAgents | |||||
| Qwen3-4B-Instruct-2507 | 4.62% | 11.06% | 18.65% | 11.03% | 27.41% |
| GPT-4o-mini | 20.36% | 29.81% | 25.64% | 18.23% | 25.63% |
| GPT-4o | 4.41% | 20.25% | 8.55% | 18.57% | 15.86% |
| AlphaQuanter w/o RL | |||||
| Qwen3-4B-Instruct-2507 | 19.43% | 29.70% | 15.23% | 19.51% | 21.01% |
| Ours | |||||
| FinHarness | 18.32% | 31.03% | 18.34% | 17.01% | 15.80% |
| FinHarnessSkill | 7.40% | 20.42% | 7.88% | 19.19% | 8.80% |
In the original manuscript, Table 13 and Table 14 present experimental results using Qwen2.5-7B. According to the official website [1], this model was publicly deployed in 2024. This timeline strictly precedes our 2025 backtesting period and effectively eliminates any risk of training data leakage. To address the concerns regarding potential data leakage introduced by Qwen3-4B-Instruct-2507, we designed an additional out-of-sample evaluation. We evaluated the same five stocks over an additional period from January 2 to June 30, 2026, covering 123 trading days. In the additional backtest, FinHarnessSkill achieved the highest average annualized rate of return of 5.26% and the highest average Sharpe ratio of 0.239 across the five stocks. It also reduced average maximum drawdown from 20.10% for FinHarness to 12.74%, the lowest among the evaluated methods. These results suggest that financial skills distilled from verified financial reasoning trajectories can transfer to trading and improve performance in an additional market period. We have included the main results of the additional backtest in Table 15 of the appendix to the revised manuscript.
W2. Addressing Variability Across Stocks in the Trading Backtest.
We sincerely appreciate the reviewer's suggestions for a variance analysis of our trading results. To quantify how performance varies across stocks, we report a paired comparison between FinHarness and FinHarnessSkill on the same five stocks with the same backbone in the 2026 backtest.
Table 7: Stockwise ARR and Variance Comparison in the 2026 Trading Backtest.
| Stock | FinHarness ARR | FinHarnessSkill ARR | ARR difference |
|---|---|---|---|
| GOOGL | 20.64% | 30.38% | +9.74% |
| META | -43.00% | -14.27% | +28.73% |
| MSFT | -15.91% | 7.88% | +23.79% |
| NVDA | -22.02% | 9.21% | +31.23% |
| TSLA | -15.08% | -6.89% | +8.19% |
| Mean across stocks | -15.07% | 5.26% | +20.34% |
| Standard deviation across stocks | 22.94% | 17.19% | 10.73% |
For each stock, the ARR difference is computed by subtracting the FinHarness ARR from the FinHarnessSkill ARR. The mean difference is calculated across the five stock-wise differences before rounding. FinHarnessSkill improves ARR on all five stocks, with gains ranging from 8.19% to 31.23% and an average gain of 20.34%. Its ARR is positive on three stocks and ranges from −14.27% to 30.38%. These results confirm that verified financial reasoning trajectories enhance returns and mitigate cross-stock volatility.
W1: Addressing Concerns on Component Isolation: Fair Comparisons and Ablation Validity.
We are deeply grateful for your valuable recommendations. Inspired by your meticulous observations, we have provided further detailed clarifications and expanded our comparisons to rigorously address your concerns regarding component isolation and the underlying rationale for our ablation design. We sincerely hope these targeted updates fully resolve your doubts and meet your expectations.
Table 1: Ablation of Experience Learning Frameworks with the FinHarness Baseline on ProFinR.
To address the reviewer's concern about component isolation and framework consistency, we clarify a crucial detail from the original manuscript. As shown in Table 5 of the original manuscript, which we present as Table 1 above, all experience-learning baselines, including ACE, DC (CU), and ICL, were integrated into and evaluated within the same FinHarness framework. By strictly controlling this architectural variable, we ensure an absolutely fair comparison. Under this fair setup, FinHarnessSkill achieves a total score of 52.08%, comprehensively outperforming all other experience-learning methods. In addition, our method maintains a substantial 19.88% margin over the ACE baseline implemented within the same framework. This rigorous ablation definitively isolates the variables and proves that the performance improvements stem directly from the accumulated financial reasoning experience. The gains do not emerge from structural differences across disparate frameworks.
Table 2: Soft ablation of the Dynamic Context Orchestrator (DCO) using context thresholds.
To address the reviewer's concern that completely removing the Dynamic Context Orchestrator (DCO) might disrupt the execution pipeline, we conducted a "soft ablation" experiment. Instead of leaving the agent with an unbounded context that causes systemic failures, we replaced the DCO with a standard heuristic context-bounding strategy. Specifically, we limited inputs to 10,000 characters per document and applied LlamaIndex for semantic matching with a 0.5 similarity threshold. This setup ensures the agent's pipeline remains fully functional while strictly isolating the necessity of DCO's explicit verification, as reflected in the severe performance drop to 2.33% on L3 tasks.
Table 3: Ablation experiments on DeepSeek-V4 Flash.
To evaluate architectural dependencies across model scales, we further analyze the ablation results for DeepSeek V4 Flash. Removing the AER module leads to a severe performance drop on Level 3 tasks across both models, underscoring the necessity of dynamic subagent activation and iterative repair. Because complex financial tasks demand multi-step evidence acquisition where initial tool executions frequently produce incomplete or noisy data, removing AER causes the reasoning chain to break permanently at the first point of failure. Conversely, removing the DCO module reveals distinct scale-dependent resilience. While removing DCO forces the smaller Qwen3-4B-Instruct-2507 into a systemic L3 crash with a drastic proliferation of hallucinated responses, DeepSeek-V4-Flash sustains moderate performance and resists this hallucination spike because its superior context capacities and robust attention mechanisms prevent severe attention sparsity under unfiltered noisy context. This contrast demonstrates that DCO is indispensable for filtering evidence and enabling smaller models to execute reliable financial reasoning.