acceptodds
Under review as a conference paper at ICLR 2027

Towards Reliable Financial Agents: Mitigating Financial Hallucinations through Evidence-Grounded Execution and Verified Experience

Abstract

Despite rapid progress in LLM capabilities, constructing financial agents remains a long-standing challenge due to hallucinations caused by evidence misalignment, context noise accumulation, and unreliable experience reuse. To address these issues, we introduce FinHarness, a financial agent framework that improves workflow reliability by integrating task-adaptive execution, candidate evidence verification, and verified skill accumulation. Extensive experiments show that FinHarness consistently enhances financial reasoning across various LLM backbones on the professional financial reasoning benchmark, achieving gains of up to 19.89% over state-of-the-art baselines while reducing the hallucination rate among erroneous responses by up to 73.23%. Moreover, incorporating accumulated skills further increases FinHarness's financial reasoning score from 41.86% to 52.08%. To evaluate cross-task generalization, we transfer FinHarness and the accumulated skills to the out-of-domain stock-trading task, where the skill-augmented framework outperforms all training-free baselines. These results show that evidence-aligned financial agent workflows can improve financial reasoning while reducing financial hallucinations across execution, context construction, and experience reuse.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Agents, Financial Agents, Financial Hallucination, Agentic Workflows, Verified Skill Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dUQKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify three main sources of hallucinations that limit effectiveness of financial agents: evidence misalignment, context noise accumulation, and unreliable experience reuse. FinHarness is the proposed agent framework to address the three limitations using (a) an adaptive evidence retriever that allocates agents and tools considering task-specific evidence requirements, (b) a dynamic context orchestrator for controlling which outputs are exposed, and (c) a skill-based experience manager that compiles verified trajectories into parameterized skills. Empirical results on ProFinR demonstrate that FinHarness raises the overall score over ProFinAgent across four backbones with reduced hallucination rates. In a 6-month, 5-stock backtest, FinHardness with a 4B model shows notable performance improvement over AlphaQuanter models.

Critical strengths
  1. Strong empirical results with ProFinR. Across a series of both open and proprietary models, FinHarness shows notable gains in average accuracy with largely consistent gains across task types (Table 1). In terms of error and hallucination rates, FinHarness also shows substantial improvements, supporting the core claim that the framework is designed to address sources hallucinations in financial agents (Table 2).

  2. Favorable comparison against training-based methods. In a 6-month, 5-stock backtest, the authors compare FinHarness against multiple variants of AlphaQuanter to show that the framework compares favorably against backbones of larger scale as well as RL-trained trading policy.

Critical weaknesses
  1. Some comparisons do not adequately isolate the respective components. First, the claim that skills "outperform the strongest experience baseline by 20.07%" is based on comparing FinHarnessSkill (52.08%) with ACE (32.01%). However, ACE, DC, and ICL are implemented with the fixed-role workflow, whereas the skills are integrated into FinHarness. Thus, a substantial portion of the improvement may be due to the framework rather than the memory component. Second, removing the dynamic context orchestrator reduces performance significantly, well below all tool-using baselines. That suggests the ablation breaks the pipeline rather than isolating the value of filtering.

  2. While the backtest results show useful comparison, several concerns exist. One is that the main backbone (Qwen3-4B) was released after the test window, so some data contamination is potentially possible. The test covers only 5 large-cap US stocks over a rather short period of time, with no variance analysis. Generalization claims based on the results therefore seem to require additional support.

Questions for authors

Q. For trading, how can you rule out potential data contamination from backbones released after the test window? Can we evaluate other backbones released earlier (or, evaluate the same models on a more recent window)?

5 comments
AuthorsOpenReview ↗

W1: Addressing Concerns on Component Isolation: Fair Comparisons and Ablation Validity.

We are deeply grateful for your valuable recommendations. Inspired by your meticulous observations, we have provided further detailed clarifications and expanded our comparisons to rigorously address your concerns regarding component isolation and the underlying rationale for our ablation design. We sincerely hope these targeted updates fully resolve your doubts and meet your expectations.

Table 1: Ablation of Experience Learning Frameworks with the FinHarness Baseline on ProFinR.

MethodL1L2L3AVG.Total Score
FinHarness59.62%32.46%27.13%39.74%41.86%
FinHarnessSkill66.83%45.55%37.98%50.12%52.08%
ACE39.42%28.27%26.36%31.35%32.20%
DC (CU)56.25%45.55%32.56%44.79%46.59%
ICL55.29%38.22%26.36%39.96%42.05%

To address the reviewer's concern about component isolation and framework consistency, we clarify a crucial detail from the original manuscript. As shown in Table 5 of the original manuscript, which we present as Table 1 above, all experience-learning baselines, including ACE, DC (CU), and ICL, were integrated into and evaluated within the same FinHarness framework. By strictly controlling this architectural variable, we ensure an absolutely fair comparison. Under this fair setup, FinHarnessSkill achieves a total score of 52.08%, comprehensively outperforming all other experience-learning methods. In addition, our method maintains a substantial 19.88% margin over the ACE baseline implemented within the same framework. This rigorous ablation definitively isolates the variables and proves that the performance improvements stem directly from the accumulated financial reasoning experience. The gains do not emerge from structural differences across disparate frameworks.

Table 2: Soft ablation of the Dynamic Context Orchestrator (DCO) using context thresholds.

ModelFrameworkL1L2L3AVG.Total Score
Qwen3-4B-Instruct-2507w/o DCO (Soft Ablation)32.69%27.75%2.33%20.92%23.48%

To address the reviewer's concern that completely removing the Dynamic Context Orchestrator (DCO) might disrupt the execution pipeline, we conducted a "soft ablation" experiment. Instead of leaving the agent with an unbounded context that causes systemic failures, we replaced the DCO with a standard heuristic context-bounding strategy. Specifically, we limited inputs to 10,000 characters per document and applied LlamaIndex for semantic matching with a 0.5 similarity threshold. This setup ensures the agent's pipeline remains fully functional while strictly isolating the necessity of DCO's explicit verification, as reflected in the severe performance drop to 2.33% on L3 tasks.

Table 3: Ablation experiments on DeepSeek-V4 Flash.

FrameworkL1L2L3AVG.Total Score
Baseline12.50%7.85%3.10%7.82%8.52%
FinHarnessSkill82.21%72.77%83.72%79.57%79.17%
w/o AER58.65%46.60%9.30%38.18%42.23%
w/o DCO55.77%42.93%31.01%43.24%45.08%
w/o SBEM73.08%61.26%72.09%68.81%68.56%

To evaluate architectural dependencies across model scales, we further analyze the ablation results for DeepSeek V4 Flash. Removing the AER module leads to a severe performance drop on Level 3 tasks across both models, underscoring the necessity of dynamic subagent activation and iterative repair. Because complex financial tasks demand multi-step evidence acquisition where initial tool executions frequently produce incomplete or noisy data, removing AER causes the reasoning chain to break permanently at the first point of failure. Conversely, removing the DCO module reveals distinct scale-dependent resilience. While removing DCO forces the smaller Qwen3-4B-Instruct-2507 into a systemic L3 crash with a drastic proliferation of hallucinated responses, DeepSeek-V4-Flash sustains moderate performance and resists this hallucination spike because its superior context capacities and robust attention mechanisms prevent severe attention sparsity under unfiltered noisy context. This contrast demonstrates that DCO is indispensable for filtering evidence and enabling smaller models to execute reliable financial reasoning.

AuthorsOpenReview ↗

W2 & Q1: Clarification and Supplementary Experiments on Data Leakage and Generalization.

We sincerely appreciate the reviewer's careful observations and constructive suggestions. We have incorporated your feedback by conducting extensive supplementary experiments and providing detailed clarifications to address your concerns regarding potential data contamination and the backtesting window, and we hope these additions will successfully resolve your doubts and earn your approval.

Table 4: Performance comparison of various models and frameworks in trading simulation (2026-01-02 to 2026-06-30, 123 trading days).

Model/FrameworkGOOGLMETAMSFTNVDATSLAAverage
ARR (↑)ARR (↑)ARR (↑)ARR (↑)ARR (↑)ARR (↑)SR (↑)MDD (↓)
Market
B&H27.86%-28.28%-41.08%11.31%-16.20%-9.28%-0.03823.60%
TradingAgents
Qwen3-4B-Instruct-250725.09%-14.89%-18.57%-2.70%-41.08%-10.43%-0.25514.55%
GPT-4o-mini27.24%-24.04%-30.71%7.06%-16.19%-7.33%-0.04923.93%
GPT-4o39.28%-24.07%-6.54%-20.45%-15.67%-5.49%-0.12413.53%
AlphaQuanter w/o RL
Qwen3-4B-Instruct-250737.22%-42.15%-12.99%-9.71%-28.52%-11.23%-0.02420.98%
Ours
FinHarness20.64%-43.00%-15.91%-22.02%-15.08%-15.07%-0.42820.10%
FinHarnessSkill30.38%-14.27%7.88%9.21%-6.89%5.26%0.23912.74%

Table 5: Detailed Sharpe Ratio across various models and frameworks.

Model / FrameworkGOOGLMETAMSFTNVDATSLA
Market
B&H0.689-0.371-0.8690.3560.006
TradingAgents
Qwen3-4B-Instruct-25071.019-0.613-0.490-0.027-1.165
GPT-4o-mini0.646-0.347-0.6490.259-0.153
GPT-4o1.570-0.759-0.271-0.547-0.613
AlphaQuanter w/o RL
Qwen3-4B-Instruct-25070.085-0.102-0.025-0.013-0.065
Ours
FinHarness0.550-1.086-0.347-0.534-0.723
FinHarnessSkill1.002-0.1970.3880.316-0.316

Table 6: Detailed Maximum Drawdown across various models and frameworks.

Model / FrameworkGOOGLMETAMSFTNVDATSLA
Market
B&H20.36%28.72%26.69%18.23%24.00%
TradingAgents
Qwen3-4B-Instruct-25074.62%11.06%18.65%11.03%27.41%
GPT-4o-mini20.36%29.81%25.64%18.23%25.63%
GPT-4o4.41%20.25%8.55%18.57%15.86%
AlphaQuanter w/o RL
Qwen3-4B-Instruct-250719.43%29.70%15.23%19.51%21.01%
Ours
FinHarness18.32%31.03%18.34%17.01%15.80%
FinHarnessSkill7.40%20.42%7.88%19.19%8.80%

In the original manuscript, Table 13 and Table 14 present experimental results using Qwen2.5-7B. According to the official website [1], this model was publicly deployed in 2024. This timeline strictly precedes our 2025 backtesting period and effectively eliminates any risk of training data leakage. To address the concerns regarding potential data leakage introduced by Qwen3-4B-Instruct-2507, we designed an additional out-of-sample evaluation. We evaluated the same five stocks over an additional period from January 2 to June 30, 2026, covering 123 trading days. In the additional backtest, FinHarnessSkill achieved the highest average annualized rate of return of 5.26% and the highest average Sharpe ratio of 0.239 across the five stocks. It also reduced average maximum drawdown from 20.10% for FinHarness to 12.74%, the lowest among the evaluated methods. These results suggest that financial skills distilled from verified financial reasoning trajectories can transfer to trading and improve performance in an additional market period. We have included the main results of the additional backtest in Table 15 of the appendix to the revised manuscript.

[1] https://qwenlm.github.io/blog/qwen2.5/

AuthorsOpenReview ↗

W2. Addressing Variability Across Stocks in the Trading Backtest.

We sincerely appreciate the reviewer's suggestions for a variance analysis of our trading results. To quantify how performance varies across stocks, we report a paired comparison between FinHarness and FinHarnessSkill on the same five stocks with the same backbone in the 2026 backtest.

Table 7: Stockwise ARR and Variance Comparison in the 2026 Trading Backtest.

StockFinHarness ARRFinHarnessSkill ARRARR difference
GOOGL20.64%30.38%+9.74%
META-43.00%-14.27%+28.73%
MSFT-15.91%7.88%+23.79%
NVDA-22.02%9.21%+31.23%
TSLA-15.08%-6.89%+8.19%
Mean across stocks-15.07%5.26%+20.34%
Standard deviation across stocks22.94%17.19%10.73%

For each stock, the ARR difference is computed by subtracting the FinHarness ARR from the FinHarnessSkill ARR. The mean difference is calculated across the five stock-wise differences before rounding. FinHarnessSkill improves ARR on all five stocks, with gains ranging from 8.19% to 31.23% and an average gain of 20.34%. Its ARR is positive on three stocks and ranges from −14.27% to 30.38%. These results confirm that verified financial reasoning trajectories enhance returns and mitigate cross-stock volatility.

Reviewer_NKKQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FinHarness, a framework for improving the reliability of LLM-based financial agents by addressing evidence misalignment, context noise accumulation, and unreliable experience reuse. The framework integrates an Adaptive Evidence Retriever that constructs task-specific execution graphs and repairs evidence gaps, a Dynamic Context Orchestrator that admits evidence only after checking provenance, temporal alignment, financial-field semantics and task relevance, and a Skill-Based Experience Manager that compiles externally verified execution trajectories into reusable procedural representations. Its skill-augmented variant, FinHarnessSkill, further updates memory between benchmark queries while preventing the current query’s verification feedback from influencing its own response. The evaluation covers professional financial reasoning on ProFinR, expert-audited hallucinations, experience-reuse comparisons, and transfer to a six-month stock-trading backtest. The reported results show substantial reasoning improvements across four backbones, lower hallucination rates among erroneous responses, and additional gains from skill reuse.

Critical strengths
  1. The paper identifies an interesting and financially specific reliability problem. The distinction between successful tool execution and financially valid evidence seems compelling.

  2. The execution architecture of FinHarness has a coherent organizing principle. Planning, evidence admission, repair, and answer synthesis are organized around explicit evidence obligations.

  3. The end-to-end reasoning improvements are substantial and span multiple backbones.

Critical weaknesses
  1. Table 2 reports hallucination rates among erroneous responses, while explicit abstentions are excluded from the audit. FinHarness also instructs the answerer to state that the conclusion is unconfirmed when evidence is insufficient. This protocol may systematically favor systems that abstain. Without absolute hallucination rates and abstention rates, it is unclear how much improvement comes from fewer false claims versus changes in response behavior. For example, the Qwen3-4B naive baseline has a 98.30% error rate but only a 19.08% conditional hallucination rate. Moreover, the six hallucination cases in Table 8 closely overlap with the proposed verification dimensions, while the number of expert annotators, inter-annotator agreement, and blinding procedures are not reported.

  2. In FinHarness, the Skill-Based Experience Manager uses benchmark reference answers to verify trajectories before reusing them on later queries from the same benchmark. Although the causal protocol prevents direct answer leakage, it remains unclear whether other Experience-Learning Frameworks such as ACE, DC, and ICL receive equivalent correctness feedback and experience budgets. Thus, the effects of skill representation and additional supervision are not clearly separated. In addition, Table 5 shows that DC (46.59%) and ICL (42.05%) both outperform FinHarness without skills (41.86%). It suggests that generic experience reuse does not necessarily hurt performance.

  3. The paper reports results without variance, confidence intervals, or query-order sensitivity analysis. Order sensitivity is relevant since skills accumulate during a single, deterministically ordered benchmark pass. Token consumption, tool-call counts, and latency are also unreported, leaving the cost of multi-round repair unclear. Although FinHarness's end-to-end results are based on four backbones, the main multi-agent and experience-learning comparisons and component ablations concentrate on Qwen3-4B. Thus, the evidence for cross-backbone improvement is stronger than the evidence that the proposed orchestration and memory mechanisms consistently outperform alternatives at comparable cost.

  4. The ablations do not clearly isolate the mechanisms proposed to explain the gains. In particular, Table 7 reports a drop to 7.20% when the DCO is removed. However, this component supplies not only evidence filtering but also the admitted evidence interface and feedback for subsequent acquisition. Verified evidence also affects trajectory eligibility for skill construction. Removing it therefore changes several interacting mechanisms, rather than isolating context noise alone. The paper should explain more whether the ablated system retains a functional, comparable synthesis interface and how its memory is constructed.

Questions for authors
  1. For each configuration in Table 2, what are the numbers of benchmark errors, full and partial abstentions, responses audited, and confirmed hallucinations?

  2. How many experts independently annotated each candidate, how were disagreements resolved, and what inter-annotator agreement was observed?

  3. How many independent runs underlie for main experiments and what is the uncertainty level?

  4. For matched-backbone comparisons, were tool access and total token, tool-call, and execution limits equivalent across methods?

13 comments
AuthorsOpenReview ↗

Q1 & W1: Addressing Concerns Regarding Absolute Hallucination Rates and Abstention Rates.

We sincerely thank the reviewer for this insightful comment regarding our hallucination evaluation protocol.

Table 1: Rate of explicit abstentions (refusals to answer) across different backbone models and agent configurations.

ModelNo-Tool BaselineReActProFinAgentFinHarness
Qwen3-4B-Instruct-250775.38%18.18%5.87%8.52%
Qwen3-32B67.23%17.61%6.82%11.36%
Claude-Sonnet 4.628.41%4.36%1.89%1.33%
DeepSeek-V4 Flash80.30%26.33%10.23%5.11%

Table 2: Rate of incompletions due to tool execution failures across different configurations.

ModelReActProFinAgentFinHarness
Qwen3-4B-Instruct-25078.33%14.96%7.77%
Qwen3-32B8.90%12.12%4.92%
Claude-Sonnet 4.65.49%2.27%2.65%
DeepSeek-V4 Flash19.89%12.12%3.41%

Table 3: Error Counts, Hallucination Counts, and Global Hallucination Rates

ModelFrameworkErrorsHallucinationsGlobal Hallucination Rate
Qwen3-4B-Instruct-2507Baseline5199918.75%
ProFinAgent37218835.61%
FinHarness307489.09%
Qwen3-32BBaseline51715629.55%
ProFinAgent34021540.72%
FinHarness235264.92%
Claude-Sonnet 4.6Baseline30915429.17%
ProFinAgent887514.20%
FinHarness5061.14%
DeepSeek-V4 FlashBaseline4835710.80%
ProFinAgent22211822.35%
FinHarness166101.89%

The empirical results across multiple evaluation dimensions consistently demonstrate that FinHarness achieves the lowest hallucination rates among all tested frameworks. We clearly show the performance gaps across individual metrics for each framework in Figure 3 of the manuscript.

Lowest Global Hallucination Rates: The results show that FinHarness achieves a lower hallucination rate than both the no-tool baseline and ProFinAgent across all four backbones. Calculated over all 528 queries, the global hallucination rate relative to ProFinAgent decreases from 35.61% to 9.09% for Qwen3-4B-Instruct-2507 and from 22.35% to 1.89% for DeepSeek-V4 Flash.

Explicit Abstentions & Tool Execution Failures: The results show that FinHarness achieves lower explicit abstention rates than ReAct across all four backbones. Claude Sonnet 4.6 and DeepSeek V4 Flash show decreased abstention compared with ProFinAgent, with DeepSeek V4 Flash experiencing concurrent declines in hallucination and abstention, where abstention drops from 10.23% to 5.11%. Furthermore, results indicate that FinHarness leaves fewer queries incomplete due to tool execution failures than ReAct on all four backbones and than ProFinAgent on three.

AuthorsOpenReview ↗

Q2 & W1: Addressing Expert Annotation Procedures and Inter-Annotator Agreement.

We sincerely thank the reviewer for requesting further details regarding our independent expert annotation procedures, disagreement resolution protocols, and the observed inter-annotator agreement metrics. To identify financial hallucinations, four financial domain experts independently evaluated all erroneous responses under strictly blinded conditions against a unified reference standard. For example, evaluating the 483 erroneous responses from the DeepSeek V4 Flash naive baseline identified 417 unanimous non-hallucination classifications and 48 unanimous hallucination labels alongside 18 cases of disagreement, achieving a Fleiss Kappa of 0.881 and an average pairwise agreement of 97.52%. During disagreement resolution, the committee systematically reviewed the query requirements, gold answers, system responses, and source tool records, escalating any remaining deadlocks to an independent adjudicator. All agreement scores were calculated exclusively on the initial blinded labels, entirely avoiding inflating the scores through discussion consensus.

AuthorsOpenReview ↗

W2: Addressing Experience Accumulation Fairness and Generic Experience Reuse.

We sincerely thank the reviewer for raising this insightful point regarding experience learning fairness and the impact of generic experience reuse on overall performance.

Strict Fairness in Experience Accumulation and Feedback: We explicitly confirm that all compared experience learning frameworks, including ACE, DC, and ICL, accumulate experience in the same way as FinHarness. Every baseline operates under the identical online causal construction protocol, receives equivalent correctness feedback verified against benchmark reference answers, and shares the same experience memory budgets. This strict fairness guarantees that the gains of FinHarnessSkill stem entirely from our structured procedural skill representation rather than additional supervision.

Why ACE Degrades Performance: The ACE framework exhibits significant declines across L1, L2, and overall accuracy compared to FinHarness. This occurs because ACE retrieves broad textual guidelines based primarily on semantic overlap, which introduces unstructured context noise that actively disrupts the rigid tool schemas and strict reasoning paths required for financial reasoning tasks.

Why ICL Struggles on Complex L3 Tasks: While the ICL framework improves overall performance, it underperforms FinHarness on L3 tasks. We found ICL prepends static generic demonstrations that consume critical context space. These fixed examples fail to provide the dynamic error recovery mechanisms and precise numerical constraints necessary for navigating highly complex L3 scenarios, ultimately misleading the agent during intricate multi-step reasoning.

Why FinHarnessSkill Is Effective: FinHarnessSkill achieves superior performance by explicitly decoupling financial skills from task-specific facts. During skill compilation, the framework removes concrete query content, historical answers, specific financial entities, dates, and observed values, while schema validation and leakage audits ensure skills retain only reusable procedural parameters, tool preferences, and evidence requirements. Consequently, these financial skills guide complex reasoning without context noise or interference from past answers, enabling robust generalization across diverse tasks.

Reviewer_RPjsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FinHarness, a framework for reducing financial hallucinations in tool-using agents without updating model parameters. It combines three components: an Adaptive Evidence Retriever for task-dependent planning and repair, a Dynamic Context Orchestrator that checks evidence before admitting it into the reasoning context, and a Skill-Based Experience Manager that extracts reusable procedures from validated trajectories. The authors evaluate financial reasoning on ProFinR and transfer the resulting skills to stock trading. They report improved task accuracy and reduced hallucination rates across several backbone models, together with better backtesting results.

Critical strengths
  1. The paper addresses a practical and worthwhile problem. Successful tool calls do not necessarily provide financially valid evidence: the entity, reporting period, price basis, or metric definition can still be wrong. Sections 3 and 4.2.2 make this distinction concrete. Connecting evidence checks to targeted repair is a sensible direction for improving financial agents.
  2. The reported gains are encouraging and cover multiple models. Table 1 shows improvements over ProFinAgent across four backbone models. Tables 5 and 7 also include experience-learning comparisons and component ablations. These results provide useful initial evidence that the complete system works, even though they do not fully establish which design choices account for the gains.
Critical weaknesses

The methodological novelty appears limited, and the comparisons do not clearly establish the added value of the proposed design. ProFinAgent already combines tool retrieval, DAG-based planning, and case memory. FinHarness adds task contracts, evidence admission, repair, and verified procedural skills, but the contribution still looks largely like a combination of familiar mechanisms with financial validation rules. For example, Eq. (3) specifies which checks evidence should pass, while the difficult part is implementing those checks reliably. I appreciate that the paper includes tool-using ReAct baselines and existing agent frameworks. My concern is whether it improves over a strong general-purpose agent given the same tools, explicit financial checks, feedback access, and comparable execution budget. The comparisons do not clearly establish parity in prompts, retries, tool calls, or feedback used to construct memory. Component removal alone does not resolve this issue. A simpler agent with matched resources and validation information would be a more convincing reference point. A method does not need to be complicated to be valuable, but the current evidence leaves its distinct contribution unclear.

  1. The reliability and backtesting evaluations need stronger controls and clearer reporting. In Section 4.2.2 and Table 2, hallucination rates are calculated among erroneous responses, with abstentions and tool failures excluded from the hallucination count. This is useful, but it differs from the hallucination rate across all queries. The simultaneous accuracy improvements are encouraging; still, overall hallucination frequency, abstention/incompletion rates, and clearer expert-auditing details are needed to assess the reliability claim. The evidence-admission checks also need more concrete implementation details and an analysis of incorrectly accepted or rejected evidence. The trading evidence is limited to five stocks over one six-month period. Following AlphaQuanter’s protocol does not by itself establish that every input was available at the historical decision time. Publication timestamps, revised data, decision/execution timing, and transaction costs should be explicitly documented. Freezing the skill registry prevents updates during evaluation, but does not by itself establish historical point-in-time validity. These gaps do not demonstrate leakage, but they make the stronger claims about reliable and generalizable trading performance difficult to assess.
Questions for authors
  1. Contribution: What is the main methodological advance over ProFinAgent and a tool-using agent with explicit financial validation rules? Which existing comparison best isolates this advance?
  2. Comparison fairness: Which comparisons use matched backbone models, tools, validation information, execution budgets, and feedback for memory construction? Can you provide tool-call/token usage and runtime summaries from the existing runs, and clarify any differences?
  3. Reliability measurement: How are the source, temporal, financial-field, and slot-support checks implemented? Which are programmatic and which rely on an LLM? For Table 2, could you provide the underlying counts, abstention/incompletion rates, and the expert annotation and disagreement-resolution procedure?
9 comments
AuthorsOpenReview ↗

Q1 & W1: Methodological Advances of FinHarness over Existing Tool-Using Financial Agents and General Baseline Comparisons.

We sincerely thank the reviewer for raising these questions regarding the methodological advances of FinHarness over existing financial tool-using agents, as well as the importance of baseline comparisons with matched resources and financial checks. While ProFinAgent focuses on optimizing tool-planning pathways, FinHarness establishes verified execution control and dynamic adaptability through three core advances.

Dynamic SubAgent & Tool Routing: The fixed-role baseline uses predefined agent roles, and ProFinAgent schedules tool calls through a DAG. FinHarness selects subagents and tools based on the evidence required by the task contract. For an exact closing price query, the Dynamic Context Orchestrator rejects records from other dates, and the Adaptive Evidence Retriever searches for the missing price. This limits unnecessary tool calls and reduces the risk of using a mismatched price in the final answer.

Closed-Loop Evidence Admission: ProFinAgent lacks strict verification bounds after tool planning, allowing noisy or misaligned tool outputs to enter synthesis and cause financial hallucinations. FinHarness enforces explicit evidence gating and context constraints to filter irrelevant noise and admit only verified factual evidence into the reasoning stream.

Verified General Financial Skills: Instead of reusing raw trajectories, FinHarness extracts validated, anonymized procedural skills from verified trajectories, preventing information leakage and generalizing effectively to out of sample trading without overfitting.

Table 1: Performance comparison of Qwen3-4B-Instruct-2507 across Baseline, ReAct, ProFinAgent, their Financial Validation-Enhanced Variants, and FinHarness on ProFinR.

FrameworkL1L2L3AVG.Total Score
Baseline4.32%0.00%0.00%1.44%1.70%
ReAct31.73%13.08%3.87%16.23%18.18%
ReAct + Financial Validation39.42%17.80%1.55%19.59%22.35%
ProFinAgent40.38%28.79%13.17%27.45%29.55%
ProFinAgent + Financial Validation53.85%31.41%10.08%31.78%35.04%
FinHarness59.62%32.46%27.13%39.74%41.86%

To guarantee strict execution parity, the system enforces a maximum context capacity of 131,072 tokens. We specifically cap individual prompt inputs at 86,150 tokens and generation outputs at 6,144 tokens. All inference procedures use a temperature of zero alongside a strict 600s timeout threshold. We also integrated the FinHarness financial validation module into ReAct and ProFinAgent and applied the same limit on repair iterations to both baselines.

Experimental setup. Under the same retry budget, we applied the Dynamic Context Orchestrator’s financial evidence admission checks to both baselines and enabled ReAct to make additional tool calls after detecting an error and ProFinAgent to construct and execute a repair DAG.

Accuracy improvements and limitations. While adding the financial validation module improves the overall accuracy of both baselines, L3 accuracy decreases from 3.87% to 1.55% for ReAct and from 13.17% to 10.08% for ProFinAgent. ReAct uses the DCO’s evidence admission checks to filter tool outputs. However, L3 tasks also require planning dependent subtasks, selecting tools for each evidence requirement, and diagnosing why earlier calls failed. The results suggest that filtering and additional retries alone are insufficient for these tasks. ProFinAgent can replan and execute a repair DAG after a failure, but its downstream context selection still relies on context threshold limits and semantic matching. Therefore, additional retrieval can add redundant information while leaving the evidence needed for the answer unresolved.

Additional execution cost. ProFinAgent triggered repair on 64 queries, adding an average of 12,084.67 tokens, 2.16 tool calls, and 13.81 seconds per repaired query. ReAct triggered repair on 78 queries, adding an average of 9,869.79 tokens, 2.72 tool calls, and 16.14 seconds per repaired query. Thus, both repair extensions incur additional computation.

AuthorsOpenReview ↗

Q2: Addressing Comparison Fairness and Execution Budgets.

We sincerely thank the reviewer for raising this insightful question regarding resource parity and the necessity for matched baseline comparisons. Except for the naive baseline presented in Table 1 of the original manuscript, all evaluated frameworks strictly operate using the exact same set of tools to ensure there is no discrepancy in tool access. The system strictly caps the overall execution budget at a maximum of six total tool invocations per query. To ensure strict resource parity and prevent unbounded execution loops, the repair program governed by the manager agent is permitted to activate exactly once per task.

Table 2: Comparison of execution overhead and efficiency metrics (average tokens, model calls, tool calls, and latency per query) across FinHarness, ReAct, and ProFinAgent using the Qwen3-4B-Instruct-2507 backbone.

FrameworkTokensModel CallsTool CallsLatency
FinHarness27,4805.803.8930.70 s
ReAct36,0244.503.4042.66 s
ProFinAgent29,8404.002.8047.85 s

With Qwen3-4B-Instruct-2507 as the shared backbone, FinHarness uses 27,480 tokens and takes 30.70 s per query, reducing token usage by 23.7% relative to ReAct and 7.9% relative to ProFinAgent, and latency by 28.0% and 35.8%, respectively. This superior performance is primarily because FinHarness dynamically plans and activates task-relevant subagents for parallel execution based on specific task requirements. However, ReAct struggles with continuous reflection loops, and ProFinAgent is constrained by a fixed four-stage invocation pipeline. Furthermore, rigorous evidence control and filtering mechanisms in FinHarness drastically compress the input context length. This strict context minimization successfully overcomes the processing bottlenecks prevalent in ProFinAgent. Specifically, ProFinAgent frequently exceeds single-document context thresholds and resorts to LlamaIndex chunking to extract partial snippets, a workaround that inherently discards critical information and incurs substantial semantic embedding delays. By limiting downstream context strictly to admitted evidence and directing targeted repair toward unresolved requirements, FinHarness accelerates the overall evidence-acquisition process. This architectural advantage explains why these profound efficiency gains occur despite FinHarness making an average of 5.80 model calls and 3.89 tool calls per query.

AuthorsOpenReview ↗

W2: Addressing Backtesting Evaluation Controls. (Part 1/2)

We sincerely thank the reviewer for highlighting the necessity of a verifiable historical backtesting protocol and apologize for lacking these critical implementation details in our original manuscript. To ensure strict historical point-in-time validity, our framework enforces rigorous temporal boundaries on all data inputs. In addition, we use the same trade execution and transaction cost settings as AlphaQuanter.

Decision timing: The agent generates a buy, sell, or hold decision for trading day tt.

Execution timing: Buy and sell decisions made on day tt are executed at the close of the following trading day, t+1t+1.

Execution price: Both buy and sell orders are filled at the closing price of the asset on day t+1t+1.

Transaction fees: Each executed buy or sell incurs a fee of 0.1% of its transaction value.

Slippage buffer: We use the same capital utilization ratio of 0.9 for buy orders. Order size is calculated using 90% of the available cash, with the remaining cash providing a buffer. The simulator uses the closing price directly as the fill price.

Market Price and Volume Data: Historical prices and trading volumes are strictly limited to the inclusive date range from fourteen calendar days before the specified cutoff date through that date, using the same lookback parameter of 14 as AlphaQuanter. Therefore, technical indicators evaluated by the manager agent derive from this restricted period, meaning metrics requiring longer windows remain intentionally inaccessible.

Macro and Fundamental Filings: For macro and fundamental analysis, financial reports and non-news corporate announcements are restricted to a one-year lookback period before the decision date.

Real-Time News and Web Search: News and web search results accessed by the fundamental event agent are constrained to the preceding two days.

Rolling Agent Memory Context: Furthermore, the memory module maintains a rolling context of up to seven historical summaries per agent role instead of a fixed seven-day chronological window.

Strict Prevention of Lookahead Bias: To completely prevent any potential lookahead bias, every single temporal window strictly terminates at the close of the preceding trading day, guaranteeing that the agent formulates decisions based exclusively on information genuinely public before market open.

Table 3: Performance comparison of various models and frameworks in trading simulation (2026-01-02 to 2026-06-30, 123 trading days).

Model/FrameworkGOOGLMETAMSFTNVDATSLAAverage
ARR (↑)ARR (↑)ARR (↑)ARR (↑)ARR (↑)ARR (↑)SR (↑)MDD (↓)
Market
B&H27.86%-28.28%-41.08%11.31%-16.20%-9.28%-0.03823.60%
TradingAgents
Qwen3-4B-Instruct-250725.09%-14.89%-18.57%-2.70%-41.08%-10.43%-0.25514.55%
GPT-4o-mini27.24%-24.04%-30.71%7.06%-16.19%-7.33%-0.04923.93%
GPT-4o39.28%-24.07%-6.54%-20.45%-15.67%-5.49%-0.12413.53%
AlphaQuanter w/o RL
Qwen3-4B-Instruct-250737.22%-42.15%-12.99%-9.71%-28.52%-11.23%-0.02420.98%
Ours
FinHarness20.64%-43.00%-15.91%-22.02%-15.08%-15.07%-0.42820.10%
FinHarnessSkill30.38%-14.27%7.88%9.21%-6.89%5.26%0.23912.74%

To address the insightful query regarding the backtesting time span and validate historical point-in-time validity, we designed an additional out-of-sample evaluation. We followed the evaluation protocol described in the original manuscript and backtested the same five stocks over an additional period from January 2 to June 30, 2026, covering 123 trading days. The empirical outcomes conclusively demonstrate the robust generalization capabilities of FinHarnessSkill under completely unseen market dynamics. FinHarnessSkill achieved the highest average annualized rate of return of 5.26% and the highest average Sharpe ratio of 0.239 across the five stocks. It also reduced average maximum drawdown from 20.10% for FinHarness to 12.74%, the lowest among the evaluated methods. These results suggest that financial skills distilled from verified financial reasoning trajectories can transfer to trading and improve performance in an additional market period. We have included the main results of the additional backtest in Table 15 of the appendix to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.