acceptodds
Under review as a conference paper at ICLR 2027

FinAgent-Reliance: Benchmarking Human Reliance and Error Inheritance in Financial LLM Decision Support

Abstract

Existing financial LLM benchmarks typically evaluate whether a model produces the correct answer, but stop before asking whether model failures propagate into human decisions. We introduce FinAgent-Reliance, a source-grounded benchmark and experimental framework for evaluating reliability at the human-AI system level. The framework links authoritative financial evidence to human-validated answers, natural and controlled agent failures, blinded correctness assessment, and separate coding of AI-specific error inheritance. Its frozen Stress-Test v2 contains 47 human-validated records: 12 correct parent responses and 35 isolated controlled failures spanning ten failure categories, with leakage-resistant splits across parent families, companies, and source documents. Stress-Test v2 is difficult for both evaluated open-weight 7B instruction models. Qwen2.5-7B achieves balanced accuracy 0.576 (95% parent-family cluster-bootstrap CI: 0.486-0.681), while Mistral-7B reaches 0.515 on parseable outputs and 0.475 when unparseable outputs are counted as failures. At the failure-typing level, neither model correctly identifies the target category for any of the 20 controlled failures spanning seven semantically subtle categories. Exploratory human studies show that the reliance measurement and evidence-inspection protocols can be implemented in practice; these studies are not used to estimate intervention effects. FinAgent-Reliance evaluates an additional question beyond model correctness: whether a known model error is detected, resisted, corrected, or inherited after it is presented to a human decision-maker.

Primary area:
datasets and benchmarks
Keywords:
human-AI reliability, financial LLM agents, datasets and benchmarks, error inheritance, source-grounded evaluation, controlled failures, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7ZhFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new source-grounded benchmark and experimental framework for evaluating the relability of finanical LLMs at the human-AI system level. The key aspect of the decision making process that this framework seeks to address is how a LLM response influences the recipient of the response; it is possible for an incorrect LLM response to prompt the user to switch to an incorrect response with an error differing from that inherent to the LLM response. The key contributions of this work include an evaluation of how model error is propagatd or mitigated at the decision-making level, meausrement of AI reliance and a source-grounded finanical failure benchmark.

Two 7B models, Qwen2.5-7B and Mistral-7B, were evaluated on the proposed Stress-Test v2 benchmark. Both models were shown to be unable identify the target category for any of the 20 controlled failure types included.

Critical strengths

The question being addressed is one that should be of interest to many AI practitioners, particularly as AI is increasingly used and may influence decision-making. In highly regulated industries, such as finance, this aspect carries even greater importance as even subtle shifts or failure modes introduced due to the addition of AI to the decision-making workflow could carry significant consequences, both on the basis of outcomes and regulations.

To that end, the construction of FinAgent-Reliance appears to be sound and comprehensive in how it seeks to evaluate and benchmark the LLM-human system. The stratification of takss by task types, failure variants and assistance conditions provides a fairly holsitic evaluation of the AI-human system.

Critical weaknesses

I believe there are two glaring weaknesses in this manuscript, both somewhat linked to the readability.

  1. The main maunscript references too many numbers and terms with little to no grounding for a reader. A good example of this is the first dicsussion of the conditions on Line 168; these are central to the construction of the proposed benchmark but the only visible reference to them. before this point is in Figure 1. These should be clearly stated and justified due to their importance to the subsequent description and discussion. Along a similar line, the reader is not given any examplars from Gold Core v1 or the Financial-Agent Failure Library. This lends a sense of arbitrariness to these objects despite the justification presented for their uses in the paper. More examples of this are provided in the next section (in the form of questions).

  2. The manuscript is structured confusingly in parts. The abstract contains several metrics which read like a recap of a technical report rather than substantiating the key takeaways. Aspects neeeded to understand Sections 2 and 3 are explained in Section 5. While some of writing is a personal stylistic choice, an effort has to be made to enable a reader to develop and follow the reasoning presented by the authors as they read the paper as it is presented.

Questions for authors
  1. The term "parent-family" or "parent" is used multiple times in the manuscript. What does this mean?
  2. The discussion in Section 7 is very general. In its current form, it is difficult to grasp what the numbers mean and how they link to the proposed objectivs of this work. 3, What would be needed for the authors to make more general remarks about LLM-human systems?
  3. Results and metrics are mostly scattered trhough the text; this makes it difficult to reference these results and subsequently, follow the discussion.
  4. What is the "Experimental Stimuli v1"?
1 comment

We thank the reviewer for the detailed suggestions on the presentation of the framework, benchmark composition, results and examples. They identified places where the original manuscript was insufficiently clear.

  1. Experimental conditions C1–C4 We agree that the conditions must be defined before they are used. Section 3 now introduces them: C1 allows human-only reconsideration; C2 shows a fixed AI response without citations; C3 adds optional inspection of cited evidence; C4 adds a mandatory checklist and a neutral challenge prompt and, in Pilot v2, requires evidence inspection before submission. Initial judgements are locked before the intervention, the AI response is identical across C2–C4, and C3 and C4 share the same frozen citations.

  2. Parent families and benchmark construction Section 4 now defines a parent task as a validated financial question with a correct reference response and authoritative evidence. A controlled variant introduces exactly one material error while preserving the question and evidence. Stress-Test v2 contains 12 parent families: 12 correct responses and 35 controlled failures, 47 records in total. These are not 47 independent tasks, so uncertainty is estimated with parent-family cluster bootstrapping. The separate roles of Gold Core v1, the Human-Validated Expansion Base, Experimental Stimuli v1 and Stress-Test v2 are now explicit in Section 4 and Appendix B.

  3. Experimental Stimuli v1 The two resources serve different purposes. Experimental Stimuli v1 (24 tasks) supports the human pilots, measuring changes between initial and final judgements under C1–C4. Stress-Test v2 is a failure-enriched diagnostic benchmark for automated error detection and classification. Using different resources prevents a direct item-level comparison between automated error recognition and human error inheritance. Sections 5.1 and 7 now discuss this limitation explicitly.

  4. Presentation and interpretation of results The Results and Discussion now separate binary error detection from exact failure-category classification. Qwen2.5-7B achieves balanced accuracy of 0.576, and Mistral-7B 0.475 when unparseable outputs are counted as errors. Both have zero exact-category recall on 20 controlled failures in seven categories. This does not mean all 20 errors went undetected; some were flagged as errors but assigned the wrong category. [If 32B is reported: Qwen2.5-32B-Instruct reaches 0.775 and correctly classifies 9 of these 20; because it ran in BF16 and the 7B models were 4-bit quantised, we do not attribute this to scale.]

The human studies are described as exploratory protocol-validation pilots, not confirmatory tests of intervention effectiveness. In Pilot v2, evidence was viewed in 38/64 C3 trials and 64/64 C4 trials, where inspection was required, while final accuracy was 76.6% in both conditions. These observations are descriptive and do not show that either interface changed decision quality.

  1. Representative benchmark examples We agree that real records demonstrate source grounding better than schematic examples. Appendix B now replaces the synthetic illustrations with verified records from the frozen Gold Core and Failure Library. Each gives the question, the SEC filing accession numbers, the extracted figures, the validated Gold answer and, for the failure record, the controlled modification and its category. Appendix A.4 additionally presents two observed participant cases on real filings (Goldman Sachs/BNY Mellon and American Express/Discover).

  2. Generalisability Sections 4, 7 and 8 now discuss generalisability more fully. Stress-Test v2 has only 12 independent parent families, several categories have 2–4 examples, and the pilots are small with repeated observations per participant. We therefore present the findings as diagnostic evidence about the tested configurations and the feasibility of the measurement protocol, not as general conclusions about financial LLMs or human–AI decision-making.

We appreciate the reviewer’s suggestions, which have helped us present the framework and its limitations more clearly.

Reviewer_SDLAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework for evaluating whether errors will propagate into humans' decisions when AI makes mistakes. The work focuses on financial agents. Empirical evaluations are conducted on two 7B instruction LLMs. The results conclude that these two models are not good at detecting several subtle AI error types, and human pilots show that increasing evidence would not lead to higher accuracy.

Critical strengths
  • The topic is quite important and the exploration for inheritance of AI error is interesting.
  • This work provides well-designed evaluation conditions C1-C4.
Critical weaknesses
  • Why is there a stress test V2? There is no information about V1.
  • Even though a framework is presented in this paper, its theoretical contribution is weak.
  • As only one coder for reliance labels is involved in the pilot study, the paper lacks an independent agreement assessment, which makes the evaluation results unreliable.
Questions for authors

None

1 comment

We thank the reviewer for raising concerns about benchmark versioning, the conceptual contribution, and the reliability of the human reliance annotations.

  1. Relationship between Stress-Test v1 and Stress-Test v2 We agree that the original naming could suggest the paper compares benchmark releases. We have clarified this under Benchmark Versioning in Section 4. Stress-Test v2 is the frozen 47-record benchmark (12 validated parent responses and 35 controlled single-error variants) evaluated throughout the manuscript. The suffix identifies the artifact release, not an experimental condition. We report no comparison with an earlier stress-test release, and no conclusion depends on improvement between versions. Appendix B now summarises the composition and role of each frozen artifact (Gold Core v1, Failure Library v1, Experimental Stimuli v1, Expansion Base v1 and Stress-Test v2).

  2. Theoretical and methodological contribution We do not claim a new general theory of human reliance. FinAgent-Reliance builds on established human–AI decision-making and appropriate-reliance research. Our contribution is methodological: a source-grounded procedure that separates changes in answer correctness from reproduction of an identifiable error in AI-generated financial advice.

The framework compares three observable components: the participant’s initial response, a fixed AI response whose error status is established against authoritative evidence, and the participant’s final response. Because the AI error is known and source-grounded, we can distinguish (i) a final answer that is incorrect from (ii) one that reproduces the specific material error the AI presented. Aggregate before–after accuracy treats these two outcomes as the same. The procedure also separates resistance, AI-supported correction and failure recoverability, so a correct final answer is not automatically credited to the AI. Much prior reliance work measures agreement with AI advice. Our measurement instead targets the propagation of a particular, verifiable financial error, which matters where specific misstatements (a wrong metric, period or comparison) carry consequences.

We have strengthened this positioning in Section 2 (Related Work) and Section 3 (Framework). We state explicitly that observed error correspondence is consistent with inheritance but does not establish the participant’s internal reasoning or a causal mechanism. Appendix A.4 illustrates the distinction with two observed Pilot v1 cases: one where the final answer reproduces the AI’s substituted metric, and one where the participant explicitly rejects an AI comparison reversal.

  1. Independent validation of reliance annotations We agree that reliability is important, because distinguishing AI-specific inheritance from ordinary incorrectness requires judgement. Pilot v1 and Pilot v2 reliance labels were assigned by a single blinded reviewer using a frozen rubric. The reviewer had no access to condition, telemetry, confidence or participant identity. No independent second coder was used, so we cannot report percentage agreement or Cohen’s kappa for these labels. Section 5.3 and Appendix A.3 now state this explicitly. The 100% agreement (κ = 1.0) reported for benchmark failure validation comes from a separate review process and is not evidence of agreement on reliance labels. We therefore treat the human reliance results as exploratory single-coder measurements and do not claim reproducibility. An independently double-coded sample with agreement statistics is a necessary next step.

  2. Scope of the conclusions We have revised the Discussion, Limitations and Conclusion to avoid overstating the findings. Our contribution is a reproducible framework and a diagnostic benchmark, supported by exploratory human pilots. Model results are reported as specific to the evaluated configurations, given only 12 independent parent families. The pilots (24 and 16 participants, repeated trials) are described as feasibility studies. We do not claim general intervention effectiveness, causal reliance mechanisms, or a universal estimate of evaluator reliability.

Reviewer_W87vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FinAgent-Reliance which is a framework for studying whether errors in financial LLM responses end up in people's decisions. The main idea is to seperate a person giving an incorrect answer from a person adopting a specific mistake from the AI's response. These are not necessarily the same thing, and the framework aims to make this distinction measurable.

The authors construct a benchmark using financial information from authoritative sources, introduce controlled errors into otherwise correct answers, and have these errors reviewed by humans. They also propose an experimental procedure in which participants answer financial questions before and after receiving AI assistance. Correctness and reliance are assessed separately.

The evaluation includes a 47-record stress test, two 7B language models, and two small human pilot studies using a different set of questions. Together, these experiments examine the models' ability to identify financial errors and the extent to which specific AI errors appear in participants' final answers. While Model experiments test how well models identify and classify financial errors, human experimenrs test if participants adopt AI errors and how their access to evidence ( as the paper points it out different evidence-checking requirements) can impact their final responses.

Critical strengths
  1. The strongest aspect of the paper is its focus on AI-specific error inheritance. Rather than treating every incorrect human answer as evidence for over-reliance, the authors attempt to understand if a particular AI mistake appears in the user's final answer.

  2. I would consider the overall experimental setup desirable, especially comparing participants' answers before and after AI assistance as well as the four experimental conditions that let examining different types of assistance and evidence verification. The separate evaluation of correctness and reliance is also a good design choice.

  3. The benchmark uses authoritative financial evidence and the controlled errors are independently reviewed.

Critical weaknesses
  1. I think the framework is interesting, but I think it is not immediately fully clear to the reader what it adds to earlier studies on human reliance on AI. A clearer comparison with prior works would help.

  2. The benchmark includes a small number of independent tasks, and some error categories have very few examples. This makes it hard to understand if the findings generalize to other tasks and financial questions (Even though acknowledged in the limitation)

  3. Only two 7B models are tested. It would help to include at least one larger or stronger model, to see whether the same difficulties also appear in more capable systems.

  4. Also, the human pilots involve very few participants. Even though the authors acknowledge that these studies are exploratory, still the small samples limit what can be concluded about human reliance and error inheritance.

  5. The reliance outcomes were coded by only one reviewer, without an independent agreement check. Deciding whether a participant adopted a specific AI error can be subjective, so it is difficult to know how consistantly another reviewer would assign the same labels. For me, this is an important limitation.

  6. I like to see a few complete examples here, i.e. showing the introduced AI error, how it appears in a participant's final answer and how the labels are assigned would make the evaluation easier to understand.

  7. The model evaluation and human studies use different datasets, but the reason for this is not fully clear. A short explanation of this choice would help.

Questions for authors

Addressing these concerns would make the paper stronger, and I’d be open to raising my score substantially if the authors provide convincing answers in the rebuttal:

  1. Could the authors use a second independent reviewer to check the reliance labels and report inter-annotator agreement?

  2. Why do the model evaluation and human studies use different datasets rather than the same set of questions?

  3. Could the authors provide a few examples showing the original AI error, the participant's final answer, and how error inheritance was identified?

  4. Could the authors include a larger or stronger language model to check whether the findings also apply to more capable models?

  5. Did participants have access to financial documents or other resources when answering the questions before seeing the AI response?

2 comments

We thank the reviewer for the detailed, actionable suggestions. Below we explain what we have changed and state which limitations remain.

Q1. Independent annotation and inter-annotator reliability We agree that independent annotation matters most for AI-specific error inheritance, because it requires judging whether the participant’s final answer corresponds to the AI response. In Pilot v1 and Pilot v2, reliance labels were assigned by one blinded reviewer using the frozen rubric. No second coder was used, so inter-annotator agreement cannot be reported. Section 5.3 and Appendix A.3 now state this explicitly and separate these single-coder labels from the independently reviewed benchmark failures (κ = 1.0), which do not transfer to the reliance labels. This does not resolve the reproducibility concern. We therefore report all human reliance outcomes as exploratory and identify independent double-coding as a requirement for further validation.

Q2. Different datasets for model and human evaluations The two experiments measure different outcomes. The 24-item Experimental Stimuli v1 supports the before–after human protocol (correctness, AI-specific reliance and evidence interaction under C1–C4). The 47-record Stress-Test v2 (12 validated parents, 35 controlled failures) deliberately contains several variants per parent to probe ten failure mechanisms, and is analysed by parent family. Sections 4 and 5.1 now clarify this. We acknowledge the consequence: the current experiments cannot link an evaluator’s ability to recognise an error with a participant’s likelihood of inheriting it at the item level. Matched human and automated evaluation on the same controlled failures is an important next step.

Q3. Worked examples of inheritance and resistance Appendix A.4 now presents two anonymised observed cases from Pilot v1. Each gives the question, evidence, Gold answer, initial response, AI response, final response and the original coding rationale.

Case 1, inheritance (A4-V1-03): the AI substituted BNY Mellon’s net income applicable to shareholders for the Gold metric. The participant’s initial answer used about $2.05B and $2.19B. The final answer reproduced the AI’s distinctive $2.054B and $2.193B. Correctness scores did not change, so the case is not counted as a harmful-advice opportunity. Case 2, resistance (A4-V1-02): the AI reversed the 2023 American Express–Discover financing comparison (Gold: American Express higher by $2,620M). The participant explicitly rejected the reversal and kept the correct conclusion.

These cases show how the rubric was applied. They do not establish causal mechanisms or annotation reliability. The pilot records contained no verified case of a fully incorrect final answer with a distinct non-inherited error, so we do not present a constructed case as an observed outcome.

Q4. Stronger language model We evaluated Qwen2.5-32B-Instruct on the same frozen benchmark with the same prompt, schema and scoring rules. It achieves balanced accuracy of 0.775 (MCC 0.506), compared with 0.576 for Qwen2.5-7B. It correctly classifies 9 of the 20 failures in the seven categories where both 7B models had zero exact-category recall, and misclassifies the other 11. The 32B model ran in BF16 with vLLM and the 7B models with 4-bit quantisation, so the comparison does not isolate model scale. We report it as configuration-specific, with parent-family bootstrap CIs and per-category results in the appendix.

Q5. Document access before AI assistance Section 5.1 and Appendix A.2 now clarify this procedure. In the initial stage, the interface showed only the question, with no financial documents, AI answer or citations, and the response was locked before the intervention. C3 then allowed optional evidence inspection. In Pilot v2, C4 required evidence inspection before submission. External resource use was not monitored, and some initial responses contain precise figures consistent with outside lookups. We therefore describe initial responses as pre-AI-response judgements, not verified unaided decisions.

Sample size and generalisability Pilot v1 (24 participants) and Pilot v2 (16) involve repeated trials per participant and were designed to test feasibility, not to estimate intervention effects. Stress-Test v2 has 47 records but only 12 independent parent families. We have strengthened the limitations throughout and avoid causal or population-level claims. We thank the reviewer again. The feedback has sharpened what the framework establishes and what still requires validation.

I thank the authors for the detailed rebuttal and the improvements, particularly the additional model evaluation and clearer explanations. The 32B results mostly address my concern about stronger models (although 11 of the 20 previously challenging cases remain misclassified), but the different quantization settings and small benchmark limit the conclusions. However, some concerns (pivotal to the study) still remain. Given that AI-specific error inheritance is central here, I would have liked to see a convincing example in the main body rather than only in the appendix. Moreover,the example provided is not entirely convincing, as the participant's initial answer was already very close to the AI's response. More importantly, the lack of independent validation of reliance labels and the small human studies remain significant limitations. As a result, I appreciate the authors' efforts, but unfortunately I am not convinced enough to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.