acceptodds
Under review as a conference paper at ICLR 2027

Assess, Answer, Audit: A Point-in-Time Benchmark for Evidence-Grounded Financial Agents From Earnings Calls

Abstract

When assisting financial analysts, agents may engage in multi-turn interactions as analysts clarify and refine their requests. In this scenario, agents are required to interpret evolving requests, make decisions based on available evidence, and evaluate the reliability of answers. Yet existing benchmarks rarely assess these capabilities jointly in real multi-turn financial interactions. Inspired by earningscall dialogue, we introduce AAA-FIN (Assess–Answer–Audit for Finance), a benchmark built on real analyst–management exchanges, preceding Q&A, and the latest pre-call SEC 10-K. Three linked tasks, ASSESS, ANSWER, and AUDIT, evaluate question interpretation and evidence-based response decisions, complete and grounded answer generation, and judgments of historical answers’ evidence support and responsiveness. Together with complementary metrics, these tasks provide a multidimensional profile of financial agents’ strengths and weaknesses. We develop a reusable annotation pipeline combining LLM proposals, critic feedback, and deterministic validation, followed by review and adjudication by three domain experts. Evaluation of five API-accessible and open-weight models reveals uneven strengths across tasks, with no model consistently leading across the evaluated capabilities. Persistent gaps in evidence assessment, answer completeness, and citation coverage identify priorities for improving financial agents.

Primary area:
datasets and benchmarks
Keywords:
Agents, Benchmark, Financial Agent, AI for Fianance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pv2rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AAA-FIN, a benchmark built from real earnings-call exchanges, preceding dialogue, and pre-call 10-K filings. Its three tasks assess whether a model can determine what the evidence supports, generate an appropriate answer, and audit the support and responsiveness of a historical management answer. The authors construct 560 specialist-reviewed exchanges, yielding 1,680 task instances, and evaluate five models. The reported results show different strengths across tasks, with persistent gaps in answer completeness, evidence assessment, and citation coverage.

Critical strengths

The questions come from real earnings calls and retain the preceding dialogue, providing useful examples of follow-ups and compound requests.

Separating whether an answer addresses the question from whether the evidence supports it is a useful design choice.

Critical weaknesses

1.Annotation reliability is not established. Both specialists review LLM-generated proposals, but the paper reports neither pre-adjudication agreement nor an independent audit without access to those proposals.

2.The sample selection and label coverage are unclear. The reviewed candidates are restricted to exchanges with one to three question parts and selected provisional actions. The paper does not explain the selection procedure sufficiently or report the final label distributions.

3.The paper omits complete candidate prompts, tool budgets, context-truncation rules, and so on.

4.The experiments do not identify the source of the low scores. Task B uses predicted Task A assessments, but there is no comparison using gold evidence.

Questions for authors

Can you provide agreement results from an independent specialist audit of the reference annotations, together with agreement between the semantic evaluator and independent specialists? For Task B, do the main rankings and conclusions remain stable after human correction or replacement of the evaluator?

How were the 600 review exchanges selected? What are the final counts for each support label and answer action, and does the benchmark contain authentic cases requiring complete abstention?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pv2r - Part 1

Dear Reviewer pv2r,

Thank you for recognizing the value of natural earnings-call questions and the distinction between responsiveness and evidence support. We hope the responses below address the concerns raised in your weaknesses and questions sections.


W1 & Q1 (Part 1). The paper reports neither pre-adjudication agreement nor an independent audit without access to those proposals.

R1 to W1&Q1 (Part 1). Thanks for this constructive comment. We have conducted an inter-annotator agreement analysis on the final 560 cases involving our three professional financial specialists (Specialists A, B, and Adjudicator C). Specifically, we computed Cohen's Kappa coefficients (κ\kappa) to measure independent agreement before adjudication (A-B) and post-discussion consistency against the adjudicator (A-C and B-C) across evidence sufficiency, answer action, and answer completeness.

The independent specialist reviews show high consistency (κ>0.81\kappa > 0.81, indicating strong agreement) even before adjudication (row A-B). This confirms the reliability of the annotation process. Furthermore, it demonstrates that the final reference annotations remain fully defensible and reliable, unaffected by potential bias from the shared initial machine proposals. This table is now updated as Table 7 in the revised version.

ComparisonEvidence sufficiency κAnswer action κAnswer completeness κ
A–B0.8350.8360.911
A–C0.8110.8210.923
B–C0.8780.8800.914

Q1 (Part 2). For Task B, do the main rankings and conclusions remain stable after human correction or replacement of the evaluator?

R2 to Q1 (Part 2). We agree that verifying the reliability of our automatic LLM evaluator against human specialists is crucial to confirming the stability of our paper's main conclusions. Due to the limited rebuttal window, we re-engaged one specialist to independently evaluate 100 samples of model outputs across Tasks A, B, and C, and we applied GPT-6.1 Sol (xhigh) as the second LLM evaluator to validate the capability gaps. The results comparing the automatic evaluators' assessments with human specialist evaluations are summarized below:

Task modelA Support F1A Action F1B Grounded F1B Unsupported rate ↓C Citation accuracy
Evaluators: GPT-6 Astra / GPT-6.1 Sol / Specialist.
GPT-5.6 Sol51.35 / 53.61 / 54.0056.73 / 57.05 / 60.4332.82 / 33.98 / 35.2124.52 / 22.44 / 28.5968.27 / 62.89 / 63.79
GPT-6 Astra46.60 / 50.11 / 47.3144.88 / 45.79 / 45.5846.98 / 43.36 / 44.7510.34 / 10.52 / 10.5887.50 / 87.27 / 87.42
Qwen3.8-27B43.09 / 43.43 / 43.4343.52 / 43.87 / 43.8725.12 / 26.02 / 24.8735.09 / 35.50 / 38.3584.16 / 86.19 / 83.67
Gemma4-31B14.49 / 14.80 / 14.4913.11 / 13.41 / 13.1114.68 / 13.68 / 13.9214.20 / 17.58 / 16.900.00 / 0.00 / 0.00
GLM-4.7-Flash7.66 / 7.66 / 7.8713.29 / 13.55 / 13.4613.21 / 12.38 / 12.0434.26 / 39.57 / 38.218.15 / 8.15 / 8.15

Each cell reports the evaluated results using GPT-6 Astra (xhigh), GPT-6.1 Sol (xhigh), and specialists regarding 5 major metrics of Tasks A, B, and C.

As shown in the results above, both automatic evaluators exhibit strong alignment with human specialists, confirming their reliability on nuanced outputs. Crucially, the main ranking, conclusions, and relative capability gaps remain stable: GPT-6 Astra consistently leads in generation (Task B) and auditing (Task C), while GPT-5.6 Sol performs best on Task A. This demonstrates that our core findings are consistent among the evaluator choices.


Q2 (Part 1) & W2 (Part 1) How were the 600 review exchanges selected? The paper does not explain the exchanges selection procedure sufficiently.

R1 to Q2 (Part 1) & W2 (Part 1). Thanks for catching this point. During annotation, we selected the 600 exchanges by keeping those with one to three question parts (Sec. 3.2.2) to reduce annotation difficulty and potential bias from the specialists, as more question parts will encounter more subjective annotations. We also tried to balance the number of selected exchanges regarding the number of question parts in them. We now report the distribution of exchanges with one to three question parts as follows, and these results are also updated in the revision in Table 8.

Question parts per exchangeExchangesPercentage
120135.89%
220736.96%
315227.14%
Total560100%
AuthorsOpenReview ↗

Response to Reviewer pv2r - Part 2

(Continued from Part 1)

Q2 (Part 2) and W2 (Part 2). What are the final counts for each support label and answer action? Report the final label distributions.

R2 to Q2 (Part 2) & W2 (Part 2). Point taken. We report the support label and answer action statistics as follows, and they are in Tables 9 and 10 in the revision.

Support label statistics

Support label#Question partsPercentage
Directly supported454.20%
Supported by calculation20.19%
Partially supported98892.25%
Not found in available evidence70.65%
Contradicted by evidence191.77%
Cannot determine100.93%
Total1,071100%

Answer action statistics

Answer action#Question partsPercentage
Answer with evidence474.39%
Answer with qualification98892.25%
Correct the question premise191.77%
Do not answer171.59%
Total1,071100%

W3. The paper omits complete candidate prompts, tool budgets, context-truncation rules, and so on.

R1 to W3. Thanks for catching this point. We now append task prompts in Appendix H. For tool budgets and context-truncation rules, since we only test the base model performance, no additional harness, excluding the BM25 retriever, was involved in our experiment. To remove confusion, we now explicitly state these in the experiment settings in Sec. 4.1.


W4. The experiments do not identify the source of the low scores. Task B uses predicted Task A assessments, but there is no comparison using gold evidence.

R1 to W4. Thanks for this comment. When designing AAA-Fin, our primary goal was to measure an agent’s end-to-end financial analysis performance in a realistic deployment setup. Thus, the low scores actually reflect the unsatisfying performance of the base models.

We assume The experiments do not identify the source of the low scores. in your comment, is to analyze if errors propagating from Task A harm Task B's performance. To break this down, we compared GPT-6 Astra (performs best on Task B) on the final 560 exchanges using either its predicted Task A or the ground-truth (reference) of Task A and report the overall Task B changes as follows:

Experiment results across all 560 exchanges

Task B metricInput: A’s predictionInput: A’s ground-truth (change %)
Grounded F148.8751.16 (+2.29)
Proposition recall37.1039.88 (+2.78)
Factual precision91.3492.37 (+1.03)
Unsupported rate ↓8.667.63 (−1.03)
Action compliance62.7467.77 (+5.03)
Numerical accuracy27.7431.31 (+3.57)

From the results above, we see that providing reference Task A assessments and supporting evidence improves Task B across the reported metrics. The largest gain is in action compliance (+5.03 points), indicating closer adherence to the required response behavior. Higher proposition recall (+2.78 points) and factual precision (+1.03 points), alongside the Grounded F1 gain (+2.29 points), indicate more complete and better-grounded answers. Numerical accuracy also increases by 3.57 points. This pattern suggests that better upstream inputs help the agent both identify the content to communicate and carry out the required response actions, supporting the value of evaluating assessment and answering jointly.

We further examined the 123 exchanges containing 142 question parts with incorrect original Task A actions to assess how improving upstream inputs affects answer generation. We compared Task B responses generated using the predicted and ground-truth Task A outputs.

Task B metricInput: A’s predictionInput: A's ground-truth (change %)
Grounded F147.8554.78 (+6.93)
Proposition recall36.4743.20 (+6.73)
Factual precision87.3589.50 (+2.15)
Unsupported rate ↓12.6510.50 (−2.15)
Action compliance71.2778.02 (+6.75)
Numerical accuracy23.5836.86 (+13.28)

On this subset, reference inputs increase Grounded F1 from 47.85 to 54.78 (+6.93 points) and proposition recall from 36.47 to 43.20 (+6.73 points). These improvements indicate that better upstream assessments and evidence help Task B convey more required information and produce more complete answers.

We appreciate the thorough review and valuable comments from Reviewer pv2r and hope our responses address all your concerns effectively.

Yours sincerely,

The authors of Submission 41002

Reviewer_5KzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AAA-FIN, a benchmark for evaluating evidence-grounded financial agents using naturally occurring earnings-call interactions. Each example includes an analyst’s question, earlier exchanges with management, and the company’s most recent eligible 10-K filing available before the call. For the Audit task, it also includes management’s actual response to the question. The benchmark decomposes evaluation into three linked tasks. Assess requires the model to decompose the analyst request, determine whether the company’s filings or earlier dialogue provide enough evidence to answer the question, and select an appropriate response action. Answer requires a complete response constrained to permitted evidence. Audit requires judging statement-level support in the historical management answer and how completely that answer resolves each question part.

The construction pool contains 7750 exchanges from 662 calls and 208 companies. The authors select 600 exchanges for expert review and retain 560 exchanges, corresponding to 1680 task views, 1071 question parts, 7844 required propositions, and 6763 historical statements. Each reviewed record is independently handled by two experts, with a third expert adjudicating disagreements; 40 exchanges are ultimately removed. The final benchmark therefore uses human-reviewed gold annotations rather than directly promoting model-generated labels.

Five models are tested using a common BM25 retrieval setup, including two proprietary models such as GPT-6 Astra and three open-weighted models such as Qwen3.8-27B. GPT-5.6 Sol performs best overall on Assess, whereas GPT-6 Astra performs best on Answer and Audit, illustrating that evidence assessment, grounded generation, and answer auditing do not rank models identically. In Answer, proposition recall remains below 38% for every model despite substantially higher factual precision, while Audit shows much stronger resolution judgments than statement-support judgments.

Critical strengths
  • The benchmark makes evidence-sensitive behavior explicit and helps distinguish different sources of answer failure. AAA-FIN separates question interpretation, evidence support, response actions, and answer auditing. These dimensions distinguish an answer that addresses a request but lacks support from one that is grounded yet incomplete. Assess feeds evidence judgments into Answer, allowing uncertainty about support to affect the response. This makes evidence-sensitive behavior an explicit part of the agent’s task.
  • The benchmark combines specialist review, traceable evidence, and explicit scoring rules to support reliable annotations and inspectable evaluations. Two specialists independently review each selected record, with a third adjudicating disagreements. Filing chunks retain source metadata and availability dates, making the permitted evidence boundary inspectable. Explicit scoring rules and evaluator prompts support tracing errors back to question decomposition, support judgments, or answer construction.
Critical weaknesses
  • The conceptual contribution appears incremental, centered on integrating evidence assessment, grounded answering, and historical-response auditing in one financial benchmark.
  • The reported Task B results do not distinguish errors inherited from Task A from errors made during answer generation. Task B receives the predicted assessments from Task A but is evaluated against human-reviewed reference requirements. Although it can revisit the original question and evidence, the aggregate scores do not reveal whether it corrects or propagates upstream mistakes.
Questions for authors
  • Could the authors clarify what distinct evaluation insight follows from combining Assess, Answer, and Audit beyond reporting their separate capability scores?
  • Could the authors clarify how they interpret Task B performance when the preceding Task A assessment is incorrect?
  • Could the authors please clarify how differences between the three conversation-dependence groups (independent, history-helpful, history-required) might affect the comparison?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5Kze - Part 1

Dear Reviewer 5Kze,

Thank you for recognizing the value of our work in benchmarking financial agents from multiple dimensions through the designed tasks. We hope the following responses address your concerns.


W1 & Q1. The conceptual contribution appears incremental because it integrates assessment, grounded answering, and historical-response auditing.

R1 to W1&Q1. We appreciate this comment, but would like to highlight AAA-Fin's main contribution in comprehensively benchmarking financial agents in real multi-turn analytical scenarios. To be specific, AAA-Fin provides a shared evaluation setting in which financial requests, response requirements, and answer judgments are anchored to the same authentic exchange, prior dialogue, and pre-call 10-K. Question-part annotations allow support decisions and answer actions to be compared with what the generated answer conveys, while the historical response provides an independent target for evaluating the agent's auditing ability.

Nevertheless, AAA-Fin is directly shaped by the regulatory landscape governing financial disclosures, such as Regulation FD (17 C.F.R. § 243.100) and Rule 10b-5 (17 C.F.R. § 240.10b-5). During benchmarking, the financial agent represents an issuer. It needs to confine its response to the parts of a question the disclosed record supports and state the qualifications that record requires. By holding the natural multi-turn earnings-call exchange and the pre-call 10-K constant across all three designed tasks, our joint setup unlocks three evaluation insights that isolated scores completely miss.

AuthorsOpenReview ↗

Response to Reviewer 5Kze - Part 2

(Continued from Part 1)

W2 & Q2. The aggregate scores of Task B do not reveal whether it corrects or propagates upstream mistakes. Could the authors clarify how they interpret Task B performance when the preceding Task A assessment is incorrect?

R1 to W2&Q2. We thank the reviewer for highlighting the interaction between assessment and answering, as it hits on a core challenge in evaluating LLM agents on tasks that need multi-hop reasoning. When designing AAA-Fin, our primary goal was to measure an agent’s end-to-end financial analysis performance in a realistic deployment setup, where Task B measures final-answer quality when the agent receives its own predicted Task A assessment (Sec. 2.4). The original question, prior dialogue, and 10-K retrieval remain available, and the answer is evaluated against the human reference requirements. Task B can therefore receive credit for an appropriate, supported answer even when the preceding assessment is incorrect. We also admit that the aggregate scores in the submission cannot distinguish whether the agent propagates errors or fixes the mistakes from the upstream Task A.

To break this down, we compared GPT-6 Astra (performs best on Task B) on the final 560 exchanges using either its predicted Task A or the ground-truth (reference) of Task A and report the overall Task B changes as follows:

Experiment results across all 560 exchanges

Task B metricInput: A’s predictionInput: A’s ground-truth (change %)
Grounded F148.8751.16 (+2.29)
Proposition recall37.1039.88 (+2.78)
Factual precision91.3492.37 (+1.03)
Unsupported rate ↓8.667.63 (−1.03)
Action compliance62.7467.77 (+5.03)
Numerical accuracy27.7431.31 (+3.57)

From the results above, we see that providing reference Task A assessments and supporting evidence improves Task B across the reported metrics. The largest gain is in action compliance (+5.03 points), indicating closer adherence to the required response behavior. Higher proposition recall (+2.78 points) and factual precision (+1.03 points), alongside the Grounded F1 gain (+2.29 points), indicate more complete and better-grounded answers. Numerical accuracy also increases by 3.57 points. This pattern suggests that better upstream inputs help the agent both identify the content to communicate and carry out the required response actions, supporting the value of evaluating assessment and answering jointly.

We further examined the 123 exchanges containing 142 question parts with incorrect original Task A actions to assess how improving upstream inputs affects answer generation. We compared Task B responses generated using the predicted and ground-truth Task A outputs.

Task B metricInput: A’s predictionInput: A's ground-truth (change %)
Grounded F147.8554.78 (+6.93)
Proposition recall36.4743.20 (+6.73)
Factual precision87.3589.50 (+2.15)
Unsupported rate ↓12.6510.50 (−2.15)
Action compliance71.2778.02 (+6.75)
Numerical accuracy23.5836.86 (+13.28)

On this subset, reference inputs increase Grounded F1 from 47.85 to 54.78 (+6.93 points) and proposition recall from 36.47 to 43.20 (+6.73 points). These improvements indicate that better upstream assessments and evidence help Task B convey more required information and produce more complete answers.


Q3. Could the authors please clarify how differences between the three conversation-dependence groups (independent, history-helpful, history-required) might affect the comparison?

R1 to Q3. Thanks for this comment. In Sec. 4.3 and Table 16, we reported results separately for independent, history-helpful, and history-required exchanges, containing 94, 204, and 262 exchanges, respectively. All five models are evaluated on the same exchanges within each group under the same scoring protocol. The breakdown reveals that relative model strengths can change across conversational settings. For example, Astra exceeds Sol in Task A Action F1 on independent exchanges (42.52 versus 30.97), whereas Sol performs better on history-required exchanges (56.38 versus 49.07). All five models also achieve higher Task B Grounded F1 on history-required than independent exchanges.

We thank you again for the valuable comments and review efforts. We are happy to answer any further questions.

Yours sincerely,

The authors of Submission 41002

Thank you for the detailed response and the additional experiments. The comparison between predicted and reference Task A inputs is useful and partially addresses my concern about interpreting Task B. I also appreciate the clarification that Task B evaluates final-answer quality against the reference requirements, allowing an appropriate answer to receive credit even when the preceding assessment is incorrect.

1. Joint evaluation and the Task A–B connection.

The new experiment provides a more concrete justification for evaluating assessment and answering together. For GPT-6 Astra, replacing predicted Task A inputs with reference inputs improves Grounded F1 by 2.29 percentage points overall and by 6.93 points on the subset containing incorrect Task A actions. These results support the claim that the upstream inputs affect final-answer quality. At the same time, proposition recall remains low even with reference inputs, suggesting that the observed incompleteness cannot be attributed solely to errors in Task A.

However, this comparison does not yet directly distinguish error correction from error propagation. Could you clarify whether the intervention replaces only the predicted assessment and response action, or also supplies different supporting evidence? If both change, the improvements should be interpreted as the combined effect of better assessments and evidence, rather than as an isolated effect of correcting Task A decisions. A question-part-level breakdown showing how often Task B follows an incorrect action, corrects it, or introduces a new error would address the original concern more directly. In particular, the exchange-level action-compliance score on the selected subset does not by itself establish how the 142 incorrectly assessed question parts were handled.

2. Interpretation of the conversation-dependence groups.

The response clarifies that models are evaluated on the same examples within each group, which makes within-group model comparisons meaningful. My concern, however, was about comparisons between groups. The independent, history-helpful, and history-required groups contain different questions and may differ in difficulty, answer requirements, and evidence availability. Consequently, higher scores on history-required exchanges do not establish that providing history improves performance.

A paired evaluation of the same exchanges with and without prior dialogue would be needed to isolate that effect. I would not require such an experiment simply to justify reporting the group breakdown, but the paper should explicitly describe it as a descriptive comparison across different subsets, rather than evidence of a causal benefit from conversation history.

3. Contribution and remaining assessment.

I agree that anchoring the tasks to the same exchange and evidence boundary is a useful benchmark design choice. The new Task A–B experiment strengthens this aspect of the contribution. Nevertheless, the regulatory motivation and shared setting do not, on their own, establish additional joint evaluation insights beyond the separate task scores. The response mentions three such insights but does not enumerate them. It would help to state these explicitly and distinguish insights demonstrated by the current experiments from analyses that the benchmark enables in future work.

Overall, the rebuttal has partially addressed my concern about Task B, and I recognize the value of the additional experiment. The interpretation of the conversation-group comparison remains unresolved, and my assessment of the conceptual contribution remains largely unchanged. I am therefore maintaining my current rating of 2 (weak rejection).

Reviewer_i6VLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AAA-FIN evaluates financial agents using natural earnings-call exchanges, preceding dialogue, and filings available before the call. It separates interpreting a request and assessing its evidential support, generating an appropriately grounded and complete answer, and auditing a historical management response. Its distinctive focus is the separation of responsiveness, evidence support, and answer completeness within a realistic conversational setting. The paper provides specialist-reviewed annotations and evaluates models across these complementary tasks.

Critical strengths
  1. The dataset construction includes substantive safeguards. Assessment and answer annotations are produced without access to the historical management answer. Two specialists review each record separately, with a third adjudicating disagreements. The scoring rules accept valid alternative wording and evidence, distinguish citation precision from completeness, and account for missing outputs.
  2. The experiments reveal useful distinctions between capabilities: strong answer generation does not necessarily imply strong assessment, factual precision can coexist with incomplete answers, and recognizing responsiveness is easier than verifying evidential support. These results support evaluating the capabilities separately, even though further validation is needed to establish the reliability of the measured gaps.
Critical weaknesses
  1. The reliability of the reference annotations is not sufficiently quantified for the judgments central to this benchmark. Separate specialist review and adjudication are meaningful safeguards, but the reported revision rate measures changes to the machine proposal rather than agreement between specialists. Evidence sufficiency, required qualifications, and answer completeness can admit reasonable disagreement. Without agreement results and an analysis of disputed cases, it is difficult to assess how consistently these concepts are operationalized. Access to a shared machine proposal does not invalidate the process, but makes evidence about independent agreement particularly important.
  2. The automatic evaluator also needs more direct validation. Semantic matching and evidence-related judgments depend on an LLM evaluator, yet the paper does not report an independent human-agreement study or an alternative-evaluator analysis establishing the robustness of the main conclusions. This is especially consequential for a benchmark that itself finds weaknesses in answer auditing. The evaluator is also among the evaluated models; this does not establish bias, but further motivates independent checks. As a result, the reported gaps are informative but cannot yet be confidently separated from ambiguity in the references or errors and preferences of the evaluator.
Questions for authors
  1. What do the separate specialist reviews reveal about agreement before adjudication, particularly on evidence sufficiency, answer actions, and required answer content? An analysis of representative disagreements would help clarify whether reasonable alternative interpretations are accommodated by the final references.
  2. How well does the automatic evaluator agree with independent specialists on a representative sample of model outputs, including partial answers, valid alternative formulations, and borderline support judgments? Do the principal model comparisons and capability gaps remain stable with another evaluator?
  3. How should the claims about answerability be scoped given the selection of evidence-backed, qualified, and premise-correcting cases and the exclusion of uncertain candidates? In particular, please clarify the coverage of cases requiring an answer to be withheld and distinguish lack of support in the permitted filings from factual falsity.
4 comments
AuthorsOpenReview ↗

Response to Reviewer i6VL - Part 1

Dear Reviewer i6VL,

Thank you for recognizing the safeguards in our dataset construction and the distinctions among financial-agent capabilities. We appreciate your comments on annotation and evaluator reliability. Below, we clarify what the reported results establish and outline the additional validation needed to address your concerns.


W1 & Q1. Reference reliability is not quantified by specialist agreement, and shared machine proposals may influence the annotations.

R1 to W1&Q1. We agree that quantifying independent agreement among specialists, beyond the revision rates relative to machine proposals shown in Table 12, is crucial for establishing the benchmark's reliability against subjective interpretation. To address this, we conducted an inter-annotator agreement analysis (added this as Table 7 in revision) on the final 560 cases involving our three professional financial specialists (Specialists A, B, and Adjudicator C). Specifically, we computed Cohen's kappa coefficients (κ\kappa) to measure independent agreement before adjudication (A-B) and post-discussion consistency against the adjudicator (A-C and B-C) across evidence sufficiency, answer action, and answer completeness.

As shown in the table below, the independent specialist reviews show high consistency (κ>0.81\kappa > 0.81, indicating high agreement) even before adjudication. This confirms that our operationalized definitions of evidence boundaries, required actions, and completeness are robust and objective. It also demonstrates that the final reference annotations remain fully defensible and reliable, unaffected by potential bias from the shared initial machine proposals.

ComparisonEvidence sufficiency κAnswer action κAnswer completeness κ
A–B0.8350.8360.911
A–C0.8110.8210.923
B–C0.8780.8800.914

W2 & Q2. The automatic evaluator needs independent validation, and the capability gaps remain stable with another evaluator.

R1 to W2&Q2. We agree that verifying the reliability of our automatic LLM evaluator against human specialists is important to confirming the stability of our paper's main conclusions. Due to the limited rebuttal time window, we re-engaged one specialist to independently evaluate 100 samples of model outputs across Tasks A, B, and C, and we additionally applied GPT-6.1 Sol (xhigh) as the second LLM evaluator to validate the capability gaps. The results comparing the automatic evaluators' assessments with human specialist evaluations are summarized below:

Task modelA Support F1A Action F1B Grounded F1B Unsupported rate ↓C Citation accuracy
GPT-5.6 Sol51.35 / 53.61 / 54.0056.73 / 57.05 / 60.4332.82 / 33.98 / 35.2124.52 / 22.44 / 28.5968.27 / 62.89 / 63.79
GPT-6 Astra46.60 / 50.11 / 47.3144.88 / 45.79 / 45.5846.98 / 43.36 / 44.7510.34 / 10.52 / 10.5887.50 / 87.27 / 87.42
Qwen3.8-27B43.09 / 43.43 / 43.4343.52 / 43.87 / 43.8725.12 / 26.02 / 24.8735.09 / 35.50 / 38.3584.16 / 86.19 / 83.67
Gemma4-31B14.49 / 14.80 / 14.4913.11 / 13.41 / 13.1114.68 / 13.68 / 13.9214.20 / 17.58 / 16.900.00 / 0.00 / 0.00
GLM-4.7-Flash7.66 / 7.66 / 7.8713.29 / 13.55 / 13.4613.21 / 12.38 / 12.0434.26 / 39.57 / 38.218.15 / 8.15 / 8.15

Each cell reports the evaluated results from GPT-6 Astra / GPT-6.1 Sol / Specialist regarding 5 major metrics of Tasks A, B, and C.

As shown in the results above, both automatic evaluators exhibit strong alignment with human specialists, confirming their reliability on nuanced outputs such as partial answers and borderline support judgments. Crucially, the principal model comparisons and relative capability gaps remain entirely stable: GPT-6 Astra consistently leads in generation (Task B) and auditing (Task C), while GPT-5.6 Sol performs best on Task A. This demonstrates that our core findings are consistent among the evaluator choice.

AuthorsOpenReview ↗

Response to Reviewer i6VL - Part 2

(Continued from Part 1)

Q3. Answerability scope, and distinguish lack of support in the permitted filings from factual falsity.

R1 to Q3. We thank the reviewer for this insightful question regarding the scope of answerability, the handling of withholding cases, and the distinction between filing support and factual falsity. We clarify these points as follows:

1. Scope of Answerability and Excluded Uncertain Candidates.
Our exclusion of uncertain candidates during specialist review (moving the review cohort from 600 to 560 exchanges) was intended to improve the reliability of the reference annotations. Reference clarity does not require a fully answerable question: a clear judgment can establish that only part of a request is supported or that its premise conflicts with the evidence.

2. Coverage of Cases Requiring Information Withholding. Notably, we design AAA-Fin to evaluate evidence boundaries at both the response-planning (in Task A) and answer-auditing stages (Task C). In Task A, 988 of 1,071 question parts (92.25%) were partially supported and required further qualification before generating an answer, and 17 parts (1.59%) required explicit withholding. Statistics of answer action cases are reported in the following table, Answer action statistics, and the row Do not answer denotes the number and ratio of cases requiring information withholding.

3. Lack of Filing Support vs. Factual Falsity We fully agree with the reviewer's distinction, which is anchored in the regulatory nature of SEC filings. Governed by the Securities Exchange Act of 1934, Sarbanes-Oxley (SOX) executive certifications, and Regulation S-K disclosure standards, Form 10-K serves as a rigorously audited, legally binding baseline. Consequently, under our benchmark policy, a "lack of filing support" (Unsupported) reflects the distinct reporting boundaries between statutory periodic disclosures and real-time earnings communications rather than factual falsity or fraud in the real world.

Answer action statistics (Table 10 in revision)

Answer action#Question partsPercentage
Answer with evidence474.39%
Answer with qualification98892.25%
Correct the question premise191.77%
Do not answer171.59%
Total1,071100%

Support label statistics (Table 9 in revision)

Support label#Question partsPercentage
Directly supported454.20%
Supported by calculation20.19%
Partially supported98892.25%
Not found in available evidence70.65%
Contradicted by evidence191.77%
Cannot determine100.93%
Total1,071100%

We thank you again for the constructive comments and review efforts, and we are happy to answer any further questions.

Yours sincerely,

The authors of Submission 41002

Thanks for the response. While my concern on W1 is somewhat addressed, but I find it difficult to believe the specialist evaluation for W2. In my experience, expert evaluations can be very different in investment analysis setting, because in many cases, there is no concrete answer, and different analysts can have different viewpoints and rules. But in the new table, expert evaluations are too much aligned with AI evaluation. I think you should test it with more human specialists. I am maintaining my score.

AuthorsOpenReview ↗

Respone to follow-up comments

Dear Reviewer i6VL,

Thank you for clarifying your concern. We assume that the statement analysts can have different viewpoints and rules means specialists can have different views.

We fully agree that financial specialists may reasonably differ in their investment judgments, but to remove confusion, we want to first clarify that investment analysis involves both:

  • Investment judgment, such as forecasting, valuation, and recommendations.
  • Evidence assessment, such as checking what a disclosure supports and whether an answer addresses a request.

The raw earnings calls can inform both activities, but AAA-Fin focuses on evidence assessment and the quality of responses to analysts' questions.

Secondly, while the agent's predicted answer is open-ended in its wording, all evaluators' (LLMs and specialist) evaluations check the evidence and response requirements, not the language style or effecacy of a forecast or investment recommendation.

Specifically, for Task A, the reported Support F1 and Action F1 compare the agent's predicted combined evidence-support labels and response actions with the reference annotations. The evaluator first identifies predicted and reference question parts that request the same information; the subsequent label comparisons and score calculations are deterministic. Thus, evaluating these outputs does not require specialists to independently formulate an investment opinion or select a new reference action. Differences in question-part alignment can affect the scores, while the reliability of the reference labels is addressed separately through specialist annotation and adjudication.

For Task B, it evaluates whether a generated answer:

  1. conveys the required information;
  2. makes factual claims supported by the selected 10-K or explicitly attributed prior management dialogue; and
  3. preserves necessary qualifications and source attribution. The scoring criteria also do not require agreement with an investment viewpoint or reproduction of the historical management response. An agent's outputs are assessed against predefined criteria for evidence support, content coverage, and necessary qualifications.

Nevertheless, Task C’s Citation Accuracy measures whether cited passages contribute to the agent’s predicted support judgments. Evaluators may differ in their assessments of individual passages while still producing similar aggregate scores.

When we designed the three Tasks in AAA-Fin, we already considered such uncertainty raised in your prior response and avoided it in the evaluation protocol. Our prior results on the 100 samples already validate this and we believe that adding more specialists won't change this fact. To fully address your concerns, the rest two specialists in our team are working on the re-evaluation following the same rules, and we will report the results when they are ready.

Yours sincerely,

The authors of Submission 41002

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.