- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AAA-FIN, a benchmark built from real earnings-call exchanges, preceding dialogue, and pre-call 10-K filings. Its three tasks assess whether a model can determine what the evidence supports, generate an appropriate answer, and audit the support and responsiveness of a historical management answer. The authors construct 560 specialist-reviewed exchanges, yielding 1,680 task instances, and evaluate five models. The reported results show different strengths across tasks, with persistent gaps in answer completeness, evidence assessment, and citation coverage.
Critical strengths
The questions come from real earnings calls and retain the preceding dialogue, providing useful examples of follow-ups and compound requests.
Separating whether an answer addresses the question from whether the evidence supports it is a useful design choice.
Critical weaknesses
1.Annotation reliability is not established. Both specialists review LLM-generated proposals, but the paper reports neither pre-adjudication agreement nor an independent audit without access to those proposals.
2.The sample selection and label coverage are unclear. The reviewed candidates are restricted to exchanges with one to three question parts and selected provisional actions. The paper does not explain the selection procedure sufficiently or report the final label distributions.
3.The paper omits complete candidate prompts, tool budgets, context-truncation rules, and so on.
4.The experiments do not identify the source of the low scores. Task B uses predicted Task A assessments, but there is no comparison using gold evidence.
Questions for authors
Can you provide agreement results from an independent specialist audit of the reference annotations, together with agreement between the semantic evaluator and independent specialists? For Task B, do the main rankings and conclusions remain stable after human correction or replacement of the evaluator?
How were the 600 review exchanges selected? What are the final counts for each support label and answer action, and does the benchmark contain authentic cases requiring complete abstention?
2 comments
Response to Reviewer pv2r - Part 2
(Continued from Part 1)
Q2 (Part 2) and W2 (Part 2). What are the final counts for each support label and answer action? Report the final label distributions.
R2 to Q2 (Part 2) & W2 (Part 2). Point taken. We report the support label and answer action statistics as follows, and they are in Tables 9 and 10 in the revision.
Support label statistics
| Support label | #Question parts | Percentage |
|---|---|---|
| Directly supported | 45 | 4.20% |
| Supported by calculation | 2 | 0.19% |
| Partially supported | 988 | 92.25% |
| Not found in available evidence | 7 | 0.65% |
| Contradicted by evidence | 19 | 1.77% |
| Cannot determine | 10 | 0.93% |
| Total | 1,071 | 100% |
Answer action statistics
| Answer action | #Question parts | Percentage |
|---|---|---|
| Answer with evidence | 47 | 4.39% |
| Answer with qualification | 988 | 92.25% |
| Correct the question premise | 19 | 1.77% |
| Do not answer | 17 | 1.59% |
| Total | 1,071 | 100% |
W3. The paper omits complete candidate prompts, tool budgets, context-truncation rules, and so on.
R1 to W3. Thanks for catching this point. We now append task prompts in Appendix H. For tool budgets and context-truncation rules, since we only test the base model performance, no additional harness, excluding the BM25 retriever, was involved in our experiment. To remove confusion, we now explicitly state these in the experiment settings in Sec. 4.1.
W4. The experiments do not identify the source of the low scores. Task B uses predicted Task A assessments, but there is no comparison using gold evidence.
R1 to W4. Thanks for this comment. When designing AAA-Fin, our primary goal was to measure an agent’s end-to-end financial analysis performance in a realistic deployment setup. Thus, the low scores actually reflect the unsatisfying performance of the base models.
We assume The experiments do not identify the source of the low scores. in your comment, is to analyze if errors propagating from Task A harm Task B's performance. To break this down, we compared GPT-6 Astra (performs best on Task B) on the final 560 exchanges using either its predicted Task A or the ground-truth (reference) of Task A and report the overall Task B changes as follows:
Experiment results across all 560 exchanges
| Task B metric | Input: A’s prediction | Input: A’s ground-truth (change %) |
|---|---|---|
| Grounded F1 | 48.87 | 51.16 (+2.29) |
| Proposition recall | 37.10 | 39.88 (+2.78) |
| Factual precision | 91.34 | 92.37 (+1.03) |
| Unsupported rate ↓ | 8.66 | 7.63 (−1.03) |
| Action compliance | 62.74 | 67.77 (+5.03) |
| Numerical accuracy | 27.74 | 31.31 (+3.57) |
From the results above, we see that providing reference Task A assessments and supporting evidence improves Task B across the reported metrics. The largest gain is in action compliance (+5.03 points), indicating closer adherence to the required response behavior. Higher proposition recall (+2.78 points) and factual precision (+1.03 points), alongside the Grounded F1 gain (+2.29 points), indicate more complete and better-grounded answers. Numerical accuracy also increases by 3.57 points. This pattern suggests that better upstream inputs help the agent both identify the content to communicate and carry out the required response actions, supporting the value of evaluating assessment and answering jointly.
We further examined the 123 exchanges containing 142 question parts with incorrect original Task A actions to assess how improving upstream inputs affects answer generation. We compared Task B responses generated using the predicted and ground-truth Task A outputs.
| Task B metric | Input: A’s prediction | Input: A's ground-truth (change %) |
|---|---|---|
| Grounded F1 | 47.85 | 54.78 (+6.93) |
| Proposition recall | 36.47 | 43.20 (+6.73) |
| Factual precision | 87.35 | 89.50 (+2.15) |
| Unsupported rate ↓ | 12.65 | 10.50 (−2.15) |
| Action compliance | 71.27 | 78.02 (+6.75) |
| Numerical accuracy | 23.58 | 36.86 (+13.28) |
On this subset, reference inputs increase Grounded F1 from 47.85 to 54.78 (+6.93 points) and proposition recall from 36.47 to 43.20 (+6.73 points). These improvements indicate that better upstream assessments and evidence help Task B convey more required information and produce more complete answers.
We appreciate the thorough review and valuable comments from Reviewer pv2r and hope our responses address all your concerns effectively.
Yours sincerely,
The authors of Submission 41002
Response to Reviewer pv2r - Part 1
Dear Reviewer pv2r,
Thank you for recognizing the value of natural earnings-call questions and the distinction between responsiveness and evidence support. We hope the responses below address the concerns raised in your weaknesses and questions sections.
W1 & Q1 (Part 1). The paper reports neither pre-adjudication agreement nor an independent audit without access to those proposals.
R1 to W1&Q1 (Part 1). Thanks for this constructive comment. We have conducted an inter-annotator agreement analysis on the final 560 cases involving our three professional financial specialists (
Specialists A, B, andAdjudicator C). Specifically, we computedCohen's Kappacoefficients (κ) to measure independentagreement before adjudication(A-B) andpost-discussion consistencyagainst the adjudicator (A-C and B-C) acrossevidence sufficiency,answer action, andanswer completeness.The independent specialist reviews show high consistency (κ>0.81, indicating strong agreement) even before adjudication (row A-B). This confirms the reliability of the annotation process. Furthermore, it demonstrates that the final reference annotations remain fully defensible and reliable, unaffected by potential bias from the shared initial machine proposals. This table is now updated as Table 7 in the revised version.
Q1 (Part 2). For Task B, do the main rankings and conclusions remain stable after human correction or replacement of the evaluator?
R2 to Q1 (Part 2). We agree that verifying the reliability of our automatic LLM evaluator against human specialists is crucial to confirming the stability of our paper's main conclusions. Due to the limited rebuttal window, we re-engaged one specialist to independently evaluate 100 samples of model outputs across Tasks A, B, and C, and we applied GPT-6.1 Sol (xhigh) as the second LLM evaluator to validate the capability gaps. The results comparing the automatic evaluators' assessments with human specialist evaluations are summarized below:
Each cell reports the evaluated results using GPT-6 Astra (xhigh), GPT-6.1 Sol (xhigh), and specialists regarding 5 major metrics of Tasks A, B, and C.
As shown in the results above, both automatic evaluators exhibit strong alignment with human specialists, confirming their reliability on nuanced outputs. Crucially, the main ranking, conclusions, and relative capability gaps remain stable: GPT-6 Astra consistently leads in generation (Task B) and auditing (Task C), while GPT-5.6 Sol performs best on Task A. This demonstrates that our core findings are consistent among the evaluator choices.
Q2 (Part 1) & W2 (Part 1) How were the 600 review exchanges selected? The paper does not explain the exchanges selection procedure sufficiently.
R1 to Q2 (Part 1) & W2 (Part 1). Thanks for catching this point. During annotation, we selected the 600 exchanges by keeping those with one to three question parts (Sec. 3.2.2) to reduce annotation difficulty and potential bias from the specialists, as more question parts will encounter more subjective annotations. We also tried to balance the number of selected exchanges regarding the number of question parts in them. We now report the distribution of exchanges with one to three question parts as follows, and these results are also updated in the revision in Table 8.