acceptodds
Under review as a conference paper at ICLR 2027

FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

Abstract

Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. Long-horizon writer–reviewer loops research and verify every expected value and escalate failures to a human. On three expert-authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator while stating the expert rubric's expected value for more criteria, their scores agree with human grading, and in-house analysts prefer them in a blind review. Removing any component lowers agreement. The released 100-query FinAutoRubric Benchmark, built from in-house analysts' key questions across 78 tasks and eight asset classes, shows that rubrics from an earlier model generation still leave headroom for a later one.

Primary area:
datasets and benchmarks
Keywords:
financial reasoning, LLMs for finance, financial NLP, finance benchmark, financial analysis agents, rubric-based evaluation, LLM-as-a-judge, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pNLdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FinAutoRubric is an expert-guided framework that automatically generates evidence-grounded, context-specific rubrics for evaluating financial research agents. It combines reusable expert guidance, a Task Bank, query contextualization, and writer–reviewer validation loops to produce adaptable rubrics without per-item expert authoring. The accompanying FinAutoRubric Benchmark contains 100 analyst-derived queries spanning 78 tasks and eight asset classes, enabling broad evaluation of finance agents. Experiments show that FinAutoRubric rubrics closely match expert scoring, improve criterion coverage, align with human judgments, and support effective benchmarking of financial research agents.

Critical strengths
  1. The authors identify the problem of automatic rubric synthesis in the financial domain and incorporate expert knowledge into the process.
  2. The rubrics generated by FinAutoRubric are more highly recognized by financial experts compared with previous works.
  3. The authors construct the FinAutoRubric Benchmark based on questions from financial analysts.
Critical weaknesses
  1. Although the involvement of financial experts improves the quality of rubrics generated by FinAutoRubric, it also reduces the efficiency of the overall system and makes generalization more difficult. The authors do not compare FinAutoRubric with previous works in terms of efficiency and cost, including the time required for financial expert annotation.
  2. I am curious about how the quality of rubrics generated by FinAutoRubric compares with the built-in rubrics provided by BigFinanceBench, FrontierFinance, and Finance Agent Benchmark when used to evaluate models.
  3. Does the improvement of rubrics lead to significant changes in model performance, ranking, or gradients, or does it only introduce minor score perturbations? Can rubrics generated by FinAutoRubric identify certain model behaviors or weaknesses that previous rubrics fail to detect? The paper lacks related discussions.
Questions for authors
  1. I would like the authors to provide more information about the financial expert annotation process, including the number of experts involved, the annotation time, and whether annotations were conducted independently or followed by agreement/reconciliation.
  2. Since involving financial experts may increase the cost and reduce the efficiency of the rubric generation process, could the authors provide a detailed comparison of the time and computational/human costs between FinAutoRubric and previous rubric generation approaches?
  3. Does FinAutoRubric-generated rubric evaluation reveal different model behaviors or failure cases compared with existing benchmark rubrics? Specifically, can the authors provide examples where FinAutoRubric identifies weaknesses of financial agents that previous rubrics fail to capture?
Reviewer_gyLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FinAutoRubric, an expert-guided framework for automatically generating rubrics to evaluate financial research agents. It aims to reduce the repetitive work of authoring rubrics item by item, and to support different query contexts, information cutoffs, and institutional evaluation requirements. The framework consists of three stages. It first builds a reusable, expert-screened Task Bank. It then decomposes each query into a task and its specific context and matches the task to task-level criteria. Finally, writer–reviewer loops across different model families retrieve evidence and compute and verify expected values, while programmatic rules handle rubric-item checks, point allocation, and final acceptance. The authors evaluate generation quality on three financial benchmarks with expert-authored rubrics, using score-ranking agreement, mean absolute score error, and coverage of expert rubric items. They supplement these with a blind analyst review, a human-grading agreement experiment, and component ablations. The results show that the method outperforms the compared baselines on coverage that requires matching both the evaluation demand and the expected value, while its scoring agreement is on par with the strongest baseline overall.

Critical strengths
  1. The paper establishes a clear division of labor between reusing expert knowledge and generating query-level rubrics, which is of practical value. It separates reusable expert principles and task-level evaluation requirements from the query-specific entities, periods, assumptions, and expected answers. This design addresses the need in financial research evaluation for rubrics to adapt as information is updated and as institutional requirements change, and it offers a reasonable path toward reducing repetitive annotation work. For queries that do not completely and uniquely match a task in the Task Bank, the framework also allows task-level criteria to be generated from the expert principles, rather than forcing existing templates onto them.

  2. The experiments adopt multiple complementary quality assessments and provide a thorough uncertainty analysis. The main experiments compare different rubric sources on the same response pool under a fixed three-judge ensemble, measuring ranking agreement, score deviation, and expected-value-matched coverage. FinAutoRubric achieves expected-value-matched coverage of 60.8%, 40.7%, and 80.8% on the three benchmarks, higher than all compared methods on each. The paper further provides paired bootstrap analysis, a blind review of 45 queries by three analysts, and an agreement experiment against human grading of responses. These assessments support the practical value of the generated rubrics from different perspectives and avoid reliance on a single automatic metric.

Critical weaknesses
  1. Figure 2a presents on-demand generation for new queries or information cutoff dates, together with institution-specific guidance, as key properties of the method. Although the paper evaluates rubric generation across different queries, it does not include controlled experiments that regenerate rubrics after changes to the principle library or cutoff to verify whether the grading requirements, evidence, and expected values are updated correctly. In Section 4, the authors explicitly limit the relevant analyses to compliance with the current rules and the sensitivity of fixed rubrics to answers that violate task requirements. Reducing the cost of expert rubric authoring is also an important motivation for this work. However, Appendix B reports only cumulative time budgets: 3,600 seconds per query for the writer and 3,000 seconds per query shared across all reviewer roles. Actual token usage, end-to-end runtime, the rate of human intervention, and the cost of recovering failed runs in the full system are not reported. Consistency across repeated rubric generations for the same query is not directly evaluated either.

  2. FinAutoRubric employs a long-horizon agent capable of web retrieval and Python execution and introduces an additional model for review. Although the authors use the same generation model and reasoning effort across methods and provide EvalAgent with the same web access and cutoff settings, they do not report actual resource consumption, nor include a comparison under a matched total budget. Consequently, the gain in expected-value-matched coverage may partly stem from additional retrieval and computation and cannot be attributed solely to expert guidance. Meanwhile, on coverage that does not require expected values, FinAutoRubric’s point estimates are lower than those of Qworld and EvalAgent on all three benchmarks. The advantage in scoring agreement also appears mainly on BigFinanceBench, while on the other two benchmarks its differences from the strongest baseline are within noise. The ablation study is conducted only on BigFinanceBench and does not report confidence intervals for the differences. Removing the Task Bank lowers τb by 0.035 but yields a lower MAE. The variant without contextualization also removes the validator and the composition review, and the 11 of 50 failed generations are scored as zero, making it difficult to identify the independent contribution of contextualization.

  3. Weak and potentially biased human evaluation. The blind review relies on only three in-house analysts, potentially sharing the institutional standards used to develop the framework. Differences in rubric point formats may compromise blinding (Appendix D). Analysts were also permitted AI assistance, with analyst C using it to draft rationales. Inter-analyst agreement is extremely low (Fleiss’ κ = 0.03), raising concerns about the reliability of their preferences. The ProfBench comparison relies on only 26 decided response pairs, and the authors acknowledge that FinAutoRubric’s lead is within statistical uncertainty.

  4. Lack of independent validation of the released benchmark. The paper does not report independent expert validation of the 100 FinAutoRubric Benchmark rubrics. The authors acknowledge that the results represent “judge scores against the released rubrics rather than expert-verified correctness.” Moreover, the two highest-ranked solvers belong to the same model families used to generate and review the rubrics (Table 3), creating a potential model-family alignment confound. The queries originate from an unnamed in-house analyst team, making their representativeness difficult to assess.

Questions for authors
  1. Does the improvement survive a compute-matched comparison? Please evaluate a controlled baseline using the same agent harness, web and Python access, 6,600-second budget, and cross-family review pass as FinAutoRubric, but without its Task Bank or expert principle library. Report actual runtime, token usage, and cost for all methods. If the advantage in expected-value-matched coverage largely disappears, what evidence remains that expert guidance, rather than additional computation and review, drives the improvement?
  2. Are the two headline properties actually supported? Figure 2a claims on-demand generation and custom guidance, but neither is tested. Please regenerate rubrics for the same queries (a) under a later information cutoff and (b) under a modified principle library, for example a different materiality or evidence rule. Please show that expected values and criteria change correctly while nothing unrelated drifts, and report the variance across repeated generations of the same query.
  3. What fraction of the released rubrics are independently correct? Please have finance experts not involved in this work audit a random sample of the 100 benchmark rubrics for correctness of expected values, consistency with source evidence, whether each criterion is justified by the query, and appropriateness of point allocation. Report criterion-level error rates and reassess model rankings after correcting identified errors. Without independent validation, on what basis can the automatically generated rubrics serve as reliable ground truth for a new benchmark?
1 comment

Thank you for the response and the new analyses. The authors made a good effort to address my comments. But if I understand correct, several points are not empirically resolved. The cost table and the expert assessment are helpful, but my main concerns remain. Under the same budget ceiling, the full method reaches 60.8% expected-value coverage versus 57.9% for the closest ablation. This ablation also removes several components at once and does not match actual compute, so the contribution of expert guidance remains unclear. The incremental benefit of the Task Bank is statistically inconclusive. In addition, evidence for adaptation to new cutoffs remains limited, and adaptation to modified guidance and repeatability have not been tested. The expert assessment covers only 20 of the 100 released rubrics, and it is unclear whether the two experts are independent of this work. I will maintain my score.

Reviewer_DcV7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FinAutoRubric, which is a framework for generating rubrics for evaluating financial research agents. First, experts provide reusable guidance and review a Task Bank of reusable criteria, while writer and reviewer agents research answers, check sources, and generate criteria for questions. The paper includes comparisons of generated and expert-written rubrics by financial analysts, and tests whether rubric-based scores agree with human rankings of answers, finding that analysts generally prefer the generated rubrics and rubric-based scores agree with human rankings on 84.6% of the 26 evaluated answer pairs. The paper also introduces a 100 question benchmark.

Critical strengths

The paper addresses an important problem around accurately evaluating financial research agents. Creating reliable financial rubrics requires significant expert effort and answers can change as new information becomes available. Therefore, grounding rubric criteria in sources and calculations makes the rubrics easier to verify. The analyst comparisons and agreement with human grading provide reasonable evidence of their effectiveness.

Critical weaknesses
  1. Rubric reliability is questionable. The human evaluations are a good sign, but they don't establish the correctness of the benchmark’s rubrics, particularly on difficult tasks where the generating agents may make mistakes. Appendix J also documents an accepted rubric that misinterprets a question, showing that the review process can miss errors in the grading standard.
  2. The coverage metric could be misleading. It measures how many expert criteria are recovered with matching expected values, but doesn't penalize additional criteria that are incorrect or unnecessary. Higher coverage does not establish that the full rubric is a reliable grading standard. Although the paper also evaluates score agreement and human preferences, it does not directly measure how often generated criteria are incorrect or unnecessary.
  3. The advantage over simpler approaches is unclear. I think more baselines should be studied. The paper does not compare against experts working interactively with agents in terms of rubric quality, expert time, and total cost. Its strongest coverage baseline, Direct-Generate, uses a single generation call. Although other baselines are included, the comparisons do not separate the benefits of the proposed framework from those of spending more effort on research and review.
  4. Limited novelty: The contribution is mostly a workflow combining expert guidance, retrieval, rubric generation, and iterative review. I’m not convinced this work makes a significant contribution beyond existing work in generating rubrics for evaluation or for evaluating financial research agents in particular.
Questions for authors
  1. Have experts independently checked the new benchmark’s rubrics, particularly for difficult tasks? How often do errors survive the process?
  2. How often do generated rubrics include incorrect or unnecessary criteria, or reject valid alternative answers? Coverage alone would not capture these problems.
  3. How does FinAutoRubric compare with experts working interactively with agents, or a simpler research-and-review workflow, in rubric quality, expert time, and total cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.