- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FOMCBench; a collection of 2,154 Q&A pairs from 89 Federal Reserve press conferences (2011-2025).
Three finance professionals labeled every answer as direct, intermediate or fully evasive, giving 6,462 released judgments (Fleiss kappa= 0.79, three-class).
The primary task is binary: direct vs. non-direct. The data is split into Train (1,554), Dev (200), Test-Std (200) and Test-Challenge (200); the test splits were selected using labels from three LLMs (GPT 5.2, Gemini 3 Pro and Opus 4.6).
The authors evaluate eight open-weight models, three API models and the earnings-call model Eva-4B, each given either a single Q&A pair or the full meeting transcripts. They also show that a three-LLM consensus disagrees with the human labels on 21.5% of Train pairs.
Critical strengths
- Dataset and human annotation: Every Q&A pair has 3 independent human labels. This is stronger contribution compared to the related benchmarks (EvasionBench gold set has one annotator, and CLARITY test set has two.)
- The supplementary material: The zip file is complete and helps reproducibility. Data, scorer, cached predictions, exact prompts and checksums. The provided scripts reproduce the reported numbers.
Critical weaknesses
- Practical use is not stated: The paper does not say who would use FOMCBench or what decision its scores would inform. For someone that is less familiar with the topic, the use cases and practically are not clear. How a Fed-only classifier has real practical value? (since there are only about 200 new pairs a year)
- Compared to the prior work, the contribution is mostly the human labels.
- The test sets are small and close to their ceiling
- Presentation could be much better, writing sometimes feel hard to follow (feels like an AI generated essay) and tables packs too many numbers. Tables 4 and 5 are hard to read, and their per-cell intervals are hard to check against the paper claim. A dot plot of the paired differences in the main text, with the full tables moved to the appendix, would make the result much clearer.
Questions for authors
- Who is the intended user of FOMCBench, and what decision would its scores inform?
- What makes central-bank answers different from earnings-call or political answers? For example, the chair is often expected not to answer certain questions.
- What is the exact overlap with the FOMC data in
Al Nuaimi et al. (2025), and what is new beyond the human labels? - How will the benchmark stay useful as models approach human agreement?
- Could you please check all citations in the paper? I noticed there is a note in the
README.mdof the supplementary materials states that "No verified 2023 source supports that string", followed by some explanations. Please check all the citations in the paper and in the supplementary materials. (that citation probably won't matter that much but just in case please check citations)
3 comments
I thank authors for their response. The updated PDPF substantially improves the paper. My remaining concerns are practical significance and the contribution relative to prior work. Most pairs overlap with "Al Nuaimi et al. (2025)", and the strongest closed model already reaches the professional's level on the binary task. The benchmark looks most useful for open-weight models and for auditing LLM-generated labels. I am leaning toward a weak accept because of the dataset's potential value for open-weight models.
Thank you; on overlap and practical significance
Thank you for rereading the revised paper and for raising your score. We are glad its value for open-weight models and for auditing LLM-generated labels came through. On your two remaining points:
Contribution relative to Al Nuaimi et al. (2025). That work used Claude 4 Sonnet to label evasiveness, validated the labels on a stratified human subset whose size and agreement it did not report, and studied their association with market reactions. FOMCBench asks a measurement question that such work depends on: how reliably does a model judge whether an answer supplies the requested information, compared with a finance professional on the same exchange? Unlike the earlier release, every one of the 2,332 exchanges carries three professional votes, the test uses later meetings, and a held-out professional is scored against the same two agreeing votes as each model (§§3–4). The overlap shows why this reference matters: on the 198 held-out pairs shared verbatim with the earlier release, its model labels agree with the professional binary majority on 48.0% (Appendices D and L).
Practical significance. With meeting input and the professionals' definitions, only GPT-6.1 Sol reaches the professional's range (92.1 against 94.1). With the released prompt it scores 89.0, the other closed models stay below the professional in every run, Sol scores 89.5 against 91.4 on three-class degree, and the best prompted open-weight model reaches 77.6 (§5.2). The benchmark is what makes these differences visible: it lets anyone check whether a labeller, open or closed, is reliable before relying on its labels. For open-weight work, researchers can train on the individual votes, select on development meetings and test on later conferences with the supplied scorer. On acceptance, we also plan to release our fine-tuned Qwen3-4B checkpoint and Gemma 4 31B adapter on Hugging Face. The FOMC-trained Gemma classifiers already transfer to a fixed 148-pair European Central Bank (ECB) subset, gaining 18.6–24.1 binary Macro-F1 points over the unadapted model (Appendix K.2).
Thank you again for the careful reading and the constructive suggestions that shaped the revision.
Response to Reviewer tMit
Thank you for the practical-use questions and paired-difference plot suggestion. Performance: Macro-F1 (%), 95% intervals; the label audit reports disagreement rates.
Q1 / W1: Users and decisions (§§1, 5.1, 5.4; §7). FOMCBench lets financial-Q&A researchers assess how closely a model or its labels reproduce this professional reference for whether an answer supplies requested information. The motivation is financial: vocal/facial cues are associated with asset-price movements, and earnings-call non-answers with lower returns and higher volatility (Section 1). Its scores inform two decisions.
Q2: What makes central-bank answers different? (§3.1; §§5.2–5.3). Like politicians facing conflicting pressures (Bavelas et al., 1986), chairs may legitimately withhold future policy decisions or individual members' views. The label frequencies describe responsiveness: 36.9% of pairs are intermediate and 7.5% fully evasive. The task asks whether the requested information is supplied, not whether withholding is justified or what the speaker intends. Eva-4B, trained for earnings calls, reaches 72.1 on FOMC answers against 94.1 for a professional. In every open-weight Test-Held run with the released prompts, most binary errors are answers that professionals judged direct and the model called non-direct.
Q3 / W2: Dataset and benchmark contributions (§2; Appendix L). FOMCBench provides 2,332 curated Q&A pairs spanning 2011–2026 with three finance professionals labelling every pair. Tables 1 and 42 compare resources. We add meeting-date splits, including meetings labelled after submission; matched human/model label audits and training tests; cached predictions and an executable scorer. Against Al Nuaimi et al. (2025a), 1,323 of our 2,154 pairs from 2011–2025 match by date and exact Q&A text. The other 831 come from six later meetings or cleaned/resegmented exchanges; the 178 pairs from 2026 are new. Prior model labels agree with the professional reference on 48.0% of the 198 matched test pairs.
W3 / Q4: Test size and future usefulness (§5.2; §7). Test-Held contains all 680 pairs from the 23 press conferences of 2024–2026, split by date. Scored against the same two agreeing professionals as the held-out vote (94.1 [92.9, 95.3] binary), prompted open-weight models reach at most 77.6 [74.1, 80.7], even with the professionals' own definitions in the prompt. With meeting input and the professionals' definitions, GPT-6.1 Sol reaches 92.1 [89.5, 94.4] (92.0–92.1 in three runs) and overlaps the professional range (Section 5.2; Table 31). Three uses remain. New conferences (eight a year) can extend the date-based test, as the 2026 meetings show. Prompted open-weight models retain a gap. The votes support audits of labellers and prompts (16.4% vs 5.0% disagreement; a median 16.5-point effect of label definitions). Native Gemma improves from 71.0 to 80.6–82.2 ordinary binary Macro-F1 after professional-label training (Table 3; Appendix K.2).
W4: Presentation (Figure 3; §6). Findings lead the paper; takeaway captions and the requested paired-difference plot replace dense comparisons; detailed tables are in the appendix. The AI-use statement describes assistance; authors remain responsible.
Q5: Citations (§3.1; Appendix N). The note concerns a string in our released prompts, "Rasiah et al. 2023", for which we found no published work. The paper cites the intended source, Rasiah (2010), Journal of Pragmatics 42(3), 664–680, and presents our degree definitions as an adaptation of it; the prompts stay unchanged because they are the exact experimental inputs. Bibliographic identities and claim support were checked with full texts where available, otherwise abstracts/source descriptions.
We hope the benchmark comparison (Table 3), human reference (§5.2) and uses (§7) now answer your questions.