acceptodds
Under review as a conference paper at ICLR 2027

FOMCBench: A Human-Annotated Benchmark for Federal Reserve Q&A Responsiveness

Abstract

Cautious language can answer a question, while a fluent, on-topic response can leave it unanswered. We introduce FOMCBench, a human-annotated benchmark of answer responsiveness: 2,154 English question–answer pairs from 89 Federal Reserve press conferences (2011–2025). Three finance professionals independently annotated every pair, blind to model predictions. The release provides 6,462 judgments, fixed splits, and human-majority references for binary responsiveness and three-class evasion degree. Three-class inter-annotator agreement is 0.791 (Fleiss’ κ). Human training labels disagree with a three-model majority on 26.3% of pairs for three-class degree and 21.5% for binary responsiveness. We compare eight open-weight and three API models under pair-only and full-meeting inputs, alongside Eva-4B’s native pair-input baseline. All use common references, scoring and meeting-bootstrap uncertainty. On the primary test, meeting-minus-pair binary Macro-F1 differences range from −8.0 to +24.9 percentage points; three-class comparisons can reverse direction. The contrast does not isolate context or model openness. Versioned votes, cached predictions and executable scoring support reproducible comparisons, with explicit limits from shared meetings, subjective judgments and source-text fidelity.

Primary area:
datasets and benchmarks
Keywords:
evasion detection, central bank communication, datasets and benchmarks, evaluation validity, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tMitOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FOMCBench; a collection of 2,154 Q&A pairs from 89 Federal Reserve press conferences (2011-2025).

Three finance professionals labeled every answer as direct, intermediate or fully evasive, giving 6,462 released judgments (Fleiss kappa= 0.79, three-class).

The primary task is binary: direct vs. non-direct. The data is split into Train (1,554), Dev (200), Test-Std (200) and Test-Challenge (200); the test splits were selected using labels from three LLMs (GPT 5.2, Gemini 3 Pro and Opus 4.6).

The authors evaluate eight open-weight models, three API models and the earnings-call model Eva-4B, each given either a single Q&A pair or the full meeting transcripts. They also show that a three-LLM consensus disagrees with the human labels on 21.5% of Train pairs.

Critical strengths
  • Dataset and human annotation: Every Q&A pair has 3 independent human labels. This is stronger contribution compared to the related benchmarks (EvasionBench gold set has one annotator, and CLARITY test set has two.)
  • The supplementary material: The zip file is complete and helps reproducibility. Data, scorer, cached predictions, exact prompts and checksums. The provided scripts reproduce the reported numbers.
Critical weaknesses
  • Practical use is not stated: The paper does not say who would use FOMCBench or what decision its scores would inform. For someone that is less familiar with the topic, the use cases and practically are not clear. How a Fed-only classifier has real practical value? (since there are only about 200 new pairs a year)
  • Compared to the prior work, the contribution is mostly the human labels.
  • The test sets are small and close to their ceiling
  • Presentation could be much better, writing sometimes feel hard to follow (feels like an AI generated essay) and tables packs too many numbers. Tables 4 and 5 are hard to read, and their per-cell intervals are hard to check against the paper claim. A dot plot of the paired differences in the main text, with the full tables moved to the appendix, would make the result much clearer.
Questions for authors
  • Who is the intended user of FOMCBench, and what decision would its scores inform?
  • What makes central-bank answers different from earnings-call or political answers? For example, the chair is often expected not to answer certain questions.
  • What is the exact overlap with the FOMC data in Al Nuaimi et al. (2025), and what is new beyond the human labels?
  • How will the benchmark stay useful as models approach human agreement?
  • Could you please check all citations in the paper? I noticed there is a note in the README.md of the supplementary materials states that "No verified 2023 source supports that string", followed by some explanations. Please check all the citations in the paper and in the supplementary materials. (that citation probably won't matter that much but just in case please check citations)
3 comments
AuthorsOpenReview ↗

Response to Reviewer tMit

Thank you for the practical-use questions and paired-difference plot suggestion. Performance: Macro-F1 (%), 95% intervals; the label audit reports disagreement rates.

Q1 / W1: Users and decisions (§§1, 5.1, 5.4; §7). FOMCBench lets financial-Q&A researchers assess how closely a model or its labels reproduce this professional reference for whether an answer supplies requested information. The motivation is financial: vocal/facial cues are associated with asset-price movements, and earnings-call non-answers with lower returns and higher volatility (Section 1). Its scores inform two decisions.

  • Whether a model-built label set can serve as the reference. Researchers who label central-bank communication with LLMs can measure the drift first. Against the label two professionals agree on, a three-LLM consensus disagrees on 16.4% of Train binary judgments (632 of 3,861 matched instances) and a held-out professional on 5.0%; consensuses of closed models given the professionals' definitions still disagree about two to three times as often as a professional in each of three runs (9.7–14.7%). The consensus and professionals respectively label 38 and 87 Train answers fully evasive. Section 5.1 reports the period reversal.
  • Which input protocol and label definitions to use and report when models judge long transcripts. With the released prompt, giving a model the whole press conference instead of one exchange raises held-out binary scores for all eight open-weight models, six with intervals excluding zero. On 4,490 ECB pairs (same panel; text-version bindings and validation rules in Appendix J), meeting input again raises binary scores for all eight models, every interval excluding zero (§5.5; Table 33).

Q2: What makes central-bank answers different? (§3.1; §§5.2–5.3). Like politicians facing conflicting pressures (Bavelas et al., 1986), chairs may legitimately withhold future policy decisions or individual members' views. The label frequencies describe responsiveness: 36.9% of pairs are intermediate and 7.5% fully evasive. The task asks whether the requested information is supplied, not whether withholding is justified or what the speaker intends. Eva-4B, trained for earnings calls, reaches 72.1 on FOMC answers against 94.1 for a professional. In every open-weight Test-Held run with the released prompts, most binary errors are answers that professionals judged direct and the model called non-direct.

Q3 / W2: Dataset and benchmark contributions (§2; Appendix L). FOMCBench provides 2,332 curated Q&A pairs spanning 2011–2026 with three finance professionals labelling every pair. Tables 1 and 42 compare resources. We add meeting-date splits, including meetings labelled after submission; matched human/model label audits and training tests; cached predictions and an executable scorer. Against Al Nuaimi et al. (2025a), 1,323 of our 2,154 pairs from 2011–2025 match by date and exact Q&A text. The other 831 come from six later meetings or cleaned/resegmented exchanges; the 178 pairs from 2026 are new. Prior model labels agree with the professional reference on 48.0% of the 198 matched test pairs.

W3 / Q4: Test size and future usefulness (§5.2; §7). Test-Held contains all 680 pairs from the 23 press conferences of 2024–2026, split by date. Scored against the same two agreeing professionals as the held-out vote (94.1 [92.9, 95.3] binary), prompted open-weight models reach at most 77.6 [74.1, 80.7], even with the professionals' own definitions in the prompt. With meeting input and the professionals' definitions, GPT-6.1 Sol reaches 92.1 [89.5, 94.4] (92.0–92.1 in three runs) and overlaps the professional range (Section 5.2; Table 31). Three uses remain. New conferences (eight a year) can extend the date-based test, as the 2026 meetings show. Prompted open-weight models retain a gap. The votes support audits of labellers and prompts (16.4% vs 5.0% disagreement; a median 16.5-point effect of label definitions). Native Gemma improves from 71.0 to 80.6–82.2 ordinary binary Macro-F1 after professional-label training (Table 3; Appendix K.2).

W4: Presentation (Figure 3; §6). Findings lead the paper; takeaway captions and the requested paired-difference plot replace dense comparisons; detailed tables are in the appendix. The AI-use statement describes assistance; authors remain responsible.

Q5: Citations (§3.1; Appendix N). The note concerns a string in our released prompts, "Rasiah et al. 2023", for which we found no published work. The paper cites the intended source, Rasiah (2010), Journal of Pragmatics 42(3), 664–680, and presents our degree definitions as an adaptation of it; the prompts stay unchanged because they are the exact experimental inputs. Bibliographic identities and claim support were checked with full texts where available, otherwise abstracts/source descriptions.

We hope the benchmark comparison (Table 3), human reference (§5.2) and uses (§7) now answer your questions.

I thank authors for their response. The updated PDPF substantially improves the paper. My remaining concerns are practical significance and the contribution relative to prior work. Most pairs overlap with "Al Nuaimi et al. (2025)", and the strongest closed model already reaches the professional's level on the binary task. The benchmark looks most useful for open-weight models and for auditing LLM-generated labels. I am leaning toward a weak accept because of the dataset's potential value for open-weight models.

AuthorsOpenReview ↗

Thank you; on overlap and practical significance

Thank you for rereading the revised paper and for raising your score. We are glad its value for open-weight models and for auditing LLM-generated labels came through. On your two remaining points:

Contribution relative to Al Nuaimi et al. (2025). That work used Claude 4 Sonnet to label evasiveness, validated the labels on a stratified human subset whose size and agreement it did not report, and studied their association with market reactions. FOMCBench asks a measurement question that such work depends on: how reliably does a model judge whether an answer supplies the requested information, compared with a finance professional on the same exchange? Unlike the earlier release, every one of the 2,332 exchanges carries three professional votes, the test uses later meetings, and a held-out professional is scored against the same two agreeing votes as each model (§§3–4). The overlap shows why this reference matters: on the 198 held-out pairs shared verbatim with the earlier release, its model labels agree with the professional binary majority on 48.0% (Appendices D and L).

Practical significance. With meeting input and the professionals' definitions, only GPT-6.1 Sol reaches the professional's range (92.1 against 94.1). With the released prompt it scores 89.0, the other closed models stay below the professional in every run, Sol scores 89.5 against 91.4 on three-class degree, and the best prompted open-weight model reaches 77.6 (§5.2). The benchmark is what makes these differences visible: it lets anyone check whether a labeller, open or closed, is reliable before relying on its labels. For open-weight work, researchers can train on the individual votes, select on development meetings and test on later conferences with the supplied scorer. On acceptance, we also plan to release our fine-tuned Qwen3-4B checkpoint and Gemma 4 31B adapter on Hugging Face. The FOMC-trained Gemma classifiers already transfer to a fixed 148-pair European Central Bank (ECB) subset, gaining 18.6–24.1 binary Macro-F1 points over the unadapted model (Appendix K.2).

Thank you again for the careful reading and the constructive suggestions that shaped the revision.

Reviewer_oJveOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces FOMCBench, a benchmark of question–answer pairs from Federal Reserve press conferences, where professional annotators label how directly each answer responds to the question. The authors use the human labels to audit LLM-consensus labels and find that the model consensus systematically under-identifies fully evasive answers. They also evaluate open-weight and API LLMs with pair-only versus full-meeting input, and conclude that scores depend on the input protocol and label granularity, while explicitly declining to claim a model ranking.

Critical strengths

Full-coverage professional annotation with an explicit audit against LLM-consensus labels. It shows that model-generated references change which examples count as evasive, which matters for any benchmark built on such labels.

Critical weaknesses
  1. The test splits are not independent of the evaluated systems. They were built from the evaluated models' disagreements, share meetings with training, and the primary metric was chosen after seeing results; this undermines the benchmark role and needs new prospective data to fix.
  2. The evaluation is confounded and yields little insight. The with/without-context comparison changes context, prompt wrapper and batching together, several no-context baselines are near-constant predictors, and the audit's consequence for training is never tested. The paper's length and many conditions also obscure what it actually claims.
Questions for authors
  1. How much did the "professional review" step change the labels, and was it done before or after model predictions were visible to the authors?
  2. Among the many conditions, which results are your primary claims? Do the no-context baselines of several open-weight models predict "non-direct" for nearly everything, so that context gains reflect recovery from format collapse?
  3. Do the conclusions hold on a meeting-disjoint subset and without the models used to select the test data?
1 comment
AuthorsOpenReview ↗

Response to Reviewer oJve

Thank you for the careful review and three questions (Macro-F1 %, 95% intervals).

Q3 / W1: Date-disjoint test (§3.3; Table 2). We replaced model-selected splits with a meeting-disjoint split by date: Train 2011–2022, Dev 2023 and Test-Held, the primary test, all 23 of 2024–2026 (680 pairs). No meeting is shared; no model output or label entered the split. The three original consensus APIs are outside the prompt panel; the four new closed models selected no test items. On Test-Held, with the released prompt, meeting input raises binary Macro-F1 for all eight open-weight models, six intervals excluding zero.

W1: New data (Appendices D, F, I). We added 178 pairs from the seven 2026 press conferences, labelled after submission. The authors state that the same three professionals each worked independently in a spreadsheet copy of the assignment file. On these meetings, meeting input raises binary Macro-F1 for all eight open-weight models (signs only: seven meetings give unstable intervals). Disclosure: pair- and meeting-input results on each part of Test-Held, and context-ablation results on 94 of its pairs (Appendix G), were known before the pooled test was defined. On all 2,154 pairs of 2011–2025 (descriptive; includes training meetings), all eight binary meeting-input gains have intervals above zero.

W1: Metric and task choices (§4; §5.4). We clarify the submitted p. 9 timing statement: both tasks and Macro-F1, with binary as primary, were chosen before inspecting model results. Sections 3.3 and 4 now distinguish these design choices from the later definition of the pooled test. Binary addresses the small fully evasive class. Under three-class scoring on Test-Held, seven of eight differences are positive, and Apertus falls (−6.4 [−11.2, −2.0]) while its binary score rises.

Q2: Primary claims and class collapse (§§5.1–5.4; Tables 16–17). Three claims. (1) On Train, the audited three-LLM consensus disagrees with two agreeing professionals on 16.4% of binary judgments, versus 5.0% for a held-out professional. Planned consensuses of three closed models using professional definitions disagree about two to three times as often in each of three runs (9.7–14.7% vs 5.0%). (2) In matched scoring on Test-Held, prompted open-weight models remain well below professionals (at best 77.6 vs 94.1); the strongest closed model, GPT-6.1 Sol, with meeting input and the professionals' definitions, reaches 92.1 [89.5, 94.4] (three runs: 92.0–92.1), overlapping the held-out professional. (3) The median open-weight meeting-input gain is smaller with the professionals' definitions (Section 5.4). On class collapse, yes for the affected runs. With released pair prompts, five models call at least 92.5% of Test-Held answers non-direct (professionals: 42.6%), with no invalid outputs. Meeting input brings Nemotron and GPT-OSS to about 54%, and they show the two largest gains. Mistral, Apertus and GLM stay above 89% non-direct or invalid outputs.

W2: Context and training controls (Tables 4–5; Appendices G, K). The ablation keeps prompt, wrapper and output format fixed (headline-rule timing: Appendix G), comparing B0 (target exchange only) with B1 (whole press conference). On Test-Held, context alone (B1−B0) raises binary Macro-F1 for five of eight models, each by at least 5 points with intervals excluding zero, and lowers one (DeepSeek, −3.3). Across models, the 92.5-point gain comprises 62.4 context, 7.4 prompt/format and 22.7 joint-labelling points. We fine-tuned Qwen3-4B-Instruct-2507 on the same Train pairs with professional or consensus labels (one recipe, three seeds; exploratory, planned before fitting), scored once on Test-Held. In every seed, professional labels recalled more of the 67 fully evasive answers (49, 40, 28 vs 28, 13, 18) and gave higher three-class scores; binary scores were similar. A DeBERTa-v3 encoder favours consensus labels, so the benefit depends on the learner. Appendix K.2: native 4B/31B fine-tuning and transfer, all seeds.

Q1: The professional review step. It affected four of the 2,154 pairs, those on which the three professionals gave three different labels. In the submitted version the review settled each as intermediate; it changed no other label and no binary reference. The revision removes the step. Current binary references (§3.2; Appendix B.2) collapse each vote before majority voting. Three-class references require a majority of the three released votes; those four ties are excluded only from three-class scoring. Agreement uses those votes; current scores do not depend on the review. The authors confirm that they had not seen model predictions when this review took place. The authors state that the three professionals initially labelled every pair independently, seeing neither model outputs nor others' labels.

Clarity (§1; §6). The introduction states the three findings; §6 consolidates recurring limits. We hope §§3.3, 5.1 and 5.4 answer your questions.

Reviewer_i7drOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FOMCBench, a benchmark for answer responsiveness: whether an answer provides the information requested by a question. It contains 2,154 question–answer pairs from 89 Federal Reserve press conferences between 2011 and 2025, each independently labeled by three finance professionals as direct, intermediate, or fully evasive. The benchmark has potential, particularly through its expert annotations and analysis of label sources. I personally love this kind of benchmark to be posted in the future. However, the current experiments do not establish a substantial advantage over existing benchmarks or a sufficiently distinctive complementary role. The evaluation remains largely limited to coarse classification, and the observed performance gaps offer limited insight beyond differences in model capability on finance-domain language.

Critical strengths
  1. Meaningful task definition. The paper clearly distinguishes hedging from evasion and evaluates whether the requested information is provided, without inferring speaker intent or honesty. This distinction is particularly important for central bank communication, where cautious language is common.

  2. Strong annotation design. All 2,154 pairs receive three independent expert annotations, collected without access to model predictions. This is an improvement over EvasionBench’s Gold-1K, which primarily relies on a single annotator. Retaining individual votes also supports research on annotator disagreement.

  3. Useful label-source audit. Table 3 shows that model consensus can obscure distinctions recognized by experts: 202 human-labeled direct examples and 71 fully evasive examples are assigned to the intermediate category. This raises an important concern about treating LLM consensus as ground truth.

Critical weaknesses
  1. Model-dependent test construction compromises the main comparisons. Test-Std is stratified by Condition-D model consensus, while Test-Challenge is selected using disagreements among API models and between Eva-4B and Opus 4.6. These systems are subsequently evaluated on the resulting sets, creating selection bias that complicates both the API/open-weight comparison and Eva-4B’s relative standing. Because the splits precede full human annotation, their human-label distributions—including only 10 fully evasive examples in Test-Std—are also consequences of model-based selection. The main findings should be validated on a model-agnostic random split.

  2. The G–B comparison is heavily confounded. The protocols simultaneously change context, batching, message wrappers, and prompts, preventing attribution of performance changes to context. Moreover, annotators saw ordered meeting-level exchanges and could consult subsequent follow-ups under FC1, making condition B more closely aligned with the reference-labeling procedure. A controlled ablation should hold the prompt and wrapper fixed while varying context availability.

  3. The contribution relative to existing data remains unclear. Al Nuaimi et al. (2025a) already provides 1,728 pairs from 83 FOMC press conferences using the same three-axis framework. Given the shared domain and taxonomy, leaving exact record overlap unverified is a significant omission. The label-source analysis could offer a stronger contribution, but the roughly 20% human-disagreement rate and 25% model–human disagreement rate are not directly comparable. A leave-one-annotator-out evaluation, with an explicit rule for disagreements between the remaining two annotators, would provide a more informative human reference.

  4. The central claim about label sources lacks a direct downstream test. Since the training set has complete human annotations, the natural experiment is to fine-tune the same model separately on expert labels and model-consensus labels, then evaluate both on a human-labeled test set. Deferring this experiment leaves the practical benefit of the improved annotations unestablished and weakens the case for a substantially more valuable benchmark.

  5. Some open-weight results require diagnostic analysis. Mistral-Small-4 pair and Apertus achieve binary Macro-F1 scores of 37.1 and 35.9, respectively, compared with 30.6 for the majority baseline and 53.5 for TF-IDF+LR. These results could reflect class collapse, prompt sensitivity, or output-parsing failures rather than limited semantic understanding alone. Prediction distributions, confusion matrices, and invalid-output rates are needed to determine whether meeting-level input improves responsiveness judgments or mainly alleviates such failures.

  6. Excessive qualifications obscure the main findings. Repeated caveats such as “pointwise, exploratory, unadjusted” and “does not establish…” make the main text difficult to follow. The approximately 45-page appendix also devotes substantial space to provenance details. The paper would benefit from consolidating recurring limitations, moving implementation records out of the main narrative, and stating its supported conclusions more directly. Where feasible, stronger experimental controls should resolve uncertainties that are currently addressed only through disclaimers.

  7. The evaluation provides limited diagnostic depth. Binary and three-way classification alone do not show what distinctive challenges the benchmark captures. The paper lacks prompt-sensitivity analysis and finer-grained breakdowns by topic, requested information, or evasion strategy. Such analyses could reveal which distinctions models struggle with and whether errors reflect financial knowledge, contextual reasoning, or responsiveness judgments, making the benchmark substantially more informative.

Questions for authors

Please refer to the weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer i7dr

Thank you for asking for independent tests, isolated context and a training-label comparison (Macro-F1 %, 95% intervals). FOMCBench combines curated 2011–2026 FOMC Q&A, three professional votes per pair and a matched human Macro-F1 reference and trained classifiers (Tables 1, 3). The EvasionBench release holds 16,726 model-labelled pairs, not the human gold set.

W1 Meeting-disjoint test (§3.3; Table 2; Appendix D). Split membership now uses meeting date alone, without model outputs or labels: Train 2011–2022, Dev 2023 and Test-Held, the primary test, all 23 press conferences of 2024–2026 (680 pairs, 67 fully evasive). Random pair splits can leak shared meeting context (Søgaard et al., 2021; Lazaridou et al., 2021). Results on each test part were known before pooling.

W2 G–B confound (§5.4; Table 5; Appendix G). The controlled ablation fixes prompt, wrapper and format, with the target exchange alone or its whole conference. Appendix G dates the headline rule. On Test-Held, context alone raises binary Macro-F1 for five of eight models, each by at least 5 points with intervals excluding zero, and lowers one (DeepSeek, −3.3). Across models, the 92.5-point gain comprises 62.4 context, 7.4 prompt/format and 22.7 joint-labelling points. Professionals could access full conference context (Section 3.2). Without continuation-screen pairs, the same six intervals exclude zero (Appendix G).

W3 Overlap and matched human reference (§5.1; Appendices D, L; Table 42). The advance is the professional reference on every pair. Exact date-and-text matching finds 1,323 of our 2,154 pairs from 2011–2025 in the prior corpus. The copy has model labels; they agree with the professional reference on 48.0% of the 198 matched test pairs. We ran leave-one-annotator-out: a held-out professional's vote and the three-LLM consensus are compared with the label the other two agree on; pairs where those two split are excluded. On Train the professional disagrees on 5.0% of binary judgments (192 of 3,861) and the consensus on 16.4% (632). Under a plan fixed before their requests, consensuses of three closed models given the professionals' definitions still disagree about two to three times as often as a professional in each of three runs (9.7–14.7% vs 5.0%). Section 5.1 reports the period reversal.

W4 Fine-tuning (Table 4; Appendix K). We fine-tuned Qwen3-4B-Instruct-2507 on the same Train pairs with professional or consensus labels (one recipe, three matched seeds; exploratory, planned before fitting), scored once on Test-Held. Professional labels recalled more of the 67 fully evasive answers in every seed (49, 40, 28 vs 28, 13, 18) and gave higher three-class scores; binary scores were similar. Consensus training had lower fully evasive recall but higher precision. A DeBERTa-v3 encoder favours consensus labels on both targets, so the benefit depends on the learner. Separate 4B/31B professional-label fine-tuning and transfer: Appendix K.2 (all seeds).

W5 Mistral and Apertus. On Test-Held, both models collapse toward non-direct with released pair prompts; neither has invalid pair-input outputs. They call 98.2% and 99.6% of answers non-direct (professionals: 42.6%), scoring 32.9 and 30.8 binary (constant direct 36.4; TF-IDF 60.1). With meeting input they stay at 89.1% and 96.8% non-direct or invalid, so their gains (+10.2, +2.0) are shifts within near-constant output. Nemotron and GPT-OSS leave the collapse and show the two largest gains. Tables 13, 16–17, 19 give scores, distributions and per-class recall; the scorer (Appendix O) prints confusion matrices.

W6 Qualifications (§1; §6). Agreed. Findings lead §1; takeaway captions guide results; §6 consolidates recurring limits; the appendix guide is on p. 16.

W7 Diagnostic depth (§§5.2–5.4; Tables 27, 31). We changed the degree-definition block under a plan fixed before these runs. Topic/requested-information/strategy breakdowns remain future work; human annotations cover degree. Chair/period results are in Tables 24, 29. The professionals' definitions raise ordinary single-exchange binary scores, with intervals excluding zero, for seven of eight open-weight and all four closed models, and lower them for none (median gains +16.5 and +11.4); every model calls fewer direct answers non-direct. On error sources, §5.3 reports that most binary errors call professionally judged direct answers non-direct, and that the professionals' definitions lower this share for all twelve models. We would be glad to deepen this error analysis if helpful. Using matched scoring with meeting input and the professionals' definitions, the best prompted open-weight model reaches 77.6 and GPT-6.1 Sol 92.1 [89.5, 94.4] (three runs: 92.0–92.1), overlapping the held-out professional (94.1 [92.9, 95.3]).

We hope the new test (§3.3), fine-tuning (§5.1) and context controls (§5.4) answer your concerns.

Remaining Concerns

I appreciate the authors' response and additional experiments. While several points have been clarified, some important concerns remain regarding the benchmark's validity, novelty, and interpretation of the results. I do not think some fundamental issues can be resolved in this case and recommend the authors reconsider this paper for an appropriate workshop. I will maintain my rating.

W1. Test-set construction and potential contamination. The newly added 2026 conference data fall outside the original corpus cutoff (December 2025). Were these samples annotated by the same three professionals under the same protocol, without access to model outputs? Moreover, the original paper acknowledges that test samples had been submitted to three API providers during corpus construction. How many Test-Held samples were previously exposed to these providers? Since the 2024–2026 transcripts are publicly available, potential contamination also deserves attention. Finally, the statement "Results on each test part were known before pooling" raises concerns about post-hoc test-set construction.

W2. Unresolved confounding between G and B. The authors acknowledge that professional annotators had access to full conference context, but do not examine how annotations would change when only individual Q&A pairs are available. Consequently, the B condition may inherently align better with the annotation setting, making it difficult to isolate the effect of contextual information. This concern remains unresolved.

W3. Limited novelty and potential benchmark saturation. The response clarifies that 1,323 of 2,154 Q&A pairs (~61%) overlap with Al Nuaimi et al., suggesting that the main contribution lies in professional annotations rather than new data. The leave-one-annotator-out results (5.0% disagreement for professionals vs. 16.4% for model consensus) support the value of these annotations. However, disagreement remains 2–3 times higher for models even when given professional definitions. The benchmark may also be approaching saturation: GPT-6.1 Sol achieves 92.1 [89.5, 94.4] in binary classification, compared with 94.1 [92.9, 95.3] for held-out professionals.

W4. Limited evidence for downstream benefits of professional labels. For Qwen3-4B, professional-label training improves fully evasive recall across three seeds (49/40/28 vs. 28/13/18), but the results show substantial seed variability and similar binary classification performance. DeBERTa-v3 instead favors consensus labels on both tasks. Thus, the claimed downstream benefits appear limited to minority-class recall in one model, rather than demonstrating consistent improvements.

W5. Class collapse undermines the reported contextual gains. Mistral and Apertus classify 98–99.6% of samples as non-direct, performing even worse than an always-direct baseline (36.4). Their apparent improvements with meeting-level context therefore largely reflect fluctuations in near-constant predictions rather than meaningful classification gains. This weakens the claim that meeting context consistently improves all open-source models.

AuthorsOpenReview ↗

Response to remaining concerns

Thank you for reading our response so carefully and for engaging with the new experiments. We were glad to see that you find the professional annotations and the leave-one-annotator-out audit valuable. We take each remaining concern in turn.

W1 Test construction and exposure. Yes, the 2026 pairs were labelled by the same three professionals under the same guide. The authors state that each worked independently and saw neither model outputs nor the others' labels (§3.2). Agreement on 2026 is somewhat lower (three-class Fleiss' κ 0.647 against 0.791 for 2011–2025; §6, L2).

On exposure, it helps to separate three things. First, the three consensus models (Claude Opus 4.6, GPT-5.2 and Gemini 3 Pro) labelled the 2011–2025 pairs (Appendix B.3), so the 502 test pairs from 2024–2025 were sent to those providers; the 178 pairs from 2026 were not. Second, a public transcript is not a public label. All 502 of the 2024–2025 pairs appear in an earlier public release, but only 280 of them appear there with labels (136 with professional labels, the rest model-generated), and the 2026 pairs appear in neither release. When we drop those 280 pairs, all eight meeting-input gains stay positive, seven intervals still exclude zero, and matched model scores move by at most 4.8 points (Appendix D). The 2026 meetings alone show the same direction for all eight models (Appendix I). Third, whether models saw the public transcripts in training cannot be verified, and §6 (L1) says so.

On pooling, we disclosed that the results for each part were known before the pooled test was defined, and the paper keeps that disclosure. The test itself is simply every pair from every 2024–2026 conference, chosen by date alone and never by label or model output. The plan for the pooled analysis was fixed before any pooled result was computed, no other date cut was scored under it, and each part is also reported on its own (Appendix D; Tables 14, 29 and 32).

W2 G–B and the annotation setting. The professionals saw each exchange and could open the full conference whenever they wanted (§3.2), so neither model input matches their setting exactly. You are right that we did not collect pair-only professional labels, and we list that as open. What we could control is the model side: keeping prompt, wrapper and output format fixed and adding only the conference raises binary Macro-F1 by at least 5 points for five of eight models, with intervals excluding zero, and lowers it for one (Table 5; Appendix G).

W3 Novelty and saturation. We agree that the professional reference is the heart of the contribution, and we built it from the ground up: we collected every FOMC press conference from April 2011 to October 2026 (96 conferences, 2,332 pairs, including 178 new pairs from 2026), curated the exchanges, and had three finance professionals label every pair. The benchmark evaluates 13 open-weight and closed models plus baselines against a professional scored on the same items, with the matched audit, date-based splits, fine-tuned classifiers and an executable scorer (Table 1). On the test set itself, 198 of the 680 pairs overlap the earlier corpus. On saturation, only one setting comes close: GPT-6.1 Sol, given the professionals' definitions and the whole conference, reaches 92.1 [89.5, 94.4], overlapping the professional's 94.1 [92.9, 95.3] (§5.2). The other closed models score 83.2–88.0, below the professional in every run; Sol scores 89.0 with the released prompt and the whole conference; and the best open-weight model reaches 77.6 even with the professionals' definitions. On Train, closed-model consensuses given the professionals' definitions still disagree about two to three times as often as a professional; on 2026 the order reverses, which §5.1 reports.

W4 Downstream benefit. We agree, and the paper says the same. The fine-tuning is exploratory, and we show every seed precisely because recall varies: professional labels raise fully evasive recall in each Qwen3-4B seed and raise three-class Macro-F1, binary scores are similar, and DeBERTa-v3 leans the other way (§5.1; Appendix K.1). We do not claim a general improvement.

W5 Class collapse. We agree that these two models are dominated by non-direct predictions, and that is why we report them: collapse is one of the paper's findings (§5.3; seven of 17 open-model runs with the released or native prompts put at least 90% of pairs in one class). With meeting input both stay above 89% non-direct; Mistral gains +10.2 [4.3, 16.0] and Apertus +2.0 [−1.0, 4.6], an interval that includes zero (Table 13). The claim itself says six of eight intervals exclude zero, and the largest gains come from Nemotron and GPT-OSS, whose non-direct shares fall to about 54%. We are happy to add this note beside the claim in the final version.

Thank you again for staying engaged with the paper; we would be glad to answer any further question during the discussion.

Comments on the paper

4 comments
Area_Chair_2Lx1OpenReview ↗

Discussion Period - No Response from Authors Yet

Dear Authors,

We have not yet received a response from you yet. Please respond as soon as possible so that the reviewers have a chance to read and respond to your rebuttal during the discussion period.

AC

AuthorsOpenReview ↗

To all reviewers and the area chair: thank you and a guide to the revision

Thank you to the area chair for the reminder, and to the reviewers for the time, care and expertise you gave the paper. We took every comment seriously, and together they led us to substantially revise the work. We replaced the test with a date-based split, added seven 2026 press conferences labelled after submission, ran the controls and fine-tuning tests you asked for, added four closed models and the professionals' own definitions, checked the results on European Central Bank (ECB) press conferences, and rewrote the manuscript. We believe FOMCBench, with three professional votes on every answer, gives the community a solid reference for judging whether an answer gives what was asked, and we remain committed to it and attentive to any further guidance from the reviewers and the area chair.

The resource behind the findings. FOMCBench is a professionally annotated benchmark covering 2011–2026. It contains 2,332 Q&A pairs from 96 FOMC press conferences. Three finance professionals labelled every pair, yielding 6,996 professional judgments. These votes support a matched human reference, training-label comparisons and controlled model evaluation.

A guide to rereading the paper A good route is the abstract and Figure 1 (p. 1), then the contribution and benchmark comparison in §1 and Table 1 (pp. 2–3). The following passages address the central review concerns:

  • Test independence: §3.3 and Table 2 (pp. 4–5) replace the model-selected splits with meeting-disjoint date splits. Seven 2026 conferences, labelled after submission, extend the test (Appendix I).
  • Context and training: §5.4 and Table 5 (p. 8) separate prompt/format, context and joint labelling; Appendix G dates the headline rule (general response, item 6). §5.1 and Table 4 (pp. 6–7) compare training on each label source. The encoder instead favours consensus labels; both results are retained (Appendix K.1).
  • Model and prompt coverage: §5.2, Figure 2 (p. 7) and Appendix H add the professionals' definitions and four closed models. Each guide-aligned closed condition and the consensus audit ran three times, under plans fixed before those requests.
  • Readability: §1 states the findings and contributions; §6 consolidates recurring limits; Figure 3 (p. 9) shows paired differences. The appendix guide is on p. 16.

What the revised evidence supports

  1. On training meetings, an LLM consensus disagrees with two agreeing professionals about three times as often as a held-out professional. Guide-aligned closed consensuses disagree about two to three times as often in each of three runs. Period results are reported in full (§5.1; Tables 15, 30).
  2. On matched Test-Held instances, with meeting input and professional definitions, prompted open-weight models reach at best 77.6 against the professional's 94.1. The strongest closed model, given the professionals' definitions, reaches 92.1, with an interval overlapping the professional's (§5.2; Figure 2; Table 31).
  3. Professional definitions add a median 16.5 points to open-weight single-exchange scores; the median meeting-input gain is larger under the released prompt (§§5.2, 5.4; Table 27).

Practical value and robustness (§5.5, p. 10). Table 3 (p. 6) now puts constant, random, TF-IDF, encoder and prompted models in one benchmark comparison. The separate native-classifier study raises Gemma's ordinary FOMC binary score from 71.0 to 80.6–82.2 across all selected seeds and tests transfer to 148 ECB pairs (Appendix K.2). Excluding whole meetings with documented source-text differences retains 621 test pairs: the headline orderings and all eight positive meeting-input point gains persist (§5.5; Appendix D).

The general response gives shared evidence; each reviewer reply answers the original points with paper locations. We explain the date split in place of a random pair split, and identify the topic/strategy breakdowns not performed (i7dr reply). The revised paper and supplement contain the results described here, with ECB evidence aggregate-only (Appendices J, K.2). The reviewed FOMC package will be made public on acceptance, as stated in the reproducibility statement. We would welcome any further questions during the discussion and will carry agreed clarifications into the final version.

AuthorsOpenReview ↗

General response: a date-based test and the requested checks

Thank you for identifying the test-design and context confounds and asking about training-label consequences. The revised evaluation replaces model-selected splits sharing training meetings (i7dr, oJve). Item 8 gives the new matched model-performance comparison. Performance: Macro-F1 (%), 95% whole-meeting intervals; the audit uses disagreement rates.

1. Meeting-date splits (§3.3; Table 2; Appendix I). Train holds the 65 conferences of 2011–2022, Dev the 8 of 2023 and Test-Held all 23 of 2024–2026 (680 pairs, 67 fully evasive). Meetings are disjoint; no model output or label entered the split. The seven 2026 conferences (178 pairs) were labelled after submission. Pair- and meeting-input results on each test part, and context-ablation results on 94 test pairs (Appendix G), were known before pooling. We also report every 2011–2025 pair (item 3). Score levels across versions use different test populations. Shared texts/binary references and all 6,400 overlapping original G/B predictions are unchanged; see supplement VERSION_CHECK.json.

2. Test-Held (§§5.2–5.4; Figures 2–3).

  • Released/native conditions in Figure 2: scored against the same two agreeing professionals as the held-out vote, the best open model, Eva-4B, reaches 72.1 [68.3, 75.8] binary, against the professional’s 94.1 [92.9, 95.3].
  • Meeting input raises binary scores for all eight open-weight models: +2.0 [−1.0, 4.6] (Apertus) to +24.4 [19.7, 28.7] (Nemotron); six intervals exclude zero.
  • Most binary errors in every open-weight released-prompt run call professionally judged direct answers non-direct.

3. Every 2011–2025 pair (Appendix F; Table 21). On all 2,154 pairs (prompted models; descriptive, including training meetings), meeting input raises all eight binary scores, every interval excluding zero (+3.8 to +24.9). Whole-meeting source-text exclusions preserve the headline orderings (Appendix D).

4. Matched label audit (§5.1; Tables 15, 30). A held-out professional and the three-LLM consensus are scored against the other two agreeing votes. On Train, the professional disagrees on 5.0% of binary judgments (192 of 3,861), the consensus on 16.4% (632 of 3,861). Planned three-closed-model consensuses given professional definitions disagree about two to three times as often in each of three runs (9.7–14.7% vs 5.0%). On 2026 with meeting input (post hoc), all four point rates reverse (4.2–8.0% vs 9.1%).

5. Training (exploratory; §5.1; Table 4; Appendix K). We fine-tuned Qwen3-4B-Instruct-2507 on the same Train pairs with professional or consensus labels (one recipe, three seeds), scored once on Test-Held. Professional labels recall more of the 67 fully evasive answers in every seed (49, 40, 28 vs 28, 13, 18) and give higher three-class Macro-F1; binary scores are similar. DeBERTa-v3 favours consensus on both targets. Separate native Gemma fine-tuning raises ordinary FOMC binary Macro-F1 from 71.0 to 80.6–82.2; all seeds and 148-pair ECB transfer appear in Appendix K.2.

6. Controlled context ablation (§5.4; Table 5; Appendix G). B0 and B1 keep prompt, wrapper and output format fixed: target exchange alone versus its whole conference. The headline rule was fixed after scoring 94 test pairs, before the full analysis. Context alone (B1−B0) raises five of eight Test-Held binary scores by at least 5 points with intervals excluding zero and lowers one (DeepSeek, −3.3 [−5.7, −0.9]). Summed over models, prompt/format contributes 7.4 points, context 62.4 and joint labelling 22.7 of the 92.5-point gain.

7. Second reference (§5.5; Appendix J; Table 33). On 4,490 ECB pairs (same three professionals; text-version bindings and validation rules in Appendix J), meeting input raises all eight open-weight binary scores, every interval excluding zero.

8. Professional definitions and closed models (§§5.2, 5.4; Tables 27, 29, 31). Under a plan fixed before these runs, we replaced Test-Held label definitions and added Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol and GPT-6.1 Terra. Professional definitions raise ordinary pair-input binary scores for seven of eight open-weight and all four closed models, with intervals excluding zero (median +16.5 and +11.4 points); every model calls fewer direct answers non-direct. With meeting input, the best prompted open-weight matched score is 77.6 [74.1, 80.7], below the professional’s interval. GPT-6.1 Sol reaches 92.1 [89.5, 94.4] (92.0–92.1 across three runs), overlapping the professional on Test-Held. By period: 90.6 vs 95.2 on 2024–2025; 96.2 vs 90.9 on 2026. Finding 2 states this comparison. With these definitions, the median open-weight meeting-input gain is smaller (three of eight intervals exclude zero; one closed model falls); claim 3 is scoped to the released prompt.

We hope the new test, context controls and training evidence now answer the central questions.

AuthorsOpenReview ↗

List of changes in the revised paper

We thank the reviewers for the feedback that guided these revisions. The changes below summarise our response.

  1. Splits. The model-selected test splits are removed. Splits are by meeting date: Train 2011–2022, Dev 2023, Test-Held 2024–2026 (680 pairs, 23 press conferences), including seven 2026 conferences labelled after submission (Sections 3.3, 5; Appendix I).
  2. Label audit. A held-out professional is now the matched baseline for the three-LLM consensus (Section 5.1), with a fine-tuning test of each label source and, under a plan fixed in advance, the same audit for consensuses of three closed models given the professionals' definitions, repeated in three runs. Section 5.1 also states the 2026 reversal with meeting input (post hoc; 4.2–8.0% vs 9.1%).
  3. Finding 2: matched comparison. New runs with the professionals' own label definitions in the prompt, and four closed API models, are reported in Section 5.2, Figure 2 and Appendix H, with three runs of each guide-aligned closed-model condition. Prompted open-weight models remain well below professionals; the strongest closed model, with the professionals' definitions, has an interval overlapping the professional's.
  4. Claim 3 scoped. The meeting-input result is stated for the released prompt; with the professionals' definitions the median open-weight gain is smaller (Section 5.4).
  5. Context ablation. A controlled ablation with pre-planned arms separates prompt and format, context and joint labelling (Section 5.4, Table 5; Appendix G, which also dates its headline rule).
  6. Robustness and generalisation (Section 5.5). The meeting-input effect is checked on each part of Test-Held, on the 2011–2023 meetings, on European Central Bank press conferences (aggregates only; source-text bindings and validation rules in Appendix J). Separately, the earnings-call classifier Eva-4B is evaluated on FOMC answers. A post hoc whole-meeting source-text exclusion retains 621 test pairs: all eight meeting-input point gains stay positive, and the audit and matched-human orderings persist (Appendix D).
  7. Presentation. Findings and contributions stated in the introduction, including the downstream label-source test; a table comparing the nearest benchmarks (Table 1: year, size, models evaluated, annotators, human score); context-comparison grids replaced by a paired-difference plot; a main benchmark table (Table 3) gives point scores and model coverage, including OpenAI Decisions with its distinct request format and native 4B/31B before/after professional-label adaptation; limitations on exposure, class/chair imbalance and evaluation design paired with completed checks; baselines (constant, random, TF-IDF, DeBERTa) and the earnings-call classifier Eva-4B included. The matched encoder study favours consensus training and is retained alongside the positive Qwen result (Section 5.1; Appendix K.1).
  8. Abstract, introduction, Figure 1 and conclusion rewritten to match the above, in plainer language: the introduction now states why responsiveness at the FOMC matters, and the conclusion adds future work (shared annotation standards, broader tests across central banks and earnings calls, a test set that new conferences can extend, learning from the three votes, tone and facial expression).
  9. Native-classifier development. Qwen3-4B and Gemma 4 31B use Train/Dev development with every selected seed, both Dev metrics and loss curves reported. Native Gemma’s ordinary binary FOMC score rises from 71.0 to 80.6–82.2; a fixed 148-pair ECB subset tests transfer, with rare-class uncertainty retained (Table 3; Appendix K.2).
  10. Reference construction and version consistency. The shared texts and binary references are unchanged. The four three-class no-majority cases are excluded from that scoring; model results are rescored on the new date splits. The original and revised tests overlap partially and use different populations.
  11. Supplement. All 6,996 professional votes, the annotation guide, date-based splits, the FOMC model outputs used for scoring (including the new runs), prompts, the scorer and numerical verification; ECB outputs remain aggregate-only (Appendices J, K.2).

The paper and supplementary documents now distinguish the shared LLM prompts from Eva's native prompt, and adaptive from default reasoning. The supplement describes the construction pipeline separately from the later pooled-test design and documents exact overlap with the prior FOMC corpus; overlap counts are unchanged.

The revised paper contains the analyses listed here. We will carry agreed presentation clarifications into the final version if accepted. Changes are listed by section rather than marked in colour in the PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.