acceptodds
Under review as a conference paper at ICLR 2027

HAMLET: Hallucination Assessment in LLMs for Multi-party, Long-horizon, Evolving meeting Threads

Abstract

Existing meeting benchmarks primarily evaluate models on a single meeting, treating each meeting as a self-contained document. This misses a key property of real-world meetings: facts and decisions evolve across a series of meetings for one project, where a proposal may be introduced, revised, and eventually adopted or remain unresolved. Hallucinations in this setting can mislead users and carry incorrect information into subsequent meetings, yet existing benchmarks provide limited analysis of such hallucination behaviour. We address these gaps with HAMLET (Hallucination Assessment in LLMs for Multi-party, Long-horizon, Evolving meeting Threads), which treats evolving discussion threads as the unit of evaluation. HAMLET extracts threads from AMI and ICSI meeting series and generates deterministic probes from their evolving states. Candidate answers are human-verified against the full series transcript, and a two-stage judge evaluates both reference-level correctness and unsupported content against the transcript. We further introduce HAMLET-Aug, which doubles thread length with synthetic on-topic discussion while preserving the thread state. Across 13 models, even frontier models find this setting challenging: the best model achieves 64.0% correctness. We find models can often track discussions to a resolution but tend to over-commit when no resolution exists, while many hallucinated details appear beyond the reference answer. On HAMLET-Aug, doubling thread length shifts hallucination towards additional content. These findings highlight the importance of measuring hallucination in evolving multi-meeting discussions.

Primary area:
datasets and benchmarks
Keywords:
LLM hallucination, hallucination evaluation, long context, meeting assistant

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tkt1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HAMLET evaluates factual consistency across evolving meeting discussions. It constructs 175 human-verified probes from seven meeting series, covering retrieval, absence, contradiction, and ambiguity. A two-stage judge checks reference coverage and additional claims against transcript evidence. Experiments evaluate 13 models, with 104 probes reused on synthetically extended discussions in HAMLET-Aug.

Critical strengths
  • The evaluation unit is the evolving thread, not a single meeting. That exposes failures which span-level retrieval benchmarks miss: a model can retrieve an outdated proposal and still look extractive.
  • The Contradiction vs Ambiguity contrast is clean: similar threads and question forms, differing mainly in whether a decision exists. The finding that models over-commit on unresolved threads is a real diagnostic result.
  • The human verification against full series transcripts, with evidence citations and double annotation, is a serious annotation effort.
  • The audit of prior meeting benchmarks is useful: many items labeled unanswerable are actually answerable. That motivates HAMLET rather than just adding another leaderboard.
  • The two-stage judge is a sensible design: separate reference coverage from transcript grounding, so extra content is not ignored.
  • HAMLET-Aug reuses the same probes and preserves the thread state, which makes the length comparison paired rather than cross-dataset.
  • The failure-mode study goes beyond aggregate scores and shows where the errors live (decision status, timing, speaker attribution).
Critical weaknesses
  • Stage 2 is described as transcript grounding, but the Appendix K prompt gives the judge only a transcript excerpt, and the excerpt construction is not specified. Since halluc.2 counts only contradicted extra claims and explicitly treats uncovered claims as unverifiable, the reported rate is a lower bound on unsupported content, not a measure of it. The paper should report the supported/contradicted/unverifiable proportions and audit a sample against the full transcript.
  • halluc.2 is answer-level, so it is confounded with answer length and claim count. A longer answer has more chances to contain one contradicted claim. The HAMLET-Aug result ("longer context shifts hallucination into extra content") could partly be a verbosity effect. Report claim-level rates and control for claim count.
  • Thread extraction and move labels are produced by GPT-5.4 and, as far as I can tell, not human-audited; the reported agreement is for answerability and verdict, not for the thread structure. If the extracted state is wrong, the probes and gold answers inherit the error. The "deterministic" claim applies to template filling, not to the extraction pipeline.
  • Absence and Ambiguity probes appear thread-scoped, but the model sees the full series. A true fact from another thread can be counted as a hallucination, and the templates do not make the scope explicit. Please clarify the scope or evaluate both.
  • The judge validation is limited: 50 GPT-5.2 answers, sampled by the judge's own labels (25/25), with the human seeing the judge's decision and rationale. The 94%/90% is therefore a balanced, anchored agreement rate, not an independent error rate; there is no validation on HAMLET-Aug or on other model families. A blind, stratified sample with confidence intervals is needed.
  • Table 1 has no confidence intervals or run-to-run variance, and the 175 probes are clustered in seven series. Temperature 0 is not a guarantee of determinism for proprietary models, so differences such as 64.0 vs 62.3 may be noise. Report series-clustered uncertainty and repeated runs.
  • The open models are extended to 128K with RoPE scaling while frontier models use native context. That mixes model quality with context handling, so the leaderboard is not apples-to-apples.
  • HAMLET-Aug inserts GPT-5.4-generated synthetic dialogue and validates it with a GPT-5.4 judge; with 19 threads and 104 probes, the length effect may be a synthetic-style or leakage effect rather than a pure context-length effect.
Questions for authors
  1. How is the Stage 2 excerpt constructed, and how much of the full series does it cover? Please report supported/contradicted/unverifiable proportions and claim-level hallucination rates, and say whether the HAMLET-Aug shift survives when answer length/claim count is matched.

  2. Were thread extraction and move labels human-audited? Are Absence and Ambiguity probes thread-scoped or series-scoped, and how many "hallucinations" are true facts from another thread in the same series?

  3. Can the judge be validated on a blind, stratified sample across model families and on HAMLET-Aug, with confidence intervals? Does the ranking change?

  4. Please report series-clustered uncertainty and run-to-run variance, native-context vs RoPE-scaled results, exact model versions, and human-verified state-preservation/leakage checks for HAMLET-Aug. Does the length effect survive synthetic-artifact controls?

6 comments

We thank the reviewer for the thoughtful and detailed feedback, and for the very careful reading of the evaluation protocol. We address each concern below.

1. Stage-2 excerpt and the proportions of claim verdicts (W1, Q1)

In our paper, halluc.2_2 is designed to check hallucination with respect to the discussion thread that the probe asks about, not the whole series. As the title "Stage 2: transcript evidence grounding" in §3 indicates, Stage 2 checks the extra content against the relevant evidence in the transcript.

(a) Excerpt construction. The excerpt contains the utterances that annotators cited for the probe, each with 30 utterances of context on either side, and overlapping windows are merged. It covers a median of 6.4% of the lines in a series. We have described this in Appendix J and revised the main text to "against human-cited transcript evidence."

(b) Proportions. Over all HAMLET answers of the 13 models, 54.9% of the extra claims are supported, 39.2% are unverifiable, and 5.9% are contradicted. We add Table 18 listing the proportions for every configuration on both datasets.

(c) Audit against the full transcript. On 50 probes × 3 models (GPT-5.2, DeepSeek-V4-Flash, Qwen2.5-32B), we re-ran Stage 2 with the full series. Of the 190 excerpt-unverifiable claims, 92% become supported and 1% become contradicted. Human annotators checked a stratified sample of these claims: an estimated 82.9% [75.1, 90.3] are supported and 2.4% contradicted, and the full-series judge agrees with the annotators on 88.7%.

We agree that halluc.2_2 should be read as a conservative rate of contradictions with the human-cited evidence, not as a measure of all unsupported content. We state this in §3 and the Limitations. We keep the excerpt for the main metric because it covers a median of 6.4% of the series lines, so Stage 2 reads about 15× less input, and switching to the full series leaves the Overall-correct ranking unchanged. We additionally report the full-series judge results in Appendix J as a reference.

2. Answer length and claim count (W2, Q1)

We added per-claim halluc.2_2 (Appendix J.3, Table 19). The per-claim rate does not grow with the number of claims, and it ranks the models consistently with the answer-level rate (τ = .79 on HAMLET, .92 on HAMLET-Aug). GPT-5.1 gives the longest answers, and its high answer-level rate mainly reflects answer length.

HAMLET-Aug shift under length matching. We also checked whether the higher halluc.2_2 on HAMLET-Aug is a verbosity effect.

  • First, HAMLET-Aug answers are barely longer than HAMLET answers (3.61 vs. 3.67 extra claims per answer).
  • Second, we paired each model's answers to the same probe on HAMLET and HAMLET-Aug and kept only the pairs whose claim counts differ by at most one (698 of 1,352 pairs; 2.95 claims per answer on both sides). On these pairs, halluc.2_2 rises from 13.8% on HAMLET to 16.3% on HAMLET-Aug (+2.6 pp), close to the rise over all pairs (14.8% to 17.2%, +2.4 pp), so answer length does not explain the shift. Because the shift is modest, we have revised Finding 3 to "longer threads change where hallucination appears," without attributing it to context length alone.

3. Human audit of thread extraction (W3, Q2)

We previously did not report a human audit of the threads because threads are an intermediate product used to generate questions. The gold answers do not inherit extraction errors: annotators verify every probe and its answer against the full series transcript (not extracted threads) and rewrite incorrect candidate answers. Before running the full extraction, we ran a small pilot study and refined the extraction prompt and strategy based on it, and after extraction we reviewed the resulting threads before generating questions.

We added a audit section with quantitative results (Appendix D.1):

  • Move labels: on 40 events stratified by move, GPT-5.4 agrees with human labels on 37 (κ = .91). The three disagreements all assign a stronger move, e.g., DECISION for SUPPORT (Table 7).
  • Thread state: agrees for 7 of 10 threads. In the other three, the pipeline marks the thread as open while annotators judge that it reached a decision.
  • Effect on scores: removing all probes of any single thread changes no model's Overall-correct by more than 2.3 pp, and the ranking agrees with Kendall's τ ≥ .968.

We have also clarified in §1 that "deterministic" refers only to "deterministic template instantiation over these extracted threads." The threads are used only to construct the questions, and every resulting probe is verified by human annotators.

4. Thread scope vs. series scope (W4, Q2)

Annotators judge answerability and gold answers over the full series, and they confirmed that every Absence probe has no answer anywhere in the full series transcript (§2.2). A true fact from another thread falls into one of two cases:

  • It answers the question. Since annotation covers the full series, the annotators would have marked the probe answerable, included the fact in the gold answer, and cited its utterances as evidence. The fact is then part of the reference in Stage 1 and of the evidence in Stage 2, so it is supposed to be judged correct.
  • It does not answer the question. The fact is then extra content not directly answer the probe. It lies outside the cited evidence, so Stage 2 labels it unverifiable rather than contradicted, and it does not count toward halluc.2_2.

Therefore, none of the hallucinations are true facts from another thread: neither Stage 1 nor Stage 2 counts such a fact as a hallucination.

5. Judge validation (W5, Q3)

We agree that the original validation was balanced by the judge's own call and that the annotator saw the judge's decision. We now state this in §3, and we added two independent checks (Appendix I, Table 14):

(a) Blind, stratified annotation across model families and datasets. 40 answers (20 HAMLET, 20 HAMLET-Aug; five per category in each), covering five model families. Annotators labelled without seeing the judge's output or the model's identity.

Table R1: Agreement between blind human labels and the GPT-5.4 judge on overall correctness(Wilson 95% CI)

SetAgreementκ
All (40).850 [.709, .929].71
HAMLET (20).750 [.531, .888].52
HAMLET-Aug (20).950 [.764, .991].89

(b) Cross-family judge. We re-ran all answers with DeepSeek-V4-Flash as the judge. The two judges agree on Overall-correct for 89.5% of answers (κ = .79). The DeepSeek judge is more lenient overall, but the ranking barely changes (Kendall's τ = .974 on HAMLET, .897 on HAMLET-Aug; Figure 8). The GPT-5.4 judge doesn't show measurable preference for GPT answers (Δ\Delta = −1.4 pp, 95% CI [−3.6, 0.8]).

We also report traditional metrics (ROUGE, BERTScore) in Table 16. They are not sufficient to score individual answers, which is why we use an LLM judge.

6. Confidence intervals and run-to-run variance (W6, Q4)

We added Bootstrap CIs and repeated runs in Appendix K.1:

  • Bootstrap CIs and three runs per model (Table 20). Run-to-run sd of Overall-correct is 1–3 pp for most models.
  • Paired McNemar tests with Holm correction (Figure 9). The larger gaps hold when pooled over three runs, e.g., GPT-5 vs. GPT-5-mini +9.3 pp [4.7, 14.3] and GPT-5-mini vs. Qwen2.5-32B +14.7 pp [6.8, 24.9] (series-clustered CIs).
  • Series clustering. Probes from the same series may be correlated, so we also computed CIs that resample whole series instead of individual probes. These CIs are no wider than the probe-level ones (mean SE of Overall-correct 2.8 vs. 3.4 pp), so the series structure does not make the reported CIs overconfident. Table 20 reports the wider, probe-level CIs.

In the revised text, we base model comparisons on the differences that these tests support.

7. RoPE scaling for open models (W7, Q4)

We applied YaRN to every open model so that all systems can read the full input (refer to Qwen report https://arxiv.org/pdf/2412.15115). The leaderboard therefore compares each model in its best available configuration. To check whether YaRN itself is the main reason for the lower scores of the open models, we ran two additional experiments:

  • Native long context (Appendix H). Qwen3-30B-A3B-Instruct-2507 reads long inputs natively, without YaRN. It does not score higher than Qwen2.5-32B with YaRN; its Overall-correct is lower (McNemar p=.004p=.004).

Table R2: Open models with and without YaRN on HAMLET

ModelYaRNComplete ↑halluc.1_1 ↓halluc.2_2 ↓Overall-correct ↑
Qwen2.5-32Byes.411.474.126.326
Qwen3-30B-A3B-Instruct-2507no.394.549.309.206
  • Shorter inputs (Appendix K.2). Qwen2.5-72B with YaRN and Qwen3-30B-A3B without YaRN gain similarly with the oracle input (+13.7 and +12.6 pp on Overall-correct), so the gain is not specific to YaRN. Even with the oracle input, no open model exceeds .36, far below the best model on the full series (.640).

Exact model versions and context settings are listed in Table 11.

8. HAMLET-Aug: synthetic style and leakage (W8, Q4)

(a) Construction and verification. The generator is restricted in what it may insert: it is instructed not to introduce decisions, commitments, owners, or final values, and it is not shown the discussion after the insertion point, so it cannot deliberately leak later outcomes. Rule-based checks then confirm that inserted turns use only allowed moves, existing speakers, and ordered timestamps, and that every original utterance is unchanged (Appendix G). Most importantly, we re-annotated all 104 reused probes and candidate answers under the same human protocol as HAMLET, so the HAMLET-Aug gold labels are verified against the augmented transcripts. We understand that an insertion cannot be guaranteed never to change a correct answer, which is why we re-annotated rather than reused the HAMLET gold labels.

(b) Do the synthetic turns drive the effect? We took all answers, across the 13 models, that have no halluc.2_2 on HAMLET but do on HAMLET-Aug (167 answers, 193 contradicted claims). For each contradicted claim, we looked at the transcript line that the judge quoted as the contradiction. Only 7% of these lines are synthetic turns. 72% are original utterances that were also in the HAMLET excerpt, and most of these claims do not appear in the HAMLET answer. In other words, on HAMLET-Aug the models write new wrong claims, and original utterances refute them.

(c) Synthetic dialogue. Building conversation benchmarks with LLM-generated dialogue is common practice, both for meetings [1, 2] and for long multi-session conversations [3, 4]. In HAMLET-Aug, only the inserted segment is synthetic, every original utterance is unchanged, and the inserted turns are naturalized with hesitations and backchannels such as "um" and "mm-hmm" (Appendix G). Moreover, HAMLET probes are factual questions about specific events in a thread, so the answers depend on the content of the discussion rather than its style.

References

[1] Kirstein et al. You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with Multi-Agent Conversations. Findings of ACL, 2025.

[2] Golany et al. Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts. Findings of EMNLP, 2024.

[3] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL, 2024.

[4] Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR, 2025.


We thank the reviewer again for the detailed questions, which helped us make the evaluation protocol much more transparent! We hope these responses address the concerns, and we are happy to discuss any further questions.

Reviewer_4Da3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HAMLET, a benchmark that evaluates hallucination in multi-party meeting series rather than in isolated meetings. Discussion threads are extracted from AMI and ICSI series, annotated with typed moves, and converted into four categories of probes—retrieval, absence, contradiction, and ambiguity—that target different aspects of thread-state tracking. Candidate answers are verified against the series and scored by a two-stage judge that separates reference-level correctness from transcript grounding. The authors also construct HAMLET-Aug, which doubles the length of selected threads while preserving their trajectory, and evaluate thirteen models, none of which exceeds 64.0% overall correctness.

Critical strengths
  • The thread-level formulation is the paper's main conceptual contribution. By making the evolution of a proposal, rather than a single meeting, the unit of analysis, the benchmark can distinguish failures that single-meeting QA cannot, such as treating a superseded proposal as final. The contrast between contradiction and ambiguity probes is particularly well designed, since both involve a change of direction but only one terminates in a decision.
  • The annotation protocol is more rigorous than is common in benchmark papers. Two independent annotators label each probe with supporting utterances, disagreements are adjudicated, and Gwet's AC1 is reported for both verdict and answerability (0.786 and 0.878).
  • The audit of existing benchmarks is a useful secondary contribution. The finding that 58% of sampled MeetingQA items and 34% of sampled MISeD items labelled unanswerable are in fact answerable cautions against using those labels as a measure of abstention.
Critical weaknesses
  • The scale is insufficient for the model-level claims. HAMLET contains 175 probes from seven series, and the per-category cells are considerably smaller; several reported rates repeat exactly across models, which is consistent with cell sizes in the low tens. Table 1 reports no confidence intervals, yet the text draws conclusions from differences of 0.02–0.08. The absence of a human ceiling further complicates the interpretation of 64.0%.
  • The Stage-2 judge does not have access to the full series. It receives an evidence excerpt and is instructed that a claim not covered by the excerpt is "unverifiable, not contradicted" (Appendix K). Claims contradicted elsewhere in a 24.5k–106k-token series are therefore not detected, and halluc.2 is biased downward. This is also inconsistent with the main text's statement that answers are checked against the full transcript.
  • The abstention and length analyses are confounded. The answering prompt does not state that "not determined" is an acceptable response, so Absence performance reflects both over-commitment and ordinary instruction-following. In addition, extra content is present in 77–93% of answers and halluc.2 is computed only over that extra content, which entangles verbosity with hallucination. HAMLET-Aug changes length and content simultaneously, so the reported shift cannot be attributed to thread length alone.
Questions for authors
  • Please report per-category probe counts and bootstrap confidence intervals, and indicate whether the GPT-family ordering survives paired tests.
  • How large is halluc.2 when Stage 2 is given the full series rather than the evidence excerpt?
  • Does an explicit abstention instruction change Absence accuracy, and what is human accuracy on a stratified sample?
3 comments

We thank the reviewer for the thoughtful and detailed feedback, and for recognizing the thread-level formulation, the annotation protocol, and the benchmark audit. We address each concern below.

1. Scale, confidence intervals, paired tests, and human ceiling

(a) Per-category counts. HAMLET has 84 Retrieval, 19 Absence, 39 Contradiction, and 33 Ambiguity probes. HAMLET-Aug adds 104 probes, so 279 probes are human-verified in total. Absence has the smallest size of probes because we filter them strictly: of the 48 probes generated from Absence templates initially, annotators found 29 answerable after checking the full series, and they are not used in Absence category.

(b) Bootstrap CIs and paired tests (Appendix K.1). Table 20 adds 95% bootstrap CIs (about ±6.6 pp on Overall-correct) and the mean ± sd over three runs. We also test every pair of models with a paired exact McNemar test with Holm correction (Figure 9).

(c) Does the GPT-family ordering survive? The main orderings keep the same. Pooled over three runs, GPT-5 is above GPT-5-mini (+9.3 pp, 95% CI [4.7, 14.3]) and GPT-5.6-Terra is above Luna (+6.1, [2.0, 10.4]). At the family level, the three GPT-5.6 models score about 9 pp above the GPT-5.x models (+9.3, [5.0, 13.5]), in the same direction in every run. We agree that small gaps between adjacent ranks should not be over-interpreted, and we have revised the text to draw conclusions at the family level rather than from adjacent ranks.

(d) Human ceiling. Human annotators answered a stratified sample of 20 probes (five per category) blind: they saw only the question and the full series transcript, with no candidate answer, reference answer, or cited evidence. The annotation interface provided a search tool and a notepad, and answering one probe still took about 30 minutes or more. We scored their answers with the same two-stage judge. The annotators reach 85% Overall-correct (17/20, 95% CI [64.0, 94.8]), while the best model reaches 65% on the same 20 probes.

2. Stage-2 judge and the full series

The phrase "full transcript" at the end of the first paragraph of §3 was a typo, and we have corrected it. halluc.2_2 is designed to check hallucination with respect to the discussion thread that the probe asks about, so Stage 2 compares extra claims only against the evidence cited by the annotators. We also added a Limitations paragraph and Appendix J on this design.

How large is halluc.2_2 with the full series? We re-ran Stage 2 with the full series on 50 probes × 3 models from different strength levels and families (GPT-5.2, DeepSeek-V4-Flash, Qwen2.5-32B).

Table R2: halluc.2_2 with the excerpt vs. the full series

Modelexcerptfull series
GPT-5.2.02.04
DeepSeek-V4-Flash.06.04
Qwen2.5-32B.12.18

Halluc.2_2 changes by a few points in both directions, the model ranking by Overall-correct is unchanged (τ = 1.0), and the halluc.2_2 ranking is largely preserved (τ = .82). Of the 190 claims that are unverifiable against the excerpt, 92% become supported and only 1% become contradicted under the full series. A human check of these claims gives a consistent picture: an estimated 82.9% [75.1, 90.3] are supported and 2.4% contradicted. Contradictions outside the excerpt therefore exist but are rare: they shift halluc.2_2 by only a few points and do not change the model ranking. In exchange, the excerpt covers a median of 6.4% of the series lines, so Stage 2 reads about 15× less input than a full-series judge.

3. Confounds in the abstention and length analyses

(a) Explicit abstention instruction (Appendix K.3). We ran GPT-5-mini and Qwen2.5-32B with the original answering prompt (P0) and with a prompt that adds "If the transcript does not contain the answer, say that the meeting does not contain the answer to this question" (P1; Table 22), which is also the MISeD prompt in our prompt-robustness study (Table 13). Only the system prompt changes.

Table R3: Effect of an explicit abstention instruction (Table 23)

ModelPromptAbsence correctOverall-correctAbstention rate on Retrieval
GPT-5-miniP0.105.4743.6%
P1.211.4639.5%
Qwen2.5-32BP0.526.3266.0%
P1.579.19435.7%

Absence correctness rises only slightly, but abstention on the answerable Retrieval probes also rises. The instruction thus makes the models abstain more often, not more accurately, and Overall-correct does not improve for GPT-5-mini and drops by 13 pp for Qwen2.5-32B. This may be a form of sycophancy: the models follow the hint in the prompt rather than checking whether the series contains the answer. This is why we keep the answering prompt minimal. The low Absence scores should not explained by the missing permission to abstain, and Absence measures whether a model can tell that the series contains no answer.

(b) Answer length influence. We agree that answer-level hallucination number can be affected by answer length: when the answer is longer, it's more likely it contains hallucination. We regard this as part of model behavior rather than a flaw of the metric. A user reads the whole answer, so a single contradicted claim makes the answer wrong however long it is, which is why we evaluates at the answer level. To study the effect further, we added a per-claim halluc.2_2, the share of extra claims that are contradicted (Appendix J.3, Table 19). We find the per-claim rate does not grow with the number of claims, and the model ranking agrees with the answer-level one (τ = .79 on HAMLET, .92 on HAMLET-Aug). The main change is GPT-5.1, which gives the longest answers of all models: its high answer-level rate mainly reflects answer length.

(c) HAMLET-Aug changes length and content. We agree, and we have revised Finding 3 to "longer threads change where hallucination appears," without attributing the shift to length alone. Two observations help separate the effects:

  • Answer length. HAMLET-Aug answers are barely longer (3.61 vs. 3.67 extra claims per answer), so the halluc.2_2 shift is not a verbosity effect.

  • Content. We traced every contradiction that appears only on HAMLET-Aug to the evidence the judge quoted: 72% quote original utterances that were already in the HAMLET excerpt, most of them claims that are new in the HAMLET-Aug answer, and only 7% quote synthetic turns. The shift therefore mainly reflects what the models write on the longer threads, not the inserted content itself.

  • Effect away from the insertion. The insertion also changes answers to probes whose evidence it does not touch. For these probes, the meeting that holds the evidence is the same in both versions, yet halluc.1_1 falls by 7.0 pp (p<.001p<.001). For probes whose evidence the insertion does touch, halluc.1_1 does not change (+0.3 pp; Figure 7).


We thank the reviewer again for the constructive suggestions, which helped us make the evaluation more precise! We hope these responses address the concerns, and we are happy to discuss any further questions.

AuthorsOpenReview ↗

A Gentle Reminder Before the Discussion Deadline

Dear reviewer 4Da3,

Thank you once more for your valuable comments and the time you have spent on our submission!

As the discussion period is nearing its end, we would like to gently follow up on our response. We hope the additional clarifications and experiments have addressed your concerns, and we would sincerely appreciate it if you could reconsider your assessment in light of our revised response. Any further feedback before the deadline would be very helpful to us.

Thank you again for your consideration!

Best regards,

Author of submission 34033

Reviewer_1Th6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HAMLET, a benchmark for evaluating hallucinations in evolving discussions across multiple meetings. It constructs human-verified probes from discussion threads in the AMI and ICSI corpora, covering retrieval, absence, contradiction, and ambiguity. A two-stage evaluation checks answers against both reference answers and transcript evidence. The paper also introduces HAMLET-Aug to study longer discussions. Experiments across 13 models highlight difficulties in recognizing unresolved information and avoiding hallucinated details beyond the reference answer.

Critical strengths
  1. The paper addresses a practically relevant limitation of existing meeting benchmarks by examining how models track proposals, revisions, and decisions across multiple meetings.

  2. The paper contributes a human-verified benchmark for evaluating hallucinations in evolving multi-meeting discussions, with four probe categories and a two-stage evaluation protocol that provides a detailed breakdown of answer errors.

Critical weaknesses
  1. It remains unclear precisely how the benchmark operationalizes cross-meeting state tracking. The observed failures could equally stem from the difficulty of long-context retrieval, incomplete evidence integration, or the model's tendency to answer rather than abstain. You should add some controlled comparisons, for instance providing relevant evidence to the model or varying context length while preserving the required state transitions. It would help better isolate the challenges that are genuinely distinctive to this setting.

  2. The evaluation framework does not fully address the unsupported content that the paper places particular emphasis on. In Stage 2, only additional claims that directly contradict the transcript evidence are counted as hallucinations, whereas claims that cannot be verified are not penalized. As a result, an answer judged "Overall-correct" may still contain assertions lacking verified support. I would recommend separating unverifiable claims for dedicated analysis and explicitly clarifying how their treatment affects the interpretation of the headline metrics.

  3. The scale of the evaluation somewhat limits the strength of several conclusions. The test set comprises only 175 probes drawn from seven meeting series, of which a mere 19 are Absence probes. This provides limited evidence for broader interpretations such as greater capability not implying greater honesty, or helpfulness tuning causing over-answering. These interpretations would be more convincing with broader coverage and targeted controls; otherwise, they should be presented more cautiously as hypotheses.

  4. Some of the increase in halluc.2 on HAMLET-Aug may just reflect that the evidence itself changed, rather than the model becoming more prone to erroneous generation. Table 15 makes this worry concrete: a claim that basically stays the same flips from "unverifiable" to "contradicted" after augmentation. So before we conclude that models start generating more incorrect details in longer discussions, it would be better to separate these two effects, for example by scoring the same fixed answers against both transcript versions.

Questions for authors

See Critical Weaknesses.

4 comments

We thank the reviewer for the thoughtful and detailed feedback. We address each concern below.

1. Cross-meeting state tracking vs. retrieval, evidence integration, and abstention

How HAMLET operationalizes state tracking. A probe is designed to ask about one or several states of a thread, and the four categories separate what is required while tracking these states:

  • a Retrieval probe needs evidence found and combined in meetings;
  • a Contradiction probe needs the outcome after earlier proposals were revised or overturned;
  • an Ambiguity probe has a similar thread and question form but no decision;
  • an Absence probe asks for a detail the thread never establishes.

Answering therefore requires locating the relevant moves in a long series, integrating moves spread across meetings, and recognizing when no conclusion exists. These abilities are components of cross-meeting state tracking rather than alternative explanations of its failures, and HAMLET is designed to measure them together.

Still, for completeness, we followed the suggestion and ran controlled comparisons to see how much each component contributes (Appendix K.2 and K.3, summarized in §4 "What makes HAMLET hard?").

(a) Providing the relevant evidence. We tested three open models: Qwen2.5-32B and Qwen2.5-72B with YaRN RoPE scaling, as in the main experiments, plus Qwen3-30B-A3B-Instruct-2507, which reads the full series natively and so rules out any effect of YaRN. Each model received three inputs of decreasing length, all of which keep the state transitions needed to answer: the full series (full, 51.7K tokens on average), only the meetings that contain the cited evidence (meetings, 27.6K), and only the cited evidence with ±30 utterances of context (oracle, 6.5K), the same window that Stage 2 uses as evidence.

Table R1: Overall-correct under shorter inputs (change from full in pp; Table 21)

Modelfullmeetingsoracle
Qwen2.5-32B (YaRN).326.280 (−4.6).354 (+2.9)
Qwen2.5-72B (YaRN).223.320 (+9.7).360 (+13.7)
Qwen3-30B-A3B-Ins (native).206.263 (+5.7).331 (+12.6)

Qwen2.5-72B and Qwen3-30B-A3B improve with shorter inputs, by 13.7 and 12.6 pp with the oracle input, and their gains appear in all four categories (Figure 10). A long input is therefore part of the difficulty for some open models. Even with the oracle input, however, no model exceeds an Overall-correct of .36, far below the best model on the full series (.640), so most of the difficulty remains when the evidence is given. This again supports the motivation of HAMLET: knowing where the evidence is does not tell a model how the discussion evolved or whether it reached a conclusion.

(b) Evidence across meetings. Pooled over the 13 models, probes whose cited evidence spans several meetings are answered correctly about 11 pp less often than probes whose evidence lies in one meeting (95% CI [−19.6, −2.6] pp; Figure 11).

(c) Tendency to answer rather than abstain. We added an explicit abstention instruction to the answering prompt (Appendix K.3; details in point 3(c) below). Absence correctness rises only slightly, and Overall-correct does not improve. So the low Absence scores are not explained by a missing permission to abstain.

Together, these results suggest that the main difficulty lies in tracking how a discussion changes across meetings and whether it reached a conclusion, rather than only in input length or retrieval.

2. Unverifiable claims in Stage 2

We agree that unverifiable claims deserve dedicated analysis. We added Appendix J.

(a) Claim-level separate proportions. Pooling the extra claims in all HAMLET answers of the 13 models, 54.9% are supported by the evidence excerpt, 39.2% are unverifiable, and 5.9% are contradicted. Table 18 lists the three proportions, as a share of claims, for every configuration on HAMLET and HAMLET-Aug.

(b) What the unverifiable claims are. On a common sample (50 probes × 3 models from different families), we checked the 190 excerpt-unverifiable claims in two ways:

  • Full-series judge: re-running Stage 2 with the full series instead of the excerpt, 92% of them become supported and 1% become contradicted.
  • Human check: on a stratified sample, an estimated 82.9% are supported by the transcript and 2.4% are contradicted.

So most claims that the excerpt cannot verify are in fact supported. The excerpt covers a median of 6.4% of the series lines, so Stage 2 reads about 15× less input than a full-series judge (running Stage 2 on the full series for all configurations would take roughly 264M input tokens). During our test, switching to the full series leaves the model ranking by Overall-correct unchanged.

We have also clarified in §3 and in the Limitations that halluc.2_2 measures contradictions with the human-cited evidence.

3. Scale and Absence probes

(a) Scale. HAMLET has 175 probes and HAMLET-Aug has 104, i.e., 279 human-verified probes, each double-annotated against the full series and adjudicated. This is comparable to widely used evaluation sets that require careful annotation, such as HumanEval (164 problems) [1], GPQA-Diamond (198 questions) [2], and MT-Bench (80 questions) [3]. To make the reliability explicit, we now report bootstrap CIs, paired tests, and three runs per model (Appendix K.1, Table 20, Figure 9). The large gaps between model families are stable across runs, and the conclusions of the paper rest on these overall trends rather than on small gaps between adjacent ranks.

(b) Small Absence probe size. Absence probes are hard to construct reliably. Prior benchmarks show this: in our audit of 50 sampled unanswerable items each, 58% in MeetingQA and 34% in MISeD are in fact answerable from the transcript (Appendix B). In HAMLET, we construct 48 probes generated from Absence templates first, and then annotators found that 29 are answerable once the full series is checked, so we remove them from Absence category. This filtering is deliberately strict, and it trades size for label reliability. HAMLET has 19 Absence probes, and HAMLET-Aug has another 17 (re-annotated on the extended series), so 36 human-verified Absence probes in total. For comparison, LongMemEval [3] has 30 abstention questions. In our setting the probes themselves are easy to extend. Using the same pipeline, we generated 100 additional Absence probes from 155 threads not used in HAMLET (provided here: https://anonymous.4open.science/r/hamlet_rebuttal_data-11BB/), and we release them after human filtering, together with the construction code, the two-stage judge, the rubric, and our manual audit of the MeetingQA and MISeD unanswerable labels.

Pooled over the eight GPT and DeepSeek models, correctness is .694 on the 29 answerable probes and .191 on the 19 Absence probes (−50.3 pp, 95% CI [−66.4, −33.1], permutation p < .001), and the drop is significant for each of the eight models individually (Fisher p < .01). The 19 probes therefore separate answering when an answer exists from answering when none exists.

(c) Abstention prompt (Appendix K.3). We ran GPT-5-mini and Qwen2.5-32B with the original answering prompt (P0) and with a prompt that explicitly allows abstention (P1: "If the transcript does not contain the answer, say that the meeting does not contain the answer to this question"; Table 22). Only the system prompt changes.

Table R2: Effect of an explicit abstention instruction (Table 23)

ModelPromptAbsence correctOverall-correctAbstention rate on Retrieval
GPT-5-miniP0.105.4743.6%
P1.211.4639.5%
Qwen2.5-32BP0.526.3266.0%
P1.579.19435.7%

Absence correctness rises only slightly, while both models abstain much more on answerable Retrieval probes, so Overall-correct does not improve for GPT-5-mini and drops by 13 pp for Qwen2.5-32B. An explicit abstention instruction may steers models toward abstaining and hurts their answers on the other categories. Whether to abstain should be the model's own judgment after reading the series, and this judgment is what Absence measures.

4. Halluc.2_2 on HAMLET-Aug: evidence change vs. answer change

We thank the reviewer for this observation and followed the suggestion directly. The question is whether the higher halluc.2_2 on HAMLET-Aug reflects a change in the evaluation (different transcript evidence, including synthetic turns) or a change in the model answers. For three models of different strength and families (GPT-5.6-Sol, GPT-5.2, Qwen2.5-32B), we took their answers on the 104 probes shared by both datasets and crossed answers and evidence in a 2×2 design: each set of answers (HAMLET or HAMLET-Aug) is judged by Stage 2 against each evidence excerpt (HAMLET or HAMLET-Aug), with the same prompt and judge (Table R3).

Table R3: halluc.2_2 (%) with answers and evidence crossed

HAMLET evidenceHAMLET-Aug evidence
HAMLET answers9.311.2
HAMLET-Aug answers14.411.9

This separates the two effects:

  • Answer effect (change the answers, hold the evidence fixed): +5.1 pp against the HAMLET evidence and +0.6 pp against the HAMLET-Aug evidence, +2.9 pp on average.
  • Evidence effect (change the transcript evidence, hold the answers fixed): +1.9 pp for the HAMLET answers and −2.6 pp for the HAMLET-Aug answers, −0.3 pp on average.

The increase comes from the answers, not from the evidence. Whichever evidence they are judged against, HAMLET-Aug answers have higher halluc.2_2 than the HAMLET answers, while swapping the evidence for the same answers barely moves it (−0.3 pp on average). The gap is largest on the 47 probes whose evidence the insertion touches: judged against the same HAMLET evidence, the HAMLET-Aug answers have 10.6 pp higher halluc.2_2 than the HAMLET answers (19.1% vs. 8.5%). The higher halluc.2_2 on HAMLET-Aug reflects what models write after reading the longer series, not an artifact of the evaluation. In Table 25 (revised version), this example shows that a claim can be judged differently when the evidence changes, which is exactly the evidence effect that the 2×2 design measures.

5. Revisions to the wording

  • Capability and helpfulness tuning (§4.1). We now write that "even the most capable models often give an answer when the meeting provides no evidence for one," and present helpfulness tuning as "one possible explanation."
  • Finding 3. We have revised it to "longer threads change where hallucination appears."

References

[1] Chen et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.

[2] Rein et al. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. COLM, 2024.

[3] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks Track, 2023.


We thank the reviewer again for helping us improve the evaluation and its interpretation! We hope these responses address the concerns, and we are happy to discuss any further questions.

AuthorsOpenReview ↗

A Gentle Reminder Before the Discussion Deadline

Dear reviewer 1Th6,

Thank you again for your time and thoughtful review of our paper!

As the discussion deadline is approaching, we wanted to send a gentle reminder. We hope our response, together with the additional clarifications, experiments and revisions in paper, has addressed your concerns. If so, we would be grateful if you could reconsider your assessment in light of the revised response.

We would also greatly appreciate any further feedback before the deadline, and we are happy to provide more clarification if needed.

Best regards,

Authors of paper 34033

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for the careful and constructive reviews!

We are encouraged that all reviewers find the thread-level formulation a practically relevant and conceptually clean contribution, the Contradiction vs. Ambiguity contrast well designed (4Da3, tkt1), the annotation protocol rigorous (4Da3, tkt1), and the audit of prior meeting benchmarks useful (4Da3, tkt1).

We would like to highlight the core gaps HAMLET fills:

  • Evolving Cross-Meeting Dynamics: Existing benchmarks treat meetings as isolated, static records, failing to test how facts are proposed, revised, or resolved over time. HAMLET shifts the evaluation unit to cross-meeting discussion threads (up to 106K tokens).
  • Noise-Free, Diagnostic Probes: Prior datasets rely on coarse factuality scores and noisy rule-based negative labels (our audit shows 58% of MeetingQA and 34% of MISeD "unanswerable" items are actually answerable). HAMLET introduces four targeted probe types (Retrieval, Absence, Contradiction, Ambiguity), with all 175 HAMLET and 104 HAMLET-Aug probes double-verified with utterance-level citations.
  • Two-Stage Attribution & Key Insights: By judging against both references and transcripts, we uncover why models fail. Across 13 models (best: 64.0% correctness), models frequently over-commit when discussions remain unresolved and fabricate extra details beyond the reference, and further analysis could be seen in paper.

We believe this would be a useful resource for the community to study hallucination and evolving multi-meeting discussions.

We have uploaded a revised PDF version. All revised and added text is highlighted in blue. The main changes are:

  • Limitations. A new section on the scale, the Stage-2 evidence scope, and the automatic thread extraction.
  • Thread extraction human audit (Appendix D.1). A human audit of move labels and thread states, and the effect of each thread on the scores.
  • Open-model context and model versions (Appendix H). A natively long-context open model compared with the YaRN-extended ones, and list the exact versions of all models.
  • Judge validation (Appendix I). A blind human annotation across five model families and both datasets, and a second judge from another model family.
  • Stage-2 evidence scope (Appendix J). Supported / unverifiable / contradicted proportions for all configurations, a full-series Stage-2 judge, a human check of unverifiable claims, and per-claim halluc.2_2.
  • Additional analyses (Appendix K).
    • Statistical reliability (K.1): bootstrap CIs, paired McNemar tests with Holm correction, three runs per model, leave-one-thread-out sensitivity, and a human reference point.
    • Controlled context ablation (K.2, §4 "What makes HAMLET hard?"): full series vs. evidence meetings vs. oracle evidence, and stratification by how many meetings the evidence spans.
    • Absence probes and abstention (K.3): an explicit abstention prompt for GPT-5-mini and Qwen2.5-32B, with abstention rates on answerable probes.
  • Wording and typos.: Fixed typos and polished phrasing across the paper, including an added paragraph in Section 4.

Detailed responses to each point are given in the replies to the individual reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.