- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HAMLET evaluates factual consistency across evolving meeting discussions. It constructs 175 human-verified probes from seven meeting series, covering retrieval, absence, contradiction, and ambiguity. A two-stage judge checks reference coverage and additional claims against transcript evidence. Experiments evaluate 13 models, with 104 probes reused on synthetically extended discussions in HAMLET-Aug.
Critical strengths
- The evaluation unit is the evolving thread, not a single meeting. That exposes failures which span-level retrieval benchmarks miss: a model can retrieve an outdated proposal and still look extractive.
- The Contradiction vs Ambiguity contrast is clean: similar threads and question forms, differing mainly in whether a decision exists. The finding that models over-commit on unresolved threads is a real diagnostic result.
- The human verification against full series transcripts, with evidence citations and double annotation, is a serious annotation effort.
- The audit of prior meeting benchmarks is useful: many items labeled unanswerable are actually answerable. That motivates HAMLET rather than just adding another leaderboard.
- The two-stage judge is a sensible design: separate reference coverage from transcript grounding, so extra content is not ignored.
- HAMLET-Aug reuses the same probes and preserves the thread state, which makes the length comparison paired rather than cross-dataset.
- The failure-mode study goes beyond aggregate scores and shows where the errors live (decision status, timing, speaker attribution).
Critical weaknesses
- Stage 2 is described as transcript grounding, but the Appendix K prompt gives the judge only a transcript excerpt, and the excerpt construction is not specified. Since halluc.2 counts only contradicted extra claims and explicitly treats uncovered claims as unverifiable, the reported rate is a lower bound on unsupported content, not a measure of it. The paper should report the supported/contradicted/unverifiable proportions and audit a sample against the full transcript.
- halluc.2 is answer-level, so it is confounded with answer length and claim count. A longer answer has more chances to contain one contradicted claim. The HAMLET-Aug result ("longer context shifts hallucination into extra content") could partly be a verbosity effect. Report claim-level rates and control for claim count.
- Thread extraction and move labels are produced by GPT-5.4 and, as far as I can tell, not human-audited; the reported agreement is for answerability and verdict, not for the thread structure. If the extracted state is wrong, the probes and gold answers inherit the error. The "deterministic" claim applies to template filling, not to the extraction pipeline.
- Absence and Ambiguity probes appear thread-scoped, but the model sees the full series. A true fact from another thread can be counted as a hallucination, and the templates do not make the scope explicit. Please clarify the scope or evaluate both.
- The judge validation is limited: 50 GPT-5.2 answers, sampled by the judge's own labels (25/25), with the human seeing the judge's decision and rationale. The 94%/90% is therefore a balanced, anchored agreement rate, not an independent error rate; there is no validation on HAMLET-Aug or on other model families. A blind, stratified sample with confidence intervals is needed.
- Table 1 has no confidence intervals or run-to-run variance, and the 175 probes are clustered in seven series. Temperature 0 is not a guarantee of determinism for proprietary models, so differences such as 64.0 vs 62.3 may be noise. Report series-clustered uncertainty and repeated runs.
- The open models are extended to 128K with RoPE scaling while frontier models use native context. That mixes model quality with context handling, so the leaderboard is not apples-to-apples.
- HAMLET-Aug inserts GPT-5.4-generated synthetic dialogue and validates it with a GPT-5.4 judge; with 19 threads and 104 probes, the length effect may be a synthetic-style or leakage effect rather than a pure context-length effect.
Questions for authors
-
How is the Stage 2 excerpt constructed, and how much of the full series does it cover? Please report supported/contradicted/unverifiable proportions and claim-level hallucination rates, and say whether the HAMLET-Aug shift survives when answer length/claim count is matched.
-
Were thread extraction and move labels human-audited? Are Absence and Ambiguity probes thread-scoped or series-scoped, and how many "hallucinations" are true facts from another thread in the same series?
-
Can the judge be validated on a blind, stratified sample across model families and on HAMLET-Aug, with confidence intervals? Does the ranking change?
-
Please report series-clustered uncertainty and run-to-run variance, native-context vs RoPE-scaled results, exact model versions, and human-verified state-preservation/leakage checks for HAMLET-Aug. Does the length effect survive synthetic-artifact controls?
6 comments
4. Thread scope vs. series scope (W4, Q2)
Annotators judge answerability and gold answers over the full series, and they confirmed that every Absence probe has no answer anywhere in the full series transcript (§2.2). A true fact from another thread falls into one of two cases:
- It answers the question. Since annotation covers the full series, the annotators would have marked the probe answerable, included the fact in the gold answer, and cited its utterances as evidence. The fact is then part of the reference in Stage 1 and of the evidence in Stage 2, so it is supposed to be judged correct.
- It does not answer the question. The fact is then extra content not directly answer the probe. It lies outside the cited evidence, so Stage 2 labels it unverifiable rather than contradicted, and it does not count toward halluc..
Therefore, none of the hallucinations are true facts from another thread: neither Stage 1 nor Stage 2 counts such a fact as a hallucination.
5. Judge validation (W5, Q3)
We agree that the original validation was balanced by the judge's own call and that the annotator saw the judge's decision. We now state this in §3, and we added two independent checks (Appendix I, Table 14):
(a) Blind, stratified annotation across model families and datasets. 40 answers (20 HAMLET, 20 HAMLET-Aug; five per category in each), covering five model families. Annotators labelled without seeing the judge's output or the model's identity.
Table R1: Agreement between blind human labels and the GPT-5.4 judge on overall correctness(Wilson 95% CI)
| Set | Agreement | κ |
|---|---|---|
| All (40) | .850 [.709, .929] | .71 |
| HAMLET (20) | .750 [.531, .888] | .52 |
| HAMLET-Aug (20) | .950 [.764, .991] | .89 |
(b) Cross-family judge. We re-ran all answers with DeepSeek-V4-Flash as the judge. The two judges agree on Overall-correct for 89.5% of answers (κ = .79). The DeepSeek judge is more lenient overall, but the ranking barely changes (Kendall's τ = .974 on HAMLET, .897 on HAMLET-Aug; Figure 8). The GPT-5.4 judge doesn't show measurable preference for GPT answers ( = −1.4 pp, 95% CI [−3.6, 0.8]).
We also report traditional metrics (ROUGE, BERTScore) in Table 16. They are not sufficient to score individual answers, which is why we use an LLM judge.
6. Confidence intervals and run-to-run variance (W6, Q4)
We added Bootstrap CIs and repeated runs in Appendix K.1:
- Bootstrap CIs and three runs per model (Table 20). Run-to-run sd of Overall-correct is 1–3 pp for most models.
- Paired McNemar tests with Holm correction (Figure 9). The larger gaps hold when pooled over three runs, e.g., GPT-5 vs. GPT-5-mini +9.3 pp [4.7, 14.3] and GPT-5-mini vs. Qwen2.5-32B +14.7 pp [6.8, 24.9] (series-clustered CIs).
- Series clustering. Probes from the same series may be correlated, so we also computed CIs that resample whole series instead of individual probes. These CIs are no wider than the probe-level ones (mean SE of Overall-correct 2.8 vs. 3.4 pp), so the series structure does not make the reported CIs overconfident. Table 20 reports the wider, probe-level CIs.
In the revised text, we base model comparisons on the differences that these tests support.
7. RoPE scaling for open models (W7, Q4)
We applied YaRN to every open model so that all systems can read the full input (refer to Qwen report https://arxiv.org/pdf/2412.15115). The leaderboard therefore compares each model in its best available configuration. To check whether YaRN itself is the main reason for the lower scores of the open models, we ran two additional experiments:
- Native long context (Appendix H). Qwen3-30B-A3B-Instruct-2507 reads long inputs natively, without YaRN. It does not score higher than Qwen2.5-32B with YaRN; its Overall-correct is lower (McNemar ).
Table R2: Open models with and without YaRN on HAMLET
| Model | YaRN | Complete ↑ | halluc. ↓ | halluc. ↓ | Overall-correct ↑ |
|---|---|---|---|---|---|
| Qwen2.5-32B | yes | .411 | .474 | .126 | .326 |
| Qwen3-30B-A3B-Instruct-2507 | no | .394 | .549 | .309 | .206 |
- Shorter inputs (Appendix K.2). Qwen2.5-72B with YaRN and Qwen3-30B-A3B without YaRN gain similarly with the oracle input (+13.7 and +12.6 pp on Overall-correct), so the gain is not specific to YaRN. Even with the oracle input, no open model exceeds .36, far below the best model on the full series (.640).
Exact model versions and context settings are listed in Table 11.
8. HAMLET-Aug: synthetic style and leakage (W8, Q4)
(a) Construction and verification. The generator is restricted in what it may insert: it is instructed not to introduce decisions, commitments, owners, or final values, and it is not shown the discussion after the insertion point, so it cannot deliberately leak later outcomes. Rule-based checks then confirm that inserted turns use only allowed moves, existing speakers, and ordered timestamps, and that every original utterance is unchanged (Appendix G). Most importantly, we re-annotated all 104 reused probes and candidate answers under the same human protocol as HAMLET, so the HAMLET-Aug gold labels are verified against the augmented transcripts. We understand that an insertion cannot be guaranteed never to change a correct answer, which is why we re-annotated rather than reused the HAMLET gold labels.
(b) Do the synthetic turns drive the effect? We took all answers, across the 13 models, that have no halluc. on HAMLET but do on HAMLET-Aug (167 answers, 193 contradicted claims). For each contradicted claim, we looked at the transcript line that the judge quoted as the contradiction. Only 7% of these lines are synthetic turns. 72% are original utterances that were also in the HAMLET excerpt, and most of these claims do not appear in the HAMLET answer. In other words, on HAMLET-Aug the models write new wrong claims, and original utterances refute them.
(c) Synthetic dialogue. Building conversation benchmarks with LLM-generated dialogue is common practice, both for meetings [1, 2] and for long multi-session conversations [3, 4]. In HAMLET-Aug, only the inserted segment is synthetic, every original utterance is unchanged, and the inserted turns are naturalized with hesitations and backchannels such as "um" and "mm-hmm" (Appendix G). Moreover, HAMLET probes are factual questions about specific events in a thread, so the answers depend on the content of the discussion rather than its style.
References
[1] Kirstein et al. You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with Multi-Agent Conversations. Findings of ACL, 2025.
[2] Golany et al. Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts. Findings of EMNLP, 2024.
[3] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL, 2024.
[4] Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR, 2025.
We thank the reviewer again for the detailed questions, which helped us make the evaluation protocol much more transparent! We hope these responses address the concerns, and we are happy to discuss any further questions.
We thank the reviewer for the thoughtful and detailed feedback, and for the very careful reading of the evaluation protocol. We address each concern below.
1. Stage-2 excerpt and the proportions of claim verdicts (W1, Q1)
In our paper, halluc.2 is designed to check hallucination with respect to the discussion thread that the probe asks about, not the whole series. As the title "Stage 2: transcript evidence grounding" in §3 indicates, Stage 2 checks the extra content against the relevant evidence in the transcript.
(a) Excerpt construction. The excerpt contains the utterances that annotators cited for the probe, each with 30 utterances of context on either side, and overlapping windows are merged. It covers a median of 6.4% of the lines in a series. We have described this in Appendix J and revised the main text to "against human-cited transcript evidence."
(b) Proportions. Over all HAMLET answers of the 13 models, 54.9% of the extra claims are supported, 39.2% are unverifiable, and 5.9% are contradicted. We add Table 18 listing the proportions for every configuration on both datasets.
(c) Audit against the full transcript. On 50 probes × 3 models (GPT-5.2, DeepSeek-V4-Flash, Qwen2.5-32B), we re-ran Stage 2 with the full series. Of the 190 excerpt-unverifiable claims, 92% become supported and 1% become contradicted. Human annotators checked a stratified sample of these claims: an estimated 82.9% [75.1, 90.3] are supported and 2.4% contradicted, and the full-series judge agrees with the annotators on 88.7%.
We agree that halluc.2 should be read as a conservative rate of contradictions with the human-cited evidence, not as a measure of all unsupported content. We state this in §3 and the Limitations. We keep the excerpt for the main metric because it covers a median of 6.4% of the series lines, so Stage 2 reads about 15× less input, and switching to the full series leaves the Overall-correct ranking unchanged. We additionally report the full-series judge results in Appendix J as a reference.
2. Answer length and claim count (W2, Q1)
We added per-claim halluc.2 (Appendix J.3, Table 19). The per-claim rate does not grow with the number of claims, and it ranks the models consistently with the answer-level rate (τ = .79 on HAMLET, .92 on HAMLET-Aug). GPT-5.1 gives the longest answers, and its high answer-level rate mainly reflects answer length.
HAMLET-Aug shift under length matching. We also checked whether the higher halluc.2 on HAMLET-Aug is a verbosity effect.
3. Human audit of thread extraction (W3, Q2)
We previously did not report a human audit of the threads because threads are an intermediate product used to generate questions. The gold answers do not inherit extraction errors: annotators verify every probe and its answer against the full series transcript (not extracted threads) and rewrite incorrect candidate answers. Before running the full extraction, we ran a small pilot study and refined the extraction prompt and strategy based on it, and after extraction we reviewed the resulting threads before generating questions.
We added a audit section with quantitative results (Appendix D.1):
We have also clarified in §1 that "deterministic" refers only to "deterministic template instantiation over these extracted threads." The threads are used only to construct the questions, and every resulting probe is verified by human annotators.