- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HALT introduces held-out-source evaluation to test whether hallucination detectors for forensic video reports transfer to unseen generating models, and examines reliance on source identity, report length, and the labeling mechanism. It builds on an existing corpus of 807 UCF-Crime videos and 19,361 reports from three generators and eight prompting strategies, adding splits, detector comparisons, reliability tiers, and diagnostic analyses. TF-IDF ANY AUC falls from 0.820 in standard evaluation to 0.566 under source holdout, with larger degradation for fabrication labels than for omission and distortion. Strict video-disjoint evaluation, human agreement, length associations, and judge-pair confounding provide further context. The main contribution is an evaluation and diagnostic framework, rather than a wholly new corpus or generation model.
Critical strengths
- The evaluation question matters to the community. When the same generators appear on both sides of a split, high scores do not establish transfer to an unseen generator. HALT combines source holdout, a source-base-rate control, and type-specific transfer analysis into a reusable diagnostic protocol. The fall in TF-IDF ANY AUC from 0.820 to 0.566 makes the concern concrete and testable. Its value does not depend on proposing a complex new model.
- The experiments and self-audit provide useful evidence. The crossed corpus enables comparisons across videos, generators, and prompting strategies; linear, reference-conditioned, and fine-tuned encoder detectors show a consistent pattern of cross-source degradation. Fabrication AUC drops by 0.104–0.191, versus 0.035–0.105 for omission/distortion, showing what aggregate scores can hide. The authors disclose human agreement, aggregation sensitivity, and judge-pair confounding, and release splits and detector implementations; the existing per-judge verdicts also enable reanalysis. These candid limitations and reusable resources strengthen the contribution.
Critical weaknesses
- Label validity and source–judge confounding limit the core interpretation. In Table 1, H1 panel–human κ is 0.150, while H5/H6 human–human κ is only 0.082/0.192. Higher panel–consensus agreement is calculated on a subset where the humans already agree and therefore does not establish reliability across all reports. MACRO-HQ excludes only H1 and retains H5/H6 despite weak human agreement. More fundamentally, preventing self-evaluation assigns a different judge pair to each generator; H1 pairwise κ values are 0.850, 0.415, and 0.148. Source holdout thus changes both report distributions and the label mechanism, preventing separation of factual errors, writing style, and judge preferences. The authors acknowledge this and appropriately qualify the result as degradation of “fabrication labels,” but the identifiability limitation remains. In particular, Section 4.1’s assertion that a shared bias leaves relative source comparisons unaffected needs evidence that the bias is source-invariant. This does not invalidate the observed degradation in label transfer, but it limits interpretation as a benchmark for factual video hallucination detection. Narrower claims, per-judge and aggregation analyses, and results excluding H1/H5/H6 are feasible revisions. Stronger factual claims require fixed external judges or a stratified sample independently annotated from the videos. This is a structural limitation of the labeling design; adding detectors alone cannot resolve it.
- Evidence for strict generalization and uncertainty remains incompletely reported. The main source-holdout experiment deliberately shares videos to hold the input fixed, but it does not directly represent deployment on both a new generator and new videos. Appendix D already provides strict video-disjoint corroboration: TF-IDF fabrication/omission/ANY AUC is 0.668/0.780/0.559. The concern is not absence of this check, but absence of complete results across detectors, types, folds, and training settings. Each video yields up to 24 correlated reports, yet the bootstrap resampling unit is unspecified. Independent report-level resampling could produce overly narrow intervals; the manuscript alone does not establish that this error actually occurred. Complete strict-split results, paired bootstrap intervals clustered by video, and variation across training seeds would establish the stability of the type differences and detector ranking. The cross-generation pilot has only 15 videos and 360 reports, with generators and judges changing together, so it offers exploratory corroboration rather than a standalone basis for broad cross-generation claims. Most issues here are addressable through reporting and reanalysis; the pilot’s representativeness requires a larger sample with controlled judges.
Questions for authors
- Labels and confounding: Can you report per-judge metrics and cross-source differences under AND/OR aggregation within matched video–prompt conditions, with sensitivity results excluding H1 alone versus H1/H5/H6? Do fixed external judges or independent video annotations support the ranking? If unavailable, will you explicitly limit the claims to transfer under this labeling pipeline and revise the assertion that relative comparisons are unaffected?
- Strict splits and statistics: Please report fold-level and type-level AUC, sample sizes, and class prevalence for all detectors with both source and video disjointness, and explain video grouping in training/validation/test. Is bootstrap resampling performed over reports or videos? Please provide paired difference intervals clustered by video, and clarify seeds, hyperparameter selection, and reference-text isolation across folds.
- The “70–73%” interpretation: (AUC_base − 0.5) / (AUC_tfidf − 0.5) is a descriptive ratio of above-chance AUC, not a causal contribution or explained-variance decomposition. Can you provide paired confidence intervals for the ratio, test sensitivity to fold/type aggregation, and adjust the “signal share” wording accordingly?
- Length association: On H1, length predicts panel and human labels at AUC 0.727 and 0.566. Does the difference persist after controlling for source, video, prompting strategy, and truncation? Can you report paired uncertainty on the same human-annotated sample? This determines whether the evidence supports a length association or a stronger attribution to judge length bias.
- Reproducibility and comparisons: Please complete model versions, epochs, seeds, validation selection, truncation rules, and tuning procedures for the strict splits. The existing external zero-shot Qwen detector is a useful control. Any added baseline should match report/reference inputs and available generation information; semantic entropy requiring multiple sampled generations is not directly comparable to a single-report detector. PR-AUC, calibration, and false-positive rates at fixed recall are useful future additions, not independent grounds for rejection. The cross-generation pilot should retain its exploratory framing; a larger sample with fixed judges is future work.
3 comments
Part 2 of 2
W1 / Q1 (continued)
- Where the asymmetry does not appear: The judges read at most 3,000 characters of each report. On the 6,712 reports short enough to be read in full, the asymmetry disappears (ratio 1.10 [0.65, 1.54]). This subset is 68% GPT, so it does not prove that truncation causes the asymmetry, but it is consistent with the reviewer's concern, and we report it.
- What we changed in the claims: We removed the Section 4.1 sentence saying a shared bias leaves comparisons between sources unaffected. The main result is now stated for this labelling pipeline, with the same-judge ratios reported beside the original.
W2 / Q2: Strict splits and statistics
"Is bootstrap resampling performed over reports or videos? Please provide paired difference intervals clustered by video, and clarify seeds, hyperparameter selection, and reference-text isolation."
- Bootstrap: The submission resampled reports, so the reviewer's concern was correct. All intervals now resample whole videos, with the same resamples for every comparison. The main gap becomes −0.134 [−0.159, −0.110] (submission: [−0.148, −0.121]).
- Strict splits, all detectors (Appendix D): Here training and test share neither a source LLM nor a video. We train on the other two sources' training videos (about 9,000 reports) and test on the held-out source's test videos (about 1,015 reports from 127 videos). The ratio is 2.33 [1.97, 2.79] for TF-IDF, 1.75 [1.45, 2.14] for DeBERTa (1.70 across three seeds, SD 0.05) and 1.69 [1.45, 2.00] for ModernBERT. Appendix D tabulates AUC and prevalence for every held-out source and type.
- Seeds and settings (Appendices D and K): Across three seeds, DeBERTa's fabrication AUC on the standard split is 0.879 (SD 0.003). The encoders use one fixed training schedule, with no model selection on validation data and no tuning on any test fold. A second ModernBERT seed showed that one failed fold in the first run was a one-off.
- Video grouping: The standard split puts 563 videos in training, 117 in validation and 127 in test, with all 24 reports of a video in the same part. Five other random video splits change the results by an SD of at most 0.017.
- Reference-text isolation (Appendix K): Each expert reference belongs to one video. Under the standard and strict splits, no test reference is seen in training. Under held-out source, the video and its reference are intentionally the same and only the source LLM changes; the strict split removes even that overlap, and the asymmetry remains.
- TF-IDF vocabulary (Appendix I): The vocabulary is built from all report text, without labels. Building it from training text only leaves every result pointing the same way (ratio 3.24).
Q3: The "70 to 73%" interpretation
"A descriptive ratio of above-chance AUC, not a causal contribution or explained-variance decomposition."
Agreed. It is a descriptive ratio, not a causal share. We renamed it the base-rate fraction, removed "signal share", and report all six types with intervals: H1 73% [68, 77], H5 70% [65, 76], H6 45% [40, 51], H2 35% [26, 43], H4 31% [24, 37] and H3 13% [7, 19].
It is also stable (Appendix I). Across six different video splits it changes by an SD of at most 0.043. Computed per error group instead of per type, it is 63% [59, 67] for fabrication, 33% [27, 38] for distortion and 13% [7, 18] for omission, the same order.
Q4: Length association
"Does the difference persist after controlling for source, video, prompting strategy, and truncation?"
Partly, and not once the video is controlled. On the released labels, report length predicts the panel's scene-fabrication label with AUC 0.651 and the human label with 0.566 (130 reports from 62 videos). The difference is 0.085 [−0.13, 0.30] as tabulated, and 0.007 within each source LLM. Controlling for source, prompting strategy and truncation, it is 0.264 [0.03, 0.48]. Severity minimization shows a difference both before (0.178 [0.03, 0.33]) and after (0.156 [0.04, 0.31]) these controls.
When we also control for video, comparing only reports of the same video, the difference disappears: 0.105 [−0.11, 0.31] for scene fabrication and 0.066 [−0.09, 0.24] for severity minimization. So the link with length runs between videos, not within one. Section 4.6 now reports a length association, not a judge bias.
Q5: Reproducibility and comparisons
Appendix K lists the model versions, epochs, seeds, model selection, truncation and tuning for every detector. We keep the zero-shot Qwen detector as the outside control and add no baseline that needs inputs this task lacks. PR-AUC, Brier score, calibration error (ECE) and the false-positive rate at 80% recall are in Appendix I. The cross-generation pilot is still labelled exploratory and is no longer in the abstract.
Thank you again. In a final update to the revised PDF:
- Strict splits for all detectors (Appendix D, Table 11): we added the remaining linear detectors. The asymmetry holds for each, with every ratio interval above one: surface 1.40 [1.16, 1.66], surface-noid 1.34 [1.04, 1.67], grounded 1.77 [1.29, 2.47]. The appendix also explains the three configurations not in the strict tables (the prompted detector is never trained, so a strict split changes only its test set; the embedding features are not part of the released corpus; ModernBERT at 2,048 tokens was not rerun with saved predictions).
- A correction: running these showed that the "surface" row of the held-out-source feature table reported one fold rather than the three-fold mean. It is corrected (0.616 / 0.507 / 0.546), with no effect on any conclusion.
- Independent video annotations: the limitations now state explicitly that no new labels were collected from the videos. We answer this question with the fixed external judge, the existing 130 human-labelled reports and RAGTruth's human labels.
Part 1 of 2
We thank the reviewer for an exceptionally careful review. We agree with the central point. Because no model judges itself, holding out a source also changes who labels it, so the result must be tested with the labelling held fixed. We ran those tests, and where the effect shrank, we narrowed our claims. (The two terms we use below, the fabrication asymmetry ratio and the base-rate control, are defined in our general comment.)
W1 / Q1: Labels and source-judge confounding
(continued in Part 2)