- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ThreadNIAH, a video needle-in-a-haystack benchmark for long-video understanding that evaluates locating, linking, and combining evidence. It organizes five task types into three levels, instantiates them through 29 templates over synthesized videos of 40 to 130 minutes built from 199 source videos, and pairs each main question with diagnostic sub-questions forming a QA chain, scored by Main Acc, Sub Acc, and All Pass Rate. The benchmark contains 870 chains with 3,450 questions.
Critical strengths
- The QA chain design is a genuine addition to video NIAH evaluation: probing intermediate evidence and operations separately exposes failures that final-answer accuracy hides, and the prefix-survival analysis makes the chain view quantitative.
- The evaluation covers current frontier and open-weight models with documented inference settings, and the generator model is excluded from the baselines to avoid self-bias.
Critical weaknesses
- Ground truth is generated by Gemini 3.1 Pro and quality-controlled by whether GPT-5.2 can answer from needle frames. This biases the pool toward questions one model family can solve from sparse frames, and residual label errors compound in APR, which requires every question in a chain to be correct. The human audit pass rate of about 94 percent implies a per-question error level that materially deflates APR.
- The evidence that ThreadNIAH difficulty transfers to natural videos is weak. The task-type correlation of 0.70 has a confidence interval of 0.10 to 0.90 over only five points, and the template-level correlation of 0.569 has a lower bound of 0.111. The abstract's claim of similar difficulty patterns on real-world videos rests on these two unstable estimates.
Questions for authors
- What sampling settings were used for GLM-5.3-Flash, and can the two MiMo variants be reported separately in the thinking row?
- Could you human-verify a random subset of chains and report how the residual label error rate shifts APR? A per-question error level near the human-audit failure rate would materially deflate the chain metrics.
3 comments
Response to reviewer 5RQx (part 2)
W2. Substantiate the claimed difficulty transfer with stronger evidence, as the current correlations are based on unstable, wide-confidence-interval estimates.
We acknowledge that the limited data points (5 task types; 5 questions per template) result in wide confidence intervals. However, we would like to emphasize two points:
(1) The direction of the positive difficulty correlation is stable.
The lower bounds of both confidence intervals (0.10 and 0.111) are greater than 0, indicating that the difficulty under the two settings is positively correlated. Although the exact correlation magnitude is imprecise, the direction of this relationship is robust.
(2) Failure-mode analysis provides evidence independent of the correlation coefficients.
Beyond the correlation values, we compared the failure modes of each model under the two settings individually—a comparison that does not rely on correlation coefficients. On natural videos, 6 out of 7 models achieve their highest accuracy on Details; once cross-segment association or evidence integration is required, accuracy drops markedly. For example, Gemini-3.6-flash drops from 70.0% on Details to 26.7% on Counting, and Qwen3.5-27B from 52.5% to 20.0% (Table 6). On ThreadNIAH, 6 out of the same 7 models also achieve their highest accuracy on Details (Table 2). This indicates that both settings expose the same failure pattern: models can localize a single piece of evidence, but struggle to associate and integrate multiple pieces of evidence across clips.
Response to reviewer 5RQx (part 3)
W3. What sampling settings were used for GLM-5.3-Flash, and can the two MiMo variants be reported separately in the thinking row?
GLM-5.3-Flash's official API does not expose a video frame-rate parameter, nor does it publish a default sampling rate. We therefore submit the raw videos directly without setting an FPS, and the actual frame sampling is determined server-side (Appendix C). The standard setting uses low reasoning effort, while the thinking setting uses high reasoning effort.
For MiMo V2.6 Pro, the thinking results merge responses from the standard endpoint and the ultraspeed endpoint of the same model. Ultraspeed is an officially provided acceleration tier backed by the same underlying model, and both endpoints share identical reasoning settings (1 fps, thinking enabled, JSON Object output). Therefore, they are not two distinct model variants, and we report them jointly.
Response to reviewer 5RQx (part 1)
Dear Reviewer 5RQx, thank you for your time and thoughtful review. Below, we address your concerns.
W1. Human-verify a random subset of chains and quantify how residual label errors (implied by the ~94% audit pass rate) deflate the strict APR metric.
We thank the reviewer for raising this point, but we would like to clarify a key distinction regarding the reported human audit rate. The 94.4% in Appendix B is a per-judgment pass rate (1,086 out of 1,150 individual judgments passed), not a per-question error rate. When counted by question, only 10 out of 575 questions (1.7%) were jointly flagged as failing by both annotators.
To directly quantify the impact of residual label errors on APR, we recomputed APR on these 145 audited chains, retaining only chains in which every question was judged as passing by both annotators, and compared this with the un-pruned results. Because the reasons for failing the audit extend beyond label errors, this comparison overestimates the impact of label errors and thus serves as an upper-bound estimate. Note that all numbers below are computed on these 145 chains and are therefore not directly comparable to the 870-chain results in the main text. As shown, the changes in APR are minimal: the largest improvement is 6.6 percentage points, most models change by less than 2 percentage points, and some even show a slight decrease. This confirms that residual label errors do not materially deflate APR.