acceptodds
Under review as a conference paper at ICLR 2027

ThreadNIAH: Reasoning over Relational Needles in Long Video Haystacks

Abstract

Multimodal large language models can increasingly process hour-long videos, where relevant evidence may be sparse and distributed over time. Effective long-video understanding therefore requires models to locate relevant evidence, link related evidence across the video, and combine multiple pieces of evidence. Building on the controlled setting of Video Needle-in-a-Haystack (NIAH), we introduce ThreadNIAH to systematically evaluate these capabilities. ThreadNIAH organizes five task types into three levels that progressively evaluate locating, linking, and combining evidence: Single-Needle Retrieval (Details), Multi-Needle Threading (Tracking and Comparison), and Multi-Needle Synthesis (Counting and Logic). To instantiate these task types in a controlled and scalable manner, we design 29 task templates that specify the evidence-bearing needles, their relations, and the questions defined over them. Since a correct final answer does not necessarily imply correct intermediate steps, each Main QA is paired with diagnostic Sub-QAs that probe its supporting evidence and intermediate steps. Together, they form a QA chain, the basic evaluation unit of ThreadNIAH. We use Main Acc, Sub Acc, and All Pass Rate (APR) to measure final-task, intermediate-step, and all-question chain accuracy, respectively. The resulting benchmark contains 870 QA chains, comprising 870 Main QAs and 2,580 Sub-QAs. Across ten video MLLMs and five thinking variants, most configurations perform substantially better on single-needle retrieval than on multi-needle tasks. Moreover, thinking does not ensure consistent success across QA chains: even Gemini 3.6 Flash with thinking achieves 67.7% Main Acc but only 49.6% APR. Similar difficulty patterns on real-world long-video questions further support the relevance of ThreadNIAH to real-world long-video understanding.

Primary area:
datasets and benchmarks
Keywords:
Long Video Understanding, Multimodal Large Language Models, Needle-in-a-haystack, Video Question Answering, Video Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5RQxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ThreadNIAH, a video needle-in-a-haystack benchmark for long-video understanding that evaluates locating, linking, and combining evidence. It organizes five task types into three levels, instantiates them through 29 templates over synthesized videos of 40 to 130 minutes built from 199 source videos, and pairs each main question with diagnostic sub-questions forming a QA chain, scored by Main Acc, Sub Acc, and All Pass Rate. The benchmark contains 870 chains with 3,450 questions.

Critical strengths
  • The QA chain design is a genuine addition to video NIAH evaluation: probing intermediate evidence and operations separately exposes failures that final-answer accuracy hides, and the prefix-survival analysis makes the chain view quantitative.
  • The evaluation covers current frontier and open-weight models with documented inference settings, and the generator model is excluded from the baselines to avoid self-bias.
Critical weaknesses
  • Ground truth is generated by Gemini 3.1 Pro and quality-controlled by whether GPT-5.2 can answer from needle frames. This biases the pool toward questions one model family can solve from sparse frames, and residual label errors compound in APR, which requires every question in a chain to be correct. The human audit pass rate of about 94 percent implies a per-question error level that materially deflates APR.
  • The evidence that ThreadNIAH difficulty transfers to natural videos is weak. The task-type correlation of 0.70 has a confidence interval of 0.10 to 0.90 over only five points, and the template-level correlation of 0.569 has a lower bound of 0.111. The abstract's claim of similar difficulty patterns on real-world videos rests on these two unstable estimates.
Questions for authors
  • What sampling settings were used for GLM-5.3-Flash, and can the two MiMo variants be reported separately in the thinking row?
  • Could you human-verify a random subset of chains and report how the residual label error rate shifts APR? A per-question error level near the human-audit failure rate would materially deflate the chain metrics.
3 comments
AuthorsOpenReview ↗

Response to reviewer 5RQx (part 1)

Dear Reviewer 5RQx, thank you for your time and thoughtful review. Below, we address your concerns.

W1. Human-verify a random subset of chains and quantify how residual label errors (implied by the ~94% audit pass rate) deflate the strict APR metric.

We thank the reviewer for raising this point, but we would like to clarify a key distinction regarding the reported human audit rate. The 94.4% in Appendix B is a per-judgment pass rate (1,086 out of 1,150 individual judgments passed), not a per-question error rate. When counted by question, only 10 out of 575 questions (1.7%) were jointly flagged as failing by both annotators.

To directly quantify the impact of residual label errors on APR, we recomputed APR on these 145 audited chains, retaining only chains in which every question was judged as passing by both annotators, and compared this with the un-pruned results. Because the reasons for failing the audit extend beyond label errors, this comparison overestimates the impact of label errors and thus serves as an upper-bound estimate. Note that all numbers below are computed on these 145 chains and are therefore not directly comparable to the 870-chain results in the main text. As shown, the changes in APR are minimal: the largest improvement is 6.6 percentage points, most models change by less than 2 percentage points, and some even show a slight decrease. This confirms that residual label errors do not materially deflate APR.

ModelUnfilteredChain drop
Proprietary Models (Thinking)
Gemini 3.6 Flash55.762.3 (+6.6)
Qwen3.7-Plus39.644.2 (+4.5)
Qwen3.7-Max32.133.3 (+1.3)
MiMo V2.6 Pro5.26.4 (+1.2)
GLM-5.3-Flash6.36.0 (-0.3)
Proprietary Models
Gemini 3.6 Flash24.126.5 (+2.3)
Qwen3.7-Max21.723.1 (+1.4)
Qwen3.7-Plus19.422.8 (+3.3)
Doubao Seed 2.1 Pro10.712.3 (+1.6)
GLM-5.3-Flash8.37.9 (-0.4)
MiMo V2.6 Pro4.35.0 (+0.7)
Open-Weight Models
Qwen3.5-27B9.78.8 (-0.8)
Qwen3.5-9B3.43.9 (+0.5)
VideoChat-Flash-7B3.42.0 (-1.5)
LongVT-RFT0.71.1 (+0.3)
AuthorsOpenReview ↗

Response to reviewer 5RQx (part 2)

W2. Substantiate the claimed difficulty transfer with stronger evidence, as the current correlations are based on unstable, wide-confidence-interval estimates.

We acknowledge that the limited data points (5 task types; 5 questions per template) result in wide confidence intervals. However, we would like to emphasize two points:

(1) The direction of the positive difficulty correlation is stable.

The lower bounds of both confidence intervals (0.10 and 0.111) are greater than 0, indicating that the difficulty under the two settings is positively correlated. Although the exact correlation magnitude is imprecise, the direction of this relationship is robust.

(2) Failure-mode analysis provides evidence independent of the correlation coefficients.

Beyond the correlation values, we compared the failure modes of each model under the two settings individually—a comparison that does not rely on correlation coefficients. On natural videos, 6 out of 7 models achieve their highest accuracy on Details; once cross-segment association or evidence integration is required, accuracy drops markedly. For example, Gemini-3.6-flash drops from 70.0% on Details to 26.7% on Counting, and Qwen3.5-27B from 52.5% to 20.0% (Table 6). On ThreadNIAH, 6 out of the same 7 models also achieve their highest accuracy on Details (Table 2). This indicates that both settings expose the same failure pattern: models can localize a single piece of evidence, but struggle to associate and integrate multiple pieces of evidence across clips.

AuthorsOpenReview ↗

Response to reviewer 5RQx (part 3)

W3. What sampling settings were used for GLM-5.3-Flash, and can the two MiMo variants be reported separately in the thinking row?

GLM-5.3-Flash's official API does not expose a video frame-rate parameter, nor does it publish a default sampling rate. We therefore submit the raw videos directly without setting an FPS, and the actual frame sampling is determined server-side (Appendix C). The standard setting uses low reasoning effort, while the thinking setting uses high reasoning effort.

For MiMo V2.6 Pro, the thinking results merge responses from the standard endpoint and the ultraspeed endpoint of the same model. Ultraspeed is an officially provided acceleration tier backed by the same underlying model, and both endpoints share identical reasoning settings (1 fps, thinking enabled, JSON Object output). Therefore, they are not two distinct model variants, and we report them jointly.

Reviewer_waJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ThreadNIAH, a new benchmark for evaluating long-video understanding and reasoning over multiple pieces of evidence. The benchmark organizes five task types into three levels, including single-needle retrieval, multi-needle threading, and multi-needle synthesis. The major novelty of the paper lies in introducing QA chains, which combine main questions with diagnostic sub-questions to evaluate final-answer accuracy and intermediate reasoning capabilities. The authors construct 870 QA chains using 29 task templates and evaluate ten video MLLMs. Experimental results demonstrate that current models struggle with multi-needle reasoning, particularly when combining evidence across different video clips.

Critical strengths
  1. The proposed ThreadNIAH provides a systematic framework for evaluating different levels of long-video understanding, from single-needle retrieval to multi-needle reasoning. The controlled task construction enables targeted evaluation of different capabilities.

  2. The introduction of QA chains provides an interesting way to evaluate supporting evidence and intermediate reasoning capabilities rather than relying solely on final-answer accuracy. Experiments on various video MLLMs demonstrate the challenges of multi-needle reasoning.

Critical weaknesses
  1. Limited relevance to natural long-video understanding: Many tasks rely heavily on artificially overlaid symbols, such as codewords, colored shapes, and dots, to establish relations between video clips. As a result, the benchmark may primarily evaluate symbolic cue recognition and rule-based reasoning rather than understanding natural relationships between video events. The effectiveness of this design in evaluating real-world long-video understanding requires further justification.

  2. Insufficient analysis of QA-chain evaluation: Although QA chains provide additional diagnostic information, the lower All Pass Rate (APR) may partly result from requiring multiple questions to be answered correctly. Since each question is evaluated independently, it remains unclear to what extent QA chains reflect actual intermediate reasoning failures. Further analysis would help clarify their diagnostic advantages over conventional question-level evaluation.

  3. Potential influence of visual marker recognition: The benchmark requires models to distinguish fine-grained visual markers, including different colors, shapes, and spatial positions. However, the experiments do not sufficiently investigate whether performance degradation comes from failures in recognizing these markers or from cross-clip reasoning. Additional experiments with ground-truth marker information would help clarify this issue.

  4. Limited validation on natural long-video tasks: Although Appendix D reports correlations between synthetic and natural long-video task difficulty, the evaluation includes only 145 natural-video questions. Moreover, the analysis mainly focuses on difficulty rankings rather than whether the two settings reveal similar model failures. More comprehensive validation would strengthen the practical relevance of the benchmark.

Questions for authors
  1. Why are artificial visual markers necessary for constructing multi-needle reasoning tasks? Could natural visual attributes be used instead?

  2. Can the authors provide experiments to distinguish visual marker recognition errors from cross-clip reasoning errors?

  3. How does QA-chain length affect APR? Can the authors further demonstrate the diagnostic advantages of QA chains beyond question-level accuracy?

  4. Can the authors provide more evidence that ThreadNIAH reflects similar reasoning difficulties and failure patterns in natural long videos?

5 comments
AuthorsOpenReview ↗

Response to reviewer waJJ (part 1)

Dear Reviewer waJJ, thank you for your time and thoughtful review. Below, we address your concerns.

W1. Relying on artificial overlays primarily tests symbolic cue recognition rather than natural long-video understanding. Why not use natural visual attributes instead?

1. Artificial overlays are a design choice for controllability, scalability, and robustness.

We do not use natural visual attributes to link needles because the needles are drawn from randomly selected videos with highly diverse content. Automatically generating natural-attribute cues would introduce additional error risks into the construction pipeline. Visual overlays allow any video to be automatically used as a haystack or a needle, while keeping the evidence relations and ground-truth answers fully controllable. This makes our benchmark low-cost, controllable, and scalable. In contrast, manually annotating cross-needle evidence relations is highly labor-intensive and becomes increasingly difficult to scale as the evidence structure grows more complex. Furthermore, it is important to note that overlays are only responsible for linking the needles: in most templates, the Main QA asks about the natural visual content of the target clips, and some templates directly rely on natural visual attributes (Appendix E).

2. To verify that this design reflects genuine long-video understanding, we additionally conducted a short-context experiment on the 145 audited chains from Appendix B (29 templates × 5).

When the needles were provided directly to the models in their order of appearance, all models showed significant improvements across the three metrics. This indicates that recognizing overlays and reasoning itself is not the bottleneck; rather, the core difficulty lies in cross-segment understanding over long videos.

ModelMainSubAPR
Proprietary Models (Thinking)
Qwen3.7-Plus80.0 (+26.6)95.6 (+10.5)69.8 (+33.1)
Qwen3.7-Max79.9 (+30.0)93.7 (+9.2)66.7 (+37.4)
Gemini 3.6 Flash77.6 (+14.2)92.2 (+2.8)60.8 (+9.9)
GLM-5.3-Flash74.0 (+44.8)89.8 (+48.3)48.6 (+43.3)
MiMo V2.6 Pro62.5 (+35.0)80.5 (+40.6)36.0 (+31.9)
Proprietary Models
Doubao Seed 2.1 Pro74.2 (+36.3)80.6 (+25.7)38.8 (+28.6)
Gemini 3.6 Flash73.9 (+24.8)76.2 (+16.1)36.7 (+16.9)
GLM-5.3-Flash73.6 (+36.2)83.4 (+36.1)39.0 (+32.3)
Qwen3.7-Max73.0 (+22.1)77.8 (+17.8)35.6 (+18.0)
Qwen3.7-Plus71.6 (+24.7)77.6 (+20.0)33.5 (+17.4)
MiMo V2.6 Pro64.0 (+33.1)67.8 (+34.3)23.7 (+20.2)
Open-Weight Models
Qwen3.5-27B72.2 (+31.0)72.0 (+23.5)30.9 (+23.8)
Qwen3.5-9B61.7 (+33.0)56.9 (+24.4)16.3 (+13.8)
VideoChat-Flash-7B47.9 (+19.1)50.6 (+18.1)10.1 (+7.6)
LongVT-RFT39.6 (+1.4)46.2 (+17.7)6.7 (+6.2)

Please refer to oNNW#3 for more experimental details.

AuthorsOpenReview ↗

Response to reviewer waJJ (part 2)

W2. Clarify how QA-chain length affects APR and demonstrate the diagnostic advantages of QA chains beyond conventional question-level accuracy.

1. Impact of Chain Length on APR:

Overall, the longer the chain, the more difficult it is to achieve a high APR. We will include an analysis of this chain-length effect on APR in the revised paper. For example, the breakdown results for Gemini-3.6-flash thinking are shown below. Even at the same chain length, substantial APR gaps persist across task types (e.g., at L=4, Tracking achieves 72.0, whereas Counting and Logic only reach 37.8 and 39.7, respectively). This indicates that the performance gaps stem not solely from chain length, but also from inherent task difficulty.

typeL=2L=3L=4L=5L=6
details66.178.9———
tracking——72.025.7—
comparison—76.5—70.335.7
counting——37.8—8.3
logic——39.735.773.7

2. Diagnostic Advantages of QA Chains:

The diagnostic advantage of QA chains lies in their ability to reveal how model capabilities degrade as reasoning depth increases. As shown in the prefix survival rates in the bottom row of Figure 4, questions that mainly probe evidence localization exhibit high survival rates in the early stages, but these rates drop sharply with increasing depth. This demonstrates that failures are concentrated in the later stages of cross-segment association and integration—information that single-question accuracy cannot provide. It also suggests that current models already possess strong retrieval capabilities; the primary bottleneck is their reasoning ability across multiple segments.

AuthorsOpenReview ↗

Response to reviewer waJJ (part 3)

W3. Disentangle visual marker recognition errors from cross-clip reasoning errors.

We thank the reviewer for this constructive suggestion. To disentangle visual marker recognition errors from cross-clip reasoning errors, we conducted an additional analysis.

In summary, we categorized the Sub-QAs into perception and reasoning types and computed their respective accuracies. This categorization is based on the operation type specified by the template during construction: perception tasks only require information from within a single needle (e.g., locating the needle, reading the marker text, recognizing marker attributes such as color, shape, and position), whereas reasoning tasks require cross-needle operations (e.g., filtering or constructing sets under conditions, and counting/aggregation).

The results for Gemini-3.6-flash thinking are shown below. As can be seen, in long videos, perception accuracy reaches 82.9%–94.4% across all task types, indicating that the model is capable of recognizing these markers. In contrast, for Tracking, Comparison, and Counting, reasoning accuracy is notably lower than perception accuracy (e.g., 71.9% vs. 94.4% on Tracking). Logic is an exception, where reasoning accuracy is not lower than perception accuracy. This gap indicates that the primary challenge lies in cross-needle reasoning rather than in marker perception.

Task typePerceptionReasoning
Details88.586
Tracking94.471.9
Comparison91.483.5
Counting82.973.4
Logic86.992.1
Overall89.580.8

Furthermore, the short-context experiment in W1 also supports this conclusion: when needles with markers were provided directly to the models in their order of appearance, all models showed significant improvements across the three metrics, with Gemini-3.6-flash thinking achieving 92.2% Sub Acc and 77.6% Main Acc (+14.2). This indicates that, under short-context settings, models can correctly recognize markers and perform cross-needle reasoning, and that performance degradation mainly stems from cross-segment reasoning in long videos rather than marker recognition.

Reviewer_oNNWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ThreadNIAH extends video Needle-in-a-Haystack to relational, multi-needle reasoning over long videos (40–130 min, mean 72.2). It defines three levels and five task types (L1 Details; L2 Tracking, Comparison; L3 Counting, Logic), instantiated by 29 templates that specify needle roles (target/reference/decoy), relations, questions, and diagnostic sub-QA steps. Each Main QA is paired with Sub-QAs forming a QA chain (870 chains = 870 Main + 2,580 Sub = 3,450 questions), answered independently and scored by Main Acc, Sub Acc, and All Pass Rate (APR). Ten MLLMs plus thinking variants and a human baseline are evaluated; a natural-video companion set checks ecological validity.

Critical strengths

1.Clear, useful taxonomy. Locate/link/combine maps cleanly onto the five task types and fills a real gap versus V-NIAH/VNBench/VideoChat-Flash. 2.The QA-chain + APR diagnostic is a genuine contribution. Showing that Gemini 3.6 Flash (thinking) reaches 67.7% Main but only 49.6% APR (an 18.1-point gap) is exactly what final-answer-only benchmarks hide; the prefix-survival analysis is a nice complement. 3.Reasonable scale and construction discipline: 199 diverse sources, controlled insertion, two automatic QC checks (visual reproducibility, text-only shortcut), and human review with reported agreement (Gwet's AC1 = 0.914, 94.4% pass). 4.A thoughtful ecological-validity check against natural long-video QA.

Critical weaknesses

1.The frame-sampling confound is not controlled — possibly the biggest threat. Videos are sampled at 1 fps but capped at 512–768 frames; for a 130-minute video this is roughly one frame every ~10 seconds. If needle clips are short, the evidence may never be sampled, especially for open-weight models. Poor Counting/Logic scores could reflect missing frames rather than reasoning failure. The paper never reports needle-clip duration vs. effective sampling, nor whether the needle frames were actually presented. 2.APR is not comparable across task types. Chain lengths differ systematically (Details 2.12 up to Counting 5.00), and APR is a strict conjunction, so longer chains mechanically depress it. Reporting APR by task type without length normalization conflates difficulty with chain length; the headline cross-type comparisons remain apples-to-oranges. 3.Artificial overlay convention. Reasoning is heavily mediated by burned-in symbols (code words, shapes, fill modes, marker counts, positions) and template logic, risking a measure of "can the model parse a synthetic puzzle" rather than natural long-video understanding. The natural-video companion (145 questions) is too small and its correlations too weak to fully rebut this: task-type correlation of 0.70 with CI [0.10, 0.90], and template-level 0.569 with CI [0.111, 0.611] (the latter CI looks implausibly narrow given the point estimate — please verify). 4.Generative/verification model dependence. QA content is generated by Gemini 3.1 Pro and QC'd by GPT-5.2. Gemini is (sensibly) excluded from baselines, but natural-scene answer correctness still rests on Gemini's generation, and only two text-only-shortcut calls is a weak filter (random double-match is about 6% for four options).

Questions for authors

1.What is the needle-clip duration, and what fraction of needles are actually captured at each model's frame cap? Can you show accuracy conditioned on needle coverage? 2.Can you report a chain-length-normalized APR (or per-depth accuracy) so task types are comparable? 3.For the natural-video correlations, please clarify the bootstrap and the template-level interval. 4.How long are the source clips, and how does 1 fps interact with the visibility of small overlays?

6 comments
AuthorsOpenReview ↗

Response to reviewer oNNW (part 1)

Dear Reviewer oNNW, thank you for your time and thoughtful review. Below, we address your concerns.

W1. Needle Coverage Under Frame-Sampling Caps: Needle-Clip Duration, Capture Rate, and Coverage-Conditioned Accuracy

Most models are able to sample the needle frames. Under each model's frame cap, we computed the coverage rate, i.e., the fraction of needles for which at least one frame is sampled (each needle is a 5-second clip):

  1. API-based large models (full coverage): For the APIs of Gemini (1 fps, up to 3 h), Qwen (1 fps, server-side cap of 8,000 frames), Mimo (1 fps, server-side cap of 2,048 frames), and Doubao (1 fps, server-side cap of 1,280 frames), all needles are fully covered.

  2. Locally deployed models (partial coverage): For the four locally deployed models, full coverage is not achievable due to their maximum frame-sampling limits. We report their accuracy computed only on the samples where at least one needle frame was sampled, as shown below. We observe only marginal differences in performance.

ModelFramesCov. (macro, ≥1 frame/needle)Main (Δ)Sub (Δ)APR (Δ)
Qwen3.5-9B76885.4%32.7 (+1.7)37.6 (+0.2)3.7 (+0.2)
Qwen3.5-27B76885.4%42.4 (+2.1)51.6 (+3.2)9.6 (+1.0)
VideoChat-Flash-7B51262.7%33.3 (+2.7)35.0 (+2.9)4.1 (+0.6)
LongVT-RFT76885.4%30.3 (+1.7)30.3 (+0.5)3.1 (+0.7)
AuthorsOpenReview ↗

Response to reviewer oNNW (part 2)

W2. APR is not comparable across task types as chain length varies.

We thank the reviewer for this insightful comment. We agree that raw APR across task types without considering length may conflate difficulty with chain length.

To address this, we have computed per-depth APR. We will add a detailed analysis of this breakdown to the revised paper. For instance, the table below reports the per-depth accuracy of Gemini-3.6-flash thinking across different chain lengths:

typeL=2L=3L=4L=5L=6
details66.178.9———
tracking——72.025.7—
comparison—76.5—70.335.7
counting——37.8—8.3
logic——39.735.773.7

We acknowledge that the inherent chain length varies systematically across task types. However, when conditioned on the same chain length, the performance trends across task types remain broadly comparable. For example, at a comparable depth, the difficulty ordering of tasks is largely consistent with our main results. We will explicitly clarify this point and include the per-depth analysis in the revised manuscript to provide a more rigorous and fine-grained comparison.

AuthorsOpenReview ↗

Response to reviewer oNNW (part 3)

W3. Overlay reliance may undermine natural long-video evaluation, while the limited natural-video companion and an implausibly narrow template-level CI weaken the results.

1. Heavy reliance on artificial overlays and template logic may measure synthetic puzzle parsing rather than natural long-video understanding

We decouple the needle recognition from long-video retrieval and find that overlays and visual element reasoning are not the bottleneck. We run this experiment on the 145 audited chains from Appendix B (29 templates × 5), providing the needles directly to the model in the order they appear in the haystack. This evaluates whether the model can correctly recognize and reason across different needles in a short-context setting, and allows us to observe how long context affects performance. As shown below, compared to Needles-in-a-Haystack setting, answering questions directly based on the needles leads to a significant improvement. This demonstrates that recognizing overlays and reasoning itself is not the bottleneck; Rather, the core difficulty lies in cross-segment understanding in long videos.

ModelMainSubAPR
Proprietary Models (Thinking)
Qwen3.7-Plus80.0 (+26.6)95.6 (+10.5)69.8 (+33.1)
Qwen3.7-Max79.9 (+30.0)93.7 (+9.2)66.7 (+37.4)
Gemini 3.6 Flash77.6 (+14.2)92.2 (+2.8)60.8 (+9.9)
GLM-5.3-Flash74.0 (+44.8)89.8 (+48.3)48.6 (+43.3)
MiMo V2.6 Pro62.5 (+35.0)80.5 (+40.6)36.0 (+31.9)
Proprietary Models
Doubao Seed 2.1 Pro74.2 (+36.3)80.6 (+25.7)38.8 (+28.6)
Gemini 3.6 Flash73.9 (+24.8)76.2 (+16.1)36.7 (+16.9)
GLM-5.3-Flash73.6 (+36.2)83.4 (+36.1)39.0 (+32.3)
Qwen3.7-Max73.0 (+22.1)77.8 (+17.8)35.6 (+18.0)
Qwen3.7-Plus71.6 (+24.7)77.6 (+20.0)33.5 (+17.4)
MiMo V2.6 Pro64.0 (+33.1)67.8 (+34.3)23.7 (+20.2)
Open-Weight Models
Qwen3.5-27B72.2 (+31.0)72.0 (+23.5)30.9 (+23.8)
Qwen3.5-9B61.7 (+33.0)56.9 (+24.4)16.3 (+13.8)
VideoChat-Flash-7B47.9 (+19.1)50.6 (+18.1)10.1 (+7.6)
LongVT-RFT39.6 (+1.4)46.2 (+17.7)6.7 (+6.2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.