acceptodds
Under review as a conference paper at ICLR 2027

HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

Abstract

Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, ex- isting evaluations largely focus on short-term reasoning, failing to assess a criti- cal capability: maintaining cumulative temporal consistency over extended time horizons. To close this evaluation gap, we introduce HeiCo-FOCUS, a clinically grounded dataset for evaluating long-context video understanding through the task of **F**oreign **O**bject **C**ontextual **U**nderstanding in **S**urgery. Built on a dataset of **He**idelberg **Co**lorectal surgeries, this task requires models to continuously track multiple objects as they are inserted, manipulated, occluded, and removed over procedures lasting up to hours. HeiCo-FOCUS comprises 30,000 visual question answerings (VQAs) pairs covering five core capabilities: object recognition, tem- poral grounding, aggregation, event and procedural understanding, and complex reasoning. The dataset was constructed through a rigorous multi-stage annotation pipeline involving large-scale crowd annotation and 39 surgical domain experts to ensure high quality and clinical relevance. To systematically probe model behav- ior, we introduce a multi-track evaluation framework that progressively increases temporal and contextual demands from single frames to full procedures. Experi- ments with ten frontier VLMs show that HeiCo-FOCUS tasks are far from solved: only around half of the models clearly outperform a text-only baseline. Across the video tracks, models perform best on event and procedural understanding (mean Accuracy: 56.5% across all models), while temporal grounding remains particu- larly challenging for all evaluated models (mean Accuracy: 19.7%). We therefore expect HeiCo-FOCUS to serve as a catalyst for the development of models capa- ble of reliable, temporally consistent reasoning over hours-long videos.

Primary area:
datasets and benchmarks
Keywords:
Surgical Data Science, Vision-Language Model Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8GdLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HeiCo-FOCUS builds on the 30 surgical videos in HeiCo, adding foreign-object annotations and 30,000 VQA pairs. It links object instances across time to support questions about identity, insertion, retrieval, and counting. The authors evaluate ten VLMs on FRAME, SEGMENT, and PROCEDURE tasks, with a GPT-5.6 Sol text-only baseline for comparison. Temporal grounding is a particular weakness across the video tracks.

Critical strengths
  1. Tracking foreign objects is a useful application of long-context video understanding. Establishing whether an object has been retrieved can require linking earlier and later observations.

  2. The annotation effort is substantial: 79,682 foreign-object boxes, including 55,629 with instance identities across time. These add useful information beyond the original HeiCo labels.

  3. Reporting video-level accuracy with bootstrapped confidence intervals is also appropriate given the shared video sources.

Critical weaknesses
  1. Sparse video sampling makes the temporal-grounding results difficult to interpret.

PROCEDURE windows extend to 296 minutes but are represented by only 300 uniformly sampled frames. Meanwhile, time-point and duration answers are scored with a tolerance capped at five seconds. For a 296-minute window, the interval between neighboring samples is approximately 59 seconds. An insertion or retrieval may therefore happen between sampled frames, leaving insufficient evidence to locate it within the required tolerance. Brief appearances may be missed altogether.

  1. The annotation process is substantial, but the reliability of the final question pool remains insufficiently quantified.

Stage 6 verifies the generation pipeline using about 500 questions, after which the pipeline produces 30,000 VQA pairs. This is useful quality control, but it does not establish the error rate of the final pool. The paper should report the errors found during verification, their distribution across question types, and whether revised templates were checked again.

Appendix I acknowledges cases where sponges are difficult to associate across time or where removal is uncertain, especially around recording interruptions. It is unclear how such uncertainty was handled when generating reference answers. Human difficulty on aggregation and duration questions does not independently validate those answers.

  1. The experiments provide limited evidence for the central claim about cumulative temporal consistency.

The benchmark contains relevant long-context questions, but its main metrics score answers independently. They do not directly measure whether a model maintains a consistent history for the same object across multiple time points. A model could give incompatible answers about insertion, retrieval, and later presence without that contradiction being explicitly measured.

The three tracks are useful for describing performance, but they also differ in question composition and sampling. Their score differences therefore do not isolate the effect of context length. Controlled history-removal tests or paired questions about the same instance would provide stronger evidence that the benchmark measures dependence on distant evidence and cumulative state consistency. The current analysis mainly establishes that the tasks are difficult, rather than explaining the source of that difficulty. [Secs. 3, 5–6]

  1. The public evaluation example does not reproduce the stated judge configuration.

The paper specifies gpt-oss-120b for semantic grading and validates it against human raters. However, the supplementary package defaults to Qwen3.5-4B, including when using the Evaluator() call shown in Appendix B. The evaluator supports custom judges, so this does not establish that the reported experiments used the wrong model. Still, the exact configuration used for the paper needs to be supplied; validation of one judge cannot be assumed to apply to another.

Questions for authors

See Weakness.

11 comments
AuthorsOpenReview ↗

#1 Addressing W3

We thank Reviewer 8GdL for their critical and constructive review, which has helped us improve the paper substantially. As part of our rebuttal, we now provide (1) analyses verifying the long-context requirements of the VQA and (2) model evaluations on full procedures at 1 fps (where possible). We have further recruited human annotators to conduct a final verification on a sample of the dataset.

Critical Weaknesses

  • W3: The experiments provide limited evidence for the central claim about cumulative temporal consistency. The benchmark contains relevant long-context questions, but its main metrics score answers independently. They do not directly measure whether a model maintains a consistent history for the same object across multiple time points. A model could give incompatible answers about insertion, retrieval, and later presence without that contradiction being explicitly measured. The three tracks are useful for describing performance, but they also differ in question composition and sampling. Their score differences therefore do not isolate the effect of context length. Controlled history-removal tests or paired questions about the same instance would provide stronger evidence that the benchmark measures dependence on distant evidence and cumulative state consistency. The current analysis mainly establishes that the tasks are difficult, rather than explaining the source of that difficulty. [Secs. 3, 5–6]

We thank the reviewer for this valuable feedback. To estimate the distribution of temporal context required to answer PROCEDURE-track questions, we systematically analyzed all test-set questions with automatically generated reference answers for windows of at least 2 hours. Specifically, we truncated the available history to the final L minutes and recomputed the reference answers. By comparing these with the original answers, we quantified the proportion of questions that depend on information beyond the retained temporal context. Keeping only the last 30 minutes changes 51 % of the answers; keeping the last 60 minutes still changes 38 % (Tab. R1). These shares are lower bounds, since an answer such as “no” or “0” can stay the same after the cut even when the question concerns the whole video.

Cut-off LProportion of reference answers that change
5 min68 %
15 min62 %
30 min51 %
60 min38 %

Tab. R1. Proportion of reference answers that change when only the last L minutes of the video are kept (test PROCEDURE questions with windows of at least 2 h). About half of the answers depend on more than the last 30 minutes.

To test whether the models use this history, we cut their input in the same way. On the same questions, we removed every frame older than 30 or 5 minutes before the window end from the 300-frame input and re-ran three models with the same prompt (Tab. R2).

ModelFull videoLast 30 min onlyLast 5 min only
GPT-5.6 Sol48.5 %40.8 %39.1 %
Gemini 3.8 Flash52.8 %46.7 %45.2 %
Claude Sonnet 540.2 %37.1 %37.5 %
Text-only baseline33.4 %--

Tab. R2. Accuracy (%) on all PROCEDURE questions with windows of at least 2 h when only the frames from the last 30 or 5 min of the 300-frame input are kept. Accuracy falls for all three models already at 30 minutes, highlighting the importance of long context in our dataset.

AuthorsOpenReview ↗

#2 Addressing W2 and W1

  • W2: The annotation process is substantial, but the reliability of the final question pool remains insufficiently quantified. Stage 6 verifies the generation pipeline using about 500 questions, after which the pipeline produces 30,000 VQA pairs. This is useful quality control, but it does not establish the error rate of the final pool. The paper should report the errors found during verification, their distribution across question types, and whether revised templates were checked again.
    • We thank the reviewer for this feedback. To clarify, while the final pool was not checked again after fixing templates, our internal pipeline included multiple rounds of reviews at all stages. The bounding boxes and instance linkings were verified by experienced annotators; the question generation included multiple iterations of creating and checking templates and question generation. To fully address the reviewer’s comment, however, we are currently conducting a review of the final dataset and will post the results before the end of the rebuttal period.
    • Appendix I acknowledges cases where sponges are difficult to associate across time or where removal is uncertain, especially around recording interruptions. It is unclear how such uncertainty was handled when generating reference answers. Human difficulty on aggregation and duration questions does not independently validate those answers.
      • Validity conditions for question generation: Questions were instantiated from predefined templates that enforce strict validity conditions to avoid ambiguous scenarios. For instance, surgical videos may contain "blue screen" interruptions where the visual feed is lost. In that case for any question involving event aggregation, our question templates explicitly require that the selected video clip contains zero blue screen events.
      • Handling uncertain instance IDs: Whenever annotators were uncertain about a specific object's identity (such as a sponge), they logged an explicit uncertainty event in the annotations. If such an event occurred within a given video snippet, our generation pipeline automatically suppressed any questions requiring instance-level tracking or identification for that segment.
  • W1: Sparse video sampling makes the temporal-grounding results difficult to interpret. PROCEDURE windows extend to 296 minutes but are represented by only 300 uniformly sampled frames. Meanwhile, time-point and duration answers are scored with a tolerance capped at five seconds. For a 296-minute window, the interval between neighboring samples is approximately 59 seconds. An insertion or retrieval may therefore happen between sampled frames, leaving insufficient evidence to locate it within the required tolerance. Brief appearances may be missed altogether.
    • We thank the reviewer for the raised concern. Importantly, the primary contribution of HeiCo-FOCUS is the dataset itself, rather than the initial model evaluation. The dataset provides temporally grounded annotations and reference answers derived from the original videos, independent of the frame-sampling strategy used for model inference. The 300-frame limit reflects a practical constraint of the evaluated models and APIs, not an inherent limitation of the benchmark. Future models with more efficient long-video processing or adaptive frame selection can exploit the dataset's richer temporal information. Every frame carries a burned-in hh:mm:ss clock and the window start and end times are given in the prompt. For the PROCEDURE track, full answerability cannot currently be guaranteed for most models because APIs limit the number of images per request and overall request size. However, as Gemini models are able to take more context, we reran Gemini 3.8 Flash and Gemini 3.5 Flash Lite at 1 fps on all test PROCEDURE questions with windows of at most 1 hour (Tab. R3). Crucially, at 1 fps, time answers improve by 4-6 pp, but overall accuracy does not improve.
Gemini 3.8 FlashGemini 3.5 Flash Lite
All questions, 300 frames65.8 %40.8 %
All questions, 1 fps65.8 %37.3 %
Time questions, 300 frames28.2 %14.6 %
Time questions, 1 fps34.0 %18.4 %

Tab. R3. Accuracy (%) of Gemini 3.8 Flash and Gemini 3.5 Flash Lite on all test PROCEDURE questions with windows of at most 1 h. The models received either the standard 300 frames or every frame at 1 fps. The input using 1 fps improves answers to time questions by 4-6 pp, but not overall accuracy.

AuthorsOpenReview ↗

#3 Addressing W4

  • W4: The public evaluation example does not reproduce the stated judge configuration. The paper specifies gpt-oss-120b for semantic grading and validates it against human raters. However, the supplementary package defaults to Qwen3.5-4B, including when using the Evaluator() call shown in Appendix B. The evaluator supports custom judges, so this does not establish that the reported experiments used the wrong model. Still, the exact configuration used for the paper needs to be supplied; validation of one judge cannot be assumed to apply to another.
    • The reviewer is right that the example in Appendix B does not reproduce the paper’s grading. All reported results were graded by gpt-oss-120B. The package defaults to Qwen3.5-4B so that it can run without an API key. We have amended Appendix B to use the paper judge.
Reviewer_ZPoWOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HeiCo-Focus, a benchmark built from 30 colorectal surgery videos, with 30,00 questions covering individual frames, short segments and extended procedures. It has great contribution of the annotation including persistent objects, and individual foreign objects across time. These annotations support questions about object identification, counting, timing and reasoning. Ten VLMs are evaluated, with the best reaching 56.7% accuracy.

Critical strengths
  1. The motivation is clear and clinically meaningful. I appreciate the significant annotation effort behind this dataset and think it can be valuable in this field of research.
  2. The evaluation includes several useful checks: a text only baseline, a human baseline, capability-specific results and uncertainty across videos. The LLM judge is also compared with human grading.
Critical weaknesses
  1. The sampling and scoring may not be well matched, based on my understanding, procedure videos are represented by only 300 sampled frames, while timing answers must be within 5 second accuracy. For a two-hour input, i feel it is impossible to get within 5 seconds (bc each frame is 24 seconds apart). I doubt the low accuracy may come from this setting.
  2. The experiments do not clearly isolate which capability the models are missing. Comparisons using relevant short clips, shuffled frames, or other experiments can help distinguish if it is perception error or temporal ordering or miss of long term memory.
Questions for authors

The annotation effort is substantial, but if any video is annotated by more than one expert? I am interested in the measuring agreement between readers as well, thus can help us understand the ceiling of the models.

4 comments
AuthorsOpenReview ↗

#1 Addressing W1

We thank Reviewer ZPoW for their constructive review, which has helped us improve the paper substantially. As part of our rebuttal, we now provide (1) analyses verifying the long-context requirements of the VQA, (2) model evaluations on full procedures at 1 fps (where possible), and (3) resolution experiments below. We have further recruited human annotators to conduct a final verification on a sample of the dataset and to provide a new human baseline.

Critical Weaknesses

  • W1: The sampling and scoring may not be well matched, based on my understanding, procedure videos are represented by only 300 sampled frames, while timing answers must be within 5 second accuracy. For a two-hour input, i feel it is impossible to get within 5 seconds (bc each frame is 24 seconds apart). I doubt the low accuracy may come from this setting.
  • We thank the reviewer for the feedback. For the PROCEDURE track, full answerability cannot currently be guaranteed for most models because APIs limit the number of images per request and overall request size. However, as Gemini models are able to take more context, we reran Gemini 3.8 Flash and Gemini 3.5 Flash Lite at 1 fps on all test PROCEDURE questions with windows of at most 1 hour (Tab. R3). At 1 fps, time answers improve by 4-6 pp, but overall accuracy does not improve.
Gemini 3.8 FlashGemini 3.5 Flash Lite
All questions, 300 frames65.8 %40.8 %
All questions, 1 fps65.8 %37.3 %
Time questions, 300 frames28.2 %14.6 %
Time questions, 1 fps34.0 %18.4 %

Tab. R3. Accuracy (%) of Gemini 3.8 Flash and Gemini 3.5 Flash Lite on all test PROCEDURE questions with windows of at most 1 h. The models received either the standard 300 frames or every frame at 1 fps. The input using 1 fps improves answers to time questions by 4-6 pp, but not overall accuracy.

AuthorsOpenReview ↗

#2 Addressing W2 and Q1

  • W2: The experiments do not clearly isolate which capability the models are missing. Comparisons using relevant short clips, shuffled frames, or other experiments can help distinguish if it is perception error or temporal ordering or miss of long term memory.
    • We thank the reviewer for the suggestion. We have added three experiments to address the points:
    • Memory: To estimate the distribution of temporal context required to answer PROCEDURE-track questions, we systematically analyzed all test-set questions with automatically generated reference answers for windows of at least 2 hours. Specifically, we truncated the available history to the final L minutes and recomputed the reference answers. By comparing these with the original answers, we quantified the proportion of questions that depend on information beyond the retained temporal context. Keeping only the last 30 minutes changes 51 % of the answers; keeping the last 60 minutes still changes 38 % (Tab. R1). These shares are lower bounds, since an answer such as “no” or “0” can stay the same after the cut even when the question concerns the whole video.
Cut-off LProportion of reference answers that change
5 min68 %
15 min62 %
30 min51 %
60 min38 %

Tab. R1. Proportion of reference answers that change when only the last L minutes of the video are kept (test PROCEDURE questions with windows of at least 2 h). About half of the answers depend on more than the last 30 minutes.

To test whether the models use this history, we cut their input in the same way. On the same questions, we removed every frame older than 30 or 5 minutes before the window end from the 300-frame input and re-ran three models with the same prompt (Tab. R2).

ModelFull videoLast 30 min onlyLast 5 min only
GPT-5.6 Sol48.5 %40.8 %39.1 %
Gemini 3.8 Flash52.8 %46.7 %45.2 %
Claude Sonnet 540.2 %37.1 %37.5 %
Text-only baseline33.4 %--

Tab. R2. Accuracy (%) on all PROCEDURE questions with windows of at least 2 h when only the frames from the last 30 or 5 min of the 300-frame input are kept. Accuracy falls for all three models already at 30 minutes, highlighting the importance of long context in our dataset.

  • Perception
    • Input format: the three Gemini models (3.8 Flash, 3 Flash and 3.5 Flash Lite) received the clips as video files, all other models as frame sequences. We resampled the Gemini clips to 1 fps so that all models see the same frames.
    • Resolution: we used 360 × 210 because larger inputs exceed the request-size limits of most model APIs (a 300-frame request at 960 × 540 is about 32 MB). To test whether more spatial detail helps, we used FRAME questions, which refer to a single frame and therefore stay within these limits. We sampled 200 FRAME questions stratified by subcapability and evaluated three models at 360 × 210, 480 × 270 and 960 × 540 (Tab. R4). We report object-attribute and needle questions separately. Needle questions are all questions whose text or reference answer names a needle; needles are among the smallest foreign objects, so they should benefit most from a higher resolution.
    • Increasing the resolution translates to modest improvements in overall performance. Accuracy on object-attribute questions does not improve consistently, needle questions only improve clearly for Claude Sonnet 5, however at massively increased token usage and cost. For example for GPT-5.6 Sol, the image tokens per frame grow 5.6x.
Accuracy (%), 360x210 / 480x270 / 960x540GPT-5.6 SolGemini 3.8 FlashClaude Sonnet 5
All 20039.0 / 39.5 / 45.043.5 / 45.0 / 47.022.5 / 24.5 / 27.0
Object attributes (29)41.4 / 41.4 / 34.548.3 / 44.8 / 37.917.2 / 17.2 / 24.1
Needle questions (53)39.6 / 39.6 / 37.741.5 / 43.4 / 45.39.4 / 11.3 / 18.9

Tab. R4. Accuracy (%) on 200 FRAME questions at three input resolutions. Higher resolution brings modest overall gains, but no consistent gain on object attributes; needle questions improve clearly only for Claude Sonnet 5.

Questions For Authors

  • Q1: The annotation effort is substantial, but if any video is annotated by more than one expert? I am interested in the measuring agreement between readers as well, thus can help us understand the ceiling of the models.
    • We are currently conducting both a verification (using all available information including bounding boxes) and a human baseline on a subset of the final set and will post the results before the end of the discussion period.
Reviewer_ZPoWOpenReview ↗

Thank you for the author's detailed response.

I would thank you for the authors for adding these experiments and it definitely make things cleaner. As i am already at 3 out of 4 at the beginning. I would need a little more time to look back to the paper to see if i would like to raise to 4. Will get back to the authors by deadline.

AuthorsOpenReview ↗

Addressing Q1 with extra verification and a new human baseline

We thank the reviewer for the kind reply. We will monitor OpenReview until the end of the discussion period and are happy to answer any remaining questions.

As promised, we have now run both a verification and a human baseline for 300 questions (100 per track), drawn at random from the test split, stratified by capability.

  • Human baseline: Medical students answered all 300 questions (two to four students per track). Every question came with the frame, segment or procedure video at full resolution and frame rate, without bounding boxes or other annotations.
  • Verification: Two technical researchers checked all 300 questions, with the full video and all box and instance annotations (1 fps). For questions requiring medical expertise, a surgical expert was consulted.

Human baseline

Accuracy (%)FrameSegmentProcedure
Students, mean (range)55.0 (52.0–59.0)76.3 (70.0–79.0)64.0 (62.0–66.0)
Best model41.062.048.0
Median model28.039.038.5
Text only (no video)19.025.026.0

Tab. R5. Accuracy (%) of medical students (3, 4 and 2 per track) and models on the same 300 test questions (100 per track). On average, the students outperform the best model by 14–16 pp.

Every student outperforms every model on every track; the gap to the best model is 11–18 pp (frame), 8–17 (segment) and 14–18 (procedure). Students agree on 68–89 % of questions (Fleiss' κ 0.45, 0.58 and 0.61 for frame, segment and procedure).

Verification

Verdict (100 per track)FrameSegmentProcedure
Incorrect011
Correct, no issue869596
Correct, but very hard to answer1043
Correct, ambiguous phrasing400

Tab. R6. Verification verdicts for the 300 test questions (100 per track). 298 of the 300 reference answers (99 %) were confirmed as correct and only 2 as incorrect.

Out of the 300 reference answers, only 2 are wrong (0.7 %): one because of a missing bounding box, one because the question turned out to be underspecified. This question and the 4 ambiguous frame questions (1.3 %) can be fixed by simple rephrasing. Most correct but hard-to-answer questions involve small surgical clips that are hard to recognize without temporal context.

We thank the reviewers for their constructive feedback. The verification and the new human baseline will be added to the final version of the paper.

Reviewer_AHZdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HeiCo-FOCUS, a long-context surgical visual question answering (VQA) benchmark centered on tracking foreign objects through insertion, manipulation, occlusion, and retrieval, and reasoning about their states over time. Built on 30 colorectal procedures from HeiCo, the dataset comprises approximately 96 hours of video and 30,000 VQA pairs across three tracks: FRAME, SEGMENT, and PROCEDURE. It covers five capabilities: object recognition and identity matching, temporal grounding, aggregation, event and procedural understanding, and complex reasoning. The main contribution is a multi-stage annotation pipeline that establishes persistent object identities and uses them to generate clinically grounded long-video questions. The authors evaluate ten VLMs, alongside a text-only baseline, a human baseline, and capability-level analyses. Results reveal substantial limitations in temporal grounding and long-horizon aggregation.

Critical strengths
  1. The benchmark targets long-video understanding through foreign object tracking in surgery, providing a clinically meaningful application.
  2. The annotation pipeline is systematic, and the costly instance-level annotations constitute a valuable and scarce resource.
  3. The dataset and accompanying evaluation package are provided for release, offering a useful contribution to the community.
Critical weaknesses
  1. The PROCEDURE track represents videos lasting up to 296 minutes using only 300 uniformly sampled frames, yielding sampling intervals of nearly one minute. Yet, the tolerance for time-point and duration answers is capped at ±5 seconds. An insertion or retrieval event may therefore occur between sampled frames or be entirely absent from the input.

  2. Long inputs do not necessarily imply long-context requirements: some questions may be answerable from only a few local frames. The paper neither quantifies the temporal evidence required to answer questions nor compares full-context inputs against local-window inputs. The extent to which the benchmark specifically tests long-context understanding therefore remains unverified.

  3. Approximately 500 questions were used to validate the question-generation pipeline, but the paper does not clarify whether the final 30,000 VQA pairs underwent an independent random audit. Neither question ambiguity rates nor reference-answer error rates are reported. These measurements are important given the paper’s emphasis on a “quality-first approach.”

  4. Although preventing retained foreign objects motivates the benchmark, overall accuracy combines material recognition, functional explanations, temporal grounding, and other tasks. The paper does not separately evaluate which objects remain in the body at the end of surgery or how often models incorrectly report an unretrieved object as retrieved. The latter is particularly important to the stated safety objective.

  5. The test set contains only ten sigmoid resections from a single institution. Bootstrap confidence intervals are consequently based on a small number of videos, and overlapping intervals leave the robustness of several model rankings unclear. Foreign-object diversity in sigmoid resections may also be lower than in the full dataset, as suggested by the examples in Figure 6, although a systematic breakdown is needed to establish this.

Questions for authors
  1. What proportion of questions genuinely require long-term history rather than a local segment? Could the authors estimate or stratify the required evidence spans and provide representative ablations using local windows, history truncation, or frame-order shuffling to establish the importance of persistent memory and temporal order?

  2. Do the 300-frame PROCEDURE inputs include per-frame timestamps or an explicit temporal mapping? When sampling intervals substantially exceed five seconds and key events may be omitted, how is answerability ensured under the ±5-second tolerance? Could ablations involving timestamps, sampling-aware tolerances, and different frame counts establish whether the temporal grounding conclusions are robust?

  3. Our understanding is that the approximately 500 Stage 6 questions were used to validate the pipeline before final dataset generation. Is this correct? What proportion were marked “Needs revision” or received answers inconsistent with the references, and what changes followed? Were the final 30,000 VQA pairs independently audited through random sampling? If so, please report question ambiguity and reference-answer error rates.

  4. What proportions of the 30,000 VQA pairs are expert-written versus template-generated, and how do these proportions vary across capability categories? Please distinguish expert authorship from expert verification and report the question counts associated with the 85 templates to clarify source diversity and template concentration.

  5. Did human raters view the original videos or the same downsampled inputs as the models? Was replay allowed, and what was the average time spent per question? Given the 35% human accuracy on event aggregation, was an error analysis conducted to distinguish task difficulty, evaluation constraints, question ambiguity, and reference-answer errors? Were all 300 questions retained in the final release?

  6. Which models received native video versus frame sequences, and were there additional internal sampling or input constraints? Could the authors test ranking stability under a common input configuration on representative models and questions? What motivated the 210 × 360 resolution, and does increasing spatial resolution substantially improve performance on object attributes and needle-related questions?

6 comments
AuthorsOpenReview ↗

#1 Addressing W1 and W2

We thank Reviewer AHZd for their thorough and constructive review! Prompted by their questions, we show experimentally that (1) about half of the PROCEDURE questions with windows of at least 2 h depend on video content older than 30 minutes, and model accuracy drops when this history is removed (Tab. R1, R2); (2) sampling at 1 fps instead of 300 frames improves time answers but not overall accuracy (Tab. R3); and (3) a higher input resolution brings only modest gains (Tab. R4). We are also auditing a sample of the final dataset and creating a new human baseline, and will post both before the end of the discussion period.

Critical Weaknesses

  • W1: The PROCEDURE track represents videos lasting up to 296 minutes using only 300 uniformly sampled frames, yielding sampling intervals of nearly one minute. Yet, the tolerance for time-point and duration answers is capped at ±5 seconds. An insertion or retrieval event may therefore occur between sampled frames or be entirely absent from the input.
    • We agree that uniformly sampling 300 frames may miss brief events, making precise temporal localization challenging or even impossible for some questions. Importantly, the primary contribution of HeiCo-FOCUS is the dataset itself, rather than the initial model evaluation. The dataset provides temporally grounded annotations and reference answers derived from the original videos, independent of the frame-sampling strategy used for model inference. The 300-frame limit reflects a practical constraint of the evaluated models and APIs, not an inherent limitation of the benchmark. Future models with more efficient long-video processing or adaptive frame selection can exploit the dataset's richer temporal information. However, to address the reviewer’s comment, we reran Gemini 3.8 Flash and Gemini 3.5 Flash Lite at 1 fps on all test PROCEDURE questions with windows of at most 1 hour (Tab. R3). At 1 fps, time answers improve by 4-6 pp, but overall accuracy does not improve.
  • W2: Long inputs do not necessarily imply long-context requirements: some questions may be answerable from only a few local frames. The paper neither quantifies the temporal evidence required to answer questions nor compares full-context inputs against local-window inputs. The extent to which the benchmark specifically tests long-context understanding therefore remains unverified.
    • To estimate the distribution of temporal context required to answer PROCEDURE-track questions, we systematically analyzed all test-set questions with automatically generated reference answers for windows of at least 2 hours. Specifically, we truncated the available history to the final L minutes and recomputed the reference answers. By comparing these with the original answers, we quantified the proportion of questions that depend on information beyond the retained temporal context. Keeping only the last 30 minutes changes 51 % of the answers; keeping the last 60 minutes still changes 38 % (Tab. R1). These shares are lower bounds, since an answer such as “no” or “0” can stay the same after the cut even when the question concerns the whole video.
Cut-off LProportion of reference answers that change
5 min68 %
15 min62 %
30 min51 %
60 min38 %

Tab. R1. Proportion of reference answers that change when only the last L minutes of the video are kept (test PROCEDURE questions with windows of at least 2 h). About half of the answers depend on more than the last 30 minutes.

To test whether the models use this history, we cut their input in the same way. On the same questions, we removed every frame older than 30 or 5 minutes before the window end from the 300-frame input and re-ran three models with the same prompt (Tab. R2).

ModelFull videoLast 30 min onlyLast 5 min only
GPT-5.6 Sol48.5 %40.8 %39.1 %
Gemini 3.8 Flash52.8 %46.7 %45.2 %
Claude Sonnet 540.2 %37.1 %37.5 %
Text-only baseline33.4 %--

Tab. R2. Accuracy (%) on all PROCEDURE questions with windows of at least 2 h when only the frames from the last 30 or 5 min of the 300-frame input are kept. Accuracy falls for all three models already at 30 minutes, highlighting the importance of long context in our dataset.

AuthorsOpenReview ↗

#2 Addressing W3 and W4

  • W3: Approximately 500 questions were used to validate the question-generation pipeline, but the paper does not clarify whether the final 30,000 VQA pairs underwent an independent random audit. Neither question ambiguity rates nor reference-answer error rates are reported. These measurements are important given the paper’s emphasis on a “quality-first approach.”
    • We thank the reviewer for this important point. Quality control ran at every stage, not only at the end:
      • Annotation: professional annotators first marked all foreign objects with bounding boxes at 1 fps, and an automated method corrected and completed these boxes. 18 medical students then checked every box and linked the boxes of each object across the whole video, in two passes (annotation and validation).
      • Question design: questions are generated automatically from templates, at random time points and at clinically relevant moments marked by annotators, such as the insertion or retrieval of an object. In several review rounds, clinical and technical experts inspected samples of generated questions and reworded or restricted templates that produced ambiguous questions. An additional LLM-based check flagged potentially ambiguous questions for review.
      • Pipeline check before release: before generating the final dataset, 22 external raters (surgical AI researchers, medical students and expert surgeons) answered about 500 generated questions covering all capabilities and object classes; their disagreements with the reference answers led to further template fixes (see Q3).
      • Final pool: we are currently auditing a sample of the released test questions and will post ambiguity and reference-error rates before the discussion ends.
  • W4: Although preventing retained foreign objects motivates the benchmark, overall accuracy combines material recognition, functional explanations, temporal grounding, and other tasks. The paper does not separately evaluate which objects remain in the body at the end of surgery or how often models incorrectly report an unretrieved object as retrieved. The latter is particularly important to the stated safety objective.
    • We thank the reviewer for this important point. Retained foreign objects are rare (about 1 in 10,000 operations [doi:10.1056/NEJMsa021721]) and videos documenting such errors are understandably difficult to obtain due to their sensitive nature. Furthermore, reference counts are impossible to generate in many shared surgical videos because shared videos may contain "blue screen" or "video blur" interruptions (for de-identification reasons) where the visual feed is lost, and there is no visual control about inserted or retrieved object instances. We accounted for these ambiguities by instantiating questions from predefined templates that enforce strict validity conditions (e.g. no blue screen event occurring in an input clip). Broadly speaking, we deliberately designed HeiCo-FOCUS to assess the underlying capabilities required for detecting retained foreign objects rather than only the final outcome: recognizing and tracking foreign objects over time, understanding their function, and distinguishing objects that must be retrieved from those intentionally left in the body, such as surgical clips.
AuthorsOpenReview ↗

#3 Addressing W5, Q1 and Q2

  • W5: The test set contains only ten sigmoid resections from a single institution. Bootstrap confidence intervals are consequently based on a small number of videos, and overlapping intervals leave the robustness of several model rankings unclear. Foreign-object diversity in sigmoid resections may also be lower than in the full dataset, as suggested by the examples in Figure 6, although a systematic breakdown is needed to establish this.
    • We thank the reviewer for this valuable suggestion. Appendix D (Table 3) now provides a breakdown of foreign object prevalence separately for the three procedure types. The test videos cover 6 of the 7 foreign-object classes (all except silicone loops), with clips, drains and sponges each appearing in 8-9 of the 10 videos. We kept the original HeiCo split to stay aligned with the intended use and community derivatives.
    • The limited number of videos follows from a deliberate choice. The mission of this work was to provide a benchmark that simultaneously addresses a hard technical challenge (long-context video understanding) and a globally relevant clinical problem (avoiding retained foreign objects). We therefore deliberately chose colorectal surgery as the foundation of our work, despite its substantially higher per video annotation cost. The majority of current surgical video datasets are heavily skewed towards cholecystectomies. Colorectal surgery, however, is substantially more complex, with longer operative times, greater blood loss, and a higher rate of complications (16–35 % of patients, doi:10.1055/s-0036-1584501). Thus, the risk of retained foreign objects is higher, and the importance of tracking them throughout a case is greater.
    • A main criterion was to maximize diversity of foreign objects and questions. For example, the average procedure length of HeiCo was five times higher than in a private dataset of 100 cholecystectomy videos, while the average number of object instances was three times higher. With the same annotation budget, we could therefore have annotated approximately 100-150 cholecystectomy videos. However, numerous cholecystectomy datasets already exist. Instead, we deliberately focused on a diverse set of complex, clinically highly relevant colorectal procedures.

Questions For Authors

  • Q1: What proportion of questions genuinely require long-term history rather than a local segment? Could the authors estimate or stratify the required evidence spans and provide representative ablations using local windows, history truncation, or frame-order shuffling to establish the importance of persistent memory and temporal order?

We thank the reviewer for this valuable suggestion. See our answer to AHZd W2 above (Tab. R1, R2).

  • Q2: Do the 300-frame PROCEDURE inputs include per-frame timestamps or an explicit temporal mapping? When sampling intervals substantially exceed five seconds and key events may be omitted, how is answerability ensured under the ±5-second tolerance? Could ablations involving timestamps, sampling-aware tolerances, and different frame counts establish whether the temporal grounding conclusions are robust?
    • Every frame carries a burned-in hh:mm:ss clock and the window start and end times are given in the prompt. For the PROCEDURE track, full answerability cannot currently be guaranteed for most models because APIs limit the number of images per request and overall request size. However, as Gemini models are able to take more context, we reran Gemini 3.8 Flash and Gemini 3.5 Flash Lite at 1 fps on all test PROCEDURE questions with windows of at most 1 hour (Tab. R3). Crucially, at 1 fps, time answers improve by 4-6 pp, but overall accuracy does not improve.
Gemini 3.8 FlashGemini 3.5 Flash Lite
All questions, 300 frames65.8 %40.8 %
All questions, 1 fps65.8 %37.3 %
Time questions, 300 frames28.2 %14.6 %
Time questions, 1 fps34.0 %18.4 %

Tab. R3. Accuracy (%) of Gemini 3.8 Flash and Gemini 3.5 Flash Lite on all test PROCEDURE questions with windows of at most 1 h. The models received either the standard 300 frames or every frame at 1 fps. The input using 1 fps improves answers to time questions by 4-6 pp, but not overall accuracy.

Comments on the paper

2 comments

Dear Reviewers,

As we are approaching the end of the discussion period, I would like to kindly remind everyone to review the authors' responses and follow up on any outstanding questions or concerns. If you have not yet engaged in the discussion, please take a moment to do so before the deadline.

Thank you all for your time and efforts throughout the review process! Your AC

AuthorsOpenReview ↗

Overview of improvements in response to the reviews

Dear Area Chair,

We thank the reviewers for their constructive feedback, which has helped us improve the paper substantially. During the discussion period, we ran four new analyses (Tab. R1-R4), manually verified 300 test questions and collected a new human baseline. They address the four main concerns of the reviews:

  • Long context (all three reviewers): For questions covering at least 2 hours of video, 51 % of the reference answers change when only the last 30 minutes are kept, and all three tested models lose accuracy under the same cut (Tab. R1, R2). The benchmark therefore requires long context.
  • Frame sampling (all three reviewers) and resolution (AHZd): Sampling at 1 fps instead of 300 frames improves time answers by 4-6 pp but not overall accuracy (Tab. R3), and a higher resolution brings only modest gains (Tab. R4). The evaluation settings therefore explain only a small part of the model errors.
  • Reference answers (AHZd, 8GdL): In a random sample of 300 test questions, checked against the full videos and all annotations, only 2 reference answers are wrong and 4 questions are ambiguously phrased (Tab. R6). We will be able to fix these 4 questions and one of the errors with simple rephrasing.
  • Human performance (AHZd, ZPoW): Medical students answered the same 300 questions on the full-resolution videos without annotations. Every student outperforms every model on every track (on average by 14-16 pp over the best model), and the students agree on 68-89 % of the questions (Tab. R5). Current models thus remain clearly below human performance.

The revised paper also reports the foreign-object prevalence per procedure type and the question counts per sub-capability (Appendix D, Tables 3 and 4), and the evaluation example in Appendix B now uses the judge model of the paper.

We hope these additions resolve the reviewers’ concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.