- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HeiCo-FOCUS builds on the 30 surgical videos in HeiCo, adding foreign-object annotations and 30,000 VQA pairs. It links object instances across time to support questions about identity, insertion, retrieval, and counting. The authors evaluate ten VLMs on FRAME, SEGMENT, and PROCEDURE tasks, with a GPT-5.6 Sol text-only baseline for comparison. Temporal grounding is a particular weakness across the video tracks.
Critical strengths
-
Tracking foreign objects is a useful application of long-context video understanding. Establishing whether an object has been retrieved can require linking earlier and later observations.
-
The annotation effort is substantial: 79,682 foreign-object boxes, including 55,629 with instance identities across time. These add useful information beyond the original HeiCo labels.
-
Reporting video-level accuracy with bootstrapped confidence intervals is also appropriate given the shared video sources.
Critical weaknesses
- Sparse video sampling makes the temporal-grounding results difficult to interpret.
PROCEDURE windows extend to 296 minutes but are represented by only 300 uniformly sampled frames. Meanwhile, time-point and duration answers are scored with a tolerance capped at five seconds. For a 296-minute window, the interval between neighboring samples is approximately 59 seconds. An insertion or retrieval may therefore happen between sampled frames, leaving insufficient evidence to locate it within the required tolerance. Brief appearances may be missed altogether.
- The annotation process is substantial, but the reliability of the final question pool remains insufficiently quantified.
Stage 6 verifies the generation pipeline using about 500 questions, after which the pipeline produces 30,000 VQA pairs. This is useful quality control, but it does not establish the error rate of the final pool. The paper should report the errors found during verification, their distribution across question types, and whether revised templates were checked again.
Appendix I acknowledges cases where sponges are difficult to associate across time or where removal is uncertain, especially around recording interruptions. It is unclear how such uncertainty was handled when generating reference answers. Human difficulty on aggregation and duration questions does not independently validate those answers.
- The experiments provide limited evidence for the central claim about cumulative temporal consistency.
The benchmark contains relevant long-context questions, but its main metrics score answers independently. They do not directly measure whether a model maintains a consistent history for the same object across multiple time points. A model could give incompatible answers about insertion, retrieval, and later presence without that contradiction being explicitly measured.
The three tracks are useful for describing performance, but they also differ in question composition and sampling. Their score differences therefore do not isolate the effect of context length. Controlled history-removal tests or paired questions about the same instance would provide stronger evidence that the benchmark measures dependence on distant evidence and cumulative state consistency. The current analysis mainly establishes that the tasks are difficult, rather than explaining the source of that difficulty. [Secs. 3, 5–6]
- The public evaluation example does not reproduce the stated judge configuration.
The paper specifies gpt-oss-120b for semantic grading and validates it against human raters. However, the supplementary package defaults to Qwen3.5-4B, including when using the Evaluator() call shown in Appendix B. The evaluator supports custom judges, so this does not establish that the reported experiments used the wrong model. Still, the exact configuration used for the paper needs to be supplied; validation of one judge cannot be assumed to apply to another.
Questions for authors
See Weakness.
11 comments
#2 Addressing W2 and W1
- W2: The annotation process is substantial, but the reliability of the final question pool remains insufficiently quantified. Stage 6 verifies the generation pipeline using about 500 questions, after which the pipeline produces 30,000 VQA pairs. This is useful quality control, but it does not establish the error rate of the final pool. The paper should report the errors found during verification, their distribution across question types, and whether revised templates were checked again.
- We thank the reviewer for this feedback. To clarify, while the final pool was not checked again after fixing templates, our internal pipeline included multiple rounds of reviews at all stages. The bounding boxes and instance linkings were verified by experienced annotators; the question generation included multiple iterations of creating and checking templates and question generation. To fully address the reviewer’s comment, however, we are currently conducting a review of the final dataset and will post the results before the end of the rebuttal period.
- Appendix I acknowledges cases where sponges are difficult to associate across time or where removal is uncertain, especially around recording interruptions. It is unclear how such uncertainty was handled when generating reference answers. Human difficulty on aggregation and duration questions does not independently validate those answers.
- Validity conditions for question generation: Questions were instantiated from predefined templates that enforce strict validity conditions to avoid ambiguous scenarios. For instance, surgical videos may contain "blue screen" interruptions where the visual feed is lost. In that case for any question involving event aggregation, our question templates explicitly require that the selected video clip contains zero blue screen events.
- Handling uncertain instance IDs: Whenever annotators were uncertain about a specific object's identity (such as a sponge), they logged an explicit uncertainty event in the annotations. If such an event occurred within a given video snippet, our generation pipeline automatically suppressed any questions requiring instance-level tracking or identification for that segment.
- W1: Sparse video sampling makes the temporal-grounding results difficult to interpret. PROCEDURE windows extend to 296 minutes but are represented by only 300 uniformly sampled frames. Meanwhile, time-point and duration answers are scored with a tolerance capped at five seconds. For a 296-minute window, the interval between neighboring samples is approximately 59 seconds. An insertion or retrieval may therefore happen between sampled frames, leaving insufficient evidence to locate it within the required tolerance. Brief appearances may be missed altogether.
- We thank the reviewer for the raised concern. Importantly, the primary contribution of HeiCo-FOCUS is the dataset itself, rather than the initial model evaluation. The dataset provides temporally grounded annotations and reference answers derived from the original videos, independent of the frame-sampling strategy used for model inference. The 300-frame limit reflects a practical constraint of the evaluated models and APIs, not an inherent limitation of the benchmark. Future models with more efficient long-video processing or adaptive frame selection can exploit the dataset's richer temporal information. Every frame carries a burned-in hh:mm:ss clock and the window start and end times are given in the prompt. For the PROCEDURE track, full answerability cannot currently be guaranteed for most models because APIs limit the number of images per request and overall request size. However, as Gemini models are able to take more context, we reran Gemini 3.8 Flash and Gemini 3.5 Flash Lite at 1 fps on all test PROCEDURE questions with windows of at most 1 hour (Tab. R3). Crucially, at 1 fps, time answers improve by 4-6 pp, but overall accuracy does not improve.
| Gemini 3.8 Flash | Gemini 3.5 Flash Lite | |
|---|---|---|
| All questions, 300 frames | 65.8 % | 40.8 % |
| All questions, 1 fps | 65.8 % | 37.3 % |
| Time questions, 300 frames | 28.2 % | 14.6 % |
| Time questions, 1 fps | 34.0 % | 18.4 % |
Tab. R3. Accuracy (%) of Gemini 3.8 Flash and Gemini 3.5 Flash Lite on all test PROCEDURE questions with windows of at most 1 h. The models received either the standard 300 frames or every frame at 1 fps. The input using 1 fps improves answers to time questions by 4-6 pp, but not overall accuracy.
#3 Addressing W4
- W4: The public evaluation example does not reproduce the stated judge configuration. The paper specifies gpt-oss-120b for semantic grading and validates it against human raters. However, the supplementary package defaults to Qwen3.5-4B, including when using the Evaluator() call shown in Appendix B. The evaluator supports custom judges, so this does not establish that the reported experiments used the wrong model. Still, the exact configuration used for the paper needs to be supplied; validation of one judge cannot be assumed to apply to another.
- The reviewer is right that the example in Appendix B does not reproduce the paper’s grading. All reported results were graded by gpt-oss-120B. The package defaults to Qwen3.5-4B so that it can run without an API key. We have amended Appendix B to use the paper judge.
#1 Addressing W3
We thank Reviewer 8GdL for their critical and constructive review, which has helped us improve the paper substantially. As part of our rebuttal, we now provide (1) analyses verifying the long-context requirements of the VQA and (2) model evaluations on full procedures at 1 fps (where possible). We have further recruited human annotators to conduct a final verification on a sample of the dataset.
Critical Weaknesses
We thank the reviewer for this valuable feedback. To estimate the distribution of temporal context required to answer PROCEDURE-track questions, we systematically analyzed all test-set questions with automatically generated reference answers for windows of at least 2 hours. Specifically, we truncated the available history to the final L minutes and recomputed the reference answers. By comparing these with the original answers, we quantified the proportion of questions that depend on information beyond the retained temporal context. Keeping only the last 30 minutes changes 51 % of the answers; keeping the last 60 minutes still changes 38 % (Tab. R1). These shares are lower bounds, since an answer such as “no” or “0” can stay the same after the cut even when the question concerns the whole video.
Tab. R1. Proportion of reference answers that change when only the last L minutes of the video are kept (test PROCEDURE questions with windows of at least 2 h). About half of the answers depend on more than the last 30 minutes.
To test whether the models use this history, we cut their input in the same way. On the same questions, we removed every frame older than 30 or 5 minutes before the window end from the 300-frame input and re-ran three models with the same prompt (Tab. R2).
Tab. R2. Accuracy (%) on all PROCEDURE questions with windows of at least 2 h when only the frames from the last 30 or 5 min of the 300-frame input are kept. Accuracy falls for all three models already at 30 minutes, highlighting the importance of long context in our dataset.