- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EgoEasy Benchmark (EEB) for query-free, complete annotation of egocentric workflows. Its 472 videos span 38.15 hours and contain 2891 episodes, with human-authored references covering global summaries, typed episode timelines, role-bound facts, navigation, and evidence. The evaluator separates workflow structure, fine-grained proposition recovery, and annotation reliability through three dashboards. It combines order-preserving temporal alignment, prediction-side semantic canonicalization, capacity-constrained proposition matching, and explicit distinctions between confirmed unsupported claims, unresolved extras, and duplicates. Seven video-language systems are evaluated on a test set. Results reveal substantially weaker complete workflow recovery than coarse summary generation.
Critical strengths
It introduces a meaningful task that evaluates complete annotation rather than selected answers. EEB requires systems to recover an entire typed workflow, including task execution, navigation, idle periods, and visually unidentifiable intervals, without video-specific queries or reference boundaries. Its structured facts distinguish object visibility, carrying, and functional tool use. This matters because successful question answering or coarse captioning does not establish that a system can produce complete and reliable supervision from an untrimmed recording.
A comparatively careful evaluation metrics with substantive human validation. The evaluator measures prediction-side precision as well as required-reference recall, constrains both sides of proposition matching to capacity one, distinguishes unresolved extras from confirmed hallucinations, and includes adaptive boundary tolerances. Five independent human annotations on the development set provide evidence of target reproducibility, while held-out review of 1000 frozen judge decisions yields 94.3% confirmation. These safeguards strengthen the benchmark's diagnostic value and address several common weaknesses of recall-oriented, purely text-similarity-based evaluation. Such reliable evaluation is often what is most lacking in the field of egocentric video.
Critical weaknesses
- The baseline comparison does not isolate model capability from input budgets and annotation protocols. Systems differ in sampling density, frame captions, visual resolution, and generation strategy. Four models use staged annotation after direct outputs repeatedly truncated, whereas the other systems generate the full schema directly. The paper acknowledges these differences and appropriately avoids causal comparisons across protocol groups, but lacks matched experiments on the same backbone and controlled sampling, output-length, and call budgets. Consequently, low scores can reflect output bottlenecks or propagated timeline errors as well as deficiencies in video understanding. Controlled comparisons and resource reporting are necessary to identify which limitations the benchmark is actually exposing.
- Data provenance and release details are insufficiently explicit for a benchmark contribution. The manuscript describes scene coverage and annotation structure, but does not sufficiently specify traceable video sources, acquisition conditions, participant or recording-session grouping, or the access and licensing arrangements for the corpus. It also does not establish whether development and test videos are separated by wearer, location, or session, or audit possible exposure in previously trained models. Hidden references are a useful safeguard against label access, but do not by themselves establish that the footage is unseen. Clear provenance, split manifests, privacy and access policies, and verifiable release arrangements would improve reproducibility and clarify the benchmark's generalization claims. These omissions leave risks unresolved; they do not demonstrate that contamination actually occurred.
Questions for authors
Please see the weaknesses. I will decide the final score based on the opinions of other reviewers and the rebuttal.
4 comments
Response to 9EBi: Data Provenance, Split Scope, and Controlled Release(W2)
Thank you for asking for clearer provenance, split scope, prior-exposure information, and release conditions. These address different questions: where EEB's recordings came from, what its test split measures, and what safeguards govern access.
1. New Collection and Recording Permissions
All 472 EEB videos were newly commissioned for the benchmark from paid collectors performing everyday first-person workflows; none were repackaged from existing datasets. Collector agreements cover recording, research use, and controlled distribution of privacy-processed copies to approved researchers. Relevant site permissions were obtained, including for hotel-room and restroom recordings. Privacy masking has been applied to prepared footage.
The data are traceable through video IDs and collection records rather than inferred from previously released benchmark sources. Identifying collector information and precise source paths are not part of the research release.
2. Evaluation Scope and Development/Test Split
EEB uses 48 development videos and 424 hidden-reference test videos, partitioned at the video level.
Importantly, EEB evaluates annotation of new video instances within the covered workflow domains, rather than specifically testing generalization to unseen wearers, locations, or recording sessions. Wearers and locations may recur across splits; wearer-, site-, and session-disjointness are not requirements of the benchmark's evaluation target.
Familiarity with a kitchen, hotel room, or cleaning workflow does not reveal a particular recording's event sequence, episode boundaries, object/tool roles, or uncertainty intervals. Each system must still recover these video-specific facts from the recording itself, without access to hidden test references. The split is defined by stable video identifiers and development/test membership. Available source/session information and pseudonymous wearer/site groupings provide additional provenance context without disclosing personal identities. These groupings describe recording relationships rather than define the benchmark's generalization objective.
Accordingly, EEB's results characterize in-domain, video-level workflow annotation performance, not a separate claim of participant- or site-disjoint generalization.
3. Prior Exposure and Contamination
EEB's recordings were privately collected for this benchmark and were not sourced from publicly released corpora. To our knowledge, they were not publicly distributed before the reported evaluations. This collection process reduces the risk of direct exposure through previously published video benchmarks. Hidden test references protect evaluation annotations, while newly collected footage provides an additional safeguard against known public-dataset overlap.
We distinguish this from claims about proprietary pretraining corpora, which cannot be fully audited from the available model information. EEB's evaluation conclusions concern the reported model configurations on newly collected video instances, rather than requiring an unverifiable guarantee about every model's historical training exposure.
4. Privacy and Controlled Research Access
EEB adopts application-based research access under an EEB Research Data Use Agreement. The policy permits non-commercial research, requires attribution and reasonable data-security safeguards, and prohibits unauthorized redistribution, re-identification, identity linkage, and attempts to reverse privacy masking.
Only privacy-processed recordings and corresponding safe annotations are included in the research release; hidden test references remain withheld. Development references are available through the controlled-access process.
The release policy covers audiovisual material, annotation text, and metadata while preserving timestamp-to-evidence correspondence. Material that cannot be safely distributed without obscuring required evidence is withheld or handled under the documented masking policy. These arrangements provide traceable provenance, explicit evaluation scope, and controlled research access. EEB evaluates complete, evidence-grounded annotation of new video instances within represented workflow domains.
Thank you for the responses. The within-backbone controls and resource reporting substantially alleviate my concern about baseline comparability, while the provenance, split-scope, and controlled-access clarifications address most of my data-related concerns.
My remaining concerns are the robustness of the evaluation to temporal thresholds and boundary tolerances, and the validation of the newly introduced independent reference-coverage judgments discussed in the response to Reviewer b7vN. Please ensure that the revised scoring formulas, implementation, validation, and reported results are consistent, and incorporate the data-governance documentation into the revision. Overall, the rebuttal strengthens my assessment, and I am inclined to raise my rating from weak rejection to weak acceptance.
Response to 9EBi: Thank You to Reviewer 9EBi and Planned Revisions
We sincerely thank you for the careful reading and for raising the rating. We are glad the within-backbone controls and provenance clarifications addressed the main concerns.
For the remaining points, the revision will:
- add a sensitivity analysis over temporal thresholds and boundary tolerances;
- report human validation of the new reference-coverage judgments;
- check that the scoring formulas, the released evaluator, and all reported results are consistent;
- add the data-governance documentation (provenance, split scope, privacy, and access policy) to the paper.
Thank you again for the constructive feedback, which has substantially improved the paper.
Response to 9EBi: Within-Backbone Controls and Output Bottlenecks(W1)
We appreciate the distinction between video understanding and annotation-pipeline constraints. As in our response to Reviewer aj6N, we conducted within-backbone controls for Qwen3-VL-8B/32B, varying frame sampling (0.5/1 FPS) and Direct/Staged generation. Here we focus on output bottlenecks, propagated timeline errors, and resource comparability.
1. Matched Controls and Temporal Results
Within each backbone/FPS setting, Direct and Staged global passes used the same source-frame indices and preprocessing, verified by run manifests. We fixed checkpoints, greedy decoding, repetition penalty (1.0), XGrammar (0.2.7), and the global visual budget (25,165,824 pixels; at most 512 frames). Staged detail calls receive only model-predicted intervals, never reference boundaries.
The table reports video-macro Strict Episode, Typed Segment, and Boundary F1 (%). The 1-FPS 8B-Staged and 32B-Direct entries match the published Table 3; the other configurations are the additional full-test controls.
At 1 FPS, 8B Staged exceeds Direct by 2.14, 6.05, and 3.59 percentage points for the three metrics, consistent with staging's intended benefit in reducing per-call complete-schema truncation. By contrast, 32B Direct performs better in Strict and Typed F1, while Staged raises Boundary F1 (20.76 vs. 18.50). Increasing FPS does not consistently improve 32B scores. Thus, protocol effects are backbone- and metric-dependent; staging is not claimed to be universally superior.
2. Output Constraints and Resource Accounting
The original staged protocol was adopted after direct complete-schema responses from smaller backbones repeatedly truncated. Staged generation reduces per-call output length, but cannot repair an omitted or mistyped episode: detail generation is conditioned on the model's own timeline. Hence the benchmark can expose both generation bottlenecks and propagated structural errors, not just video-recognition difficulty.
At 1 FPS, mean recorded resource use was:
Direct uses a 32,768-token output cap, while Staged uses per-call summary/timeline/detail caps of 2,048/8,192/8,192. Although global observations are matched, aggregate calls, visual exposure, and output budgets are not: Staged also revisits predicted local windows. The pixel budget couples FPS with spatial resolution.
3. Implications for Baseline Evaluation
EEB compares configured end-to-end annotation systems under one schema, reference set, and evaluator, rather than intrinsic backbones at identical total compute.
The controls highlight two distinct challenges. First, generating a complete structured annotation can be limited by output constraints even when the model can recognize portions of the video. Second, successfully generating a complete schema does not guarantee accurate temporal segmentation: errors in the initial timeline propagate into subsequent detail extraction. These findings reinforce EEB's evaluation objective. The benchmark measures whether a complete annotation system can produce reliable, temporally grounded supervision, including generation completeness and structural correctness.