acceptodds
Under review as a conference paper at ICLR 2027

EgoEasy Benchmark: Evaluating Automated Annotation of Egocentric Workflows

Abstract

Egocentric videos provide a scalable source of real-world activity data for embodied learning, yet converting unedited footage into reliable supervision remains a major bottleneck. Existing benchmarks evaluate only isolated tasks, query-driven segments, or predefined perception outputs, failing to assess whether a system can recover complete, evidence-grounded workflows while limiting unsupported claims. To address this gap, we introduce EgoEasy Benchmark (EEB), a comprehensive benchmark and evaluation suite for query-free annotation of egocentric workflows. EEB comprises 472 videos totaling 38.15 hours (2,891 episodes across 10 environmental and 13 workflow families), capturing rich transitions, navigation, interruptions, and unidentifiable intervals. We establish a unified annotation schema and evaluation protocol that decouples reference-content recovery from evidential reliability via prediction-side semantic canonicalization, evidence-aware adjudication, and capacity-constrained matching. Three complementary dashboards evaluate hierarchical workflow structure, fine-grained propositions, and annotation reliability. Evaluation across seven video-language model baselines reveals that current systems remain weak on complete temporal coverage, role grounding, and evidential calibration, highlighting the need for EEB in advancing robust video-based supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
egocentric video, automated video annotation, embodied AI, long-video understanding, temporal segmentation, video-language models, annotation reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9EBiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EgoEasy Benchmark (EEB) for query-free, complete annotation of egocentric workflows. Its 472 videos span 38.15 hours and contain 2891 episodes, with human-authored references covering global summaries, typed episode timelines, role-bound facts, navigation, and evidence. The evaluator separates workflow structure, fine-grained proposition recovery, and annotation reliability through three dashboards. It combines order-preserving temporal alignment, prediction-side semantic canonicalization, capacity-constrained proposition matching, and explicit distinctions between confirmed unsupported claims, unresolved extras, and duplicates. Seven video-language systems are evaluated on a test set. Results reveal substantially weaker complete workflow recovery than coarse summary generation.

Critical strengths

It introduces a meaningful task that evaluates complete annotation rather than selected answers. EEB requires systems to recover an entire typed workflow, including task execution, navigation, idle periods, and visually unidentifiable intervals, without video-specific queries or reference boundaries. Its structured facts distinguish object visibility, carrying, and functional tool use. This matters because successful question answering or coarse captioning does not establish that a system can produce complete and reliable supervision from an untrimmed recording.

A comparatively careful evaluation metrics with substantive human validation. The evaluator measures prediction-side precision as well as required-reference recall, constrains both sides of proposition matching to capacity one, distinguishes unresolved extras from confirmed hallucinations, and includes adaptive boundary tolerances. Five independent human annotations on the development set provide evidence of target reproducibility, while held-out review of 1000 frozen judge decisions yields 94.3% confirmation. These safeguards strengthen the benchmark's diagnostic value and address several common weaknesses of recall-oriented, purely text-similarity-based evaluation. Such reliable evaluation is often what is most lacking in the field of egocentric video.

Critical weaknesses
  • The baseline comparison does not isolate model capability from input budgets and annotation protocols. Systems differ in sampling density, frame captions, visual resolution, and generation strategy. Four models use staged annotation after direct outputs repeatedly truncated, whereas the other systems generate the full schema directly. The paper acknowledges these differences and appropriately avoids causal comparisons across protocol groups, but lacks matched experiments on the same backbone and controlled sampling, output-length, and call budgets. Consequently, low scores can reflect output bottlenecks or propagated timeline errors as well as deficiencies in video understanding. Controlled comparisons and resource reporting are necessary to identify which limitations the benchmark is actually exposing.
  • Data provenance and release details are insufficiently explicit for a benchmark contribution. The manuscript describes scene coverage and annotation structure, but does not sufficiently specify traceable video sources, acquisition conditions, participant or recording-session grouping, or the access and licensing arrangements for the corpus. It also does not establish whether development and test videos are separated by wearer, location, or session, or audit possible exposure in previously trained models. Hidden references are a useful safeguard against label access, but do not by themselves establish that the footage is unseen. Clear provenance, split manifests, privacy and access policies, and verifiable release arrangements would improve reproducibility and clarify the benchmark's generalization claims. These omissions leave risks unresolved; they do not demonstrate that contamination actually occurred.
Questions for authors

Please see the weaknesses. I will decide the final score based on the opinions of other reviewers and the rebuttal.

4 comments
AuthorsOpenReview ↗

Response to 9EBi: Within-Backbone Controls and Output Bottlenecks(W1)

We appreciate the distinction between video understanding and annotation-pipeline constraints. As in our response to Reviewer aj6N, we conducted within-backbone controls for Qwen3-VL-8B/32B, varying frame sampling (0.5/1 FPS) and Direct/Staged generation. Here we focus on output bottlenecks, propagated timeline errors, and resource comparability.

1. Matched Controls and Temporal Results

Within each backbone/FPS setting, Direct and Staged global passes used the same source-frame indices and preprocessing, verified by run manifests. We fixed checkpoints, greedy decoding, repetition penalty (1.0), XGrammar (0.2.7), and the global visual budget (25,165,824 pixels; at most 512 frames). Staged detail calls receive only model-predicted intervals, never reference boundaries.

The table reports video-macro Strict Episode, Typed Segment, and Boundary F1 (%). The 1-FPS 8B-Staged and 32B-Direct entries match the published Table 3; the other configurations are the additional full-test controls.

BackboneFPSProtocolStrict F1Typed F1Boundary F1
8B0.5Direct11.5326.4314.58
8B0.5Staged12.2228.6515.70
8B1.0Direct12.9627.1519.11
8B1.0Staged15.1033.2022.70
32B0.5Direct11.1729.3722.00
32B0.5Staged9.7827.8519.91
32B1.0Direct10.7030.9018.50
32B1.0Staged8.3825.8720.76

At 1 FPS, 8B Staged exceeds Direct by 2.14, 6.05, and 3.59 percentage points for the three metrics, consistent with staging's intended benefit in reducing per-call complete-schema truncation. By contrast, 32B Direct performs better in Strict and Typed F1, while Staged raises Boundary F1 (20.76 vs. 18.50). Increasing FPS does not consistently improve 32B scores. Thus, protocol effects are backbone- and metric-dependent; staging is not claimed to be universally superior.

2. Output Constraints and Resource Accounting

The original staged protocol was adopted after direct complete-schema responses from smaller backbones repeatedly truncated. Staged generation reduces per-call output length, but cannot repair an omitted or mistyped episode: detail generation is conditioned on the model's own timeline. Hence the benchmark can expose both generation bottlenecks and propagated structural errors, not just video-recognition difficulty.

At 1 FPS, mean recorded resource use was:

Backbone / ProtocolCallsInput tokens
8B Direct1.0025,946
8B Staged5.8894,398
32B Direct1.0025,946
32B Staged7.75104,733

Direct uses a 32,768-token output cap, while Staged uses per-call summary/timeline/detail caps of 2,048/8,192/8,192. Although global observations are matched, aggregate calls, visual exposure, and output budgets are not: Staged also revisits predicted local windows. The pixel budget couples FPS with spatial resolution.

3. Implications for Baseline Evaluation

EEB compares configured end-to-end annotation systems under one schema, reference set, and evaluator, rather than intrinsic backbones at identical total compute.

The controls highlight two distinct challenges. First, generating a complete structured annotation can be limited by output constraints even when the model can recognize portions of the video. Second, successfully generating a complete schema does not guarantee accurate temporal segmentation: errors in the initial timeline propagate into subsequent detail extraction. These findings reinforce EEB's evaluation objective. The benchmark measures whether a complete annotation system can produce reliable, temporally grounded supervision, including generation completeness and structural correctness.

AuthorsOpenReview ↗

Response to 9EBi: Data Provenance, Split Scope, and Controlled Release(W2)

Thank you for asking for clearer provenance, split scope, prior-exposure information, and release conditions. These address different questions: where EEB's recordings came from, what its test split measures, and what safeguards govern access.

1. New Collection and Recording Permissions

All 472 EEB videos were newly commissioned for the benchmark from paid collectors performing everyday first-person workflows; none were repackaged from existing datasets. Collector agreements cover recording, research use, and controlled distribution of privacy-processed copies to approved researchers. Relevant site permissions were obtained, including for hotel-room and restroom recordings. Privacy masking has been applied to prepared footage.

The data are traceable through video IDs and collection records rather than inferred from previously released benchmark sources. Identifying collector information and precise source paths are not part of the research release.

2. Evaluation Scope and Development/Test Split

EEB uses 48 development videos and 424 hidden-reference test videos, partitioned at the video level.

Importantly, EEB evaluates annotation of new video instances within the covered workflow domains, rather than specifically testing generalization to unseen wearers, locations, or recording sessions. Wearers and locations may recur across splits; wearer-, site-, and session-disjointness are not requirements of the benchmark's evaluation target.

Familiarity with a kitchen, hotel room, or cleaning workflow does not reveal a particular recording's event sequence, episode boundaries, object/tool roles, or uncertainty intervals. Each system must still recover these video-specific facts from the recording itself, without access to hidden test references. The split is defined by stable video identifiers and development/test membership. Available source/session information and pseudonymous wearer/site groupings provide additional provenance context without disclosing personal identities. These groupings describe recording relationships rather than define the benchmark's generalization objective.

Accordingly, EEB's results characterize in-domain, video-level workflow annotation performance, not a separate claim of participant- or site-disjoint generalization.

3. Prior Exposure and Contamination

EEB's recordings were privately collected for this benchmark and were not sourced from publicly released corpora. To our knowledge, they were not publicly distributed before the reported evaluations. This collection process reduces the risk of direct exposure through previously published video benchmarks. Hidden test references protect evaluation annotations, while newly collected footage provides an additional safeguard against known public-dataset overlap.

We distinguish this from claims about proprietary pretraining corpora, which cannot be fully audited from the available model information. EEB's evaluation conclusions concern the reported model configurations on newly collected video instances, rather than requiring an unverifiable guarantee about every model's historical training exposure.

4. Privacy and Controlled Research Access

EEB adopts application-based research access under an EEB Research Data Use Agreement. The policy permits non-commercial research, requires attribution and reasonable data-security safeguards, and prohibits unauthorized redistribution, re-identification, identity linkage, and attempts to reverse privacy masking.

Only privacy-processed recordings and corresponding safe annotations are included in the research release; hidden test references remain withheld. Development references are available through the controlled-access process.

The release policy covers audiovisual material, annotation text, and metadata while preserving timestamp-to-evidence correspondence. Material that cannot be safely distributed without obscuring required evidence is withheld or handled under the documented masking policy. These arrangements provide traceable provenance, explicit evaluation scope, and controlled research access. EEB evaluates complete, evidence-grounded annotation of new video instances within represented workflow domains.

Thank you for the responses. The within-backbone controls and resource reporting substantially alleviate my concern about baseline comparability, while the provenance, split-scope, and controlled-access clarifications address most of my data-related concerns.

My remaining concerns are the robustness of the evaluation to temporal thresholds and boundary tolerances, and the validation of the newly introduced independent reference-coverage judgments discussed in the response to Reviewer b7vN. Please ensure that the revised scoring formulas, implementation, validation, and reported results are consistent, and incorporate the data-governance documentation into the revision. Overall, the rebuttal strengthens my assessment, and I am inclined to raise my rating from weak rejection to weak acceptance.

AuthorsOpenReview ↗

Response to 9EBi: Thank You to Reviewer 9EBi and Planned Revisions

We sincerely thank you for the careful reading and for raising the rating. We are glad the within-backbone controls and provenance clarifications addressed the main concerns.

For the remaining points, the revision will:

  • add a sensitivity analysis over temporal thresholds and boundary tolerances;
  • report human validation of the new reference-coverage judgments;
  • check that the scoring formulas, the released evaluator, and all reported results are consistent;
  • add the data-governance documentation (provenance, split scope, privacy, and access policy) to the paper.

Thank you again for the constructive feedback, which has substantially improved the paper.

Reviewer_aj6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce EgoEasy Benchmark (EEB) to evaluate automated annotation of egocentric workflows. EEB contains 472 videos totaling 38.15 hours, covering diverse environments and activities. The authors develop a unified annotation and evaluation framework that assesses workflow structure, fine-grained semantic understanding, and annotation reliability. Experiments on seven video-language models reveal significant limitations in temporal localization, semantic grounding, and reliable annotation, highlighting the challenges of automated egocentric workflow understanding.

Critical strengths

1/ The paper is well-motivated, and evaluating the reliability of automated annotation is particularly important in the current field of egocentric video understanding.

2/ The authors evaluate seven representative video-language models and conduct additional validation studies to demonstrate the annotation consistency and reliability of the evaluation protocol.

3/ The authors conduct extensive evaluations across multiple VLMs, making the experimental results more convincing.

Critical weaknesses

1/ My primary concern is the contribution of the proposed benchmark. EEB contains only 38.15 hours of egocentric videos collected from relatively limited scenarios, while several existing human-annotated egocentric datasets are substantially larger. For example, the EgoMe dataset contains a longer total duration and also provides detailed annotations and temporal annotations covering the entire video span. Therefore, it would be helpful if the authors could further clarify the distinctive advantages of EEB.

2/ Another concern is the fairness of the baseline comparisons. Different models adopt different video preprocessing strategies, including spatial resolutions, frame sampling rates, and input frame limits, which may affect their performance, particularly on temporal localization tasks. I suggest that the authors consider investigating how these differences may influence the fairness of the comparisons.

3/ Although the authors provide a comprehensive evaluation protocol, Table 3 reports relatively high Review Rates for several models. It would be helpful if the authors could further discuss the implications of these unresolved claims and how they may affect the reliability of the evaluation results.

I will consider raising my rating according to the authors’ rebuttal and other reviewers’ comments.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to aj6N: EEB's Contribution Relative to EgoMe (W1)

Appreciate the suggestion to clarify EEB's contribution relative to other egocentric datasets, particularly EgoMe. EEB's primary contribution is not dataset scale, but a unified benchmark for evaluating complete, evidence-grounded workflow annotation. Its central question is whether a model can independently recover an entire observable workflow, including temporal structure, semantic roles, and evidential reliability.

1. EEB Compared with EgoMe

EgoMe is a valuable dataset for observation, imitation, and procedural video understanding. It provides detailed timestamped action descriptions, object attributes and relations, and imitation-verification annotations. We recognize these contributions rather than claiming that EEB introduces temporal annotation or fine-grained descriptions for the first time.

The distinction lies in the evaluation objective and the requirements imposed on generated annotations.

DimensionEgoMeEEB
Primary objectiveObservation/imitation understanding and procedural descriptionComplete, query-free workflow annotation
Temporal targetTimestamped action descriptionsGapless typed partition covering tasks, navigation, idle, and unidentifiable intervals
Semantic targetDetailed action descriptions and object relationsExplicit role-bound propositions with localized evidence
EvaluationProcedural captioning and imitation-related tasksJoint temporal, semantic, and evidential evaluation
ReliabilityImitation verification and caption qualityCandidate support, reference recall, confirmed unsupported claims, unresolved claims, and duplicates

EgoMe contains 82 h 46 min across observation and imitation videos, including 44 h 54 min of egocentric footage. Its individual clips span 3–60 seconds, mostly 10–25 seconds. EEB contains 38.15 hours across 472 recordings, averaging approximately 5 minutes each. These differences reflect complementary task settings rather than a claim of greater dataset scale. EgoMe, Sections 3.2–3.3, 4.6 and 6.

2. What EEB Contributes Beyond Dense Annotation

First, complete workflow recovery is an explicit requirement. Given only an untrimmed video and a shared schema, models must produce a gapless, temporally ordered annotation. Navigation, inactivity, and unidentifiable intervals cannot simply be omitted. An accurate caption of several salient actions is insufficient if substantial portions of the recording are missing or mistyped.

Second, EEB evaluates semantic relationships, not merely descriptive overlap. Actions, affected entities, tools, locations, and navigation endpoints are represented as role-bound propositions. Identifying an object is insufficient if its functional role is incorrect. Temporal eligibility and capacity-one matching further prevent misaligned or repeated claims from receiving inappropriate recovery credit.

Third, EEB explicitly evaluates evidential reliability alongside completeness. It distinguishes supported claims, confirmed unsupported content, unresolved assertions, duplicates, and missed required reference facts. This enables evaluation of both what a model successfully recovers and whether its additional claims are justified by evidence. EgoMe's imitation-verification labels address a different question: whether an imitated activity matches its demonstration.

Fourth, the evaluation target is independently validated. Five trained annotators achieved 90.2% Strict Episode F1 and 96.3% Proposition F1 on the development set, while a separate 1,000-decision held-out audit confirmed 94.3% of the frozen semantic judge's decisions.

3. Contribution and Positioning

EEB contributes a joint evaluation contract for temporal completeness, role-preserving semantic recovery, and evidential reliability. Its value is demonstrated by the seven-model evaluation: fluent high-level summaries coexist with poor strict temporal recovery, incomplete role grounding, and substantial unresolved content. These failures are difficult to characterize through procedural caption quality alone.

We will strengthen the Introduction, Related Work, and benchmark comparison to acknowledge EgoMe's annotation richness while emphasizing EEB's distinct evaluation objective and complementary contribution to reliable automated supervision.

AuthorsOpenReview ↗

Response to aj6N: Baseline Fairness and Paired Sensitivity Results (W2)

Thank you for raising baseline fairness. Frame sampling, spatial resolution, and generation protocols can affect temporal annotation. We used paired within-backbone controls for Qwen3-VL-8B and Qwen3-VL-32B at 0.5/1 FPS under Direct/Staged generation, matching global visual observations and recording inference resources.

1. Temporal Sensitivity

The table reports Strict Episode, Typed Segment, and Boundary F1 (%). The 1-FPS 8B-Staged and 32B-Direct rows reproduce published Table 3 results. The remaining rows are newly measured full-test ablations.

BackboneFPSProtocolStrict F1Typed F1Boundary F1
8B0.5Direct11.5326.4314.58
8B0.5Staged12.2228.6515.70
8B1.0Direct12.9627.1519.11
8B1.0Staged15.1033.2022.70
32B0.5Direct11.1729.3722.00
32B0.5Staged9.7827.8519.91
32B1.0Direct10.7030.9018.50
32B1.0Staged8.3825.8720.76

Generation protocol sensitivity. Qwen3-VL-8B benefits consistently from Staged generation across both sampling rates. At 1 FPS, staging improves Strict Episode, Typed Segment, and Boundary F1 by 2.14, 6.05, and 3.59 percentage points, respectively. This supports our original motivation for staging: mitigating complete-schema output truncation in models that struggle with Direct generation. Qwen3-VL-32B exhibits a different pattern. Direct generation achieves higher Strict Episode and Typed Segment F1 at both sampling rates, while Staged improves Boundary F1 at 1 FPS (20.76 versus 18.50). This indicates that protocol effectiveness depends on the backbone and evaluation metric, rather than staging being universally superior.

Frame sampling sensitivity. Increasing FPS improves all three temporal metrics for 8B under both protocols, whereas 32B shows mixed changes. For example, 32B Direct improves Typed Segment F1 but decreases Strict Episode and Boundary F1. Higher sampling density therefore does not guarantee uniformly better temporal annotation.

These results are consistent with the original baseline configurations: 8B uses Staged generation to address output-completion difficulties, while 32B supports Direct complete annotation. The additional evaluations clarify how alternative configurations affect performance under the same benchmark protocol.

2. Controlled Inputs and Computational Resources

Within each backbone/FPS comparison, Direct and Staged global passes used identical source-frame indices and visual preprocessing, verified through run manifests. Checkpoints, greedy decoding, repetition penalty 1.0, and XGrammar 0.2.7 were fixed. The global visual budget was capped at 25,165,824 pixels and 512 frames. Staged detail calls operated only on model-predicted intervals, without access to reference episode boundaries.

At 1 FPS, recorded mean resource consumption was:

Backbone / ProtocolCallsInput tokens
8B Direct1.0025,946
8B Staged5.8894,398
32B Direct1.0025,946
32B Staged7.75104,733

Direct generation used a 32,768-token output cap; Staged used per-call limits of 2,048/8,192/8,192 for summary, timeline, and detail generation.

Matched global observations do not imply identical total compute, since Staged additionally processes predicted local windows. Moreover, the fixed pixel budget couples temporal sampling with spatial resolution. Reporting these resources makes the operating differences explicit.

3. Implications for Baseline Fairness

EEB evaluates complete annotation systems under a unified schema, common reference annotations, and consistent scoring rules. Its objective is to assess end-to-end workflow annotation quality, rather than isolate backbone performance under identical inference budgets.

The full-test comparisons establish three observations. First, generation protocol affects temporal performance, with Staged benefiting 8B more consistently than 32B. Second, increasing frame sampling does not uniformly improve annotation quality. Third, the performance of a configured system reflects the interaction of backbone capability, visual preprocessing, and generation strategy.

Importantly, these findings support the interpretation of our original baseline results. Staged generation was introduced to improve the feasibility of complete structured annotation, not to guarantee higher scores on every temporal metric. Different models may therefore benefit from different operating configurations. By applying the same annotation requirements and evaluator across configurations, EEB provides a consistent basis for comparing complete annotation systems while making their computational and preprocessing differences transparent.

AuthorsOpenReview ↗

Response to aj6N: Implications of High Review Rates for Reliability (W3)

We agree that high Review Rates require careful interpretation. EEB separates unresolved adjudication from confirmed factual errors; these signals should be interpreted jointly with reference recovery and temporal accuracy.

1. What High Review Rates Mean

The published 39.8–73.3% video-macro Review Rates quantify claims that the frozen evaluator cannot resolve using eligible reference evidence. They are not hallucination rates. A claim may describe a real event but fail temporal or episode-type eligibility, or add specificity that an eligible reference cannot verify. Conversely, a genuine core action may still contain incorrect roles, details, or temporal extent.

This distinction is supported by our stratified source-video audit of 500 unresolved claims across seven models. Of these, 182 (36.4%) contained semantically supported content, and another 144 (28.8%) contained a supported core event with role, detail, or temporal errors. Thus, 326/500 (65.2%) contained some video-supported content, although this does not imply fully correct annotations. The audit demonstrates why unresolved assertions should not automatically be counted as fabricated events.

2. Joint Interpretation of Reliability Metrics

Grounded Precision reflects candidate-support credit under EEB's evidence-adjudication protocol; Reference Fact Recall measures recovery of required reference facts. Confirmed Hallucination Rate counts adjudicated unsupported or contradicted claims, while Review Rate identifies unresolved assertions. The conservative upper unsupported-fact diagnostic includes both confirmed unsupported and unresolved claims.

These measures are complementary. A low Review Rate alone does not establish annotation quality: a model may avoid uncertain details while missing required events or producing inaccurate temporal structure. Conversely, a detailed prediction may contain a supported core event even when some additional assertion remains unresolved. Reference recall and temporal metrics make these distinctions visible.

For a video with a fixed candidate denominator, adjudicating one unresolved claim as supported reduces Review Rate and the upper unsupported-fact diagnostic by the same amount. Adjudicating it as unsupported decreases Review Rate and increases Confirmed Hallucination Rate by that amount, leaving their sum unchanged. Unverifiable assertions remain unresolved. The published metrics are independently video-macro averaged.

3. Implications for EEB Evaluation

EEB deliberately reports confirmed unsupported content, unresolved assertions, reference coverage, and temporal structure separately, rather than collapsing uncertainty and error into a single binary score. Source-video support of an action does not automatically validate its episode type, boundaries, or required-reference recall; supported extras follow the documented extra-evidence policy.

We will clarify this joint interpretation in the Discussion. High Review Rate is a transparent indicator of remaining adjudication uncertainty, not evidence that EEB conflates unresolved claims with hallucinations.

Reviewer_b7vNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EEB is a benchmark for query-free, complete annotation of egocentric workflows: 472 first-person videos, split into 48 public development videos and 424 hidden-reference test videos. Given only the video and a JSON schema, a system must output global summary facts, a gapless typed episode timeline, role-bound atomic facts, and evidence windows. Scoring combines temporal structure, fine-grained semantics, and reliability. Seven zero-shot VLMs peak at 15.1 Strict Episode F1 and 25.7 proposition F1; five trained annotators re-annotating the development set reach 90.2 and 96.3; 1,000 held-out judge decisions are human-confirmed at 94.3%.

Critical strengths
  1. A useful target with a carefully specified protocol. Requiring a gapless typed partition that includes navigation, idle and unidentifiable intervals, role-preserving proposition matching with capacity-one assignment, duration-adaptive boundary tolerances, and separate accounting of confirmed-unsupported, unresolved and duplicate claims is a meaningful step beyond query-conditioned QA and dense-captioning evaluation.
  2. Substantive validation of the target and the judge. Five annotators, blind to the frozen references, reproduce the development-set annotations at 90.2 Strict Episode F1 and 96.3 proposition F1, which shows the target is well defined under the guidelines. The per-category human confirmation of 1,000 held-out judge decisions (94.3%) also goes beyond an unvalidated LLM judge.
Critical weaknesses
  1. The semantic-judge instructions explicitly distinguish candidate support from reference coverage. However, the final scoring uses their minimum, wij=min⁡(sijc,sijr)w_{ij}=\min(s^c_{ij},s^r_{ij}), as the shared matching credit for both precision and recall. Consequently, Grounded Precision may be reduced by incomplete reference coverage even when the candidate claim is fully supported. The authors should quantify how frequently these judgments differ and examine whether scoring them separately changes the results or model rankings. This matters because Grounded Precision is the Reliability dashboard’s headline evidence that generated claims are supported.
  2. The results show the complete task is hard, but do not separate why. Video-macro Review Rates of 39.8–73.3% leave a substantial share of claims unresolved, and the paper does not estimate how much of this is missing reference detail, temporal ineligibility, or claims that remain unverifiable after watching the video. The consequence is that the paper reads these numbers as evidence of hallucination and weak role grounding. An unresolved claim may be true but unannotated, true but placed in a misaligned episode, or false; until these are told apart, the Reliability dashboard describes the limits of the evaluator as much as the limits of the models.
Questions for authors

Scoring: How often do candidate support and reference coverage judgments differ for each model? What happens to precision, recall, F1, and model rankings if candidate support and reference coverage are scored separately, rather than using their minimum for both metrics?

Unresolved claims: Please investigate and quantify the main sources of unresolved claims, potentially including incomplete reference coverage, temporal alignment failures, granularity mismatches, unsupported predictions, or genuinely ambiguous video evidence. A stratified human audit against the source videos should accompany this analysis and report how these factors affect the reliability metrics.

Data provenance and release: Were the videos newly collected or sourced from existing datasets? Please clarify their provenance, recording permissions, consent and privacy safeguards, and the licences governing video and annotation distribution.

4 comments
AuthorsOpenReview ↗

Response to b7vN: Directional Scoring and Ranking Robustness (W1/Q1)

Thank you for identifying an important distinction: candidate support measures whether a generated claim is supported by evidence, whereas reference coverage measures how fully it recovers a required reference fact. We agree that Precision and Recall should reflect these directions separately.

1. Directional Scoring Formulation

The submitted paper defines shared pair credit:

wij=min⁡(sijc,sijr).w_{ij}=\min(s^{c}_{ij},s^{r}_{ij}).

Here, sijcs^{c}_{ij} denotes candidate support and sijrs^{r}_{ij} denotes reference coverage.

The adjudication protocol asks the judge to assess both directions separately. In the original scoring implementation, however, coverage used for metric computation was derived from support: entailed → covered; partial → partial if no entailed judgment exists; otherwise → missed. For allocated positive matches, the shared minimum therefore equaled candidate-support credit. The reported Grounded Precision was not reduced by incomplete independent reference coverage.

For the revised analysis, we retained candidate-support judgments and replaced support-derived coverage with independently adjudicated reference-coverage labels from retained judge records, supplemented where necessary by additional model-assisted adjudication. Model outputs, temporal/type eligibility, capacity-one matching, denominators, and supported-extra treatment remained fixed.

For video vv, let MvM_v be its fixed matching, CvC_v its non-masked candidate propositions, and GvreqG_v^{\mathrm{req}} its required reference propositions. Let ci,rj∈0,0.5,1c_i,r_j\in\\{0,0.5,1\\} denote support and independent coverage credits. We compute:

Pvdir=∑(i,j)∈Mvci+Tvextra∣Cv∣.P_v^{\mathrm{dir}}=\frac{\sum_{(i,j)\in M_v}c_i+T_v^{\mathrm{extra}}}{|C_v|}.

Rvdir=∑(i,j)∈Mv,;j∈Gvreqrj∣Gvreq∣.R_v^{\mathrm{dir}}=\frac{\sum_{(i,j)\in M_v,\\;j\in G_v^{\mathrm{req}}}r_j}{|G_v^{\mathrm{req}}|}.

F1,vdir=2PvdirRvdirPvdir+Rvdir.F_{1,v}^{\mathrm{dir}}=\frac{2P_v^{\mathrm{dir}}R_v^{\mathrm{dir}}}{P_v^{\mathrm{dir}}+R_v^{\mathrm{dir}}}.

Here, TvextraT_v^{\mathrm{extra}} preserves precision credit for adjudicated supported extras. Unmatched candidates remain in the precision denominator; unmatched required references receive zero recall credit; optional references are excluded from recall. F1 is zero when both terms are zero. P, R, and F1 are independently video-macro averaged.

2. Directional Rescoring

We recomputed directional P/R/F1 over all videos among all seven models. Credit disagreement is measured only on allocated, temporally eligible pairs with valid reciprocal links and positive judgments in both directions.

ModelDisagreementOriginal P/R/F1Revised P/R/F1
GPT-4.123.93%31.59/27.71/25.7531.59/23.56/23.61
Gemini 3 Flash19.78%21.29/20.58/19.5121.29/19.79/19.18
Qwen3-VL-32B21.24%16.10/20.84/17.0916.10/16.77/15.60
GLM-4.6V-Flash36.14%8.05/16.44/9.008.05/12.13/8.11
Qwen3-VL-8B33.72%5.94/21.54/8.435.94/17.74/7.90
InternVL3.5-8B35.94%4.15/11.54/4.274.15/8.95/3.89
LongVA-7B-DPO4.28%3.05/3.98/2.573.05/3.81/2.48

All scores are percentages. Disagreement is a directional-credit diagnostic, not a hallucination rate.

3. Findings and Manuscript Revision

Precision is unchanged for all seven models because support credits and matches were fixed. Recall and F1 change because independent coverage replaces support-derived coverage, not because removing min alone lowers scores. Directional disagreements range from 4.28% to 36.14%. Crucially, all seven models retain their original F1 ranking, and Hierarchy metrics are unaffected. Our main conclusion that current models struggle with complete, evidence-grounded workflow annotation remains unchanged.

In the revision, we will correct the §4.3, Appendix A.2.2 to distinguish the judge's independent coverage judgments from the support-derived labels actually used in the original scoring implementation, specify the revised directional formulas, and update the reported results accordingly.

AuthorsOpenReview ↗

Response to b7vN: Unresolved Claims: Human Audit and Reliability (W2/Q2)

We agree that unresolved claims are not necessarily hallucinations. The frozen evaluator assigns an unresolved label when eligible reference evidence cannot establish support or contradiction. We therefore reviewed source videos to distinguish factual support from temporal, structural, and role-level errors.

1. Human Audit of 500 Claims

We conducted a stratified audit of 500 unresolved claims, covering all seven models, navigation/non-navigation episodes, and reference-eligibility states. Reviewers checked video evidence, claimed intervals, episode types, and semantic roles. Each claim received one principal finding; secondary issues could coexist.

Principal findingClaimsSample share
Semantically supported content18236.4%
Event absent in claimed interval9819.6%
Incorrect navigation classification5310.6%
Supported core; incorrect roles, details, or extent14428.8%
Unverifiable model-added detail234.6%
Total500100%

326/500 claims (65.2%) contained video-supported semantic content or core events. Among these, 182 had supported semantic content, while 144 contained a valid core event with incorrect roles, details, or temporal extent. This distinction separates semantic support from complete annotation correctness. The 98 absent-event findings concern the claimed interval rather than necessarily the entire recording. Navigation errors can also describe real turning or reaching with an incorrect episode type, while unverifiable details remain uncertain rather than automatically classified as hallucinations.

2. Representative Failure Modes

MechanismExampleInterpretation
Temporal mismatchElevator exit aligned with elevator rideReal event, wrong interval
Episode-type errorTurning/reaching labeled navigationLocal motion is not travel between work areas
Object/role errorSteel-wool scrubber called sponge; pushed cart labeled toolCore activity may be real, but asserted roles are wrong
Missing event in intervalCloth retrieval, door closure, two-plate cleaningNo supporting action at predicted time
Temporal extentShort mopping event assigned broad intervalDuration overstated
Unverifiable extraModel predicts “floor 3”Neither reference requires nor video establishes this number

The floor-number example illustrates unverified model-added specificity rather than a ground-truth error. The reference describes pressing an elevator button, riding the elevator, and entering the lobby without asserting an exact floor. Since the video does not establish “floor 3,” the evaluator conservatively leaves this additional detail unresolved rather than declaring it false.

Eligibility and factuality are distinct. Temporal overlap and episode-type compatibility determine which reference units the semantic judge can consider. A genuine event may remain unresolved when its relevant reference is ineligible, or when an eligible reference lacks the detail needed to verify a model-added assertion.

Similarly, a correctly identified core action may contain an incorrect object, role, or temporal qualifier. A reference can accurately describe a broader activity without enumerating every supported subaction. These examples demonstrate how temporal eligibility, reference granularity, and prediction-level errors can contribute to unresolved outcomes. The audit categories capture observed findings, while underlying mechanisms may overlap.

3. Implications for Reliability

The published 39.8–73.3% Review Rates measure claims requiring further adjudication, not observed hallucination prevalence. EEB separately reports confirmed unsupported/contradicted claims, unresolved claims, and duplicates. Its conservative upper unsupported-fact rate also includes unresolved claims and therefore should not be interpreted as a measured hallucination rate.

The metric implications follow directly from the evaluator's accounting. For a video with NC,vN_{C,v} non-masked candidates, resolving one unresolved claim as supported reduces both Review Rate and the upper unsupported-fact rate by 1/NC,v1/N_{C,v}. Resolving it as unsupported transfers the same amount from Review Rate to Confirmed Hallucination Rate, leaving their sum unchanged. Genuinely unverifiable claims remain unresolved.

Because EEB reports video-macro metrics, the claim-level audit findings provide complementary evidence for interpreting the published scores rather than a direct adjustment to those aggregate values. Supported extras follow the documented evidence policy, and video-level support alone does not grant correct episode boundaries or required-reference recall credit. These distinctions reinforce EEB's separate treatment of evidential reliability, temporal correctness, and reference recovery.

AuthorsOpenReview ↗

Response to b7vN: Data Provenance, Privacy, and Release Policy (Q3)

Thank you for raising these questions about dataset provenance and redistribution rights, particularly for hotel rooms, public restrooms, and other indoor settings. We distinguish collection authorization, privacy safeguards, and downstream distribution rights.

1. Data Provenance and Recording Authorization

All 472 EEB videos were newly commissioned and collected for this benchmark; none were reused or repackaged from existing datasets. Paid collectors recorded everyday first-person workflows across 10 environment families and 13 workflow families. The recordings were annotated under EEB's unified schema and divided into 48 development videos and 424 hidden-reference test videos.

Collectors signed recording-consent and video-authorization agreements covering collection, research use, and controlled distribution of privacy-processed recordings to approved researchers. Recording permissions were also obtained from relevant site managers or other authorized parties, including for the hotel-room and restroom settings identified by the reviewer.

These agreements establish the necessary authorization for both collection and controlled research distribution. The downstream Data Use Agreement provides additional restrictions on researchers' use of the released data.

2. Privacy Safeguards and Data Processing

EEB has completed privacy masking of sensitive visual information in the prepared recordings. The release copies use privacy-processed footage rather than unredacted originals. Privacy safeguards address identifiable individuals and sensitive information in the recording environments, including hotel rooms, restrooms, and public indoor areas.

Privacy protection also applies to accompanying annotations and metadata, which must not disclose identifying information removed from the footage. Identifying or sensitive audio is excluded or redacted in the distributed material. Unredacted original recordings are not part of the research release.

Because EEB evaluates temporally localized episodes and evidence windows, privacy processing preserves the correspondence between video timestamps and annotations. Material whose supporting evidence is obscured by privacy masking is excluded from scoring or release, rather than being evaluated against unavailable evidence.

This maintains consistency between the privacy-protected recordings and the evidence used for benchmark evaluation.

3. Research Access and Data Use Agreement

EEB adopts controlled, application-based research access under an EEB Research Data Use Agreement, rather than unrestricted public redistribution. The agreement governs downstream use of released recordings and annotations within the scope of the collection authorizations.

ItemRelease policy
AccessResearcher application, identity/affiliation verification, and agreement acceptance
Permitted useNon-commercial research, including model development and evaluation
RedistributionNo redistribution or sublicensing without separate authorization
PrivacyNo re-identification, identity linkage, or reversal of masking
SecurityReasonable storage/access safeguards and reporting of privacy issues
AttributionEEB citation and attribution required
Test integrityHidden test reference annotations remain withheld

The 48-video development references are designated for controlled research access, while the 424-video test references remain hidden to preserve evaluation integrity.

These established collection authorizations and privacy safeguards, together with the controlled-access policy, provide a documented framework for permissioned, privacy-protected research use while preserving hidden-reference evaluation integrity.

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer b7vN,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards

Comments on the paper

1 comment

Dear Reviewers,

Authors posted their responses to your queries. Please read these messages and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.