acceptodds
Under review as a conference paper at ICLR 2027

EgoEasy Annotator: Evidence-Governed Task-Navigation Timelines from In-the-Wild Egocentric Video

Abstract

Human egocentric video could scale robot learning, but an untrimmed recording is not a ready-made training example. Recovering tasks, operations, and movement between work areas requires reading fleeting or obscured clues without inventing details or missing events. We introduce EgoEasy Annotator (EEA), which checks proposed descriptions against the video and separately searches for missed events. When interpretations would change the annotation, it asks what the video must show to distinguish them and opens suitable new views. A video model describes these views without seeing the descriptions being checked. A controller retains a specific description when the evidence distinguishes it from the alternatives, or shared details when only those are supported. Unclear details that still affect the annotation receive local human review; unsupported guesses are omitted. Thus, uncertain object or work-area names need not erase supported actions or navigation. Each newly found event undergoes the same checks before entering one source-timestamped task–navigation timeline. Averaging HD-EPIC and EgoEasy-Bench scores, EEA improves episode segmentation F1 by 12.34 percentage points over the best baseline, while recovering more events with fewer unsupported claims. In a 36-video study, it saves 32.14% of human review and correction time over the development-selected baseline while meeting the preset requirements for average final quality across both benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video, Long Video Understanding, Video Annotation, Temporal Understanding, Task Segmentation, Navigation Understanding, Robot Learning, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tZGSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EgoEasyAnnotator (EEA), which builds a timestamped task–navigation timeline from untrimmed egocentric video. EEA first records visible contact, motion, and state changes with their timestamps. The claim closure stage then checks ambiguities that could change the annotation. The observation call sees the selected frames or clips but not the candidate labels or the initial annotation. A rule-based controller keeps an exact description, falls back to a supported broader description, or sends the unresolved detail for local human review. A separate coverage closure pass searches for missed events, and each new finding goes through the same checks before the final timeline is assembled. On 488 videos, EEA performs better than the comparison baselines on the main annotation metrics. In a 36-video human study, correcting EEA outputs took 32.14% less time than correcting Dense-Group outputs.

Critical strengths
  1. The main idea is useful. The paper separates two common failures in long-video annotation: a description may lack enough evidence, and the initial annotation may miss an event. Claim closure checks the first problem, while coverage closure searches for the second. New findings go through the same evidence checks before they enter the final timeline. This lets EEA control unsupported descriptions without giving up coverage.
  2. The experiments support the method’s practical effect. On HD-EPIC and EgoEasy-Bench, EEA improves episode segmentation F1 by 12.34 percentage points and Event Recall by 13.52 points, while lowering the upper bound of Unsupported Field Rate by 7.67 points. The ablations suggest that claim checking and omission search address different failure modes. The human study also reports 32.14% less correction time for EEA outputs than for Dense-Group outputs.
Critical weaknesses
  1. The generator and evaluator share the same backbone. Both EEA and the automatic evaluator use Qwen3.8-27B. The evaluator extracts required facts and operations from the references, checks visual support, and matches predicted fields and events. EEA may therefore benefit from the evaluator’s model-specific preferences. The human calibration is useful, but its sample is too small to rule out shared errors.
  2. Unexplained numerical instability limits confidence in the empirical results. On identical test videos, Dense-Group’s HD-EPIC Event Recall changes from 33.51% and 30.81% to 98.92% across seeds, while EgoEasy recall drops from 68.63% to 1.96% (Table 43). These opposing changes yield an aggregate standard deviation of only 0.68 percentage points (Table 42), concealing substantial instability within each benchmark. The authors acknowledge the variation but do not explain whether it arises from generation, matching, or evaluation. Moreover, the main generator and semantic evaluator use the same Qwen checkpoint; the pooled 240-assertion calibration does not establish that comparative gains survive independent assessment. Please audit the extreme runs using saved predictions and human judgments, provide the corresponding matching decisions and counts, and verify the relative gains with an independent evaluator.

Without this diagnosis, I have low confidence in the reliability of the reported comparisons.

Questions for authors
  1. Could the authors compare EEA with Dense-Group and Dense-Group+SR at similar token or inference budgets, to show how much of the gain comes from the proposed design?
  2. EEA stops visual verification and requests human review after reaching a preset inspection budget. Could the authors explain how these budgets are chosen and how changing them affects annotation quality and the amount of human review?
  3. The figures are not clear, seem like purely AI-generated.
  4. See weaknesses.
7 comments
AuthorsOpenReview ↗

Response to tZGS (1/3): diagnosing the seed anomaly (W2)

Thank you for identifying the mirrored Dense-Group results and specifying the audit needed to assess them. You were right that a small aggregate standard deviation could not justify those benchmark-level extremes. We traced the saved outputs, audited the matching decisions with humans, and repeated scoring and generation. Part 2 addresses independent evaluation; Part 3 addresses resources, stopping budgets, and figures.

Cause and corrected counts

For Dense-Group (DG), seed 20260918, the result loader selected non-final attempts instead of the designated final outputs:

BenchmarkIncorrectly selected outputCorrect final outputEvent matches: old -> corrected
HD-EPICPre-grouping recovery artifact, attempt 0Attempt 2183/185 -> 60/185
EEBIncomplete partial output, attempt 0Attempt 31/51 -> 35/51

The HD file was an overcomplete pre-grouping recovery artifact concatenating window-level DG observations. It retained event descriptions before the required grouping stage; evaluating those observations is not evaluating the completed DG timeline. Here, "final" means the completed grouping output in the common task/navigation timeline schema. The recovery concatenation was not an eligible final timeline.

The final output for each run was designated before that run was scored. The correction restores those assignments; evaluation scores were not used to choose the final attempt.

The error was the association between the stated run and the output passed to evaluation. We take responsibility for it. Selecting the designated final outputs and their saved judgments reconstructs the corrected counts:

DG Event RecallSeed 20260916Seed 20260917Seed 20260918Across-seed SD
HD-EPIC33.51% (62/185)30.81% (57/185)32.43% (60/185)1.36 pp
EEB68.63% (35/51)68.63% (35/51)68.63% (35/51)0.00 pp

Corrected seed-20260918 benchmark average: 50.53% (published 50.44%); three-seed mean: 50.44% (published 50.41%), with SD 0.68 pp. The opposing errors had largely canceled in the average. We will correct Tables 42-43 and retain separate benchmark results for every seed.

Human audit of the extreme outputs

Humans checked all 236 reference-event matching decisions on the two wrongly selected outputs:

OutputMachine decisionsHuman re-check
HD-EPIC183 matches: 179 unique candidates, 4 fixed tie-breaks. Two nonmatches: 1 missing candidate, 1 semantic failure.181 matches confirmed, 2 ambiguous; both nonmatches confirmed
EEB1 match. Fifty nonmatches: 47 missing candidates, 2 timing failures, 1 semantic failure.Match confirmed; 48 nonmatches confirmed, 2 ambiguous

Thus 232/236 decisions were confirmed and four remained ambiguous, with none reversed. The audit confirms that most matching events were present in the mistakenly selected artifacts; it does not make those intermediate artifacts eligible final timelines. The local matching ambiguities do not explain the mirrored extremes.

Replays and audit scope

CheckResult
Five fresh scoring replays: support labels reproduced1,997/2,000
Five fresh scoring replays: matching decisions reproduced1,994/2,000
Ten fresh DG generations on the same 12 videosHD: 57-63/185 matches; EEB: 34-36/51

These checks support the diagnosis of this DG anomaly. The revised tables retain each method's observed benchmark-level seed variation; the DG correction does not establish universal seed stability.

We re-checked file identity for every other entry of Tables 42-43, including all EEA seeds, and for all main-table runs; no other entry was affected. The next reply tests the comparative gains independently of the original Qwen judge.

AuthorsOpenReview ↗

Response to tZGS (2/3): independent model and human assessment (W1, W2)

Your key question is whether EEA's comparative gains survive a change of evaluator. We therefore tested the method differences with a different-family judge and direct human assessments.

DG is Dense-Group; SR is DG with two self-refinement rounds. Differences below are EEA minus control, in percentage points (pp). Event counts confirmed reference-event matches; UFR is the conservative rate of unsupported or unresolved core assertions, so lower is better.

Independent Gemma evaluation and human Event Recall

We rescored frozen outputs on 60 videos, 30 per benchmark, using google/gemma-3-27b-it with deterministic decoding and method identities hidden. Its configuration was selected on 240 newly labeled development assertions: 40 per method per benchmark from 16 development videos, then frozen before any test scoring. These are separate from the paper's original calibration sample. Development agreement with humans was 217/240 (90.42%), with similar rates across methods: EEA 91.25%, DG 88.75%, and SR 91.25%.

Humans also adjudicated Event Recall on the same 60 videos, covering 1,188 reference events.

Assessment, difference [95% CI]Versus DGVersus SR
Gemma: Event Recall+12.67 [7.02,18.32]+12.30 [6.62,17.98]
Gemma: UFR-8.39 [-13.74,-3.04]-6.76 [-12.10,-1.42]
Human: Event Recall+12.62 [7.34,17.90]+12.27 [6.95,17.59]

The event-recovery benefit is therefore supported separately by Gemma and humans, with each interval excluding zero. Under Gemma, necessary-field F1 and no-edit episode recall also improve: versus DG, +9.16 [3.43,14.89] and +9.69 [6.35,13.03] pp, respectively.

Direct human support audit: 7,052 assertions on 36 videos

Humans labeled assertions as supported, contradicted, insufficiently evidenced, or unresolved. Conservative UFR counts the last three categories; unresolved items are not called confirmed errors.

MethodUnsupported or unresolved / audited assertionsHuman conservative UFR
EEA247/2,4859.94%
DG436/2,28119.11%
SR402/2,28617.59%

These are pooled audit proportions. EEA's human UFR is lower by 9.17 pp versus DG and 7.65 pp versus SR. Even assigning every unresolved EEA assertion to unsupported and every unresolved control assertion to supported leaves sample advantages of at least 4.40 pp and 2.79 pp. These last numbers bound interpretation of unresolved labels within this sample; they are not sampling confidence intervals.

Conclusion. EEA's Event gains remain under independent Gemma and human assessment. Its support gains remain under Gemma and are corroborated by the lower observed UFR in the human assertion audit. These complementary assessments show that the observed advantages are not confined to the original Qwen judge.

AuthorsOpenReview ↗

Response to tZGS (3/3): token controls, budgets, and figures (Q1-Q3)

Q1. Do the gains remain at comparable token use?

Using the same generator backbone, we increased baseline resources toward EEA's cumulative token budget on the same respective panels: DG on 60 videos (30 per benchmark), SR on 24 (12 per benchmark).

DG used 60-s/2-fps windows with 25% overlap and fixed additional grouping/consensus calls. SR retained its original DG frames and completed two full critique-revision rounds over every window, followed by fixed same-frame consolidation calls. The final call was capped at the remaining token allowance. Totals count input (text and visual) plus output across all ordinary, repair, and failed calls.

Control / EEATotal tokens, millionsGPU-hoursSummed wall time, hours
DG / EEA, 60 videos123.7 / 123.793.20 / 96.9052.59 / 57.79
SR / EEA, 24 videos49.5 / 49.537.70 / 38.8020.45 / 23.12

This compares total-token use, not equal FLOPs or runtime. Both the token comparisons and the budget sweep use the same development-frozen Gemma judge for semantics; segmentation matching is deterministic. UFR includes unsupported and unresolved assertions; NEER is the recall of reference episodes needing no core edit or substantive review.

EEA minus control, pp [95% CI]DG, 60 videosSR, 24 videos
Segmentation F1+3.87 [-1.66,9.40]+9.86 [3.30,16.42]
Necessary-field F1+2.83 [-1.24,6.90]+5.54 [0.44,10.64]
UFR-3.88 [-7.62,-0.14]-4.73 [-9.20,-0.26]
Event Recall+3.73 [0.31,7.15]+8.61 [3.55,13.67]
NEER+4.76 [2.21,7.31]+6.46 [3.38,9.54]

At these totals, extra baseline tokens do not fully remove the event, support, and no-edit-episode gains. DG's Segmentation/Field intervals include zero. These are pointwise intervals, without an additional multiplicity adjustment.

The original Qwen-scored component controls explain contributing mechanisms: targeted versus equally many uniform views lowers UFR by 4.11 pp and raises Event by 4.95 pp on full EEA's fixed investigation schedule. The 2x2 ablation separately shows verification reducing UFR by 11.08 pp and coverage raising Event by 11.14 pp when each is added alone. EEA couples these obligations: newly found events must pass the same support checks.

Q2. Budget choice and sensitivity

The fixed caps depend on source length, not test scores. With n scheduling windows (each at most 30 seconds and 900 native frames), s=max(1,ceil(n/24)). Caps include 48 x s further requests, 24 x s investigations with at most two requests each, and two coverage rounds with at most 4 x s requests each. These are ceilings, not mandatory consumption.

On 12 videos, six per benchmark, low/default/high runs share the initial observations and draft, with unchanged evidence rules. Actual token use is 0.62x/1.00x/1.35x default.

Change, pp [95% CI]Default minus lowHigh minus default
Event Recall+4.64 [1.02,8.26]+1.66 [-0.72,4.04]
UFR-4.62 [-7.88,-1.36]-1.02 [-3.11,1.07]
Video duration flagged for review-8.64 [-13.40,-3.88]-2.96 [-6.10,0.18]

Budget-terminated videos fall from 8 to 2 to 0; unresolved questions from 24 to 7 to 1. Default improves quality and reduces flagged duration versus low; further high-budget gains remain uncertain. Exhaustion stops inspection and retains local review for important unresolved details without relaxing the evidence rules.

Q3. Figure clarity and provenance

We agree that Fig. 1 should be clearer. Its stages, arrows, and text are our own and follow Sec. 3; we assembled it after discussing layout with an AI assistant. The small thumbnails, however, are AI-generated illustrations, not benchmark frames. The caption and AI-use statement should have said so; we will correct both.

We will replace the thumbnails with real, timestamped EEB frames, label the evidence passed between stages, and show that candidate labels are hidden from the observer and newly found events return to the same verification gate. The diagram will illustrate exact, supported broader, and local-review decisions. We will also simplify Fig. 2 and enlarge its labels. Anonymous EEB samples can be provided.

Thank you again. We hope the concrete error diagnosis, independent model/human assessments, and comparable-token controls address the reliability concerns underlying your recommendation. We would appreciate reconsideration of the soundness assessment and overall recommendation in light of this evidence.

Reviewer_YrXzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work turns untrimmed videos into navigation dataset for robot learning. This is achieved by time stamping task episodes in an automated pipeline that could flag a request for human reviewing. The video observer is a model that is fed the frames of the video and describes contact, motion and state change to propose episodes and claims. A claim is a proposed detail in the pipeline that is linked to a start and end time while an episode is an interval within the video that has a start and end time. A rule based controller keeps the claims that meet evidence rules or flag for human review if the difference would change an action or a boundary. It is noted that paraphrases alone do not require human review. The annotation requirement is divided into claim closure and coverage closure. A claim closure checks what the claim description says and compare it what is evident visually. A coverage closure is to ensure that there are no missing details in the video in the claim made. On 428 EEB files and 60 HD-EPIC recordings, EEA outperforms five baselines on 5 metrics reported as the average on both benchmarks.

Critical strengths
  • The claim closure and the coverage closure breaks down the annotation criteria into its main two subcomponents to ensure the pipeline produces descriptions aligning with the video
  • Ambiguous, not visible and absent are kept distinct which improves handling of uncertainty in this pipeline
  • Requesting a new view from a separate model call to investigate a missing operation inside a long task episode strengthen the annotations with longer tasks the model is more likely to miss key temporal information.
  • Not including description or proposed labels reduces bias in model’s investigate of missed events
  • Ablations demonstrates the value of both verification and coverage on three different metrics which empirically supports the claims of the paper
  • Strong agreement between human and evaluator labels (218/240) with reporting of disagreements strengthens the reliability of the automated pipelines
Critical weaknesses
  • The pipeline is heavily automated. Despite flagging cases for human reviews, this is still based on a cascade of automated stages which might miss a significant portion of cases where human verification is required.
  • Only one episode is covered in b_k, b_k+1 when there could be overlapping episodes which will not be captured in this setup
  • Both generators and evaluators are Qwen models. If there is a certain bias or limitation of the Qwen family of models, it will not be caught in the evaluation.
  • Loose definition of state change and how it correctly correspond to the target goal. How is a state change evaluated to exactly match what the target goal was? There could be a state change that results in the opposite of the desired task’s goal e.g. spilling water rather than pouring water are very similar states changes that might vary in correspondence to a given goal for a task
  • The definition of contact, motion and state change are independent. Could there be a state change during a motion e.g. flattening dough or without contact e.g. where water boiling is evaporating
  • Despite the fixed ontology in table 6 being reasonably comprehensive for kitchen domains, the fixed setup makes it challenging to generalise this framework to more diverse scenarios e.g. object interaction could be drilling into an object or kicking an object. While these could be included in “handle object”, that term is too vague compared to more precise ones such as close a container or door.
Questions for authors
  • How do the results differ when using different models for the evaluator than the generator?
  • Have you considered human reviewing of some samples that did not get flagged by the pipeline to ensure the reliability of the automation?
  • Are there qualitative results for the annotations of overlapping episodes? e.g. if each hand is performing different actions simultaneously towards the same task
  • Have you considered using the dense narrations of HD-EPIC instead of your annotations pipeline to construct the episodes and comparing the HD-EPIC benchmark with your pipeline to using the original narrations?
  • How does EEA handle state changes that occur without direct contact or are delayed from the wearer’s action (e.g., water boiling after the hob is turned on)? Could you share qualitative examples of such cases and whether they were accepted, broadened, or sent to review?
  • How would the results change if a different model is used as an evaluator that is not from the same family as the generator?
  • How does the fixed broadening ontology generalize beyond the current domains? How can it be scaled for future work where it could be common to encounter new scenarios that are not covered within the initial fixed broadening ontology?
7 comments
AuthorsOpenReview ↗

Response to YrXz (1/3): independent assessment and unflagged errors (W1, W3; Q1, Q2, Q6)

Thank you for the careful reading and for recognizing the separate claim/coverage checks, label-hidden observation, and new-view acquisition. Your questions led to independent model and human assessment, direct audits outside flags, and a narration comparison. Part 2 addresses states and concurrency; Part 3 addresses the ontology and narrations.

Does the advantage survive a different-family evaluator?

We evaluated frozen Dense-Group (DG), DG with self-refinement (SR), and EEA outputs on 60 videos, 30 per benchmark, using google/gemma-3-27b-it with method identities hidden. The configuration was selected on 240 newly human-labeled assertions from 16 separate development videos, then frozen before test evaluation.

Humans also adjudicated Event Recall on the same 60 videos, covering 1,188 reference events. UFR denotes the conservative unsupported-field rate, including unresolved judgments; lower is better. Differences are EEA minus control, in percentage points (pp).

Assessment, difference [95% CI]Versus DGVersus SR
Gemma: Event Recall+12.67 [7.02,18.32]+12.30 [6.62,17.98]
Gemma: UFR-8.39 [-13.74,-3.04]-6.76 [-12.10,-1.42]
Human: Event Recall+12.62 [7.34,17.90]+12.27 [6.95,17.59]

A separate human support audit covers 7,052 assertions on the 36 human-study videos:

MethodUnsupported or unresolved / audited assertionsConservative UFR
EEA247/2,4859.94%
DG436/2,28119.11%
SR402/2,28617.59%

These pooled audit rates favor EEA by 9.17/7.65 pp. Even assigning unresolved labels least favorably to EEA leaves sample advantages of at least 4.40/2.79 pp; these are semantic bounds, not sampling CIs. This provides direct human corroboration of the support difference seen under independent Gemma evaluation.

What remains wrong outside the flags?

We blindly reviewed 200 randomly sampled unflagged assertions per method:

MethodIncorrect unflagged assertions
DG34/200 (17%)
SR28/200 (14%)
EEA6/200 (3%)

An assertion audit cannot detect events never mentioned. We therefore also used the independent substantive-problem audit of the initial outputs on the 36 videos, and tested whether the flag locations capture those problems beyond chance.

For DG and EEA, we circularly shifted each video's complete flag pattern 10,000 times, preserving its lengths and union area.

Flag-location testDGEEA
Video time actually marked13.56%22.89%
Actual problems captured47/218 (21.56%)61/81 (75.31%)
Mean capture after random shifts13.84%23.47%
Shifts reaching actual capture4/10,0000/10,000

Flag area is duration-weighted; capture is problem-weighted, so the random mean need not equal the area. EEA's observed capture exceeds all 10,000 shifted placements (add-one randomization p=0.00010).

These tests show lower observed error among unflagged assertions and targeted localization of substantive problems. Twenty of EEA's 81 problems still fall outside its flags, so final inspection remains necessary. After the same stricter inspection and all production time counted, EEA still saves 30.72% [23.9%,37.5%] versus corrected DG within the prespecified aggregate final-quality tolerances.

AuthorsOpenReview ↗

Response to YrXz (2/3): state changes, delayed effects, and concurrency (W2, W4, W5; Q3, Q5)

Does a state change establish the intended task outcome?

Thank you for the pouring/spilling example: a visible change alone is not enough to justify completion. EEA checks the proposed outcome against the before-state, the action relation, and the resulting state. Liquid motion alone cannot establish transfer into a pot; the receiving destination and outcome must be supported. If only a spill is visible, the claimed transfer is unsupported. If the destination is unclear, the exact claim remains unresolved.

Supported broader facts can remain, with local review for uncertainty that affects the action or boundary. This checks the stated outcome against the video; it does not infer the wearer's unobservable intention.

Contact, motion, and state can co-occur or be separated in time

They are jointly usable observations, not mutually exclusive categories or a mandatory three-step sequence. Flattening dough can simultaneously involve contact, motion, and a state change. Boiling can be visible without current hand contact. The relevant question is which visual relations support the particular claim.

We checked 50 video cases per category; categories can overlap:

Video-checked auditResult
Goal-consistent state/completion decisions correct46/50
Opposite outcome incorrectly accepted as completion2/50
Delayed/no-contact state decisions correct44/50
Joint contact-motion-state handling correct44/50

The audit measures how often the rules work in practice, including residual completion errors.

Concrete delayed-effect decisions

Visible evidenceActual EEA decision
HD: hob activation at 03:40.2-03:43.8Accept the supported activation event
Same video: boiling visible at 05:09.0-05:17.0Accept boiling as a later observable state; do not assert its unseen onset, precise causal delay, or cooking completion
EEB video: boiling onset not visibleKeep onset timing unresolved and flagged for review

Thus a delayed effect can support a separately observed state without requiring contemporaneous hand contact or inventing the missing transition. Supported generalization is used when a broader description is justified; otherwise the unresolved detail remains for review.

How are overlapping actions represented?

For simultaneous actions contributing to one goal, the episode contains concurrent key moments. In the HD example at 06:22.0-06:41.0, the left hand stabilizes and rotates a bowl while the right hand stirs. EEA retains both actions within one shared-goal episode. The corresponding audit recovers same-goal two-hand concurrency in 40/50 cases.

Your concern about independent overlapping episodes is valid. The current output is a single non-overlapping episode timeline, so it cannot fully encode two independently sustained goals at the same time. Concurrent key moments address the shared-goal example you asked about; full multi-goal concurrency requires a multilayer output representation. We will make this distinction explicit in the revision.

The evidence rules can check each individual claim, while the output representation determines how much parallel activity can be retained. Separating these issues makes the method's behavior and its present scope clearer.

AuthorsOpenReview ↗

Response to YrXz (3/3): broadening beyond the ontology and narration comparison (W6; Q4, Q7)

Does the fixed ontology restrict every action description?

Table 6 defines allowed broader fallbacks, not an exhaustive action vocabulary. Exact descriptions are open-vocabulary: a precise action can be retained when its positive evidence, counterevidence checks, and alternative-resolution requirements are satisfied, even without a broader parent in Table 6.

If exact evidence is insufficient, the fallback must both be an allowed common parent of the remaining interpretations and have its own visual support. Otherwise the detail stays unknown or requires local review. The system does not invent a parent to force an answer.

Code/log and video audit, 50 cases per rowResult
Supported open-vocabulary exact descriptions retained40/50
Invalid broader parent generated0/50
No valid parent: routed to review/unknown50/50

Audit categories can overlap. For example, in an EEB video, a power tool contacts a panel, but drilling and fastening cannot be distinguished. With no approved common parent, the interval was sent to local review. Conversely, visible cutting with unsupported ingredient/tool subtype retains "cuts an ingredient with a knife."

To expand fallback coverage, new parent links and their visual evidence requirements can be added and validated. This does not require replacing the two closure checks. The audit establishes current routing behavior; quantitative accuracy in new domains remains a separate empirical question.

What if we use HD-EPIC's dense narrations?

We made this comparison on the same 30 HD videos. The same generation model converts the existing official dense narrations into the common timeline format; EEA uses video alone. Both outputs use the same evaluation protocol.

Field is necessary-fact F1; NEER is recall of reference episodes requiring no core edit or substantive review. Differences are EEA minus the narration-based timeline, in percentage points (pp).

EndpointDifference [95% CI]
Segmentation F1+7.04 [2.10,11.98]
Boundary F1+10.07 [4.80,15.34]
Navigation F1+19.12 [12.05,26.19]
Complete round-trip recall+23.46 [15.20,31.72]
Field+1.36 [-2.12,4.84]
Event Recall-3.92 [-7.86,0.02]
Conservative UFR, lower is better+3.92 [1.20,6.64]
NEER+8.77 [4.10,13.44]

Existing human narrations yield fewer unsupported claims and are cheaper to reuse: 2.84M processed tokens versus 91.08M for video-only EEA. Their event-recall point estimate is higher, but the difference interval includes zero.

EEA's advantage is the recovery of temporal organization and navigation directly from raw video: stronger segmentation, boundaries, round trips, and no-edit episode recall. Thus narration is a useful input when already available, while EEA provides an annotation route for video without it. The comparison establishes their measured strengths rather than requiring EEA to win every endpoint.

Thank you again for the detailed questions. We hope the independent assessments, unflagged-error audit, concrete behavior examples, and narration comparison address your concerns and support a more favorable assessment.

Reviewer_8K6pOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EGOEASY ANNOTATOR (EEA), which which checks proposed descriptions against the video and separately searches for missed events. It produces a complete task–navigation timeline directly from untrimmed egocentric video, and substantially reduce the human correction needed afterward without any drop in final annotation quality.

Critical strengths
  1. The paper recovers more important events and observable detail while reducing unsupported claims On HD-EPIC and EgoEasy-Bench.
  2. This paper is well organized and written.
Critical weaknesses
  1. The methodological novelty mainly lies in workflow and rule design, while the core technical contribution needs further clarification. The method primarily combines existing video models, multi-round evidence retrieval, independent verification, omission scanning, and a rule-based controller. The authors should clearly identify their original contributions and distinguish them from engineering implementation.
  2. Although robot learning is a motivation of the paper, no downstream experiments demonstrate whether these annotations improve task planning, action learning, or world model training.
  3. The dataset excludes samples with severe occlusion or insufficient visible activity, yet these are challenging cases that automatic annotation systems need to handle. The authors should evaluate them as a separate challenge set to assess whether the system appropriately preserves uncertainty and flags cases for human review rather than generating incorrect descriptions.
  4. After human correction of EEA’s outputs, event recall on EgoEasy-Bench remains lower than that of both control workflows. Could the reported human-time savings partly result from more omissions remaining undetected and uncorrected?
  5. The paper lacks qualitative examples of the annotation outputs and failure-case analysis. Representative examples showing predicted timelines, descriptions, and reference annotations would help illustrate the method’s strengths and limitations.
Questions for authors

Please refer to weakness.

7 comments
AuthorsOpenReview ↗

Response to 8K6p (1/3): the original contribution and its evidence (W1)

Thank you for recognizing the results and clarity, and for asking us to distinguish the research contribution from implementation. Your questions prompted a stricter human study, difficult-video tests, downstream learning, and failure analysis. We clarify the contribution here; Part 2 addresses omissions and human time (W4-W5), and Part 3 covers downstream learning and excluded videos (W2-W3).

What is original?

A description can be well supported while its timeline still misses an event. Conversely, finding an omitted event does not establish that the proposed description of it is correct. EEA makes these two obligations explicit and connects them through one release rule:

  1. Claim closure: verify proposed content with visual evidence.
  2. Coverage closure: search beyond the draft for missing events.
  3. Shared release rule: newly found events must pass the same checks as existing claims; preserve the most specific supported content and mark important unresolved details for local review.

The observer receives selected views and a neutral question, without candidate labels or the draft. The controller distinguishes supported exact descriptions, supported broader descriptions, and unresolved details. The contribution is this executable annotation protocol and the demonstrated roles of its components. The video model and retrieval tools are existing building blocks.

Relative to the question-conditioned agents discussed in Sec. 2, EEA must discover which events and distinctions need checking without a supplied question. Our SR comparator can also find omissions, but only in its original frames. EEA adds a separate coverage search that can acquire new evidence, and applies the same exact/broader/review rules to every new finding.

Do the two obligations actually address different failures?

The original 60-video study, using the submitted Qwen evaluator, starts all conditions from the same observations and draft, then runs them independently. UFR is the conservative unsupported-field rate; Event is confirmed event recall. The following contrasts are in percentage points (pp), with 95% confidence intervals (CIs).

Add to a system with neither mechanismUFR changeEvent change
Verification only-11.08 [-15.17,-6.99]+1.51 [-3.57,6.59]
Coverage only-1.25 [-5.36,2.86]+11.14 [5.93,16.35]

Verification clearly improves support without a confirmed event-recovery gain; coverage clearly improves recovery without a confirmed support gain. Using both raises no-edit episode recall to 14.21%, versus 8.61% with verification alone, 7.22% with coverage alone, and 4.18% with neither. This counts reference episodes meeting the temporal, required-content, support, and review criteria.

Which design choices make the protocol useful?

The original component controls keep full EEA's investigation schedule fixed:

Controlled changeResultMechanism supported
Targeted versus equally many uniform new viewsUFR -4.11 pp; Event +4.95 ppThe evidence selected matters
Hide versus show candidate labels, with views fixedInitially wrong candidates corrected: 183/250 (73.20%) versus 151/250 (60.40%)Separating observation from the suggested answer helps correct errors
Keep versus withhold supported broader descriptionsEvent +5.94 pp; flagged duration -15.78 pp; 55/60 audited descriptions supported and usefulRetain useful information when exact detail is unsupported

These are the submitted component experiments, not new rebuttal runs. In the 2x2 study, a disabled mechanism's budget is not given to the other mechanism. The schedule-fixed controls test components under full EEA's investigation schedule.

Our new resource control separately asks whether more baseline tokens can replace the design. With comparable total tokens and independent Gemma scoring on 60 videos, EEA still improves over Dense-Group by +3.73 [0.31,7.15] pp Event and -3.88 [-7.62,-0.14] pp UFR.

The scientific lesson is specific: verifying what is already written and searching for what is missing require different checks; targeted evidence and label-hidden observation help those checks, while supported broadening preserves useful content. The contribution is established through these testable distinctions, beyond assembling standard tools.

AuthorsOpenReview ↗

Response to 8K6p (2/3): omissions, complete human time, and examples (W4-W5)

W4. Does leaving omissions explain the human-time saving?

Thank you for this question. We traced all 15 previously unmatched EEB events and applied the same stricter final inspection to all three workflows: annotation from scratch, corrected Dense-Group (DG), and corrected EEA. All production, inspection, repair, disagreement-resolution, and rework time was counted.

The 108 final annotations were frozen before research scoring. Later event adjudication changes evaluation labels, not the annotations or their recorded production time.

Cause of the original EEB missEventsSubsequently matched
Coverage search did not propose the event55
Event found but incorrectly treated as already covered33
Flagged, but first human correction incomplete22
Boundary/merge error21
Content present, but a different occurrence was matched11
Onset unseen or essential evidence occluded20

Eleven matches came from content repair; one came from occurrence rematching, an evaluation correction. The accounting is 139 + 11 + 1 = 151 matched events, giving 151/154 = 98.05%, versus the original 139/154 (90.26%). The remaining three events stay unmatched and in the denominator; their intervals are flagged.

Final quality and total production time

We used prespecified non-inferiority limits on benchmark-averaged quality: the 95% CI lower endpoint must exceed -2 pp for Segmentation/Field/Event, and the upper endpoint must be below +1 pp for UFR.

Final quality: EEA minus control, pp [95% CI]Corrected DGScratch
Segmentation F1+0.50 [-0.34,1.34]+0.35 [-0.48,1.18]
Necessary-field F1+0.80 [0.06,1.54]+0.68 [-0.05,1.41]
Event Recall-0.28 [-1.05,0.49]-0.37 [-1.13,0.39]
Conservative unsupported-field rate (UFR)+0.20 [-0.12,0.52]+0.33 [0.01,0.65]
ComparatorEEA time difference, person-min/video-hour [95% CI]Time saved [95% CI]
Corrected DG-145 [-177,-113]30.72% [23.9%,37.5%]
Scratch-235 [-277,-193]41.81% [35.0%,48.6%]

All eight aggregate quality criteria pass. EEA's UFR is slightly higher than scratch (+0.33 pp; 95% CI [0.01,0.65]), while remaining within the prespecified +1 pp margin. Thus, after the same final inspection and with its full cost included, EEA retains substantial human-time savings within the prespecified quality tolerances. The original 32.14% saving versus DG and the new 30.72% refer to different inspection protocols.

W5. Concrete outputs and failures

These video-checked examples show how the evidence changes the annotation:

Visible evidence and intervalActual output
HD, 04:12.0-04:18.5: hand contact, door motion, closed end stateExact: "closes the oven door"
HD, 02:31.0-02:46.4: cutting visible; precise ingredient/tool subtype and "finely" unsupportedBroader: "cuts an ingredient with a knife"
EEB, 02:08.0-02:31.5: essential cleaning contact occludedLocal review of these 23.5 seconds
HD, 05:16.1-05:19.8: knob turns and flame disappears; initially omittedCoverage finds the event; verification approves "turns off the hob"
HD, 07:52.6-07:55.2: brief ingredient addition during prolonged stirringMissed and not flagged; retained as a false negative

The last case explains why support alone is insufficient: an omitted event cannot appear as an unsupported assertion. The event audit and complete correction-time experiment directly test this failure mode. The revision will pair representative output timelines with their reference timelines and include these failures.

AuthorsOpenReview ↗

Response to 8K6p (3/3): downstream learning and excluded difficult videos (W2-W3)

W2. Do the labels improve a model trained on them?

We tested downstream temporal learning using pseudo-labels from Dense-Group (DG), DG with self-refinement (SR), or EEA across the 488-video collection. The same temporal convolutional network (TCN) and 2 fps features were used for each label source, with three training seeds. The held-out predictions cover all 18 source groups: seven HD wearer groups and eleven EEB collection groups. Each prediction is produced by a learner trained without that video's source group.

All label sources use the same 0.5-second boundary grid; the primary boundary tolerance is +/-1 second. Predictions on the held-out videos were scored against their independent human benchmark references; DG/SR/EEA pseudo-labels were used only as training targets.

EndpointDG labelsSR labelsEEA labelsEEA minus DG, pp [95% CI]
Navigation F127.126.834.3+7.20 [6.79,7.64]
Boundary F131.030.838.1+7.10 [6.74,7.47]
Segmentation F135.234.340.9+5.70 [5.34,6.05]
Complete round-trip recall19.419.027.8+8.40 [8.00,8.83]

DG is the stronger control on all four endpoints. Each metric improves in all 18 held-out groups and for all three training seeds. CIs resample source groups after averaging the three seeds, with trained predictions fixed.

This establishes that EEA's annotations improve downstream recovery of temporal structure and navigation. It is evidence for temporal learning; robot policies, task planning, and world-model training remain distinct downstream evaluations.

W3. Does the method retain uncertainty on the excluded videos?

We evaluated all 93 excluded difficult EEB files, alongside 93 duration-matched clear files (7.75 hours per set). The difficult set includes severe occlusion and insufficiently visible or structured activity. Humans audited, per method, 200 exact descriptions, 100 cases needing a review decision, 100 review requests, and 250 reference events.

Difficult-set auditDGSREEA
Incorrect exact descriptions46/200 (23.0%)38/200 (19.0%)17/200 (8.5%)
Cases needing review correctly flagged29/10036/10078/100
Review requests judged appropriate45/10049/10072/100
Reference events recovered90/250 (36%)95/250 (38%)110/250 (44%)

These complementary measurements matter: EEA makes fewer incorrect precise claims, identifies more cases needing intervention, and retains useful events. Of EEA's review requests, 72/100 were judged to correspond to a genuine need for human intervention. Event recovery is 65% on the clear control set and 44% on the difficult set, which quantifies the remaining challenge.

Together with the fully counted human workflow in Part 2, these experiments provide evidence for practical annotation utility, appropriate handling of difficult observations, and transfer to a learner trained on the labels.

Thank you again for the suggestions. We hope the clarified contribution and completed experiments address your concerns and support a more favorable assessment of the paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response: main additions and corrections

We thank the reviewers for the careful reading and concrete suggestions. We added independent evaluation, resource controls, a stricter human study, difficult-video tests, and downstream learning. The reviewer-specific replies give the settings, denominators, and uncertainty. DG denotes Dense-Group; SR adds self-refinement. pp denotes percentage points; brackets give 95% confidence intervals.

Main questionNew evidence
Can the anomalous seed results be explained?Two non-final DG outputs were misloaded. Saved-output reconstruction and human matching audits identify the error; the other seed and main-table file identities were checked.
Do the gains survive independent assessment?On 60 videos, human Event Recall favors EEA over DG by 12.62 pp [7.34,17.90]. On 36 videos, human conservative unsupported-field rates are 9.94% for EEA and 19.11% for DG.
Is the gain just more tokens?With comparable recorded input-plus-output tokens and independent Gemma scoring, EEA retains Event, support, and no-edit-episode gains over DG and SR.
Is time saved by leaving omissions?After the same final inspection and all production time counted, EEA saves 30.72% [23.9%,37.5%] versus corrected DG and passes all eight prespecified aggregate quality criteria.
Do the labels help learning?The same temporal learner, tested against independent human references, improves four held-out metrics by 5.70-8.40 pp with EEA rather than DG training labels.
What happens on excluded difficult videos?Human audit of 93 excluded files: incorrect exact descriptions are 8.5% for EEA versus 19-23% for the controls, with 44% event recovery.
Are errors hidden outside flags?Blind audit of 200 unflagged assertions per method: 3% incorrect for EEA versus 14-17%; a separate location test verifies targeted flag placement.
Why not use existing narrations?On the same 30 HD videos, narration has lower UFR; video-only EEA improves temporal boundaries, navigation, and no-edit episode recall.

The contribution is supported by complementary evidence: checking proposed claims reduces unsupported content, searching beyond the draft recovers omissions, and newly found content must pass the same evidence rules. We will incorporate the corrections and additional results into the revision.

Dear Reviewers,

Authors posted their responses to your queries. Please read these messages and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

AuthorsOpenReview ↗

EgoEasy-Bench: 50 anonymous samples and practical annotation challenges

Thank you for your thoughtful feedback and requests for concrete examples (8K6p W5; YrXz Q3). We have curated 50 EgoEasy-Bench (EEB) video samples to illustrate diverse scenes and challenging task structure:

https://drive.google.com/drive/folders/14Ul9YtuP_RRIvs-MB9IJhAHJ71Ny3fYz?usp=sharing

The shared materials and accompanying information have been carefully checked for identifying details to preserve double-blind anonymity.

The collection spans home cooking and dining (20), household chores and care (9), hotel housekeeping (6), public-facility cleaning and maintenance (10), outdoor cleaning and pet care (4), and fine manual work (1).

In these 50 clips (3.6 hours in total), the references contain 488 task episodes, 302 navigation episodes, and 1,933 key moments, with 180 navigation episodes lasting at most 5 seconds. Short navigation is also common in the full valid EEB collection: 534/992 episodes (53.83%; Table 9). 33 clips include leaving one space, working elsewhere, and returning.

Clip and timestampAnnotation challenge
04, 03:26.14-03:33.06The wearer interrupts sweeping, checks a stove control in another room, returns within 7 seconds, then resumes sweeping. Both navigation legs and the intervening action need distinct boundaries.
20, 03:59.54-04:00.21A 0.67-second caution-sign pickup must be distinguished from the much longer carrying and subsequent placement.
49, 00:30.01-04:52.70Six kitchen-table round trips follow the same route, with different carried objects and repeated returns to stirring. Each occurrence must remain distinct.
08, 02:05.00; 03:31.53Tool contact with a nearly transparent line requires close visual inspection.

These examples motivate EEA's complementary checks: coverage search looks for omitted events and transitions; claim verification checks actions, object roles, and states against visible evidence, with local review for important unresolved details.

Our separate 36-video human-workflow study quantifies the practical benefit. After the same final inspection and with all production time counted, EEA reduces human time by 30.72% versus corrected Dense-Group (95% CI: 23.9%-37.5%) and 41.81% versus annotation from scratch (95% CI: 35.0%-48.6%), while meeting all eight prespecified aggregate quality non-inferiority criteria.

We hope these samples make the annotation problem, the design rationale, and the measured practical benefits easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.