- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents EgoEasyAnnotator (EEA), which builds a timestamped task–navigation timeline from untrimmed egocentric video. EEA first records visible contact, motion, and state changes with their timestamps. The claim closure stage then checks ambiguities that could change the annotation. The observation call sees the selected frames or clips but not the candidate labels or the initial annotation. A rule-based controller keeps an exact description, falls back to a supported broader description, or sends the unresolved detail for local human review. A separate coverage closure pass searches for missed events, and each new finding goes through the same checks before the final timeline is assembled. On 488 videos, EEA performs better than the comparison baselines on the main annotation metrics. In a 36-video human study, correcting EEA outputs took 32.14% less time than correcting Dense-Group outputs.
Critical strengths
- The main idea is useful. The paper separates two common failures in long-video annotation: a description may lack enough evidence, and the initial annotation may miss an event. Claim closure checks the first problem, while coverage closure searches for the second. New findings go through the same evidence checks before they enter the final timeline. This lets EEA control unsupported descriptions without giving up coverage.
- The experiments support the method’s practical effect. On HD-EPIC and EgoEasy-Bench, EEA improves episode segmentation F1 by 12.34 percentage points and Event Recall by 13.52 points, while lowering the upper bound of Unsupported Field Rate by 7.67 points. The ablations suggest that claim checking and omission search address different failure modes. The human study also reports 32.14% less correction time for EEA outputs than for Dense-Group outputs.
Critical weaknesses
- The generator and evaluator share the same backbone. Both EEA and the automatic evaluator use Qwen3.8-27B. The evaluator extracts required facts and operations from the references, checks visual support, and matches predicted fields and events. EEA may therefore benefit from the evaluator’s model-specific preferences. The human calibration is useful, but its sample is too small to rule out shared errors.
- Unexplained numerical instability limits confidence in the empirical results. On identical test videos, Dense-Group’s HD-EPIC Event Recall changes from 33.51% and 30.81% to 98.92% across seeds, while EgoEasy recall drops from 68.63% to 1.96% (Table 43). These opposing changes yield an aggregate standard deviation of only 0.68 percentage points (Table 42), concealing substantial instability within each benchmark. The authors acknowledge the variation but do not explain whether it arises from generation, matching, or evaluation. Moreover, the main generator and semantic evaluator use the same Qwen checkpoint; the pooled 240-assertion calibration does not establish that comparative gains survive independent assessment. Please audit the extreme runs using saved predictions and human judgments, provide the corresponding matching decisions and counts, and verify the relative gains with an independent evaluator.
Without this diagnosis, I have low confidence in the reliability of the reported comparisons.
Questions for authors
- Could the authors compare EEA with Dense-Group and Dense-Group+SR at similar token or inference budgets, to show how much of the gain comes from the proposed design?
- EEA stops visual verification and requests human review after reaching a preset inspection budget. Could the authors explain how these budgets are chosen and how changing them affects annotation quality and the amount of human review?
- The figures are not clear, seem like purely AI-generated.
- See weaknesses.
7 comments
Response to tZGS (2/3): independent model and human assessment (W1, W2)
Your key question is whether EEA's comparative gains survive a change of evaluator. We therefore tested the method differences with a different-family judge and direct human assessments.
DG is Dense-Group; SR is DG with two self-refinement rounds. Differences below are EEA minus control, in percentage points (pp). Event counts confirmed reference-event matches; UFR is the conservative rate of unsupported or unresolved core assertions, so lower is better.
Independent Gemma evaluation and human Event Recall
We rescored frozen outputs on 60 videos, 30 per benchmark, using google/gemma-3-27b-it with deterministic decoding and method identities hidden. Its configuration was selected on 240 newly labeled development assertions: 40 per method per benchmark from 16 development videos, then frozen before any test scoring. These are separate from the paper's original calibration sample. Development agreement with humans was 217/240 (90.42%), with similar rates across methods: EEA 91.25%, DG 88.75%, and SR 91.25%.
Humans also adjudicated Event Recall on the same 60 videos, covering 1,188 reference events.
| Assessment, difference [95% CI] | Versus DG | Versus SR |
|---|---|---|
| Gemma: Event Recall | +12.67 [7.02,18.32] | +12.30 [6.62,17.98] |
| Gemma: UFR | -8.39 [-13.74,-3.04] | -6.76 [-12.10,-1.42] |
| Human: Event Recall | +12.62 [7.34,17.90] | +12.27 [6.95,17.59] |
The event-recovery benefit is therefore supported separately by Gemma and humans, with each interval excluding zero. Under Gemma, necessary-field F1 and no-edit episode recall also improve: versus DG, +9.16 [3.43,14.89] and +9.69 [6.35,13.03] pp, respectively.
Direct human support audit: 7,052 assertions on 36 videos
Humans labeled assertions as supported, contradicted, insufficiently evidenced, or unresolved. Conservative UFR counts the last three categories; unresolved items are not called confirmed errors.
| Method | Unsupported or unresolved / audited assertions | Human conservative UFR |
|---|---|---|
| EEA | 247/2,485 | 9.94% |
| DG | 436/2,281 | 19.11% |
| SR | 402/2,286 | 17.59% |
These are pooled audit proportions. EEA's human UFR is lower by 9.17 pp versus DG and 7.65 pp versus SR. Even assigning every unresolved EEA assertion to unsupported and every unresolved control assertion to supported leaves sample advantages of at least 4.40 pp and 2.79 pp. These last numbers bound interpretation of unresolved labels within this sample; they are not sampling confidence intervals.
Conclusion. EEA's Event gains remain under independent Gemma and human assessment. Its support gains remain under Gemma and are corroborated by the lower observed UFR in the human assertion audit. These complementary assessments show that the observed advantages are not confined to the original Qwen judge.
Response to tZGS (3/3): token controls, budgets, and figures (Q1-Q3)
Q1. Do the gains remain at comparable token use?
Using the same generator backbone, we increased baseline resources toward EEA's cumulative token budget on the same respective panels: DG on 60 videos (30 per benchmark), SR on 24 (12 per benchmark).
DG used 60-s/2-fps windows with 25% overlap and fixed additional grouping/consensus calls. SR retained its original DG frames and completed two full critique-revision rounds over every window, followed by fixed same-frame consolidation calls. The final call was capped at the remaining token allowance. Totals count input (text and visual) plus output across all ordinary, repair, and failed calls.
| Control / EEA | Total tokens, millions | GPU-hours | Summed wall time, hours |
|---|---|---|---|
| DG / EEA, 60 videos | 123.7 / 123.7 | 93.20 / 96.90 | 52.59 / 57.79 |
| SR / EEA, 24 videos | 49.5 / 49.5 | 37.70 / 38.80 | 20.45 / 23.12 |
This compares total-token use, not equal FLOPs or runtime. Both the token comparisons and the budget sweep use the same development-frozen Gemma judge for semantics; segmentation matching is deterministic. UFR includes unsupported and unresolved assertions; NEER is the recall of reference episodes needing no core edit or substantive review.
| EEA minus control, pp [95% CI] | DG, 60 videos | SR, 24 videos |
|---|---|---|
| Segmentation F1 | +3.87 [-1.66,9.40] | +9.86 [3.30,16.42] |
| Necessary-field F1 | +2.83 [-1.24,6.90] | +5.54 [0.44,10.64] |
| UFR | -3.88 [-7.62,-0.14] | -4.73 [-9.20,-0.26] |
| Event Recall | +3.73 [0.31,7.15] | +8.61 [3.55,13.67] |
| NEER | +4.76 [2.21,7.31] | +6.46 [3.38,9.54] |
At these totals, extra baseline tokens do not fully remove the event, support, and no-edit-episode gains. DG's Segmentation/Field intervals include zero. These are pointwise intervals, without an additional multiplicity adjustment.
The original Qwen-scored component controls explain contributing mechanisms: targeted versus equally many uniform views lowers UFR by 4.11 pp and raises Event by 4.95 pp on full EEA's fixed investigation schedule. The 2x2 ablation separately shows verification reducing UFR by 11.08 pp and coverage raising Event by 11.14 pp when each is added alone. EEA couples these obligations: newly found events must pass the same support checks.
Q2. Budget choice and sensitivity
The fixed caps depend on source length, not test scores. With n scheduling windows (each at most 30 seconds and 900 native frames), s=max(1,ceil(n/24)). Caps include 48 x s further requests, 24 x s investigations with at most two requests each, and two coverage rounds with at most 4 x s requests each. These are ceilings, not mandatory consumption.
On 12 videos, six per benchmark, low/default/high runs share the initial observations and draft, with unchanged evidence rules. Actual token use is 0.62x/1.00x/1.35x default.
| Change, pp [95% CI] | Default minus low | High minus default |
|---|---|---|
| Event Recall | +4.64 [1.02,8.26] | +1.66 [-0.72,4.04] |
| UFR | -4.62 [-7.88,-1.36] | -1.02 [-3.11,1.07] |
| Video duration flagged for review | -8.64 [-13.40,-3.88] | -2.96 [-6.10,0.18] |
Budget-terminated videos fall from 8 to 2 to 0; unresolved questions from 24 to 7 to 1. Default improves quality and reduces flagged duration versus low; further high-budget gains remain uncertain. Exhaustion stops inspection and retains local review for important unresolved details without relaxing the evidence rules.
Q3. Figure clarity and provenance
We agree that Fig. 1 should be clearer. Its stages, arrows, and text are our own and follow Sec. 3; we assembled it after discussing layout with an AI assistant. The small thumbnails, however, are AI-generated illustrations, not benchmark frames. The caption and AI-use statement should have said so; we will correct both.
We will replace the thumbnails with real, timestamped EEB frames, label the evidence passed between stages, and show that candidate labels are hidden from the observer and newly found events return to the same verification gate. The diagram will illustrate exact, supported broader, and local-review decisions. We will also simplify Fig. 2 and enlarge its labels. Anonymous EEB samples can be provided.
Thank you again. We hope the concrete error diagnosis, independent model/human assessments, and comparable-token controls address the reliability concerns underlying your recommendation. We would appreciate reconsideration of the soundness assessment and overall recommendation in light of this evidence.
Response to tZGS (1/3): diagnosing the seed anomaly (W2)
Thank you for identifying the mirrored Dense-Group results and specifying the audit needed to assess them. You were right that a small aggregate standard deviation could not justify those benchmark-level extremes. We traced the saved outputs, audited the matching decisions with humans, and repeated scoring and generation. Part 2 addresses independent evaluation; Part 3 addresses resources, stopping budgets, and figures.
Cause and corrected counts
For Dense-Group (DG), seed 20260918, the result loader selected non-final attempts instead of the designated final outputs:
The HD file was an overcomplete pre-grouping recovery artifact concatenating window-level DG observations. It retained event descriptions before the required grouping stage; evaluating those observations is not evaluating the completed DG timeline. Here, "final" means the completed grouping output in the common task/navigation timeline schema. The recovery concatenation was not an eligible final timeline.
The final output for each run was designated before that run was scored. The correction restores those assignments; evaluation scores were not used to choose the final attempt.
The error was the association between the stated run and the output passed to evaluation. We take responsibility for it. Selecting the designated final outputs and their saved judgments reconstructs the corrected counts:
Corrected seed-20260918 benchmark average: 50.53% (published 50.44%); three-seed mean: 50.44% (published 50.41%), with SD 0.68 pp. The opposing errors had largely canceled in the average. We will correct Tables 42-43 and retain separate benchmark results for every seed.
Human audit of the extreme outputs
Humans checked all 236 reference-event matching decisions on the two wrongly selected outputs:
Thus 232/236 decisions were confirmed and four remained ambiguous, with none reversed. The audit confirms that most matching events were present in the mistakenly selected artifacts; it does not make those intermediate artifacts eligible final timelines. The local matching ambiguities do not explain the mirrored extremes.
Replays and audit scope
These checks support the diagnosis of this DG anomaly. The revised tables retain each method's observed benchmark-level seed variation; the DG correction does not establish universal seed stability.
We re-checked file identity for every other entry of Tables 42-43, including all EEA seeds, and for all main-table runs; no other entry was affected. The next reply tests the comparative gains independently of the original Qwen judge.