acceptodds
Under review as a conference paper at ICLR 2027

EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos

Abstract

We present EgoAction, a unified benchmark and model family that directly infers and evaluates all key dense actions in first-person video, including semantic interactions, hand assignments, and temporal boundaries, from continuous egocentric clips. This approach is a step forward in egocentric video understanding, where models and metrics have typically been constrained to recognizing coarse activities or generating holistic summaries that conflate distinct failure modes. The benchmark, EgoActionBench, is highly structured and diagnostic, representing actions as parallel hand tracks of atomic events, and effectively disentangling discovery, recognition, localization, and laterality errors that traditional caption-similarity metrics obscure. The EgoAction model family achieves significant performance gains in multiple dense action tasks, including semantic F1, atomic-action recall, and precise temporal boundary localization. We also show that adapting standard vision-language models, such as Qwen3-VL and LLaVA-OneVision-2, with our curated dense-action corpus significantly enhances their structured-output capabilities, raising semantic recall to over 20% on human-annotated sets and reducing median boundary error by 66% on held-out sets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video Understanding, Dense Video Captioning, Hand-Object Interaction, Temporal Action Localization, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sZjzOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces EgoActionBench, a benchmark for dense per-hand action captioning in egocentric human videos. Given a complete clip without other information, a model must generate temporally localized atomic action descriptions on Left, Right, and Both hands. The benchmark comprises 1407 episodes and 40384 human-refined action annotations curated from Ego4D, EgoLife, and EPIC-KITCHENS-100. Its ground-truth-anchored evaluation retrieves temporally eligible predictions, selects one prediction by field-level semantic agreement, and evaluates that same prediction for semantic content, temporal IoU, and hand assignment. Experiments compare 19 baseline models and adapt Qwen3-VL-4B and LLaVA-OneVision-2-8B using automatically annotated training episodes. Adaptation improves EgoActionScore by 10.4 and 20.0 points, respectively, with additional evidence from frame-sampling sweeps and transfer to HanDyVQA.

Critical strengths
  • An important and well-specified task beyond coarse video understanding. The benchmark requires autonomous discovery of atomic manipulation actions, temporal boundaries, and anatomical hand assignments. It explicitly preserves sustained single-hand actions alongside shorter actions of the other hand and distinguishes coordinated bimanual manipulation from unrelated simultaneous actions. This work also introduces a multi-stage human verification pipeline, which makes this a potentially useful resource for studying the detailed structure of human manipulation. This matters because coarse activity descriptions can conceal precisely the action transitions and hand roles needed for learning from human demonstrations.
  • Conducted meaningful and sufficient ablation experiments. Both adapted backbones improve across Semantic, Temporal, and Hand scores, with particularly substantial Temporal gains of 16.8 and 25.7 points. The adapted models outperform all open-source baselines evaluated in the paper, and sampling sweeps show that these gains are not replaced by simply providing the original backbones with denser observations. Transfer evaluation on HanDyVQA uses identical video inputs and recordings disjoint from adaptation data, increasing accuracy from 61.01% to 63.80% without further VQA fine-tuning, although Objects AP declines.
Critical weaknesses

Data source concern. The paper describes automatically annotated adaptation episodes from Ego4D, EPIC-KITCHENS, and EgoLife, but does not clearly identify the models used to generate and refine the training labels or provide a sufficiently detailed training-specific annotation procedure, including prompts, filtering criteria, and quality checks. The manuscript also does not clearly state whether the adaptation annotations, data-generation code, and source-video split manifests are publicly available or will be released. Regarding evaluation, the authors state that adaptation and benchmark data are separated by source-video identifiers, which is a useful safeguard against direct fine-tuning overlap. However, the benchmark reuses videos from existing datasets, and the paper neither specifies their original official splits nor audits possible exposure during the evaluated models' pretraining or prior task-specific training. This does not establish that leakage occurred, but leaves contamination risk unresolved. Is there any special consideration or method for the selection of test set video sources here?

The metric's treatment of prediction errors and temporal boundary ambiguity needs stronger justification. I am concerned that EgoActionScore does not fully capture the correctness of a generated action sequence: it applies no independent false-positive penalty, allows a prediction to support multiple ground-truth actions, and lets Temporal and Hand contribute even when semantic agreement is zero. A further concern is that atomic action boundaries are not necessarily uniquely identifiable timestamps. During continuous manipulation, reasonable annotators may place a transition slightly differently depending on whether they emphasize contact onset, motion onset, or object-state change. A difference of a few tenths of a second may therefore be acceptable in some cases rather than a genuine localization error. However, the protocol scores temporal IoU against a single reference interval without an explicit boundary-tolerance or uncertainty model. With a median action duration of only 0.63 seconds, such differences can substantially affect the temporal score, potentially conflating annotation ambiguity with model error. The candidate-overlap threshold provides flexibility in retrieving matches, but does not remove this issue from the final IoU score, and threshold sensitivity alone does not validate the treatment of boundary uncertainty.

Questions for authors

Please see the weaknesses

5 comments
AuthorsOpenReview ↗

Response to the Review[1/2]

Thank you for recognizing the importance of the task, the preservation of independent hand timelines, and the value of the annotation pipeline and ablation studies. We appreciate your detailed comments on data provenance and evaluation design.

1. Data provenance and reproducibility. (W1)

Both the initial generation and subsequent automatic refinement of the adaptation labels use GPT-6 Astra. The annotations contain timestamped action captions organized into Left, Right, and Both tracks, providing supervision for action descriptions, temporal intervals, and hand assignments.

Candidate benchmark episodes are first selected automatically and then manually screened for coherent manipulation tasks and verifiable hand–object interactions. The adaptation and benchmark sets are separated by original source-video ID. All episodes extracted from the same source recording remain on the same side of the partition, including episodes with different temporal ranges or episode identifiers.

This source-video separation controls direct overlap with our adaptation data. If a model’s pretraining data are not disclosed, we cannot verify whether it saw the benchmark’s source videos during pretraining. The benchmark evaluates newly constructed, fine-grained action annotations over existing video sources. And we will release the adaptation training annotations and source-video identifiers documenting the adaptation–benchmark split.

2. False-positive penalties and reuse of predictions. (W2)

We have added action-level F1 through one-to-one matching to measure action coverage and prediction redundancy. GT-anchored semantic assessment can reuse a prediction when it contains information relevant to multiple reference actions. In the separate F1 calculation, each prediction can match at most one GT action. Unmatched predictions count as false positives, and unmatched GT actions count as false negatives.

For illustration, if three duplicate predictions describe one GT action, at most one receives a match: the other two count as false positives, giving an F1 of 50%. If one prediction spans two GT actions, it can contribute at most one true positive, leaving one false negative and giving an F1 of 66.7%. These examples illustrate how the revised metric penalizes redundant predictions and under-segmentation.

The revised Temporal score averages m-tIoU and F1, and EgoActionScore averages Temporal and Semantic. These changes are summarized in Section 3.4 and detailed in Appendix C, with illustrative examples in Figure 6. Representative results under the updated evaluation are:

Model / systemm-tIoU ↑F1 ↑EgoActionScore ↑
Human Expert88.493.190.4
GPT-6 Astra40.668.659.6
EgoAction (LLaVA-OV2-8B)32.357.047.9
EgoAction (Qwen3-VL-4B)26.645.941.8
Flora18.646.937.8
Qwen3-VL-4B9.928.225.9
LLaVA-OV2-8B9.826.724.7

3. Content requirements for action matching. (W2)

The revised protocol requires positive content agreement before a prediction can be selected. This follows our action-first objective: identifying the action provides the basis for evaluating its timing and acting hand. A prediction with zero content agreement is ineligible for both GT-anchored selection and one-to-one F1 matching. If a GT action has no positive-content candidate, its Semantic, temporal IoU, and Hand scores are all zero, and it remains in the evaluated denominator. It is also unmatched in the F1 calculation. Thus, a prediction with the correct interval and hand label but no agreement with the reference action’s content receives no credit for that action.

AuthorsOpenReview ↗

Response to the Review[2/2]

4. Boundary ambiguity and interpretation of temporal scores. (W2)

We acknowledge that reasonable annotators can place some action transitions at slightly different timestamps. Our annotation guidelines use visible changes in hand–object contact, motion phase, and object state to identify boundaries. Annotators inspect native video frames around these transitions, and ambiguous cases undergo further review. This provides a consistent annotation procedure at the video’s native temporal resolution.

Temporal IoU measures the overlap between predicted and reference action intervals. Its sensitivity is particularly relevant for brief actions, where a timing offset can span a substantial part of the action or extend into a neighboring operation. Human performance provides an empirical reference under the same scoring protocol.

The human baseline covers all benchmark videos, each independently annotated by three annotators. Their predictions are evaluated separately against the fixed references, and the reported scores average the three annotators’ results:

Evaluation setHuman m-tIoU ↑GPT-6 Astra m-tIoU ↑Human F1 ↑
Overall88.440.693.1
Ego4D92.037.596.2
EgoLife94.440.397.3
EPIC-KITCHENS-10075.945.584.2

These results show the level of human agreement with the references under strict interval scoring and the remaining model–human gap across sources.

To directly examine temporal localization with allowance for boundary shifts, we additionally evaluate Boundary-F1 at tolerances of 0.1, 0.2, and 0.5 seconds. Boundaries extracted from predicted and reference action intervals are matched one-to-one within each video and hand track. A match is accepted when the absolute timing difference falls within the specified tolerance. Unmatched predicted boundaries count as false positives, and unmatched reference boundaries count as false negatives. Counts are aggregated across videos to compute micro-F1. For this additional validation, we compare Qwen3.5-4B before and after task-specific SFT, with both models using 2 FPS video inputs:

ModelBoundary-F1 (0.1 s) ↑Boundary-F1 (0.2 s) ↑Boundary-F1 (0.5 s) ↑
Qwen3.5-4B Base10.8118.2129.89
Qwen3.5-4B + SFT23.4435.7954.37

SFT improves Boundary-F1 across all three tolerances, including the more permissive 0.5-second setting. These results complement the strict interval-based scores by showing that adaptation improves boundary detection even when the evaluation explicitly accepts temporal offsets within these windows.

We appreciate the time and thought you have devoted to evaluating our work. Your suggestions have helped us clarify the annotation process and strengthen the evaluation protocol. We hope our responses address your concerns and would be glad to clarify any remaining questions.

Dear Reviewer,

Thank you for your careful assessment of our work. Following your suggestions, we have provided additional experimental results and clarified the relevant aspects of the paper. We would welcome your thoughts on these updates whenever you have an opportunity to review them. Please let us know if there is anything further we could explain or discuss.

We sincerely appreciate your time and consideration.

Reviewer_4RU8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A benchmark for per-hand action recognition in egocentric videos that focuses on atomicity. temporal precision and hand assignment. EgoActionBench is constructed via a semi-automated pipeline where off the shelf models are used with human in the loop to caption actions densely for the given three criteria. It is based on EgoLife, EPIC-Kitchens and Ego4D and contains 1,407 episodes and 40,384 action captions across left, right and both hands tracks. The task is that model needs to temporally localise and describe the action visible in either or both hands while associating these predictions with the corresponding hand. The paper provides analysis of 19 baselines with varying capabilities including proprietary models, open source VLM, video captioning and video understanding models. Further, they train their own EgoAction with 2 different visual backbones. Their main findings are: (1) dense action recovery is challenging for those large models that are capable of video understanding (2) temporal localisation is the primary bottleneck: even the best model GPT-6 Astra trails humans by a significantly larger margin on temporal, compared to the smaller gaps on semantic and hand (3) temporal precision and motion remain challenging even when training with dense actions

Critical strengths
  • Authors identified a sound gap in video understanding concerning the granularity of actions.
  • The authors introduce criteria to assess this limitation and define the concepts of atomicity, temporal precision and hand assignment and clearly explain why these are valid criteria to evaluate the models for the gap they are addressing
  • The authors evaluate four categories of baselines on EgoActionBench and demonstrate they struggle when measured with fine grained actions which strengthen their paper hypothesis and the need EgoActionBench
  • Breaking down evaluation into verbs, objects, tools and motion makes it more interpretable and helps identify where a given model falls short.
Critical weaknesses
  • The paper is hard to follow in places, particularly the notation and the evaluation details, much of which sits in Appendix C rather than the main text. It also gives little justification for the choice of models at key pipeline stages: CaRe-Ego in Stage 1, the unnamed VLM in Stage 2, and the agents in Stage 4.
  • The first two stages in the annotation pipeline are automatic and there is no human verification in between making the whole pipeline fragile and highly susceptible to cumulative errors from these initial stages
  • Median action duration is short (0.63 seconds) while several action categories might have a median of much longer than 0.63 seconds that might not make this benchmark suitable to evaluate them
  • In stage 1: candidate retrieval, the method is only evaluating for the segments correctly retrieved above a certain threshold t_sem, however, this discards all other predictions with a lower overlap from the evaluation metric
  • In L917, the motion values are required to belong to a certain set of values which makes it harder to generalises for actions that do not result in a motion that belong in this set of values.
  • GPT Astra 6 is used in creating the benchmark but is also the best performing baseline in the main results table. The usage of GPT Astra 6 for annotation is only mentioned in the AI Use Statement but the details of how it is used is not mentioned in the main text.
Questions for authors
  • Why is CaRe-Ego the model chosen for hand aware preprocessing? How often do its masks fail or swap the left and right hands, and how do such errors affect the hand assignments in later stages?
  • Further details on the choice of VLM in step 2 of the annotation pipeline and its setup (e.g. prompt, in-context learning, etc) are needed
  • Further details and justifications on choices and setup of agentic model in stage 4 for QC are needed.
  • What fraction of agents’ proposed revisions did human reviewers accept in Stage 5?
  • L214, on data sources and selection, how is it screened for suitable for dense hand objection annotation? What determines its suitability? How are episodes with severe occlusion, poor lightening, etc discarded?
  • Does the quality of the VLM description deteriorate for actions lower than its sliding window interval? Can you provide qualitative results for the VLM outputs for actions that span a duration larger than and shorter than the sliding window interval for comparison?
  • Can you provide more qualitative samples for figure 4 that shows samples with both hands in action when (1) both hands are handling the same object (2) when they are handling different objects and carrying out different actions on different objects.
  • How was GPT Astra 6 used in annotation? Would it maintain highest performance if tested on a subset of the data without using it in the annotation?
  • Without a false-positive penalty, how does EgoActionScore distinguish accurate per-hand predictions from systematic over-prediction across both hand tracks?
  • Could you report results by action-duration bin and by activity category?
  • How many samples were included in the human baseline?
5 comments
AuthorsOpenReview ↗

Response to the Review[1/4]

Thank you for recognizing the importance of action atomicity, temporal precision, and hand–action association, as well as the interpretability of our evaluation.

1. CaRe-Ego selection and hand-aware preprocessing. (W1, Q1)

We use CaRe-Ego because it explicitly distinguishes left and right hands and has demonstrated strong hand-segmentation performance on egocentric benchmarks. The results reported in its original paper are summarized below [1]:

Evaluation setLeft-hand IoU ↑Right-hand IoU ↑Left-hand pixel Acc. ↑Right-hand pixel Acc. ↑
EgoHOS in-domain92.3493.6496.6496.81
EgoHOS out-of-domain94.4794.4197.0996.69
mini-HOI4D70.3989.7697.7993.09

These published segmentation results motivate its use for providing explicit hand cues. In our pipeline, we compare detections across neighboring frames to filter out brief, isolated detections. We then check suspicious regions and remove those identified as non-hand regions. The resulting hand overlays guide draft caption generation. To assess reliability in our pipeline, we manually inspected 675 sampled frames from 30 videos. The frame-level hand misassignment rate was 1.1%. In the inspected videos, these mask errors did not lead to incorrect hand assignments in the subsequent captions. The errors occurred intermittently, while most frames correctly distinguished the two hands. The captioning model considers multiple frames within each sliding window, so surrounding frames provide correct hand cues even when an individual frame contains an error. Human annotators also review the captions against the original videos and correct hand-assignment errors.

2. Caption-generation and agentic-QC setup. (W1, W6, Q2–Q3)

As noted in the AI Use Statement, GPT-6 Astra assists with benchmark annotation. We clarify its specific roles in Stage 2 caption generation and Stage 4 agentic QC below:

StageInputsOutput and role
Stage 2: Caption generationHand-highlighted video frames sampled at 5 fps, video summary, and recent captionsCaption proposals for the Left, Right, and Both tracks, using one-second windows with a one-second stride and no overlap
Stage 4: Agentic QCHuman-revised annotations and timestamped frames from the original videoProposed corrections to captions, hand assignments, and boundaries, with supporting frame references and explanations

The caption-generation and temporal-merging prompts are provided in Appendix H.2. The captioning prompt uses recent captions to establish continuity while requiring descriptions to be supported by the current window’s visual evidence. It distinguishes visible actions from idle or invisible hands and requests separate descriptions for the three tracks. Consecutive proposals are merged only when visual evidence supports the same uninterrupted action. During QC, agents inspect hand–object contact, hand-motion phases, and object-state changes, examining consecutive native frames around ambiguous boundaries.

3. Human verification and error propagation. (W2, Q4)

The automatic stages produce drafts that undergo human revision against the original videos, followed by agentic QC and final human verification. Annotators can revise captions, hand assignments, and temporal boundaries, remove unsupported annotations, and add records. To quantify the extent of this intervention, we compared recoverable initial annotations with their completed human-edited versions for 457 benchmark clips:

CategoryCount
Initial10,733
Revised or removed10,285 (95.83%)
Added6,757

These results demonstrate substantial human intervention in the final annotations. In Stage 5, human reviewers accepted 6,206 of the 6,446 agent-proposed revisions (96.28%) after checking them against the original videos.

References

[1] Su, Y., Wang, Y., and Chau, L.-P. (2026). CaRe-Ego: Contact-aware relationship modeling for egocentric interactive hand-object segmentation. Expert Systems with Applications, 296, 129148.

AuthorsOpenReview ↗

Response to the Review[2/4]

4. Episode selection and visual quality. (Q5)

Candidate episodes are first selected automatically and then undergo a round of manual screening. Episodes are selected to capture a coherent manipulation task or subtask with sufficient context to distinguish the actions and their transitions. The selection criterion is whether the visible hand–object interactions and action boundaries can be verified from the video. Clips with severe occlusion, unverifiable actions, or unrelated task transitions are excluded. Source identifiers and temporal ranges are retained for traceability. Annotation uses the native video frames, with the following temporal resolutions:

SourceEpisodesFrame rateDuration of one frame
Ego4D45130 fps33.3 ms
EgoLife48520 fps50 ms
EPIC-KITCHENS-10047150 fps20 ms

These frame-level coordinates support boundary inspection beyond the one-second interval used for initial caption proposals.

5. Action duration and annotation granularity. (W3, Q6, Q10)

The reported median summarizes the duration distribution across all annotated actions and does not represent the typical duration of every action category. Actions such as holding or stirring can last substantially longer and are annotated over their full continuous intervals. Their boundaries follow meaningful changes in the operation, manipulated object, contact relation, or motion, rather than a fixed duration limit. The benchmark includes actions lasting at least five seconds, and the duration-specific results below explicitly report performance on these longer actions.

The one-second proposals belong to draft generation. Final annotations have variable-length intervals because annotators add omitted actions, split compound events, merge uninterrupted actions, and refine boundaries using native video frames. The proposal interval therefore does not impose a one-second minimum duration or quantize the final action boundaries.

We report the evaluated models’ temporal localization performance across five action-duration bins below. All entries are m-tIoU:

Model<0.5 s0.5–1 s1–2 s2–5 s≥5 s
Qwen3-VL-4B3.67.913.421.339.1
EgoAction (Qwen3-VL-4B)18.426.130.340.456.1
LLaVA-OV2-8B3.27.613.621.840.5
EgoAction (LLaVA-OV2-8B)20.333.540.348.959.3
Qwen3-VL-32B4.39.615.925.045.3
GPT-6 Astra23.342.755.762.768.1

Adaptation improves temporal localization in every duration bin on both backbones. For actions shorter than 0.5 seconds, m-tIoU increases from 3.6 to 18.4 for Qwen and from 3.2 to 20.3 for LLaVA. For actions lasting at least five seconds, the corresponding improvements are 39.1 to 56.1 and 40.5 to 59.3. These results demonstrate that the benchmark evaluates both brief and sustained actions, while revealing the greater difficulty of precisely localizing brief actions.

6. Activity coverage and performance by data source. (Q10)

We complement the duration-based results in Point 5 with performance breakdowns by activity category and data source. All entries below are EgoActionScore (0–100; higher is better), using the same evaluated records and scoring protocol as the main table.

Performance by activity category. The benchmark’s existing taxonomy organizes 1,407 episodes into 17 scene categories and six activity groups.

Model / systemKitchen and diningHousehold and daily lifeConstruction and repairGardening and outdoorCrafts and laboratoryDesk, devices, and leisure
GPT-6 Astra61.658.057.156.657.059.6
Qwen3-VL-32B-Instruct31.326.727.226.126.329.9
Qwen3-VL-4B-Instruct27.525.622.823.223.127.2
EgoAction (Qwen3-VL-4B)42.140.442.838.942.241.7
LLaVA-OV2-8B26.222.323.523.622.326.1
EgoAction (LLaVA-OV2-8B)48.946.546.747.846.349.5

Performance by data source. We also provide a complementary breakdown across Ego4D, EgoLife, and EPIC-KITCHENS-100 to examine performance across different activity distributions.

Model / systemEgo4DEgoLifeEPIC-KITCHENS-100
Human Expert92.893.883.0
GPT-6 Astra57.459.063.6
EgoAction (LLaVA-OV2-8B)47.546.350.5
EgoAction (Qwen3-VL-4B)42.040.243.6
Flora34.936.144.0
Qwen3-VL-4B-Instruct24.724.529.3
LLaVA-OV2-8B23.922.728.2

Both EgoAction variants achieve higher scores than their corresponding backbones in all six activity groups and across all three data sources.

AuthorsOpenReview ↗

Response to the Review[3/4]

7. Candidate filtering and false-positive penalties. (W1, W4, Q9)

The overlap threshold determines whether a prediction is eligible to correspond to a particular GT action. A GT action without a positive-content candidate receives zero and remains in the evaluated denominator. A prediction that is ineligible for one GT action may still correspond to another. Predictions that remain unmatched are accounted for by the newly added action-level F1 through one-to-one matching: each prediction can match at most one GT action, unmatched predictions count as false positives, and unmatched GT actions count as false negatives. Systematically generating additional captions across both hand tracks therefore incurs a penalty when those captions do not recover additional GT actions. The revised Temporal score averages m-tIoU and F1, and EgoActionScore averages Temporal and Semantic. The updated results are provided below:

Model / systemm-tIoU ↑F1 ↑EgoActionScore ↑
Human Expert88.493.190.4
GPT-6 Astra40.668.659.6
GPT-5.6 Sol31.057.649.8
EgoAction (LLaVA-OV2-8B)32.357.047.9
EgoAction (Qwen3-VL-4B)26.645.941.8
Flora18.646.937.8
Gemini 3.1 Pro Preview18.141.834.8
CapRL-Video-4B16.940.132.4
Qwen3-VL-32B-Instruct11.832.029.2
Qwen3-VL-4B-Instruct9.928.225.9
Qwen3-VL-8B-Instruct10.328.125.2
Qwen2.5-VL-72B-Instruct8.226.124.7
LLaVA-OV2-8B9.826.724.7
OmniCaptioner-IF-7B10.125.023.0
EgoThinker-7B13.724.120.8
InternVL3.5-8B8.825.420.4
OmniCaptioner-IF-3B6.625.720.0
MiniCPM-V-4.5-8B6.720.319.4
Molmo2-8B8.021.218.8
EgoGPT-7B4.516.015.4
VideoLLaMA3-7B4.411.114.2
RynnBrain-4B10.413.313.9
video-SALMONN-2+-3B3.514.210.8

The main comparative conclusions are preserved under the updated evaluation. GPT-6 Astra remains the highest-scoring evaluated model, and both EgoAction variants achieve higher overall scores than all evaluated open-source baselines. Both variants also improve m-tIoU and F1 over their respective backbones.

8. Motion categories and qualitative hand-interaction examples. (W5, Q7)

The Motion field describes the basic movement in a ground-truth action, such as lifting, rotating, or pressing. Models generate free-form captions with temporal intervals and hand labels. They are not required to select a label from the eight Motion categories. We agree that the eight categories limit the detail captured by the Motion score. Verb, Objects, and Tools capture complementary semantic information, so actions sharing a coarse movement pattern remain distinguishable. For example, lifting a cup and lifting a lid share a Motion category but involve different objects.

The qualitative examples in Appendix J (Dataset Examples, Figures 14–16) illustrate both types of hand interaction requested by the reviewer. In the EgoLife example, handling a vase transitions from one-handed manipulation to coordinated two-handed manipulation of the same object. In the Ego4D example, the left hand holds the steering wheel while the right hand handles a writing board. These are represented as independent, overlapping hand timelines. The EPIC-KITCHENS example further shows joint shaking of a cake mold followed by separate actions involving other objects.

9. Human-baseline coverage and results. (Q11)

The human baseline covers all 1,407 benchmark videos. Every video is independently annotated by three annotators in a group separate from benchmark construction. Each annotator’s predictions are scored separately against the fixed benchmark references using the same protocol as the models, and the reported values average the three annotators’ scores for each metric.

Evaluation setm-tIoU ↑F1 ↑Hand ↑Semantic ↑EgoActionScore ↑
Overall88.493.195.590.190.4
Ego4D92.096.296.891.592.8
EgoLife94.497.397.391.893.8
EPIC-KITCHENS-10075.984.291.485.983.0

Thank you again for your careful review and constructive feedback. We hope that our responses and revisions address the concerns raised, and we would be happy to provide further clarification on any remaining questions.

Reviewer_XJiyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for dense per-hand action captioning in egocentric video, named EgoActionBench. Given a full clip, a model must output timestamped atomic action captions for the Left, Right, and Both hand tracks, without action queries or reference boundaries. The benchmark contains 1,407 episodes and 40,384 human-refined annotations from Ego4D, EgoLife, and EPIC-KITCHENS-100. It also proposes a GT-anchored progressive evaluation protocol that retrieves temporally eligible predictions, selects one by LLM-judged semantic agreement, and scores Semantic, Temporal IoU, and Hand assignment. The paper benchmarks 19 VLMs and fine-tunes Qwen3-VL-4B and LLaVA-OneVision-2-8B on automatically annotated training episodes, finding large gaps to human performance, especially in temporal localization and motion description.

Critical strengths
  1. This paper defines a relevant and challenging egocentric benchmark requiring joint recovery of action content, temporal extent, and acting hand from continuous video.
  2. It provides a substantial human-refined annotation set with per-hand tracks and fine temporal granularity, plus a broad evaluation of 19 VLMs and adapted baselines. The GT-anchored evaluation separates semantic, temporal, and hand errors more cleanly than strict one-to-one or merge-then-match protocols.
Critical weaknesses
  1. Since extra predictions are free and one prediction can match multiple GT actions, models may score well by emitting many plausible captions. This undermines the claim of measuring dense action recovery and may change model rankings.
  2. A single text-only LLM judge, without human agreement statistics or multi-judge robustness, is a weak basis for free-form caption scoring. No inter-annotator agreement is reported for the benchmark or human baseline. Adaptation and benchmark use the same source datasets; video-level disjointness is not fully demonstrated. Fine-tuned models also benefit from task-specific supervision while open-source baselines do not, so the main comparison should be framed more carefully.
Questions for authors
  1. Can you report precision/F1 or enforce a fixed prediction budget, and show whether model rankings remain stable? This directly affects whether the benchmark measures dense recovery rather than over-generation.
  2. Can you validate the LLM semantic judge against human judgments, report agreement, and test robustness to a different judge?
  3. Are adaptation and benchmark episodes disjoint at the source-video level, not just episode level?
  4. What is the inter-annotator agreement for benchmark annotations and the human baseline, especially for the Both track and short actions?
  5. How sensitive are results to the semantic-first selection rule, which ignores temporal IoU except for ties? Could a temporally poor but semantically similar prediction be selected over a better-localized one?
8 comments
AuthorsOpenReview ↗

Response to the Review[1/2]

We thank the reviewer for the constructive feedback and for recognizing the relevance of dense per-hand action captioning, the fine-grained annotations, and the diagnostic value of EgoActionBench.

1. Prediction redundancy and model rankings. (W1, Q1)

We agree with the reviewer that GT-anchored scoring alone does not adequately penalize redundant predictions, particularly when one prediction can support multiple GT actions. We have therefore added action-level F1 based on one-to-one matching. Semantic assessment retains GT-anchored correspondence, while F1 allows each prediction to match at most one GT action. Unmatched predictions count as false positives, and unmatched GT actions count as false negatives. This penalizes redundant predictions and missed actions, complementing the interval alignment measured by m-tIoU. Temporal averages m-tIoU and F1, and EgoActionScore averages Temporal and Semantic, so these penalties contribute to the overall score.

The updated results for all evaluated models are provided below, ordered by EgoActionScore.

Model / systemm-tIoU ↑F1 ↑EgoActionScore ↑
Human Expert88.493.190.4
GPT-6 Astra40.668.659.6
GPT-5.6 Sol31.057.649.8
EgoAction (LLaVA-OV2-8B)32.357.047.9
EgoAction (Qwen3-VL-4B)26.645.941.8
Flora18.646.937.8
Gemini 3.1 Pro Preview18.141.834.8
CapRL-Video-4B16.940.132.4
Qwen3-VL-32B-Instruct11.832.029.2
Qwen3-VL-4B-Instruct9.928.225.9
Qwen3-VL-8B-Instruct10.328.125.2
Qwen2.5-VL-72B-Instruct8.226.124.7
LLaVA-OV2-8B9.826.724.7
OmniCaptioner-IF-7B10.125.023.0
EgoThinker-7B13.724.120.8
InternVL3.5-8B8.825.420.4
OmniCaptioner-IF-3B6.625.720.0
MiniCPM-V-4.5-8B6.720.319.4
Molmo2-8B8.021.218.8
EgoGPT-7B4.516.015.4
VideoLLaMA3-7B4.411.114.2
RynnBrain-4B10.413.313.9
video-SALMONN-2+-3B3.514.210.8

The main comparative conclusions are preserved under the updated evaluation. GPT-6 Astra remains the highest-scoring evaluated model, and both EgoAction variants retain higher overall scores than all evaluated open-source baselines. Both variants also improve m-tIoU and F1 over their respective backbones. These results show that the benefits of task-specific adaptation persist when prediction redundancy and missed actions contribute to the evaluation.

2. Robustness to the semantic judge. (W2, Q2)

We have added a controlled cross-judge evaluation in Appendix D. We evaluate fixed predictions from six model configurations on the same 99 clips, using Qwen3-32B and GPT-5.6 Sol as alternative semantic judges. The GT annotations and candidate sets are held fixed, while prediction selection and one-to-one matching are recomputed for each judge. The results are shown below. “Qwen” and “Sol” denote the two semantic judges.

Model configurationEgoActionScore (Qwen)EgoActionScore (Sol)Semantic (Qwen)Semantic (Sol)
EgoAction (LLaVA-OV2-8B, SFT)48.941.752.743.1
EgoAction (Qwen3-VL-4B, SFT)41.236.046.739.6
Qwen3-VL-4B27.323.734.128.2
LLaVA-OV2-8B26.823.033.127.0
OmniCaptioner-IF-7B22.020.827.326.0
EgoGPT-7B16.915.623.120.6

Although absolute scores differ between judges, the ordering of all six configurations is unchanged for both EgoActionScore and Semantic, with Spearman’s rank correlation of 1.0 for both metrics. Both judges also preserve the improvements from supervised adaptation on both backbones. These results support the robustness of the tested model comparisons to replacing the semantic evaluator.

3. Source-video-level separation. (W2, Q3)

The adaptation and benchmark sets are disjoint at the original source-video level, not merely at the episode level. We use the source-video identifier as the grouping unit when constructing the split. All episodes extracted from the same original video are kept on the same side of the adaptation–benchmark partition, even when they correspond to different time intervals or have different episode IDs.

Ego4D, EgoLife, and EPIC-KITCHENS appear in both source lists because we draw adaptation and evaluation examples from these datasets. Within each dataset, however, the two sets use different original videos. The split is constructed specifically for our dense per-hand action-captioning task, rather than by independently assigning extracted episodes to training or evaluation. This prevents different segments of the same source recording from appearing on both sides.

AuthorsOpenReview ↗

Response to the Review[2/2]

4. Human annotation and the Human Expert baseline. (W2, Q4)

We have expanded Appendix G to clarify the annotation responsibilities and the construction of the Human Expert baseline. For benchmark construction, each video is assigned to two team members: one revises the initial annotations against the source video, and the other checks the revisions for errors and omissions. Agentic QC then proposes corrections with supporting frame references, which human reviewers verify against the original video before approving the annotations.

The Human Expert baseline uses a separate group of three annotators. Every video is independently annotated by all three annotators. Each annotator’s predictions are scored separately against the fixed benchmark references using the same evaluation protocol as the models. The reported Human Expert result is the arithmetic mean of the three annotators’ scores for each metric.

The resulting human-baseline scores are summarized below, both overall and by data source. Each annotator is evaluated separately against the fixed benchmark references, and the reported values average the three annotators’ scores.

Evaluation setm-tIoUF1HandSemanticEgoActionScore
Overall88.493.195.590.190.4
Ego4D92.096.296.891.592.8
EgoLife94.497.397.391.893.8
EPIC-KITCHENS-10075.984.291.485.983.0

For the Both track, the guidelines distinguish coordinated manipulation of a shared object from simultaneous independent hand actions. For short actions, annotators inspect native video frames around action transitions to refine the start and end boundaries. These procedures are described in the annotation and quality-control appendices.

5. Sensitivity to semantic-first selection. (Q5)

We acknowledge that semantic-first selection can favor a prediction with higher content agreement but poorer temporal localization. To quantify sensitivity to this choice, we additionally evaluated temporal-first selection, prioritizing temporal IoU over content agreement. Across the six evaluated models, EgoActionScore decreases by 0.1–1.3 points, while the model ranking remains unchanged. Both EgoAction variants continue to outperform their respective backbones. Temporal-first selection increases m-tIoU but lowers Semantic, making the trade-off between action-content correspondence and temporal overlap explicit.

Each entry below reports semantic-first / temporal-first scores.

Model / systemEgoActionScorem-tIoUSemantic
GPT-6 Astra59.6 / 58.340.6 / 42.664.6 / 61.1
EgoAction (LLaVA-OV2-8B)47.9 / 47.132.3 / 33.751.2 / 49.0
EgoAction (Qwen3-VL-4B)41.8 / 40.926.6 / 28.347.4 / 44.7
Qwen3-VL-32B-Instruct29.2 / 28.911.8 / 12.036.6 / 35.9
Qwen3-VL-4B-Instruct25.9 / 25.89.9 / 10.032.8 / 32.5
LLaVA-OV2-8B24.7 / 24.59.8 / 9.931.2 / 30.7

We adopt semantic-first selection because our evaluation is action-first: the correspondence should identify the action being described before assessing its temporal boundaries. A prediction with greater temporal overlap may describe a different action, such as stirring instead of holding a bowl. Selecting it would establish the wrong action correspondence for temporal evaluation. The selected prediction retains its full original interval, so inaccurate boundaries still receive a lower tIoU. This design gives priority to action identity while explicitly scoring the resulting localization error. Appendix I illustrates this trade-off through qualitative examples.

6. Interpretation of task-specific adaptation. (W2)

We agree that task-specific supervision should be explicit when interpreting the comparisons. EgoAction is trained for dense per-hand action captioning, while the pretrained baselines are evaluated without this adaptation. Accordingly, the main table lists EgoAction separately and includes its corresponding unadapted Qwen3-VL-4B and LLaVA-OV2-8B backbones. The cross-model benchmark characterizes the capabilities of the evaluated systems. The adaptation analysis examines the change within each backbone after task-specific training. We therefore frame the finding as improved dense action captioning through task-specific adaptation, and report the temporal and semantic components separately to identify where the improvements occur.

Thank you again for your careful review and constructive feedback. We hope that our responses and revisions address the concerns raised. We would be happy to provide further clarification on any remaining questions.

Dear Reviewer,

Thank you again for your thoughtful review and constructive feedback. We have provided additional experiments and clarifications in our responses and revised manuscript. We would be grateful for your thoughts on whether these updates address your concerns. We would also be happy to provide any further clarification that you may find helpful.

Thank you very much for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear Reviewers and Area Chairs,

We thank the reviewers for their careful review and constructive feedback. We have revised the manuscript, with major changes highlighted in blue. The main updates are:

  1. Updated the evaluation suite and its illustration, clarified matching and score aggregation, and added action-level F1 to account for missed actions and redundant predictions (Section 3.4, Figure 4, and Appendix C).

  2. Expanded the benchmark comparison to 20 baseline models and systems, and updated the main results and analysis under the revised evaluation protocol (Sections 5.1–5.2 and Table 3).

  3. Updated the sensitivity analyses for candidate overlap thresholds and video frame sampling, and provided detailed results by action duration and data source (Section 5.3, Table 4, and Appendices A.1 and K).

  4. Added comparisons across semantic judges to assess the stability of model scores and rankings using fixed predictions on the same evaluation subset (Appendix D).

  5. Added an annotation-model sensitivity experiment on 30 videos, comparing annotation granularity and eight models’ scores and rankings under the original and Opus-5.5 annotations (Appendix F, Tables 10–11).

  6. Expanded dataset statistics and annotation details, including quality control, prompts, and examples, and clarified how Human Expert scores are averaged (Sections 3.2–3.3 and Appendices B, E, G, H, and J).

  7. Expanded the description of supervised fine-tuning and GRPO, including the reward formulation, training configuration, and comparison between SFT and SFT + GRPO (Section 4 and Appendix A).

  8. Clarified the distinction between actions and atomic actions, updated comparisons with related benchmarks, and added extended related-work discussion (Sections 2 and 3.1, Table 1, and Appendix L).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.