- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces EgoActionBench, a benchmark for dense per-hand action captioning in egocentric human videos. Given a complete clip without other information, a model must generate temporally localized atomic action descriptions on Left, Right, and Both hands. The benchmark comprises 1407 episodes and 40384 human-refined action annotations curated from Ego4D, EgoLife, and EPIC-KITCHENS-100. Its ground-truth-anchored evaluation retrieves temporally eligible predictions, selects one prediction by field-level semantic agreement, and evaluates that same prediction for semantic content, temporal IoU, and hand assignment. Experiments compare 19 baseline models and adapt Qwen3-VL-4B and LLaVA-OneVision-2-8B using automatically annotated training episodes. Adaptation improves EgoActionScore by 10.4 and 20.0 points, respectively, with additional evidence from frame-sampling sweeps and transfer to HanDyVQA.
Critical strengths
- An important and well-specified task beyond coarse video understanding. The benchmark requires autonomous discovery of atomic manipulation actions, temporal boundaries, and anatomical hand assignments. It explicitly preserves sustained single-hand actions alongside shorter actions of the other hand and distinguishes coordinated bimanual manipulation from unrelated simultaneous actions. This work also introduces a multi-stage human verification pipeline, which makes this a potentially useful resource for studying the detailed structure of human manipulation. This matters because coarse activity descriptions can conceal precisely the action transitions and hand roles needed for learning from human demonstrations.
- Conducted meaningful and sufficient ablation experiments. Both adapted backbones improve across Semantic, Temporal, and Hand scores, with particularly substantial Temporal gains of 16.8 and 25.7 points. The adapted models outperform all open-source baselines evaluated in the paper, and sampling sweeps show that these gains are not replaced by simply providing the original backbones with denser observations. Transfer evaluation on HanDyVQA uses identical video inputs and recordings disjoint from adaptation data, increasing accuracy from 61.01% to 63.80% without further VQA fine-tuning, although Objects AP declines.
Critical weaknesses
Data source concern. The paper describes automatically annotated adaptation episodes from Ego4D, EPIC-KITCHENS, and EgoLife, but does not clearly identify the models used to generate and refine the training labels or provide a sufficiently detailed training-specific annotation procedure, including prompts, filtering criteria, and quality checks. The manuscript also does not clearly state whether the adaptation annotations, data-generation code, and source-video split manifests are publicly available or will be released. Regarding evaluation, the authors state that adaptation and benchmark data are separated by source-video identifiers, which is a useful safeguard against direct fine-tuning overlap. However, the benchmark reuses videos from existing datasets, and the paper neither specifies their original official splits nor audits possible exposure during the evaluated models' pretraining or prior task-specific training. This does not establish that leakage occurred, but leaves contamination risk unresolved. Is there any special consideration or method for the selection of test set video sources here?
The metric's treatment of prediction errors and temporal boundary ambiguity needs stronger justification. I am concerned that EgoActionScore does not fully capture the correctness of a generated action sequence: it applies no independent false-positive penalty, allows a prediction to support multiple ground-truth actions, and lets Temporal and Hand contribute even when semantic agreement is zero. A further concern is that atomic action boundaries are not necessarily uniquely identifiable timestamps. During continuous manipulation, reasonable annotators may place a transition slightly differently depending on whether they emphasize contact onset, motion onset, or object-state change. A difference of a few tenths of a second may therefore be acceptable in some cases rather than a genuine localization error. However, the protocol scores temporal IoU against a single reference interval without an explicit boundary-tolerance or uncertainty model. With a median action duration of only 0.63 seconds, such differences can substantially affect the temporal score, potentially conflating annotation ambiguity with model error. The candidate-overlap threshold provides flexibility in retrieving matches, but does not remove this issue from the final IoU score, and threshold sensitivity alone does not validate the treatment of boundary uncertainty.
Questions for authors
Please see the weaknesses
5 comments
Response to the Review[2/2]
4. Boundary ambiguity and interpretation of temporal scores. (W2)
We acknowledge that reasonable annotators can place some action transitions at slightly different timestamps. Our annotation guidelines use visible changes in hand–object contact, motion phase, and object state to identify boundaries. Annotators inspect native video frames around these transitions, and ambiguous cases undergo further review. This provides a consistent annotation procedure at the video’s native temporal resolution.
Temporal IoU measures the overlap between predicted and reference action intervals. Its sensitivity is particularly relevant for brief actions, where a timing offset can span a substantial part of the action or extend into a neighboring operation. Human performance provides an empirical reference under the same scoring protocol.
The human baseline covers all benchmark videos, each independently annotated by three annotators. Their predictions are evaluated separately against the fixed references, and the reported scores average the three annotators’ results:
| Evaluation set | Human m-tIoU ↑ | GPT-6 Astra m-tIoU ↑ | Human F1 ↑ |
|---|---|---|---|
| Overall | 88.4 | 40.6 | 93.1 |
| Ego4D | 92.0 | 37.5 | 96.2 |
| EgoLife | 94.4 | 40.3 | 97.3 |
| EPIC-KITCHENS-100 | 75.9 | 45.5 | 84.2 |
These results show the level of human agreement with the references under strict interval scoring and the remaining model–human gap across sources.
To directly examine temporal localization with allowance for boundary shifts, we additionally evaluate Boundary-F1 at tolerances of 0.1, 0.2, and 0.5 seconds. Boundaries extracted from predicted and reference action intervals are matched one-to-one within each video and hand track. A match is accepted when the absolute timing difference falls within the specified tolerance. Unmatched predicted boundaries count as false positives, and unmatched reference boundaries count as false negatives. Counts are aggregated across videos to compute micro-F1. For this additional validation, we compare Qwen3.5-4B before and after task-specific SFT, with both models using 2 FPS video inputs:
| Model | Boundary-F1 (0.1 s) ↑ | Boundary-F1 (0.2 s) ↑ | Boundary-F1 (0.5 s) ↑ |
|---|---|---|---|
| Qwen3.5-4B Base | 10.81 | 18.21 | 29.89 |
| Qwen3.5-4B + SFT | 23.44 | 35.79 | 54.37 |
SFT improves Boundary-F1 across all three tolerances, including the more permissive 0.5-second setting. These results complement the strict interval-based scores by showing that adaptation improves boundary detection even when the evaluation explicitly accepts temporal offsets within these windows.
We appreciate the time and thought you have devoted to evaluating our work. Your suggestions have helped us clarify the annotation process and strengthen the evaluation protocol. We hope our responses address your concerns and would be glad to clarify any remaining questions.
Dear Reviewer,
Thank you for your careful assessment of our work. Following your suggestions, we have provided additional experimental results and clarified the relevant aspects of the paper. We would welcome your thoughts on these updates whenever you have an opportunity to review them. Please let us know if there is anything further we could explain or discuss.
We sincerely appreciate your time and consideration.
Response to the Review[1/2]
Thank you for recognizing the importance of the task, the preservation of independent hand timelines, and the value of the annotation pipeline and ablation studies. We appreciate your detailed comments on data provenance and evaluation design.
1. Data provenance and reproducibility. (W1)
Both the initial generation and subsequent automatic refinement of the adaptation labels use GPT-6 Astra. The annotations contain timestamped action captions organized into Left, Right, and Both tracks, providing supervision for action descriptions, temporal intervals, and hand assignments.
Candidate benchmark episodes are first selected automatically and then manually screened for coherent manipulation tasks and verifiable hand–object interactions. The adaptation and benchmark sets are separated by original source-video ID. All episodes extracted from the same source recording remain on the same side of the partition, including episodes with different temporal ranges or episode identifiers.
This source-video separation controls direct overlap with our adaptation data. If a model’s pretraining data are not disclosed, we cannot verify whether it saw the benchmark’s source videos during pretraining. The benchmark evaluates newly constructed, fine-grained action annotations over existing video sources. And we will release the adaptation training annotations and source-video identifiers documenting the adaptation–benchmark split.
2. False-positive penalties and reuse of predictions. (W2)
We have added action-level F1 through one-to-one matching to measure action coverage and prediction redundancy. GT-anchored semantic assessment can reuse a prediction when it contains information relevant to multiple reference actions. In the separate F1 calculation, each prediction can match at most one GT action. Unmatched predictions count as false positives, and unmatched GT actions count as false negatives.
For illustration, if three duplicate predictions describe one GT action, at most one receives a match: the other two count as false positives, giving an F1 of 50%. If one prediction spans two GT actions, it can contribute at most one true positive, leaving one false negative and giving an F1 of 66.7%. These examples illustrate how the revised metric penalizes redundant predictions and under-segmentation.
The revised Temporal score averages m-tIoU and F1, and EgoActionScore averages Temporal and Semantic. These changes are summarized in Section 3.4 and detailed in Appendix C, with illustrative examples in Figure 6. Representative results under the updated evaluation are:
3. Content requirements for action matching. (W2)
The revised protocol requires positive content agreement before a prediction can be selected. This follows our action-first objective: identifying the action provides the basis for evaluating its timing and acting hand. A prediction with zero content agreement is ineligible for both GT-anchored selection and one-to-one F1 matching. If a GT action has no positive-content candidate, its Semantic, temporal IoU, and Hand scores are all zero, and it remains in the evaluated denominator. It is also unmatched in the F1 calculation. Thus, a prediction with the correct interval and hand label but no agreement with the reference action’s content receives no credit for that action.