acceptodds
Under review as a conference paper at ICLR 2027

ActionRefiner: Refining Predictions with Inverse Verbs for Action Recognition in Egocentric Videos

Abstract

Despite advances in egocentric action recognition using vision-language models, distinguishing actions with identical nouns but inverse verbs (e.g., "open fridge" vs. "close fridge") remains challenging because these actions often only differ in their temporal order of execution. We present ActionRefiner, an approach to improve action recognition via a Causal Temporal Generation Process (CTGP) and a residual classification module. Given action intermediate spatio-temporal features, CTGP learns latent temporal representations for verbs and nouns conditioned on action context via VAE-based modelling and a domain estimator. A residual classifier then produces corrective logits that are fused with the baseline logits to predict actions. ActionRefiner consistently improves strong baselines and establishes new state-of-the-art performance on Epic-Kitchens-100 and EGTEA Gaze+. In particular, it achieves a new state-of-the-art 57.7 Top-1 accuracy on Epic-Kitchens-100 and 82.7 on EGTEA Gaze+. Notably, the majority of improvements over LaViLa on EGTEA Gaze+ is on inverse-verb actions, demonstrating the importance of learning action-aware temporal representations of verbs and nouns.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
action recognition, egocentric videos

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DHQKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActionRefiner, a plug-in framework for improving egocentric action recognition, particularly for actions involving inverse verbs that share the same noun but differ in temporal dynamics. The method introduces a Causal Temporal Generation Process (CTGP) to learn verb- and noun-specific latent temporal representations using VAE-based models conditioned on action-level context. A residual classification module then generates corrective logits, incorporating posterior averaging, negative verb–noun swapping, and top-k masked fusion to refine predictions from a frozen action recognition baseline. Experiments on EPIC-Kitchens-100 and EGTEA Gaze+ demonstrate improvements across multiple baselines, including LaViLa, AVION, TIM, and MTCN. The method achieves 57.7% and 82.7% Top-1 accuracy on the two datasets, respectively. Additional ablation studies and inverse-verb analyses are presented to support the effectiveness of the proposed components.

Critical strengths
  1. Consistent improvements across multiple baselines and datasets. ActionRefiner demonstrates consistent performance gains when integrated with different action recognition models, including TIM, AVION, LaViLa, and MTCN. For example, it improves Top-1 accuracy by 1.3% over TIM and 2.5% over LaViLa on EPIC-Kitchens-100. These results support the practical effectiveness and general applicability of the proposed refinement framework.

  2. Well-motivated problem formulation with targeted empirical validation. The paper addresses the challenging problem of distinguishing actions involving inverse verbs, which often share similar visual appearances but differ in temporal dynamics. The analysis on EGTEA Gaze+ shows larger improvements on inverse-verb actions than on other actions, providing empirical support for the proposed motivation. This task-specific analysis strengthens the practical significance of the work beyond overall accuracy improvements.

Critical weaknesses
  1. Insufficient justification for the causal interpretation of CTGP. The paper presents Causal Temporal Generation Process (CTGP) as a central contribution. However, the proposed transition distributions primarily characterize action-conditioned temporal dependencies, and it remains unclear what distinguishes them from conventional conditional temporal modeling. The paper does not sufficiently establish the causal interpretation or identifiability of the learned latent representations. Although comparisons with Mamba and vanilla VAE demonstrate performance improvements, they do not necessarily validate the claimed causal mechanism. This weakens the theoretical foundation and novelty of the paper's main contribution.

  2. Insufficient experimental evidence to isolate the contribution of CTGP. ActionRefiner introduces additional verb/noun representations extracted from powerful visual encoders, including V-JEPA 2.1 Gigantic (2B). While the reported improvements are encouraging, the existing ablations do not fully establish whether these gains primarily arise from the proposed CTGP or from additional visual representations, supervision, and model capacity. More controlled comparisons with simpler action-conditioned temporal models using identical features and training objectives would help establish the necessity of the proposed framework. This limits the strength of the empirical evidence supporting its methodological contribution.

Questions for authors
  1. Clarification of the causal interpretation. What specifically makes the proposed CTGP a causal temporal generation process rather than a conventional action-conditioned temporal model? Could the authors clarify the causal assumptions underlying the transition distributions and whether any identifiability guarantees or empirical evidence support the causal interpretation of the learned latent variables? This distinction is important for assessing the theoretical contribution and novelty of the method.

  2. Isolating the contribution of CTGP. While the comparisons with Mamba and vanilla VAE are useful, they do not fully isolate the contribution of the proposed temporal mechanism. Could the authors provide a comparison with a simpler action-conditioned temporal model (e.g., a conditional GRU, Transformer, or VAE) using the same verb/noun features, supervision, and comparable model capacity? Such a comparison would help determine whether the improvements arise specifically from CTGP or from additional conditioning and representation learning.

  3. Statistical significance of inverse-verb improvements. In Table 5, the improvements on inverse-verb actions are relatively modest compared to those on other actions (e.g., +1.1% versus +0.8% for LaViLa). Could the authors report the sample sizes of these subsets and provide uncertainty estimates or statistical significance analyses? This would help establish whether the proposed method specifically addresses inverse-verb confusion, rather than providing a general improvement in action recognition.

1 comment

We thank the reviewer for suggestions and comments.

Causal interpretation. Conventional conditional temporal models learn one unconstrained joint mapping to produce the whole vector ztz_t from its history and context, there is no structure about how individual components of ztz_t relate to others. CTGP employs a factorized component-wise transition. Each coordinate of ztz_t is generated from the noise term, lagged history and action context, this is the content in nonlinear ICA in causal literature. The underlying assumptions are that verb and noun latent variables are influenced by time-delayed history, context and exogenous action features.

The identifiability theorems in Yao et al. (2022) are proven under the assumption that the domain variable is given and exogenous, i.e. a known, discrete domain/regime index. However, there are no such indices in action recognition task/dataset, this is the reason we use a continuous GRU. The assumption in Yao et al. (2022) does not hold once domain vector is replaced by a value estimated by a network, since it is no longer independent of the estimation process the theorem is meant to prove. Therefore the identifiability experiments in Yao et al. (2022) (e.g., Mean Correlation Coefficient) is not feasible on egocentric video. Li et al. (2024a) also use a continuous GRU to estimate domain variables with the assumption that they can be learned empirically and do not provide theoretical proof on identifiability. CTGP essentially is inline with this assumption and we do not claim identifiability either. We will revise the manuscript to state this directly, i.e. ActionRefiner uses transition-prior VAE as a structured conditioning mechanism, not as a source of identifiability guarantees.

Isolating CTGP contribution. We additionally test GRU and transformer to compare with CTGP. We replace CTGP with GRU or transformer and keep the rest components and settings the same as ActionRefiner. Concretely, verb feature from visual encoder are concatenated with action feature and fed to verb GRU/Transformer. The same concatenation is applied to noun GRU/Transformer. The model sizes are on the same level as CTGP.

ModelTop-1 Acc
LaViLa81.8
ActionRefiner82.7
Mamba81.7
GRU81.7
Transformer81.8

Both GRU and transformer are on par with the base model LaViLa. The results confirm the contribution of CTGP and show the necessity of CTGP.

Statistical test. The total test set has 2022 videos, the inverse verb subset has 1381 videos and the reset subset has 641 videos. We run McNemar’s test on both subsets with LaViLa.

ModelInverse Verb Top-1 (ΔTop-1)Inverse Verb pRest Top-1 (ΔTop-1)Rest p
LaViLa + ActionRefiner81.1 (+1.1)0.07886.1 (+0.8)0.18

For the inverse verb subset, p=0.078 and the rest subset has p=0.18. On inverse verb subset it is approaching significance while not significant on the rest set.

Reviewer_LSnROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a key challenge in egocentric action recognition: distinguishing semantically similar actions with identical nouns but inverse verbs, which existing video models struggle with as differences mainly stem from subtle temporal motion patterns. The authors propose ActionRefiner, a plug‑in residual refinement module built upon a Causal Temporal Generation Process (CTGP). CTGP employs VAE‑based latent modelling to learn verb‑noun temporal representations conditioned on holistic action context. The module produces corrective logits that are fused with frozen backbone predictions via top‑k masked fusion. Evaluated on Epic‑Kitchens‑100 and EGTEA Gaze+, ActionRefiner shows accuracy gains over multiple backbones.

Critical strengths
  1. This paper is clearly written with well‑organized figures, tables and appendix, easing understanding of its pipeline and experiments.
  2. The authors identify a real pain point in egocentric action recognition: existing models fail heavily on inverse‑verb actions sharing identical nouns, where discriminative signals only lie in subtle temporal dynamics. They propose ActionRefiner as a model‑agnostic plug‑in refinement module targeting exactly this failure mode.
Critical weaknesses
  1. Missing comparisons against modern multimodal large‑language‑model baselines. Recent MLLMs such as Qwen‑VL are capable of video action recognition. The paper provides no empirical comparison against these strong alternatives, leaving unclear whether the proposed module still brings value against MLLM‑based solutions.
  2. Modest overall performance improvements. ActionRefiner yields limited accuracy gains: most improvements range from 1‑2.5 points over baselines, and only 0.9 top‑1 accuracy gain on the strong LaViLa backbone. Such small margins raise concerns about practical significance.
  3. Lack of empirical validation for causal identifiability of CTGP. Though drawing upon causal temporal representation literature, the paper provides no task‑specific verification that the learned latent variables satisfy identifiability assumptions. The causal claim remains mainly a design motivation rather than experimentally validated property of CTGP.
Questions for authors

Q1. Recent multimodal large‑language models such as Qwen‑VL support video‑based action recognition. Why are these MLLM baselines not included in your evaluations? Could you clarify whether your ActionRefiner still yields meaningful improvements compared with MLLM‑based solutions under your experimental setup?

Q2. Can you provide task‑specific empirical evidence to verify that the learned verb‑noun latent variables satisfy the causal identifiability assumptions in this egocentric action recognition task?

1 comment

We thank the reviewer for suggestions and comments.

Comparison Qwen-VL. MLLMs such as Qwen-VL are trained mainly for open-ended video understanding (QA, grounding, captioning, reasoning) on long, untrimmed videos. Egocentric action recognition is a classification task, it instead requires choosing one of action classes from short, trimmed clips.

We now finetune Qwen3-VL-8B-Instruct on EK100 with sft and qlora. We ask the model to describe the action in the video and restrict the output to the action phrases. We calculate the accuracy by two methods. First we compare the generated text against ground truth action text and second, we use a Bert text encoder to encode the generated text and all action text, then we compare the predicted text embedding against full action text embedding and choose the most similar one as the predicted action. Qwen3-VL-8B-Instruct has 20.0 accuracy for exact text comparison and 17.6 for text embedding comparison, compared to ActionRefiner’s 57.7. This shows the MLLM model struggles at action recognition despite its capability on video understanding tasks. A general-purpose MLLM remains far from action recognition models under the closed-set classification. ActionRefiner will not work with MLLMs. Our ActionRefiner needs temporal features and action logits of all actions. MLLMs do not have action logits.

Performance improvement. We emphasize that the gains are consistent and complementary. ActionRefiner improves all five backbone spreads over two datasets, The model architectures across video-language (LaViLa, AVION), long-context (TIM) and multimodal (MTCN).

Identifiability. The identifiability theorems in Yao et al. (2022) are proven under the assumption that the domain variable is given and exogenous, i.e. a known, discrete domain/regime index. However, there are no such indices in action recognition task/dataset. The identifiability experiments in Yao et al. (2022) (e.g., Mean Correlation Coefficient) is not feasible on egocentric video, those experiments use synthetic or instrumented data (modified Cartpole, motion capture) where the true generating latents are known by construction. EPIC-Kitchens and EGTEA Gaze+ do not have such ground truth, therefore we do not claim a formal identifiability verification. We will revise the manuscript to state this directly, i.e. ActionRefiner uses transition-prior VAE as a structured conditioning mechanism, not as a source of identifiability guarantees.

Reviewer_L7bXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ActionRefiner, a plug-in module for egocentric action recognition. A frozen action baseline (TIM, AVION, LaViLa, or MTCN) supplies action logits and temporal features. Two separately fine-tuned visual encoders (V-JEPA 2.1 Gigantic, 2B parameters, on EPIC-Kitchens-100; MViTv2-S on EGTEA Gaze+) supply verb and noun features. A “Causal Temporal Generation Process” (CTGP) encodes verb and noun features with two VAEs whose transition priors are conditioned on an action-context vector from a GRU over the baseline features. A residual MLP head maps the latents to correction logits, which are averaged over K posterior samples, masked to the baseline’s top-k classes, and added to the baseline logits. Training uses cross-entropy, a verb–noun swap margin loss, reconstruction, two KL terms, and verb/noun supervision on latent means.

Results: +1.3 Top-1 on TIM (56.4 → 57.7), +2.2 on AVION, +2.5 on LaViLa on EPIC-Kitchens-100; +0.9 on LaViLa (81.8 → 82.7) and +2.0 on MTCN on EGTEA Gaze+. Ablations compare CTGP with vanilla VAEs and Mamba, remove the domain estimator, and split EGTEA into “inverse-verb” and “rest” subsets.

Critical strengths
  • Gains are consistent across five baselines on two datasets. The module improves every baseline it is attached to, and Table 11 reports mean ± std over runs (CTGP 57.66 ± 0.11 vs. vanilla VAE 56.56 ± 0.03). Table 8 shows the EK-100 gain persists with a 300M encoder instead of the 2B one. This is the main evidence that the module does something beyond noise.
  • Reporting is transparent on reproduced baselines. The paper gives its own reproductions (TIM 56.6, LaViLa 50.7) next to the published numbers and gives full compute and hyperparameter details (Appendix A, B).
Critical weaknesses
  • The key control is missing: simple fusion of the verb/noun encoder with the baseline. The method adds a fine-tuned 2B V-JEPA 2.1 encoder (72.9% verb, 62.3% noun on EK-100, Table 7) to a smaller baseline. The gain may come from ensembling a second strong model, not from CTGP. The paper never reports (a) late fusion of the V-JEPA verb and noun probabilities with the baseline action logits (p(verb)·p(noun) combined with ℓ_base), (b) a plain MLP on concatenated V-JEPA features and ℓ_base with the same top-k masking, or (c) the action accuracy of the V-JEPA encoder trained directly on action labels. Table 11 shows that two vanilla VAEs on the same features give 0.0 gain over the reproduced TIM; this is surprising if the features carry useful information, and suggests the gain depends on the L_vn supervision (Table 3: 56.5 → 57.4), not on the “causal” transition. Without baseline (a) or (b), the contribution of CTGP cannot be judged.
  • The inverse-verb claim is not supported by the numbers. The abstract states “the majority of improvements over LaViLa on EGTEA Gaze+ is on inverse-verb actions”. From Table 5 (full 81.8, inverse 80.0, rest 85.3), the inverse-verb subset is about 66% of the test set. The per-subset gains are +1.1 and +0.8, a difference of 0.3 points on a subset whose size is not stated. Tables 6 and 12 show the domain estimator changes the inverse-verb number by 0.3 and the verb-only/noun-only ablations are within 0.1 of each other. These differences are at or below the run-to-run std reported in Table 11. The claim that ActionRefiner specifically targets inverse verbs is the paper’s stated motivation and is not demonstrated.
  • The “causal” framing adds notation but no verified content. Equations 1–6 are taken from Yao et al. (2022) and Li et al. (2024a). The paper does not use or test any identifiability result. The “domain” variable z^u in those works is a discrete non-stationarity index; here it is a continuous GRU output over features from a model already trained on the action label, so it mainly injects the action prediction into the prior. How the inverse transition f^{-1} in Eqs. 5–6 is implemented is not stated. Removing the domain estimator costs 0.2 points (Table 6). The method is a conditioned VAE with a residual head; the paper should describe it as such.
  • Hyperparameters are backbone-specific and tuned on the evaluation split. k = 25 for TIM but k = C_a for AVION and LaViLa; α = 1 on EK-100 and 1.5 on EGTEA; the swap loss helps TIM by 0.1 and hurts LaViLa by 0.1 (Tables 4, 10). EK-100 results are reported on the validation set, and Appendix B says k was chosen by “validation performance”. If the same split was used for selection and reporting, the +1.3 is partly selection. The ablation increments (57.4 → 57.6 → 57.7) are within one std.

Minor: TIM is cited as “Cai et al. (2025)” in Sec. 1 and 2.2 (should be Chalk et al., 2024). “Mamba transformer” is not a transformer. Two of three qualitative examples (Fig. 2) are failures explained as label noise. Only Top-1 action accuracy is reported; verb and noun Top-1 and Top-5, and the standard EK-100 unseen/tail-class splits, are missing. The total cost (V-JEPA 2B fine-tuning ≈ 40 h, 27 h per feature stream, ~500 H200 hours) is large for a +1.3 gain and weakens the “plug-in” description.

Questions for authors
  • Report, on EK-100 with TIM, the Top-1 of: (a) late fusion ℓ_base + α·log(p_verb ⊗ p_noun) from the V-JEPA probes, with the same top-k mask; (b) an MLP on [V-JEPA verb feats; noun feats; ℓ_base] with the same mask; (c) V-JEPA 2.1 Gigantic fine-tuned directly on action labels.
  • State the number of clips in the EGTEA inverse-verb and rest subsets. Give a paired significance test (e.g., McNemar) for the +1.1 vs +0.8 gain and for the domain-estimator ablation.
  • Which split was used to select k, α, λ, and the schedule on EK-100? If it was the validation split, report results with a held-out portion of the training set used for selection.
  • How is f^{-1} in Eqs. 5–6 implemented? What does the Jacobian term contribute to the loss in practice?
  • Why do two vanilla VAEs on the same V-JEPA features give zero gain (Table 11) while CTGP without L_vn gives +0.9 on LaViLa (Table 9) but −0.1 on TIM (Table 3)?
2 comments
AuthorsOpenReview ↗

Reply Part 1.

We thank the reviewer for suggestions and comments.

Late Fusion. We did run the requested late fusion experiments.

ModelActionVerbNoun
(a) late fusion ℓ_base + α·log(p_verb ⊗ p_noun)58.777.268.9
(b) an MLP on [V-JEPA verb feats; noun feats; ℓ_base]56.775.766.4
(c) V-JEPA 2.1 Gigantic fine-tuned directly on action labels49.268.960.7
ActionRefiner + (a) late fusion59.177.569.1

For late fusion on Tim, we add to each TIM action logit the log-probabilities of its verb and noun from the V-JEPA probes. These act as penalties: when TIM's top action is wrong but the probes recognise the correct verb or noun, the wrong action loses more than the correct one, which can move to the top. Late fusion can be seen as a stronger ℓ_base. Adding ActionRefiner's residual on top further raises accuracy further to 59.1. ActionRefiner is trained on action labels and fixes errors that late fusion does not, it adds a complementary signal, although a smaller one than on TIM alone . The MLP stays at TIM’s level 56.7. A simple MLP does not improve the base model, this shows the necessity of ActionRefiner. We retrained V-JEPA Gigantic and fine-tuned it directly on the action labels. Where it reaches only 49.2 accuracy.

Hyperparameters and held-out portion. EK-100 and EGTEA differ substantially in label-space size, which changes the relative scale of the verb-noun prior and the action logits. Dataset dependence of α and k is therefore expected for fusion weights.

We selected k, α\alpha and λ\lambda on the validation split. This follows the standard protocol for EK-100 action recognition. The labels of test are not public, and most state-of-the-art methods report results only on the validation split (Chalk et al., 2024). All baselines in our Table 1, including LaViLa, AVION and TIM, are validation results. The official training code of LaViLa, AVION and TIM also keeps the checkpoint with the best validation accuracy. Nevertheless, we ran the requested experiment.

As requested, we held out 10% of the training videos of each participant (7,030 clips) and retrained ActionRefiner on the remaining clips. We selected k, λvn\lambda_{vn}, α\alpha and the checkpoint on the held-out clips with six 400-epoch runs ( k∈k \in{10, 15, 25, 50, CaC_a}, λvn∈\lambda_{vn} \in {0.1, 0.5} and α∈\alpha \in {0.5, 0.75, 1, 1.25, 1.5}). The selected model reaches 57.2 Top-1 on val versus TIM’s 56.6 and ActionRefiner’s 57.7).

However, these clips are only new to ActionRefiner, TIM and the V-JEPA probes were already trained on the whole train set. We therefore run a more informative check. We selected on data that no component has seen, i.e. we split the validation participants into two halves, select on one half and report on the other, and obtain 57.6 (mean over 1,000 random splits).

Inverse transition and Jacobian term. f^i−1\hat f_i^{-1} is implemented as a separate MLP for each latent dimension. It takes z^t−L,…,z^t−1\hat z_{t-L}, \dots, \hat z_{t-1}, z^t(i)\hat z_t(i) and z^tu\hat z_t^u as input, and its output is the estimated noise term. For each latent dimension i, the inverse transition estimates the process noise,

ϵ^t(i)=f^i−1(z^t(i)∣z^t−1,z^tu).\hat\epsilon_t(i) = \hat f_i^{-1}\left(\hat z_t(i) \mid \hat z_{t-1}, \hat z_t^u\right).

Since ϵ^t(i)\hat\epsilon_t(i) depends on z^t\hat z_t only through its i-th coordinate, the Jacobian is triangular. Its determinant therefore reduces to the product of the diagonal terms. The transition prior is the sum of a noise likelihood term and a Jacobian term,

log⁡pϕ(z^t∣z^t−1,z^tu)=∑i=1nlog⁡p(ϵ^t(i))+∑i=1nlog⁡∣∂f^i−1∂z^t(i)∣.\log p_\phi\left(\hat z_t \mid \hat z_{t-1}, \hat z_t^u\right) = \sum_{i=1}^{n} \log p\left(\hat\epsilon_t(i)\right) + \sum_{i=1}^{n} \log \left| \frac{\partial \hat f_i^{-1}}{\partial \hat z_t(i)} \right|.

This term replaces the intractable prior in the KL term of the ELBO,

LKLtrans=Ez^∼q[log⁡q(z^t∣xt)−log⁡pϕ(z^t∣z^t−1,z^tu)].\mathcal L_{\mathrm{KL}}^{\mathrm{trans}} = \mathbb E_{\hat z \sim q}\left[\log q\left(\hat z_t \mid x_t\right) - \log p_\phi\left(\hat z_t \mid \hat z_{t-1}, \hat z_t^u\right)\right].

AuthorsOpenReview ↗

Reply Part 2.

Identifiability. The identifiability theorems in Yao et al. (2022) are proven under the assumption that the domain variable is given and exogenous, i.e. a known, discrete domain/regime index. However, there are no such indices in action recognition task/dataset, this is also the reason to use a continuous GRU. The assumption does not hold once domain vector is replaced by a value estimated by a network, since it is no longer independent of the estimation process the theorem is meant to prove. Li et al. (2024a) also use a continuous GRU to estimate domain variables with the assumption that they can be learned empirically and do not provide theoretical proof on identifiability. CTGP essentially is inline with this assumption and we do not claim identifiability either. We will revise the manuscript to state this directly, i.e. ActionRefiner uses transition-prior VAE as a structured conditioning mechanism, not as a source of identifiability guarantees.

Statistical test. The total test set has 2022 videos, the inverse verb subset has 1381 videos and the reset subset has 641 videos. We run McNemar’s test on both subsets with LaViLa.

ModelInverse Verb Top-1 (ΔTop-1)Inverse Verb pRest Top-1 (ΔTop-1)Rest p
LaViLa + ActionRefiner81.1 (+1.1)0.07886.1 (+0.8)0.18
LaViLa + ActionRefiner w/o ds80.8 (+0.8)0.5186.1 (+0.8)1

For the inverse verb subset, p=0.078 and the rest subset has p=0.18. On inverse verb subset it is approaching significance while not significant on the rest set.

CTGP performance difference. CTGP uses beta VAEs instead of vanilla VAEs. Moreover, TIM is the stronger backbone, the residual head has less corrective signal.

Minor comment. We will fix the wrong citation and Mamba term and add more qualitative examples. The verb and noun top-1 and top-5 accuracy are shown in the table below.

For unseen and tail class splits, we follow previous works to report only results on the validation set on EK100. The ‘plug-in’ property we describe is not computational efficient but generalizable across different base models. In addition, V-JEPA only needs to be finetuned once and can be applied to multiple baseline.

ModelAction@1Action@5Verb@1Verb@5Noun@1Noun@5
LaViLa50.769.771.790.562.282.7
LaViLa + ActionRefiner53.572.074.192.264.383.8
AVION53.971.472.689.765.081.8
AVION + ActionRefiner56.674.475.592.067.185.0
TIM56.473.975.691.166.383.8
TIM + ActionRefiner57.774.675.990.467.283.2

Comments on the paper

1 comment

Dear Reviewers,

Authors posted their responses to your queries. Please read these messages and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.