acceptodds
Withdrawn from ICLR 2027

TVTA: Trajectory-Aware Viseme-Guided Temporal Aggregation for Event-Based Lip Reading

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Event-based lip reading has recently emerged as a promising direction for visual speech recognition, benefiting from the high temporal resolution and motion sensitivity of event cameras. However, existing methods typically perform spatial compression before sufficient temporal modeling, which may suppress sparse and localized motion trajectories that are crucial for distinguishing similar lip movements. Moreover, most current approaches optimize temporal representations mainly at the word-classification level, leaving the underlying articulatory structure weakly constrained. To address these limitations, we propose a temporal aggregation framework that preserves localized motion trajectories and introduces viseme-level sequence guidance. First, we introduce Trajectory-Aware Differential Aggregation (TDA), which performs local temporal modeling at each spatial location before adaptive spatial aggregation. Second, we propose Viseme-Guided Aggregation (VGA), which introduces viseme-level sequence supervision to guide temporal aggregation for word recognition. Third, we employ EMA teacher-student consistency to regularize training and improve generalization. Extensive experiments on DVS-Lip and DVS-LRW100 show that the proposed method establishes new best results under the corresponding evaluation settings, while qualitative analyses further reveal that the learned temporal representations capture meaningful viseme-aware structure from event streams.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Lip Reading, Event Camera, Temporal Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pycYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines five off-the-shelf components into a complex multi-branch pipeline: DropPath, BiMamba, Word-to-viseme lookup + CTC loss, Mean Teacher EMA consistency for lip reading.

Critical strengths

I think the paper provides sufficient details to understand the proposed combination, making it friendly to readers who have a limited background in lip reading and necessay compoents.

Critical weaknesses
  1. I feel the paper does not provide sufficient scientific value and reads like an engineering integration paper. Every component is a direct adaptation of well-known techniques. Combing existing methods without new insights and motivations does not meet the ICLR standards.

  2. Evaluation is flaw. Based on my search, baselines like FAST-HG do not leverage pre-training. The authors claim "new best results" solely by pretraining. Claiming SOTA by adding more training data than baselines is a critical evaluation flaw.

Questions for authors

Please see Weaknesses.

Reviewer_LSxAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new method for event-based lipreading. Trajectory-Aware Differential Aggregation (TDA) and Viseme-Guided Aggregation (VGA) are introduced, which model location-wise trajectories on the encoder feature map before global spatial aggregation and incorporate viseme-aware sequence supervision, respectively. This improves temporal representation learning for word recognition and leads to better performance. A comparison with other SOTA methods on two datasets is provided together with an ablation study.

Critical strengths

The paper proposes a new approach for event-based lipreading which is an underexplored area.

Critical weaknesses
  • Performance improvements over SOTA methods are marginal. The improvement in Acc1 in Table 1 is 0.2% and in Acc in Table 2 is 0.18. Are these improvement statistically significant?

  • It is questionable if the comparison with other SOTA methods in Table 1 is fair. The proposed method is the best only when pretraining is used. In case the other methods do not use pretraining then this is not a fair comparison, as the proposed method is trained on more data. It would be useful if datasets and the number of hours used for training each method are mentioned in the table.

  • The ablation study is not very informative. It would be better if only one component was removed at a time. This would reveal the impact of each of them separately. Removing multiple components at the time does not demonstrate how important each of them is.

  • It is not entirely clear why a student-teacher architecture is used. Why not training a single model? Even if the proposed approach results in better results (should be verified using the ablation study proposed above) a motivation for using a teacher-student approach should be provided.

  • Fig 5., it seems that the model which performs better (teacher or student) depends on the temporal modelling approach used (LSTM or GRU or Mamba). Please provide some insights on why this might happen.

Questions for authors

Please see critical weaknesses section.

Reviewer_p8fYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In my view, this paper proposes TVTA for event-based lip reading by combining trajectory-aware temporal modeling, viseme-guided aggregation, and teacher-student consistency learning. I find the problem meaningful, and the method achieves competitive results on two benchmarks. However, I feel the technical novelty is somewhat limited, and the experimental comparisons do not fully convince me of the advantages of the proposed approach.

Critical strengths

I think the paper addresses an interesting problem in event-based lip reading, particularly the preservation of fine-grained temporal information. The trajectory-aware temporal modeling and viseme-guided supervision are reasonably motivated, in my view. I also find the experiments on DVS-Lip and DVS-LRW100 competitive, and the component ablations are helpful. I believe the proposed framework mainly combines bidirectional Mamba, spatial attention, CTC-based viseme supervision, and EMA teacher-student learning. In my view, the differences from existing event-based temporal modeling methods are not sufficiently demonstrated.

Critical weaknesses

I believe the proposed framework mainly combines bidirectional Mamba, spatial attention, CTC-based viseme supervision, and EMA teacher-student learning. In my view, the differences from existing event-based temporal modeling methods are not sufficiently demonstrated. In my view, there are the following weak points: (1) Fairness of experimental comparisons. I am concerned that on DVS-Lip, the proposed method achieves 77.49% accuracy without pretraining, below FAST-HG (79.37%). Its best result of 80.04% relies on additional pretraining using DVS-LRW100. I do not think comparing this result directly with methods without equivalent pretraining is entirely fair. (2) Insufficient validation of trajectory modeling. I find that TDA models temporal features at fixed spatial locations, whereas lip movements involve spatial displacement. In my view, the paper does not clearly establish whether these location-wise features can effectively represent actual lip-motion trajectories. (3) Limited analysis of viseme supervision. I note that the viseme labels are derived from word pronunciations rather than annotated visual articulations. I feel the reliability of these labels and their actual contribution to temporal reasoning need further justification. (4) Missing efficiency analysis. I think applying bidirectional Mamba across spatial locations introduces additional computational costs, yet the paper does not provide sufficient comparisons of parameters, inference time, and computational complexity.

Questions for authors

(1) Fairness of experimental comparisons. I am concerned that on DVS-Lip, the proposed method achieves 77.49% accuracy without pretraining, below FAST-HG (79.37%). Its best result of 80.04% relies on additional pretraining using DVS-LRW100. I do not think comparing this result directly with methods without equivalent pretraining is entirely fair. (2) Insufficient validation of trajectory modeling. I find that TDA models temporal features at fixed spatial locations, whereas lip movements involve spatial displacement. In my view, the paper does not clearly establish whether these location-wise features can effectively represent actual lip-motion trajectories. (3) Limited analysis of viseme supervision. I note that the viseme labels are derived from word pronunciations rather than annotated visual articulations. I feel the reliability of these labels and their actual contribution to temporal reasoning need further justification. (4) Missing efficiency analysis. I think applying bidirectional Mamba across spatial locations introduces additional computational costs, yet the paper does not provide sufficient comparisons of parameters, inference time, and computational complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.