- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a flow matching approach for sign language production by combining Riemannian and metric flow matching, enabling modelling directly on the rotation space. To do that they develop a two stage pipeline where a correction network is initially trained to as a naturalness-guided term that will guide the flow matching model to remain on the 6D rotation data manifold. Experimentally the authors demonstrate that the proposed approach can achieve SOTA results on multiple benchmark datasets.
Critical strengths
- The paper is very polished and well written
- The paper is the first work to combine Riemannian on Sing Language Production, enabling modelling directly on the rotation space.
- The intuition is very clear and scientifically sound.
- The proposed method achieves state-of-the-art results, outperforming previous approaches by a large margin.
Critical weaknesses
- Naturaleness
a) Training a prior on joint rotations alone assumes that the joints are independent. However joints belong in a kinematic tree and a combination of plausible joint rotations can still lead to infeasible body poses. What do the authors do to mitigate such cases? Could the authors discuss this?
b) Unless I am mistaken, the k-means needs to be trained per dataset. This raises a question about how k-means was trained for the mutli-lingual model. Could the authors clarify?
c) Would the k-value be affected when more datasets are added?
- Unjustified Motivation.
a) The motivation of modelling directly on the 6d data space can lead to off-manifold poses is clear and valid theoretically. However, most prior works that use linear FM rely on Gram–Schmidt transform that will always map the generated rotations to valid 6d rotations. In addition, the authors never demonstrate qualitatively and quantitatively the intermediate poses of the proposed model. Are the intermediate generation indeed valid 6d rotations that produces feasible body poses?
b) Followin the above, an ablation of 6D with linear FM and Gram–Schmidt transform would justify the need of the proposed Riemannian FL.
- Experimental Results
a) The results on BLEU are surprisingly good. Could the authors comment or attribute this improvement?
- Novelty
The paper combines Riemannian FM [1] with Metric FM [2]. The proposed approach is a straightforward combination of the previous approaches and hence the claims should be toned down. In addition, the authors do not discuss or compare directly with any of those works explicitly.
[1]: Chen et al. Flow Matching on General Geometries, ICLR 2024 [2]: Kapuśniak et al.: Metric Flow Matching for Smooth Interpolations on the Data Manifold, NeurIPS 2024
Questions for authors
I would appreciate if the authors could clarify my concerns especially regarding naturaleness and the intermediate 6d rotations produced by their method, as I described in the previous section.