- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents M2E, which encodes compensated onset–apex optical flow into 12-dimensional regional motion features and uses a class-weighted RBF-SVM for micro-expression recognition. The authors evaluated benchmark performance, component ablations, cross-dataset transfer, SHAP explanations, and Raspberry Pi deployment.
Critical strengths
- The method addresses small-sample recognition by reducing dense facial motion to a compact set of regional measurements. Head-motion compensation and sample-wise normalization address nuisance variation before classification, reducing what the classifier needs to learn from limited data.
- The 12-dimensional encoding combines regional vertical motion with left–right horizontal differences and inner–outer eyebrow differences. This is a compact way to retain both local movement and relationships between regions without AU labels. It also gives the SHAP analysis concrete facial-motion variables to explain.
- The anchor-based version reports 76.94% UF1 and 79.99% UAR on Composite, exceeding the lightweight baselines listed in Table 1. The authors also test it on a Raspberry Pi and examine performance without CK+ augmentation. Together, these results support the practical value of a simple representation and classifier, even though the relative efficiency needs a fairer comparison.
Critical weaknesses
- M2E assumes the onset and apex frames are known and thus calculate the optical flow between the onset and apex. However, detecting the onset, apex, and offset is very challenging for MER/FER. This has been demonstrated from the experimental results on the SMIC, which is the only dataset without groundtruth apex labeling. The performance of the propose method is inferior to FRL-DGT.
- In Figure 1, TV-L1 and MEFlowNet can be mistaken for sequential steps, although they are alternative configurations. The regional abbreviations need a legend, and “Magnitude weighted” and “Direction mapping” do not explain how the 12 features are constructed. The symbols should also match the equations.
- In Section 3.4, the bilateral horizontal difference is described as capturing asymmetry. However, symmetric motion (−1, 1) and unilateral motion (−2, 0) both give −2. The feature cannot distinguish these patterns, so this interpretation should be revised.
- The claim that Equations (9)–(11) suppress isolated high-magnitude noise needs qualification. As a counterexample, 99 aligned flow values of 1 and one outlier of 1000 yield a response of approximately 10, compared with 1 without the outlier. This does not establish a failure on the benchmark, but shows that the aggregation is not generally robust to large outliers.
- Table 6 compares variants of the proposed features but does not include MDMO, a closely related regional flow descriptor. Table 3 tests arithmetic averaging, but this does not replace a comparison with an existing descriptor. A matched comparison using the same flow, compensation, normalization, and classifier is needed to establish the benefit of the proposed encoding.
- Tables 1 and 2 use different front ends, but Raspberry Pi measurements cover only the anchor-based version. The runtime comparison with LAENet in Appendix E.9 does not provide matched hardware measurements and includes training-data preparation. Table 12 reports memory increases rather than total peak memory. These measurements do not establish the full pipeline’s efficiency advantage.
Questions for authors
- How does the 12-dimensional representation compare with MDMO under identical preprocessing, training data, and classification settings?
- Can you clarify or revise the asymmetry and noise-suppression claims discussed above?
- Please report complete inference latency and total peak memory for both configurations and LAENet on the same device.
- Please specify the ROI expansion parameter, eye-closure threshold, regional landmark indices, and the conversion to the upward-positive coordinate convention.
8 comments
Reply to Reviewer 1BJE [Part 2/2]
W6 / Q3 Latency and memory
On the same desktop, the complete anchor-based M2E pipeline, from the given onset and apex frames to the predicted label, takes less time per sample than LAENet preprocessing alone on the three MER datasets. Feature construction, timed from reading the two frames and including face cropping, takes 10.00–35.00% less time than LAENet preprocessing (Table R2). The SVM stage adds almost nothing. Training and testing over all LOSOCV folds of 3DB take 2.05 s in total, which is under 5 ms per test sample even with training included (Table R4). M2E anchor and LAENet run on the CPU, and MEFlowNet uses the GPU.
Table R2. Per-sample time on the same desktop, including face cropping (s).
| Dataset | Samples | M2E anchor (CPU) | LAENet (CPU) | Reduction | M2E MEFlowNet (GPU) |
|---|---|---|---|---|---|
| CASME II | 145 | 0.52 | 0.80 | 35.00% | 0.51 |
| SAMM | 133 | 0.86 | 1.31 | 34.35% | 0.76 |
| SMIC | 164 | 0.45 | 0.50 | 10.00% | 0.44 |
| CK+ | 289 | 1.33 | 2.96 | 55.07% | 0.51 |
Table R3 gives the mean of the per-sample peak host RSS, in addition to the memory increase in revised Table 17 (originally Table 12).
Table R3. Mean per-sample peak host RSS during feature construction or preprocessing (MB).
| Dataset | M2E anchor | LAENet | M2E MEFlowNet |
|---|---|---|---|
| CASME II | 140.86 | 144.74 | 2223.17 |
| SAMM | 152.82 | 157.86 | 2225.34 |
| SMIC | 134.64 | 142.57 | 2223.33 |
| CK+ | 137.12 | 140.91 | 2220.63 |
Table R4. Downstream learning and evaluation.
| Workload | Time (s) | Host RSS (MB) | Memory aggregation |
|---|---|---|---|
| M2E SVM LOSOCV (anchor) | 2.05 | 124.69 | Whole-experiment peak |
| M2E SVM LOSOCV (MEFlowNet) | 1.89 | 124.55 | Whole-experiment peak |
| LAENet CK+ pre-training | 10.81 | 2251.46 | Whole-pre-training peak |
| LAENet MER LOSOCV | 557.29 | 2255.28 | Mean of fold peaks |
The anchor pipeline also runs on the Raspberry Pi (revised Table 17), and MEFlowNet was evaluated on the desktop GPU. All of these numbers are in Appendix E.10 (originally E.9).
Q4 Reproducibility details
Appendix A now specifies the ROI sizes, the eye-closure threshold, and the coordinate convention, with the FaceMesh indices in Table 6. Eyebrow boxes are expanded by δ = 4 pixels on each side. Cheeks, mouth corners, and chin use 8 × 8-pixel boxes around their landmarks. The weighting uses ε = 10⁻⁶, and the closed-eye threshold is EAR = 0.20 (Soukupova & Cech, CVWW 2016). We flip the sign of the vertical flow during aggregation so that positive means upward, and the feature names and SHAP labels follow this convention. These are the values in the released code.
We hope these additional results address your questions. We will respond promptly to any further questions during the discussion period.
References
[R1] X. Fan et al. SelfME: Self-Supervised Motion Learning for Micro-Expression Recognition. In CVPR, 2023.
Follow-up on our response
Dear Reviewer 1BJE,
Your questions led to several concrete changes, including the matched comparison with MDMO in Appendix E.9, the corrected wording in Section 3.4, the revised Figure 1, and the same-platform latency and memory measurements in Appendix E.10.
We would appreciate hearing whether these resolve the points you raised, and we can provide further details if needed.
If you find these changes satisfactory, we hope they merit a reconsideration of your assessment. Thank you very much.
Reply to Reviewer 1BJE [Part 1/2]
We thank the reviewer for the detailed and technical questions, which helped us tighten several parts of the paper. Revised text is in orange, and the new measurements are in Appendices E.9–E.10.
W1 Dependence on onset and apex localization
M2E recognizes a micro-expression from an annotated or estimated onset–apex pair, which is also the setting of FRL-DGT (Zhai et al., CVPR 2023) and SelfME [R1]. We use official apex labels where they exist and estimate the SMIC apex with D&C-ROIs (Appendix D.3). The SMIC comparison does not isolate the effect of apex estimation, because the two methods also differ in motion representation, training objective, and model capacity.
W2 Figure 1
The two front ends are alternatives, and the revised Figure 1 draws them as parallel branches. MEFlowNet works on the onset and apex frames directly and never sees TV-L1 flow. The figure now outlines the ROIs on the face. The caption spells out the region abbreviations and defines μx1 / μx2 and μy1 / μy2 as the positive and negative parts of μx(R) and μy(R) in Eq. (11). Section 3.4 and Table 5 show how the regional responses become the 12 features.
W3 / Q2 Bilateral horizontal difference
Section 3.4 now calls this feature "the relative horizontal motion of the two sides" instead of "asymmetry". Keeping the left and right components separately, as a 15-dimensional variant, does not improve results consistently across the three datasets (revised Table 8, originally Table 6).
W4 / Q2 Outlier robustness of the aggregation
We removed the claim about isolated noise from Section 3.4. The text now only says that the weighting in Eqs. (9)–(11) gives more weight to strong motion vectors.
W5 / Q1 Comparison with MDMO
With the same flow, compensation, classifier, splits, and training pool, M2E outperforms MDMO (Liu et al., IEEE TAFFC 2016) on every dataset, with and without CK+. MDMO keeps its own 36 regions and its own magnitude normalization, which is part of how its features are built, and M2E keeps max-abs.
Table R1. Matched-input comparison (UF1 / UAR, %).
M2E is also higher in the source-only setting, by 8.73 / 7.76 points on MMEW and 28.09 / 24.07 on CK+. A subject-level bootstrap on 3DB gives 95% intervals for the difference that all exclude zero. Details are in Appendix E.9 (Tables 14–16).