- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies label-efficient activity recognition from paired geometric (pose/skeleton) and inertial inputs. F2B jointly trains a fusion predictor and two independently deployable modality branches. On selected unlabeled pairs, the detached weak-view fusion distribution supervises the strongly augmented branches, alongside supervised training and fusion pseudo-label learning. The paper evaluates five datasets and four label budgets, with teacher-source ablations on three datasets and comparisons with unimodal specialists. The complete training recipe improves most branches over supervised training; matched ablations show that the incremental benefit of the F2B loss depends on the dataset and target branch.
Critical strengths
-
A useful joint training and deployment objective. One training run retains both complete-input fusion and independently usable single-sensor predictors, without a separately pretrained teacher. This addresses a concrete deployment need when sensor availability changes, and its value does not require every branch to outperform a dedicated unimodal model.
-
Meaningful controls and transparent reporting. The paper includes matched no-distillation controls, teacher-source comparisons, specialist comparisons, and evaluation-unit-based uncertainty estimates. Table 5 provides positive evidence for the F2B loss on CZU-MHAD: geometric and inertial branches improve by 3.71 and 6.48 percentage points, respectively, with unadjusted 95% confidence intervals excluding zero. These results establish that the core loss can be beneficial, while the reported negative and uncertain results help delimit its applicability.
Critical weaknesses
-
The broad gains of the complete recipe are stronger than the evidence for the core F2B loss. The 38/40 branch improvements over supervised training also include the contribution of unlabeled fusion training, as the authors acknowledge. In the matched 10%-label ablation, the geometric branch changes by −0.51 pp on MM-Fit and −0.67 pp on mRI; inertial gains are +2.83 and +2.50 pp, but all four intervals include zero. Together with the positive CZU-MHAD results, this supports conditional effectiveness rather than a consistently established incremental benefit. Tables 2 and 3 evaluate the complete training recipe at four label budgets. However, the comparison that removes only the F2B loss while retaining the remaining training components is reported at the 10% budget in Tables 4 and 5. The independent contribution of the distillation loss at other budgets therefore remains unresolved.
-
The added value of the full fusion teacher remains insufficiently established relative to close alternatives. Online ensemble-to-branch distillation has been studied in ONE, while prior incomplete-modality learning and multimodal distillation methods have explored knowledge transfer under missing-input settings. The paper acknowledges related work in both areas. F2B’s sensor-specific branches and deployment design are meaningful differences. However, the clearest teacher-source finding—other-branch teaching outperforming same-branch teaching—supports cross-modal transfer without isolating the advantage of the full fusion teacher. The paper also reports that no reduced teacher dominates across datasets. Given the overlap with prior work, stronger evidence that the full fusion teacher improves on simpler cross-branch teaching is needed to establish the method’s contribution.
Questions for authors
-
Clarification using existing controls: What are the paired performance differences between the full fusion teacher and the other-branch teacher for the direct branches, and how variable are these differences across evaluation units? Do the existing results show an additional benefit from including the student’s own modality in the teacher?
-
Isolating the distillation effect across label budgets: Tables 2 and 3 report results at four label budgets. Beyond the 10% setting, are there comparisons that remove only the F2B loss while retaining the same class-adaptive threshold and other training components? If available, do these comparisons show the same dataset- and branch-dependent pattern as Table 5?
2 comments
Thank you for your response.
Thank you for the careful reading and for two questions that go straight to the core of the paper. Your distinction between the full training recipe and the F2B loss itself was especially useful and shaped the revision. In short:
Q2: the F2B loss at every label budget
As you suggested, we removed only the F2B loss and kept the class-adaptive threshold and every other component, at all four budgets on four datasets, in two pipelines. The main pipeline is the setup of Tables 2–3 (batch size 32 on CZU-MHAD; the 25% representation selector on mRI and UTD Kinect V2) and reproduces their F2B means at every budget. The control pipeline is the setup of Table 5.
F2B minus no distillation, in pp (geometric / inertial):
The gains that survive Holm correction are MM-Fit inertial at 15% in both pipelines, CZU-MHAD inertial at 25% in the main pipeline, and CZU-MHAD inertial at 10% and 15%, CZU-MHAD geometric at 25% and UTD Kinect V2 geometric at 15% in the control pipeline. Each is positive in every held-out unit. The one significant decrease is MM-Fit geometric at 50%.
Your expectation of a dataset- and branch-dependent effect holds. CZU-MHAD gains at every budget in both pipelines, while the mRI geometric branch stays near zero in the control pipeline. The loss can also hurt: at 50% labels it lowers both MM-Fit branches. On the segmented datasets, the main pipeline has more positive means (22 vs 18 of 24) but fewer significant gains (1 vs 4), and no cell is Holm-significant in both. The CZU-MHAD 10% gains you highlighted (+3.7 / +6.5) become +3.7 / +2.8. Section 5.7, Table 7 and supplementary Sections I.2–I.3 report all values.
Q1: fusion teacher versus other-branch teacher
This is the right test, and the answer narrows our claim. Other-branch minus fusion teacher at 10% labels (control pipeline at Section 5.4), as mean [unadjusted 95% CI] and the units where the other-branch teacher is higher. A negative value means the fusion teacher is better.
The fusion teacher has the higher mean in two of these six cells, and only MM-Fit inertial has an interval that excludes zero. Across all budgets in the main pipeline, it has the higher mean in 16 of 24 comparisons, and none survives Holm correction. There, cross-sensor teaching has the higher mean than self-teaching in 18 of 24 comparisons (one survives Holm correction), but in only 3 of 8 on mRI. The student's own modality in the teacher has a branch-dependent effect: removing its direct logit changes MM-Fit geometric by +0.98 pp (5/5 units) and inertial by −1.34 pp (0/5) (supplementary Table 3), the gate and residual still see both sensors.
We agree that the added value of the fusion teacher is not established, and the revision removes that claim (abstract, introduction, Section 5.4, conclusion). We keep the fusion head as the teacher because it is also the two-sensor predictor (Section 3.2) and unlike in ONE, each branch observes a different sensor and is evaluated alone. This does not affect Q2, which compares F2B with no distillation. The contribution we claim is the one your review identified: one training run that keeps the fusion predictor and both single-sensor predictors without a pretrained teacher, together with the matched evidence above for the loss.
Your review made the paper more precise. We would appreciate a reassessment in light of the matched ablations now covering every label budget under the main-table protocol.