- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SPAFT, a method to prune and then fine-tune PPG foundation models. Initially, the paper identifies that existing CNN based PPG FMs have significant redundancies and many blocks and filters do not contribute toward task prediction. Therefore, SPAFT first identifies redundancies in resnet blocks by structurally comparing cosine similarities of the filters within a block. Then, these are pruned and fine-tuned. Experiments demonstrate that SPAFT outperforms baselines across both classification and regression with rates of 82.8% and 87.5%, respectively.
Critical strengths
- SPAFT is well-motivated through experiments with PPG FMs. It addresses an important problem of adapting FMs for different health tasks that could require different PPG characteristics for prediction.
- Comprehensive baselines: The paper compares SPAFT against well-justified baselines across four PPG foundation models and many fine-tuning paradigms, demonstrating an extensive experimental design.
Critical weaknesses
Addressing these weaknesses will help me reconsider my score:
- Statistical Significance: The results reported in Table 1 and Table 2 are single values which are close. For example, for Ring-PPG HR, SPAFT obtains 58.447 for AnyPPG and the 2nd best is 59.285. Consequently, it is unclear if SPAFT's improvement is meaningful. In particular, the win rates are calculated based on which method achieves the best numerical result. This approach treats negligible differences as wins, which is not meaningful. To address this issue, include statistical tests between the top 2 results to identify an useful win rate metric.
- Dataset provenance: Dataset details such as the total number of samples, people, devices are necessary to contextualize the result. This will help the reader understand whether the improvements are meaningful. For example, an improve of 0.1 on 10 people is less likely to be significant compared to one evaluated across 1000. Furthermore, the paper emphasizes heterogeneous deployments, however, four of the evaluation tasks are on a private dataset and most tasks focus on either heart rate or blood pressure. I recommend either tempering the claims on the cardiovascular domain or diversifying the evaluation by adding more tasks. For example, glucose or respiration rate from the already used BUT-PPG dataset, or depression from the ECSMP dataset (https://www.sciencedirect.com/science/article/pii/S2352340921009355).
- Overclaims:
- (1) Efficiency: The experiments in the paper do not support the efficiency claims. Perhaps, these results are in the appendix, which is missing. Also, I recommend including the primary supporting evidence in the main text instead of the appendix.
- (2) Task-relevant substructures: The paper indicates that SPAFT finds task-relevant substructures. While the prune-then-finetune approach could learn these structures, additional evidence such as representations before and after SPAFT that demonstrate the substructure is necessary to substantiate this claim (e.g., a heart rate substructure).
- (3) Case Study: In Figure 4, QLoRA does show inter-class separation, hence saying it fails to distinguish the two waveforms is an overclaim. Visually, SPAFT is only marginally better. Could you provide quantitative metrics to support this case study?
Questions for authors
Questions and suggestions that will affect my assessment are in the Weaknesses section.
3 comments
Thank you for the additional experiments and clarifications. My concerns have been addressed and I have increased my score.
We sincerely thank the reviewer for carefully reviewing our work and increasing the score. We appreciate the constructive feedback, which helped us strengthen both the analysis and the experiments.
Thank you again for your time and review
Dear Reviewer JRyB,
Thank you again for your time and review. Have we addressed your concerns? We would appreciate the opportunity to further clarify any remaining perceived issues as you finalize your assessment.
Best,
Authors of submission 25063