- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes NIRP-Net for remote photoplethysmography (rPPG) estimation from synchronized RGB and near-infrared facial videos under challenging illumination. The method uses two priors: the relative stability of actively illuminated 940-nm NIR observations and the absence of target physiological signals in background regions. Its prior-guided feature disentanglement module subtracts background-derived disturbance from NIR features, then combines RGB–NIR feature differences with RGB background features to suppress disturbance in the RGB representation. A gated local cross-attention module subsequently incorporates complementary NIR information, followed by a temporal Transformer and waveform regression head.
Critical strengths
-
Relevant problem and coherent design. Illumination interference is a meaningful obstacle to reliable rPPG. Refining each modality before fusion is well motivated, and the architecture connects background-based disturbance estimation with adaptive use of NIR information. The manuscript appropriately describes NIR as relatively stable rather than illumination invariant.
-
Promising reported accuracy. In Table 2, NIRP-Net reduces driving-dataset MAE from 2.33 bpm for the strongest listed multimodal baseline to 1.73 bpm, approximately a 26% reduction. It also reports the lowest MAE and RMSE on the indoor and FaceIVL datasets. These consistent improvements are encouraging.
-
Useful experimental breadth. The evaluation includes three datasets, a cross-dataset experiment, and ablations of feature refinement, fusion, and temporal modeling. Table 4 is particularly informative: RGB refinement alone does not improve the GCA baseline, whereas combining NIR refinement, RGB refinement, and GCA produces the strongest result. This supports an interaction between the components.
-
Potentially valuable dataset contribution. FaceIVL provides synchronized RGB–NIR observations, contact PPG references, and a subject-independent evaluation split. Acquisition through a shared optical path is useful for studying spatially corresponding multimodal features. Ethics approval and participant consent are reported.
Critical weaknesses
-
The claimed physiological–illumination disentanglement is not directly demonstrated. Equations (1)–(4) implement learned feature subtraction, but the manuscript does not establish that the removed components specifically represent illumination or that the retained features preserve physiological information. RGB–NIR differences can also contain modality-specific pulse responses, appearance differences, and alignment errors. Improved heart-rate accuracy demonstrates the usefulness of the architecture, but does not by itself validate the proposed interpretation. Controlled illumination perturbations, measurements of pulse and illumination content before and after refinement, or targeted representation analyses would strengthen the central claim.
-
Important training and preprocessing details are missing. Appendix B introduces auxiliary regularization terms for “global-local disturbance balancing” and “local illumination consistency,” but provides neither their equations nor their weights. These terms may materially affect the claimed separation mechanism. The manuscript also leaves unclear how facial masks are generated, how background pooling is normalized, and how much background remains after facial cropping. Because masking occurs after the convolutional encoders, background features may also contain facial information through their receptive fields. The initialization and constraints of the learnable coefficients α and β should be specified; without a constraint, Equation (3) need not behave as a convex balance between disturbance references.
-
The evaluation does not sufficiently isolate illumination robustness. FaceIVL results are aggregated across normal and low-light recordings. Separate results for each condition are necessary to show where the improvement arises. The acquisition description also does not establish whether illumination changes dynamically within recordings or primarily differs between two recording conditions. Illumination intensity, spectrum, and temporal variation are not quantified. Furthermore, the solar-spectrum motivation does not directly explain robustness under indoor lighting. Tests varying ambient illumination and active NIR strength would better establish the operating conditions of the proposed prior.
-
Key comparisons are absent, limiting the novelty and complementarity claims. The paper identifies Shao and Hu’s NIR-prior method as closely related, but does not include it in the quantitative comparisons. The “Diff” ablation is insufficiently specified to determine whether it faithfully represents that approach. Matched RGB-only and NIR-only variants of the proposed backbone are also missing. An NIR-only baseline is especially important under low visible light: it would establish how much improvement comes from cross-modal complementarity rather than access to a stronger sensing modality. These omissions make the advantage over prior-guided alternatives difficult to assess.
-
Split definitions and statistical uncertainty require clarification. FaceIVL explicitly uses a subject-independent split, but the manuscript does not state whether the driving dataset’s 7:3 split or the indoor dataset’s eight folds separate subjects. Subject and recording identities, clip overlap, and the splitting procedure should be documented to rule out leakage. Appendix B reports a single random seed, with no variation across runs or uncertainty estimates for the performance differences. This matters particularly for the smaller improvements on Indoor and FaceIVL. Cross-dataset evaluation is also limited to Driving → Indoor; transfers involving FaceIVL would provide stronger evidence of generalization.
-
There are reporting inconsistencies and incomplete dataset documentation. Table 7 appears to exchange the IENet and FusionPhys MAE/RMSE values relative to Table 2. Its computational measurements use 160-frame clips, whereas training and architecture details specify 128-frame clips; this difference needs explanation. Throughput measurements should specify batch size, precision, and whether preprocessing is included. For FaceIVL, age and skin-tone distributions, failure or exclusion criteria, and a data-release plan are not provided, limiting assessment of its coverage and reproducibility.
Questions for authors
Please see the weakness part.
4 comments
Response to Reviewer wDkY (Part 2/3)
3. Illumination Conditions and Robustness Evaluation (W3)
Evaluation by lighting condition. We separately evaluate normal-light and low-light recordings. NIRP-Net achieves the lowest MAE and RMSE under both conditions. Its advantage therefore extends to side-lit conditions with facial shadows and uneven illumination, beyond normal-light recordings.
| Method | Normal-Light MAE ↓ (bpm) | Normal-Light RMSE ↓ (bpm) | Low-Light MAE ↓ (bpm) | Low-Light RMSE ↓ (bpm) |
|---|---|---|---|---|
| PhysNet | 0.68 | 0.97 | 1.68 | 4.75 |
| RhythmMamba | 1.04 | 2.74 | 1.51 | 3.20 |
| IENet | 0.65 | 1.12 | 1.60 | 3.13 |
| FusionPhys | 0.61 | 1.24 | 1.64 | 3.56 |
| NIRP-Net | 0.56 | 0.71 | 1.33 | 3.12 |
Illumination conditions and temporal variation. Normal lighting uses indoor ceiling lights, with approximately 200 lx at the face. Low lighting uses a visible-light source positioned approximately 45° to the side, providing about 50 lx and creating facial shadows and uneven illumination. The conditions are recorded separately, without deliberately introducing substantial illumination changes within individual recordings. Small head and facial movements are allowed. FaceIVL therefore evaluates performance across lighting conditions and under spatially nonuniform illumination, while MR-NIRP Driving provides complementary evaluation under complex illumination variations during real-world driving.
Indoor NIR prior. Solar-spectrum attenuation around 940 nm motivates the outdoor setting. Indoors, active 940-nm illumination provides the auxiliary NIR observation. Paired FaceIVL statistics show that, from normal to low lighting, RGB pulse-band SNR decreases by 1.96 dB on average, whereas NIR decreases by only 0.17 dB, supporting its relative stability across visible-light conditions. We use a fixed NIR illumination intensity during acquisition. Experiments with different NIR illumination intensities are planned for future work.
4. Comparison with Related Work and Cross-Modal Complementarity (W4)
NIRP-Net achieves lower estimation errors, demonstrating its advantage over both the simplified Diff baseline and our implementation of this NIR-prior method. When preparing the original manuscript, we could not obtain official code for Shao and Hu’s method. We therefore constructed a Diff baseline within our backbone for a controlled comparison, without presenting an uncertain implementation as a full reproduction. Following a similar principle, Diff separately projects RGB and NIR facial features, subtracts them, and applies temporal modeling and waveform regression to the resulting difference features. Following the reviewer’s suggestion, we additionally implemented Shao and Hu’s method based on the published details and evaluated it on MR-NIRP Driving using the same protocol.
| Method | MAE ↓ (bpm) | RMSE ↓ (bpm) | MAPE ↓ (%) | ρ ↑ |
|---|---|---|---|---|
| Shao and Hu (our implementation) | 2.60 | 6.27 | 3.96 | 0.64 |
| NIRP-Net | 1.73 | 4.17 | 2.51 | 0.83 |
The full model outperforms both single-modal variants under normal and low lighting, including NIR-only under low lighting, demonstrating the benefit of RGB–NIR complementarity.
| Input Setting | Normal-Light MAE ↓ (bpm) | Normal-Light RMSE ↓ (bpm) | Low-Light MAE ↓ (bpm) | Low-Light RMSE ↓ (bpm) |
|---|---|---|---|---|
| RGB-only | 0.675 | 1.187 | 3.690 | 8.834 |
| NIR-only | 0.810 | 1.359 | 2.205 | 4.593 |
| RGB+NIR (Full model) | 0.563 | 0.706 | 1.328 | 3.120 |
These variants use matched encoder architectures, temporal modeling modules, and regression heads, with consistent data splits, training protocols, and checkpoint selection rules. Each single-modal variant removes processing that depends on the other modality. The full model outperforms both single-modal variants under normal and low lighting, including NIR-only under low lighting. These results demonstrate that the improvement comes not merely from access to NIR observations, but also from exploiting RGB–NIR complementarity.
Our response continues in the next comment.
Response to Reviewer wDkY (Part 3/3)
5. Data Splits, Statistical Stability, and Cross-Dataset Evaluation (W5)
Subject-independent splits. All datasets use subject-independent splits, with no overlap in subject identities, recordings, or clips between training and testing. For MR-NIRP Driving, we group all recording identifiers belonging to the same subject before applying an approximately 7:3 training/test split, rounding down the number of training subjects. MR-NIRP Indoor uses eight-fold leave-one-subject-out evaluation, with seven subjects for training and one for testing in each fold. FaceIVL uses a gender-stratified, subject-independent split. All recordings and clips from each subject remain within the same partition, preventing leakage through shared identities or adjacent clips from the same recording.
NIRP-Net achieves the lowest mean MAE and RMSE on both Indoor and FaceIVL across three random seeds. We conduct repeated experiments with three random seeds on Indoor and FaceIVL, keeping data splits, method-specific hyperparameters, and checkpoint selection rules fixed. The results below show that NIRP-Net’s advantage is not limited to a single random initialization. Values are reported as mean ± standard deviation across three seeds.
| Method | Indoor MAE ↓ (bpm) | Indoor RMSE ↓ (bpm) | FaceIVL MAE ↓ (bpm) | FaceIVL RMSE ↓ (bpm) |
|---|---|---|---|---|
| RhythmMamba | 0.748 ± 0.110 | 1.125 ± 0.179 | 1.286 ± 0.248 | 2.824 ± 0.621 |
| FusionPhys | 0.540 ± 0.042 | 0.763 ± 0.028 | 1.181 ± 0.048 | 2.681 ± 0.019 |
| IENet | 0.795 ± 0.094 | 1.180 ± 0.125 | 1.219 ± 0.134 | 2.648 ± 0.575 |
| NIRP-Net | 0.495 ± 0.045 | 0.690 ± 0.058 | 0.979 ± 0.049 | 2.367 ± 0.122 |
NIRP-Net achieves the lowest overall MAE and RMSE on Driving → FaceIVL. This extends the original Driving → Indoor evaluation. The normal- and low-light breakdown is also provided in our response to Reviewer gJu1.
| Method | MAE ↓ (bpm) | RMSE ↓ (bpm) | MAPE ↓ (%) | ρ ↑ |
|---|---|---|---|---|
| RhythmFormer | 4.86 | 11.67 | 4.48 | 0.72 |
| RhythmMamba | 5.76 | 12.22 | 5.34 | 0.70 |
| IENet | 4.50 | 10.67 | 4.16 | 0.76 |
| FusionPhys | 3.89 | 11.02 | 3.47 | 0.75 |
| NIRP-Net | 3.62 | 10.17 | 3.30 | 0.78 |
6. Reporting Consistency and Dataset Documentation (W6)
The MAE/RMSE entries for IENet and FusionPhys were swapped in Table 7 (Table 9 in the revised manuscript). “160 frames” was a typographical error, the actual clip length for computational-cost evaluation is 128 frames. Both errors have been corrected in the revised manuscript. Computational cost is measured on an RTX A6000 with batch size 1, FP32 precision, and 128×128 spatial resolution. We perform 10 warm-up runs followed by 100 timed runs, measuring network forward-pass time only and excluding preprocessing and data transfer. Throughput is reported as processed frames per second.
FaceIVL participants are aged 18–28 years, with skin types predominantly Fitzpatrick III–IV. Recording quality is checked for consecutive zeros or abnormally large values in the fingertip PPG signal, and all recordings identified as acquisition failures are excluded. FaceIVL will be released upon paper acceptance through an application process requiring a signed research-use agreement, in accordance with ethics committee requirements.
Dear Reviewer wDkY,
With only approximately an half hour remaining before thediscussion period closes, we hope you can understand our eagerness to hear your thoughts.
We would be truly grateful for any further feedback youmay have. Thank you again for your time and efforts inreviewing our work.
We wish you all the best!
Response to Reviewer wDkY (Part 1/3)
We thank the reviewer for recognizing our motivation, design, experiments, and FaceIVL contribution. Relevant revisions are included in the revised manuscript.
1. Physiological–Illumination Feature Disentanglement (W1)
We compute the local RGB–NIR discrepancy on 8×8 regional feature maps after landmark-based coarse alignment, without requiring strict pixel-level registration of the original images. This discrepancy is processed to obtain the local illumination-related representation, combined with the global disturbance reference, and projected to produce the projected disturbance feature. Thus, modality-specific pulse responses or appearance differences are not directly treated as noise. The orthogonality loss reduces the similarity between the refined RGB representation and the projected disturbance feature, while waveform supervision encourages accurate recovery of the ground-truth PPG and preservation of physiological information.
With the main network frozen, we fit separate linear probes on the RGB facial features, projected disturbance feature, and refined RGB representation for two tasks: (1) PPG recovery: reconstruct PPG from each feature and compute the Pearson correlation between the reconstructed waveform and synchronized fingertip PPG. (2) Illumination recovery: apply a known brightness modulation only to RGB, reconstruct this modulation from the feature difference between perturbed and unperturbed inputs, and compute the Pearson correlation between the reconstructed and applied modulation curves. These results show that PFD improves PPG recovery and reduces illumination recovery in FRGB. The projected disturbance feature P(Fnoise) shows higher illumination recovery and lower PPG recovery than the refined RGB representation.
2. Training and Preprocessing Details (W2)
Loss functions. In addition to the negative Pearson loss, we use three auxiliary losses:
Lα=max(0,0.1−α)2+max(0,α−0.9)2.
The local illumination consistency loss is defined as
Llocal=SmoothL1(FlocalIllum,sg(ΔF)).
For orthogonality, features are spatially averaged within the facial mask before computing squared cosine similarity,
Lorth=2T1∑t=1T[(∣FtRGB∣∣P(Fnoise)t∣FtRGB⋅P(Fnoise)t)2+(∣FtNIR∣∣(βDgNIR)t∣FtNIR⋅(βDgNIR)t)2].
These losses regularize the mixing coefficient, local disturbance consistency, and feature orthogonality, respectively. Here, sg denotes stop-gradient.
Learnable coefficients. α and β are sigmoid-parameterized and initialized to 0.5, ensuring values in (0,1) and convex global–local mixing.
Masks and global disturbance references. Facial masks are generated from the convex hull of MediaPipe facial contour landmarks, undergo the same geometric transformation as the images, and are resized to the feature resolution before binarization at a threshold of 0.5. Each global disturbance reference is obtained by spatially summing (1−M)⊙F and dividing by the background area, yielding dimensions B×C×T×1×1. Frame-wise statistics over all preprocessed clips show mean RGB/NIR non-facial area proportions of 57.31%/59.44% on MR-NIRP Driving.
We apply masking after encoding to select the spatial source of the reference, without truncating the convolutional receptive fields. Facial and background regions share an encoder within each modality, placing them in a common feature space before masking separates their spatial locations. Hard masking before encoding would alter the observed image structure and introduce artificial zero-valued boundaries.
To account for facial context in background features, the model separates reference-region selection from disturbance estimation. The NIR branch uses the learnable coefficient β to control background-reference subtraction, the RGB branch uses α to balance the global background reference and local illumination-related representation, followed by a learnable projection to produce the projected disturbance feature.
Our response continues in the next comment.