acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Near-Infrared Priors for Illumination-Robust rPPG Estimation

Abstract

Remote photoplethysmography (rPPG) enables contact-free physiological measurement from facial videos, but remains vulnerable to illumination variations that can easily overwhelm subtle pulse-related appearance changes. RGB-NIR sensing provides a promising solution by introducing near-infrared observations that are relatively more stable than visible-light imaging. However, raw RGB-NIR features still contain illumination-related disturbance, and direct fusion may propagate it into the final prediction. To address this problem, we propose the Near-Infrared Prior Network (NIRP-Net), a prior-guided framework for illumination-robust RGB-NIR rPPG estimation. NIRP-Net first performs prior-guided feature disentanglement to derive disturbance-suppressed RGB and NIR physiological representations by exploiting the 940-nm low-irradiance and physiology-irrelevant background priors. It then introduces gated cross-attention aggregation to selectively retrieve and aggregate complementary NIR cues for temporal prediction. We further collect FaceIVL, a new RGB-NIR facial rPPG dataset under controlled illumination variations. Experiments on public datasets and FaceIVL demonstrate state-of-the-art accuracy and greater computational efficiency than existing RGB-NIR approaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
rPPG, Heart rate estimation, Near-infrared dataset

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wDkYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NIRP-Net for remote photoplethysmography (rPPG) estimation from synchronized RGB and near-infrared facial videos under challenging illumination. The method uses two priors: the relative stability of actively illuminated 940-nm NIR observations and the absence of target physiological signals in background regions. Its prior-guided feature disentanglement module subtracts background-derived disturbance from NIR features, then combines RGB–NIR feature differences with RGB background features to suppress disturbance in the RGB representation. A gated local cross-attention module subsequently incorporates complementary NIR information, followed by a temporal Transformer and waveform regression head.

Critical strengths
  1. Relevant problem and coherent design. Illumination interference is a meaningful obstacle to reliable rPPG. Refining each modality before fusion is well motivated, and the architecture connects background-based disturbance estimation with adaptive use of NIR information. The manuscript appropriately describes NIR as relatively stable rather than illumination invariant.

  2. Promising reported accuracy. In Table 2, NIRP-Net reduces driving-dataset MAE from 2.33 bpm for the strongest listed multimodal baseline to 1.73 bpm, approximately a 26% reduction. It also reports the lowest MAE and RMSE on the indoor and FaceIVL datasets. These consistent improvements are encouraging.

  3. Useful experimental breadth. The evaluation includes three datasets, a cross-dataset experiment, and ablations of feature refinement, fusion, and temporal modeling. Table 4 is particularly informative: RGB refinement alone does not improve the GCA baseline, whereas combining NIR refinement, RGB refinement, and GCA produces the strongest result. This supports an interaction between the components.

  4. Potentially valuable dataset contribution. FaceIVL provides synchronized RGB–NIR observations, contact PPG references, and a subject-independent evaluation split. Acquisition through a shared optical path is useful for studying spatially corresponding multimodal features. Ethics approval and participant consent are reported.

Critical weaknesses
  1. The claimed physiological–illumination disentanglement is not directly demonstrated. Equations (1)–(4) implement learned feature subtraction, but the manuscript does not establish that the removed components specifically represent illumination or that the retained features preserve physiological information. RGB–NIR differences can also contain modality-specific pulse responses, appearance differences, and alignment errors. Improved heart-rate accuracy demonstrates the usefulness of the architecture, but does not by itself validate the proposed interpretation. Controlled illumination perturbations, measurements of pulse and illumination content before and after refinement, or targeted representation analyses would strengthen the central claim.

  2. Important training and preprocessing details are missing. Appendix B introduces auxiliary regularization terms for “global-local disturbance balancing” and “local illumination consistency,” but provides neither their equations nor their weights. These terms may materially affect the claimed separation mechanism. The manuscript also leaves unclear how facial masks are generated, how background pooling is normalized, and how much background remains after facial cropping. Because masking occurs after the convolutional encoders, background features may also contain facial information through their receptive fields. The initialization and constraints of the learnable coefficients α and β should be specified; without a constraint, Equation (3) need not behave as a convex balance between disturbance references.

  3. The evaluation does not sufficiently isolate illumination robustness. FaceIVL results are aggregated across normal and low-light recordings. Separate results for each condition are necessary to show where the improvement arises. The acquisition description also does not establish whether illumination changes dynamically within recordings or primarily differs between two recording conditions. Illumination intensity, spectrum, and temporal variation are not quantified. Furthermore, the solar-spectrum motivation does not directly explain robustness under indoor lighting. Tests varying ambient illumination and active NIR strength would better establish the operating conditions of the proposed prior.

  4. Key comparisons are absent, limiting the novelty and complementarity claims. The paper identifies Shao and Hu’s NIR-prior method as closely related, but does not include it in the quantitative comparisons. The “Diff” ablation is insufficiently specified to determine whether it faithfully represents that approach. Matched RGB-only and NIR-only variants of the proposed backbone are also missing. An NIR-only baseline is especially important under low visible light: it would establish how much improvement comes from cross-modal complementarity rather than access to a stronger sensing modality. These omissions make the advantage over prior-guided alternatives difficult to assess.

  5. Split definitions and statistical uncertainty require clarification. FaceIVL explicitly uses a subject-independent split, but the manuscript does not state whether the driving dataset’s 7:3 split or the indoor dataset’s eight folds separate subjects. Subject and recording identities, clip overlap, and the splitting procedure should be documented to rule out leakage. Appendix B reports a single random seed, with no variation across runs or uncertainty estimates for the performance differences. This matters particularly for the smaller improvements on Indoor and FaceIVL. Cross-dataset evaluation is also limited to Driving → Indoor; transfers involving FaceIVL would provide stronger evidence of generalization.

  6. There are reporting inconsistencies and incomplete dataset documentation. Table 7 appears to exchange the IENet and FusionPhys MAE/RMSE values relative to Table 2. Its computational measurements use 160-frame clips, whereas training and architecture details specify 128-frame clips; this difference needs explanation. Throughput measurements should specify batch size, precision, and whether preprocessing is included. For FaceIVL, age and skin-tone distributions, failure or exclusion criteria, and a data-release plan are not provided, limiting assessment of its coverage and reproducibility.

Questions for authors

Please see the weakness part.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wDkY (Part 1/3)

We thank the reviewer for recognizing our motivation, design, experiments, and FaceIVL contribution. Relevant revisions are included in the revised manuscript.

1. Physiological–Illumination Feature Disentanglement (W1)

We compute the local RGB–NIR discrepancy on 8×8 regional feature maps after landmark-based coarse alignment, without requiring strict pixel-level registration of the original images. This discrepancy is processed to obtain the local illumination-related representation, combined with the global disturbance reference, and projected to produce the projected disturbance feature. Thus, modality-specific pulse responses or appearance differences are not directly treated as noise. The orthogonality loss reduces the similarity between the refined RGB representation and the projected disturbance feature, while waveform supervision encourages accurate recovery of the ground-truth PPG and preservation of physiological information.

With the main network frozen, we fit separate linear probes on the RGB facial features, projected disturbance feature, and refined RGB representation for two tasks: (1) PPG recovery: reconstruct PPG from each feature and compute the Pearson correlation between the reconstructed waveform and synchronized fingertip PPG. (2) Illumination recovery: apply a known brightness modulation only to RGB, reconstruct this modulation from the feature difference between perturbed and unperturbed inputs, and compute the Pearson correlation between the reconstructed and applied modulation curves. These results show that PFD improves PPG recovery and reduces illumination recovery in F~RGB\widetilde{\text{F}}^{\mathrm{RGB}}. The projected disturbance feature P(Fnoise)\mathcal{P}(\text{F}_{\mathrm{noise}}) shows higher illumination recovery and lower PPG recovery than the refined RGB representation.

FeaturePPG RecoveryIllumination Recovery
FfaceRGB\text{F}_{\mathrm{face}}^{\mathrm{RGB}}0.76930.8684
P(Fnoise)\mathcal{P}(\text{F}_{\mathrm{noise}})0.30430.9391
F~RGB\widetilde{\text{F}}^{\mathrm{RGB}}0.85800.3862

2. Training and Preprocessing Details (W2)

Loss functions. In addition to the negative Pearson loss, we use three auxiliary losses:

Lα=max⁡(0,0.1−α)2+max⁡(0,α−0.9)2.\mathcal{L}_{\alpha} = \max(0,0.1-\alpha)^2 + \max(0,\alpha-0.9)^2.

The local illumination consistency loss is defined as

Llocal=SmoothL1(FlocalIllum,sg(ΔF)).\mathcal{L}_{\mathrm{local}} = \mathrm{SmoothL1}(\text{F}_{\mathrm{local}}^{\mathrm{Illum}},\mathrm{sg}(\Delta \text{F})).

For orthogonality, features are spatially averaged within the facial mask before computing squared cosine similarity,

Lorth=12T∑t=1T[(F~tRGB⋅P(Fnoise)t∣F~tRGB∣∣P(Fnoise)t∣)2+(F~tNIR⋅(βDgNIR)t∣F~tNIR∣∣(βDgNIR)t∣)2].\mathcal{L}_{\mathrm{orth}} = \frac{1}{2T}\sum_{t=1}^{T}\left[\left(\frac{\widetilde{\text{F}}_t^{\mathrm{RGB}}\cdot\mathcal{P}(\text{F}_{\mathrm{noise}})_t}{\\|\widetilde{\text{F}}_t^{\mathrm{RGB}}\\|\\|\mathcal{P}(\text{F}_{\mathrm{noise}})_t\\|}\right)^2 + \left(\frac{\widetilde{\text{F}}_t^{\mathrm{NIR}}\cdot(\beta \text{D}_g^{\mathrm{NIR}})_t}{\\|\widetilde{\text{F}}_t^{\mathrm{NIR}}\\|\\|(\beta \text{D}_g^{\mathrm{NIR}})_t\\|}\right)^2\right].

These losses regularize the mixing coefficient, local disturbance consistency, and feature orthogonality, respectively. Here, sg denotes stop-gradient.

Learnable coefficients. α\alpha and β\beta are sigmoid-parameterized and initialized to 0.5, ensuring values in (0,1) and convex global–local mixing.

Masks and global disturbance references. Facial masks are generated from the convex hull of MediaPipe facial contour landmarks, undergo the same geometric transformation as the images, and are resized to the feature resolution before binarization at a threshold of 0.5. Each global disturbance reference is obtained by spatially summing (1−M)⊙F(1-\text{M})\odot \text{F} and dividing by the background area, yielding dimensions B×C×T×1×1B\times C\times T\times1\times1. Frame-wise statistics over all preprocessed clips show mean RGB/NIR non-facial area proportions of 57.31%/59.44% on MR-NIRP Driving.

We apply masking after encoding to select the spatial source of the reference, without truncating the convolutional receptive fields. Facial and background regions share an encoder within each modality, placing them in a common feature space before masking separates their spatial locations. Hard masking before encoding would alter the observed image structure and introduce artificial zero-valued boundaries.

To account for facial context in background features, the model separates reference-region selection from disturbance estimation. The NIR branch uses the learnable coefficient β\beta to control background-reference subtraction, the RGB branch uses α\alpha to balance the global background reference and local illumination-related representation, followed by a learnable projection to produce the projected disturbance feature.

Our response continues in the next comment.

AuthorsOpenReview ↗

Response to Reviewer wDkY (Part 2/3)

3. Illumination Conditions and Robustness Evaluation (W3)

Evaluation by lighting condition. We separately evaluate normal-light and low-light recordings. NIRP-Net achieves the lowest MAE and RMSE under both conditions. Its advantage therefore extends to side-lit conditions with facial shadows and uneven illumination, beyond normal-light recordings.

MethodNormal-Light MAE ↓ (bpm)Normal-Light RMSE ↓ (bpm)Low-Light MAE ↓ (bpm)Low-Light RMSE ↓ (bpm)
PhysNet0.680.971.684.75
RhythmMamba1.042.741.513.20
IENet0.651.121.603.13
FusionPhys0.611.241.643.56
NIRP-Net0.560.711.333.12

Illumination conditions and temporal variation. Normal lighting uses indoor ceiling lights, with approximately 200 lx at the face. Low lighting uses a visible-light source positioned approximately 45° to the side, providing about 50 lx and creating facial shadows and uneven illumination. The conditions are recorded separately, without deliberately introducing substantial illumination changes within individual recordings. Small head and facial movements are allowed. FaceIVL therefore evaluates performance across lighting conditions and under spatially nonuniform illumination, while MR-NIRP Driving provides complementary evaluation under complex illumination variations during real-world driving.

Indoor NIR prior. Solar-spectrum attenuation around 940 nm motivates the outdoor setting. Indoors, active 940-nm illumination provides the auxiliary NIR observation. Paired FaceIVL statistics show that, from normal to low lighting, RGB pulse-band SNR decreases by 1.96 dB on average, whereas NIR decreases by only 0.17 dB, supporting its relative stability across visible-light conditions. We use a fixed NIR illumination intensity during acquisition. Experiments with different NIR illumination intensities are planned for future work.

4. Comparison with Related Work and Cross-Modal Complementarity (W4)

NIRP-Net achieves lower estimation errors, demonstrating its advantage over both the simplified Diff baseline and our implementation of this NIR-prior method. When preparing the original manuscript, we could not obtain official code for Shao and Hu’s method. We therefore constructed a Diff baseline within our backbone for a controlled comparison, without presenting an uncertain implementation as a full reproduction. Following a similar principle, Diff separately projects RGB and NIR facial features, subtracts them, and applies temporal modeling and waveform regression to the resulting difference features. Following the reviewer’s suggestion, we additionally implemented Shao and Hu’s method based on the published details and evaluated it on MR-NIRP Driving using the same protocol.

MethodMAE ↓ (bpm)RMSE ↓ (bpm)MAPE ↓ (%)ρ ↑
Shao and Hu (our implementation)2.606.273.960.64
NIRP-Net1.734.172.510.83

The full model outperforms both single-modal variants under normal and low lighting, including NIR-only under low lighting, demonstrating the benefit of RGB–NIR complementarity.

Input SettingNormal-Light MAE ↓ (bpm)Normal-Light RMSE ↓ (bpm)Low-Light MAE ↓ (bpm)Low-Light RMSE ↓ (bpm)
RGB-only0.6751.1873.6908.834
NIR-only0.8101.3592.2054.593
RGB+NIR (Full model)0.5630.7061.3283.120

These variants use matched encoder architectures, temporal modeling modules, and regression heads, with consistent data splits, training protocols, and checkpoint selection rules. Each single-modal variant removes processing that depends on the other modality. The full model outperforms both single-modal variants under normal and low lighting, including NIR-only under low lighting. These results demonstrate that the improvement comes not merely from access to NIR observations, but also from exploiting RGB–NIR complementarity.

Our response continues in the next comment.

AuthorsOpenReview ↗

Response to Reviewer wDkY (Part 3/3)

5. Data Splits, Statistical Stability, and Cross-Dataset Evaluation (W5)

Subject-independent splits. All datasets use subject-independent splits, with no overlap in subject identities, recordings, or clips between training and testing. For MR-NIRP Driving, we group all recording identifiers belonging to the same subject before applying an approximately 7:3 training/test split, rounding down the number of training subjects. MR-NIRP Indoor uses eight-fold leave-one-subject-out evaluation, with seven subjects for training and one for testing in each fold. FaceIVL uses a gender-stratified, subject-independent split. All recordings and clips from each subject remain within the same partition, preventing leakage through shared identities or adjacent clips from the same recording.

NIRP-Net achieves the lowest mean MAE and RMSE on both Indoor and FaceIVL across three random seeds. We conduct repeated experiments with three random seeds on Indoor and FaceIVL, keeping data splits, method-specific hyperparameters, and checkpoint selection rules fixed. The results below show that NIRP-Net’s advantage is not limited to a single random initialization. Values are reported as mean ± standard deviation across three seeds.

MethodIndoor MAE ↓ (bpm)Indoor RMSE ↓ (bpm)FaceIVL MAE ↓ (bpm)FaceIVL RMSE ↓ (bpm)
RhythmMamba0.748 ± 0.1101.125 ± 0.1791.286 ± 0.2482.824 ± 0.621
FusionPhys0.540 ± 0.0420.763 ± 0.0281.181 ± 0.0482.681 ± 0.019
IENet0.795 ± 0.0941.180 ± 0.1251.219 ± 0.1342.648 ± 0.575
NIRP-Net0.495 ± 0.0450.690 ± 0.0580.979 ± 0.0492.367 ± 0.122

NIRP-Net achieves the lowest overall MAE and RMSE on Driving → FaceIVL. This extends the original Driving → Indoor evaluation. The normal- and low-light breakdown is also provided in our response to Reviewer gJu1.

MethodMAE ↓ (bpm)RMSE ↓ (bpm)MAPE ↓ (%)ρ ↑
RhythmFormer4.8611.674.480.72
RhythmMamba5.7612.225.340.70
IENet4.5010.674.160.76
FusionPhys3.8911.023.470.75
NIRP-Net3.6210.173.300.78

6. Reporting Consistency and Dataset Documentation (W6)

The MAE/RMSE entries for IENet and FusionPhys were swapped in Table 7 (Table 9 in the revised manuscript). “160 frames” was a typographical error, the actual clip length for computational-cost evaluation is 128 frames. Both errors have been corrected in the revised manuscript. Computational cost is measured on an RTX A6000 with batch size 1, FP32 precision, and 128×128 spatial resolution. We perform 10 warm-up runs followed by 100 timed runs, measuring network forward-pass time only and excluding preprocessing and data transfer. Throughput is reported as processed frames per second.

FaceIVL participants are aged 18–28 years, with skin types predominantly Fitzpatrick III–IV. Recording quality is checked for consecutive zeros or abnormally large values in the fingertip PPG signal, and all recordings identified as acquisition failures are excluded. FaceIVL will be released upon paper acceptance through an application process requiring a signed research-use agreement, in accordance with ethics committee requirements.

Dear Reviewer wDkY,

With only approximately an half hour remaining before thediscussion period closes, we hope you can understand our eagerness to hear your thoughts.

We would be truly grateful for any further feedback youmay have. Thank you again for your time and efforts inreviewing our work.

We wish you all the best!

Reviewer_gJu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a physical priori guided multimodal rPPG measurement framework based on RGB-NIR. The main contributions of the method can be divided into two points: A priori guided feature decoupling, which uses the information of NIR in 940nm band to supplement the deficiency of RGB mode in this band, and uses the background noise feature to separate the physiological signal feature from the facial region signal; Gated cross modal attention aggregation, using RGB modal information to adjust NIR information, and aggregating to RGB modal information through gating selection. In addition, we also collected a dataset FaceIVL in the laboratory environment based on RGB-NIR. Finally, experiments are carried out on three datasets, MR-NIRP Driving, MR-NIRP Indoor, and FaceIVL. Compared with existing methods using RGB single-mode and RGB-NIR dual-mode, the performance of these methods is improved.

Critical strengths
  1. This work mainly focuses on the impact of lighting changes on rPPG measurement; specifically, the different characteristics of RGB and NIR modal data are used to complement each other, which improves the measurement results under complex lighting conditions.
  2. The framework explicitly separates the background region information in each modal information and uses it to extract the required physiological signals from the facial information. At the same time, the attention mechanism with gating is used to integrate the complementary physiological information between NIR and RGB modes.
  3. The experiments in this work tested the proposed method on three RGB-NIR datasets and carried out a comparative experiment across datasets. The performance of this method has been significantly improved. In addition, the ablation experiments further proved the effectiveness of the proposed components.
Critical weaknesses
  1. Many experimental details are missing. It is unclear whether the methods in Table 2 use RGB, NIR, or RGB-NIR inputs. Moreover, the training and testing splits are inconsistent across the three datasets, and, most importantly, none of the experiments appears to use a validation set. This is methodologically questionable for deep learning experiments and makes it difficult to determine whether the comparisons are fair.
  2. The cross-dataset evaluation is too limited. The paper evaluates only a very small number of cross-dataset settings, which is insufficient to support the claimed generalization ability.
  3. The terminology of "feature disentanglement" is not well justified. Feature disentanglement typically refers to separately modeling different factors, such as background and facial features, often with explicit constraints such as orthogonality or independence objectives. In contrast, the proposed method appears to perform disturbance suppression using background references rather than strict feature disentanglement in the conventional sense.
  4. The background prior may be inaccurate in realistic scenarios. The proposed method assumes that non-facial background regions mainly contain illumination and sensor-related disturbances. However, real backgrounds may exhibit different reflectance properties, spatially varying illumination, shadows, motion, occlusions, or unrelated moving objects. Therefore, using a global background feature to represent the disturbances affecting facial skin may be inaccurate.
  5. The design of GCA is not sufficiently motivated. The RGB-guided cross-attention module always uses RGB features as the queries and NIR features as the keys and values. This design implicitly assumes that RGB is the appropriate aggregation anchor. However, the paper also states that RGB is more vulnerable to illumination variations, which appears inconsistent with using RGB as the primary guide.
Questions for authors
  1. Please clarify whether each method in Table 2 uses RGB, NIR, or RGB-NIR inputs. Why are validation sets not used, and how can the authors ensure that the comparisons are fair?
  2. Can the authors provide more comprehensive cross-dataset evaluations, including additional source-target combinations, FaceIVL-based transfer experiments, and diverse illumination conditions?
  3. How strongly are background features correlated with the illumination disturbances affecting facial regions? Please evaluate the sensitivity of the method to background reflectance, spatially varying illumination, shadows, motion, occlusions, and unrelated moving objects.
  4. Why are RGB features always used as the queries in GCA when RGB is considered more sensitive to illumination changes? How does the method compare with NIR-guided, symmetric, or reliability-adaptive cross-attention designs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer gJu1 (Part 1/2)

We thank the reviewer for recognizing our motivation to address illumination interference, RGB–NIR complementarity modeling, gated aggregation design, and the multi-dataset evaluation and component ablations. The corresponding changes have been incorporated into the revised manuscript. We respond point by point below.

1. Input Modalities, Data Splits, and Comparison Fairness (W1 & Q1)

In Table 2, the first ten methods use RGB inputs, while IENet, FusionPhys, and NIRP-Net use synchronized RGB–NIR inputs. We will explicitly identify each method’s input modalities in the revised manuscript.

MR-NIRP Driving and Indoor contain only 18 and 8 independent participants, respectively. Reserving additional participants for validation would further reduce the already limited training population. Therefore, the original experiments did not use a separate validation set, preserving sufficient training data. Indoor uses eight-fold LOSO evaluation, with seven participants for training and one for testing in each fold. All splits are subject-independent, with no participant shared between training and testing.

The original FaceIVL experiments use a subject-independent 6:4 training/test split. To further evaluate performance with an independent validation set, we additionally conduct experiments using a gender-stratified 5:1:4 training/validation/test split, containing 30, 6, and 24 participants, respectively. The results are shown below. NIRP-Net achieves the best results across all reported metrics, demonstrating that its performance advantage is maintained when an independent validation set is introduced.

MethodInputMAE ↓ (bpm)RMSE ↓ (bpm)MAPE ↓ (%)ρ ↑
POSRGB7.3514.966.660.60
CHROMRGB10.1118.879.190.42
OMITRGB18.3026.6417.080.12
PhysNetRGB1.283.561.170.98
PhysFormerRGB2.909.032.570.82
RhythmFormerRGB4.689.804.300.78
RhythmMambaRGB1.233.031.160.98
IENetRGB–NIR1.403.311.310.98
FusionPhysRGB–NIR1.233.151.220.98
NIRP-NetRGB–NIR1.112.411.100.99

2. Cross-Dataset Evaluation (W2 & Q2)

NIRP-Net achieves the lowest low-light MAE and RMSE on Driving → FaceIVL, demonstrating transfer to low-light conditions in an unseen dataset. We extend the original Driving → Indoor evaluation with Driving → FaceIVL and report normal- and low-light results separately. All methods use the same source-data split and evaluation protocol.

Normal LightLow Light
MethodMAE ↓ (bpm)RMSE ↓ (bpm)MAPE ↓ (%)ρ ↑MAE ↓ (bpm)RMSE ↓ (bpm)MAPE ↓ (%)ρ ↑
RhythmFormer4.6811.844.260.715.0511.514.690.73
RhythmMamba4.6510.104.310.796.8714.036.380.61
IENet4.069.283.750.834.9311.894.570.70
FusionPhys3.7410.183.440.794.0411.803.510.70
NIRP-Net3.609.883.310.803.6410.453.300.76

3. Feature Disentanglement (W3)

PFD combines prior-guided disturbance estimation and explicit feature subtraction, reinforced by orthogonality losses in both RGB and NIR branches. Conventional feature disentanglement aims to represent underlying factors separately, often using orthogonality or independence constraints. Our datasets provide reference PPG waveforms but no separately labeled physiological and illumination components. We therefore use physical priors to guide their separation under waveform supervision.

PFD uses the NIR background reference to refine NIR features, then combines the refined NIR reference with RGB background information to estimate RGB disturbance. Subtracting these estimated components produces the refined modality-specific representations. Orthogonality losses provide auxiliary constraints in both RGB and NIR branches by minimizing the squared cosine similarity between spatially pooled refined features and the estimated subtracted components. Thus, explicit subtraction implements the separation, orthogonality regularization reinforces the distinction between retained and removed representations, and waveform supervision preserves their relevance to physiological prediction.

Our response continues in the next comment, where we address Points 4–5 on the background reference and attention direction in GCA.

AuthorsOpenReview ↗

Response to Reviewer gJu1 (Part 2/2)

4. Effectiveness of the Background Reference (W4 & Q3)

The background reference need not capture all facial disturbances: it targets shared illumination variations, while the local branch estimates additional facial disturbances. For each MR-NIRP Driving video, we compute the Pearson correlation between the mean intensity curve of the facial region and that of the background region after low-pass filtering both curves. The median correlations are 0.855 for RGB and 0.431 for NIR, showing that facial and background brightness changes are correlated but not identical.

The local branch uses the cross-modal discrepancy to estimate facial disturbances not fully captured by the background reference,

ΔF=AR(FfaceRGB)−AN(F~NIR).\Delta \text{F} = \mathcal{A}_R(\text{F}_{\mathrm{face}}^{\mathrm{RGB}}) - \mathcal{A}_N(\widetilde{\text{F}}^{\mathrm{NIR}}).

The global and local estimates are combined to refine the RGB features,

F~RGB=FfaceRGB−P(αDgRGB+(1−α)FlocalIllum).\widetilde{\text{F}}^{\mathrm{RGB}} = \text{F}_{\mathrm{face}}^{\mathrm{RGB}} - \mathcal{P}(\alpha \text{D}_g^{\mathrm{RGB}} + (1-\alpha)\text{F}_{\mathrm{local}}^{\mathrm{Illum}}).

With model weights fixed, we modify only the background through static spatially varying brightness gains, low-frequency brightness modulation, or a moving occlusion patch. The results support the effectiveness of combining an auxiliary background reference with local disturbance estimation under the tested background mismatches.

SettingsMAE ↓ (bpm)RMSE ↓ (bpm)
Original input1.734.17
Static spatially nonuniform brightness changes, up to 20%1.714.13
Low-frequency temporal background brightness changes, 20% amplitude2.044.80
Moving background occlusion patch2.355.04

5. Attention Direction in GCA (W5 & Q4)

Illumination stability and absolute pulse SNR are distinct properties. In FaceIVL, the RGB has a higher pulse-band SNR, supporting RGB as the aggregation anchor. NIR has a lower absolute SNR but is less affected by visible-light conditions: from normal to low lighting, RGB pulse-band SNR decreases by 1.96 dB, whereas NIR decreases by only 0.17 dB. Accordingly, GCA uses refined RGB features as queries to retrieve complementary information from NIR, with gating controlling its contribution.

Our ablation shows that using refined RGB features as queries and NIR features as keys and values achieves lower MAE and RMSE under both normal and low lighting than reverse-direction aggregation and bidirectional cross-attention, supporting the attention direction adopted in GCA.

Aggregation DirectionNormal-Light MAE ↓ (bpm)Normal-Light RMSE ↓ (bpm)Low-Light MAE ↓ (bpm)Low-Light RMSE ↓ (bpm)
NIR-query, RGB-key/value0.6531.1721.4183.283
Bidirectional cross-attention0.6080.8751.4403.187
RGB-query, NIR-key/value (Ours)0.5630.7061.3283.120

Dear Reviewer gJu1,

With only approximately an half hour remaining before thediscussion period closes, we hope you can understand our eagerness to hear your thoughts.

We would be truly grateful for any further feedback youmay have. Thank you again for your time and efforts inreviewing our work.

We wish you all the best!

Reviewer_3t5KOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method for fusing information from synchronous RGB and NIR videos for rPPG estimation. The proposed method, NIRP-Net, uses the 940 nm NIR signal (which is relatively free of noise due to illumination variation) as a reference signal to remove noise from the RGB video using prior-guided disentanglement fusion. It also uses variations in the background (non-face) part of the image frames to help remove non-pulsatile variations from the face signals in both NIR and RGB. The paper also introduces FaceIVL, a new RGB-NIR facial rPPG dataset with two different visible lighting conditions, thought it is not clear from the submitted version of the paper whether the authors will publicly release the new dataset.

Critical strengths
  1. The prior-guided feature disentanglement is well thought out and is explained well in Section 3.3. The idea of using the NIR features as a relatively stable reference signal is a good one, and the way it is combined with the RGB features is clever.

  2. The gated cross-attention is also a good idea, as it clearly improves the results.

  3. The paragraph describing the conclusions from the results of the overall-pipeline ablation studies (lines 419-429) is particularly well done, both concise and informative.

  4. The authors included their training and testing code, including a checkpoint of the model, in the supplementary material, and they say they plan to release their code after the paper is accepted.

Critical weaknesses
  1. Regarding the FaceIVL dataset:

a. The paper does not say whether the authors plan to make their dataset public. Given that the authors collected the new dataset because they identified a dearth of public RGB-NIR datasets, releasing FaceIVL publicly would be a good contribution to the research community.

b. The experimental conditions are not explained well enough in either the paper or the appendices. In particular, it is not clear how the visible lighting intensity is varied over time. Is it varied within each video, or does each video have constant lighting? From Table 1, I am guessing that each subject has two 1-minute videos, of which one is entirely in normal light and one is entirely in low light. However, that is not clear from the text. Also, were there any other sources of the visible light in the room? Did any of them vary over time during the course of a single video?

c. The new dataset seems to be relatively easy. It would be better if in addition to the conditions provided, it also contained some more difficult conditions, such as changing the visible lighting over time during the course of a video, and possibly some other motion conditions.

  1. The proposed algorithm's design treats NIR videos as a relatively stable reference (lines 184-187) compared to the RGB videos, and this is certainly the case for the driving data from the MR-NIRP Car dataset. However, it is not clear whether the same holds true for the FaceIVL dataset. If each video in the FaceIVL dataset has constant visible lighting throughout the video, then each RGB video should be just as stable a reference as each NIR video. In that case, what would be the benefit of treating the NIR video (which has worse SNR than the RGB video due to the reduced sensitivity of the camera sensor in NIR) as the stable reference?

  2. It is not clear how the clips from each dataset are separated into training and testing sets. In particular, it is not clear whether the subjects (people) in the training set are disjoint from the subjects in the test set. Line 334 says that FaceIVL is separated using a "subject independent" split, but the meaning of that is not clear. In the paper that introduced it, the MR-NIRP Car driving dataset was split into training and testing sets using leave-one-subject-out (LOSO) cross-validation. This ensured that the learned model would have never seen any of the test subjects before test time. This is the most difficult case, and the most realistic for most applications. If the training and testing sets come from the same videos, then it is possible that a test clip could be immediately adjacent to one of the training clips--this is clearly an easier case than would occur in any applications.

  3. The partitioning of both the RGB and NIR feature maps into w x w local windows is described in lines 272-287, ending by explaining that it "preserves the spatial correspondence between synchronized RGB and NIR facial regions...." This makes sense for the FaceIVL dataset, because the camera aligns the RGB and NIR images by construction. However, the RGB and NIR cameras in MR-NIRP Car and MR-NIRP Indoor are not so well aligned, because the RGB and NIR cameras were not collocated. For these datasets, were the camera images aligned in any way? If not, how did that affect the performance of the algorithm?

  4. For the disturbance references:

a. What type of global pooling (line 224) was used? For example, was it global average pooling, global max pooling, or global pooling with learned linear weights?

b. What are the dimensions of the disturbance references DgNIRD_g^{NIR} and DgRGBD_g^{RGB}? Is the former 1 x T?

c. A suggestion: In realistic situations, a single 1 x T time series might not be enough to explain the lighting disturbances. For example, one time series of lighting change over time might affect one part of the scene, while a different time series of lighting change over time might affect a different part of the scene. To address this, it might be possible to use more than one disturbance time series to represent multiple lighting changes over time, similar to the noise subspace in (Nowara et al., 2020).

  1. The size of each clip (each sample window) is not clear. Line 463 says that the clips are 160 frames long, but line 641 says the clips are 128-frames long. In addition to clarifying the actual clip duration in frames, it should also be expressed in seconds.

  2. The unit of the errors in tables 2-6 is presumably bpm, but it is not clear from the tables (it should be stated explicitly).

  3. The paper should point out that the Pearson correlation is the only error measure that measures the accuracy of the entire waveform, rather than just the spectral peak (heart rate). The Pearson correlation error measure should also be explained using an equation (it can be in the Appendices).

  4. What is the meaning of the two minus signs in Figure 1 (b) ? This should be explained.

MINOR ERRORS:

A. In Table 2, in the RhythmMamba row, the 0.99 in the RMSE column should be black and not boldface, but the 0.99 in the correlation column should be red and boldface.

B. In Tables 2, 3, and 7, the meaning of the horizontal rules (horizontal lines) in the middle of the tables is not clear. Do they separate one type of algorithm from another type? The categorization of algorithms by the horizontal rules should be stated explicitly.

C. The first time the "MR-NIRP Driving" dataset is referred to, the paper should explain that this refers to the Driving condition of the MR-NIRP Car dataset. The official name of the entire dataset is the "MR-NIRP Car" dataset, but this paper is referring to the part of the dataset it used as "MR-NIRP Driving", presumably to indicate that the paper used the videos from only the Driving condition (and not the Garage condition) of the MR-NIRP Car dataset. "MR-NIRP Driving" is a reasonable way to refer to the Driving (not Garage) videos of the MR-NIRP Car dataset, as long as this is explained somewhere in the paper.

Questions for authors

Please respond to the Weaknesses identified above, particularly Weaknesses 1, 2, 3, and 4.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3t5K (Part 1/2)

We thank the reviewer for recognizing our prior-guided feature processing, gated cross-attention, ablation analysis, and code reproducibility. The corresponding changes have been incorporated into the revised manuscript. We respond point by point below.

1. FaceIVL Dataset (W1)

(a) Data release. FaceIVL will be made available upon paper acceptance. Data collection has received ethics approval. Since the dataset contains identifiable faces and physiological information, access will require an application and a signed research-use agreement, following our ethics committee’s requirements.

(b) Lighting conditions and acquisition protocol. Each of the 60 participants was recorded in two approximately one-minute synchronized RGB–NIR videos, one under normal lighting and one under low lighting, yielding 120 recordings. Normal lighting used ceiling lights, while low lighting used a visible-light source positioned approximately 45° to the side, creating facial shadows and uneven illumination. The two conditions were recorded separately, without deliberately introducing substantial lighting changes within individual recordings.

(c) Dataset scale and challenges. MR-NIRP Driving and Indoor contain only 18 and 8 independent participants, respectively, limiting the data available for subject-independent evaluation. FaceIVL expands participant coverage to 60 individuals and provides high-quality synchronized data to support further research. Uneven illumination and permitted small head and facial movements introduce additional challenges relative to Indoor, while maintaining a more controlled environment than real-world driving. Dynamic illumination and broader motion conditions are planned for future extensions.

2. The Role of the NIR Reference in FaceIVL (W2)

Relative stability refers to lower sensitivity to visible-light conditions, rather than higher absolute pulse SNR. MR-NIRP Driving supports our design under varying illumination, while FaceIVL further demonstrates its benefit under fixed lighting. Across separately recorded normal and low-light videos, RGB pulse-band SNR decreases by 1.96 dB on average, whereas NIR decreases by only 0.17 dB.

The local cross-modal branch improves RGB feature refinement beyond global background estimation and final fusion, reducing low-light MAE and RMSE while maintaining normal-light performance. We remove the local cross-modal branch in PFD, including the computation of ΔF\Delta F and FlocalIllumF_{\mathrm{local}}^{\mathrm{Illum}}. RGB disturbance estimation then uses only the global RGB background reference, Fnoise=DgRGBF_{\mathrm{noise}} = D_g^{\mathrm{RGB}}. NIR refinement, the subsequent projection and subtraction, and GCA fusion remain unchanged.

SettingNormal MAE ↓ (bpm)Normal RMSE ↓ (bpm)Low-Light MAE ↓ (bpm)Low-Light RMSE ↓ (bpm)
w/o Local Branch0.6080.7561.9585.822
Full model0.5630.7061.3283.120

3. Data Partitioning (W3)

Subject-independent means that training and test participants are strictly disjoint, with all videos and clips from each participant assigned to the same partition. MR-NIRP Driving uses a fixed, approximately 7:3 subject-level split. MR-NIRP Indoor uses eight-fold LOSO evaluation. FaceIVL uses 36 participants for training and 24 for testing, with balanced gender representation. Although Driving uses a fixed split rather than LOSO, its test participants remain entirely unseen during training. Adjacent clips from the same video cannot be split between the training and test sets.

4. RGB–NIR Spatial Correspondence (W4)

We align the RGB–NIR images in MR-NIRP Driving and Indoor during preprocessing. Five facial landmarks from each modality are used to estimate a 2D similarity transform that maps the face to a common template, reducing differences in position, scale, and in-plane rotation. All compared methods use the same preprocessing for a fair comparison.

Our method does not require pixel-level alignment between the RGB and NIR images. Cross-modal subtraction operates on deep feature maps with a spatial resolution of 8×88 \times 8, where each spatial unit represents a region with a receptive field rather than an individual input pixel. Figure 4, newly added to the revised manuscript, shows similar facial-region activation patterns in the RGB and NIR features before cross-modal subtraction, suggesting spatial correspondence at the feature level. GCA then allows each RGB feature to attend to all NIR features within the corresponding local window. Please refer to Section 3.3 and Appendix A of the revised manuscript for further details.

Our response continues in the next comment, where we address Questions 5–9 and the minor corrections.

AuthorsOpenReview ↗

Response to Reviewer 3t5K (Part 2/2)

5. Construction of Background Disturbance References (W5)

(a) Pooling. We use masked average pooling normalized by the valid background area, without max pooling or learned pooling weights. For each channel and time step, background features are spatially summed and divided by the corresponding background area. Only spatial dimensions are aggregated, channel and temporal dimensions are preserved.

(b) Reference dimensions. Both background references satisfy DgRGB,DgNIR∈RB×C×T×1×1.D_g^{\mathrm{RGB}}, D_g^{\mathrm{NIR}} \in \mathbb{R}^{B\times C\times T\times1\times1}. Each sample therefore contains CC channel-specific temporal sequences, not a single 1×T1\times T sequence. These references are spatially broadcast when used for subtraction or fusion.

(c) Spatially nonuniform disturbances. In addition to the multichannel global references, the RGB branch estimates a spatially resolved local disturbance representation from cross-modal feature differences FlocalIllum∈RB×C×T×H′×W′.F_{\mathrm{local}}^{\mathrm{Illum}}\in\mathbb{R}^{B\times C\times T\times H'\times W'}. This representation preserves responses at different spatial locations and is combined with the global background reference through learnable weighting. Thus, the local branch complements the global reference by modeling spatially nonuniform variations.

6. Input Clip Length (W6)

All methods use 128-frame inputs for training and computational-cost evaluation, corresponding to approximately 4.27 seconds at 30 fps. “160 frames” was a typographical error and has been corrected in the revised manuscript.

7. Metric Units (W7)

MAE and RMSE are reported in bpm, MAPE in %, and Pearson correlation is dimensionless. These units are specified in the revised manuscript.

8. Pearson Correlation (W8)

We follow the rPPG-Toolbox evaluation protocol. In the HR evaluation tables, Pearson correlation measures the association between predicted and ground-truth HR values, not between waveforms. Using this established protocol ensures a fair comparison across methods. We will clarify its definition and provide the corresponding equation in the revised manuscript.

9. Minus Signs in Figure 1(b) (W9)

The first minus sign computes the difference between RGB and NIR features for disturbance estimation. The second subtracts the estimated disturbance from RGB features to obtain refined features. We will clarify these operations in the main text and the caption of Figure 1.

10. Minor Corrections

Our experiments use the entire MR-NIRP Car dataset, including both driving and garage recordings. The official dataset website (https://computationalimaging.rice.edu/mr-nirp-dataset/) lists this dataset under “MR-NIRP video dataset (driving),” and we follow this naming convention. We will explicitly clarify this terminology and the included recording conditions in the manuscript. The remaining minor corrections have been incorporated into the revised manuscript.

Reviewer_3t5KOpenReview ↗

Update after author responses to my original review

I appreciate the authors' detailed and thorough replies to my questions. I am glad that they plan to make the dataset public, as this will be a good contribution to the community. Having 60 subjects, with an even M/F balance, is good. Was there any coding of the skin tone of each participants (e.g., using the Monk Skin Tone Scale or the Fitzpatrick scale)?

Regarding using MR-NIRP Car with a single 7:3 subject-wise split, I think that this is not a great idea. There are few enough subjects in MR-NIRP Car that a single 30% portion of them is not representative of the entire dataset. Instead, a cross-validation approach would be best. If Leave-One-Subject-Out cross validation is too expensive, then you could do Leave-Two-Subjects-Out, Leave-Three-Subjects-Out, or even Leave-Half-of-the-Subjects out cross validation. The important thing is to use cross-validation, so that every one of the subjects is incorporated into the test results. With a small dataset like MR-NIRP Car, some subjects are easier than others, and a lucky (or even intentional) selection of a single subset of the test subjects as a test set (without cross validation) could lead to inflated results.

I think the modeling of the background illumination as a single 1x1 time series per channel is limiting, because it assumes a single global illumination variation across all pixels in the face. However, it is clearly better than not modeling the background illumination at all, so it can be thought of a a zeroth-order approximation of the lighting variation.

After reading the author responses, I would ideally like to raise my review from 3 to 3.5. I don't think the paper quite deserves a 4, but it is significantly better than the 3 rating I gave it the first time around. Torn between giving the paper a high 3 or a low 4, I will settle on a borderline 4.

Thank you for your careful review of our work. We appreciate your helpful suggestions and will incorporate additional cross-validation experiments in future work.

Recruitment was limited to participants with Fitzpatrick types III–IV, as comparable numbers from other types were difficult to recruit locally. We conducted questionnaire-based interviews, but could not confidently distinguish between types III and IV from the responses. Self-reported Fitzpatrick typing is less accurate than clinical assessment for types III–VI [1]. We therefore did not include individual Fitzpatrick labels in the dataset to avoid providing potentially inaccurate annotations to other researchers.

Thank you again for your constructive feedback.

Reference:

[1] Eilers S, Bach D Q, Gaber R, et al. Accuracy of self-report in assessing Fitzpatrick skin phototypes I through VI[J]. JAMA dermatology, 2013, 149(11).

Comments on the paper

1 comment

We thank the reviewers for their helpful feedback and recognition of our work.

NIRP-Net combines prior-guided feature disentanglement, adaptive RGB–NIR aggregation, and a new multimodal dataset for illumination-robust rPPG estimation.

  • A prior-guided framework. NIR and background priors guide feature disentanglement to obtain disturbance-suppressed RGB and NIR representations. Gated cross-attention then selectively aggregates complementary physiological information from the refined features.
  • A new RGB–NIR dataset. FaceIVL provides a valuable resource for the community and future research through high-quality synchronized RGB–NIR recordings and contact PPG references. With 60 subjects, it is larger and includes more subjects than most public RGB–NIR datasets.
  • High accuracy and computational efficiency. NIRP-Net achieves state-of-the-art accuracy, with approximately half the MACs of current RGB-NIR methods.

We appreciate the reviewers’ time and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.