acceptodds
Under review as a conference paper at ICLR 2027

4D Point Splatting for mmWave Radar Micro-Doppler Differentiable Rendering

Abstract

Millimeter-wave radar resolves the micrometer displacements of a moving body as micro-Doppler, enabling gait, gesture, activity, pose and vital-sign sensing. Each task relies on a downstream model trained on real data, yet radar datasets lag orders of magnitude behind vision and language. Brute-force collection will not close that gap, and existing synthesis methods (forward-only simulators, learned generators and static-scene inverse renderers) cannot fit a real capture of a moving body, leaving a substantial sim2real gap. We present 4D Point Splatting (4DPS), a physics-based differentiable renderer that fits an explicit moving scene to a real capture. Using the capture's synchronized camera to initialize the room and the moving body, we represent the scene as oriented points carrying material, normal and velocity, and splat each point into range and Doppler bins. Our output is the raw complex return a real radar records, so we can apply the same digital signal processing (DSP) on the rendered data, and obtain any DSP representation to run on and train any SOTA downstream model. We fit 235 sequences on HuPR and 100 on RT-Pose, two datasets with different radars. Our reconstructions set the state of the art, with median correlations of 0.940 and 0.830 on range–azimuth, 0.829 and 0.695 on range–Doppler, and 0.988 and 0.966 on micro-Doppler. Our renders also set the state of the art in downstream sim2real gap. On RT-Pose, a pose model trained on them alone reaches 21.9 cm MPJPE on real data, against 16.7 cm for real training and 127.6 cm for RF-Genesis baseline, and we show that both the rendered- and real-trained models learn nearly the same features. Finally, our custom CUDA kernels fit a sequence in 5.7 minutes on HuPR and 12.9 minutes on RT-Pose on one H100, making dataset-scale inverse rendering of mmWave radar possible for the first time. Together, this work can enable synthetic data generation for any human-centered downstream task and model, and scene editing for data augmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Differentiable rendering, mmWave radar, FMCW radar, inverse rendering, Splatting, Micro-Doppler

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hRH6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

MmWave human-sensing models require large amounts of annotated radar data, but real data are scarce and synthetic data often suffer from a substantial sim-to-real gap. This paper presents 4DPS, a differentiable radar inverse-rendering framework for dynamic human micro-Doppler. Unlike prior radar inverse renderers that mainly model static scenes, 4DPS introduces explicit velocity into a point-based scene representation, where oriented points carry material, surface normal, and motion parameters. The method initializes geometry from synchronized camera observations and optimizes the dynamic scene and radar-related parameters against real radar captures. To make dynamic fitting tractable, it derives a closed-form range–Doppler splatting formulation that avoids explicitly rendering every chirp. This enables corpus-scale reconstruction of moving humans while substantially improving radar rendering fidelity and downstream sim-to-real performance.

Critical strengths
  1. The experimental evaluation is comprehensive. The paper validates 4DPS on two radar datasets, multiple signal representations, downstream sim-to-real transfer, ablations, and runtime analysis, with a quite complete supplement.
  2. The technical novelty and motivation are clear. The paper extends differentiable radar inverse rendering from static scenes to moving humans by introducing explicit differentiable velocity into the point representation and deriving a closed-form range–Doppler splatting formulation. This directly addresses the key limitation of prior static inverse renderers and provides a convincing motivation for reducing the sim-to-real gap of synthetic human radar data.
Critical weaknesses
  1. The major concern is that the external comparison is too narrow. The quantitative evaluation relies essentially on RF-Genesis as the only baseline. Although Appendix C.2 explains why a fully matched inverse-rendering baseline is unavailable, this still weakens the empirical positioning.
  2. 4DPS optimizes materials, surface normals, gains, velocities, and a learned noise floor, while important real-world effects such as multipath reflections are not explicitly modeled. As a result, some fitted “physical” parameters may partly absorb unmodeled propagation or calibration effects rather than correspond to physically meaningful quantities. The paper does not clearly disentangle physically correct modeling from data-driven compensation for model mismatch.
  3. Although the paper models the average noise floor, it does not explicitly model complex multipath reflections that are common in real radar measurements. The rendered-trained models still underperform real-trained ones substantially, but the paper does not quantify which signal statistics or propagation effects dominate this remaining gap.
  4. 4DPS still requires a reconstructed human mesh, static scene geometry, synchronized sensing, and sequence-level calibration against real radar data. In realistic deployments with diverse subjects, clothing, body shapes, and environments, repeatedly obtaining these inputs and refitting the renderer may substantially reduce the claimed data-collection savings. The current experiments do not show how much of same- subject/scene calibration can be reused across new subjects or environments.
Questions for authors
  1. Could the authors broaden the forward-simulation comparison? In particular, adapted comparisons with mmSim (Cui et al., 2023), FMCW-MIMO-Radar-Simulation (Kurtoglu et al., 2022), Controllable Radar Simulation with Waveform Parameter Embedding (Xiao et al., 2025), and 4D Radar Simulator for Human Activity Recognition (Zhou and Le Kernec, 2024) would better isolate the benefit of real-capture fitting over conventional simulation.
  2. Have the authors considered explicitly parameterizing or modeling additional real-world radar effects, such as multipath reflections and other interference factors, rather than letting the existing fitted parameters implicitly absorb them?
  3. How well do fitted parameters transfer across different subjects and genuinely different environments? For example, can person-specific parameters calibrated on one subject or scene/radar parameters calibrated in one room be reused without full refitting?
3 comments
AuthorsOpenReview ↗

Response to Reviewer hRH6

We thank the reviewer for the detailed review and the positive assessment of the evaluation. The general response (link) answers shared points as G1–G5.

W1: “The major concern is that the external comparison is too narrow. [...]”

Q1: “Could the authors broaden the forward-simulation comparison? [...]”

Answer: The four simulators you name cannot run here, but the benefit of fitting over conventional simulation is already isolated in the paper by our unfit render. RF-Genesis is the only released simulator that takes explicit recorded motion and arrays and outputs raw ADC (App. C.2). Thank you for the four references (G3): three synthesize every chirp in CPU loops, intractable at 584 million chirps, and Ctrl-RS outputs no ADC or elevation. We will add all four to Sec. 2 and App. C.2. In the unfit render (Sec. 4.1, Table 2), fitting raises micro-Doppler ρS\rho_S from 0.754 to 0.988 (HuPR) and from 0.515 to 0.966 (RT-Pose). On scale, other radar inverse renderers fit far less data: 3DPS 9,216 chirps and mmIR 2,496, against 584 million for 4DPS over 335 sequences (Sec. 3.3).

W2: “fitted ‘physical’ parameters may partly absorb unmodeled propagation [...]”

W3: “[...] does not quantify which [...] effects dominate this remaining gap.”

Q2: “[...] explicitly parameterizing or modeling [...] multipath reflections [...]?”

Answer: We render single bounces by design, as 3DPS and RadarSplat do and as mmIR finds sufficient; fitted parameters can absorb what the model leaves out, and we will say so in App. I. Single-bounce transport is a deliberate choice that prior radar inverse renderers share: mmIR, which traces up to three bounces, finds on its seven outdoor scenes that a single bounce loses no accuracy (range–azimuth ρ\rho 0.936 vs. 0.919 full); 3DPS renders single bounces only; RadarSplat also renders single bounces only, treating multipath as noise that it filters out before fitting and adds back to its single-bounce render for evaluation (removing it moves PSNR only from 26.06 to 25.95). One bounce is part of what makes fitting 584 million chirps tractable. The parameters are constrained to stay physical (ITU-R P.2040 materials, fixed positions, bounded velocities; Table 5) and explicit, and a scene fitted on one recording explains another of the same room at 90% and 92% of its own range–azimuth ρ\rho with nothing refitted (App. G.1, Table 19).

The paper locates the remaining gap but does not separate propagation from motion error, and we will state that. The deficit sits in the arms, while hips, knees and the head keep 0.73–0.80 of real-trained AP (Fig. 4, Table 13), and rendered- and real-trained models compute the same features from the backbone onward (Sec. 4.3, Table 4).

W4: “[...] how much of same-subject/scene calibration can be reused across new subjects or environments.”

Q3: “How well do fitted parameters transfer across different subjects and genuinely different environments? [...]”

Answer: Fitted parameters do not transfer across people or environments; they transfer across motions of the same person in the same scene, which is where data is saved. Differentiable rendering is scene-specific. A fit is specific to its person and scene, as for mmIR, 3DPS, Radar Fields and RadarSplat (G5); we will state this in App. I. Per session, 4DPS needs one synchronized camera–radar recording; the mesh comes from the video, and calibration is the fit itself. These are per session, not per motion. With one radar recording per person and scene and the other motions rendered, RT-Pose error falls from 30.3 to 20.9 cm, within 4.6 cm of recording every sequence, and HuPR AP rises from 36.3 to 43.7 (G1). We will add this to Sec. 4.2.

Thank you again for the thorough review and the references.

Thank you to the authors for their detailed responses and clarifications.

One of the main contributions of this paper is to improve the convenience and diversity of mmWave radar data generation. However, the limitation that fitted parameters cannot transfer across different subjects or environments inevitably restricts these advantages. Therefore, I still believe this is an important limitation that warrants further investigation and analysis.

Nevertheless, given the theoretical soundness and the distinctive contributions of the proposed method, I will maintain my original score.

We thank the reviewer for the follow-up and for maintaining the score. We agree that transferring fitted parameters across people and environments is important: until it is possible, 4DPS increases data diversity along the motion axis only, within each recorded person and scene. Even within that scope, one radar recording per person and scene is enough to render the remaining motions, which lowers RT-Pose error from 30.3 to 20.9 cm and raises HuPR AP from 36.3 to 43.7 (Sec. 4.4), and a new person or environment needs only one synchronized camera–radar recording.

In the revised PDF (link), Limitations and future work (Sec. 5) now states that transferring fitted parameters across people and environments is out of scope and left to future work, and App. I discusses this in greater detail.

Reviewer_HSjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents 4D Point Splatting (4DPS), a differentiable renderer for millimeter-wave radar observations of moving humans. It initializes scene geometry and body motion from synchronized visual observations, represents the scene using oriented points with material and velocity attributes, and fits parameters to recorded radar measurements. The method is evaluated on HuPR and RT-Pose through reconstruction fidelity, and downstream human-pose estimation. It achieves strong agreement with the recordings used for fitting and improves downstream transfer relative to the evaluated simulation baseline. However, the experiments establish reconstruction capability more convincingly than accurate simulation of new conditions or useful expansion of the available training data.

Critical strengths
  1. The method provides an explicit and differentiable representation for reconstructing radar signals from moving humans.

  2. The reconstruction experiments cover two datasets and hundreds of sequences, with practical fitting times and detailed signal-level evaluation.

Critical weaknesses

1. The signal reconstruction shown in the paper does not demonstrate the main value of simulation.

The introduction motivates simulation as a way to generate much more data than can be collected. But the quantitative downstream experiments mainly use reconstructions of data that already exist. Why use these reconstructions instead of the original measurements? Training on them performs worse than training on real data, and adding them does not consistently help.

2. Unfair comparison with baseline methods.

The main comparison is not fair: 4DPS fits the radar recordings used for evaluation, while baseline method (RF-Genesis) simulate without equivalent fitting. Better reconstruction after seeing the target signal does not establish better simulation of unseen conditions.

Questions for authors
  1. How well does the fitted representation predict a new motion or environment without fitting to its radar measurements? Also, how reliable can motion especially with Doppler effect being realistic synthesized?

  2. What is the demonstrated benefit of using the synthetic reconstructions instead of the original real data? Can additional, previously unrecorded synthetic conditions improve downstream performance with a fixed amount of real training data?

  3. How do geometry and motion errors affect phase and Doppler at millimeter wavelengths? The mmWave radar considered in the paper have wavelength of a few mm, which is much smaller than the visual reconstruction accuracy. How would the geometrical errors affect the reconstruction?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HSjA (1/2)

We thank the reviewer for the careful review; Q2 led to an important new experiment. The general response (link) answers shared points as G1–G5.

W1: “[...] Why use these reconstructions instead of the original measurements? Training on them performs worse than training on real data, and adding them does not consistently help.”

Q2: “What is the demonstrated benefit of using the synthetic reconstructions instead of the original real data? [...]”

Answer: The benefit is rendering motions that were never recorded: with downstream models trained on one radar recording per (person, scene) pair plus many 4DPS-rendered motions, RT-Pose error falls from 30.3 to 20.9 cm and HuPR AP rises from 36.3 to 43.7 (G1). We will add the result to Sec. 4.2. Training on reconstructions of recorded sequences is not meant to replace those recordings: it measures the sim2real gap, a secondary measure of reconstruction fidelity (G2). Renders alone reach 38.4 AP and 21.9 cm, where RF-Genesis reaches 1.8 AP and 127.6 cm (Table 3).

W2: “4DPS fits the radar recordings used for evaluation, while baseline method (RF-Genesis) simulate without equivalent fitting.”

Answer: We use RF-Genesis as a baseline as prior work does, Table 2 evaluates the quality of our fit as is standard in inverse rendering, and when both methods face unseen conditions, 4DPS outperforms RF-Genesis.

1. We follow prior work in using RF-Genesis as a baseline. RF-Genesis is the only released simulator that takes explicit recorded motion and arrays and outputs raw ADC (App. C.2), so it is the only one we can run on the recorded motions and radars of our datasets. HybridSim (Xiong et al., 2026), the closest fitted radar renderer, also compares against it; HybridSim and mmGPE release no code (G3).

2. 4DPS is a differentiable renderer, so we must evaluate the quality of our fit; this is standard practice for works in inverse rendering. Table 2 compares reconstruction by design, as is standard for differentiable rendering methods. It measures how faithfully 4DPS reconstructs a recording it has fitted, which is the paper's claim; RF-Genesis appears there as the only runnable forward simulator, not as a competing fitted method (App. C.3).

3. When both 4DPS and RF-Genesis face unseen conditions, 4DPS outperforms RF-Genesis. To test unseen conditions, we compare 4DPS and RF-Genesis on held-out motions, whose radar recordings neither method has access to. Your question is whether the advantage survives when 4DPS has not seen the target. We tested this on new motions: each is rendered from the fit of a different recording of the same person and scene, so neither 4DPS nor RF-Genesis has seen its radar. Scored against that radar, 4DPS matches range–azimuth far better (ρ\rho 0.688 vs. 0.130 on RT-Pose, 0.859 vs. 0.132 on HuPR; G4). Used as training data, they lower RT-Pose error from 30.3 to 20.9 cm and raise HuPR AP from 36.3 to 43.7 (G1).

AuthorsOpenReview ↗

Response to Reviewer HSjA (2/2)

Q1: “How well does the fitted representation predict a new motion or environment without fitting to its radar measurements? Also, how reliable can motion especially with Doppler effect being realistic synthesized?”

Answer: For a new motion of the same person and scene, the fit predicts the radar well without fitting to it; a new environment needs its own recording and fit.

New motion. We render a recorded motion from the fit of a different recording of the same person and scene, so the motion's own radar is never used. These renders match its real radar far better than RF-Genesis on range–azimuth (ρ\rho 0.688 vs. 0.130 on RT-Pose, 0.859 vs. 0.132 on HuPR; G4), and adding them to training improves pose models (G1).

New environment. Differentiable rendering is scene-specific. A fit describes the scene it was fitted to, as for mmIR and 3DPS and other inverse renderers. A new environment needs a new recording and fit (G5).

Doppler realism. For recorded motion, fitted micro-Doppler reaches ρS\rho_S 0.988 (HuPR) and 0.966 (RT-Pose), against 0.671 and 0.394 for RF-Genesis (Table 2). For new motion, it reaches 0.715 (HuPR) and 0.552 (RT-Pose), against 0.674 and 0.407 for RF-Genesis (G4).

Q3: “How do geometry and motion errors affect phase and Doppler at millimeter wavelengths? [...]”

Answer: They have little effect on what 4DPS fits: a centimeter-scale geometry error stays within one range bin and shifts the absolute phase of every antenna pair by nearly the same amount, while our loss supervises the relative phase between pairs, which sets the angle, not the absolute phase. The radar resolves range in bins of 4.29 cm (HuPR) and 4.51 cm (RT-Pose) (App. A.1). A centimeter offset at a few meters changes the direction to a point by a fraction of a degree, below the 1.3° (RT-Pose) and 14.3° (HuPR) azimuth resolution, so the relative phase between pairs barely changes. Following mmIR and 3DPS, we supervise the magnitude of each virtual antenna pair's ADC signal and the relative phase between pairs, through the range–azimuth–elevation magnitude loss (Sec. 3.3). Motion error enters through Doppler: because the velocity phase wraps (9 m/s apart on HuPR), each joint's velocity starts from the video's kinematics and is corrected by a learned residual bounded by the joint's own speed (Sec. 3.3). Fitted micro-Doppler reaches ρS\rho_S 0.988 (HuPR) and 0.966 (RT-Pose) (Table 2), and the remaining error sits in the arms, the smallest and fastest parts (Fig. 4, Table 13). Geometry is aligned to the radar in preprocessing (App. B.1); Sec. 4.2 states the fit's sensitivity to this registration, as for mmIR and 3DPS, and we will add it to App. I.

We hope these answers address your concerns, and we would be glad to discuss any of them further.

Reviewer_ev8pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 4D Point Splatting (4DPS), a physics-based differentiable renderer that fits an explicit moving scene to real mmWave radar recordings. The static scene and the moving person are initialized as oriented points using synchronized camera data (and LiDAR for RT-Pose). Each point carries material parameters based on ITU-R P.2040 and a surface normal, while the person's points also carry velocities derived from 14 joint velocities, initialized from video kinematics and refined with bounded residuals.

The method uses closed-form range–Doppler splatting to efficiently reconstruct complex radar returns, from which standard DSP representations can be derived. Materials, normals, velocities, and radar gains are optimized using magnitude-based losses on range–azimuth–elevation representations from the first chirp and Doppler planes.

The method fits 235 HuPR and 100 RT-Pose sequences, taking approximately 5.7 and 12.9 minutes per sequence on an H100 GPU, respectively. It reports improved rendering fidelity compared with RF-Genesis and the unfitted renderer, achieving micro-Doppler spectrogram correlations of 0.988 on HuPR and 0.966 on RT-Pose. In downstream pose estimation, a model trained exclusively on 4DPS renders achieves 21.9 cm MPJPE on real RT-Pose data, compared with 16.7 cm for real-data training and 127.6 cm for RF-Genesis.

Critical strengths

Prior differentiable radar inverse renderers primarily model static scenes and do not explicitly capture the slow-time dynamics needed for human micro-Doppler. 4DPS introduces closed-form Doppler splatting and differentiable velocities, enabling practical fitting of moving scenes at corpus scale (335 sequences). The fitted renders achieve high fidelity to real recordings, with micro-Doppler correlations of 0.988 on HuPR and 0.966 on RT-Pose, compared with 0.671 and 0.394 for RF-Genesis. The rendered data also achieve substantially better downstream sim-to-real transfer than RF-Genesis (21.9 vs. 127.6 cm MPJPE on RT-Pose). These results demonstrate a meaningful advance in physically grounded radar rendering of moving humans, bridging an important gap between forward-only motion simulators and static-scene inverse renderers.

Critical weaknesses

The paper is motivated by the scarcity of radar training data, but the main reconstruction and downstream experiments rely on renders fitted to existing real radar captures. Training on 4DPS renders alone still performs worse than training on real data (HuPR: 38.4 vs. 65.6 AP; RT-Pose: 21.9 vs. 16.7 cm MPJPE). More importantly, adding 4DPS renders to real training data does not demonstrate a consistent improvement in downstream performance (Table 11). The difference is small on RT-Pose (16.52 vs. 16.67 cm MPJPE), while performance decreases on HuPR (61.6 vs. 65.6 AP).

Although Appendix G demonstrates scene editing and novel-motion rendering, the novel motions are illustrated qualitatively, without quantitative fidelity evaluation or downstream task validation. These results establish the feasibility of rendering and fitting recorded motions, but provide limited evidence that generating additional synthetic samples improves downstream learning under data scarcity. This distinction matters because synthetic data generation and augmentation are central motivations of the paper.

Questions for authors
  1. Table 11 shows that adding 4DPS renders to real training data does not consistently improve downstream pose estimation performance. Have the authors evaluated whether synthetic data augmentation is more beneficial when only a small fraction of real training data is available? Such results would help establish whether 4DPS can meaningfully address radar data scarcity.

  2. Appendix G demonstrates novel-motion rendering and scene editing, but the novel motions are only illustrated qualitatively. Could the authors provide downstream evaluation using these newly generated samples, for example by augmenting a pose estimation or activity recognition model, to demonstrate their practical utility beyond fitting existing radar recordings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ev8p

We thank the reviewer for the positive assessment and two concrete questions, which led to an important new result. The general response (link) answers shared points as G1–G5.

W1: “[...] adding 4DPS renders to real training data does not demonstrate a consistent improvement in downstream performance (Table 11).”

Q1: “Have the authors evaluated whether synthetic data augmentation is more beneficial when only a small fraction of real training data is available?”

Answer: Yes, and it helps substantially: with one radar recording per person and scene plus renders of the other motions, RT-Pose error falls from 30.3 to 20.9 cm and HuPR AP rises from 36.3 to 43.7, outperforming models trained on two real recordings per pair without renders (G1). Thank you, this question led to a new result, and we will add it to Sec. 4.2. Table 11's mixed arm differs: it adds renders of motions the real data already contains, so on full data it measures fidelity, not augmentation. The downstream results are a secondary measure of reconstruction fidelity, and the goal is to come as close to real-data performance as possible, not to beat it (G2): renders alone reach 38.4 vs. 65.6 AP (HuPR) and 21.9 vs. 16.7 cm (RT-Pose), where RF-Genesis reaches 1.8 AP and 127.6 cm (Table 3). HuPR is the harder case compared to RT-Pose (Sec. 4.2): 14.3° (single-chip radar) vs. 1.3° (cascaded radar) azimuth resolution which means worse training data and worse alignment to camera initialization, 2 vs. 8 elevation bins so worse elevation resolution, less training data due to lower virtual antenna count, and 2D keypoints vs. 3D ground truth.

W2: “[...] the novel motions are illustrated qualitatively, without quantitative fidelity evaluation or downstream task validation.”

Q2: “Could the authors provide downstream evaluation using these newly generated samples [...]?”

Answer: Yes: renders of motions the fit never saw, added to one or two radar recordings per person and scene, lower RT-Pose error from 30.3 to 20.9 cm, within 4.6 cm of recording every sequence, and raise HuPR AP from 36.3 to 43.7 (G1). We rendered these motions without fitting or accessing their radar recordings. Scored against their real radar equivalents, these renders exceed RF-Genesis on range–azimuth ρ\rho (0.688 vs. 0.130 on RT-Pose, 0.859 vs. 0.132 on HuPR; G4). App. G.1 (Table 19) already scores one scene edit, at 90% and 92% of the recording's own range–azimuth ρ\rho with nothing refitted; we will point to it from Sec. 4.

Thank you again, both questions improved the paper.

Reviewer_BmGMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The objective of the proposed work in my understanding is to synthetically generate radar data for full body motion of humans given the scarcity in obtaining radar data for moving humans. To this end the authors propose 4DPS, by fitting an explicit human motion model to its real radar sequence counterpart. This is done by initialising a differentiable renderer with room geometry in addition to the body mesh for each frame in the video. The geometry is kept fixed and the method refines the material estimation, normals, velocities to match the radar output. The fitted renderer can then be used to synthesise new motions. The central contribution of the work extension to temporal motion and efficient rendering of radar chirps.

Critical strengths
  1. The method design is practical: The closed form for doppler response of each moving point is efficient. Fitting to real radar points to bridge sim to real gap is straightforward and intuitive.
  2. The authors throughly investigate the reconstruction fidelity of the radars (Table 2, main paper supported with Table 7-10 in supp mat). 4DPS + fitting shows substantial improvement across all metric specifically micro doppler spread.
Critical weaknesses
  1. The paper is somewhat difficult to follow in places. The introduction presents too many methodological details before establishing the context and positioning against the literature.

  2. Scope of novelty: In my understanding Midas [1] already introduces the idea of synthetic human dopplers from videos. Authors in [2] optimizes scattering parameters of the simulator against doppler spectograms for human data. Both of the works haven't been cited. It will be helpful to have some differentiation against these works. Although, as this is not my direct area of expertise I would like to defer to reviewers that are more aligned with the field for the novelty concerns.

  3. The authors demonstrate that the model trained on the generated synthetic data could be transferred to real data for down streaming tasks like pose estimation. However, its not clear that if synthetic data generated through the renderer for new motions can replace real data for training.

[1] Deng K, Zhao D, Han Q, Zhang Z, Wang S, Zhou A, Ma H. Midas: Generating mmwave radar data from videos for training pervasive and privacy-preserving human sensing tasks. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2023 Mar 28;7(1):1-26.

[2] Copa EI, Yildirim HC, Determe JF, Horlin F. Synthetic radar signal generator for human motion analysis. IEEE Transactions on Radar Systems. 2024 Dec 17;3:88-100.

Questions for authors

The main questions that would be helpful if answered as as follows:

  1. What differentiates 4DPS from prior works like Midas or [2] ? Does 4DPS warrant a comparison with the above mentioned methods?

  2. How much does the synthetic generated data from 4DPS help in bridging the gap to real data for tasks like pose estimation specifically for new motion absent in real data or held out ?

  3. What other down streaming tasks has 4DPS been evaluated on ?

1 comment
AuthorsOpenReview ↗

Response to Reviewer BmGM

We thank the reviewer for the review and the two references. The general response (link) answers shared points as G1–G5.

W1: “The introduction presents too many methodological details before establishing the context and positioning against the literature.”

Answer: Agreed. We will restructure Sec. 1 so that the problem and the three families of prior work (forward-only simulators, learned generators, static-scene inverse renderers) come first, with Table 1, and the three technical challenges of an explicit velocity follow.

W2: “Midas [1] already introduces the idea of synthetic human dopplers from videos. Authors in [2] optimizes scattering parameters of the simulator against doppler spectograms [...]”

Q1: “What differentiates 4DPS from prior works like Midas or [2]? Does 4DPS warrant a comparison [...]?”

Answer: 4DPS differs from both by fitting a renderer to every capture it renders, and neither can serve as a baseline; we will cite both in Sec. 2.

  • Midas computes per-vertex RCS and radial velocity on meshes estimated from video, passes them through a reflection model with a hand-set attenuation coefficient, and makes the output realistic with a Transformer trained on pairs of generated and real radar, as mmGPE's learned refiner does (Sec. 2). Both are learned generators: they train one network across many sequences so that it generalizes to new ones, and nothing is fitted to the capture being rendered. 4DPS is a differentiable renderer, which solves a different problem: it fits an explicit, physics-based model to each recording on its own, so every render is tied to that recording's measured radar.
  • [2] places scatterers on ellipsoids fitted to each body part and calibrates a few global parameters per body-part group and per propagation path by Bayesian optimization, against the measured Doppler–time spectrograms of two 1-s clips of one activity. Its authors chose Bayesian optimization because the objective's “derivatives are extremely difficult to compute”, and note the parameters may not transfer to another radar position or activity; every sequence beyond the calibration pairs is rendered forward. 4DPS instead fits every sequence it renders, with exact gradients through a differentiable renderer, and fits per-sequence materials, normals, joint velocities, gains and noise floor (Sec. 3.3, Table 5); the closed-form Doppler splat makes this tractable at dataset scale (Sec. 3.2).

Neither can be run as a baseline: Midas outputs no raw ADC or range–azimuth–elevation–Doppler cube, and [2] releases no code (G3).

W3: “its not clear [...] synthetic data [...] for new motions can replace real data for training.”

Q2: “How much does the synthetic generated data from 4DPS help [...] specifically for new motion absent in real data or held out?”

Answer: Substantially: renders of motions absent from the real training data lower RT-Pose error from 30.3 to 20.9 cm and raise HuPR AP from 36.3 to 43.7 (G1). Thank you for the question; we ran exactly this experiment. We kept only one real radar recording per (person, scene) pair and rendered every other motion by editing the fit to that recording. This model even outperforms one trained on two real recordings per pair without renders (43.7 vs. 39.3 AP; 20.9 vs. 22.2 cm). We will add it to Sec. 4.2.

Q3: “What other down streaming tasks has 4DPS been evaluated on?”

Answer: We evaluated 4DPS only on the pose estimation task, because we are restricted by the availability and compatibility of existing datasets. Of the mmWave human-sensing datasets we surveyed (App. C.1, Table 6), only RT-Pose and HuPR release raw ADC with slow time and a co-located camera, and both happen to have been collected for pose estimation. 4DPS renders the raw signal, so the representations other tasks use (range–Doppler and micro-Doppler for activity, fall and vital signs) follow from it, and we score each against the recordings (Table 2, Tables 7–10). We will add this to Sec. 5.

Thank you again for the references and the clear questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers (1/3)

We thank the reviewers for careful, constructive reviews. Reviewers found the fitting “straightforward and intuitive” (BmGM), the evaluation “comprehensive” (hRH6), the experiments to “cover two datasets and hundreds of sequences, with practical fitting times” (HSjA), and the results “a meaningful advance in physically grounded radar rendering of moving humans” (ev8p).

4DPS fits a differentiable, physics-based radar renderer to the raw radar recording of a moving person in a scene, at the scale of entire datasets (584 million chirps over 335 sequences), and can then render that person in that scene with new motions. An important new result (G1): a pose model trained on a few real radar recordings plus 4DPS renders of all the other motions beats one trained on those few recordings alone, on both datasets, and on RT-Pose comes close to one trained on every real recording. G1–G5 answer the points raised in several reviews; each reviewer thread points back to them. Numbers refer to the submitted PDF.

G1. New: Train a downstream model on one radar recording per (person, scene) pair, with the remaining motions rendered (BmGM W3/Q2, HSjA W1/Q2 and Q1, ev8p W1/Q1 and W2/Q2, hRH6 W4/Q3)

Downstream models trained on one to two real radar recordings per (scene, person) pair, plus 4DPS renders of all the other unseen motions, significantly outperform models trained only on the few real recordings. On RT-Pose, training on few real and mostly 4DPS renders comes close to training on real recordings of all motions. Radar pose datasets record every motion of every person with the radar. HuPR's 193 training sequences come from 11 recording sessions, about 18 per session; RT-Pose's 44 come from 9 sessions, about 5 per session. Within a session the person and the scene stay the same; only the motion changes, and every sequence is a different motion. Following ev8p's Q1 and Q2, we retained only one or two radar sequences per session and ignored the rest, fitted them, then edited the person's motion in the fit to match every other motion of the session, and then rendered each as a new recording, without ever accessing the real radar recordings corresponding to those motions. Pose models are trained on the recorded sequences plus these renders, with the same number of training steps as real-only training, and tested on the paper's real test set:

Training dataRT-Pose: recorded + renderedRT-Pose: MPJPE (cm) ↓HuPR: recorded + renderedHuPR: AP ↑
1 recording per session9 + 030.311 + 036.3
+ 4DPS renders of the other motions9 + 3520.911 + 18243.7
2 recordings per session17 + 022.222 + 039.3
+ 4DPS renders of the other motions17 + 2719.622 + 17145.2
Every sequence recorded (ceiling)44 + 016.3193 + 065.0

Two seeds per row (four for RT-Pose with 4DPS renders). Real-only rows are retrained under this protocol, so they differ slightly from Table 3.

Adding renders of the unrecorded motions helps on both datasets, and on RT-Pose brings the model close to the all-real ceiling:

  • RT-Pose comes close to the ceiling. With 9 of 44 sequences recorded and 35 rendered, the error is 20.9 cm, 4.6 cm from recording all 44 (16.3 cm) and down from 30.3 cm without the renders. With 17 recorded, it is 19.6 cm, 3.3 cm from the ceiling.
  • HuPR. With 11 of 193 sequences recorded (6%), AP rises from 36.3 to 43.7. HuPR is the harder case compared to RT-Pose (Sec. 4.2): a smaller share of real data (6% vs. 20%), 14.3° (single-chip radar) vs. 1.3° (cascaded radar) azimuth resolution which means worse training data and worse alignment to camera initialization, 2 vs. 8 elevation bins so worse elevation resolution, less training data due to lower virtual antenna count, and 2D keypoints vs. 3D ground truth.
  • Renders are worth more than doubling the number of real recordings. One recording per session plus renders beats two recordings per session without them, on both datasets (20.9 vs. 22.2 cm; 43.7 vs. 39.3 AP), and on every seed.
AuthorsOpenReview ↗

General response to all reviewers (2/3)

For every (scene, person) pair, 4DPS renders new motions, allowing us to significantly cut down on the number of real sequences recorded. Each (person, scene) pair is recorded with many unique motions (about 18 on HuPR, 5 on RT-Pose), and because 4DPS can render new motions, we are able to significantly reduce the amount of real data required for downstream training.

Note that this experiment differs from the mixed arm of Table 11. There, we need all real radar sequences, so we can fit every sequence and render our reconstruction. Each rendered reconstruction shows a motion the real data already contains, so the renders add no new motions. Here, because we only fit to a subset of the real radar recordings, 4DPS only sees the motions in those few radar recordings during fitting. The remaining withheld real radar recordings are never inspected or accessed. Instead, we edit the person's geometry and motion to render new motions.

In terms of runtime cost, a fit takes 5.7 (HuPR) or 12.9 (RT-Pose) minutes per recording on one H100 (Table 25), plus one render per motion.

We will add this experiment to the results in the main paper (Sec. 4.2), with the construction and per-sequence results in App. G. We thank the reviewers for suggesting this experiment, as it strengthens the paper.

G2. Downstream results on 4DPS reconstructions are a secondary measure of fidelity and the sim2real gap (BmGM W3/Q2, HSjA W1/Q2, ev8p W1/Q1)

The downstream experiments measure the sim2real gap and are a secondary measure of reconstruction fidelity beside the direct comparison with the recordings (Table 2). The goal is to come as close to the real-data performance ceiling as possible, not to beat it. Pose models trained on 4DPS renders alone reach 38.4 AP (HuPR) and 21.9 cm (RT-Pose), against 65.6 AP and 16.7 cm with real data, while RF-Genesis, the only other method that generates raw ADC from our motions, reaches 1.8 AP and 127.6 cm (Sec. 4.2, Table 3). The gap is small enough that rendered data supports the downstream task. That is what makes G1 work: for a given person and scene, 4DPS renders new motions that pose models can use, so we can dramatically reduce the amount of real data that needs expensive, time-consuming recording.

The paper never claims that renders outperform real data. Its contributions (Sec. 1) are (1) a differentiable renderer of human micro-Doppler with a velocity estimated per joint, (2) closed-form range–Doppler splatting that makes inverse rendering tractable on entire datasets, here 584 million chirps over 335 sequences (Sec. 3.3), and (3) the first sim2real analysis of physics-based radar inverse rendering at dataset scale, on two downstream models covering two different radars. We will state this in a scope paragraph after the contributions and in the abstract's last sentence.

G3. RF-Genesis is the only runnable baseline, and the six suggested methods cannot be run here (HSjA W2, hRH6 W1/Q1, BmGM W2/Q1)

RF-Genesis is our only external baseline because it is the only released simulator that takes explicit recorded motion and arrays and outputs raw ADC (App. C.2). HybridSim (Xiong et al., 2026), the closest fitted radar renderer, also compares against it; HybridSim and mmGPE release no code. Our paper isolates what fitting adds with its own control, the unfit render: the 4DPS chain with the person unfitted, i.e. a conventional simulation of the person (Sec. 4.1, Table 2). Fitting raises micro-Doppler ρS\rho_S from 0.754 to 0.988 (HuPR) and from 0.515 to 0.966 (RT-Pose). Even unfit, the render exceeds RF-Genesis (range–azimuth ρ\rho 0.878 vs. 0.134 and 0.648 vs. 0.119).

We thank the reviewers for the six additional references. None of them can be run as a baseline here:

  • Midas (Deng et al., 2023) outputs Doppler–time plots and point clouds, with no raw ADC or range–azimuth–elevation–Doppler cube.
  • Copa et al. (2024) release no code.
  • mmSim, FMCW-MIMO-Radar-Simulation and the 4D Radar Simulator synthesize every chirp in CPU loops. The 4D Radar Simulator's authors report 1.5 hours per 3-second clip at 12 GHz and running out of memory at 77 GHz, our band; at 584 million chirps this is intractable, which is the cost the closed-form Doppler splat removes (Sec. 3.2).
  • Ctrl-RS (Xiao et al., 2025) outputs a range–azimuth–Doppler cube for road scenes, with no ADC or elevation.

We will cite all six in Sec. 2 and add them to App. C.2. G4 compares 4DPS and RF-Genesis on motions neither has seen.

AuthorsOpenReview ↗

General response to all reviewers (3/3)

G4. 4DPS renders of new motions match their real radar far better than RF-Genesis (BmGM Q2, HSjA W2, ev8p W2/Q2, hRH6 Q3)

Scored against their own radar recordings, the 4DPS renders of new motions that G1 trains on match far better than RF-Genesis on range–azimuth (ρ\rho 0.688 vs. 0.130 on RT-Pose, 0.859 vs. 0.132 on HuPR). We render recording BB's motion from another recording AA's fit of the same person and scene, and compare it with BB's real recording, which neither 4DPS nor RF-Genesis sees. On micro-Doppler, the renders also score above RF-Genesis (ρS\rho_S 0.552 vs. 0.407 on RT-Pose, 0.715 vs. 0.674 on HuPR). We will add the full comparison to App. G.

Recordings per sessionBoardsMicro-Doppler ρS\rho_S: 4DPS edited renderMicro-Doppler ρS\rho_S: RF-GenesisRange–azimuth ρ\rho: 4DPS edited renderRange–azimuth ρ\rho: RF-Genesis
RT-Pose, 1340.5520.4070.6880.130
RT-Pose, 2260.6360.4030.7140.140
HuPR, 13640.7150.6740.8590.132
HuPR, 23420.7600.6740.8760.132

Medians over radar boards (HuPR: two per render). Both methods are scored on the same boards.

G5. Scope: 4DPS Fitted scenes only transfer to new motions of the same person in the same scene (HSjA Q1, hRH6 W4/Q3)

A fit does not transfer to a new person or scene; it transfers to new motions of the same person in the same scene. Differentiable rendering is scene-specific: like mmIR, 3DPS, Radar Fields and RadarSplat, 4DPS recovers the scene from that scene's own measurements. A fit therefore describes one subject in one scene, and transfer to a new environment or a new subject is outside what inverse rendering provides; we will state this in App. I. Within this (scene, person) pair, a fit can be edited to render new motions: G1 trains pose models on them, G4 scores them against the radar, and App. G.1 renders a person fitted on one recording in the scene fitted on another, keeping 90% (RT-Pose) and 92% (HuPR) of that recording's own range–azimuth ρ\rho with nothing refitted (Table 19).

Revisions we will make

We will make five revisions:

  1. Sec. 1: context and prior work before method detail (BmGM W1); a scope paragraph after the contributions; the abstract's last sentence (G2).
  2. Sec. 2, App. C.2: the six methods above (G3); RF-Genesis described as a forward simulator driven by the recorded motion, as App. C.3 does.
  3. Sec. 4.2: the new-motion training experiment (G1). App. G: its construction and per-sequence results, and the signal-level comparison (G4).
  4. App. I: a fit describes one person in one scene (G5); fitted parameters can absorb unmodeled propagation (hRH6 W2); the fit is sensitive to camera-to-radar registration, as Sec. 4.2 states (HSjA Q3).
  5. Sec. 4: a pointer to App. G.1 (ev8p W2).
AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised PDF. Changes made in response to the reviews are shown in red. Minor correctness and conciseness edits, made so that the additions fit within the 10-page limit, are shown in blue. Section and appendix numbers below refer to the revised PDF.

  1. Novel motion synthesis is now part of the main paper, and the title now ends "… Differentiable Rendering and Novel Motion Synthesis". Sec. 4.3 renders new motions by editing a fitted recording and scores them against their real radar recordings, against RF-Genesis (construction, scores and gallery in App. F). Sec. 4.4 adds the new downstream experiment (general response G1): with one or two real recordings per session plus 4DPS renders of the other motions, RT-Pose error falls from 30.3 to 20.9 cm and HuPR AP rises from 36.3 to 43.7 (training setup and per-sequence results in App. G).
  2. Introduction (BmGM W1). The problem and prior work now come before method detail, and a scope paragraph follows the contributions (G2).
  3. Related work (G3). The six suggested methods are cited in Sec. 2, and App. C.2 explains why none can serve as a baseline.
  4. Limitations (Sec. 5, App. I). A fit describes one person in one scene, and transfer to new people and environments is stated as future work (G5, hRH6 W4/Q3). App. I also states that fitted parameters can absorb unmodeled propagation (hRH6 W2) and that the fit is sensitive to camera-to-radar registration (HSjA Q3).
  5. Space. To make room, the comparison of real- and rendered-trained models (formerly Sec. 4.3) and the explanation of why HuPR is the harder downstream task (formerly Sec. 4.2) moved to App. E, marked "moved from".

We thank the reviewers again for their suggestions, which strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.