- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces AvatarPilot and formulates active 4D human avatar reconstruction as a sequential camera-control problem. A physically constrained camera observes depth, a person mask, its own pose history, and an accumulated voxelized capture history, and a PPO policy selects both motion on a viewing sphere and motion of the sphere center. Training uses a proxy reward based on newly observed direction bins of persistent surface Gaussians. The authors construct a simulator from multi-view DNA-Rendering captures, using GauHuman avatars for policy training and per-frame MaskGaussian reconstructions for evaluation. On 24 held-out sequences from 10 subjects, the learned policy is evaluated under in-place and synthetically translated “walking” conditions with both 3DGS-Avatar and template-free 4DGS. It consistently outperforms fixed, random-walk, orbit, and spiral trajectories in the main table, and the paper additionally studies capture distance, observation noise, cross-dataset transfer, fixed-camera arrays, greedy planners, and design ablations.
Critical strengths
First, the paper defines a meaningful and underexplored problem at the intersection of active vision, camera control, and 4D human reconstruction. The policy uses deployable observation types rather than ground-truth body pose, and evaluation is based on the quality of reconstructions produced from the selected images rather than only the training reward. This makes the task formulation consequential and potentially useful.
Second, the evaluation is broad within the simulator. The authors use subject-disjoint data, three policy seeds, two substantially different reconstruction back ends, several capture distances, observation-noise tests, cross-dataset transfer, fixed-camera comparisons, and design ablations. The learned policy also beats the preset trajectories consistently in Table 1, especially when the unmodified presets lose the moving person.
Critical weaknesses
First, the central real-world claim is not validated. All policy inputs, captured images, and test references are rendered from fitted studio-capture assets; no real camera or drone is run in closed loop. The “walking” condition is created by rigidly translating an in-place performance along a synthetic path, so it omits natural coupling between locomotion, heading changes, articulation, clothing dynamics, background, etc. The motion model further assumes a fixed-radius viewing sphere and look-at orientation, with physical feasibility represented mainly by a per-step displacement bound. Consequently, the experiments establish performance in a clean simulator, not the claimed autonomous physical capture setting.
The computational feasibility of the onboard perception loop is also unverified. In simulation, human masks are obtained by thresholding rendered alpha, whereas deployment would require an actual segmentation model, for which the paper cites Segment Anything (Sec. 3.3, p. 5). The reported approximately 8 ms depth/mask-to-command runtime is measured on an NVIDIA RTX 6000 Ada workstation GPU and explicitly excludes real-camera segmentation and depth sensing (App. A.1, p. 15); it therefore does not establish end-to-end performance on a power- and memory-constrained onboard platform. A lightweight or optimized segmenter might be sufficient, so this is not evidence that onboard deployment is impossible. Rather, the missing hardware-specific accuracy–latency evaluation leaves a central prerequisite for the practical claim untested.
Second, the evidence does not convincingly isolate the benefit of learned long-horizon active view selection. In the main walking comparison, the preset orbit and spiral remain centered on the person’s initial position and therefore contain no person in about 41% of frames. Once the same onboard position estimate is used to make the spiral follow the person, the gap shrinks from approximately 2.55 dB to only 0.27 dB PSNR and 0.003 LPIPS (Table A3), and this stronger comparison is reported only as means without confidence intervals or significance marks. No established learned active-view baseline is adapted. Moreover, the one- and three-step privileged planners achieve higher final proxy reward while producing worse reconstructions (Table A4/Fig. A5), showing that the reward is not monotonically aligned with the actual objective. Together, these results leave unclear how much is gained from the proposed long-horizon policy rather than from target tracking, the restricted action parameterization, or simulator-specific inductive bias.
Questions for authors
-
Can the authors provide paired per-sequence results and confidence intervals for the comparison against orbit+tracker and spiral+tracker, preferably for both 3DGS-Avatar and 4DGS and with uncertainty clustered by subject? Would the +0.27 dB / −0.003 LPIPS margin over spiral+tracker remain statistically reliable across policy seeds?
-
Do the authors have any closed-loop results using a real RGB-D camera, drone, or rail system? If not, can they evaluate on genuinely translating/turning human captures with realistic background, segmentation, depth, and dynamics errors rather than rigidly translating an in-place fitted avatar?
-
Why do the privileged greedy planners obtain higher final directional-diversity reward but worse reconstruction quality? Please report per-sequence reward–quality correlations and clarify what prevents reward exploitation.
-
Why are established active-view policies such as GenNBV/GLEAM or recent dynamic-GS next-best-view methods not adapted as quantitative baselines? If they are incompatible, please specify the exact incompatibility and compare against the strongest non-learning receding-horizon alternative under the same onboard observations.
-
Which human-segmentation model, input resolution, numerical precision, and automatic prompting/tracking mechanism are intended for deployment? Can the authors report end-to-end sensor-to-command latency, including its variability, sustained update rate, peak memory, and power on a representative onboard device rather than a workstation GPU, and evaluate framing/reconstruction under the resulting delayed or dropped masks?
-
Could the authors assess whether the policy’s gains persist with an additional recent human-specific Gaussian reconstruction method, such as Vid2Avatar-Pro (CVPR 2025), using identical learned and tracking-enhanced baseline trajectories, matched capture budgets, and held-out evaluation views? If adaptation is infeasible, please clarify the input or preprocessing incompatibilities.
7 comments
Response to Reviewer o3Gc on Question 5: onboard perception loop (Part 2/5)
Delayed or dropped masks (Q5, last part). The revised PDF adds this test, in simulation and without retraining. The policy issues one command per control step, every 133 ms (two frames of the 15 fps recordings). With a delay of steps, the camera executes at each step the command computed steps earlier; with a dropped mask, the person mask of that step does not reach the policy. A delay of one step, the worst case for a perception loop that takes up to 133 ms, lowers the PSNR by 0.19 dB in place and 0.10 dB when the person walks, and keeps the person in the image in every frame. When the person walks, dropping the mask at 10% of the steps has no measurable effect. The PSNR is measured with 3DGS-Avatar on 12 sequences that cover all ten subjects, with one training seed. The new App. E.4 and Table A17 of the revised PDF give the details and also report delays of two and three steps.
Intended deployment and onboard latency (Q5, first and second parts). The policy needs one 256 × 256 person mask and depth map per control step (Table A1). The mask can come from a person-segmentation network that runs in real time, which outputs the person mask directly, without prompts or tracking; Sec. 3.3 of the revised PDF now refers to a person-segmentation network instead of Segment Anything. For such a network (PeopleSemSegNet), NVIDIA's Isaac ROS benchmarks report 4.7 ms on a Jetson AGX Orin and 8.0 ms on a Jetson Orin Nano Super, far below the 133 ms control step. The end-to-end figures of the whole loop (latency, update rate, memory and power, at a given numerical precision) depend on the onboard device that is chosen. We will measure them when the policy is deployed on a real platform, which is the next step of this work; the limitations of the revised PDF now note this.
Response to Reviewer o3Gc on Questions 1 and 4: isolating the benefit of the learned policy (Part 3/5)
Learned policy versus target tracking. The task asks the camera to keep the person in view, to see the body from as many directions as possible, and to keep it inside the image at close range, with onboard observations only. Our policy learns all three from one reward, the gain in directional diversity. It is free to move the centre of its viewing sphere, and it learns to move it with a walking person although nothing in the reward refers to following. Following is thus one of the abilities the policy learns, and it is why its lead grows when the person walks: the plain presets capture no person in about 41% of their frames, and our policy in under 0.1%.
Comparison against orbit + tracker and spiral + tracker (Q1). These presets are our own construction rather than an existing method: a preset trajectory does not observe the person, so we let the centre of their path follow the onboard position estimate of our own system. The revised PDF now reports this comparison with intervals and per-sequence results for both reconstruction methods (new App. E.1 and Tables A13 and A14 of the revised PDF), and they are new rows of Table 1. Our policy minus each of them in the walking condition (PSNR in dB):
| Our policy minus | Mean | 95% interval, clustered by subject | Seeds with interval above zero | Sequences ahead |
|---|---|---|---|---|
| Orbit + our tracker, 3DGS-Avatar | +2.17 | [+1.30, +3.28] | 3 of 3 | 18/24 |
| Spiral + our tracker, 3DGS-Avatar | +0.27 | [+0.10, +0.45] | 2 of 3 | 15/24 |
| Orbit + our tracker, 4DGS | +1.13 | [+0.57, +1.81] | 3 of 3 | 16/24 |
| Spiral + our tracker, 4DGS | +0.71 | [+0.45, +1.03] | 3 of 3 | 19/24 |
The clustered interval covers the three training seeds and resamples the 10 subjects with all their sequences kept together; it lies above zero in all four comparisons. Table A13 gives the interval of each seed separately. In three of the four comparisons it lies above zero for all three seeds. In the fourth (spiral + our tracker, 3DGS-Avatar) it lies above zero for two seeds, and for the third seed it is [−0.02, +0.28]. App. E.1 reports the same check for LPIPS. In place the person performs the recorded motion without travelling, so there is nothing to follow, and a gain comes from choosing better viewpoints. There our policy leads the plain spiral by 0.27 dB (3DGS-Avatar) and 0.58 dB (4DGS). When the person walks and the spiral follows with our tracker, the lead is of similar size (0.27 and 0.71 dB).
GenNBV, GLEAM and dynamic-scene methods as baselines (Q4, first part). GenNBV and GLEAM scan static objects and explore static indoor scenes with a coverage reward, with no moving person to keep in view, so they cannot be run on our task as they are. The closest we can come within our task is to train our policy with a coverage reward of their kind, which is ablation (a) of the paper: it is 0.58 dB (in place) and 0.78 dB (walking) below our reward (Table A10). The recent view-selection methods for dynamic scenes (Sec. 2) select frames offline from pre-recorded multi-camera video, reconstruct a static scene while treating pedestrians as a disturbance, or plan views for a moving rigid object; none controls a camera around a moving person.
Non-learning planners under the same onboard observations (Q4, second part). The revised PDF adds three such planners (new rows of Table A4 and App. B.3 of the revised PDF). Like the paper's privileged planners they replan at every step and maximise our reward one or three steps ahead (a receding horizon), but they use only the onboard detection and depth maps; one of them also counts the space not observed yet, as classical next-best-view planning does. They are 1.2 to 3.0 dB behind our policy with both reconstruction methods, and the strongest, the one that counts unobserved space, is 1.2 to 1.7 dB behind. Unlike them, our policy is trained on the return of the whole capture.
Restricted action parameterisation and simulator-specific bias. Your review also asks whether the gain could come from the restricted action parameterisation or from a bias of the simulator. The action parameterisation is shared by every strategy: the random walk and the planners use the same capture distance and the same per-step limit, and the random walk is more than 2 dB behind our policy (Table 1). As for the simulator, the images given to the reconstruction come from per-frame reference models rather than from the avatars the policy trains on, and the policy transfers without retraining to a second dataset, MVHumanNet, with leads of similar size (Table A9).
Response to Reviewer o3Gc on Question 2: real-world validation (Part 1/5)
Thank you for the careful and detailed review. We have uploaded a revised PDF in which new text is marked in blue, and we answer in five comments, in the order of the weaknesses in your review, each with the related questions.
Closed-loop results with a real camera (Q2, first part). We have no closed-loop result with a real camera, drone or rail system: all results are obtained in simulation. The introduction and the limitations state this, and the abstract of the revised PDF now states it as well. The paper contributes the task, a simulation and evaluation platform built from real captures, and a policy learned on it. We see these results as a step toward autonomous capture in the real world, and running the policy on a real platform is the next step.
Genuinely translating and turning captures (Q2, second part). DNA-Rendering is a diverse repository of real human performances, and our held-out sequences already include turning (the heading sweeps up to 175°) and large body motion. Since the walking condition moves a performance recorded in place along a path, the revised PDF additionally evaluates on seven DNA-Rendering sequences with more walking and turning: in six the person walks, and in one the person turns a full circle on the spot. They are evaluated in the same simulator with the motion as recorded, without added displacement and without retraining; they are used neither for training nor in the 24 held-out sequences. Our policy minus each preset in PSNR (dB):
Our policy is ahead of all four presets with both methods, on six or seven of the seven sequences. The details are in the new App. E.3 of the revised PDF (Table A16, Lead of our policy on recordings of real walking and turning; Fig. A7).
Background, segmentation and depth errors (Q2, second part). The simulator renders no background. The policy does not see colour, so a background would reach it only through errors in the person mask and the depth map. The submission tests such errors in synthetic form, without retraining (Sec. 4.3; App. B.4, Table A5): noise on the depth map, the mask and the camera's own pose estimate, and missed detections. At the heaviest of four levels the reconstruction loses at most 0.41 dB (one training seed). Dropped masks are in our next comment.
Motion model (fixed radius, look-at orientation, per-step bound). The fixed capture distance and the look-at orientation are design choices that keep the task simple and let every strategy capture from the same distance; Table A10 ablates the look-at constraint, and letting the policy also turn the camera by up to ±30° lowers PSNR by 0.68 dB in place and 0.62 dB walking. The per-step limit is derived from the thrust-to-weight ratio of a consumer multirotor (App. A.2); as you note, it stands in for flight dynamics rather than simulating them, and the limitations of the revised PDF now state this. A first test in this direction is in our next comment: how the policy tolerates commands that are executed late.