acceptodds
Under review as a conference paper at ICLR 2027

AvatarPilot: Learning a Camera Policy for Active 4D Human Avatar Reconstruction

Abstract

Capturing a moving person for 4D reconstruction typically relies on studio rigs with dozens of fixed cameras, which are costly and confine capture to the studio. An autonomous camera that moves around the person would be a far more flexible and cheaper alternative, but the quality of the reconstruction then depends on where the camera goes. We formulate this as active 4D human avatar reconstruction, a task in which a physically constrained camera decides at every step where to move next using only its onboard observations, to achieve a better 4D reconstruction from the images it captures. We solve this task by learning a camera policy with reinforcement learning. To train it, we turn hundreds of existing multi-view captures of real people into renderable 4D avatar assets made of 3D Gaussians, and build on them a pipeline from simulated onboard observations to camera actions, along with an evaluation protocol. We define the training reward directly on the avatars' surface Gaussians, which encourages the policy to see the body from new viewing directions. Experiments show that on held-out subjects, the learned policy outperforms every preset trajectory we compare against. When the person walks, the reconstructions from its captures exceed those from the strongest preset trajectory by 2.5 dB PSNR. When the person stays in place, one camera under our policy matches the reconstruction quality of six to eight fixed cameras.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Active Reconstruction, Next-Best-View, Reinforcement Learning, 4D Human Reconstruction, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o3GcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AvatarPilot and formulates active 4D human avatar reconstruction as a sequential camera-control problem. A physically constrained camera observes depth, a person mask, its own pose history, and an accumulated voxelized capture history, and a PPO policy selects both motion on a viewing sphere and motion of the sphere center. Training uses a proxy reward based on newly observed direction bins of persistent surface Gaussians. The authors construct a simulator from multi-view DNA-Rendering captures, using GauHuman avatars for policy training and per-frame MaskGaussian reconstructions for evaluation. On 24 held-out sequences from 10 subjects, the learned policy is evaluated under in-place and synthetically translated “walking” conditions with both 3DGS-Avatar and template-free 4DGS. It consistently outperforms fixed, random-walk, orbit, and spiral trajectories in the main table, and the paper additionally studies capture distance, observation noise, cross-dataset transfer, fixed-camera arrays, greedy planners, and design ablations.

Critical strengths

First, the paper defines a meaningful and underexplored problem at the intersection of active vision, camera control, and 4D human reconstruction. The policy uses deployable observation types rather than ground-truth body pose, and evaluation is based on the quality of reconstructions produced from the selected images rather than only the training reward. This makes the task formulation consequential and potentially useful.

Second, the evaluation is broad within the simulator. The authors use subject-disjoint data, three policy seeds, two substantially different reconstruction back ends, several capture distances, observation-noise tests, cross-dataset transfer, fixed-camera comparisons, and design ablations. The learned policy also beats the preset trajectories consistently in Table 1, especially when the unmodified presets lose the moving person.

Critical weaknesses

First, the central real-world claim is not validated. All policy inputs, captured images, and test references are rendered from fitted studio-capture assets; no real camera or drone is run in closed loop. The “walking” condition is created by rigidly translating an in-place performance along a synthetic path, so it omits natural coupling between locomotion, heading changes, articulation, clothing dynamics, background, etc. The motion model further assumes a fixed-radius viewing sphere and look-at orientation, with physical feasibility represented mainly by a per-step displacement bound. Consequently, the experiments establish performance in a clean simulator, not the claimed autonomous physical capture setting.

The computational feasibility of the onboard perception loop is also unverified. In simulation, human masks are obtained by thresholding rendered alpha, whereas deployment would require an actual segmentation model, for which the paper cites Segment Anything (Sec. 3.3, p. 5). The reported approximately 8 ms depth/mask-to-command runtime is measured on an NVIDIA RTX 6000 Ada workstation GPU and explicitly excludes real-camera segmentation and depth sensing (App. A.1, p. 15); it therefore does not establish end-to-end performance on a power- and memory-constrained onboard platform. A lightweight or optimized segmenter might be sufficient, so this is not evidence that onboard deployment is impossible. Rather, the missing hardware-specific accuracy–latency evaluation leaves a central prerequisite for the practical claim untested.

Second, the evidence does not convincingly isolate the benefit of learned long-horizon active view selection. In the main walking comparison, the preset orbit and spiral remain centered on the person’s initial position and therefore contain no person in about 41% of frames. Once the same onboard position estimate is used to make the spiral follow the person, the gap shrinks from approximately 2.55 dB to only 0.27 dB PSNR and 0.003 LPIPS (Table A3), and this stronger comparison is reported only as means without confidence intervals or significance marks. No established learned active-view baseline is adapted. Moreover, the one- and three-step privileged planners achieve higher final proxy reward while producing worse reconstructions (Table A4/Fig. A5), showing that the reward is not monotonically aligned with the actual objective. Together, these results leave unclear how much is gained from the proposed long-horizon policy rather than from target tracking, the restricted action parameterization, or simulator-specific inductive bias.

Questions for authors
  1. Can the authors provide paired per-sequence results and confidence intervals for the comparison against orbit+tracker and spiral+tracker, preferably for both 3DGS-Avatar and 4DGS and with uncertainty clustered by subject? Would the +0.27 dB / −0.003 LPIPS margin over spiral+tracker remain statistically reliable across policy seeds?

  2. Do the authors have any closed-loop results using a real RGB-D camera, drone, or rail system? If not, can they evaluate on genuinely translating/turning human captures with realistic background, segmentation, depth, and dynamics errors rather than rigidly translating an in-place fitted avatar?

  3. Why do the privileged greedy planners obtain higher final directional-diversity reward but worse reconstruction quality? Please report per-sequence reward–quality correlations and clarify what prevents reward exploitation.

  4. Why are established active-view policies such as GenNBV/GLEAM or recent dynamic-GS next-best-view methods not adapted as quantitative baselines? If they are incompatible, please specify the exact incompatibility and compare against the strongest non-learning receding-horizon alternative under the same onboard observations.

  5. Which human-segmentation model, input resolution, numerical precision, and automatic prompting/tracking mechanism are intended for deployment? Can the authors report end-to-end sensor-to-command latency, including its variability, sustained update rate, peak memory, and power on a representative onboard device rather than a workstation GPU, and evaluate framing/reconstruction under the resulting delayed or dropped masks?

  6. Could the authors assess whether the policy’s gains persist with an additional recent human-specific Gaussian reconstruction method, such as Vid2Avatar-Pro (CVPR 2025), using identical learned and tracking-enhanced baseline trajectories, matched capture budgets, and held-out evaluation views? If adaptation is infeasible, please clarify the input or preprocessing incompatibilities.

7 comments
AuthorsOpenReview ↗

Response to Reviewer o3Gc on Question 2: real-world validation (Part 1/5)

Thank you for the careful and detailed review. We have uploaded a revised PDF in which new text is marked in blue, and we answer in five comments, in the order of the weaknesses in your review, each with the related questions.

Closed-loop results with a real camera (Q2, first part). We have no closed-loop result with a real camera, drone or rail system: all results are obtained in simulation. The introduction and the limitations state this, and the abstract of the revised PDF now states it as well. The paper contributes the task, a simulation and evaluation platform built from real captures, and a policy learned on it. We see these results as a step toward autonomous capture in the real world, and running the policy on a real platform is the next step.

Genuinely translating and turning captures (Q2, second part). DNA-Rendering is a diverse repository of real human performances, and our held-out sequences already include turning (the heading sweeps up to 175°) and large body motion. Since the walking condition moves a performance recorded in place along a path, the revised PDF additionally evaluates on seven DNA-Rendering sequences with more walking and turning: in six the person walks, and in one the person turns a full circle on the spot. They are evaluated in the same simulator with the motion as recorded, without added displacement and without retraining; they are used neither for training nor in the 24 held-out sequences. Our policy minus each preset in PSNR (dB):

Our policy minusMean of all 7 sequencesMean of the 6 walking sequencesThe sequence with a full turn on the spot
Orbit, 3DGS-Avatar+2.08+1.80+3.72
Spiral, 3DGS-Avatar+0.66+0.61+0.98
Orbit + our tracker, 3DGS-Avatar+1.70+1.35+3.82
Spiral + our tracker, 3DGS-Avatar+0.48+0.26+1.81
Orbit, 4DGS+1.01+0.91+1.59
Spiral, 4DGS+0.78+0.81+0.63
Orbit + our tracker, 4DGS+0.67+0.51+1.66
Spiral + our tracker, 4DGS+0.52+0.49+0.74

Our policy is ahead of all four presets with both methods, on six or seven of the seven sequences. The details are in the new App. E.3 of the revised PDF (Table A16, Lead of our policy on recordings of real walking and turning; Fig. A7).

Background, segmentation and depth errors (Q2, second part). The simulator renders no background. The policy does not see colour, so a background would reach it only through errors in the person mask and the depth map. The submission tests such errors in synthetic form, without retraining (Sec. 4.3; App. B.4, Table A5): noise on the depth map, the mask and the camera's own pose estimate, and missed detections. At the heaviest of four levels the reconstruction loses at most 0.41 dB (one training seed). Dropped masks are in our next comment.

Motion model (fixed radius, look-at orientation, per-step bound). The fixed capture distance and the look-at orientation are design choices that keep the task simple and let every strategy capture from the same distance; Table A10 ablates the look-at constraint, and letting the policy also turn the camera by up to ±30° lowers PSNR by 0.68 dB in place and 0.62 dB walking. The per-step limit is derived from the thrust-to-weight ratio of a consumer multirotor (App. A.2); as you note, it stands in for flight dynamics rather than simulating them, and the limitations of the revised PDF now state this. A first test in this direction is in our next comment: how the policy tolerates commands that are executed late.

AuthorsOpenReview ↗

Response to Reviewer o3Gc on Question 5: onboard perception loop (Part 2/5)

Delayed or dropped masks (Q5, last part). The revised PDF adds this test, in simulation and without retraining. The policy issues one command per control step, every 133 ms (two frames of the 15 fps recordings). With a delay of kk steps, the camera executes at each step the command computed kk steps earlier; with a dropped mask, the person mask of that step does not reach the policy. A delay of one step, the worst case for a perception loop that takes up to 133 ms, lowers the PSNR by 0.19 dB in place and 0.10 dB when the person walks, and keeps the person in the image in every frame. When the person walks, dropping the mask at 10% of the steps has no measurable effect. The PSNR is measured with 3DGS-Avatar on 12 sequences that cover all ten subjects, with one training seed. The new App. E.4 and Table A17 of the revised PDF give the details and also report delays of two and three steps.

Intended deployment and onboard latency (Q5, first and second parts). The policy needs one 256 × 256 person mask and depth map per control step (Table A1). The mask can come from a person-segmentation network that runs in real time, which outputs the person mask directly, without prompts or tracking; Sec. 3.3 of the revised PDF now refers to a person-segmentation network instead of Segment Anything. For such a network (PeopleSemSegNet), NVIDIA's Isaac ROS benchmarks report 4.7 ms on a Jetson AGX Orin and 8.0 ms on a Jetson Orin Nano Super, far below the 133 ms control step. The end-to-end figures of the whole loop (latency, update rate, memory and power, at a given numerical precision) depend on the onboard device that is chosen. We will measure them when the policy is deployed on a real platform, which is the next step of this work; the limitations of the revised PDF now note this.

AuthorsOpenReview ↗

Response to Reviewer o3Gc on Questions 1 and 4: isolating the benefit of the learned policy (Part 3/5)

Learned policy versus target tracking. The task asks the camera to keep the person in view, to see the body from as many directions as possible, and to keep it inside the image at close range, with onboard observations only. Our policy learns all three from one reward, the gain in directional diversity. It is free to move the centre of its viewing sphere, and it learns to move it with a walking person although nothing in the reward refers to following. Following is thus one of the abilities the policy learns, and it is why its lead grows when the person walks: the plain presets capture no person in about 41% of their frames, and our policy in under 0.1%.

Comparison against orbit + tracker and spiral + tracker (Q1). These presets are our own construction rather than an existing method: a preset trajectory does not observe the person, so we let the centre of their path follow the onboard position estimate of our own system. The revised PDF now reports this comparison with intervals and per-sequence results for both reconstruction methods (new App. E.1 and Tables A13 and A14 of the revised PDF), and they are new rows of Table 1. Our policy minus each of them in the walking condition (PSNR in dB):

Our policy minusMean95% interval, clustered by subjectSeeds with interval above zeroSequences ahead
Orbit + our tracker, 3DGS-Avatar+2.17[+1.30, +3.28]3 of 318/24
Spiral + our tracker, 3DGS-Avatar+0.27[+0.10, +0.45]2 of 315/24
Orbit + our tracker, 4DGS+1.13[+0.57, +1.81]3 of 316/24
Spiral + our tracker, 4DGS+0.71[+0.45, +1.03]3 of 319/24

The clustered interval covers the three training seeds and resamples the 10 subjects with all their sequences kept together; it lies above zero in all four comparisons. Table A13 gives the interval of each seed separately. In three of the four comparisons it lies above zero for all three seeds. In the fourth (spiral + our tracker, 3DGS-Avatar) it lies above zero for two seeds, and for the third seed it is [−0.02, +0.28]. App. E.1 reports the same check for LPIPS. In place the person performs the recorded motion without travelling, so there is nothing to follow, and a gain comes from choosing better viewpoints. There our policy leads the plain spiral by 0.27 dB (3DGS-Avatar) and 0.58 dB (4DGS). When the person walks and the spiral follows with our tracker, the lead is of similar size (0.27 and 0.71 dB).

GenNBV, GLEAM and dynamic-scene methods as baselines (Q4, first part). GenNBV and GLEAM scan static objects and explore static indoor scenes with a coverage reward, with no moving person to keep in view, so they cannot be run on our task as they are. The closest we can come within our task is to train our policy with a coverage reward of their kind, which is ablation (a) of the paper: it is 0.58 dB (in place) and 0.78 dB (walking) below our reward (Table A10). The recent view-selection methods for dynamic scenes (Sec. 2) select frames offline from pre-recorded multi-camera video, reconstruct a static scene while treating pedestrians as a disturbance, or plan views for a moving rigid object; none controls a camera around a moving person.

Non-learning planners under the same onboard observations (Q4, second part). The revised PDF adds three such planners (new rows of Table A4 and App. B.3 of the revised PDF). Like the paper's privileged planners they replan at every step and maximise our reward one or three steps ahead (a receding horizon), but they use only the onboard detection and depth maps; one of them also counts the space not observed yet, as classical next-best-view planning does. They are 1.2 to 3.0 dB behind our policy with both reconstruction methods, and the strongest, the one that counts unobserved space, is 1.2 to 1.7 dB behind. Unlike them, our policy is trained on the return of the whole capture.

Restricted action parameterisation and simulator-specific bias. Your review also asks whether the gain could come from the restricted action parameterisation or from a bias of the simulator. The action parameterisation is shared by every strategy: the random walk and the planners use the same capture distance and the same per-step limit, and the random walk is more than 2 dB behind our policy (Table 1). As for the simulator, the images given to the reconstruction come from per-frame reference models rather than from the avatars the policy trains on, and the policy transfers without retraining to a second dataset, MVHumanNet, with leads of similar size (Table A9).

Reviewer_AnvwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AvatarPilot, an RL-based framework for active 4D human avatar reconstruction. It learns a camera policy using PPO to adaptively select viewpoints based on capture history, guided by a directional coverage reward over Gaussian primitives. Extensive experiments demonstrate improved reconstruction quality over predefined camera trajectories and generalization to unseen subjects.

Critical strengths
  1. Novel problem formulation. The paper introduces active viewpoint planning for 4D human avatar reconstruction, addressing a practical problem with a clear motivation.

  2. Comprehensive experiments. The paper provides extensive comparisons, ablation studies, and cross-dataset evaluations. Results across different reconstruction methods support the effectiveness and generalizability of the proposed approach.

Critical weaknesses
  1. Limited evaluation with frontier avatar reconstruction models. The proposed camera policy is evaluated using relatively early reconstruction methods (3DGS-Avatar and 4DGS). Recent feed-forward human reconstruction models, such as LHM++ and Forge4D, have demonstrated stronger reconstruction capabilities with multi-view inputs. Furthermore, FastAvatar (https://arxiv.org/pdf/2508.19754) demonstrates that feed-forward models can achieve high-quality novel-view synthesis from a limited set of fixed input viewpoints, without requiring active viewpoint selection. The authors didn't sufficiently justify the necessity and practical benefits of active view planning compared with frontier feed-forward reconstruction models.

  2. Unclear necessity of long-horizon planning. The learned policy mainly exhibits target following and elevation sweeping, which could potentially be achieved through simpler adaptive strategies. In Table A3, adding tracking to Spiral reduces the PSNR gap to only 0.27 dB. The experiments do not sufficiently isolate the contribution of long-horizon RL planning beyond these behaviors.

  3. Limited real-world validation. All experiments are conducted in simulation, and walking scenarios are generated by translating in-place human motion sequences. The simplified camera motion constraints and simulated observations may not fully reflect real-world capture conditions. Adding some in-the-wild testing would make the proposed method more convincing.

Questions for authors
  1. Since both training and evaluation are conducted in simulation, do the authors anticipate sim-to-real gap when deploying the learned policy in the real world situation? What would be the main challenges in transferring the policy to real-world scenarios?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Anvw on the first weakness: evaluation with frontier avatar reconstruction models (Part 1/3)

Thank you for your constructive review. We have uploaded a revised PDF in which new text is marked in blue. The paper evaluates with two public methods with opposite assumptions, one with a body template and one without, to measure how the captured views affect the reconstruction. Following your suggestion, the revised PDF adds two more recent methods: the feed-forward model LHM++ and FlexiAvatar (new App. E.5, Tables A18 and A19 and Fig. A8 of the revised PDF). In short, LHM++ indeed depends little on the capture, but it reproduces the recorded performance less closely than a reconstruction fitted to the sequence, for which the capture matters.

Evaluation with a frontier feed-forward model. On the 24 MVHumanNet sequences of our transfer experiment, in the in-place condition, LHM++, 3DGS-Avatar and 4DGS each reconstruct the person from the same three captures of 76 images: that of our policy (first training seed), that of the orbit and that of the spiral. LHM++ and 3DGS-Avatar are both animated with the dataset's SMPL-X parameters. The Gaussians that LHM++ exports are rendered with the same rasteriser and scored on the same test views with the paper's metrics, which are computed on a crop around the person (App. B.2). The captures are taken from 2.0 m, the main operating point of the paper, and part of the body can lie outside an image. We use MVHumanNet because the released training configuration of LHM++ lists DNA-Rendering and does not list MVHumanNet. PSNR and LPIPS of each method from each capture:

ReconstructionPSNR (dB), orbitPSNR (dB), spiralPSNR (dB), our policyLPIPS, orbitLPIPS, spiralLPIPS, our policy
LHM++ (feed-forward)13.6313.5213.690.2550.2560.253
3DGS-Avatar (optimised per sequence)18.1820.0220.370.1880.1770.172
4DGS (optimised per sequence)15.5516.3016.760.2920.2560.252

Necessity of active view planning compared with feed-forward models. LHM++ indeed hardly depends on the capture: its PSNR differs by 0.2 dB across the three captures. From the same images, however, it reproduces the recorded performance less closely than the methods optimised on the sequence: from our capture, 3DGS-Avatar is 6.7 dB above it and closer in both metrics on all 24 sequences, and 4DGS is 3.1 dB above it in PSNR and level with it in LPIPS. Its renderings are free of artefacts but deviate from the recorded person: a large part of its error lies along the silhouette, where the animated avatar does not line up with the person (Fig. A8). Even on the pixels that show the person in both the reference and the reconstruction (12 of the 48 test views), LHM++ reaches 15.66 dB against 18.39 dB for 3DGS-Avatar (App. E.5). The LHM++ paper reports the same trend: with 64 input views the per-subject optimisation method ExAvatar overtakes it in PSNR, though not in LPIPS (its Table 2).

On these sequences, a faithful reproduction of the recorded performance thus still requires a reconstruction fitted to its images. Applications such as the free-viewpoint replay of a dance or a sports action, or a digital double of an actor in a specific costume, need such a reproduction. For such a reconstruction the capture matters: across the three captures, the PSNR of 3DGS-Avatar and of 4DGS varies by 2.2 and 1.2 dB, and both are highest with the capture of our policy. The two are thus complementary: a feed-forward model when only a few images are available, and a planned capture with a fitted reconstruction when the performance itself has to be reproduced.

A more recent avatar reconstruction method. We also add FlexiAvatar (ECCV 2026), a Gaussian avatar method that builds on the mesh–Gaussian representation of ExAvatar. On 12 of the 24 held-out DNA-Rendering sequences, in the in-place condition, the lead over the orbit persists: FlexiAvatar reconstructs 1.15 dB better from the captures of our policy (first training seed), on 11 of the 12 sequences. Against the spiral, it reconstructs 0.29 dB better from our captures, on 10 of the 12. FlexiAvatar is run without its generated auxiliary views and adapted to our data (App. E.5).

Forge4D and FastAvatar. Forge4D takes sparse-view videos of the performance from several cameras (it is trained with four), which a single moving camera does not provide. FastAvatar reconstructs head avatars, whereas our captures are of the full body. We therefore evaluated LHM++, which takes one or a few pose-free images of the person.

AuthorsOpenReview ↗

Response to Reviewer Anvw on the second weakness: necessity of long-horizon planning (Part 2/3)

Target following and elevation sweeping. Following the person and sweeping the elevation are indeed what the policy visibly does, and they are what the task requires. The task asks the camera to keep the person in view, to see the body from as many directions as possible, and to keep it inside the image at close range, with onboard observations only. Our policy learns all three from one reward, the gain in directional diversity. This reward counts from how many directions each part of the body surface has been seen and does not prescribe a path. The policy is free to move the centre of its viewing sphere, and it learns to move it with a walking person although nothing in the reward refers to following. Both behaviours are thus learned rather than designed by hand. Following is also why the lead of our policy is larger when the person walks: the presets that do not follow capture no person in about 41% of their frames, and our policy in under 0.1%.

Simpler adaptive strategies. A hand-designed strategy with exactly these two behaviours is the spiral with our tracker. It is our own construction rather than an existing method: a preset trajectory does not observe the person, so we let the centre of the spiral follow the onboard position estimate of our own system. The 0.27 dB you cite is its result with 3DGS-Avatar, the only method for which the submission reported this comparison. The revised PDF reports both reconstruction methods, in Table 1 and with intervals and per-sequence results in the new App. E.1 (Tables A13 and A14) of the revised PDF. Our policy minus each preset when the person walks (PSNR in dB; the 95% interval covers the three training seeds and resamples the 10 subjects):

Our policy minus3DGS-Avatar, mean3DGS-Avatar, 95% interval4DGS, mean4DGS, 95% interval
Spiral + our tracker (follows the person, sweeps the elevation)+0.27[+0.10, +0.45]+0.71[+0.45, +1.03]
Orbit + our tracker (follows the person)+2.17[+1.30, +3.28]+1.13[+0.57, +1.81]

The learned policy is ahead of both hand-designed strategies with both methods, and the four intervals lie above zero.

Contribution of long-horizon planning beyond these behaviours. Beyond following and elevation sweeping, what the learned policy contributes is a camera path optimised over the whole capture, rather than fixed in advance or chosen step by step. Against a path fixed in advance, its benefit is the lead in the table above. Our policy also leads in the in-place condition. There the person performs the recorded motion without travelling, so there is nothing to follow, and the spiral already sweeps the elevation; our policy still leads the spiral by 0.27 dB (3DGS-Avatar) and 0.58 dB (4DGS) (Table 1). Against step-by-step choice, the revised PDF adds planners that use the same onboard observations and the same reward as the policy but choose each move one or three steps ahead, without learning (new rows of Table A4 and App. B.3 of the revised PDF): they are 1.2 to 3.0 dB behind our policy.

AuthorsOpenReview ↗

Response to Reviewer Anvw on the third weakness and the question: real-world validation and the sim-to-real gap (Part 3/3)

Sim-to-real gap and main challenges (Question). Yes, we expect a gap. The table lists the main challenges we see in transferring the policy, and what our simulation covers for each:

Main challengeIn our simulation
Person masks and depth from real sensors against real backgroundsNoise on depth, mask and camera pose, and missed detections: at most 0.41 dB at the heaviest of four levels (Sec. 4.3, Table A5). Mask dropped at 10% of the steps when the person walks: no measurable effect (new Table A17)
Latency of onboard perception and controlCommands delayed by one control step (133 ms): 0.1 to 0.2 dB, with the person in the image in every frame (new Table A17)
Camera motion beyond our simplified constraints (flight dynamics, wind)The per-step motion limit is derived from the thrust-to-weight ratio of a consumer multirotor and is the same for every strategy (App. A.2)

These tests are run without retraining; the noise, delay and mask tests use one training seed. On the policy's side, two facts limit the gap. First, it observes only a depth map and a person mask, not colour, so appearance and background can reach it only through errors in these two. Second, it transfers without retraining to a second studio dataset, with other cameras and people, with leads of similar size (Table A9).

Walking scenarios and real walking. DNA-Rendering is a diverse repository of real human performances, and our held-out sequences already include turning and large body motion. Since the walking condition moves a performance recorded in place along a path, the revised PDF additionally evaluates on seven DNA-Rendering sequences with more walking and turning (six with walking, one with a full turn on the spot), in the same simulator with the motion as recorded, without added displacement and without retraining. They are used neither for training nor in the 24 held-out sequences. Our policy is ahead of all four presets (orbit and spiral, plain and with our tracker) with both reconstruction methods, on six or seven of the seven sequences; with 3DGS-Avatar it leads the orbit and the spiral by 2.08 and 0.66 dB on average (new App. E.3, Table A16 and Fig. A7 of the revised PDF).

In-the-wild testing. We see testing in the wild, with the policy running on a real platform, as the next step of this work. All results in the paper are obtained in simulation, as the introduction, the limitations and now the abstract state.

We hope these additions address your concerns, and we are happy to clarify anything further.

Reviewer_mDmcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AvatarPilot, a reinforcement learning-based active camera control method for dynamic 4D human avatar reconstruction. The authors construct a Gaussian avatar simulation environment from real multi-view human capture data and train a PPO policy using a surface viewing-direction diversity reward. The policy selects camera viewpoints based on observation history and the current state of the person.

The experiments evaluate different capture strategies using two reconstruction methods, 3DGS-Avatar and 4DGS. The results show that AvatarPilot outperforms predefined camera trajectories in both in-place and walking conditions, and achieves reconstruction quality comparable to that of 6–8 fixed cameras under certain settings.

Critical strengths

(1) This paper introduces an interesting and practically relevant task, extending active viewpoint planning to dynamic 4D human appearance reconstruction while considering human motion, observation history, and camera motion constraints.

(2) The authors develop a complete simulation, training, and evaluation framework based on real multi-view human capture data, providing a valuable foundation for research on active 4D human reconstruction. The evaluation with different reconstruction methods further supports the effectiveness of the proposed approach.

Critical weaknesses

(1) Limited evaluation of complex human motion. The walking condition is mainly constructed by applying global translation to human motions recorded in place. The paper lacks a dedicated analysis of challenging motions such as body turning, changes in facing direction, and large articulated movements, which may better reveal the challenges of active viewpoint planning for dynamic humans.

(2) Limited consideration of human dynamics in the directional-diversity reward. The authors record viewing directions for Gaussians in the world coordinate frame. Although this accounts for pose-dependent visibility changes, it does not explicitly consider changes in local viewing directions caused by body rotation or articulation, which may affect how accurately viewing diversity is measured for dynamic humans.

Questions for authors

(1) Why do the authors measure viewing directions in the world coordinate frame rather than in a body-local or Gaussian-local coordinate frame? How might this choice affect viewpoint planning when the human body undergoes substantial rotation or articulation?

(2) Could the authors provide a more detailed analysis of the policy's performance under challenging motions, such as body turning, changes in facing direction, or large articulated movements, to better evaluate its ability to handle complex dynamic human scenarios?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mDmc on Weakness 1 and Question 2: analysis under challenging motions (Part 1/2)

Thank you for your constructive review and for your suggestions. We have uploaded a revised PDF in which new text is marked in blue. It adds a dedicated analysis by the person's own motion, for every capture strategy of Table 1 (new App. E.2), and an evaluation on recordings of real walking and full turns (new App. E.3).

Analysis under challenging motions (Weakness 1, Question 2). The 24 held-out sequences differ widely in how much the person turns: over the 10 s, the torso heading sweeps a median of 39° and up to 175°. We rank the sequences by this range and split them into three groups of eight: calm (the heading sweeps 5–27°), moderate (33–54°) and vigorous (65–175°). The table gives the lead of our policy over each of the other strategies in the three groups (PSNR in dB, with 3DGS-Avatar):

Our policy minusIn place, calmIn place, moderateIn place, vigorousWalking, calmWalking, moderateWalking, vigorous
Fixed camera+9.60+7.10+4.06+9.07+6.58+4.95
Random walk+2.74+2.46+1.87+5.39+4.38+3.37
Orbit+4.44+2.20+0.26+6.00+3.90+1.84
Spiral+0.50+0.25+0.05+3.30+2.49+1.84
Orbit + our tracker+4.36+2.09+0.16+4.37+2.02+0.12
Spiral + our tracker+0.73+0.24+0.05+0.63+0.16+0.02
  • Calm and moderate groups. Our policy is ahead of every strategy on average.
  • Vigorous group, in place. It still leads the fixed camera and the random walk by more than 1.8 dB, and it matches the four presets (orbit and spiral, plain and with our tracker) within 0.3 dB.
  • Vigorous group, walking. It leads every strategy by more than 1.8 dB except the two presets with our tracker, which it matches.

The sequences also differ widely in articulated motion: relative to the torso, the limbs move 15 times more in the most active sequence than in the calmest one. When we group the sequences by this limb motion instead of by turning, the result is similar: our policy is ahead of every strategy in the calm and moderate groups, and it leads or matches every strategy in the vigorous group. Our policy also keeps the person in view under vigorous motion: in all three groups it loses the person in less than 0.5% of the frames.

The walking condition and real walking (Weakness 1). Since the walking condition moves a person recorded in place along a path, we additionally evaluate on seven DNA-Rendering sequences in which the person walks or turns a full circle, as recorded, in the same simulator and without retraining. In six of them the person walks and in one the person turns a full circle on the spot; in five the heading sweeps more than 300°. They are used neither for training nor in the 24 held-out sequences. Our policy is ahead of all four presets with both reconstruction methods, on six or seven of the seven sequences. With 3DGS-Avatar it leads the orbit and the spiral by 2.08 and 0.66 dB on average, and on the full turn on the spot it leads all four presets, by 1.0 to 3.8 dB.

The new Table A15 of the revised PDF (Lead of our policy by motion level) gives the analysis for both reconstruction methods, and the new Table A16 and Fig. A7 the recordings of real walking and turning.

AuthorsOpenReview ↗

Response to Reviewer mDmc on Weakness 2 and Question 1: frame of the viewing directions (Part 2/2)

Why the world coordinate frame (Question 1, first part). In Eq. (1), btb_t is the bin of the camera's azimuth and elevation about the reference point, in world axes. As you point out, this frame does not turn with the body or its limbs. This is a design choice, not a constraint: the reward is computed in simulation and could use the body pose. We chose the world frame because it is the simplest, and it is also the frame of the policy's observation: each cell of its onboard map counts the same bins. We test what this choice costs in two ways; the first is reported in the new App. D.4 and Table A12 of the revised PDF.

How accurately viewing diversity is measured (Weakness 2). Without retraining, we recomputed the directional diversity of the existing captures on the surface of the body model in three frames: world axes as in our reward, body-local (relative to the torso heading) and Gaussian-local (each surface element's own frame). The table gives the Spearman correlation between this diversity and PSNR across the capture strategies without privileged information (median over the 24 held-out sequences):

Directions recorded3DGS-Avatar, in place3DGS-Avatar, walking4DGS, in place4DGS, walking
World axes (ours)0.8130.8600.6570.685
Body-local0.8070.8570.6650.694
Gaussian-local0.8500.8720.7650.837

The body-local frame follows PSNR as closely as ours: the two differ by less than 0.01. The Gaussian-local frame follows PSNR more closely, slightly with 3DGS-Avatar and more clearly with 4DGS.

Effect on viewpoint planning (Question 1, second part). In an earlier version of our system (capture distance 3.0 m, in-place condition), we rotated the recorded person on the spot, by about one and a half turns per capture on average, just as the walking condition moves the person along a path. There we trained policies that differ only in this frame, body-local or world, with two training seeds each: the body-local ones score 0.10 dB higher in PSNR, with a 95% interval of [−0.02, +0.22] that includes zero, and the two fly alike, so the frame made no measurable difference.

In short, taking the body's rotation into account changes little in both tests. The Gaussian-local frame, which also accounts for articulation, follows PSNR more closely, so a reward in that frame is a natural refinement that we leave to future work. We are happy to clarify anything further.

Thanks for the detailed response and additional experiments. The authors have adequately addressed my concerns. I acknowledge that the proposed method can adapt to various motion conditions and achieve performance improvements. The coordinate-frame analysis also provides a reasonable justification for the current design, although the potential benefits of Gaussian-local viewing directions for policy learning remain to be explored. Overall, I maintain my positive assessment of the paper and will keep my original rating.

Thank you for reading our responses and for your positive assessment of the paper. We are glad that the revision addresses your concerns, and we are happy to clarify anything further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revised PDF and the additional experiments

We thank the reviewers for their careful and constructive comments.

Strengths noted by the reviewers.

  • The task. The reviewers describe it as "an interesting and practically relevant task" (mDmc), "a meaningful and underexplored problem" (o3Gc) and "a practical problem with a clear motivation" (Anvw).
  • The platform and the policy. Reviewer mDmc notes "a complete simulation, training, and evaluation framework based on real multi-view human capture data, providing a valuable foundation for research on active 4D human reconstruction". Reviewer o3Gc notes that the policy "uses deployable observation types rather than ground-truth body pose" and that "evaluation is based on the quality of reconstructions produced from the selected images rather than only the training reward".
  • The evaluation. It is "broad within the simulator" (o3Gc), with "extensive comparisons, ablation studies, and cross-dataset evaluations" (Anvw).
  • The presentation. All three reviewers rate the paper as clear and easy to follow.

We have uploaded a revised PDF in which new text is marked in blue, and we reply to each reviewer in a separate thread. The rest of this comment summarises the revision.

Additional experiments and analyses. They are in the appendix of the revised PDF, apart from two new rows of Table 1. The table, figure and section numbers of the submission are unchanged.

  1. Presets that follow the person (o3Gc, Anvw). Following the person is part of the task, and our policy learns it from a reward that does not refer to following. The orbit and the spiral with our own tracker, which uses the onboard position estimate of our system, are new rows of Table 1 for both reconstruction methods. When the person walks, our policy still leads the spiral with our tracker by 0.27 dB (3DGS-Avatar) and 0.71 dB (4DGS), and the orbit with our tracker by 2.17 and 1.13 dB (intervals and per-sequence results in App. E.1, Tables A13 and A14).
  2. Non-learning planners with the same onboard observations (o3Gc, Anvw). Planners that maximise our reward one or three steps ahead from onboard observations are 1.2 to 3.0 dB behind our policy (Table A4, App. B.3).
  3. Further reconstruction methods (Anvw, o3Gc). In our evaluation, the feed-forward model LHM++ hardly depends on the capture, but it reproduces the recorded performance less closely than 3DGS-Avatar and 4DGS, the two reconstruction methods used in the paper. Both are optimised on each sequence, and for them the capture matters. With the more recent FlexiAvatar (ECCV 2026), the lead of our policy over the orbit persists (App. E.5, Tables A18 and A19, Fig. A8).
  4. Recordings of real walking and turning (all three reviewers). DNA-Rendering is a diverse repository of real human performances, and our held-out sequences already include turning (the heading sweeps up to 175°) and large body motion. Since the walking condition moves a performance recorded in place along a path, we additionally evaluate on seven DNA-Rendering sequences in which the person walks (six sequences) or turns a full circle on the spot (one), in the same simulator with the motion as recorded and without retraining. Our policy is ahead of the orbit and the spiral, plain and with our tracker, with both reconstruction methods (App. E.3, Table A16, Fig. A7).
  5. Analysis by the person's own motion (mDmc). With the held-out sequences split into three groups by how much the person turns, our policy is ahead of every other capture strategy on average in the two groups that turn less, and it leads or matches them in the group that turns most (App. E.2, Table A15).
  6. Delayed commands and dropped masks (o3Gc, Anvw). A delay of one control step (133 ms) lowers the PSNR by less than 0.2 dB, and, when the person walks, dropping the person mask at 10% of the steps has no measurable effect (App. E.4, Table A17).
  7. Reward and reconstruction quality (o3Gc). A higher reward goes with a better reconstruction on 20 to 24 of the 24 held-out sequences; the two privileged planners are the main exception, and we discuss why (App. D.2, Table A11).
  8. Frame of the viewing directions (mDmc). Recomputed in a body-local frame, the directional diversity follows PSNR as closely as in world axes, and in a Gaussian-local frame more closely (App. D.4, Table A12).

Changes in the main text. The abstract now states that the experiments are in simulation. Table 1 has the two new rows, and Secs. 4.1 to 4.3 report the new comparisons in a few sentences and point to the appendix. The related work covers feed-forward human reconstruction, Sec. 3.3 now refers to a person-segmentation network, instead of Segment Anything, as the source of the person mask, and the limitations are extended.

AuthorsOpenReview ↗

Summary of the discussion

We thank the three reviewers for their engagement in the discussion. This comment summarises its outcome.

The paper poses the task of active 4D human avatar reconstruction, builds a simulation and evaluation platform from real multi-view captures, and learns a camera policy that acts on onboard observations alone. The reviews describe the task as "a meaningful and underexplored problem" (o3Gc) and the platform as "a valuable foundation for research on active 4D human reconstruction" (mDmc).

The reviewers' assessments after the discussion.

  • Reviewer mDmc considers the concerns addressed and maintains a positive assessment ("The authors have adequately addressed my concerns. [...] Overall, I maintain my positive assessment of the paper").
  • Reviewer Anvw raised the rating after the rebuttal ("Since most of my concerns have been satisfactorily addressed, I am happy to raise my score").
  • Reviewer o3Gc considers most questions answered, finds the work complete and meaningful, and gives a positive overall assessment. The rating is kept for a single remaining point ("I recognize this work as complete and meaningful, and my overall assessment is positive, my score reflects the missing real platform validation rather than a negative view of the work as a whole").

What the revision changed. The revised PDF answers the main concerns of the reviews with additional experiments (listed in our comment "Summary of the revised PDF and the additional experiments"):

  • Benefit of the learned policy beyond following the person (o3Gc, Anvw). Our policy stays ahead of presets that are given our own tracker, now with intervals and per-sequence results, and of planners restricted to onboard observations.
  • Challenging and real motion (mDmc, o3Gc, Anvw). Every capture strategy is analysed by the person's own motion. On seven added recordings of real walking and turning, our policy is ahead of all four presets.
  • Other reconstruction methods (Anvw, o3Gc). The feed-forward LHM++ hardly depends on the capture, but it reproduces the recorded performance less closely than the methods optimised on each sequence. For these methods the capture matters, so our active capture remains important: with the more recent FlexiAvatar, our policy is ahead of both the orbit and the spiral.
  • Tolerance to a real perception loop (o3Gc, Anvw). A delay of one control step (133 ms) lowers the reconstruction quality only slightly, and dropped masks have no measurable effect when the person walks.

The single remaining point is validation on a real platform (o3Gc). All results of the paper are obtained in simulation. The introduction and the limitations state this, and the abstract of the revised PDF now states it as well. Instead of closed-loop results on hardware, the revised PDF covers, one element at a time, the alternative that the reviewer's Question 2 suggested: real walking and turning, segmentation and depth errors, and delays, each in simulation. Running the policy on a real platform is the next stage of this work.

We hope this summary is helpful, and we thank the reviewers again for their constructive comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.