- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an adaptive loss-weighting scheme for hybrid mesh–Gaussian head avatars (MeGA representation). Offline, from checkpoints sampled in the first 10% of training, it applies a one-step update restricted to geometry parameters, to appearance parameters, or to both, measures per-term normalized loss improvements with a penalty for degradation of geometric terms, and converts the three utilities into a geometry–appearance preference (softmax of Qg, Qa) and a residual joint advantage ΔQj = Qj − max(Qg, Qa). A small MLP controller (30-dimensional state, hidden size 64) is trained to predict these quantities from the training state and is then used online, after 20K iterations, to scale geometry-, appearance- and consistency-oriented loss groups within [0.9, 1.1]. Experiments on eight NeRSemble subjects report small average improvements over Fresco and MeGA, plus single-subject ablations of the modulation strategy and of the route-evidence factorization.
Critical strengths
-
The question is real and the approach is reasonable in outline. Geometry, photometric and consistency objectives interact strongly in hybrid avatar training, and deciding their relative emphasis per state from post-update outcomes rather than from loss magnitudes is a sensible idea. Amortizing the decision into a controller leaves inference untouched (57 ms vs 55 ms per frame).
-
The controls are thoughtfully chosen. Fixed-GC isolates the new cross-view consistency loss, Mean Mod. tests whether a static re-weighting explains the gain, the factorization is compared against GA-only, Joint-only and Parallel-Route variants, and offline cost is reported (1 h 46 min of label generation and 3 min of controller training per subject).
Critical weaknesses
-
With a single-step lookahead, the "counterfactual" evidence is, to first order, closely related to the gradient statistics the paper positions itself against, and the residual joint advantage may largely reflect additivity. For one step of any diagonally preconditioned optimizer restricted to a parameter block S, the decrease of the training-batch loss is ≈ η⟨∇_S L, P∇_S L⟩, a preconditioned squared gradient norm of that block. Because Sg and Sa are disjoint, Qj ≈ Qg + Qa to first order, hence ΔQj = Qj − max(Qg, Qa) ≈ min(Qg, Qa) ≥ 0 whenever both routes improve, regardless of any interaction between geometry and appearance; the per-term percentile normalization and the degradation penalty in Eqs. 7–9 do not change this qualitatively. Consequently (a) the contrast drawn in Secs. 1 and 2.3 with gradient- and loss-based weighting methods may be stronger than the m = 1 setting supports; (b) it would help to show that ΔQj is not simply ≈ min(Qg, Qa) (a scatter plot from the logged labels, or labels regenerated with m ≫ 1); and (c) specifying the 30-dimensional state descriptor is important, since if it contains loss values or gradient norms the controller is effectively a learned function of those statistics. Independently of this, the offline intervention (restricting the updated parameter subset, Eq. 6) and the online action (re-weighting loss groups by at most ±10%, Eqs. 15–17) are different operations; an experiment showing that the former predicts the benefit of the latter would considerably strengthen the central mechanism.
-
The empirical evidence is not yet sufficient to establish the claims, and some effect sizes are hard to reconcile with each other. Average gains over Fresco are 0.10 dB (novel view) and 0.19 dB (self-reenactment) with LPIPS tied (Table 1), over eight subjects, with no seeds or per-subject numbers; the abstract's "consistently outperforming" may be stronger than this margin supports. Both ablations use a single subject (264 and 460), and their effect sizes are large relative to the modulation range: on subject 264 a static re-weighting within ±10% (Mean Mod.) yields +0.35 dB and its temporal variation a further +0.38 / +0.79 dB (Table 2); on subject 460 the factorization alone is worth +1.25 dB over GA-only (Table 4), more than the method's entire average gain over MeGA (+1.11 dB, Table 1). Multi-seed, all-subject ablations would be needed to separate these differences from the run-to-run variance of Gaussian avatar training. Part of the gain over the MeGA baseline (Baseline→Fixed-GC, +0.62 / +0.44 dB) comes from the group-consistency loss, which is not among the listed contributions. The controller is fitted only on states from the first 10% of training yet applied from iteration 20K onward (Appendix A); since the total iteration count is not given, it is unclear how far the controller operates outside its training regime, and a prediction-accuracy-over-time analysis would help. Mean Mod. rules out one static schedule; a progress-only (time-dependent) schedule and standard adaptive weighting baselines under equal compute would be natural additional controls, and it would help to state whether Table 3's 17 h includes the per-subject offline cost.
Questions for authors
-
Mechanism. From the logged offline labels, what is the relationship between ΔQj and min(Qg, Qa) (scatter/correlation)? Do the labels change qualitatively with m = 5–20 lookahead steps? Please list the 30 state features. Does the parameter-restricted route utility predict the benefit of the loss re-weighting actually used online? For example, generate labels by applying the (sg, sa, sj) perturbations for one step from the same state and compare them with the current labels, or report the correlation between the controller's action and the realized improvement on held-out views.
-
Controller transfer, cost and reproducibility. How many iterations does a subject's training take, what fraction is covered by the label states, and what is the controller's prediction error in early, middle and late training? Is the controller trained per subject, and does the 17 h in Table 3 include label generation and controller training? Please also specify the assignment of losses to Lg / La / Lc, the group-consistency loss, the percentile normalization, whether optimizer state and RNG are restored after each lookahead, and whether code will be released.
-
Robustness and controls. Please report per-subject results and standard deviations over at least three seeds for Tables 1, 2 and 4, extend both ablations to all eight subjects, and add equal-compute controls: a progress-only controller, GradNorm or uncertainty weighting, and a periodically re-run online lookahead.
2 comments
Responses to Reviewer FC1D (Part2)
6. Group consistency and route modulation
The original Fixed-GC control isolates the contribution of the consistency objective:
| Method | Novel PSNR | Novel SSIM | Novel LPIPS | Self PSNR | Self SSIM | Self LPIPS |
|---|---|---|---|---|---|---|
| Baseline | 30.36 | 0.952 | 0.066 | 29.60 | 0.948 | 0.067 |
| Fixed-GC | 30.98 | 0.954 | 0.060 | 30.04 | 0.950 | 0.064 |
| Route Mod. w/o GC | 31.42 | 0.953 | 0.058 | 30.44 | 0.946 | 0.065 |
| Ours | 31.71 | 0.956 | 0.057 | 30.98 | 0.954 | 0.060 |
Fixed-GC improves PSNR over Baseline by 0.62 dB on novel-view synthesis and 0.44 dB on self-reenactment. Route Mod. w/o GC further improves Baseline by 1.06 and 0.84 dB, respectively, and outperforms Fixed-GC on both tasks. Adding GC further improves PSNR from 31.42 to 31.71 dB and from 30.44 to 30.98 dB. Thus, route modulation contributes independently, while group consistency provides complementary gains.
7. Does the [0.9, 1.1] clipping make the controller too weak?
We use this conservative range to avoid excessively perturbing the original optimization dynamics. Although each step is bounded, the effect accumulates during training. On Subject 264, self-reenactment PSNR is 29.60 dB for Baseline, 30.19 dB for Mean Mod., 30.44 dB for Route Modulation w/o GC, and 30.98 dB for Full. Mean Mod. uses fixed average modulation, while Full adapts to each training state. The no-GC result also shows that bounded route modulation alone provides a clear gain. Thus, [0.9, 1.1] is conservative but still effective.
8. 30-dimensional state descriptor
The 30-dimensional descriptor contains four categories:
- current loss and regularization values, including photometric, geometry, and consistency-related terms;
- objective activation and weighting states;
- training-stage indicators;
- normalized optimization progress, including epoch, global step, and stage progress.
It contains no gradient norms or other gradient statistics. The controller predicts the route preference provided by offline counterfactual evaluation from the current training state, rather than applying a hand-designed gradient-based weighting rule. The complete 30-dimensional feature specification has been added to the revised supplementary material.
9. Per-subject robustness and component-level ablations
Our main comparison covers 8 NeRSemble subjects, while component ablations use representative subjects. This protocol is also used by several closely related person-specific head-avatar methods: GaussianAvatars uses Subject 304, TensorGA Subject 306, MeGA Subject 306, and Fresco Subject 253.
We follow the same protocol and have added complete per-subject comparisons with MeGA and Fresco to the revised supplementary material. The Fresco comparison is shown below.
Novel-view synthesis
| Subject | Fresco PSNR | SSIM | LPIPS | Ours PSNR | SSIM | LPIPS |
|---|---|---|---|---|---|---|
| 074 | 30.36 | 0.916 | 0.107 | 30.25 | 0.917 | 0.105 |
| 104 | 28.92 | 0.923 | 0.091 | 29.04 | 0.925 | 0.089 |
| 253 | 34.02 | 0.957 | 0.036 | 34.11 | 0.959 | 0.033 |
| 264 | 30.98 | 0.954 | 0.060 | 31.71 | 0.956 | 0.057 |
| 302 | 33.64 | 0.949 | 0.041 | 33.49 | 0.948 | 0.042 |
| 304 | 27.40 | 0.871 | 0.076 | 27.67 | 0.871 | 0.075 |
| 306 | 33.71 | 0.967 | 0.034 | 33.54 | 0.967 | 0.035 |
| 460 | 35.35 | 0.967 | 0.025 | 35.38 | 0.967 | 0.025 |
| Avg. | 31.80 | 0.938 | 0.058 | 31.90 | 0.939 | 0.058 |
Self-reenactment
| Subject | Fresco PSNR | SSIM | LPIPS | Ours PSNR | SSIM | LPIPS |
|---|---|---|---|---|---|---|
| 074 | 27.32 | 0.897 | 0.112 | 27.39 | 0.898 | 0.112 |
| 104 | 27.29 | 0.911 | 0.097 | 27.43 | 0.913 | 0.094 |
| 253 | 32.46 | 0.949 | 0.043 | 32.66 | 0.951 | 0.041 |
| 264 | 30.04 | 0.950 | 0.060 | 30.98 | 0.954 | 0.060 |
| 302 | 32.38 | 0.937 | 0.046 | 32.18 | 0.937 | 0.047 |
| 304 | 27.84 | 0.873 | 0.084 | 27.93 | 0.872 | 0.083 |
| 306 | 32.15 | 0.958 | 0.041 | 32.24 | 0.958 | 0.039 |
| 460 | 32.97 | 0.960 | 0.036 | 33.14 | 0.961 | 0.035 |
| Avg. | 30.30 | 0.929 | 0.064 | 30.49 | 0.931 | 0.063 |
Ours achieves higher PSNR than Fresco on 5/8 subjects for novel-view synthesis and 7/8 for self-reenactment, showing that the overall gain is not driven by a single identity.
10. Computational cost and reproducibility details
The controller is trained separately for each subject. Offline label generation takes about 1 h 46 min per subject and controller training about 3 min. The 17 h in Table 3 covers only full avatar optimization and excludes these offline costs. The revised supplementary material also provides objective grouping, percentile normalization, controller architecture, and online modulation settings. The controller remains fixed during full optimization, with no additional online lookahead.
Thank you again for the detailed analysis. These comments helped us clarify the interpretation of route evidence and strengthen the experimental validation.
Responses to Reviewer FC1D (Part1)
Thank you for the detailed and constructive comments. Following your suggestions, we added GradNorm, route modulation without group consistency, multi-step lookahead analysis, controller evaluation at later training stages, the relationship between ΔQj and min(Qg,Qa), and complete per-subject results. We also further clarified several implementation and reproducibility details in the revised manuscript.
1. Single-step route evidence and gradient-based weighting
We added GradNorm as a standard gradient-based adaptive weighting baseline under the same Subject 264 training setting as Ours. GradNorm replaces the route controller and adaptively balances the active geometry-, appearance-, and consistency-oriented objective groups throughout training, using its standard setting (α=1.5).
GradNorm provides only a limited improvement over the Baseline, while Ours achieves substantially higher PSNR on both tasks: 31.71 vs. 30.66 dB for novel-view synthesis and 30.98 vs. 29.76 dB for self-reenactment. This indicates that standard gradient-based adaptive weighting and our route-based optimization are not equivalent in practice.
Our method does not directly use loss magnitudes or gradient norms for weighting, but evaluates the realized changes of multiple objectives after restricted updates. We agree that, for a single local update, this evidence is theoretically related to gradient statistics. We have softened the original contrast and now describe route evidence as local optimization evidence based on realized post-update outcomes.
2. Relationship between ΔQj and min(Qg,Qa)
We analyzed all 5,600 offline labels of Subject 264. The Pearson and Spearman correlations between ΔQj and min(Qg,Qa) are 0.841 and 0.856, respectively, with R2=0.707.
However, among the 5,599 states where both Qg>0 and Qa>0, 1,360 states (24.29%) still have ΔQj<0. Therefore, the two quantities are strongly correlated but not equivalent.
We have accordingly revised the interpretation of ΔQj as the residual advantage or disadvantage of the joint route relative to the stronger individual route, rather than as an independent interaction signal.
3. Is single-step lookahead sufficient?
We further generated route evidence using m=5 and m=10 lookahead steps and compared them with the m=1 teacher used in our method:
The m=1 evidence is highly consistent with a moderate short-horizon rollout (m=5), supporting m=1 as a useful short-horizon approximation. As the horizon increases, the parameter trajectories of different routes gradually diverge, making the resulting utilities increasingly path-dependent. We therefore use m=1 as a practical local proxy while avoiding the additional forward/backward/update cost of longer rollouts.
4. Controller consistency at later training stages
We recomputed counterfactual evidence at the middle and late training stages and compared it with the controller predictions:
GA-Pref. Agreement measures whether the controller and the newly recomputed evidence prefer the same geometry or appearance route. Pearson measures the linear correlation between their preference scores, while Spearman measures the consistency of their ranking. These results show that the geometry–appearance preference learned from early trajectory states remains largely consistent with fresh counterfactual evidence during the middle and late stages where the controller is applied.
5. Offline restricted routes vs. online bounded modulation
The two stages serve different purposes. Offline parameter-restricted routes are used to generate discriminative route-level supervision, while the online controller converts the learned optimization preference into low-cost bounded modulation of the original training objectives. Therefore, we do not require the restricted-route utility and the bounded ±10% online re-weighting to be strictly equivalent at every state. Periodically re-running multiple explicit lookahead routes during online training would reintroduce additional forward/backward/update operations and reduce the computational advantage of the amortized controller. We have clarified this distinction in the revised manuscript.