acceptodds
Under review as a conference paper at ICLR 2027

Federated Learning of Large Novel View Synthesis Model for Better Personalized Neural Field

Abstract

Large image-conditioned novel view synthesis (NVS) models benefit from training across diverse scenes, but collecting multi-view images from many owners on a central server can be impractical due to privacy and data-ownership constraints. Keeping each scene local avoids raw-data sharing, yet personalized neural fields trained from sparse client views often suffer from insufficient geometric supervision. Moreover, sharing such neural fields is itself undesirable because their weights can constitute a directly renderable representation of the private scene. We propose a federated framework that couples a server-side large NVS model with personalized client neural fields that remain entirely local. Each client uses its neural field to generate nearby pseudo-target views and locally refine only a trainable subset of the server model using both captured and rendered supervision. Only the resulting parameter update is communicated, while raw images, pseudo-targets, and neural-field parameters remain local. The aggregated global model is then returned to clients to improve their neural fields through consistency supervision, creating a mutual-reinforcement loop between global generalization and local personalization. Experiments with 5,000 scene-level clients show substantial improvements in personalized client neural fields, while the server model achieves competitive generalized NVS performance under zero-shot, cross-domain, and non-IID evaluations. Empirical MIA/PIA and limited-context leakage evaluations further indicate lower privacy exposure than directly sharing client neural fields, without claiming formal differential privacy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Federated learning, Privacy-preserving, Novel view synthesis, Large model, Neural radiance field

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yLLWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses several challenges in large image-conditioned novel view synthesis (NVS) models. These include the reliance on centrally collected multi-view data, insufficient supervision for personalized neural fields under sparse local observations, and potential privacy risks from directly sharing neural field parameters. To address these challenges, the authors propose a federated mutual-reinforcement framework that connects a server-side image-conditioned NVS model with client-side private neural fields. Each client uses its local neural field to render pseudo-targets from nearby viewpoints. These pseudo-targets, together with captured images, are used to update a trainable subset of the server model's parameters. The server aggregates the client updates. In turn, predictions from the server model guide local neural field optimization through consistency supervision. This creates a mutual-reinforcement loop between global generalization and local personalization. Raw images, pseudo-targets, and neural field parameters remain on the clients. The paper trains the framework using 75,000 public scenes and 5,000 Objaverse scene-level clients. It evaluates the method on Objaverse, zero-shot GSO, and non-IID IBRNet-LLFF settings. The experimental results show that the server model outperforms the centralized reduced LVSM baseline with the same architecture and parameter count across all three settings. The client-side neural fields achieve the best PSNR, SSIM, and LPIPS among the compared methods on Objaverse and IBRNet-LLFF. Ablation studies support the effectiveness of pseudo-target supervision and bidirectional guidance. Privacy attack experiments also show that uploading updates to a subset of model parameters results in lower measured information exposure than directly exposing client-side neural fields.

Critical strengths
  1. Clear problem formulation and a reasonable framework connecting global generalization with local personalization. The paper addresses the limitations of centrally collecting multi-view data and the lack of supervision for personalized neural fields under sparse observations. The authors propose a federated mutual reinforcement framework that connects a server-side image-conditioned NVS model with client-side private neural fields. This allows cross-scene knowledge to guide local reconstruction while keeping client scene representations private. This design has clear practical value and represents a core contribution of the paper.
  2. The supervision interface from neural fields to NVS has a reasonable geometric basis. Clients use their neural fields to generate nearby pseudo-targets. This converts local 3D information into image-based supervision for the server model. In turn, server predictions guide client optimization through consistency supervision. Unlike independent 2D augmentation, these pseudo-targets share the geometry, visibility, and appearance modeled by the neural fields. The paper also analyzes the trade-off between pose coverage and teacher bias. This provides a reasonable explanation for the benefits of pseudo-targets and the limitations of their reliability.
  3. The experiments evaluate both server generalization and client personalization, with results largely consistent with the proposed mechanisms. The server model outperforms the centralized reduced LVSM baseline with the same architecture and parameter count across Objaverse, zero-shot GSO, and non-IID IBRNet-LLFF settings. The client-side neural fields achieve the best reconstruction metrics among the compared methods on both evaluation datasets. Ablation studies show that removing either pseudo-target supervision or consistency supervision affects performance on both sides. Together, these results support the practical effectiveness of bidirectional knowledge transfer.
Critical weaknesses
  1. The attribution of the main performance gains remains unclear due to insufficient comparability in data and computational budgets. The main experiments combine 75,000 public scenes with 5,000 client scenes and include public refinement on the server. Although Table 1 controls for model architecture and parameter count, it is unclear whether the centralized reduced LVSM baseline uses the same scene collections, amount of supervision, and total computational budget. The ablations on removing the public anchor and changing data allocation provide some relevant evidence. However, they do not fully separate the contributions of mutual reinforcement, additional pseudo-target supervision, and training resources. This directly affects the interpretation of where the performance gains come from.
  2. The theoretical analysis mainly explains individual components under fixed conditions, while guarantees for the complete mutual reinforcement process remain limited. The server generalization analysis assumes fixed client teachers, input views, and pose samplers. The client personalization convergence analysis assumes a fixed server and considers exact gradient updates for the consistency loss. However, actual training involves both server and neural field updates. It also includes reconstruction, entropy regularization, and consistency losses. The appendix discusses bidirectional error relationships and local-update drift. Nevertheless, it remains unclear how these results guarantee the stability of the complete training process. In particular, when teacher predictions are unreliable, errors on the server and client sides may reinforce each other. This possibility has not been sufficiently addressed.
  3. The privacy evaluation supports a relative reduction in information exposure, but the coverage of potential attacks remains limited. The membership inference attack (MIA) and property inference attack (PIA) experiments mainly focus on ShapeNet Cars. The limited-context reconstruction experiment evaluates reconstruction using the aggregated server model and leaked images. However, these settings do not sufficiently cover attacks that directly exploit individual client updates or sequences of updates across communication rounds. In addition, the 15 attack runs reported in the appendix use a fixed server checkpoint. They mainly capture variations in attack data splits and attacker initialization. This limits the generalizability of the privacy findings to other scenes and training states.
Questions for authors
  1. Are the main baselines matched in terms of data, supervision, and computational budgets? Please clarify whether the centralized reduced LVSM baseline and the proposed method use the same public and client scenes. Do they also use the same input-view and target-view sampling strategies and total training budget? Are there any controlled comparisons that separately evaluate pseudo-target supervision and federated mutual reinforcement under matched supervision and computational budgets? Have the main results in Tables 1 and 2 been verified through independent training runs? This information will directly affect my assessment of the reported performance gains and their attribution.
  2. What guarantees do the theoretical results provide for the actual mutual reinforcement training process? Please explain how the fixed-condition analyses of server generalization and client personalization convergence relate to the actual alternating updates and the complete client training objective. Are there any training trajectory diagnostics that track teacher errors, server prediction errors, and local-update drift? In particular, have the authors observed error accumulation or failures of bidirectional supervision when client teachers are unreliable? Such evidence would help assess the reliability of the mutual reinforcement mechanism.
  3. What attacker capabilities and training states are covered by the current privacy findings? The appendix considers an honest-but-curious server that can access individual client updates. However, the limited-context reconstruction experiment mainly uses the aggregated model and leaked images. Have the authors evaluated reconstruction attacks that directly exploit individual client updates or sequences of updates across communication rounds? In addition, do the MIA and PIA results remain consistent across different server checkpoints or independent training runs? Please clarify the scope and limitations of the privacy protection supported by the current experiments. This will affect my assessment of the strength of the paper's privacy contribution.
5 comments

We thank the reviewer for the constructive assessment. The revised paper (1) separates the sources of the server gain with matched centralized controls, (2) states the scope of the theory and adds a stability condition with measurements over rounds, and (3) extends the privacy evaluation to individual updates with controls, update inversion, combined updates, other checkpoints, and Objaverse clients. Changes are marked in blue.


W1 & Q1: Attribution of the gain

Matching. The reduced LVSM baseline shares our architecture (22.1M parameters), resolution, input views, target-view sampling, and evaluation scenes, and was trained for 9M steps on the 75K public scenes, equal to our public refinement. It differs from our server in the 5K client scenes and the pseudo-targets. The other baselines (PixelNeRF, IBRNet, LVSM (full)) were trained on the same 75K public scenes with their official configurations.

Factor separation. We trained two centralized models on all 80K scenes gathered in one place, with and without pseudo-targets on every scene, under our total budget (71.1M scene-samples), and changed one factor at a time.

FactorModels comparedServer PSNREffect
5K additional scenesCentralized 75K vs. centralized 80K38.39 vs. 38.68+0.29
Pseudo-target supervisionCentralized 80K without vs. with pseudo-targets38.68 vs. 42.48+3.80
Federated trainingCentralized 80K with pseudo-targets vs. ours42.48 vs. 41.85−0.63
Server-to-client refinement (Lcons\mathcal{L}_{\text{cons}})Ours w/o Lcons\mathcal{L}_{\text{cons}} vs. ours (Table 4)39.38 vs. 41.85+2.47

Pseudo-target supervision accounts for most of the gain, and more scenes for 0.29 dB. Federation costs 0.63 dB (paired 95% CI [−0.75, −0.51]) and retains 83% of the centralized gain. Mutual reinforcement acts through teacher quality: without Lcons\mathcal{L}_{\text{cons}} the teacher degrades (client PSNR 44.47 → 40.98) and the server drops by 2.47 dB.

These controls establish the contribution: neural-field pseudo-targets are worth 3.6 to 3.8 dB to an image-conditioned NVS model, and a federated protocol obtains 83% of that gain without collecting any client scene. The centralized control is an upper bound that needs every client's scenes in one place. When they cannot be collected, the alternative is the public-only model, which our server exceeds by 3.46 dB on Objaverse and 2.95 dB on zero-shot GSO. The statement that federated supervision improves generalization "beyond model capacity" is removed.

Independent runs. Tables 1 and 2 come from a single training run, since one full federated run takes about 16 days; the revised paper states this. The uncertainty is quantified as follows.

StatisticValue
Ours (Table 1), bootstrap 95% CI over evaluation scenes41.85 [41.27, 42.43]
Federated − centralized, paired difference on the same scenes−0.63 dB [−0.75, −0.51]
Two runs of the same public-only training (38.08 vs. 38.39)0.31 dB
LLFF, 3 seeds29.38 ± 0.11

The federated–centralized gap and the effect of pseudo-targets (+3.56 and +3.80 dB) both exceed the 0.31 dB run-to-run reference.

In the revised paper: Sec. 4.4 ("Centralized controls") and the bottom rows of Table 4; App. E (baseline setup, intervals, seeds); the conclusion (single run).

W2 & Q2: Theory and the actual training process

Theorem 3.1 (server, with the teachers fixed) and Theorem 3.3 (client, with the server fixed) describe exactly the two stages of a round: within a round, the server update uses pseudo-targets that are already rendered, and client refinement uses the server model received in that round. What they do not give by themselves is convergence across rounds. The revision states this scope at the end of Sec. 3.3 and adds the link in a Remark.

Let FtF_t be the server objective at round tt and RtR_t the teacher's rendering error. With exact gradients, as in Thm. 3.3, and the PL condition (constants μϕ\mu_\phi and μψ\mu_\psi), Prop. 3.2 and the proof of Thm. 3.3 (Eq. 39) give

Ft+1−F⋆≤a(Ft−F⋆)+γ2Jϕ2μϕRt,a=(1−ημϕ)EF_{t+1}-F^\star \le a (F_t-F^\star) + \frac{\gamma^2 J_\phi^2}{\mu_\phi} R_t, \qquad a=(1-\eta\mu_\phi)^{E}

Rt+1−R⋆≤b(Rt−R⋆)+5Jψ22μψγFt+1,b=(1−βμψ)KR_{t+1}-R^\star \le b (R_t-R^\star) + \frac{5J_\psi^2}{2\mu_\psi\gamma} F_{t+1}, \qquad b=(1-\beta\mu_\psi)^{K}

where EE and KK are the numbers of steps and η\eta and β\beta the step sizes. Here aa and bb are the rates at which each model reduces its own error, and the last terms are the error carried over from the other model. The two errors contract jointly, instead of amplifying each other, if

5γJϕ2Jψ22μϕμψ<(1−a)(1−b)\frac{5 \gamma J_\phi^2 J_\psi^2}{2 \mu_\phi \mu_\psi} \lt (1-a)(1-b)

The condition is easier to satisfy for a smaller pseudo-target weight γ\gamma, consistent with Fig. 6(a). The parts of training outside these inequalities only help: public refinement lowers the server error independently of the teacher, and Lpix\mathcal{L}_{\text{pix}} and Lent\mathcal{L}_{\text{ent}} in the full client objective (Eq. 23) anchor the field independently of the server.

Trajectory diagnostics. We rendered Objaverse ground truth at the pseudo-target poses and tracked both errors and the update norm.

RoundTeacher PSNR at pseudo-target posesServer PSNR at nearby poses∥Δϕi∥/∥ϕ∥\lVert\Delta\phi_i\rVert / \lVert\phi\rVert
10038.4724.633.8×10−23.8\times10^{-2}
50039.2131.852.1×10−22.1\times10^{-2}
1,50040.2638.020.9×10−20.9\times10^{-2}
3,00040.8341.370.4×10−20.4\times10^{-2}
  • Both errors and the drift norm decrease over rounds, and the two errors do not amplify each other. Early on the teacher pulls the server up (38.47 vs. 24.63); by the end the server overtakes it (41.37 vs. 40.83).
  • With an unreliable teacher (w/o Lent\mathcal{L}_{\text{ent}}), the server ends 0.84 dB below the variant without client training (37.24 vs. 38.08), and neither error diverges: the teacher settles near 34.0 dB and the server near 37.2 dB, which corresponds to a larger floor R⋆R^\star in the inequalities above.

In the revised paper: end of Sec. 3.3; App. B.4 (Remark B.1, Eqs. 43 and 44); App. D.3 (Tables 11 and 12).

W3 & Q3: Attacker capability and scope of the privacy evaluation

Scope. The attacker is an honest-but-curious server that knows the architecture, θbase\theta_{\text{base}}, ϕt\phi^t, and every individual Δϕit\Delta\phi_i^t; no DP or secure aggregation is applied (first paragraph of Sec. 4.3). The only thing that leaves a client is Δϕ\Delta\phi, and a scene cannot be rendered from it, because the server model needs images of that scene as input. An attacker therefore has two routes to a scene: leaked images fed to the server model, or inversion of images from Δϕ\Delta\phi. This holds for one update or for many. We tested both routes, and MIA/PIA under more conditions.

(1) Individual updates: reconstruction. The attack in the manuscript (Table 6 and Fig. 4; Table 9 and Fig. 4 in the revised paper) already used the target's individual update, ϕt+Δϕit\phi^t + \Delta\phi_i^t, as the captions say; the text of Sec. 4.3 and App. C.3 that described it as the aggregate was wrong and is corrected. We added controls that change only the attacker's model (local 4-view context, 180° target, same 100 scenes).

Attacker's modelPSNR of the 180° view
Target's own update applied (manuscript)19.01
Another client's update applied18.97
No update applied (ϕt\phi^t)18.98
Aggregated state, target included18.98
Aggregated state, target excluded18.98
Reference: leaked views changed to the surrounding 4-view configuration22.39
Reference: the client's TensoRF itself40.63

The target's update makes no difference (0.04 dB, paired 95% CI [−0.03, 0.11]). Reconstruction is determined by the coverage of the leaked views.

(2) Individual updates: inversion. We ran the optimization-based attack of the Breaching framework (Geiping et al., 2022), with the attacker also given the camera poses of the client's images and the training configuration.

Exposed updateClient's local trainingPSNR ↑SSIM ↑LPIPS ↓
Full update (all parameters)1 step11.04 ± 0.940.167 ± 0.0620.768 ± 0.047
Ours – single (Δϕ\Delta\phi)1 step6.41 ± 0.330.008 ± 0.0030.940 ± 0.033
Ours (Δϕ\Delta\phi as uploaded in the main experiment)100 steps, with pseudo-targets6.08 ± 0.290.005 ± 0.0020.906 ± 0.028

From a full update the attack recovers the colors and coarse layout of the object. From Δϕ\Delta\phi it returns noise, even after a single local step: withholding the input-side layers is enough to defeat it. What remains in an individual update is the weak membership and property signal, not image content.

(3) Sequences of updates. MIA that combines kk updates of the same client from different rounds:

Number of combined updates kkMIA Acc (chance 50)
156.8
557.9
2059.3

Twenty updates raise accuracy by 2.5 points. With only the average of a round's updates visible, as under secure aggregation, both attacks are at chance (Table 8), as Sec. 4.3 notes.

(4) Other training states and scenes.

ConditionMIA AccPIA top-1
Round 500 ("Early FL")58.933.2
Round 1,50057.532.1
Round 3,000 ("Ours")56.831.4
Round 3,000, Objaverse clients51.9 ± 0.8n/a

The values decrease as training proceeds, and on the Objaverse clients of the main experiment MIA is at chance (AUC 0.52). The 15 runs in the appendix vary attack splits and attacker initializations on one server; independent training runs were not attacked, since the full training was run once.

In the revised paper: Sec. 4.3 (scope, summary, and update inversion with Fig. 5); App. C: Table 5 (what leaves the client), C.3 (Tables 7 and 8), C.4 (Table 9), and C.5 (Table 10).

Reviewer_4iPROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a federated framework that jointly trains a server-side large image-conditioned NVS model (a reduced LVSM) and per-client personalized neural fields. Each client is one Objaverse scene with 12 posed views. A client fits a private TensoRF teacher, renders nearby pseudo-target views (mixture weight γ=0.4\gamma = 0.4), and updates only the decoder-side subset ϕ\\\phi on captured and pseudo targets. It uploads only Δϕi\Delta\phi_i. The server averages the updates, refines the model on a centrally held public set (75K scenes against 5K clients), and broadcasts the result, which clients then distill into their fields through a consistency loss. The theoretical part gives a bound for the server, a bound linking teacher error to server-gradient bias, and a biased-gradient descent bound for client refinement. Experiments report gains over a centralized reduced LVSM on Objaverse, zero-shot GSO and an LLFF non-IID cohort, and gains over per-scene NeRF baselines for client personalization. The privacy evaluation consists of MIA/PIA on ShapeNet Cars and a limited-context reconstruction attack.

Critical strengths
  1. The idea is interesting as the paper suggests per-scene neural fields serve as private teachers whose rendered pseudo-views convert local 3D structure into supervision for an image-conditioned model. This bridges two parameterizations that cannot be averaged directly, and neither the raw images nor the renderable field leaves the client.

  2. The ablation results (Table 4) are extensive and help understanding the paper. Both the server and the clients under each removal are evaluated.

Critical weaknesses
  1. The claim that federated client supervision improves the server beyond centralized training seems not to be fully supported by public data and pseudo-view augmentation. Specifically, public data makes up ρ=75/80\rho = 75/80 of the scenes and receives 3000 central steps per round (App. A.4). Removing it drops server PSNR to 32.18 (Table 4). The centralized reduced LVSM baseline (38.39, Table 1) is close to the public-only ablation (w/o Lsrv\mathcal{L}_{\mathrm{srv}}, 38.08), and its training scenes and step count do not appear to be stated. The resulting 3.5 dB gain comes from at most 5K extra scenes, delivered through a channel no more informative than central access. It seems more plausibly due to the TensoRF pseudo-view augmentation, which could equally be applied centrally: without pseudo targets the server falls back to 38.30 (Table 4). he LLFF non-IID result seems to have a similar issue. Applying the Objaverse ratio (ρ=75/80\rho = 75/80) to the ~93 (116 × 0.8) training scenes (Sec. 4.2) appears to leave ~87 central and ~6 client scenes. It is not clear whether the LVSM baselines were fine-tuned on the same scenes. The claim of better non-IID optimization therefore rests on a 0.26 dB margin over full LVSM (29.43 vs. 29.17, Table 1), with no reported variance.

  2. The threat model grants the server each individual Δϕit\Delta\phi_i^t (App. C), yet the reconstruction attack uses the aggregate ϕˉt+1\bar{\phi}^{t+1} over 117 clients (App. C.3), diluting the target’s update roughly 1/117. This seems inconsistent with the Table 6 (“intercepted client-updated model”) and Fig. 4 captions. Moreover, without a member versus non-member contrast at equal context, Table 6 seems to measure dependence on view coverage .This matters because personalization depends heavily on client-trained server updates: with server-side client training disabled (w/o Lsrv\mathcal{L}_{\mathrm{srv}}), client PSNR is 41.35 (Table 4), only ~ 0.4 dB above TensoRF (40.91, Table 2), against 44.47 (~3.6 dB) with the full pipeline. Whether this reflects client-specific content absorbed by the server or general server improvement (server PSNR 38.08 vs. 41.85, Table 4) is exactly what a member/non-member contrast would resolve. Additionally, MIA/PIA are reported only as accuracy, with no AUC or low-FPR TPR, and on ShapeNet Cars rather than Objaverse. PIA reaches 31.4% [30.4, 32.4] against 9.1% chance (1/11 balanced classes, App. C.2), and MIA reaches 56.8% [56.0, 57.6] (Table 5). These are framed as low only relative to a leaked NeRF (93.2% / 91.6%) that the protocol never transmits.

Questions for authors
  1. What was the centralized reduced LVSM trained on: the 75K public scenes only, or all 80K? Please give the number of training steps, views per scene, and whether pseudo-view augmentation was used. Are the Objaverse evaluation scenes disjoint from the 5K client scenes?
  2. Could you please report a centralized reduced LVSM trained on all 80K scenes with the same TensoRF pseudo-view augmentation (γ=0.4\gamma = 0.4) and matched compute?
  3. For LLFF, how many training scenes are public and how many are clients? Were PixelNeRF, IBRNet and both LVSM variants fine-tuned on the same LLFF training scenes, or evaluated zero-shot from Objaverse? Given the 0.26 dB margin over full LVSM, can the authors report variance across seeds?
  4. Which model do Table 6 and Fig. 4 use: the aggregate ϕˉt+1\bar{\phi}^{t+1} (App. C.3) or the target’s individual update? Can the reconstruction attack be repeated with ϕt+Δϕit\phi^t + \Delta\phi_i^t, which the stated threat model permits?
  5. Can you please add a member/non-member contrast? This would compare reconstruction of the held-out 180° view under identical leaked context, using a server that has aggregated the target client against one that has not (e.g., a client not yet sampled, or a leave-one-out run).
  6. Each upload is a single scene’s 11.2M-parameter decoder update. Can you please evaluate an update-inversion attack, for example optimizing images at known poses to reproduce Δϕit\Delta\phi_i^t?
  7. For MIA, are the candidates the client’s own training views or held-out views of the same object, and are positives and negatives balanced? Please report AUC and TPR at 0.1% and 1% FPR. Was the full federated pipeline rerun on ShapeNet Cars, and can MIA/PIA be reported on the Objaverse setting of the main experiments?
  8. In Table 4, w/o Lent\mathcal{L}_{\mathrm{ent}} yields a lower server PSNR (37.24) than disabling client training entirely (w/o Lsrv\mathcal{L}_{\mathrm{srv}}, 38.08), even though those client teachers still outperform TensoRF (41.02 vs. 40.91, Tables 2 and 4). What explains this?
7 comments

We thank the reviewer for the detailed and accurate analysis. The revised paper (1) replaces the "federated > centralized" claim with the requested matched comparison, (2) states that the reconstruction attack uses individual updates and adds member vs. non-member controls, (3) reports MIA/PIA with AUC and TPR at low FPR, on Objaverse, and on the averaged update, (4) adds an update-inversion attack, and (5) explains the "w/o Lent\mathcal{L}_{\text{ent}}" result by direct measurement. It also corrects three statements in the text: the attack model (Q4), the LLFF split (Q3), and the encoder (clients download the full model every round, so communication is 63.55 MiB per participation and 23.39 TB overall). The experiments and numbers are unchanged. Changes are marked in blue.

W1, Q1, Q2: Centralized baseline and the matched control

The reviewer's analysis is correct: the baseline in the manuscript was trained on the public scenes only, and pseudo-targets can also be used centrally. We withdrew the claim that federated supervision improves the server "beyond model capacity" and ran the requested control.

Baseline in the manuscript (Q1). Same architecture as our server (22.1M parameters, all trained); the 75K public scenes only; no pseudo-view augmentation; 9M steps with 4 scenes per step, equal to our public refinement (3,000 rounds × 3,000 steps); 12 posed views per scene at 256×256, four of them as inputs. The 100 Objaverse evaluation scenes are disjoint from both the public and the client scenes.

Matched control (Q2). One model trained on all 80K scenes gathered in one place, with TensoRF pseudo-targets (γ=0.4\gamma = 0.4) on every scene, under our total budget (71.1M scene-samples), and the same model without pseudo-targets.

ModelTraining scenesPseudo-targetsServer PSNR
Centralized, LVSM (reduced) (Table 1)75K publicNone38.39
Centralized80KNone38.68
Centralized (the requested condition)80KOn all 80K42.48
Federated, w/o pseudo target (Table 4)75K public + 5K clientNone38.30
Federated, ours75K public + 5K clientOn the 5K client scenes41.85
  • The centralized model is higher by 0.63 dB (paired 95% CI [−0.75, −0.51]). This is the cost of federation, and part of it is that the centralized model has pseudo-targets on all 80K scenes and ours on 5K.
  • Pseudo-targets add 3.80 dB centrally and 3.56 dB in federated training. Adding 5K scenes alone adds 0.29 dB, within run-to-run variation (0.31 dB).
  • Federated training retains 83% of the centralized gain.

The contribution of the paper stands on these numbers. (1) Neural fields used as private teachers are an effective supervision interface for an image-conditioned model, worth 3.6 to 3.8 dB in either setting. (2) Federated training obtains 83% of this gain while no client image or renderable field leaves the client. (3) The centralized control is an upper bound that needs every client's scenes in one place, which the setting rules out. When scenes cannot be collected, the alternative is the public-only model, and our server is 3.46 dB above it on Objaverse and 2.95 dB on zero-shot GSO, while each client gains 3.56 dB on its own field.

"w/o public data" (32.18). This ablation has no server-side training, so clients update the full model including the encoder and the model is trained on the 5K client scenes only. The reviewer's reading that public data is the foundation of server quality is correct, and the revision states this setup.

In the revised paper: Sec. 4.4 ("Centralized controls") and the bottom rows of Table 4; App. E; revised claims in the abstract, Sec. 1, Sec. 4.2, and the caption of Table 1; evaluation scenes in Sec. 4.1.

W1 & Q3: LLFF split, baselines, and seeds

The 116 scenes are split into 93 training and 23 test scenes, and all 93 training scenes are clients; there are no public LLFF scenes (Sec. 4.2 of the manuscript described this incorrectly). PixelNeRF, IBRNet, and both LVSM variants were trained on Objaverse and then fine-tuned centrally on the same 93 scenes; they are not zero-shot. Results over three seeds:

MethodHow the 93 LLFF training scenes are usedPSNRSSIMLPIPS
LVSM (full)Gathered in one place, fine-tuned centrally29.21 ± 0.130.946 ± 0.0030.082 ± 0.002
LVSM (reduced)Gathered in one place, fine-tuned centrally26.98 ± 0.150.873 ± 0.0040.102 ± 0.003
OursFederated training with 93 clients29.38 ± 0.110.953 ± 0.0020.073 ± 0.002

Our federated model stays ahead of the centrally fine-tuned full LVSM on all three metrics (+0.17 dB PSNR, +0.007 SSIM, −0.009 LPIPS; the SSIM and LPIPS margins exceed the seed deviation), with a reduced model and without gathering the LLFF scenes in one place. It is 2.40 dB above the centralized model of the same size. We keep this result and removed only the sentence attributing it to "better optimized" non-IID training, which this comparison does not isolate.

In the revised paper: Sec. 4.2 and App. E (Table 13).

Reviewer_NX3UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a federated framework coupling a shared image-conditioned novel view synthesis model with private, scene-specific neural fields. Clients render nearby pseudo-target views using their local fields and update a subset of the shared model’s parameters. Conversely, predictions from the shared model supervise local field refinement. The framework communicates parameter updates while retaining images and neural fields locally. Experiments report improved server generalization and client reconstruction, supported by component ablations, conditional theoretical analyses, and empirical privacy evaluations.

Critical strengths
  1. A coherent mechanism for collaborative scene reconstruction. I think this construction is great in that it connects two different model representations while avoiding transmission of directly renderable client fields.

  2. Substantial reported improvements with relevant ablations. The reduced server improves Objaverse PSNR from 38.391 to 41.852 over reduced LVSM, while client reconstruction improves over FrugalNeRF by approximately 3.15 dB on Objaverse and 1.91 dB on LLFF.

Critical weaknesses
  1. Unclear comparison fairness. Baselines are not clearly matched for data, supervision, and compute despite extensive public training. Relevant federated personalization baselines, such as FedMeNF [1], are also missing.

  2. Limited privacy evidence. Reconstruction attacks use the aggregated model despite the attacker’s access to individual updates. Without testing those updates or excluding the target client as a control, the experiment cannot isolate information leaked through training.

[1]: Yun, J., Hong, M., & Kim, G. (2025). FedMeNF: Privacy-Preserving Federated Meta-Learning for Neural Fields. ICCV, 2161–2171.

Questions for authors
  1. Comparison protocol: Did centralized reduced LVSM receive the same scenes, image resolution, captured-view supervision, and total training budget? What happens when these resources are matched, and how does client personalization compare with a suitable federated baseline such as FedMeNF?

  2. Implementation clarification: How is the frozen encoder initialized in the setting described as training from scratch? Is it updated during public refinement, and, if so, how are clients’ cached encoders synchronized?

  3. Privacy evidence: Can you evaluate reconstruction using the individual updates available to the stated attacker, with a control excluding the target client’s contribution? This would help separate information learned from that client from reconstruction enabled by leaked context images.

3 comments

We thank the reviewer for the constructive comments. The revised paper adds (1) budget-matched centralized controls and a FedMeNF comparison, (2) target-excluded controls for the reconstruction attack, and (3) a corrected description of the encoder and of the communication cost. Changes are marked in blue.


W1 & Q1: Comparison fairness and a federated baseline

Server. The centralized reduced LVSM has our server's architecture (22.1M parameters), resolution (256×256), input views (4), and evaluation scenes (100 scenes unseen in training). It was trained on the 75K public scenes only, for 9M steps with 4 scenes per step, which equals our server's public refinement. The other baselines (PixelNeRF, IBRNet, LVSM (full)) were trained on the same 75K public scenes with their official configurations. The reduced baseline therefore lacked the 5K client scenes and the pseudo-target supervision. We trained centralized controls that have both: one model on all 80K scenes gathered in one place, with and without TensoRF pseudo-targets (γ=0.4\gamma = 0.4) on every scene, under our total training budget (71.1M scene-samples).

Training (Objaverse PSNR)Without pseudo-targetsWith pseudo-targetsEffect of pseudo-targets
Centralized, all 80K scenes38.6842.48+3.80
Federated (ours)38.3041.85+3.56

Pseudo-target supervision is what raises performance, by 3.6 to 3.8 dB in either setting; adding the 5K scenes without it changes little. Federated training retains 83% of the centralized gain and is 0.63 dB below it (paired 95% CI [−0.75, −0.51]), while no client image or neural field leaves the client. The centralized control is an upper bound that needs every client's scenes in one place, which the setting rules out. When scenes cannot be collected, the alternative is the public-only model, and our server is 3.46 dB above it on Objaverse and 2.95 dB on zero-shot GSO (Table 1). The earlier statement that federated supervision improves generalization "beyond model capacity" is removed.

Client. We added FedMeNF, run with its public code (500 clients, 12 per round, 1,000 rounds) and compared on the same 500 clients, 12 training views, 5,000 steps, and evaluation views. Each method is shown next to local training with its own backbone.

MethodBackboneWhat the client uploadsLocal training onlyAfter federationGain
FedMeNF, public codeMLPField meta-parameters26.1428.92+2.78
OursTensoRFΔϕ\Delta\phi of the server model40.9144.47+3.56

Our method gives the higher quality and the larger gain from federation. FedMeNF shares one initialization across all scenes, whereas our server gives each scene supervision conditioned on its own views. FedMeNF also communicates field parameters, which our protocol never does.

In the revised paper: Sec. 4.4 ("Centralized controls") and the bottom rows of Table 4; App. E (details; FedMeNF in Table 14); revised claims in the abstract, Sec. 1, Sec. 4.2, and the caption of Table 1.

W2 & Q3: Reconstruction with individual updates and a target-excluded control

The reconstruction attack in the manuscript already used the target client's individual update, ϕt+Δϕit\phi^t + \Delta\phi_i^t, as the captions of Table 6 and Fig. 4 say (Table 9 and Fig. 4 in the revised paper). The text of Sec. 4.3 and App. C.3 described it as the aggregated state ϕˉt+1\bar\phi^{t+1}; this was an error in the text and is corrected. The numbers are unchanged.

We added the requested control. With the leaked views and the target view (180°) fixed, we change only the attacker's model, on the same 100 scenes.

Leaked viewsTarget's own updateAnother client's updateNo update
1-view13.9013.8713.88
2-view15.7215.7015.69
3-view18.6118.5718.58
Local 4-view19.0118.9718.98
Surrounding 4-view22.3922.3622.37
Attacker's model (local 4-view)PSNR of the 180° view
Aggregated state, target included18.98
Aggregated state, target excluded18.98
Same attack on scenes never used in training18.93
Reference: the client's TensoRF itself40.63

The target's update does not improve the reconstruction: the difference from another client's update is at most 0.04 dB, and its paired 95% CI for the local 4-view case, [−0.03, 0.11], contains zero. Excluding the target from the aggregate changes nothing, and scenes never used in training are reconstructed at the same level. Switching the leaked views to the surrounding configuration raises PSNR by 3.4 dB for every model. Reconstruction is therefore determined by the coverage of the leaked images, and the update carries no information usable for reconstructing the scene. An actual leak of the client's field gives 40.63 dB.

In the revised paper: Sec. 4.3 and App. C.4 (Table 9, all conditions in PSNR, SSIM, and LPIPS).

Q2: Encoder initialization and synchronization

The manuscript described the handling of the encoder incorrectly. The method and all reported numbers are unchanged.

  • Initialization. All parameters, including the encoder, start from random initialization. There is no pretrained checkpoint and no warm-up stage.
  • Training. The encoder (θbase\theta_{\text{base}}: image tokenizer and 6 encoder layers, 10.8M parameters) is trained only by the server's public refinement, which updates all parameters every round. Clients keep it fixed and update only ϕ\phi (decoder side, 11.2M parameters), which the server also refines.
  • Synchronization. Since θbase\theta_{\text{base}} changes every round, selected clients download the full model (θbaset,ϕt)(\theta_{\text{base}}^t, \phi^t) at the start of each round and upload only Δϕ\Delta\phi. Clients hold no cached encoder, so nothing can fall out of sync. The "cached encoder" statements in Sec. 3.1 and App. A.4 were wrong and are removed.
  • Communication. With the full model on the downlink, the cost per participation is 63.55 MiB (42.12 down, 21.43 up) instead of 42.9 MiB, and 23.39 TB overall instead of 15.78 TB.
  • Without public data. In the "w/o public data" ablation no server-side stage trains the encoder, so clients update the full model.

In the revised paper: Sec. 3.1 (Steps 3 and 5), Algorithm 1, App. A.2 and A.4, and Sec. 4.4.

Comments on the paper

1 comment
Area_Chair_6U6QOpenReview ↗

Urgent Reminder: Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly but urgent reminder that the discussion period will end on November 18, 2026.

Please carefully review the authors' rebuttals and actively participate in the ongoing discussions. If you have not yet responded to the authors or initiated a discussion, please do so as soon as possible.

In particular, reviewers are expected to address the authors' clarifications and responses, provide constructive follow-up feedback, and update their assessments and scores when appropriate.

Please ensure that your final evaluations reflect your independent judgment and take into account the information provided during the rebuttal and discussion process.

Active and timely participation is essential to ensuring a fair, thorough, and constructive review process.

Thank you for your time, effort, and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.