- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses several challenges in large image-conditioned novel view synthesis (NVS) models. These include the reliance on centrally collected multi-view data, insufficient supervision for personalized neural fields under sparse local observations, and potential privacy risks from directly sharing neural field parameters. To address these challenges, the authors propose a federated mutual-reinforcement framework that connects a server-side image-conditioned NVS model with client-side private neural fields. Each client uses its local neural field to render pseudo-targets from nearby viewpoints. These pseudo-targets, together with captured images, are used to update a trainable subset of the server model's parameters. The server aggregates the client updates. In turn, predictions from the server model guide local neural field optimization through consistency supervision. This creates a mutual-reinforcement loop between global generalization and local personalization. Raw images, pseudo-targets, and neural field parameters remain on the clients. The paper trains the framework using 75,000 public scenes and 5,000 Objaverse scene-level clients. It evaluates the method on Objaverse, zero-shot GSO, and non-IID IBRNet-LLFF settings. The experimental results show that the server model outperforms the centralized reduced LVSM baseline with the same architecture and parameter count across all three settings. The client-side neural fields achieve the best PSNR, SSIM, and LPIPS among the compared methods on Objaverse and IBRNet-LLFF. Ablation studies support the effectiveness of pseudo-target supervision and bidirectional guidance. Privacy attack experiments also show that uploading updates to a subset of model parameters results in lower measured information exposure than directly exposing client-side neural fields.
Critical strengths
- Clear problem formulation and a reasonable framework connecting global generalization with local personalization. The paper addresses the limitations of centrally collecting multi-view data and the lack of supervision for personalized neural fields under sparse observations. The authors propose a federated mutual reinforcement framework that connects a server-side image-conditioned NVS model with client-side private neural fields. This allows cross-scene knowledge to guide local reconstruction while keeping client scene representations private. This design has clear practical value and represents a core contribution of the paper.
- The supervision interface from neural fields to NVS has a reasonable geometric basis. Clients use their neural fields to generate nearby pseudo-targets. This converts local 3D information into image-based supervision for the server model. In turn, server predictions guide client optimization through consistency supervision. Unlike independent 2D augmentation, these pseudo-targets share the geometry, visibility, and appearance modeled by the neural fields. The paper also analyzes the trade-off between pose coverage and teacher bias. This provides a reasonable explanation for the benefits of pseudo-targets and the limitations of their reliability.
- The experiments evaluate both server generalization and client personalization, with results largely consistent with the proposed mechanisms. The server model outperforms the centralized reduced LVSM baseline with the same architecture and parameter count across Objaverse, zero-shot GSO, and non-IID IBRNet-LLFF settings. The client-side neural fields achieve the best reconstruction metrics among the compared methods on both evaluation datasets. Ablation studies show that removing either pseudo-target supervision or consistency supervision affects performance on both sides. Together, these results support the practical effectiveness of bidirectional knowledge transfer.
Critical weaknesses
- The attribution of the main performance gains remains unclear due to insufficient comparability in data and computational budgets. The main experiments combine 75,000 public scenes with 5,000 client scenes and include public refinement on the server. Although Table 1 controls for model architecture and parameter count, it is unclear whether the centralized reduced LVSM baseline uses the same scene collections, amount of supervision, and total computational budget. The ablations on removing the public anchor and changing data allocation provide some relevant evidence. However, they do not fully separate the contributions of mutual reinforcement, additional pseudo-target supervision, and training resources. This directly affects the interpretation of where the performance gains come from.
- The theoretical analysis mainly explains individual components under fixed conditions, while guarantees for the complete mutual reinforcement process remain limited. The server generalization analysis assumes fixed client teachers, input views, and pose samplers. The client personalization convergence analysis assumes a fixed server and considers exact gradient updates for the consistency loss. However, actual training involves both server and neural field updates. It also includes reconstruction, entropy regularization, and consistency losses. The appendix discusses bidirectional error relationships and local-update drift. Nevertheless, it remains unclear how these results guarantee the stability of the complete training process. In particular, when teacher predictions are unreliable, errors on the server and client sides may reinforce each other. This possibility has not been sufficiently addressed.
- The privacy evaluation supports a relative reduction in information exposure, but the coverage of potential attacks remains limited. The membership inference attack (MIA) and property inference attack (PIA) experiments mainly focus on ShapeNet Cars. The limited-context reconstruction experiment evaluates reconstruction using the aggregated server model and leaked images. However, these settings do not sufficiently cover attacks that directly exploit individual client updates or sequences of updates across communication rounds. In addition, the 15 attack runs reported in the appendix use a fixed server checkpoint. They mainly capture variations in attack data splits and attacker initialization. This limits the generalizability of the privacy findings to other scenes and training states.
Questions for authors
- Are the main baselines matched in terms of data, supervision, and computational budgets? Please clarify whether the centralized reduced LVSM baseline and the proposed method use the same public and client scenes. Do they also use the same input-view and target-view sampling strategies and total training budget? Are there any controlled comparisons that separately evaluate pseudo-target supervision and federated mutual reinforcement under matched supervision and computational budgets? Have the main results in Tables 1 and 2 been verified through independent training runs? This information will directly affect my assessment of the reported performance gains and their attribution.
- What guarantees do the theoretical results provide for the actual mutual reinforcement training process? Please explain how the fixed-condition analyses of server generalization and client personalization convergence relate to the actual alternating updates and the complete client training objective. Are there any training trajectory diagnostics that track teacher errors, server prediction errors, and local-update drift? In particular, have the authors observed error accumulation or failures of bidirectional supervision when client teachers are unreliable? Such evidence would help assess the reliability of the mutual reinforcement mechanism.
- What attacker capabilities and training states are covered by the current privacy findings? The appendix considers an honest-but-curious server that can access individual client updates. However, the limited-context reconstruction experiment mainly uses the aggregated model and leaked images. Have the authors evaluated reconstruction attacks that directly exploit individual client updates or sequences of updates across communication rounds? In addition, do the MIA and PIA results remain consistent across different server checkpoints or independent training runs? Please clarify the scope and limitations of the privacy protection supported by the current experiments. This will affect my assessment of the strength of the paper's privacy contribution.
5 comments
W2 & Q2: Theory and the actual training process
Theorem 3.1 (server, with the teachers fixed) and Theorem 3.3 (client, with the server fixed) describe exactly the two stages of a round: within a round, the server update uses pseudo-targets that are already rendered, and client refinement uses the server model received in that round. What they do not give by themselves is convergence across rounds. The revision states this scope at the end of Sec. 3.3 and adds the link in a Remark.
Let be the server objective at round and the teacher's rendering error. With exact gradients, as in Thm. 3.3, and the PL condition (constants and ), Prop. 3.2 and the proof of Thm. 3.3 (Eq. 39) give
where and are the numbers of steps and and the step sizes. Here and are the rates at which each model reduces its own error, and the last terms are the error carried over from the other model. The two errors contract jointly, instead of amplifying each other, if
The condition is easier to satisfy for a smaller pseudo-target weight , consistent with Fig. 6(a). The parts of training outside these inequalities only help: public refinement lowers the server error independently of the teacher, and and in the full client objective (Eq. 23) anchor the field independently of the server.
Trajectory diagnostics. We rendered Objaverse ground truth at the pseudo-target poses and tracked both errors and the update norm.
| Round | Teacher PSNR at pseudo-target poses | Server PSNR at nearby poses | |
|---|---|---|---|
| 100 | 38.47 | 24.63 | |
| 500 | 39.21 | 31.85 | |
| 1,500 | 40.26 | 38.02 | |
| 3,000 | 40.83 | 41.37 |
- Both errors and the drift norm decrease over rounds, and the two errors do not amplify each other. Early on the teacher pulls the server up (38.47 vs. 24.63); by the end the server overtakes it (41.37 vs. 40.83).
- With an unreliable teacher (w/o ), the server ends 0.84 dB below the variant without client training (37.24 vs. 38.08), and neither error diverges: the teacher settles near 34.0 dB and the server near 37.2 dB, which corresponds to a larger floor in the inequalities above.
In the revised paper: end of Sec. 3.3; App. B.4 (Remark B.1, Eqs. 43 and 44); App. D.3 (Tables 11 and 12).
W3 & Q3: Attacker capability and scope of the privacy evaluation
Scope. The attacker is an honest-but-curious server that knows the architecture, , , and every individual ; no DP or secure aggregation is applied (first paragraph of Sec. 4.3). The only thing that leaves a client is , and a scene cannot be rendered from it, because the server model needs images of that scene as input. An attacker therefore has two routes to a scene: leaked images fed to the server model, or inversion of images from . This holds for one update or for many. We tested both routes, and MIA/PIA under more conditions.
(1) Individual updates: reconstruction. The attack in the manuscript (Table 6 and Fig. 4; Table 9 and Fig. 4 in the revised paper) already used the target's individual update, , as the captions say; the text of Sec. 4.3 and App. C.3 that described it as the aggregate was wrong and is corrected. We added controls that change only the attacker's model (local 4-view context, 180° target, same 100 scenes).
| Attacker's model | PSNR of the 180° view |
|---|---|
| Target's own update applied (manuscript) | 19.01 |
| Another client's update applied | 18.97 |
| No update applied () | 18.98 |
| Aggregated state, target included | 18.98 |
| Aggregated state, target excluded | 18.98 |
| Reference: leaked views changed to the surrounding 4-view configuration | 22.39 |
| Reference: the client's TensoRF itself | 40.63 |
The target's update makes no difference (0.04 dB, paired 95% CI [−0.03, 0.11]). Reconstruction is determined by the coverage of the leaked views.
(2) Individual updates: inversion. We ran the optimization-based attack of the Breaching framework (Geiping et al., 2022), with the attacker also given the camera poses of the client's images and the training configuration.
| Exposed update | Client's local training | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| Full update (all parameters) | 1 step | 11.04 ± 0.94 | 0.167 ± 0.062 | 0.768 ± 0.047 |
| Ours – single () | 1 step | 6.41 ± 0.33 | 0.008 ± 0.003 | 0.940 ± 0.033 |
| Ours ( as uploaded in the main experiment) | 100 steps, with pseudo-targets | 6.08 ± 0.29 | 0.005 ± 0.002 | 0.906 ± 0.028 |
From a full update the attack recovers the colors and coarse layout of the object. From it returns noise, even after a single local step: withholding the input-side layers is enough to defeat it. What remains in an individual update is the weak membership and property signal, not image content.
(3) Sequences of updates. MIA that combines updates of the same client from different rounds:
| Number of combined updates | MIA Acc (chance 50) |
|---|---|
| 1 | 56.8 |
| 5 | 57.9 |
| 20 | 59.3 |
Twenty updates raise accuracy by 2.5 points. With only the average of a round's updates visible, as under secure aggregation, both attacks are at chance (Table 8), as Sec. 4.3 notes.
(4) Other training states and scenes.
| Condition | MIA Acc | PIA top-1 |
|---|---|---|
| Round 500 ("Early FL") | 58.9 | 33.2 |
| Round 1,500 | 57.5 | 32.1 |
| Round 3,000 ("Ours") | 56.8 | 31.4 |
| Round 3,000, Objaverse clients | 51.9 ± 0.8 | n/a |
The values decrease as training proceeds, and on the Objaverse clients of the main experiment MIA is at chance (AUC 0.52). The 15 runs in the appendix vary attack splits and attacker initializations on one server; independent training runs were not attacked, since the full training was run once.
In the revised paper: Sec. 4.3 (scope, summary, and update inversion with Fig. 5); App. C: Table 5 (what leaves the client), C.3 (Tables 7 and 8), C.4 (Table 9), and C.5 (Table 10).
We thank the reviewer for the constructive assessment. The revised paper (1) separates the sources of the server gain with matched centralized controls, (2) states the scope of the theory and adds a stability condition with measurements over rounds, and (3) extends the privacy evaluation to individual updates with controls, update inversion, combined updates, other checkpoints, and Objaverse clients. Changes are marked in blue.
W1 & Q1: Attribution of the gain
Matching. The reduced LVSM baseline shares our architecture (22.1M parameters), resolution, input views, target-view sampling, and evaluation scenes, and was trained for 9M steps on the 75K public scenes, equal to our public refinement. It differs from our server in the 5K client scenes and the pseudo-targets. The other baselines (PixelNeRF, IBRNet, LVSM (full)) were trained on the same 75K public scenes with their official configurations.
Factor separation. We trained two centralized models on all 80K scenes gathered in one place, with and without pseudo-targets on every scene, under our total budget (71.1M scene-samples), and changed one factor at a time.
Pseudo-target supervision accounts for most of the gain, and more scenes for 0.29 dB. Federation costs 0.63 dB (paired 95% CI [−0.75, −0.51]) and retains 83% of the centralized gain. Mutual reinforcement acts through teacher quality: without Lcons the teacher degrades (client PSNR 44.47 → 40.98) and the server drops by 2.47 dB.
These controls establish the contribution: neural-field pseudo-targets are worth 3.6 to 3.8 dB to an image-conditioned NVS model, and a federated protocol obtains 83% of that gain without collecting any client scene. The centralized control is an upper bound that needs every client's scenes in one place. When they cannot be collected, the alternative is the public-only model, which our server exceeds by 3.46 dB on Objaverse and 2.95 dB on zero-shot GSO. The statement that federated supervision improves generalization "beyond model capacity" is removed.
Independent runs. Tables 1 and 2 come from a single training run, since one full federated run takes about 16 days; the revised paper states this. The uncertainty is quantified as follows.
The federated–centralized gap and the effect of pseudo-targets (+3.56 and +3.80 dB) both exceed the 0.31 dB run-to-run reference.
In the revised paper: Sec. 4.4 ("Centralized controls") and the bottom rows of Table 4; App. E (baseline setup, intervals, seeds); the conclusion (single run).