acceptodds
Under review as a conference paper at ICLR 2027

One Capture, One Camera Family: Structured Camera Modeling for Light Field Gaussian Splatting

Abstract

3D Gaussian Splatting (3DGS) has demonstrated remarkable performance for novel view synthesis, yet its conventional camera formulation is poorly matched to light field (LF) reconstruction. In particular, treating the sub-aperture views captured in a single LF exposure as independent cameras disregards their shared acquisition geometry and leads to inconsistent scene structure across views. To address this, we propose StructLFGS, a reconstruction framework built on the principle of one capture, one camera family. Specifically, we model all sub-aperture views as observations generated by a single structured camera family, which compresses the camera solution space from per-view 6-DoF states to a compact shared parameterization. Crucially, this reduction allows us to determine the canonical camera parameters directly from the observed angular trajectories. This defines a canonical frame shared by all views and eliminates per-view camera optimization. With this canonical family fixed, the trajectories are lifted into Gaussian primitives. Their disparity estimates further constrain optimization, making cross-view geometric consistency intrinsic to the representation. Extensive experiments on four LF benchmarks show that StructLFGS outperforms GS-based methods and remains competitive against LF-specific methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Light field, Gaussian splatting, view reconstruction, structured camera modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iQk2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StructLFGS for 3DGS reconstruction from sparse light field inputs. The algorithm maps the observed robust disparity range to a pre-specified canonical inverse-depth interval, thereby avoiding the need for SfM, and incorporates disparity-consistency supervision during the optimization phase. Experiments across multiple datasets demonstrate the effectiveness of the algorithm.

Critical strengths
  1. The algorithm models all cameras as a shared structured camera family, thereby avoiding independent 6-DoF computations; the effectiveness of this approach was validated through ablation studies.
  2. The algorithm feeds the computed optical flow into the subsequent optimization stage, enhancing the geometric consistency of the overall pipeline.
Critical weaknesses
  1. The algorithm proposed in the paper relies on the "structured family" assumption; however, actual devices inevitably exhibit hardware errors and calibration errors, yet the paper lacks experiments assessing robustness against such errors.
  2. The experiments focus primarily on PSNR, SSIM, and LPIPS metrics for RGB images, lacking an evaluation of geometric accuracy.
Questions for authors

Light field images are captured using a fixed camera system that has already been calibrated; why, then, is it necessary to recover camera parameters from the images? How does the accuracy of StructLFGS compare to 3DGS methods that utilize known poses (including sparse-view 3DGS)?

2 comments

To W1

We agree that no imaging device is perfectly accurate. The two real datasets were captured with Lytro cameras. A fixed microlens array, shared lens, and sensor determine the relative arrangement of their angular views. This integrated design provides a physical basis for our structured camera family [1]. Established LF reconstruction methods also successfully exploit regular angular structure in real captures [2–5]. Their results support structured geometry as an effective approximation for practical LF reconstruction.

Our ablation further supports keeping the canonical family fixed under the evaluated conditions. After constructing the family, we allowed its shared baseline and offset parameters to be optimized. This reduced PSNR by 0.45 dB on the four-scene real-data subset compared with fixed cameras (Table 3). Thus, additional shared geometric refinement did not improve reconstruction. Together, the acquisition design, established LF results, and our controlled comparison support the use of a fixed structured family for these real captures.

[1] Dansereau, D. G., Pizarro, O., and Williams, S. B. Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras. CVPR, pp. 1027–1034, 2013.

[2] Yeung, H. W. F., Hou, J., Chen, J., Chung, Y. Y., and Chen, X. Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues. ECCV, pp. 138–154, 2018.

[3] Wang, Y., Liu, F., Zhang, K., Wang, Z., Sun, Z., and Tan, T. High-Fidelity View Synthesis for Light Field Imaging With Extended Pseudo 4DCNN. IEEE Transactions on Computational Imaging, 6:830–842, 2020.

[4] Jin, J., Hou, J., Chen, J., Zeng, H., Kwong, S., and Yu, J. Deep Coarse-to-Fine Dense Light Field Reconstruction With Flexible Sampling and Geometry-Aware Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):1819–1836, 2022.

[5] Liang, Z., Wang, Y., Wang, L., Yang, J., Guo, Y., Liu, L., Zhou, S., and An, W. Diving Into Epipolar Transformers for Light Field Super-Resolution and Disparity Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(7):8726–8743, 2026.

To W2

We have added a GT-based disparity evaluation to complement the RGB metrics (§4.5, Fig. 6, Appendix F). On StillLife, directly rendered Gaussian disparity achieves 2.69 MSE×100 and 38.61% BadPix₀.₀₇, compared with 3.10/12.90% for EPINET on the original LF. The Gaussian result has lower average squared error, although more pixels exceed the strict error threshold. Appendix F specifies the checkpoint configuration used for this geometry evaluation.

We also evaluate the geometric information preserved in the reconstructed images. With the same pretrained EPINET, StructLFGS achieves 1.30/10.62%, compared with 5.36/21.73% for FS-GAF and 28.44/23.16% for EPIT-ASR. These results show that StructLFGS supports accurate downstream disparity recovery. Together, the added measurements evaluate both the geometry encoded by the Gaussian scene and the angular information retained in its rendered views.

Table R1. Disparity evaluation on StillLife.

Input / methodDisparity measurementMSE×100 ↓BadPix₀.₀₇ (%) ↓
Original LFEPINET3.1012.90
StructLFGSDirect rendering2.6938.61
StructLFGSEPINET on rendered views1.3010.62
FS-GAFEPINET on reconstructed views5.3621.73
EPIT-ASREPINET on reconstructed views28.4423.16

To Q1

LF systems typically obtain regular angular structure through their optical design, as in Lytro cameras, or through calibration and rectification, as in camera arrays. The released images retain this structure. However, the corresponding calibration parameters may be unavailable or omitted from the dataset, as in the real datasets used here. Gaussian reconstruction still requires explicit cameras to project and render the scene. StructLFGS uses the available angular structure to construct compatible cameras and scene coordinates directly from the images. When calibrated parameters are available, our framework can use them directly.

Table 4 directly compares StructLFGS with reconstruction using GT-calibrated cameras in the same 3DGS-MCMC framework. Both use the same sparse input views. Using GT calibration improves PSNR by 0.70/0.16 dB with four/nine inputs on HCInew. Thus, our canonical construction achieves reconstruction quality close to the known-camera reference, especially with nine inputs, without requiring physical calibration parameters. This comparison isolates the camera choice within one reconstruction backend. Specialized known-pose sparse-view 3DGS methods have not been benchmarked separately.

Dear Reviewer iQk2,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_tYJ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To address the instability issues commonly associated with traditional per-view camera estimation, this paper proposes a light-field Gaussian Splatting reconstruction framework designed based on the “one capture, one camera family” principle. This method explicitly models the shared capture geometry among light-field views and, under a predefined inverse depth parameterization, constructs a canonical camera family based on angular parallax analysis, thereby avoiding the need for external SfM and per-view independent pose optimization. At the same time, the same angular trajectory is further utilized for unified camera family construction, Gaussian initialization, and disparity supervision, forming a complete reconstruction workflow. The authors conducted experiments on multiple synthetic and real-world light field datasets, achieving significant performance improvements in new-view synthesis compared to existing Gaussian-based and LF-specific methods, while analyzing the roles of components such as camera family constraints and trajectory-guided initialization through ablation experiments.

Critical strengths
  1. The paper provides a thorough comparison of the core modules through ablation studies. The experiments in the main text and the appendix distinguish between camera constraints, trajectory initialization, shuffled depth, and random initialization. Additionally, the effectiveness of the canonical camera was validated using GT calibration. Overall, these experiments largely answer the questions of whether the internal modules are effective and where their effectiveness roughly stems from.

  2. The paper is clearly written, the experimental setup is transparent, and the information for reproducibility is fairly comprehensive. The paper provides detailed descriptions of the selection of input views, held-out views, metrics, dataset preprocessing, camera handling for different baselines, and the number of optimization steps.

Critical weaknesses
  1. Since the target-camera estimation methods vary across different approaches, and the evaluation process for YoNoSplat deviates from the pose-free target-pose evaluation procedure described in its original paper, there remains a certain degree of uncertainty regarding the fairness and interpretability of the cross-method comparison in Table 1.

  2. Although the paper provides a relatively thorough validation of the camera parameterization and the final rendering quality, it lacks a direct assessment of the accuracy of the reconstructed depth geometry itself, particularly in areas with poor geometric observability, such as at long distances with narrow baselines, small parallax, or occlusions.

Questions for authors

These issues substantially affect my current evaluation of the paper. I would appreciate convincing clarification or additional evidence addressing them, and I would be willing to reconsider my current rating accordingly.

  1. The evaluation procedure used for YoNoSplat differs from the pose-free target-pose evaluation procedure described in the original YoNoSplat paper. Could the authors clarify the rationale for this modification and whether it affects the reported YoNoSplat results? In particular, would following the target-pose evaluation procedure described in the original YoNoSplat paper lead to materially different performance?

  2. Given the narrow LF baselines, a direct depth or disparity evaluation, especially in challenging regions, would help determine whether the rendering improvements also reflect accurate geometry.

  3. Although Table 1 uses the same held-out views and metric implementation for all methods, the target-camera estimation/alignment procedures differ substantially across the evaluated pipelines. Given the unusually large performance margins, could the authors clarify how much of the improvement is attributable to target-camera alignment and how much to the reconstructed Gaussian representation itself?

4 comments

Thank you for the reviews.

To W1 and Q1

YoNoSplat fits target poses to target RGB because its predicted scene and reference cameras may use different coordinate systems. This additional fitting can compensate for geometric errors or introduce alignment errors. The reported scores therefore also reflect the target-pose fitting stage.

For LF data, the known angular indices let us locate target cameras directly from the predicted input cameras. We bilinearly interpolate the target centers and assign the shared orientation to all targets. This places the target cameras in the predicted scene’s coordinate system without fitting target RGB. Our evaluation therefore measures input-camera estimation and scene reconstruction jointly, using held-out images only for scoring. Appendix B now clarifies this adaptation.

To measure its effect, we also evaluated YoNoSplat using its original target-pose alignment. The predicted Gaussians and test views remained unchanged. Appendix B, Table B.1 reports the results, reproduced below.

Table R1. YoNoSplat target-camera protocols.

DatasetTarget-camera protocolPSNR ↑SSIM ↑LPIPS ×10 ↓
HCInewLF interpolation21.300.55242.804
HCInewOriginal target-pose alignment21.650.57122.751
30scenesLF interpolation25.890.77301.973
30scenesOriginal target-pose alignment26.180.78551.931

The original protocol improves PSNR by 0.35/0.29 dB, but does not change StructLFGS’s lead on either dataset. Thus, following the original protocol improves YoNoSplat’s scores modestly while preserving the main comparative conclusion.

To W2 and Q2

We added direct disparity evaluation on StillLife (HCIold), which contains overlapping objects and depth discontinuities (§4.5, Fig. 6, Appendix F). For the StructLFGS checkpoint specified in Appendix F, Gaussian-rendered disparity achieves 2.69 MSE×100, compared with 3.10 for EPINET on the original LF. Its BadPix₀.₀₇ is higher (38.61% vs. 12.90%). Thus, direct rendering provides competitive average squared error, while fine geometric accuracy remains less consistent.

To connect geometry with rendering quality, we also applied EPINET to RGB views from the same checkpoint, obtaining 1.30/10.62%, compared with 5.36/21.73% for FS-GAF and 28.44/23.16% for EPIT-ASR. These results show that the reconstructed views preserve useful angular information for depth recovery. The matched RGB/native comparison also shows why image quality and geometric accuracy should be assessed separately. Appendix F provides the configuration, common evaluation protocol, and complete results in Table F.1.

Table R2. Disparity evaluation on StillLife.

Input / methodDisparity measurementMSE×100 ↓BadPix₀.₀₇ (%) ↓
Original LFEPINET3.1012.90
StructLFGSDirect rendering2.6938.61
StructLFGSEPINET on rendered views1.3010.62
FS-GAFEPINET on reconstructed views5.3621.73
EPIT-ASREPINET on reconstructed views28.4423.16

To Q3

We measured the effect of target-camera alignment by keeping YoNoSplat’s predicted Gaussians and test views unchanged. Using its original target-pose alignment improves PSNR by 0.35/0.29 dB on HCInew/30scenes (Appendix B, Table B.1). StructLFGS remains ahead on both datasets. Thus, the tested alignment choice accounts for only a small change in YoNoSplat’s scores and does not explain the large performance gap.

To identify the sources of improvement within StructLFGS, Table 3 compares individual components in the same 3DGS-MCMC framework. Each comparison changes one component while retaining the others. The results are summarized below.

Table R3. Controlled component ablations. PSNR drops are relative to the full model, which achieves 38.31/37.40 dB on HCInew/the four-scene real subset.

Change from the full modelHCInew PSNR drop (dB)Real subset PSNR drop (dB)
Fixed camera family → independent poses14.055.68
Trajectory initialization → random initialization6.053.22
Remove disparity supervision0.290.15

These ablations identify the shared camera constraints and geometric initialization as major contributors to reconstruction quality. Together, the two experiments show that our advantage extends beyond the tested target-alignment choice and is supported by controlled improvements within a common Gaussian framework. Table 1 compares complete pipelines, so its performance gaps cannot be divided into separate camera-alignment and Gaussian-representation contributions.

Dear Reviewer tYJ4,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

In Table 1, different methods obtain the test camera poses in different ways. Even a small error in the camera pose can affect the score because the rendered image may not align well with the GT. So it is hard to tell how much of the gap comes from the reconstruction itself. I had hoped the authors would choose scenes with calibrated cameras and compare their method with a few representative baselines. They could first align the reconstructed scenes to the same coordinate system and then render them using the same test cameras. This would show how much of the gap remains. But the rebuttal did not include this shared-camera comparison. For this reason, I am keeping my original rate.

Reviewer_AoxaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StructLFGS is a per-scene 3DGS method for sparse angular reconstruction of a rectified light field (LF): given 2x2 or 3x3 sub-aperture views, it renders the full 7x7 lattice. Instead of estimating a 6-DoF pose per view, it models all views as one "structured camera family" with shared parameters: identity rotations, coplanar centers on a lattice, and a principal-point shift for the focus offset. Each scene point then moves linearly with the angular index, with per-step disparity affine in inverse depth. Flow from a frozen RAFT network is projected onto the angular direction and fused per track with IRLS, giving a disparity, an uncertainty and a confidence. Then, fused tracks are lifted to Gaussians and optimized with 3DGS-MCMC under an L1–SSIM loss and a confidence-weighted Huber loss on rendered disparity. Extensive experiments have demonstrated the effectiveness of the proposed method.

Critical strengths

The method produces strong synthetic results without LF-specific training, at modest cost. On HCInew and HCIold, a per-scene, training-free pipeline beats nine trained LF angular super-resolution networks on all three metrics. Its PSNR significantly exceeds EPIT-ASR. It needs about 3 minutes and 1.6 GB of GPU memory per scene, much faster than the joint-optimization GS baselines (Table D.1), demonstrating that explicit Gaussian geometry with a correct camera prior can compete with learned LF priors without any training data.

Critical weaknesses

W1

The GS comparison does not isolate the claimed contribution, and the camera contribution itself is modest. Every GS baseline must recover its cameras (three of four using held-out RGB); StructLFGS imposes the known rectified geometry. Appendix B concedes that Table 1 "does not by itself isolate the effect of the camera-family parameterization." Table 3 suggests that most of the margin comes from initialization (HCInew, 2x2):

  • Free 6-DoF poses (started from the lattice) with trajectory initialization (D, 29.91 dB) already beat every GS baseline by at least 8.6 dB.

  • The fixed family with random initialization (F, 24.37 dB) beats the best baseline (YoNoSplat, 21.30 dB) by only 3.1 dB.

Meanwhile, Eq. 1 is the standard rectified LF model, and the "analytic" construction amounts to choosing a gauge. Rectified inputs already require calibration (App. B), and GT calibration does better (Table 4). Dense-correspondence initialization is common in sparse-view 3DGS. The missing baseline is 3DGS-MCMC with the known lattice cameras and an off-the-shelf dense initialization. This weighs most in my rating: Table 1 overstates what the new modeling contributes.

W2

The practical value relative to LF-specific methods is unclear. On real Lytro captures (the main use case), StructLFGS is third in PSNR, behind EPIT-ASR and Yeung et al. Its LPIPS is much worse than EPIT-ASR's and trails 5-6 of the 9 baselines. The synthetic wins may partly reflect a domain gap if the baseline checkpoints were trained on real data; the paper does not say. The comparison also covers only 2x2 inputs, reports no runtimes for these networks, and never tests what an explicit 3D representation should add: extrapolation beyond the input views, or disparity accuracy (HCI has GT disparity). It is unclear when one should prefer StructLFGS.

Questions for authors
  1. How does 3DGS-MCMC perform under the known lattice cameras (canonical or GT-calibrated) (a) with random initialization? (b) with a simple dense initialization, e.g., RAFT disparities unprojected without your track fusion? If StructLFGS still leads clearly, W1 is largely resolved.

  2. HCInew provides GT disparity. How accurate are the rendered disparities (MSEx100, BadPix) compared with standard LF depth estimators? The abstract claims that cross-view geometric consistency is "intrinsic to the representation", and this would test that claim.

  3. Sec. 4.3 attributes the real-data gap partly to inter-view capture discrepancies that the fixed model cannot represent. Did you try small per-view corrections (e.g., principal-point or photometric offsets) on top of the fixed family? Table 3 tests only shared (B, o) refinement and free 6-DoF poses.

4 comments

Thank you for your detailed and constructive comments.

To W1 and Q1

Table 1 compares complete reconstruction pipelines, while Table 3 identifies component effects within the same 3DGS-MCMC framework. The pipelines differ in camera handling, initialization, and optimization. We therefore use the controlled comparisons in Table 3 and §4.4 for component attribution. In the original table, the shared-refinement and independent-pose variants omitted disparity supervision. The revised camera comparisons retain both trajectory initialization and disparity supervision across all variants.

Table R1. Revised controlled ablations within the same 3DGS-MCMC framework.

Camera configurationInitializationDisparity supervisionHCInew PSNR ↑Real subset PSNR ↑
Fixed familyTrajectory✓38.3137.40
Shared refinementTrajectory✓37.3536.95
Independent posesTrajectory✓24.2631.72
Fixed familyRandom✓32.2634.18
Fixed familyTrajectory—38.0237.25

Under these matched settings, replacing fixed cameras with independent input poses reduces PSNR by 14.05/5.68 dB on HCInew/the four-scene real subset. For Q1(a), replacing trajectory initialization with random initialization reduces PSNR by 6.05/3.22 dB, with canonical cameras and disparity supervision retained. Removing disparity supervision reduces PSNR by 0.29/0.15 dB. Thus, the camera change produces the largest PSNR drop among these controlled alternatives on both datasets.

For Q1(b), direct RAFT unprojection without track fusion achieves 37.85 dB on HCInew, compared with 38.31 dB for trajectory initialization and 32.26 dB for random initialization. Both geometric initializations substantially outperform random initialization. The complete trajectory-based initialization retains a 0.46 dB advantage over direct unprojection under the tested settings.

LF devices obtain regular angular structure through their optical design (e.g., Lytro cameras) or calibration and rectification (e.g., camera arrays). The released images retain this structure, but the underlying camera parameters are not always provided. Our analytic construction supplies this common frame through a canonical gauge choice. The same fitted angular trajectories then support camera construction, Gaussian initialization, and disparity supervision. When calibrated parameters are available, our framework can use them directly. Table 4 compares this setting with our canonical construction within the same 3DGS-MCMC framework. GT calibration improves PSNR by 0.70/0.16 dB with four/nine inputs on HCInew. Thus, the canonical construction provides reconstruction quality close to the calibrated reference while removing the need for physical camera parameters as reconstruction inputs.

To W2

Under the standard 2×2 → 7×7 LF reconstruction protocol, StructLFGS outperforms all evaluated LF-specific methods on both synthetic benchmarks. The baseline checkpoints used in these comparisons were trained exclusively on synthetic LF data, as clarified in Appendix B. These results demonstrate the potential of Gaussian-based LF reconstruction without relying on a real-to-synthetic training mismatch. However, on real captures, StructLFGS ranks third in PSNR and trails several networks in LPIPS. Real captures also contain photometric variation, noise, and other degradations. Their individual contributions to the performance gap remain unresolved, motivating our future work on reconstruction robust to imperfect observations.

Beyond image quality, StructLFGS reconstructs a reusable Gaussian scene without external paired LF training data or LF-specific fine-tuning. A frozen pretrained RAFT model supplies correspondences, and the scene is optimized for each capture. Once constructed, the scene supports user-specified angular positions and sampling densities within the input extent, without further scene optimization. The framework is therefore useful when paired LF training data are unavailable or flexible rendering from an explicit scene is required. We have clarified this practical value in the revised Introduction and Conclusion.

To Q2

We added a GT-based comparison of directly rendered Gaussian disparity and EPINET on StillLife (§4.5, Fig. 6, Appendix F). For the StructLFGS checkpoint specified in Appendix F, direct rendering achieves 2.69 MSE×100, compared with 3.10 for EPINET on the original LF. This shows competitive average squared disparity error. Its BadPix₀.₀₇ is higher (38.61% vs. 12.90%), indicating room to improve accuracy at the strict pixel-level threshold.

The reconstructed light fields also support accurate downstream depth recovery. EPINET achieves 1.30/10.62% on RGB renders from the same checkpoint, compared with 5.36/21.73% for FS-GAF and 28.44/23.16% for EPIT-ASR. The following table summarizes the two measurements.

Table R2. Disparity evaluation on StillLife.

Input / methodDisparity measurementMSE×100 ↓BadPix₀.₀₇ (%) ↓
Original LFEPINET3.1012.90
StructLFGSDirect rendering2.6938.61
StructLFGSEPINET on rendered views1.3010.62
FS-GAFEPINET on reconstructed views5.3621.73
EPIT-ASREPINET on reconstructed views28.4423.16

These results highlight a useful strength of StructLFGS: its rendered views retain angular information that supports accurate disparity estimation, beyond per-view image quality.

To Q3

We tested per-view photometric correction on the same four real scenes used in Table 3. We observed photometric differences among the four input views and corrected them before reconstruction. This increased average PSNR by 0.38 dB. Thus, photometric correction helps, but the gain is limited. Real captures also contain noise and other degradations. We have not yet isolated their individual effects. These findings motivate our future work on reconstruction robust to imperfect observations.

For principal-point offsets, our current ablations refine the shared camera parameters while preserving the regular angular structure. The same camera family defines the geometry used for Gaussian initialization and disparity supervision. We therefore evaluated refinement within this family first. Adding independent offsets to individual views would extend this model with residual geometric corrections. That specific extension has not been tested separately.

Dear Reviewer Aoxa,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Thank you for the additional experiments, in particular the matched ablations, the direct RAFT-unprojection baseline (Q1b), and the new disparity evaluation. They have addressed my concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.