acceptodds
Under review as a conference paper at ICLR 2027

Physics-Guided Surface and Volume Gaussians for Clear Reconstruction through Smoke

Abstract

Recovering a clear scene from multiview observations of moving smoke is difficult because visible images entangle persistent surfaces with a transient participating medium. Thermal observations provide complementary structural cues, but persistent surfaces and dynamic smoke have different physical supports that call for different representations. We introduce , a framework guided by physics that models the scene with 2D Gaussian surfaces constrained by thermal observations and deformable 3D Gaussian smoke. A unified renderer composites both representations along each viewing ray, allowing their interaction to be learned consistently. To recover appearance lost during surface and smoke separation, we further use the estimated smoke contribution to guide the restoration of visible detail. Experiments on scenes with smoke show substantial improvements in the reconstruction of clear scenes across image quality metrics, together with a more compact surface representation. These results highlight the value of matching scene representations and detail recovery to the physical structure of observations through smoke.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaussian Splatting, Through-Smoke Reconstruction, Participating Media, Thermal Imaging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2puGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PhySmoke reconstructs clear images at observed camera views from smoky RGB images, registered thermal images, and known cameras. Static surfaces are represented by native 2D Gaussian surfels and dynamic smoke by deformable 3D Gaussians. A mixed renderer jointly sorts the primitives and composites them through one transmittance chain. A final reliability gate uses the estimated smoke contribution to control how much image residual is restored. On nine synthetic SmokeSeer scenes, the complete method improves the macro-average PSNR, SSIM, and LPIPS over SmokeSeer. The evaluation is restricted to observed views; the method uses the smoky RGB image at the target pose.

Critical strengths
  1. The surface/volume split matches the distinct roles of the static scene and moving smoke. The controlled mixed-renderer comparison reports a 1.17 dB PSNR gain over layer-wise compositing with the same components. This supports the claim that shared depth ordering matters, rather than merely swapping a 3D surface representation for a 2D one.

  2. The detail-recovery ablation identifies a real tradeoff. Ungated residual recovery improves PSNR but worsens SSIM and LPIPS, whereas the proposed gate retains the PSNR gain and improves those two macro metrics. The added recovery improves PSNR over the physical representation in all nine scenes, although gains in the other metrics vary.

Critical weaknesses
  1. The evidence for generalization is narrow. All nine evaluated scenes are synthetic scenes from one benchmark, and the Dense experiment changes initialization on the same scene identities. The paper reports one seed. Because the method depends on registered thermal boundaries and a smoke-based reliability estimate, I would like evidence on real RGB/thermal captures or, at least, sensitivity to registration error, thermal noise, and weaker thermal boundaries. Without it, the practical motivation remains less well supported than the in-benchmark reconstruction result.

  2. The paper gives a compactness ratio for the surface component, not cost for the complete pipeline. The method also trains a dynamic smoke representation and an additional volumetric branch for recovery. Training time, rendering time, memory, and total primitive count would clarify the practical cost of its roughly 1 dB average PSNR gain. This is an evidence gap, not a demonstrated flaw in the renderer.

Questions for authors
  1. How sensitive are the observed-view results to RGB/thermal misregistration and noisy or weak thermal boundaries? Even a controlled perturbation on the current benchmark would help establish the usable capture range.

  2. Could the authors provide complete-pipeline training time, rendering time, peak memory, and total surface-plus-smoke primitive count, using the same hardware and protocol as SmokeSeer?

  3. Do the authors have any results on real registered RGB/thermal smoke captures? If not, which capture assumptions are most likely to fail outside the synthetic benchmark?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2puG

Thank you for requesting real-data validation, controlled input perturbations, and complete-pipeline profiling. We have addressed these requests with a substantial experimental expansion: our own physical RGB–thermal capture and calibration pipeline, public real captures, 42 complete two-stage training runs covering perturbations and three seeds, and full system cost measurements. The revision reports the resulting evidence in Section 4.7 and Appendices H–I.

  1. We independently acquired real RGB–thermal data and verified the method on it.

We assembled a capture setup using a DJI Matrice 600 carrying a Zenmuse XT2, operated along the ground. We acquired paired smoky RGB–thermal observations and separate smoke-free sequences, calibrated the fixed camera rig, reconstructed poses and initial geometry, and constructed independent smoke-free reference renderings. RGB and thermal observations use their own intrinsics and calibrated poses.

Table 4 evaluates all 77 observed training views of our self-collected scene using identical object bounding boxes. Ours improves PSNR/SSIM/LPIPS from SmokeSeer's 20.10/0.7895/0.2190 to 20.48/0.8206/0.1773: +0.38 dB PSNR, +0.0311 SSIM, and a 19.0% LPIPS reduction. Figure 6 shows four matched-view comparisons across our data and the public Bathroom and Red Container captures. The magnified regions demonstrate smoke suppression and clearer surface reconstruction. Appendix H documents the full acquisition and evaluation pipeline.

  1. Controlled perturbations directly address calibration, noise, and weak thermal boundaries.

Appendix I.2, Table 10 retrains both stages of both methods on Hotdog, Room, and Forest, covering object, indoor, and outdoor content. We use the same initial cloud, full resolution, seed, and 15k thermal + 15k joint budget. Thermal-camera orientation, thermal noise, and thermal contrast are changed independently; recovery parameters retain the synthetic defaults. Every entry averages the same three scenes.

Mean PSNR in dB, reported as SmokeSeer / Ours: Clean: 21.33 / 22.32. Yaw 0.5°: 18.54 / 19.60. Yaw 1°: 17.43 / 18.00. Noise σ=0.01: 19.98 / 20.92. Contrast 0.5×: 17.07 / 17.22.

Ours achieves higher mean PSNR and SSIM and lower LPIPS in every tested condition. These results directly verify the reconstruction advantage under registration error, noisy thermal observations, and weakened thermal structure. The tables also make the dependence on input quality explicit. In real acquisition, our fixed-rig calibration and separate camera projections provide the geometric consistency needed to use both modalities effectively.

  1. We now provide complete system time, memory, and primitive accounting.

Appendix I.3, Table 11 profiles matched clean seed-zero runs on RTX A6000 GPUs. The values below are reported as SmokeSeer / Ours and include both learned branches for Ours.

Hotdog: training 25.4 / 56.1 min; output 5.7 / 55.9 ms/view; training peak memory 1.97 / 2.04 GiB; inference peak memory 0.16 / 0.36 GiB; total primitives 0.38 / 0.66 million.

Forest: training 31.5 / 66.7 min; output 9.2 / 67.3 ms/view; training peak memory 2.83 / 2.83 GiB; inference peak memory 0.51 / 0.72 GiB; total primitives 1.65 / 2.46 million.

Room: training 30.8 / 74.1 min; output 10.3 / 80.3 ms/view; training peak memory 2.74 / 2.74 GiB; inference peak memory 0.52 / 0.84 GiB; total primitives 1.54 / 2.60 million.

Training time sums the required stages, including loading and final checkpoint I/O, with validation/reporting disabled equally. Training memory is the maximum across sequential stages. Synchronized output latency includes both deformation networks, both surface/joint outputs, fusion, recovery, and required host-to-device transfers; disk writes are excluded and the scene-fit prepass is reported separately. Memory is peak PyTorch CUDA allocation, and counts include every required surface and smoke set.

This profiling quantifies the compute–quality tradeoff: the full reconstruction and recovery pipeline produces outputs in 55.9–80.3 ms and uses 2.1–2.4× baseline training time. The surface compactness measurement and the complete-pipeline profile now give explicit, separate accounts of representation size and system cost.

  1. Three-seed repeats confirm consistent improvements.

Appendix I.4, Table 12 reports fresh matched training with seeds 0, 1, and 2, including means and sample standard deviations. Ours improves all three metrics in every one of the nine scene/seed pairs. Mean paired PSNR gains are 2.62 dB for Hotdog, 1.02 dB for Room, and 0.23 dB for Forest. Together with the real acquisition study and controlled perturbations, these repetitions substantially strengthen the experimental validation of the method.

Thank you for the detailed response and additional experiments. The reported results substantially address my original concerns about real-data validation, sensitivity to thermal inputs, and complete-pipeline cost.

The self-collected real scene provides quantitative evidence beyond the synthetic benchmark: PSNR/SSIM/LPIPS improve from 20.10/0.7895/0.2190 to 20.48/0.8206/0.1773 across 77 observed views. The two public real captures add qualitative support. The controlled thermal-input perturbations directly address my robustness question, and the three-seed repeats on three selected scenes strengthen confidence in the gains.

The revision also clarifies important limitations. Halving thermal contrast reduces mean PSNR from 22.32 to 17.22 dB and narrows the advantage over SmokeSeer from 0.99 to 0.15 dB. Complete-pipeline profiling shows approximately 2.1-2.4 times the baseline training time and 7.3-9.8 times its output latency. Thus, surface compactness should not be interpreted as lower full-system cost.

One remaining clarification concerns the additional thermal-derived depth-support constraint used on the self-collected scene. A matched ablation with and without this constraint would help distinguish the original configuration's contribution from that of the real-data adaptation. I regard this as a clarification of the real-data gains rather than a prerequisite for my updated recommendation.

Overall, the added evidence, together with the joint-renderer and recovery ablations, supports the technical contribution for the stated observed-view reconstruction task. I am revising my recommendation from weak rejection (2) to weak acceptance (3), while retaining the limitations concerning real-scene coverage, thermal-input quality, and computational cost.

AuthorsOpenReview ↗

Response to Reviewer 2puG for the depth-support adaptation

Thank you for the careful reassessment and for updating your recommendation. We appreciate your recognition of the added real-data validation, thermal-input tests, and complete-pipeline profiling. We agree that surface compactness and full-system cost are separate measurements.

We have already completed the matched ablation you suggest. Both runs start from the same 15k thermal-stage checkpoint and use the same inputs, camera calibration, initialization procedure, seed, 15k joint-training budget, and recovery settings. The comparison switches the depth-support adaptation off or on, including its associated suppression of unsupported surface densification. Evaluation follows Table 4: all 77 observed training views, identical object bounding boxes, and the same independently reconstructed smoke-free reference renderings.

The matched ablation results are:

MethodPSNR ↑SSIM ↑LPIPS ↓
SmokeSeer20.100.78950.2190
Ours without depth support20.520.81990.1785
Ours with depth support20.480.82060.1773

These results directly distinguish the contributions. Without the additional constraint, our method already improves all three metrics over SmokeSeer, including a 0.42 dB PSNR gain and an 18.5% LPIPS reduction. Adding depth support further improves SSIM and LPIPS, with a small 0.04 dB PSNR decrease. The substantial improvement over SmokeSeer is therefore already present without depth support; the adaptation adds an incremental structural and perceptual improvement.

The visual motivation is that a higher PSNR does not by itself establish cleaner surface–smoke separation. Our full-image photometric objective rewards background fidelity alongside object fidelity, while some smoky appearance can still be absorbed into the surface representation. We introduced depth support to address the residual smoke and local structural blur seen in the surface renderings, explicitly guiding optimization toward depth-consistent surfaces. The SSIM and LPIPS improvements support this emphasis on structural and perceptual quality despite the small PSNR tradeoff. The object-region evaluation above quantifies the effect on the reconstruction target directly.

For real captures, whose smoke distribution need not match the simulator, this constraint uses the geometric support already available in our representation. Specifically, we use the native 2DGS depth-rendering capability to obtain a frozen surface-depth reference from the thermal stage. During joint optimization, opacity-weighted penalties discourage surface primitives from floating in front of reliable depth support, while the volumetric smoke branch remains free to explain the intervening medium. This uses the existing thermal observations and requires no external depth sensor, clear-image supervision, or smoke labels. The synthetic experiments and the two public real-capture examples use the configuration without this additional constraint.

This matched comparison clarifies the contribution of the depth-support adaptation. Thank you again for the constructive discussion.

Reviewer_g1BdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for smoked scene reconstruction, in such scenes the smoke is occuping the central object or scene elements and therefore is blocking the view. The authors ask the question how could one reconstruct and seperate static elments such as objects and dynamic elements such as smoke. They outline the key challenge, that this is not a traditional 4D Gaussian reconstruction as smoke has volumetric properties.

In order to counter this the authors propose modelling hard surface objects with 2D Gaussians and smoke with deformable 3D Gaussians. The disentanglement follows by splitting optimisation into two stages. First training on thermal images that do not capture the volumetric properties of the smoke and therefore reconstruct the geometry of the object. The second stage utilise the smoke covered RGB images, however since geometry already exist, the authors proposed optimisation and rendering strategy disentangle smoke and object appearance reconstruction.

Overall I see this as a valuable paper for the community, with enough depth that this is not solvable naively (see 3.4). The authors show a new methodology and good results.

Critical strengths

The authors propose a nice architecture to disentangle geometry + object appearance and smoke. I like the idea of using two representations for different use cases and unifying them into one system. Using thermal images as helpers seems justified as the problem is highly ill-posed. The detailed recovery in section 3.4 is very interesting and the authors utilise the physical properties of the smoke smartly.

The authors make a improvement compared to existing other methods, as shown for example in Fig 2.

Critical weaknesses

Given the thermal observations the task seems more simplistic then what the abstract suggested. Geometry can thus be more easily inferred. However as described above I think this is fair.

The experiment section could be clearer written its a bit convoluted with numbers.

The improvement over other methods is not as strong as I hoped.

Questions for authors

Did you test the results on real world objects as well? Would be interesting to see if this still works?

Why is still smoke left in some scenes e.g. in Fig 2? Is this then written to the 2D Gaussians? Where is this coming from, is the thermal imagery bad at describing geometry, has some flaws?

Are you modelling with default spherical harmonics for the 2D Gaussians? E.g. I see that the specular car in Fig 4 is slightly tinted grey, is this from then wrong spherical harmonics modelling? How could one intersept this?

1 comment
AuthorsOpenReview ↗

Response to Reviewer g1Bd

Thank you for the questions about real objects, residual smoke, and appearance modeling. We have strengthened the paper with extensive new experiments and a complete self-collected RGB–thermal validation pipeline. The revision includes real captures, controlled thermal-input perturbations, and three-seed repeats, with 42 full two-stage training runs supporting the additional synthetic tests.

  1. We now validate the method on independently acquired real objects and public real captures.

We built our own acquisition and calibration pipeline around a DJI Matrice 600 carrying a Zenmuse XT2. We collected smoky RGB–thermal observations and separate smoke-free sequences with the rig operated along the ground, calibrated the two cameras, reconstructed camera poses and initial geometry, and generated independent smoke-free reference renderings. Each modality uses its own intrinsics and calibrated pose.

Section 4.7 and Figure 6 combine this self-collected data with two public real captures, Bathroom and Red Container. Table 4 evaluates all 77 observed training views of our self-collected scene using identical object bounding boxes. Ours improves PSNR/SSIM/LPIPS from SmokeSeer's 20.10/0.7895/0.2190 to 20.48/0.8206/0.1773. The 19.0% LPIPS reduction and the matched-view enlargements demonstrate smoke suppression and clearer reconstruction of surface structure. Appendix H gives the acquisition, calibration, reference-construction, and evaluation details.

  1. Thermal guidance supplies geometry; the method resolves the joint geometry–appearance–medium problem.

Thermal observations establish structural support in the first stage. The subsequent problem is to recover visible appearance and assign attenuation and scattered radiance consistently to the persistent surface and moving medium. Our representation, shared-transmittance renderer, and reliability-guided recovery address these coupled tasks. The controlled renderer experiment improves PSNR by 1.17 dB over layer-wise compositing with unchanged components, directly demonstrating the value of shared depth ordering.

The first stage uses thermal photometric supervision. Smoky RGB supplies camera/point initialization and the joint-stage appearance supervision. The revised abstract and implementation state the multimodal inputs and their roles explicitly.

  1. The revision explains residual smoke and directly tests thermal-structure sensitivity.

The 2D representation constrains surface support, while appearance is optimized jointly with the medium. Surfels can therefore encode smoky color when the RGB evidence and thermal structure permit ambiguous assignments. Thermal radiance also reflects temperature and emissivity. These properties explain why geometric separation and appearance separation require coordinated optimization.

We address floating surface components on our self-collected data with a depth-support constraint anchored to a frozen thermal-derived surface. We also added full retraining under thermal-camera yaw errors, intensity noise, and reduced contrast (Appendix I.2, Table 10). Ours achieves higher mean PSNR and SSIM and lower LPIPS than SmokeSeer in every tested condition. For example, under 0.5°/1° yaw errors, ours reaches 19.60/18.00 dB versus 18.54/17.43 dB for SmokeSeer. These controlled experiments verify the method's reconstruction advantage when thermal support is degraded.

  1. The SH configuration and its connection to appearance are now explicit.

Both stages use degree-zero spherical-harmonic coefficients, stated in Section 4.1. This models view-independent color. Explicit view-dependent specular reflectance is a further appearance-model extension. The car's gray tint is consistent with view-independent appearance and residual surface–medium color mixing; this is distinct from the geometric support supplied by the surfels. We now explain these roles in the method and discussion.

  1. The improvements are demonstrated by complementary quantitative and visual evidence.

Across all nine synthetic scenes, the complete method raises macro PSNR from 20.61 to 21.61 dB and SSIM from 0.7569 to 0.8141, and lowers LPIPS from 0.1869 to 0.1583 (Table 1). On our independently collected real data, LPIPS decreases by 19.0%, with sharper surface structure in Figure 6. The new three-seed study further improves all three metrics in every one of the nine matched scene/seed pairs. These results establish gains in reconstruction fidelity, structure, and perceptual quality across the benchmark and the added real-data evaluation.

We also rewrote the experiment discussion to explain the mechanisms behind these improvements: shared depth ordering resolves surface–medium visibility, and reliability-guided recovery restores detail while regulating residual smoke transfer. Tables retain the numerical detail; the discussion now connects the evidence to the design choices.

Reviewer_YuUbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhySmoke, a method for recovering a clean scene from multi-view images captured through moving smoke. Since each observed image combines a persistent surface with a transient medium (smoke), PhySmoke adopts modality-specific representations to disentangle the two: 2D Gaussians for the persistent surface and 3D Gaussians for the volumetric medium. To improve this separation further, it uses the estimated smoke contribution to guide the restoration of detail obscured by smoke. On smoke-filled scenes, the authors report substantial improvements in clean-scene reconstruction across standard image-quality metrics, together with a more compact surface representation.

Critical strengths

The motivation for separating the two components is clearly argued and convincing. The opaque surface is represented by surfels, while the transient component is modeled by volumetric 3D Gaussians; this combination yields a clean separation of the scene while keeping the memory footprint compact. The method further improves the disentanglement by exploiting the differing reliability of each modality, which respects the asymmetry inherent in the image formation model.

Critical weaknesses

My main concern is that the method rests on heuristic design rather than theoretical grounds, and the paper does not establish that these heuristics are robust. Three observations support this. (i) The reliability formulation is hand-crafted: the parameter choices in Eqs. 10–12 are neither derived nor justified in their functional form. (ii) The smoke contribution appears equally heuristic. (iii) These are global parameters, yet no sensitivity analysis is reported, so it is impossible to tell whether the results depend on careful tuning. This matters most because the experiments are conducted mostly on synthetic data, where smoke statistics are far more controlled than in real captures. I would find a sensitivity study over the parameters of Eqs. 10–12, together with at least one real captured scene, sufficient to resolve this concern.

Furthermore, the experiment section reads as a narration of the tables: it walks through exact numbers without explaining why the proposed components produce the observed gains. Without such an analysis, it is difficult to tell which design choices are responsible for the improvement. The ablations already contain the material for this, and I would encourage the authors to use the space to interpret the numbers rather than restate them.

Questions for authors

See weekness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer YuUb

Thank you for highlighting the importance of parameter validation and real captures. We have addressed both points with a substantial experimental expansion: an 18-configuration study on all nine synthetic scenes, 42 complete two-stage training runs for perturbation and seed tests, and new real-data experiments. In particular, we built our own RGB–thermal acquisition, calibration, and reconstruction pipeline and collected real smoky observations to validate the method independently.

  1. The reliability rule follows the surface–volume image-formation model.

Section 3.4 connects recovery directly to the learned decomposition. The joint-minus-surface difference measures the radiance change assigned to the medium, including attenuation and added radiance. This provides the score D that controls residual transfer through C = exp(−βD). The rule gives the pointwise bound

‖I_final − S_fused‖∞ ≤ λ exp(−βD) ‖E‖∞.

The bound establishes the intended recovery behavior: the maximum admitted RGB residual decreases exponentially as the modeled medium contribution increases. The score is computed from the learned surface and volume renders, so the gate is directly tied to the scene decomposition. We have made this connection explicit and tested the gate's parameters and functional form experimentally.

  1. The new parameter study demonstrates consistent gains across a working range.

Appendix I.1, Table 9 reports every configuration in the gain/attenuation grid, fusion-threshold shifts, and linear/rational gate controls. All nine scene checkpoints are fixed, and metrics use equal-scene macro averaging.

With β = 8, all four gains λ = 0.5, 1, 2, or 3 improve PSNR, SSIM, and LPIPS over SmokeSeer.

Shifting both fusion thresholds by ±0.01 and ±0.02 gives 21.46–21.61 dB PSNR, consistently above SmokeSeer's 20.61 dB.

The exponential default achieves 21.61/0.8141/0.1583, compared with 21.56/0.8173/0.1604 for the linear gate and 21.62/0.7950/0.1666 for the rational gate (PSNR/SSIM/LPIPS).

These experiments directly answer the tuning concern. They show that the gains persist across parameter changes and explain the tradeoff: residual gain restores detail, while attenuation regulates contamination. The full grid also identifies aggressive settings, such as λ = 3, β = 4 with LPIPS 0.2041. Our original configuration delivers improvements in all three macro metrics and is retained throughout the comparisons.

  1. We independently built a capture pipeline and validated the method on real data.

We assembled a capture setup using a DJI Matrice 600 carrying a Zenmuse XT2, operated along the ground, and acquired paired smoky RGB–thermal observations and separate smoke-free sequences. We calibrated the two cameras, recovered poses and initial geometry, and constructed independent smoke-free reference renderings. This adds an end-to-end physical acquisition and evaluation study to the public real captures from SmokeSeer.

Section 4.7 and Figure 6 present our data alongside Bathroom and Red Container. Table 4 evaluates all 77 observed training views of our self-collected scene in common object bounding boxes. Ours improves PSNR from 20.10 to 20.48 dB and SSIM from 0.7895 to 0.8206, and reduces LPIPS from 0.2190 to 0.1773, a 19.0% reduction. The matched-view enlargements show smoke suppression and clearer surface structure. These results provide direct real-data validation in response to your request.

  1. The revised discussion explains why the components work.

We now organize the discussion around the mechanisms demonstrated by the ablations: thermal support strengthens the surface geometry, joint depth ordering handles surface–medium visibility, and reliability-guided recovery restores appearance detail while controlling residual transfer. The expanded parameter study, thermal-input perturbations, and repeated seeds provide additional evidence for these design choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.