- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RADIANT, an object-centric and feed-forward neural rendering framework for predicting global illumination from 3D scenes. Unlike prior approaches that represent scenes using triangle-level tokens, RADIANT operates directly on surface point clouds with known geometry and material attributes. It employs a PointNet-based encoder to aggregate surface samples into a fixed number of patch tokens per object, reducing the computational dependence on mesh complexity. A scene transformer models light transport between surface patches using object-oriented bounding-box rotary positional embeddingsand a proposed factored radiometric attention bias that incorporates material, emission, and geometric information. The paper also introduces an optional invariant form-factor attention mechanism to encode geometric relationships relevant to light transport. For view-dependent rendering, RADIANT uses point-splat rasterization to provide explicit surface visibility information, which is combined with scene features through cross-attention and a DPT decoder to generate HDR images. Experiments on a synthetic dataset of approximately 1.03 million scenes demonstrate improvements over RenderFormer and RenderFormer-V2 in rendering quality and training efficiency under matched experimental settings. The authors also report improved inference scalability with increasing scene complexity, particularly in terms of object and triangle counts.
Critical strengths
-
Unlike the RenderFormer series, which relies on explicit mesh representations, RADIANT directly performs feed-forward global illumination rendering from attributed surface point clouds without requiring mesh connectivity. This is an interesting and potentially useful direction, as it could enable neural rendering directly from point-based 3D reconstructions without an intermediate mesh reconstruction step.
-
The proposed method aggregates surface points into a compact set of patch tokens per object, making the complexity of scene attention independent of mesh tessellation. This provides a more scalable alternative to triangle-level representations, particularly for geometrically complex scenes.
-
The experimental results demonstrate substantial improvements over RenderFormer and RenderFormer-V2 under matched training settings, achieving higher rendering quality with fewer optimization steps. The ablation studies also provide evidence supporting the effectiveness of the proposed attention mechanism and rasterizer-guided decoder.
Critical weaknesses
-
The paper does not clearly establish why a mesh-free representation is needed or what advantages it offers over mesh-based approaches. More importantly, these potential benefits are not convincingly demonstrated in the experiments, leaving the practical value of the proposed mesh-free design unclear.
-
The test scenes contain new object arrangements but reuse meshes from the training pool. Therefore, the experiments do not sufficiently demonstrate generalization to entirely unseen geometries, which is important for a generalizable neural renderer.
Questions for authors
Since the evaluation uses objects from the training mesh pool, could the authors provide results on entirely unseen meshes or object categories?
7 comments
[Part 2 of 3]
Without any reconstruction, RADIANT on the raw scan is 0.51 dB above ball pivoting and 5.00 dB above the library's default closed surface. Even with the true mesh of the scanned faces, RenderFormer is 0.23 dB behind. The best reconstruction, Poisson trimmed to the scan, is level with RADIANT in PSNR and log-L1, and RADIANT is 0.012 better in LPIPS. That parity depends on the reconstruction settings: RenderFormer's PSNR moves by up to 2.9 dB across trimming radii, while RADIANT takes the points as they are, with nothing to tune. Given the same surface, RADIANT on points sampled from the trimmed mesh is 0.16 dB [0.11, 0.21] above RenderFormer on that mesh (added after the first results), so the 0.09 dB that the trimmed reconstruction gains on raw scans comes from its hole filling, which either renderer could use. Poisson reconstruction takes about 0.5 s per object on one CPU thread and failed on 2 of 3,285 objects, both with almost no scanned points.
(2) With a few reference views the two routes separate: RADIANT leads by about 2.5–2.8 dB.
Added after the zero-shot results, on 30 random scenes, we fit both models on two views of each scene and score two views never used for fitting. Both use the paper's fitting settings, tuned for neither; RenderFormer may fit better with other settings, which we have not tried.
| unseen views, after fitting | PSNR | change from zero-shot (dB) |
|---|---|---|
| RADIANT, partial scan | 29.63 | |
| RF, Poisson trimmed to the scan | 26.88 | |
| RADIANT minus RF | [2.12, 3.42] |
RADIANT is higher on 93% of these images. Fitting on GPUs is not deterministic: a repeat of the same fits gives a gap of 2.48 dB, so the gain is about 2.5–2.8 dB.
(3) Sensor effects: estimated normals and depth noise cost both models less than 0.5 dB.
A real sensor adds depth noise and needs estimated normals, so we repeated the zero-shot comparison with normals estimated from the points and noise along the scanner's line of sight, at levels fixed in advance:
| input | RADIANT | RF, trimmed Poisson | RADIANT minus RF (dB) |
|---|---|---|---|
| exact points, true normals | 26.82 | 26.91 | [, 0.02] |
| exact points, estimated normals | 26.67 | 26.70 | [, 0.07] |
| noise 0.25% of the object diagonal | 26.68 | 26.69 | [, 0.09] |
| noise 1% of the object diagonal | 26.63 | 26.48 | [0.06, 0.25] |
The two are level with estimated normals and at the lower noise level, and RADIANT is slightly ahead at the highest level tested, about one sample spacing. Outliers, missing returns from dark or specular surfaces, multi-view registration error and real scanner data remain untested.
[Part 3 of 3]
(4) Tessellation: under subdivision RADIANT's change stays within resampling noise, while every mesh baseline loses 4.4–15.6 dB, mostly from the room walls.
Real meshes come with arbitrary triangulations, so a renderer should depend on the shape of a surface, not on how it is cut into triangles. This test keeps every surface fixed and changes only its triangulation. RADIANT samples its points from the surface, so under subdivision its input keeps the same distribution; a renderer whose tokens are triangles sees a different input for the same surface. On the new-asset scenes we split every triangle along its longest edge, once, twice or four times (, and as many triangles). The new triangles lie inside the old ones and the new vertex normals are interpolated from the old ones, so the surface, its face normals and its shading are unchanged. We also decimate the objects to their triangles by quadric edge collapse; this changes the shapes, so the targets for that row are rendered again from the decimated meshes. All five models use their trained weights; RADIANT samples its points from the same meshes. Change in PSNR from each model's own original tessellation (dB):
| RADIANT | RF (retrained) | RF-V2 (retrained) | RF (released) | RF-V2 (released) | |
|---|---|---|---|---|---|
| subdivided | |||||
| subdivided | |||||
| subdivided | |||||
| objects only, | |||||
| objects only, | |||||
| room walls only, | |||||
| objects decimated to |
Under subdivision RADIANT's change is within resampling noise at every level (each 95% interval contains zero), as expected, since its points are sampled from an unchanged surface; under full-scene subdivision every mesh baseline loses 4.4–15.6 dB. The object-only and wall-only rows, added after the first results to find the cause, show that most of this loss comes from the room walls, whose triangulation never varies in training (our corpus and the authors' templates share the same wall mesh). With only the objects subdivided , the retrained baselines change by less than 0.3 dB, RF (released) loses 1.0 dB and RF-V2 (released) gains 0.7 dB; at the other three lose 2.2–7.9 dB, while RF-V2 (released) still gains 0.6 dB. So we do not claim that mesh-based renderers are unstable on objects in general. A model on mesh tokens inherits the triangulations of its training data, as the walls show. Latency moves with the triangle count: at the original tessellation the retrained baselines are faster (30–39 against 73 ms per view for RADIANT), at RADIANT is slightly faster than RenderFormer (74 against 82–95 ms) while RF-V2 (retrained) is still faster (47 ms), and at RADIANT is 4.2–10.6$\times$ faster.
We are truly grateful for your thoughtful review. Your first question made us test mesh-free input against reconstruct-then-render; the revision reports what it shows, including where a mesh route matches it, and it will include all of these experiments. If another test would show the benefit you had in mind more clearly, we would be glad to run it : )
Reply to Reviewer aL8R
[Part 1 of 3]
We sincerely thank Reviewer aL8R for the thoughtful review and for naming the use case that matters most for a mesh-free renderer: rendering "directly from point-based 3D reconstructions without an intermediate mesh reconstruction step". We are also grateful that you recognized that per-object patch tokens make scene attention independent of mesh tessellation, and for your positive assessment of our results and ablations. Both of your concerns are fair, and addressing them has strengthened the paper. Below are new test sets and direct comparisons, with the results that favor a mesh route reported alongside; all of them will be added to the revision.
Setup. Per-image PSNR at 5122 with the 600k checkpoints; all models are scored on the same images (512 samples per pixel, four views per scene). There is no per-scene fine-tuning except in the fitting test of W1. RF-V2 denotes RenderFormer-V2; "released" marks the authors' public checkpoints.
W2 & Q: Does RADIANT generalize to unseen meshes and categories?
Thank you for this important question. You are right that our test scenes reused meshes from the training pool, so we built two new test sets. On new meshes and on categories absent from training, no model loses accuracy, and RADIANT keeps its lead. New assets: 250 scenes from 500 Objaverse assets that match no training mesh by identifier, file hash or geometry; 499 of them share an LVIS category with training. Absent categories: 250 scenes from 56 assets in 14 LVIS categories with no asset in training (LVIS has 19 such categories; we kept the assets our scene generator accepts).
On the held-out split RADIANT scores 32.21 dB, so new meshes cost no accuracy. From new assets to absent categories every change is within 0.4 dB and every 95% interval contains zero (the widest, for RF (released), spans −1.36 to +0.61 dB). We do not read this as generalization to arbitrary categories: there are only 14 categories, LVIS labels are noisy, close relatives remain in training (knives, boats, shirts), and the released checkpoints were trained on the RenderFormer authors' data, which may contain these assets.
W1: Why mesh-free, and is the benefit demonstrated?
We agree that the reviewed version did not make this case clearly enough, and we thank you for pushing us to test it directly. A mesh-free renderer matters when the input is not a mesh, such as a scan, or when a mesh's triangulation is arbitrary. We tested both, and the revision states the motivation explicitly.
(1) Partial scans, no fine-tuning: RADIANT on the raw scan is as accurate as RenderFormer on the best of three reconstructions, and better than on the other two.
We apply the paper's scan simulation to all 1,028 held-out scenes: foreground objects are scanned from one view and lose 60% of their points at the median (49–79% for the middle 90% of objects), while room surfaces and lights stay complete. RADIANT gets the scanned points directly. RenderFormer gets meshes reconstructed from exactly those points, with settings fixed in advance: screened Poisson at the library's default depth (a closed surface), the same surface trimmed to the scanned region, and ball pivoting. As a reference with the true geometry of the scanned faces, it also gets the true mesh cut down to the faces the scanner saw. No model is retrained.