acceptodds
Under review as a conference paper at ICLR 2027

RADIANT: Object-Centric Feed-Forward Global Illumination

13 views

Abstract

Feed-forward neural renderers predict physically grounded global illumination across unseen 3D scenes in a single forward pass, bypassing expensive Monte Carlo path tracing and per-scene optimization. However, existing methods rely on tokenizing explicit triangle meshes, causing attention complexity to scale quadratically with primitive count. Macroscopic light transport operates predominantly between surface patches and objects rather than diffusing facet by facet across individual triangles; polygon tessellation is an artifact of representation rather than transport physics. We introduce RADIANT (RADiance-based Inter-Actor Neural Transformer), an object-centric architecture that learns mesh-free feed-forward global illumination directly from point-sampled surfaces. By pooling surface points into a small number of patch tokens per object, RADIANT decouples attention complexity from polygon discretization. Light transport is modeled in two stages: an inter-object scene transformer and a view decoder. A factored radiometric attention bias injects directional alignment and material compatibility via rotary phases without explicit pairwise matrices, while an invariant form-factor extension incorporates pairwise geometric exchange invariants directly as -invariant inner products. To resolve primary visibility without mesh connectivity, camera rays cross-attend scene tokens guided by depth-tested point-splat hit channels in a shared spherical frame. Across 1.03M scenes under matched capacity, RADIANT achieves 25 dB in 5× fewer steps and surpasses RenderFormer by 3.0 dB (and RenderFormer-V2 by 7.2 dB) at 300k steps. On scenes with hundreds of objects, RADIANT avoids quadratic primitive bottlenecks and maintains interactive rendering rates. We will release our code and data upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Global Illumination, Neural Rendering, Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aL8ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RADIANT, an object-centric and feed-forward neural rendering framework for predicting global illumination from 3D scenes. Unlike prior approaches that represent scenes using triangle-level tokens, RADIANT operates directly on surface point clouds with known geometry and material attributes. It employs a PointNet-based encoder to aggregate surface samples into a fixed number of patch tokens per object, reducing the computational dependence on mesh complexity. A scene transformer models light transport between surface patches using object-oriented bounding-box rotary positional embeddingsand a proposed factored radiometric attention bias that incorporates material, emission, and geometric information. The paper also introduces an optional invariant form-factor attention mechanism to encode geometric relationships relevant to light transport. For view-dependent rendering, RADIANT uses point-splat rasterization to provide explicit surface visibility information, which is combined with scene features through cross-attention and a DPT decoder to generate HDR images. Experiments on a synthetic dataset of approximately 1.03 million scenes demonstrate improvements over RenderFormer and RenderFormer-V2 in rendering quality and training efficiency under matched experimental settings. The authors also report improved inference scalability with increasing scene complexity, particularly in terms of object and triangle counts.

Critical strengths
  1. Unlike the RenderFormer series, which relies on explicit mesh representations, RADIANT directly performs feed-forward global illumination rendering from attributed surface point clouds without requiring mesh connectivity. This is an interesting and potentially useful direction, as it could enable neural rendering directly from point-based 3D reconstructions without an intermediate mesh reconstruction step.

  2. The proposed method aggregates surface points into a compact set of patch tokens per object, making the complexity of scene attention independent of mesh tessellation. This provides a more scalable alternative to triangle-level representations, particularly for geometrically complex scenes.

  3. The experimental results demonstrate substantial improvements over RenderFormer and RenderFormer-V2 under matched training settings, achieving higher rendering quality with fewer optimization steps. The ablation studies also provide evidence supporting the effectiveness of the proposed attention mechanism and rasterizer-guided decoder.

Critical weaknesses
  1. The paper does not clearly establish why a mesh-free representation is needed or what advantages it offers over mesh-based approaches. More importantly, these potential benefits are not convincingly demonstrated in the experiments, leaving the practical value of the proposed mesh-free design unclear.

  2. The test scenes contain new object arrangements but reuse meshes from the training pool. Therefore, the experiments do not sufficiently demonstrate generalization to entirely unseen geometries, which is important for a generalizable neural renderer.

Questions for authors

Since the evaluation uses objects from the training mesh pool, could the authors provide results on entirely unseen meshes or object categories?

7 comments
AuthorsOpenReview ↗

Reply to Reviewer aL8R

[Part 1 of 3]

We sincerely thank Reviewer aL8R for the thoughtful review and for naming the use case that matters most for a mesh-free renderer: rendering "directly from point-based 3D reconstructions without an intermediate mesh reconstruction step". We are also grateful that you recognized that per-object patch tokens make scene attention independent of mesh tessellation, and for your positive assessment of our results and ablations. Both of your concerns are fair, and addressing them has strengthened the paper. Below are new test sets and direct comparisons, with the results that favor a mesh route reported alongside; all of them will be added to the revision.

Setup. Per-image PSNR at 5122512^2 with the 600k checkpoints; all models are scored on the same images (512 samples per pixel, four views per scene). There is no per-scene fine-tuning except in the fitting test of W1. RF-V2 denotes RenderFormer-V2; "released" marks the authors' public checkpoints.

W2 & Q: Does RADIANT generalize to unseen meshes and categories?

Thank you for this important question. You are right that our test scenes reused meshes from the training pool, so we built two new test sets. On new meshes and on categories absent from training, no model loses accuracy, and RADIANT keeps its lead. New assets: 250 scenes from 500 Objaverse assets that match no training mesh by identifier, file hash or geometry; 499 of them share an LVIS category with training. Absent categories: 250 scenes from 56 assets in 14 LVIS categories with no asset in training (LVIS has 19 such categories; we kept the assets our scene generator accepts).

new assetsabsent categories
RADIANT32.5932.49
RF (retrained)29.3629.27
RF-V2 (retrained)24.9025.00
RF (released)28.8228.44
RF-V2 (released)28.7928.44
RADIANT minus RF (retrained)+3.23+3.23 [2.92, 3.55]+3.22+3.22 [2.96, 3.49]
RADIANT minus RF-V2 (retrained)+7.69+7.69 [7.27, 8.10]+7.49+7.49 [7.12, 7.87]

On the held-out split RADIANT scores 32.21 dB, so new meshes cost no accuracy. From new assets to absent categories every change is within 0.4 dB and every 95% interval contains zero (the widest, for RF (released), spans −1.36-1.36 to +0.61+0.61 dB). We do not read this as generalization to arbitrary categories: there are only 14 categories, LVIS labels are noisy, close relatives remain in training (knives, boats, shirts), and the released checkpoints were trained on the RenderFormer authors' data, which may contain these assets.

W1: Why mesh-free, and is the benefit demonstrated?

We agree that the reviewed version did not make this case clearly enough, and we thank you for pushing us to test it directly. A mesh-free renderer matters when the input is not a mesh, such as a scan, or when a mesh's triangulation is arbitrary. We tested both, and the revision states the motivation explicitly.

(1) Partial scans, no fine-tuning: RADIANT on the raw scan is as accurate as RenderFormer on the best of three reconstructions, and better than on the other two.

We apply the paper's scan simulation to all 1,028 held-out scenes: foreground objects are scanned from one view and lose 60% of their points at the median (49–79% for the middle 90% of objects), while room surfaces and lights stay complete. RADIANT gets the scanned points directly. RenderFormer gets meshes reconstructed from exactly those points, with settings fixed in advance: screened Poisson at the library's default depth (a closed surface), the same surface trimmed to the scanned region, and ball pivoting. As a reference with the true geometry of the scanned faces, it also gets the true mesh cut down to the faces the scanner saw. No model is retrained.

inputPSNRLPIPSRADIANT minus (dB)
RADIANT, full point cloud (reference)32.210.042
RF, true mesh (reference)28.950.077
RADIANT, partial scan26.820.082—
RF, Poisson, library default21.820.172+5.00+5.00 [4.81, 5.19]
RF, Poisson trimmed to the scan26.910.094−0.09-0.09 [−0.20-0.20, 0.02]
RF, ball pivoting26.310.100+0.51+0.51 [0.42, 0.60]
RF, true mesh, scanned faces only26.590.098+0.23+0.23 [0.11, 0.34]

[Part 2 of 3]

Without any reconstruction, RADIANT on the raw scan is 0.51 dB above ball pivoting and 5.00 dB above the library's default closed surface. Even with the true mesh of the scanned faces, RenderFormer is 0.23 dB behind. The best reconstruction, Poisson trimmed to the scan, is level with RADIANT in PSNR and log-L1, and RADIANT is 0.012 better in LPIPS. That parity depends on the reconstruction settings: RenderFormer's PSNR moves by up to 2.9 dB across trimming radii, while RADIANT takes the points as they are, with nothing to tune. Given the same surface, RADIANT on points sampled from the trimmed mesh is 0.16 dB [0.11, 0.21] above RenderFormer on that mesh (added after the first results), so the 0.09 dB that the trimmed reconstruction gains on raw scans comes from its hole filling, which either renderer could use. Poisson reconstruction takes about 0.5 s per object on one CPU thread and failed on 2 of 3,285 objects, both with almost no scanned points.

(2) With a few reference views the two routes separate: RADIANT leads by about 2.5–2.8 dB.

Added after the zero-shot results, on 30 random scenes, we fit both models on two views of each scene and score two views never used for fitting. Both use the paper's fitting settings, tuned for neither; RenderFormer may fit better with other settings, which we have not tried.

unseen views, after fittingPSNRchange from zero-shot (dB)
RADIANT, partial scan29.63+2.38+2.38
RF, Poisson trimmed to the scan26.88−0.95-0.95
RADIANT minus RF+2.75+2.75 [2.12, 3.42]

RADIANT is higher on 93% of these images. Fitting on GPUs is not deterministic: a repeat of the same fits gives a gap of 2.48 dB, so the gain is about 2.5–2.8 dB.

(3) Sensor effects: estimated normals and depth noise cost both models less than 0.5 dB.

A real sensor adds depth noise and needs estimated normals, so we repeated the zero-shot comparison with normals estimated from the points and noise along the scanner's line of sight, at levels fixed in advance:

inputRADIANTRF, trimmed PoissonRADIANT minus RF (dB)
exact points, true normals26.8226.91−0.09-0.09 [−0.20-0.20, 0.02]
exact points, estimated normals26.6726.70−0.03-0.03 [−0.13-0.13, 0.07]
noise 0.25% of the object diagonal26.6826.69−0.01-0.01 [−0.11-0.11, 0.09]
noise 1% of the object diagonal26.6326.48+0.15+0.15 [0.06, 0.25]

The two are level with estimated normals and at the lower noise level, and RADIANT is slightly ahead at the highest level tested, about one sample spacing. Outliers, missing returns from dark or specular surfaces, multi-view registration error and real scanner data remain untested.

[Part 3 of 3]

(4) Tessellation: under subdivision RADIANT's change stays within resampling noise, while every mesh baseline loses 4.4–15.6 dB, mostly from the room walls.

Real meshes come with arbitrary triangulations, so a renderer should depend on the shape of a surface, not on how it is cut into triangles. This test keeps every surface fixed and changes only its triangulation. RADIANT samples its points from the surface, so under subdivision its input keeps the same distribution; a renderer whose tokens are triangles sees a different input for the same surface. On the new-asset scenes we split every triangle along its longest edge, once, twice or four times (2×2\times, 4×4\times and 16×16\times as many triangles). The new triangles lie inside the old ones and the new vertex normals are interpolated from the old ones, so the surface, its face normals and its shading are unchanged. We also decimate the objects to 0.25×0.25\times their triangles by quadric edge collapse; this changes the shapes, so the targets for that row are rendered again from the decimated meshes. All five models use their trained weights; RADIANT samples its points from the same meshes. Change in PSNR from each model's own original tessellation (dB):

RADIANTRF (retrained)RF-V2 (retrained)RF (released)RF-V2 (released)
subdivided 2×2\times−0.06-0.06−7.19-7.19−4.36-4.36−7.27-7.27−7.73-7.73
subdivided 4×4\times−0.02-0.02−9.43-9.43−6.79-6.79−9.51-9.51−5.64-5.64
subdivided 16×16\times+0.05+0.05−15.62-15.62−9.15-9.15−13.08-13.08−5.46-5.46
objects only, 4×4\times−0.01-0.01−0.16-0.16−0.22-0.22−0.96-0.96+0.72+0.72
objects only, 16×16\times−0.02-0.02−2.63-2.63−2.24-2.24−7.88-7.88+0.60+0.60
room walls only, 2×2\times−0.07-0.07−7.34-7.34−4.41-4.41−7.12-7.12−7.83-7.83
objects decimated to 0.25×0.25\times+0.24+0.24−0.94-0.94−0.54-0.54−0.40-0.40−1.70-1.70

Under subdivision RADIANT's change is within resampling noise at every level (each 95% interval contains zero), as expected, since its points are sampled from an unchanged surface; under full-scene subdivision every mesh baseline loses 4.4–15.6 dB. The object-only and wall-only rows, added after the first results to find the cause, show that most of this loss comes from the room walls, whose triangulation never varies in training (our corpus and the authors' templates share the same wall mesh). With only the objects subdivided 4×4\times, the retrained baselines change by less than 0.3 dB, RF (released) loses 1.0 dB and RF-V2 (released) gains 0.7 dB; at 16×16\times the other three lose 2.2–7.9 dB, while RF-V2 (released) still gains 0.6 dB. So we do not claim that mesh-based renderers are unstable on objects in general. A model on mesh tokens inherits the triangulations of its training data, as the walls show. Latency moves with the triangle count: at the original tessellation the retrained baselines are faster (30–39 against 73 ms per view for RADIANT), at 4×4\times RADIANT is slightly faster than RenderFormer (74 against 82–95 ms) while RF-V2 (retrained) is still faster (47 ms), and at 16×16\times RADIANT is 4.2–10.6$\times$ faster.

We are truly grateful for your thoughtful review. Your first question made us test mesh-free input against reconstruct-then-render; the revision reports what it shows, including where a mesh route matches it, and it will include all of these experiments. If another test would show the benefit you had in mind more clearly, we would be glad to run it : )

Reviewer_eS5tOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper improves over RenderFormer V1/V2 by shifting the triangle-level tokenization into a mesh-free per-object sampled surface point tokenization. G-buffers from point splatting is also provided to the model. Additional inductive biases derived from light-transportation like factored radiometric attention biases and invariant form-factor features are also added. The proposed system achieved higher quality and better training efficiency.

Critical strengths
  1. The mesh-free design made the system not only more efficient but also decoupled from the mesh's subdivision pattern, decoupling the complexity of light transport from scene representation. It also enabled the model to do more extensions, like partial point cloud rendering.
  2. With the new designs, the system is substantially improved in training efficiency and rendering quality over RenderFormer V1/V2.

Other strengths like adding the gbuffers as hint and adding more rendering priors are helpful and could be recognized as important strengths.

Critical weaknesses
  1. As the method already moved to point-based tokenization, it is unclear if the concept of "object" still make sense. This could be a limitation that the model requires explicit modeling of "object"s, and it's unclear to me why the model cannot take all the sampled points in the scene as a whole. If it's due to pointnet's limitation that it could not generalized due to the pooling mechanism, maybe directly partitioning the scene according to spatial coordinates could also work.
  2. The final model is not using the optimal setting in the ablation study. There are also some misalignment with baselines in ablation/comparison, which made the improvement hard to attribute. I will list them in details in the questions for authors.
Questions for authors
  1. The RF-v2 comparison might be incorrect. Both RF-V1/V2 used DPT as decoder (not the conv2d...). The released RF-V2 has used sliding window through flex-attention. So the author's re-training could be wrong, which explains why RF-V2's training curve is bad in the paper. The authors also omit the curriculum training in RF-V2 which made the comparison unfair.
  2. The camera patch tokens used in this paper is smaller (4x4) than RF-V1/V2, which introduced an unfair advantage of the paper's method. A new comparison by setting RF's camera patch to 4x4 or setting RADIANT's to 8x8 is needed to avoid this misalignment.
  3. The ablations of camray embedding and spherical RoPE also had the gbuffers dropped, which made the ablation unable to attribute any conclusions.
  4. The final model config is not the optimal one. From tab16, OBB-RoPE, routing bias are not helping. The helpful invariant form-factor attention is not included in the final model, probably because the authors did not do a thorough ablation before launching the final training.
7 comments
AuthorsOpenReview ↗

Reply to Reviewer eS5t

[Part 1 of 2]

We sincerely thank Reviewer eS5t for an expert review that goes into the details of the baselines and the ablations. We are grateful that you recognized that the mesh-free design decouples "the complexity of light transport from scene representation" and enables partial point cloud rendering, and that the G-buffer hints and rendering priors are helpful. Your detailed questions led to several concrete corrections. Each answer below starts with what changed; results still running are marked and will follow in this thread, and all of it will be included in the revision.

Setup. Comparisons use 600k checkpoints at 5122512^2. Ablations use the paper's ablation protocol (2562256^2, 100k steps, three seeds) on the final configuration, which includes invariant form-factor attention. RF-V2 denotes RenderFormer-V2; "released" marks the authors' public checkpoints.

Q1: Is the RF-V2 comparison aligned with the authors' setup (decoder, sliding window, curriculum)?

Thank you for this careful check of the comparison. You raise three valid points, and we address each below.

(1) Decoder: both RenderFormer versions use DPT, and so did our runs. You are right, and we apologize: "Conv2D" in App. H was our error, and it has been corrected.

(2) Sliding window: the public training code trains with dense attention, and retraining on the window path does not help so far.

RF-V2 as released does run a sliding window at inference, but the public training code does not train with it. With use_volumes: true, which all eight released recipes use, train_rf2.py (line 335) takes the packed path, where use_local_attention: false sets the window to −1-1 (attention.py), so training attention is dense. The ±256\pm256-token window with attention sinks acts only on the unpacked path, which the inference documentation calls the release profile. Our retraining used the public training code with the released model configuration, so it trained densely, as the released recipes do; we score it in that path, and the released weights in their release profile. The released weights score higher in the release profile than on the packed path (28.43 against 26.61 dB on the paper's held-out images), which suggests that the RenderFormer authors trained them at least partly with the window, although their public training recipes do not use it.

We are grateful for this pointer, and we are therefore also retraining RF-V2 from scratch on the sliding-window path, following the authors' released input conventions, at the same budget. On the paper's held-out images it reaches 21.91 dB at 100k and 23.01 dB at 300k steps, against 22.06 and 23.43 dB for RF-V2 (retrained) at the same steps. We will post 600k here when it finishes.

(3) Curriculum: RADIANT stays ahead of both released checkpoints, which carry the authors' curricula; retraining with those curricula on our corpus is running.

We agree that omitting the authors' curricula leaves the comparison open, and we thank you for raising it. We did not use them: they train RenderFormer for 1.3M steps at 2562256^2 plus 100k at 5122512^2, and RF-V2 in eight stages of 300k, while we trained all models from scratch at 5122512^2 for 600k. First, the released checkpoints, evaluated with their released inference pipeline (each scene mapped to the authors' coordinate frame), paired per image with a scene bootstrap:

RADIANT minusheld-out split25–50 entities
RF (released)+3.91+3.91 [3.63, 4.20]+7.48+7.48 [7.08, 7.88]
RF-V2 (released)+3.78+3.78 [3.49, 4.07]+1.76+1.76 [1.42, 2.11]

At about 100 entities RADIANT is level with RF-V2 (released) in mean PSNR (our reply to Reviewer Jdy6 has the table). These checkpoints were trained on the authors' data, not on our corpus, so they are a reference rather than a matched comparison.

Second, we are retraining with the authors' curricula on our corpus at the same number of optimizer steps (600k): RenderFormer 500k steps at 2562256^2 followed by their 100k at 5122512^2; RF-V2 their eight stages at 75k steps each, keeping their resolutions, two views per scene and augmentation. Our batches are smaller than theirs (a quarter, and an eighth in RF-V2's 5122512^2 stages), so the number of training images differs and their learning rates are scaled down in the same proportion; every difference from their recipes will be listed with the result. Both retrainings are running, and we will post their results here when they finish.

[Part 2 of 2]

Q2: Does the camera patch size, rather than the scene representation, explain the gain?

Thank you for this suggestion; we agree that the patch size needed a direct control. At RenderFormer's ray-token budget RADIANT stays 2.42 dB ahead; patch size accounts for 0.83 of the 3.26 dB gap. We ran the control with RADIANT at 8×88\times8 and 5122512^2 for the full 600k steps (details in our reply to Reviewer Jdy6): RADIANT reaches 31.38 dB against 32.21 dB at 4×44\times4 and 28.95 dB for RF (retrained). The other option, RenderFormer at 4×44\times4 (16,384 ray tokens), is much more expensive: on one A100 at batch 3 a step takes 4.94 s against 0.87 s at 8×88\times8, about seven days per 100k steps on eight GPUs, too long for this discussion. The 8×88\times8 run already puts both models at the same resolution and ray-token budget, so we did not run it.

Q3: The CamRay and spherical RoPE ablations also removed the G-buffers.

Thank you for catching this; you are right. With the G-buffers also removed, those rows measured a different model. The revision removes them and no longer claims an effect for CamRay.

Q4 & W2: Which configuration is the final model, and which components does it claim?

You are right that the final model in the reviewed version was not the best configuration, and we thank you for pointing this out. The final model now includes invariant form-factor attention, and only components with a measured effect are claimed. It reaches 32.21 dB at 600k, and removing invariant attention costs 0.52±0.080.52 \pm 0.08 dB (three seeds). On that configuration, removing OBB-RoPE changes PSNR by −0.08±0.04-0.08 \pm 0.04 dB, replacing it by centroid RoPE by −0.04±0.04-0.04 \pm 0.04 dB, and removing routing alone by −0.04±0.05-0.04 \pm 0.05 dB, so these contribute little, and we do not claim them as contributions.

W1: Does "object" still make sense with point tokens?

(1) Grouping by object beats grouping the same points by position (mean 0.30 dB), and one group for the whole scene costs 1.05 dB. Thank you for this thoughtful question and for suggesting two concrete tests. We ran both on the final configuration, three seeds each:

  • Spatial partition. All sampled points of a scene are split by coordinates alone, with a balanced k-d tree, into as many groups as the scene has objects. The model sees the same points and the same token count as the full model, and every per-object quantity (OBB axes, FiLM and radiosity statistics, rasterizer identity, routing) is computed per group.
  • Whole scene. One group of 320 tokens and 20,480 points, the size of a median training scene of ten entities.

The decision rule is fixed in advance: a row has an effect only if all three paired differences share a sign and exceed the full model's seed standard deviation (0.011 dB). With the spatial partition, PSNR falls by 0.36, 0.28 and 0.27 dB on the three seeds (mean 0.30 dB, each paired with the full model of the same seed). With one group for the whole scene it falls by 1.07, 1.10 and 0.99 dB (mean 1.05 dB). Scope: points are still sampled per object, so groups still follow objects in part (73% of a group's points come from one object, and an object spans 2.5 groups on average). The partition tests whether object labels are needed, not whether per-object sampling is; the one-group row tests this architecture without objects, not a model designed for whole scenes.

(2) Objects are physically coherent parts of a scene. An object has one material and one rigid shape, so its tokens summarize something physically coherent: its orientation, its material and the light it receives. Objects also split a scene into the same parts however the scene is rotated, while a k-d tree split along coordinate axes changes with the rotation and cuts across objects, mixing unrelated surfaces in one summary. We think this part-level structure is one reason the model generalizes to new scenes; the drop under the spatial partition fits that view, although the experiment does not separate it from other causes.

We are sincerely grateful for your expert and detailed review. Each of your questions changed something concrete: the RF-V2 comparison, the ablation tables and the list of claimed contributions, and all of it will be included in the revision. If any comparison still looks misaligned, we would be glad to fix it while the discussion is open.

Dear Reviewer eS5t,

Many thanks again for your expert review. Your detailed questions were especially helpful to us. We know your time is limited, so even a brief note on whether our answers match what you had in mind would be very valuable. Should anything still look off, we are happy to look into it further before the discussion period ends: )

Reviewer_Jdy6OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RADIANT proposes a mesh-free, object-centric feed-forward renderer for HDR global illumination from object-wise surface samples. It compresses each object into a fixed number of patch tokens, models scene interactions through factored radiometric attention, and uses rasterized G-buffer information to guide view-dependent image reconstruction. The paper also introduces an invariant form-factor attention extension. Experiments on synthetic scenes demonstrate improved rendering accuracy over RenderFormer and RenderFormer-V2 under matched training settings. Ablation studies and additional experiments evaluate component contributions, inference efficiency and scaling, and sensitivity to incomplete or corrupted geometry. The invariant attention extension is evaluated separately at a lower resolution.

Critical strengths
  1. A compact object-centric scene representation. Aggregating surface samples into a fixed number of patch tokens per object is a useful architectural choice. It removes the need for mesh connectivity and decouples scene-attention token count from mesh tessellation, addressing a practical scalability challenge in neural rendering.
  2. Strong empirical performance under a controlled evaluation protocol. RADIANT achieves substantial rendering-quality improvements over the tested RenderFormer baselines under matched training settings and comparable model sizes, outperforming RenderFormer by 3.0 dB and RenderFormer-V2 by 7.2 dB in PSNR. The paper also provides learning-efficiency comparisons, multi-seed ablations, and statistical confidence intervals. These results offer convincing evidence of the effectiveness of the overall system, even though the contributions of individual design choices to the cross-method gains are not fully isolated.
Critical weaknesses
  1. The decoder-budget control is confounded by image resolution. Section 4.3 and Appendix H compare RADIANT at 256 × 256 with RenderFormer at 512 × 512 to match 4,096 ray tokens. This changes the prediction task and does not match tokens per output pixel, so it cannot establish that the gains are independent of decoder token density. Table 18 convincingly supports the radiometric bias, but addresses a different question. This weakens my confidence in the paper’s architectural explanation of the cross-model gains: the contribution of decoder token allocation remains unresolved, limiting how confidently those gains can be attributed to the proposed transport representation.
  2. Rendering quality is untested at the scale of the reported speedups. Table 4 evaluates scenes with 5–18 entities, whereas the 50–500-object timings in Appendix E use random weights. These measurements establish computational scaling, but not the rendering accuracy of the trained model in larger scenes. Although the paper acknowledges this limitation, it does not provide joint quality–latency evaluation beyond the training range. This is my principal concern because efficient rendering of complex scenes is a central contribution, yet the evidence does not establish whether the reported computational advantage translates into useful rendering quality at those scales.
Questions for authors
  1. Clarification of the baseline implementation. Appendix F describes RenderFormer with a DPT decoder, whereas Appendix H specifies Conv2D. Which implementation produced the main comparison results, and were the same decoder settings used throughout the reported controls?
  2. Clarification of inconsistent ablation results. Do Tables 17 and 21 use identical checkpoints and evaluation protocols? The full-model LPIPS differs (0.095/0.091), as do the “No rasterizer channels” LPIPS (0.167/0.155) and log-L1 (0.046/0.052). Table 21 also contains “[eval]” entries whose counterparts have results in Table 17. Which values are authoritative, and does reconciling these discrepancies change the conclusions about visibility guidance and scene-token contributions?
  3. Additional evidence on decoder token density. Can you provide a comparison at a common output resolution and ray-token budget, with the training protocol held fixed? This would test the token-density claim without the resolution change in Appendix H.
  4. Additional evidence on large-scene performance. Can you report rendering quality and latency for the trained model on the same scenes beyond the training range, for example 30–50 entities, without per-scene fine-tuning? This would establish whether the computational scaling translates into useful rendering quality.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer Jdy6

[Part 1 of 3]

We sincerely thank Reviewer Jdy6 for the careful and constructive review. Your two key questions go to the heart of our claims, and answering them has made the paper clearly better. We are also grateful that you found aggregating surface samples into a fixed number of patch tokens useful because it "removes the need for mesh connectivity and decouples scene-attention token count from mesh tessellation". Below we answer each point with new experiments, including the results that favor a baseline, and we will add all of them to the revision.

Setup for all numbers below. Per-image PSNR at 5122512^2 with no per-scene fine-tuning. RADIANT and the retrained baselines are scored at 600k steps: the reviewed version reported 300k, all three models were continued to 600k alike, and RADIANT now includes invariant form-factor attention. RF-V2 denotes RenderFormer-V2; "released" marks the public checkpoints of the RenderFormer authors, trained with their own curricula on their own data and evaluated with their released inference pipeline. As a check, our evaluation reproduces the paper's held-out result for RADIANT (32.21 dB).

W1 & Q3: The decoder-budget control changes the output resolution (2562256^2 against 5122512^2), so it cannot show that the gain is independent of decoder token density.

Thank you very much for this insightful comment. You are right: comparing 2562256^2 with 5122512^2 changes the prediction task and does not match tokens per output pixel. We have removed that claim, and we ran exactly the control you suggested.

(1) New control at the same resolution and ray-token budget: ray-token density explains 0.83 of the 3.26 dB gap, so about three quarters of the gap remains.

We trained RADIANT with 8×88\times8 camera patches at 5122512^2, which gives 4,096 ray tokens, the output resolution and ray-token budget of RenderFormer, with the training protocol unchanged (same corpus, global batch, learning-rate schedule and 600k steps). The reading was fixed before the result: whatever part of the gap to RenderFormer the 8×88\times8 model loses is the share due to ray-token density. Held-out PSNR on the paper's 4,112 images:

model (5122512^2)ray tokens100k300k600k
RADIANT 4×44\times416,38429.9531.1432.21
RADIANT 8×88\times8 (new)4,09629.3730.4231.38
RF (retrained)4,09625.5027.6228.95

At RenderFormer's own token budget RADIANT stays 2.42 dB ahead, and it leads at every training length.

(2) What remains. RADIANT 8×88\times8 and RenderFormer now share resolution, ray-token budget, DPT decoder, data and schedule. They still differ in the scene representation and in RADIANT's rasterized hit channels, which RenderFormer lacks; within RADIANT, removing those channels costs 3.12±0.083.12 \pm 0.08 dB (Q2). Within RADIANT the scene representation matters even more: removing the scene tokens costs 6.19±0.026.19 \pm 0.02 dB (Q2), and restricting scene attention to tokens of the same object costs 2.46±0.662.46 \pm 0.66 dB (six seeds, Sec. 4.3). These ablations show how much RADIANT relies on its own transport representation, although they do not by themselves compare it with RenderFormer's. We have not given RenderFormer hit channels, so the split of the cross-model gain between the two is not isolated. We agree that this matters for attributing the gain, thank you for prompting this analysis, and the revision states it explicitly.

[Part 2 of 3]

W2 & Q4: Rendering quality is untested at the scales of the reported speedups; quality and latency of trained models beyond the training range, without per-scene fine-tuning.

We fully agree that this is the most important question for a method whose central claim is efficient rendering of complex scenes, and we are grateful that you pressed on it. Here is the evaluation you asked for; it will be added to the revision.

(1) Quality: RADIANT leads both retrained baselines at every size, with no fine-tuning.

The new sets come from the scene generator of the corpus with only the entity count changed, objects scaled to fill the room as in an average training scene: 280 scenes with 25–50 entities (reported in the revised paper, Tab. 4) and 60 scenes with about 100 entities (rendered for this discussion; 512 samples per pixel, four views per scene). Training scenes have at most 18 entities (median 10). PSNR / LPIPS:

entitiesRADIANTRF (retrained)RF-V2 (retrained - still on going;)RF (released)RF-V2 (released)
25–5025.32 / 0.09522.50 / 0.17120.06 / 0.22717.83 / 0.23323.56 / 0.129
30–5025.15 / 0.09722.35 / 0.17419.92 / 0.23117.25 / 0.24723.56 / 0.130
46–5024.69 / 0.10422.12 / 0.18219.60 / 0.24215.22 / 0.29423.08 / 0.141
about 10024.36 / 0.11320.26 / 0.23218.32 / 0.28111.37 / 0.50424.57 / 0.123

(2) Latency: the speedups of App. E hold for trained models. Median milliseconds per view on the same scenes, batch size 1, one H100:

entitiesRADIANTRF (FlashAttention)RF (SDPA)RF-V2 (release profile)
25–5087204329127
30–5089223366134
46–5092300528164
about 1001359792,100364

At about 100 entities RADIANT is 7.2×7.2\times faster than RenderFormer with FlashAttention and 15.5×15.5\times with SDPA. Against RF-V2 in its release profile, whose attention is windowed, it is 1.45×1.45\times faster at 25–50 and 2.7×2.7\times at about 100, so the revision limits the quadratic-cost statement to dense triangle attention. The cost is GPU memory: 4.4 GiB against 2.0–2.4 GiB at 25–50 entities.

(3) Released checkpoints: ahead at 25–50 entities, level in mean quality at about 100 while 2.7× faster. RADIANT minus RF-V2 (released), paired per image with a scene bootstrap:

25–50about 100
mean [95% interval] (dB)+1.76+1.76 [1.42, 2.11]−0.21-0.21 [−0.95-0.95, 0.53]
median image (dB)+0.20+0.20−1.10-1.10
images where RADIANT is higher61%30%

On 25–50 entities much of the mean lead comes from images where RF-V2 (released) fails; the median image differs by only 0.20 dB. At about 100 the means are level within the interval, but RF-V2 (released) is higher on most images. RF (released) degrades sharply beyond 50 entities in its own pipeline (11.37 dB at about 100). These checkpoints were trained on the authors' data with their curricula, so they are a reference rather than a matched comparison; we report them in full, including where RF-V2 (released) is better.

[Part 3 of 3]

Q1: Which decoder produced the main comparison, and was it the same in every control?

Thank you for catching this inconsistency, and we apologize for it. RenderFormer used the DPT decoder in the main comparison and in every control, as all of its training configurations show. The description in App. H was our error, and it has been corrected.

Q2: Tables 17 and 21: which values are authoritative, and does resolving the differences change the conclusions?

Thank you very much for this careful reading, and we sincerely apologize for the confusion. No conclusion changes. Both tables describe the same 25k-step, seed-42 runs on the same 2,948 images, and their PSNR values agree. Table 21's "[eval]" rows are Table 17's "No rasterized splat", "No rasterizer channels or splat" and "No scene tokens". Where Table 21's LPIPS and log-L1 differ from Table 17, they match none of our evaluation records, so we withdraw them: Table 17 is authoritative and Table 21 has been removed (table numbers here are those of the reviewed version; the revision reports these ablations at 100k steps with three seeds in Tabs. 19–21). We have checked every table of the revision against our evaluation records.

Both components matter at every training length; only their relative size changes. At 25k steps (one seed), removing the rasterizer channels costs 4.07 dB and removing the scene tokens 3.68 dB; at 100k steps (three seeds), 3.12±0.053.12 \pm 0.05 and 5.66±0.065.66 \pm 0.06 dB. On the final configuration (with invariant attention; 2562256^2, 100k steps, three seeds), removing the rasterizer channels costs 3.12±0.083.12 \pm 0.08 dB, removing the scene tokens 6.19±0.026.19 \pm 0.02 dB, and removing the rasterizer channels together with the splat 3.19±0.083.19 \pm 0.08 dB, about the same as removing the rasterizer channels alone.

We are truly grateful for your detailed and constructive feedback, which led directly to the matched-budget control and the large-scene evaluation. We will incorporate all of these results and clarifications into the revision. If they address your two concerns, we would be sincerely grateful if you would reconsider your assessment, and we would be very glad to run any further test you suggest.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.