acceptodds
Under review as a conference paper at ICLR 2027

TriRender: Scaling Neural Rendering via Reweighted Hierarchical Attention

Abstract

Transformer-based neural renderers can synthesize globally illuminated images of unseen scenes, but dense attention limits their scalability. Moreover, tessellation increases the number of triangle tokens, whose features and attention logits may remain similar to those of the original triangles. Nonuniform tessellation can therefore shift softmax attention toward subdivided regions and alter predictions even when the surface geometry is unchanged. This token-count dependence also affects hierarchical attention, where coarse tokens replace groups of triangles to reduce computational cost. We propose \papername, a two-stage neural rendering pipeline that addresses these coupled challenges through reweighted hierarchical attention. Inspired by continuous attention, we use represented surface area as a quadrature weight, giving triangles and clusters a common measure across discretizations and hierarchy levels. This reduces the dependence of attention on token count when coarse and fine representations are combined. We couple this reweighting with a spatially cohesive 3D hierarchy, where grouping nearby triangles helps limit information loss when they are represented by a single coarse token. Regenerating coarse tokens from their constituent triangle features at each layer keeps them tied to local surface regions without maintaining independent cluster states. Using this hierarchy, the view-independent encoder combines attention to all coarse tokens with sliding-window triangle attention to encode global light transport, while the view-dependent decoder retains coarse global context and retrieves fine triangle features relevant to each image patch. Experiments demonstrate improved generalization to more complex scenes and substantially reduced inference time while maintaining rendering quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Rendering, Transformer, Quadrature Weight, Hierarchical Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CXGCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TriRender to address the cost and tessellation sensitivity of transformer-based rendering from triangle meshes. The core contribution is a surface-area weighting in the attention architecture, TriRender constructs spatially cohesive clusters, and regenerates coarse representations through the area-weighted pooling. The paper also utilizes a scene encoder to combine local triangle interactions with global coarse context, and an image decoder to expand the selected clusters based on visibility and attention relevance. TriRender is initialized from RenderFormer and further fine-tuned using synthetic scenes. The paper provide detailed comparisons and ablations in both the main paper and supplementary to demonstrate the improved rendering at high triangle counts and a quality–efficiency trade-off.

Critical strengths
  1. The central contribution Surface-area weighting addresses a concrete tessellation-dependent failure mode in previous pipeline and the authors provide a coherent formulation. The surface-integral interpretation supplies a common measure for coarse and fine tokens, and the provided experiments on removing reweighting demonstrates that the proposed weighting has a substantial empirical effect.

  2. The proposed hierarchical pipeline achieves a favorable measured quality–efficiency trade-off. Compared with dense attention using reweighting alone, the full model reduces runtime with minor PSNR degradation. The supplementary experiments further extend runtime evaluation to meshes containing one million triangles. These extensive experiments validates the effectiveness of the proposed framework.

  3. The spatial hierarchy is well-supported by detailed ablation. The authors provide ablations by replacing the proposed component-constrained, area-balanced grouping with uniform clustering. The results show that the PSNR reduces reduces and worsens the other reported metrics. This provide further evidence to support the importance of preserving spatially coherent regions when constructing coarse representations.

Critical weaknesses
  1. The architectural contribution is difficult to separate from additional fine-tuning. The paper fine-tuned the model on approximately 125k scenes, but comparable adaptation of RenderFormer and RenderFormer-V2 is not clearly specified. This leaves uncertainty about how much of the observed improvement comes from the proposed representation and attention mechanisms. Finetuning the baselines using the same datasets can further clarify the gains.

  2. Some details are missing to characterize the practical scope of the efficiency gains completely. It is unclear whether timings include hierarchy construction and visibility computation, while the paper acknowledges that disconnected triangles can limit cluster compression. These missing details is important for readers to assess whether the measured gains transfer to other high-resolution mesh structures. I suggest the authors provide corresponding experiments.

  3. More direct quantitative validations are required for the claimed tessellation robustness. In the paper, Figure 4 provides a controlled attention visualization, but Figure 6 compares scenes with different triangle counts. Measuring the final-image consistency under nonuniform retriangulation of fixed scenes would more directly test the central robustness claim. I also suggest the authors provide more qualitative comparison in the paper to support the robustness claim.

  4. There are some details that need to be clarified for better presentation: a) The method describes hierarchical cross-attention but Table 1 in the paper refers to "LLCA stage-2", the abbreviation should be defined or replaced with consistent terminology. b) The default attention configuration should be stated explicitly. The authors discusses different cluster counts in the paper, but does not clearly collect the default cluster count, retrieval budget, and window size used for the full model. This should be stated explicitly for better paper presentation. c) A few grammatical errors should be corrected. In Section 5.2, “but increasing inference time” should be “but increases inference time”; in Section 4.2, “possibly because the reweighted attention” should be “possibly because of the reweighted attention.”

Questions for authors

For fixed geometry, materials, lighting, and cameras, how much do rendered images change under different nonuniform triangulations, with and without area weighting? I think addressing this by adding more qualitative comparisons can directly test the claimed output robustness.

I appreciate the architectural designs of the paper, but based on the aforementioned missing details and no "borderline option", I currently voted a negative score. I would consider raising the scores if author can address my concerns during the rebuttal period.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CXGC

We thank the reviewer for their constructive feedback and the opportunity to clarify the remaining concerns. We hope the additional experiments and explanations below help address these points.

1. Architectural contribution. The main benefit of hierarchical attention is lower inference time. Tab. 1 and Fig. 7 indicate that most of the quality gain comes from surface-area reweighting. Removing reweighting reduces PSNR from 24.50 to 17.66 dB, while dense attention with reweighting reaches 25.02 dB.

We also fine-tuned RenderFormer and found that its rendering quality was close to that of the released checkpoint. We can use the fine-tuned version in the revised comparison if helpful. For RenderFormer-V2, we fine-tuned it and found that it performed worse than the released checkpoint. This degradation may be related to the broader range of scenes and higher image resolutions used in the original RenderFormer-V2 training. Notably, both TriRender and RenderFormer-V2 similarly combine local attention with scene summaries in the stage 1. Taken together, the W/o reweighting and Uniform clustering ablations help assess two design differences between the models by testing surface-area reweighting and component-constrained, area-balanced clustering, respectively.

2. Timing and visibility. TriRender's reported inference times include on-the-fly hierarchy construction and CUDA rasterization for visibility. Hierarchy construction, the Stage 1 forward pass, and the Stage 2 forward pass account for approximately 20%, 60%, and 20% of the total runtime, respectively. CUDA rasterization takes only a few milliseconds. We will include these timing details in the paper.

Disconnected triangles and cluster compression. In the Fox in the Wild scene (the first panel of supplementary Fig. 1), grouping all leaves into a single cluster reduces PSNR (by approximately 0.5dB on the entire image and 3dB measured over the region containing the fallen leaves) compared with assigning each leaf to its own cluster. With many leaves, however, one cluster per leaf can quickly exceed our target cluster budget. We will include this experiment in the revision.

3. Tessellation robustness. We compare TriRender with its W/o Reweighting variant on the original scenes (Original) and two variants with tessellated main objects (e.g., Lucy), named Dense and Very Dense. These sets contain an average of approximately 6k, 109k, and 435k triangles per scene, respectively.

Scene variantAvg. trianglesModel variantPSNR ↑SSIM ↑LPIPS ↓
Original6kW/o Reweighting21.220.90530.1066
Original6kTriRender27.400.95460.0457
Dense109kW/o Reweighting17.350.84510.1746
Dense109kTriRender27.100.95340.0502
Very Dense435kW/o Reweighting17.140.83980.1796
Very Dense435kTriRender26.660.95040.0508

We will include both quantitative and qualitative comparisons across these scene variants in the revised manuscript.

4. Terminology and defaults. LLCA stage-2 in Table 1 is a legacy abbreviation for Stage 2 hierarchical cross-attention. We use 512 clusters, retrieve the top 8 clusters per patch and attention head in Stage 2, and use a Stage 1 sliding-window radius of 64 positions in the triangle sequence. We will correct the wording in Sections 4.2 and 5.2.

Thank you for the rebuttal. I appreciate the clarifications, especially the tessellation robustness and timing and visibility part. I also appreciate the clarification for the terminology. Please include these details in the revised paper. Now I have the remaining suggestions for the authors:

  1. For the architectural contribution part, I suggest the authors providing the fine-tuned RenderFormer and RenderFormer-V2 results for a more comprehensive experiments.
  2. For the disconnected triangles and cluster compression part, I suggest adding this experiment in the paper and in the rebuttal discussion for a better presentation.
  3. Also, I think it would be better if you include more qualitative comparison for the area reweighting ablation to better present the contribution of this component.
AuthorsOpenReview ↗

Follow-up Response

Thank you for your feedback. We have uploaded a revised rebuttal addressing your comments.

  1. Table 1 now includes results for the fine-tuned versions of RenderFormer and RenderFormer-V2.
  2. Figure 11 and the accompanying text now include visual results and a more detailed description of the experiment.
  3. Table 3 and Figure 8 have been added to present the ablation results.

Thank you for the follow-up discussion and for revising the paper. I suggest highlighting the newly added figures and corresponding description in the paper so that AC can have better assessment of the paper. The results partly addressed my concerns and I have raised my score accordingly. Good luck!

Reviewer_q8ZLOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper improves over RenderFormer V1/V2 by introducing a hierarchical attention design and adding a surface-area reweighing attention bias to both stages to stabilize the model. Smaller changes like changing the grouping after sorting from fixed number of primitives to comparable surface area, and re-sorting afterwards. The proposed method made the model achieve higher rendering quality and faster speed.

Critical strengths
  1. I like the idea of surface-area based attention reweighing. This is easy to implement in a modern attention computation scheme and seems to improve the quality a lot. The examples in fig 4 and fig 5 also clearly demonstrated the functionality of the mechanism.
  2. The hierarchical attention works in both stage 1 self-attention and stage 2's cross-attention. And seems to provide a huge speed up without compromising too much on quality.
Critical weaknesses
  1. The biggest question is about the fairness of speed comparison with dense attention. It's highly suspicious that the authors did not use the optimal attention backend like flash-attention or SDPA on this, as the rendering time seems too slow for the dense backend. Also the paper did not describe how they implement the hierarchical attention, which is actually a two stage computation on token selection and would cause un-continuity in memory accessing pattern. It it worth comparing both method with fair engineering settings to see the real speed-up.
Questions for authors
  1. I think it's better to add a test to qualitatively and quantitatively demonstrate the model's invariance to tessellation patterns.
  2. It would be better if the images in supp fig1 apply proper tone-mapping for better visualization.
2 comments
AuthorsOpenReview ↗

Response to Reviewer q8ZL

We thank the reviewer for the positive assessment of our work and the constructive feedback. We respond below to the points raised under Critical Weaknesses and Questions For Authors.

Wrong timings in supplementary Fig.2. Fig. 6 and supplementary Fig. 2 use the same dataset, which contains four views per scene. In supplementary Fig. 2, we mistakenly plotted the total rendering time for all four views of each scene. We will correct the figure to show the rendering time for a single view. The timings reported in Tab. 1 are correct.

Dense baseline. We use packed FlashAttention in both stages. To retain the area bias, we append a constant α\alpha to each query and bid/αb_i\sqrt{d}/\alpha to key ii, where dd is the original head dimension and bi=log⁡Aib_i = \log A_i for a triangle of area AiA_i. With the original scale 1/d1/\sqrt{d}, the augmented dot product yields q⊤ki/d+biq^\top k_i/\sqrt{d} + b_i.

Our implementation. For stage 1, we compute local fine-grained and global coarse attention in a single GPU FlexAttention call per layer. The BlockMask skips blocks outside the attention pattern, and score_mod adds the log-area bias to each surface key's attention score. For stage 2, we follow LLSA's use of sparse indices to drive the attention loop. After coarse scoring, a Triton kernel selects the top-kk cluster indices and merges them with visibility-selected indices while removing duplicates. Each cluster is represented by a start offset and a child count in shared K/V buffers. The attention kernel uses these indices and offsets to stream the selected triangle K/V in tiles, packing short child ranges into the same tile. It accumulates softmax statistics online, and the fine and retained coarse contributions are merged under a single normalization.

Invariance to tessellation patterns. We evaluate the original scenes (Original) and two variants with tessellated main objects (e.g., Lucy), named Dense and Very Dense. These sets contain an average of approximately 6k, 109k, and 435k triangles per scene, respectively.

Scene variantAvg. trianglesPSNR ↑SSIM ↑LPIPS ↓
Original6k27.400.95460.0457
Dense109k27.100.95340.0502
Very Dense435k26.660.95040.0508

We will include both quantitative and qualitative comparisons across these scene variants in the revised manuscript.

Tone mapping in supplementary Fig. 1. We clipped HDR pixel values above 1 when preparing the images for display. We will regenerate the figure with tone mapping.

Thank you for the rebuttal. The confirmation of using flash-attention implementation removes my biggest concerns of the paper. Please include these updates and details into the revised paper. I will keep my positive scores.

Reviewer_fUYAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends RenderFormer to support the rendering of scenes with much higher triangle counts with transformer. By biasing attention with the spatial areas of the triangles (or triangle clusters) to avoid over-attending to densely tessellated regions, and uses hierarchical attention to bound the sequence length of the attention to make it more scalable. It shows better rendering quality on scenes with around 1 million triangles than the baselines, demonstrating its scalability.

Critical strengths
  1. With the proposed surface-area reweighting and hierarchical attention, the method successfully generalizes to scenes with up to 1 million triangles, even though it was only trained on scenes with 4k to 10k triangles. This generalizability shows that the model indeed learns how to gather information from different triangles across the scene, and indicates that many of these relationships between triangles are local.

  2. The ablation study shows the effectiveness of the proposed components in the method.

Critical weaknesses

There is no quantitative comparison with baselines (RenderFormer and RenderFormer v2), and exisitng simulators (e.g., Blender Cycles) on the test set proposed in the baselines, and the test set proposed in this paper (line 376). Such a comparison in a table is crucial to evaluate the rendering quality (PSNR, SSIM, LPIPS), rendering time, and computation cost of the proposed method.

Questions for authors
  1. Could author please provide the quantitative comparison mentioned above in Critical Weaknesses?

  2. Triangles that are spatially adjacent to each other may not be consecutive on the Hilbert curve, and could be split into different clusters, which may hurt the reasoning of the model. How do you resolve this?

  3. In equation 1, if the area information is encoded in the key/value, would the model be able to learn the surface-area biasing? As mentioned in line 265, can the proposed surface-area reweighting be considered as a better positional encoding?

  4. How can the model handle object inserting/removal or dynamic scenes with the current clustering method? If the clustering is calculated once at a static state, can it model the effect when the spatial relationship of the different components in the scene changes? If the clustering is re-calculated every time step, will there be jittering artifacts due to the jump of the clustering?

  5. In Figure 2 in the supplementary, what is the "dense baseline"?

Minor:

  1. Do you have ablation results for the cluster-centroid Hilbert ordering in line 254?

  2. For the uniform clustering in Table 1, how many clusters do you use?

  3. The reference to Fig. 7 in line 422 is misleading: Fig 7 doesn't show ablation on hierarchical attention.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fUYA

Thank you for the review. We respond below to the points raised under Critical Weaknesses and Questions For Authors.

1. Quantitative comparison with baselines. We compare TriRender with RenderFormer and RenderFormer-V2 on our 75-scene test set and on the baselines' test set. We use Blender Cycles reference images generated with adaptive sampling capped at 4,096 samples per pixel and denoising enabled.

Reported runtimes were measured on an NVIDIA L20 GPU and include on-the-fly Hilbert ordering wherever it is used. The RenderFormer-V2 runtimes exclude texture encoding.

75-scene test set

MethodPSNR ↑SSIM ↑LPIPS ↓Time (s) ↓
Blender Cycles (reference)———4.815
RenderFormer11.670.75180.071346.15
RenderFormer-V220.610.92750.09998.131
TriRender27.740.96450.02984.545

Baselines' test set before and after tessellation

We evaluate the original scenes (Original) and two variants with tessellated main objects (e.g., Lucy), named Dense and Very Dense. These sets contain an average of approximately 6k, 109k, and 435k triangles per scene, respectively.

MethodPSNR ↑SSIM ↑LPIPS ↓Time (s) ↓
Original (avg. 6k triangles)
Blender Cycles (reference)———2.560
RenderFormer29.530.95660.06010.173
RenderFormer-V228.410.93060.10750.151
TriRender27.400.95460.04570.105
Dense (avg. 109k triangles)
Blender Cycles (reference)———3.516
RenderFormer11.910.49510.41687.143
RenderFormer-V225.930.91080.13001.384
TriRender27.100.95340.05021.103
Very Dense (avg. 435k triangles)
Blender Cycles (reference)———4.460
RenderFormer9.120.26580.551671.81
RenderFormer-V217.640.77690.26157.946
TriRender26.660.95040.05084.295

2. Hilbert ordering and spatial locality. Hilbert ordering preserves 3D locality approximately. This trades some fine-grained spatial locality for the efficiency of windowed attention. If spatially adjacent triangles fall outside each other's sliding windows, each triangle can still attend to the other's pooled cluster token. Triangles within each other’s sliding windows can attend directly to one another even if they belong to different clusters.

3. Surface area embedding. The model might be able to learn a similar bias if area were encoded in the triangle keys. Pooling such keys, however, does not generally recover the bias corresponding to the cluster's total surface area.

An area-aware positional encoding could encode the area of a key relative to that of the query, but further experiments would be needed to determine how the model could use this information effectively. We therefore add log⁡A\log A directly to the attention logit, which multiplies the unnormalized softmax weight by the represented area AA. This applies the same weighting rule to triangles and clusters with a single precomputed scalar per surface key.

4. Object insertion/removal and dynamic scenes. We compute the clustering once per component and reuse it under rigid motion. When objects move, are inserted, or are removed, we update the global Hilbert order using the current cluster centroids. Our current treatment is limited to rigid objects. To probe sensitivity to changes in cluster order, we rotated two objects in the Cornell box scene by 180∘180^\circ. The PSNR differed by only 0.00220.0022 dB between the original and rotated configurations.

5. Dense baseline in supplementary Fig. 2. This is the Reweighting only variant in main-paper Fig. 7, labeled W/ reweighting only in Table 1. It uses RenderFormer's dense attention in both stages with surface-area reweighting added to the logits.

Minor 1. Cluster-centroid Hilbert ordering. We have not run a separate ablation of this ordering. We view it as a natural fit for sliding window attention, as it helps reduce spatial jumps in the triangle sequence at cluster boundaries.

Minor 2. Uniform clustering budget. The ablation uses 512 clusters, matching the default cluster budget of TriRender.

Minor 3. Reference to Fig. 7. The figure shows the full model, Reweighting only, and W/o Reweighting alongside the ground truth. The comparison between the full model and Reweighting only illustrates the effect of hierarchical attention, since both use area reweighting. We will clarify at line 422 which comparison the text refers to.

We hope our responses address your concerns. Please let us know if any questions remain or if further clarification would be helpful.

Thank you for the detailed rebuttal and clarifications. I appreciate the authors' efforts in addressing my concerns. I have a few remaining questions that I would appreciate further clarification on:

  1. Could the authors elaborate on why the proposed method performs worse than the baselines when the number of triangles is small? What are the main factors contributing to this performance gap?

  2. While TriRender significantly outperforms the baselines at high triangle counts, its rendering time is comparable to Blender Cycles. Could the authors clarify the practical advantages of TriRender over conventional rendering pipelines, and under what scenarios it would be preferable?

  3. Could the authors provide statistics on the percentage of spatially adjacent triangle pairs that belong to the same cluster or fall within each other's sliding attention windows? This would help quantify how effectively the proposed clustering preserves spatial locality.

AuthorsOpenReview ↗

Follow-up Response

1. Performance at low triangle counts

Our experiments focused on evaluating the effectiveness of the proposed mechanisms. We fine-tuned TriRender on a dataset 1/16 the size of RenderFormer's original training set (and 1/80 the size of RenderFormer-V2's). The smaller training set may be a major contributor to TriRender's lower PSNR on the low-triangle-count scenes.

2. Practical advantages over Blender Cycles

  • Across triangle counts, TriRender extends RenderFormer with surface-area reweighting and hierarchical attention.
  • The runtimes in our previous response were measured on an NVIDIA L20. On a single NVIDIA H200, BF16 TriRender averaged 1.586 s/image on the Very Dense test set, compared with 4.54 s/image for Blender Cycles.
  • At higher image resolutions, the runtime advantage of transformer-based rendering can become more pronounced, as illustrated by RenderFormer-V2's comparison with Blender Cycles.

We therefore hope that the mechanisms explored in TriRender will help future transformer-based renderers use modern GPUs to render large batches of views of the same scene at higher resolutions.

3. Spatial locality of clustering and sliding attention

On our test set, 94.64% of directly adjacent triangle pairs either belong to the same cluster or fall within each other's Stage 1 sliding attention windows. The corresponding fraction is 92.52% for triangle pairs identified by 8-nearest-neighbor search.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.