- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates token-based feedforward 3D Gaussians splatting and build on the observation that, in such methods, attention between tokens and image regions does not seem to correlate well (in terms of being spatially aligned). This is then demonstrated on a toy example. The authors propose Gaussian Splat Attention Bias (GSAB) that decodes intermediate 3D Gaussians at an earlier transformer layer, projects them onto the input views using ground-truth or predicted camera poses, and uses the resulting Gaussian weights (modulated by a learned scale and spatial spread) as an additive attention bias in the final cross-attention block. GSAB is evaluated on two baselines: the pose-free C3G (2K Gaussians on RealEstate10K) and the posed TokenGS (262K Gaussians on DL3DV), showing modest improvements over their reproduced baselines.
Critical strengths
- The core idea that a token responsible for predicting a specific 3D Gaussian should aggregate visual features from the 2D image regions where that Gaussian actually projects is geometrically intuitive and connects nicely to spatially modulated query mechanisms in 2D detection. However, as I will explain in the weaknesses section I am not sure whether this really is a real problem in practice.
Critical weaknesses
- The paper claims to introduce a general mechanism that solves why token-based feed-forward methods lag behind pixel-aligned ones in terms of accuracy. However, the empirical validation is simply an incremental modification of C3G. While the authors also apply GSAB to TokenGS (Table 2), their own reproduction (TokenGS*) significantly underperforms the official TokenGS numbers. Adding GSAB (TokenGS* + GSAB) merely brings the performance back to the originally reported numbers. Moreover, a substantial performance gap (~1.4-2.7 dB PSNR on RealEstate10K) remains between token-based C3G+GSAB and compact/pixel-aligned competitors like F4Splat or SPFSplatV2. Thus, GSAB comes across as an incremental and minor patch for C3G rather than a general paradigm shift for token-based 3DGS.
- The central assumption that misaligned cross-attention is a primary bottleneck in feed-forward reconstruction is treated more as a hunch than something rigorously proven. The toy experiment (Fig. 2) is too simplistic. It freezes 3D geometry and image encoder, re-initializes the transformer, and trains only color and opacity on 6 scenes for 1K iterations. Showing that local patch attention helps estimate local color when 3D geometry is fixed is trivial and does not prove that global/unconstrained cross-attention is detrimental when jointly learning 3D geometry and appearance from scratch. Let me note that multi-layer transformers already have high redundancy and often use broader context in cross-attention to resolve multi-view geometry. The fact that forcing alignment via GSAB decreases the misalignment ratio from 31.2% to 22.1% (i.e., large portion of the misaligned tokens remain "misaligned") while leading only to minor PSNR gains across the board, suggests that the network has redundancy and that strict attention-to-splat alignment is not really a bottleneck.
All in all, the paper at this point is an incremental patch to C3G rather than a mechanism to improve all token-based feedforward reconstructors. Moreover, the insight it build upon is not really analysed and proven.
Questions for authors
- In Table 2, TokenGS* + GSAB only matches the published TokenGS baseline. Can the authors clarify why the reproduction gap is so large?
- In the toy experiment (Fig. 2), geometry is frozen and only color and opacity are learned. What happens in a full training setting if attention is hard-constrained to the splat region vs. soft-biased via GSAB? Could the remaining 22.1% 'misaligned' tokens in C3G+GSAB actually be performing important non-local context aggregation (e.g., looking at adjacent textures or epipolar regions to resolve depth/pose ambiguity)?
- How does the method work with any other token-based feedforward Gaussian methods?
5 comments
Response to Reviewer TzQp (2/3)
W2. Does alignment matter beyond the toy experiment?
The toy experiment is a controlled test that isolates the effect of alignment: fixing the geometry keeps the splat regions identical across settings, so any difference in accuracy comes from the attention alone. Accordingly, we limit its conclusion to this toy experiment setting (L260), and we have revised the paragraph title (L238) to "In a controlled setting, better alignment improves reconstruction."
This controlled setting is close to the final block of C3G, where GSAB is applied. In this block, the splat regions of the intermediate and final Gaussians mostly overlap (Fig. 6), so the geometry is already largely determined by the earlier block, as in the toy experiment.
Other observations also link alignment to reconstruction quality in the full model:
- Across the scenes of the released C3G, attention precision (Eq. (6)) correlates with PSNR, with a Spearman correlation coefficient of 0.61 (Sec. 3.2).
- The intermediate reconstruction loss alone does not improve C3G reconstruction quality (Table 3), so the gain more likely comes from GSAB.
We also group the 1,000 test scenes into those whose misaligned ratio decreased and those whose misaligned ratio increased when comparing C3G*+GSAB to C3G* and to the released C3G, and report the share of scenes and the mean PSNR change of each group:
| C3G*+GSAB vs. | Misalignment decreased Scenes (%) / PSNR (dB) | Misalignment increased Scenes (%) / PSNR (dB) | Spearman (Misaligned, PSNR) |
|---|---|---|---|
| C3G* | 85.7 / +1.52 | 14.3 / +0.60 | −0.53 |
| C3G (released) | 88.5 / +0.78 | 11.5 / +0.22 | −0.36 |
PSNR increases in both groups, so factors other than alignment also contribute to the gain of GSAB. However, two results show that the increase is larger where alignment improves:
- The mean PSNR increase is 2.5 to 3.5 times larger in scenes whose misaligned ratio decreased.
- The Spearman correlation coefficient between the per-scene changes in the misaligned ratio and in PSNR is negative for both baselines, so the larger the decrease in the misaligned ratio, the larger the PSNR increase tends to be.
Response to Reviewer TzQp (3/3)
Q2a. Soft bias (GSAB) vs. hard mask.
As suggested, we have run this comparison in full training. Our first results show that encouraging alignment with the soft bias of GSAB improves over C3G, while strictly enforcing it with a hard mask does not.
We compare the two in the final block of C3G, keeping everything else identical, including the intermediate decoding and the intermediate reconstruction loss:
- Soft bias (GSAB): an attention bias added to the attention logits encourages each Gaussian token to attend more strongly to the splat region of its projected intermediate Gaussian, but the token can still attend to the patches outside it.
- Hard mask: each Gaussian token attends only to the patches within the 3σ ellipse of its projected intermediate Gaussian, as the patches outside it are masked out.
Because of the training time, we first compare at 25K iterations on the 1,000 test scenes:
| Method | PSNR |
|---|---|
| C3G | 19.50 |
| C3G+GSAB | 19.91 |
| C3G+hard mask | 19.50 |
At this early stage, GSAB increases PSNR by 0.41 dB over C3G, while the hard mask gives the same PSNR as C3G. This may be because the intermediate Gaussians are not yet well placed, so the hard mask can confine each token to a misplaced region, while the soft bias of GSAB still lets the token attend to the patches outside it.
We expect the 100K results within the discussion period and will post them in a follow-up comment.
W2 / Q2b. Could the remaining 22.1% be useful non-local context aggregation?
Some of them may be. Most of the remaining misaligned tokens (15.1% of the 22.1%) cover their own splat region and extend beyond it. This is also because GSAB reduces the other types of misalignment, where attention is placed elsewhere or covers too little.
As defined in Sec. 3.2, a token is misaligned when the precision or the recall of its attention region with respect to its splat region (Eq. (6)) falls below τ, and we split misaligned tokens into three types:
- Too wide: the recall is at least τ but the precision is below τ, so the attention region covers the splat region and extends beyond it.
- Too narrow: the precision is at least τ but the recall is below τ, so the attention region covers only part of the splat region.
- Wrong place: both are below τ, so the attention region barely overlaps the splat region.
Compared to the released C3G on the 1,000 test scenes, GSAB reduces the two types in which attention misses part or all of the splat region:
- Wrong place almost disappears (9.4% → 1.5%), so attention is directed to the right location. This reduction is linked to part of the gain of GSAB: scenes with a larger reduction of wrong place tend to show a larger PSNR increase, with a Spearman correlation coefficient of −0.34 between the per-scene change in its share and the change in PSNR.
- Too narrow is halved (10.9% → 5.6%), so attention covers more of the splat region.
The remaining too wide tokens (15.1%) may perform such non-local context aggregation, and we agree that context from adjacent textures and epipolar regions can be important to resolve geometric ambiguities.
However, we expect such context to be needed less in the final block, where GSAB is applied. The splat regions of the final Gaussians differ little from those of the intermediate Gaussians (Fig. 6), which suggests that most of the geometric ambiguity is already resolved before this block.
We deeply appreciate the careful review and the in-depth questions of TzQp, which have helped us present the claims of our paper and the evidence for them more clearly. We would be happy to answer any further questions.
Response to Reviewer TzQp (1/3)
We thank
TzQpfor the careful assessment. We address the concerns below.W1. Clarification on the comparison with pixel-aligned methods.
GSAB does not aim to resolve the difference between pixel-aligned methods and token-based ones. GSAB methodologically addresses misaligned attention in token-based models and improves them while keeping their compact set of Gaussians.
W1 / Q3. Generalization.
GSAB improves two token-based models that differ substantially, each over a baseline trained in the same setting:
We therefore ask
TzQpto consider the TokenGS result, compared on the same data, as a valid second case, and the two cases together as support that the improvement by GSAB generalizes beyond C3G.We agree that showing the improvement on a third token-based model, such as GlobalSplat, would further strengthen this claim. Training GlobalSplat with and without GSAB takes several weeks on our resources, so we cannot guarantee results within the discussion period. We plan to include the results in the camera-ready version.
W1 / Q1. TokenGS reproduction gap.
The gap between the published TokenGS results and our reproduction stems from training, not evaluation. Our evaluation code matches the official evaluation: it reproduces the published numbers when evaluating the released TokenGS checkpoint (4 views: reproduced 22.98 vs. published 23.02 dB).
The training gap is likely due to two factors:
We therefore compare TokenGS* and TokenGS*+GSAB trained in the same setting, reporting PSNR with 4 input views for each of the three seeds:
GSAB improves PSNR in every seed, and even the lowest seed of TokenGS*+GSAB (22.84) exceeds the highest seed of TokenGS* (22.70).
To let the reproductions be checked directly, we have uploaded our code as the supplementary material. It includes: