acceptodds
Under review as a conference paper at ICLR 2027

Camera Positional Embedding for Test-Time Training

Abstract

To understand the world in 3D from many images, a model must take into account the cameras that captured them. Camera embeddings for attention do so by relating two tokens only through the relative geometry of their cameras, but they also inherit the quadratic cost of attention, which limits the number of input views. Test-time training (TTT), which stores past tokens in a small network trained at test time, offers an efficient alternative with linear cost. However, how to embed cameras into TTT has not been studied, and camera embeddings designed for attention do not transfer directly. We present CaPET (Camera Positional Embedding for TTT), a camera embedding that acts inside the TTT layer. We build it by asking where a camera embedding should be placed in a TTT layer and what form it should take. For the placement, we show that a TTT layer holds implicit scores inside its fast weights, which let a camera embedding relate two views through their relative camera. For the form, we find that a TTT layer prefers orthogonal embeddings, since they preserve the norms of its keys and queries and so keep training at test time stable. On novel view synthesis across diverse datasets, CaPET improves over the TTT model without camera embedding by up to 1.80 dB PSNR and outperforms camera embeddings for attention, at the linear cost of TTT.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Test Time Training, Positional Embedding, Positional Encoding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Rj7TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed a method to integrate camera embeddings for the TTT layers. The paper carefully analyzed where and what kind of camera embedding works the best, as well as the analysis extends to a wide range of fast weight networks. The paper shows

Critical strengths
  1. The design itself is quite interesting. It integrates the camera embedding to the TTT layer which is novel and it makes sense. The proposed input, hidden, value/output embeddings with norm-preserving transformations and extension across different fast-weight networks makes it a pretty principled architecture design.
  2. The experimental results is overall solid. The comparison shows the improvements across three datasets. Multiple different fast-weight networks are tested as well.
Critical weaknesses

overall i think it is a pretty clear paper with sound ablation results, so i do not have many concerns. my main concern is on the results reported in table 3. On the baseline part, i think the absolute NVS quality is also kind of on the lower side if compared to recent papers like RayRoPE or URoPE. i understand that people establish their own baseline and do fair comparison on it, but still it raises some questions about whether the gains persist with stronger baselines. On the RayRoPE baseline: it is adapted from query-centric to world-frame coordinates, and GTA's output transformations are not clearly specified. I think more implementation details and analysis would make the results more solid.

Questions for authors

N/A

Reviewer_yUnROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CaPET (Camera Position Encoding for Test-Time Training, or TTT). Drawing inspiration from the implicit inner products found in "fast-weight" updates, the authors incorporate rotational encodings at three distinct stages: the input, the hidden features, and the value-to-output projection. These encodings are constructed using ray directions and 3D points predicted without explicit depth supervision to form orthogonal transformations. The authors evaluate various camera encoding schemes across three novel-view synthesis datasets, analyzing the impact of insertion points, geometric representations, and fast-weight network architectures. Furthermore, they extend the approach to 3D reconstruction and video generation with camera control.

Critical strengths
  1. The paper proposes a concrete, reusable design for leveraging camera geometry within the TTT framework. Unlike existing camera encoding methods—which typically operate only on the inputs or outputs of attention mechanisms—integrating encodings into the hidden features of the fast-weight network represents a valuable extension. As shown in Table 4, the addition of hidden-feature encodings alone improves the average PSNR across the three datasets from 20.08 to 20.79, demonstrating that the performance gains are not merely a result of stacking additional modules.

  2. The empirical evaluation is compelling. Section 5.1 clarifies that all compared methods share the same model architecture, training protocols, and random seeds. In Table 3, CaPET outperforms other methods across all metrics on all three datasets, achieving an average PSNR gain of 0.54 dB over RayRoPE. Figure 5 and Table 6 further demonstrate the method's robustness to varying numbers of views and different fast-weight network architectures. Additionally, the full-scale tttLRM experiments (Table 7) show that CaPET outperforms PRoPE in 137 out of 140 scenes, which strengthens the credibility of the results.

Critical weaknesses
  1. The mathematical definition of "relative camera" (pose/position) requires more precise clarification. The rotation phase in Eq. 14 is constructed axis-by-axis using rays and point coordinates defined in world space. Consequently, while the product EjTEiE_j^T E_i depends only on coordinate differences, these differences change under a global rotation. This is not equivalent to the relative camera transformation in Eq. 2, which is invariant to the choice of the world coordinate system. While Section 4.2 and the conclusion acknowledge that the TTT layer is not entirely relative, attributing this absolute dependency solely to the initial "fast weight" term is insufficient; the coordinate-system dependency inherent in the encoding itself should also be explicitly stated. This affects the understanding of the method's geometric properties, though it does not directly invalidate the reported empirical results.

  2. The mechanism behind the claim that "TTT prefers orthogonal encodings" has not been independently verified by Table 5. The table varies both the geometric information and the form of the transformation: non-orthogonal extrinsic/projection encodings differ from orthogonal ray/3D-point encodings by more than just norm preservation. Furthermore, even when both are orthogonal, using "Ray" versus "Ray + Camera origin" performs worse than the baseline (which lacks such encodings) on the Objaverse dataset. While the experiments support the overall proposed design, the observed advantages cannot yet be primarily attributed to norm preservation or training stability. A controlled experiment that keeps the geometric representation fixed while varying only the norm-handling mechanism would provide a more direct test of this explanation.

Questions for authors
  1. Does the term "relative camera" in the paper imply that the implicit scores are invariant to the world coordinate system? Please clarify the behavior of Eq. 14 under global rotation and explain how it differs from Eq. 2.

  2. Are there any results from experiments where the geometric representation is held constant while only the norm-handling approach is varied—results that could support the stability-based explanation provided in Section 4.3?

Reviewer_dkbDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses camera pose embeddings for transformers in novel-view synthesis. This has been an active and promising research topic in recent years, where explicit 3D geometric relations are used to guide attention between views, increasing robustness. This work specifically targets networks with test-time training (TTT), i.e., where normal softmax-attention is replaced with a gradient descent update, making context aggregation linear in the number of tokens. A technically sound integration of camera pose embeddings into TTT is proposed that is, in most parts, independent of the selected world frame. The authors show that implicitly, an outer product between query and key exists in TTT, and they encode both the hidden score and input score with relative positional cues here. However, this also requires rotating some single terms by the camera pose, and the authors show that in this case an orthogonal transform, as proposed in the work, is beneficial, as these transforms do not increase the norm of an embedding and thus stabilize the gradient descent updates. Through experiments on 3 datasets, the authors show that their proposed work consistently outperforms literature approaches applied to TTT networks, and careful ablations support the main design decisions.

Critical strengths
  • The paper is well written and easy to follow. The mathematical notation is simple and consistent, and good figures, like Fig. 1, help readability even more. The adaptation is clearly introduced, and the flow of the presentation is good. I also liked that the authors introduced their method on a fairly simple architecture, but later show how this generalizes to more complex, frequently used layouts.
  • The proposed method is technically reasonable. The authors motivate their work by the reduced cost of TTT methods, introduce it, and provide sound justifications for their method, e.g., where pairwise products appear, and why orthogonal matrices are required.
  • The ablations support the central design decisions. Ablation studies show that the different entry points of camera embeddings synergize (Tab. 1), and that the selected embedding format is strongest (Tab. 2). The authors also show that their proposed method works in different TTT architectures (Tab. 6) and in other, similar tasks (Tab. 8).
Critical weaknesses
  • The eval lacks comparisons to softmax-attention based methods paired with baselines, and is thus not fully fair. While the authors only claim to improve over camera embeddings designed for attention when applied to TTT, it is still unclear how the model generally performs compared to softmax-attention, with and without camera embeddings, and whether relative gains in TTT are similar to what RayRope and others achieved for softmax-attention. I believe this is critical, because the wider applicability of TTT depends on it. Comparing relative gains of camera pose embeddings applied to softmax-attention and TTT would give a scale of how effective the proposed embeddings are compared to literature. If gains are equal, this would strengthen the general applicability of the paper. At the moment, the baselines are rather weak "adaptations" to TTT, proposed by the authors.
  • Conceptually the paper does not introduce new constraints for camera embeddings, but applies existing techniques to a different architecture. The constraints used in this paper are very similar to RayRoPE and do not add additional geometric constraints. This work therefore reads more like a technically sound adaptation of those works to TTT, which limits its scope and impact in the community. It remains unclear whether the proposed overall architecture improves over SotA. Furthermore, the proposed work still does depend on world coordinate frame (as acknowledged by the authors), a weakness that softmax-attention + RayRoPE avoids.
  • The strongest listed baseline is weaker than reported in the original paper. While the adaptation of RayRoPE, the strongest baseline in Table 3, to LaCT is technically sound, the forced change from query-frame (original) to world-frame (this work) significantly weakens the baseline. This rather important change is only noted in supplementary material.
Questions for authors
  • Tab.4: Would a comparison of input + v / o be a fairer comparison to the RayRoPE adaptation?
  • The overall implementation still depends on the chosen world coordinate system. Did you analyze the impact of changing the world coordinate frame, e.g. by rescaling or random rotations / translations?
2 comments

I'd like to thank the authors for their detailed response.

My concerns about softmax-attention baselines were addressed.

My second concern about new geometric constraints was partially addressed. Applying the encoding on the hidden state is technically sound, but in my opinion an architectural requirement from TTT. The authors confirmed in their response that there is just no equivalent in softmax attention. The idea and justification of orthogonal matrices is valuable and a good insight. Furthermore, my initial comment also stated that this work "does not add additional geometric constraints." The authors did not comment on this in their response. Similar to RayRoPE, they keep ray+point embeddings, but remove invariance to the world frame. I don't see what new geometric (or projective) constraints are modelled.

The analysis on world-frame perturbation is interesting.

We thank the reviewer for the follow-up. We answer each point below.

“Applying the encoding on the hidden state is … an architectural requirement from TTT. … there is just no equivalent in softmax attention.”

We agree that the hidden embedding is an architectural requirement of TTT. Our goal is a camera embedding for TTT, not for attention, and finding such requirements is the purpose of our analysis. This requirement was not known before our analysis: the existing embeddings adapted to TTT do not use the hidden placement, and the hidden embedding alone outperforms RayRoPE (20.79 vs. 20.49 average PSNR).

 

“my initial comment also stated that this work "does not add additional geometric constraints." The authors did not comment on this in their response. … I don’t see what new geometric (or projective) constraints are modelled.”

We apologize for missing this point. We agree with the reviewer that CaPET does not model a new geometric (or projective) constraint. Instead, CaPET introduces two non-geometric constraints on a camera embedding that are important in TTT: (1) the embedding needs to be orthogonal, and (2) the embedding needs to act at the hidden score. We thank the reviewer for recognizing the orthogonal form as a good insight.

Reviewer_aHwMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to incorporate camera information into the TTT fast weight layer. The authors construct a rotation encoding using camera rays and predicted 3D points, placing it within the layer to implicitly compare information from different viewpoints. The results of novel view synthesis have improved, but existing experiments have not yet distinguished how much benefit comes from the encoding position, geometric representation, and orthogonality respectively; the improvement on some datasets is also minor, and repeated experiments for verification are lacking.

Critical strengths

The authors analysed the implicit interactions in TTT rapid weight updates, discussed where the camera encoding should be added, and constructed orthogonal RoPE using rays and predicted 3D points. The approach is relatively clear. The average PSNR on three datasets reached 21.03 dB, which is 0.95 and 0.54 dB higher than the baseline without additional encoding and RayRoPE respectively, with FLOPs only increasing by 0.13%.

Critical weaknesses
  1. The geometric encoding follows existing designs, with the new content mainly being where to place the encoding in TTT and how to coordinate it with weight updates. The authors are advised to describe the differences from existing methods more specifically, especially the new contributions in encoding mechanisms and theoretical analysis.
  2. The main text uses a two-layer MLP and standard gradient updates to analyse the effect of camera encoding, while experiments use SwiGLU and Muon. Although there are supplements in the appendix, they do not adequately explain how these differences impact theoretical conclusions. The authors are advised to clarify whether the original conclusions still hold in actual models and under which conditions. 3.Table 5 changes both the geometric representation and the transformation method, making it unclear whether the improvement comes from ray and 3D point representation or from the orthogonal transformation itself. It is recommended to keep the geometric representation unchanged and compare only orthogonal versus non-orthogonal transformations to check if orthogonal transformations can improve performance and stabilise training.
  3. On DL3DV and Objaverse, the PSNR was only 0.21 and 0.41 dB higher than RayRoPE, which is not a significant improvement. It is suggested to repeat experiments with several random seeds to confirm whether these improvements are stable.
  4. The video experiments only compare CaPET with the original scheme; it is recommended to include other camera encoding methods as a reference. Additionally, while video quality and camera rotation accuracy improved, the translation error increased from 0.468 to 0.481. The paper does not explain this result clearly, and further clarification is required.
Questions for authors

Look at the weaknesses section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.