acceptodds
Under review as a conference paper at ICLR 2027

Broader Tasks Prefer Ego-Frame Invariant Representations

Abstract

Computer vision has traditionally advanced through models optimized for individual tasks, such as object detection and semantic segmentation, while recent work increasingly seeks general representations that can support diverse downstream objectives. This raises the question of whether general representations can be guided by structural principles that hold independently of any particular downstream task. We study this question in multi-camera perception through the choice of ego coordinate frame, whose origin and axes are arbitrary. Changing this frame alters the numerical camera geometry while leaving the physical cameras, their relationships, and their visual observations unchanged. For image-projected representations, we show that consistent transformations of the underlying 3D quantities and camera geometry cancel under projection, motivating ego-frame invariance (EFI). We hypothesize that EFI is not universally advantageous for every specialized task, but becomes increasingly beneficial as a representation is required to support a broader range of predictions. To test this hypothesis in a setting where the output itself is not invariant, we study 3D object detection, an ego-frame equivariant (EFE) task whose predicted positions and orientations transform with the ego frame. We randomize the ego frame during training, and progressively broaden the task by increasing the number of target classes. Across nuScenes and Argoverse 2, using three camera-conditioning schemes, we evaluate this effect by held-out test performance obtained from selected checkpoints after training. The evaluation provides evidence that the advantage of ego-frame-invariant relative conditioning tends to grow as the prediction task becomes broader. These results support EFI as an inductive bias whose value grows as representations are required to support broader predictions, rather than as a universal advantage for every task.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Inductive bais, Symmetry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8v5nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether ego-frame invariance (EFI) becomes more beneficial as downstream tasks become broader. It mathematically describes the invariance of image-projected representations under ego-frame changes and compares three existing camera-conditioning methods: Plücker Raymap, GTA+CamRay, and PRoPE. Experiments on nuScenes and Argoverse 2 vary the number of 3D detection classes from one to four. The results suggest that EFI methods provide larger relative performance gains as the number of classes increases.

Critical strengths

The paper applies existing ego-frame-invariant camera-conditioning methods to 3D object detection, where predictions must change with the ego coordinate frame. This extends prior evaluations focusing on ego-frame-invariant outputs. Experiments on nuScenes and Argoverse 2 show that PRoPE and GTA+CamRay generally achieve larger relative performance gains over Plücker Raymap as the number of detection classes increases.

Critical weaknesses
  1. Novelty: Compared with "Cameras as Relative Positional Encoding (Li et al., 2025)", the paper reuses the same camera-conditioning methods (Plücker Raymap, GTA+CamRay, and PRoPE) without proposing a new mechanism or investigating additional methods. Its main extension is evaluating these methods in ego-frame-equivariant 3D object detection and analyzing performance across different numbers of detection classes, resulting in limited methodological novelty.

  2. Definition of task breadth: The paper approximates task breadth solely through the number of semantic classes within 3D object detection, without sufficient justification for why this reflects the diversity of downstream tasks. Evaluations on different tasks, such as occupancy prediction and HD map detection, would provide stronger evidence for the central hypothesis as broader tasks.

  3. Insufficient theoretical justification: Section 3 mathematically describes the invariance of image-projected representations under ego-frame changes. However, it does not establish why EFI should become more beneficial as task breadth increases, which is the central hypothesis of the paper.

  4. Limited ablation studies: All experiments rely on a single CNN–Transformer encoder with a BEVDepth-based detection head. Moreover, the compared conditioning methods differ not only in EFI but also in how camera geometry is injected (token-level vs. attention-level). Additional experiments with different architectures and controlled conditioning mechanisms would help establish the generalizability of the findings and isolate the effect of EFI.

Questions for authors
  1. The compared camera-conditioning methods differ not only in EFI but also in how camera geometry is injected into the encoder (token-level vs. attention-level). Why do the methods inject camera geometry at different points, and how can the observed improvements be attributed specifically to EFI rather than differences in injection points?

  2. The paper assumes that increasing the number of semantic classes corresponds to broader tasks. Under this hypothesis, how would the relative performance of EFI methods compare to Plücker Raymap when using the original 10 detection classes of nuScenes instead of the four super classes?

  3. The evaluation was conducted only in the canonical ego frame. Would the performance advantages of PRoPE and GTA+CamRay over Plücker Raymap remain consistent under different ego-frame configurations, such as 90°, 180°, or 270° rotations and translations of the ego-frame origin?

Reviewer_3zgQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether ego-frame invariance (EFI) becomes increasingly beneficial as downstream tasks require broader representations. The authors motivate EFI through the observation that changing an arbitrary ego coordinate frame alters the numerical representation of camera geometry without changing the underlying physical scene or its image projections.

To investigate this hypothesis, the paper considers ego-frame-equivariant 3D object detection and progressively increases task breadth by varying the number of target semantic classes.

The experiments compare an ego-frame-dependent camera-conditioning method (Plücker Raymap) with two ego-frame-invariant relative-conditioning methods (PRoPE and GTA+CamRay) on nuScenes and Argoverse 2. The results suggest that relative conditioning provides larger performance improvements over absolute conditioning as the number of detection classes increases.

Critical strengths

S1. Interesting investigation of representation invariance and task breadth.

The paper raises an interesting question about whether the usefulness of geometric invariance depends on the breadth of downstream tasks. Rather than assuming invariance is universally beneficial, the authors investigate its relationship with task complexity and semantic coverage.

The distinction between an ego-frame-invariant intermediate representation and an ego-frame-equivariant output is conceptually useful. The mathematical discussion of coordinate transformations and image projections provides a clear motivation for the experimental hypothesis.

S2. Consistent empirical trends across datasets and conditioning methods.

The experiments show that the advantage of relative camera conditioning generally increases as additional semantic classes are introduced.

For example, on nuScenes, PRoPE improves relative NDS* over Plücker Raymap by 0.01% in the one-class setting and 9.69% in the four-class setting. Similar trends appear on Argoverse 2 and with GTA+CamRay.

The authors also keep the training samples, model capacity, and training budget largely consistent across task configurations, which helps reduce some experimental confounding factors.

Critical weaknesses

W1. The experiments do not sufficiently isolate ego-frame invariance from other differences in camera-conditioning mechanisms.

The central claim concerns the benefit of ego-frame invariance. However, the experimental comparisons simultaneously change several architectural and representational properties.

Plücker Raymap uses absolute camera geometry injected at the token level, whereas PRoPE incorporates relative projective geometry directly into attention. GTA+CamRay also employs a different geometry-aware attention mechanism.

Consequently, the performance differences may arise from relative geometric reasoning, the attention-based injection mechanism, or other properties of the conditioning schemes rather than EFI itself.

Although evaluating two invariant methods strengthens the empirical observation, both introduce relative geometry through attention-based mechanisms, so this does not fully disentangle the contribution of invariance.

A more controlled experiment would compare invariant and non-invariant conditioning under otherwise matched architectural and geometric-information settings. Without such a comparison, the evidence supports the advantage of relative camera conditioning more directly than the specific causal role of EFI.

W2. Task breadth is confounded with semantic composition and detection difficulty.

The paper operationalizes task breadth exclusively through the number of target semantic superclasses. However, expanding the class set also changes the semantic composition, object frequencies, detection difficulty, and evaluation aggregation.

The nested class configurations are constructed by progressively adding specific categories rather than independently sampling different category combinations at each breadth level.

Therefore, the observed trend may be influenced by the particular categories introduced, especially if relative camera conditioning is more effective for certain object types.

The class-wise results suggest that gains vary across categories, reinforcing the need to distinguish general task-breadth effects from category-specific improvements.

The current experiments do not fully establish that increasing the number of classes, rather than introducing particular difficult categories, causes the growing relative advantage.

Additionally, the paper studies only one downstream task, 3D object detection, making the broader conclusions about representation generality somewhat premature.

Questions for authors

Q1. Isolating EFI. Could the authors provide a more controlled comparison that isolates ego-frame invariance from differences in geometric encoding and attention mechanisms? In particular, can the same conditioning architecture be evaluated with and without dependence on the arbitrary ego coordinate frame?

Q2. Direct verification of invariance. Could the authors explicitly measure the invariance of the learned intermediate representations under passive ego-frame transformations? For example, comparing feature differences under randomly transformed coordinate frames would help verify the central representational property and connect it to downstream performance.

Q3. Category-composition confounding. Could the authors evaluate multiple class subsets of the same cardinality rather than using only one nested sequence? This would help determine whether the performance trend depends on task breadth itself or on the particular categories added.

Q4. Generality beyond class count. Could the authors provide evidence using alternative definitions of task breadth, such as multiple prediction objectives, additional geometric properties, or different downstream tasks? The current results establish a relationship with detection-class count, but broader claims about representation generality remain insufficiently validated.

Q5. Statistical robustness. Could the authors report results across multiple random seeds, including mean performance and standard deviations? Since some improvements are small and the overall trend is not strictly monotonic, uncertainty estimates would strengthen the conclusions.

Q6. Generalization to unseen ego frames. The paper randomizes ego coordinates during training but evaluates selected checkpoints in the canonical ego frame. Could the authors additionally evaluate performance under held-out coordinate transformations or transformation ranges to clarify whether EFI provides robustness beyond the training distribution?

Reviewer_46maOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether ego-frame invariance becomes increasingly beneficial as the range of tasks a visual representation must support becomes broader. In multi-camera perception, camera positions and orientations are typically expressed relative to an arbitrary ego coordinate system. Changing this coordinate system alters the representation of the camera geometry but not the physical scene or the images captured. This work argues that a general purpose visual representation should ideally be invariant to such arbitrary coordinate choices.

However, they make a distinction between ego-frame-invariant representations (where intermediate features remain unchanged when the coordinate system is redefined) and ego-frame-equivariant predictions (where outputs such as 3D bounding boxes must change consistently with the coordinate system). The hypothesis is that a specialised representation may perform perfectly well when solving a narrow task, but an invariant representation becomes increasingly advantageous as the task demands become more diverse. Rather than testing a genuinely multitask system, they investigate this hypothesis by progressively increasing the number of object classes in a 3D detection task.

Critical strengths

Previous research, especially Li et al. (2025), has already established that relative camera conditioning provides ego-frame invariance and can outperform absolute conditioning for tasks such as novel view synthesis and depth estimation. The authors attempt to extend this understanding in two ways.

First, they distinguish between the symmetry properties of the intermediate representation and the output. They show that an invariant encoder can be useful even when the output must be equivariant.

Second, they hypothesise that the benefits of this architectural separation become greater as the prediction task becomes broader.

The paper is therefore best understood as an investigation into a proposed principle for representation learning, rather than a new 3D detection method.

Critical weaknesses

My main concern is that the authors have not investigated their central hypothesis in sufficient depth. They have two datasets, one downstream task, three previously published conditioning methods, and a single experimental variable (number of detection classes). This is a lack of experimental depth and insufficient evidence for a fairly ambitious general claim.

The entire generality hypothesis is tested by varying the number of detection classes from one to four. This is not equivalent to supporting genuinely different downstream tasks, such as segmentation, depth estimation, tracking and action prediction. It is therefore a fairly indirect test of the paper's broader claims about general purpose representations.

I'm not sure the experiments properly isolate invariance as the underlying cause. Plucker Raymap, PRoPE and GTA+CamRay differ in how geometric information enters the network, not merely whether the representations are invariant. In particular, Plucker uses token-level conditioning whereas PRoPE uses attention-level conditioning. The observed performance differences could therefore arise from other architectural differences.

Although the models are trained with randomised ego frames, validation and testing are performed in the canonical ego frame. A more direct test would evaluate equivariance accuracy and robustness under independently sampled ego-frame transformations, including transformations outside the training distribution.

Both datasets are autonomous-driving datasets, and the model always performs 3D detection. It is difficult to know whether the trends would extend to substantially different architectures or task families.

Questions for authors

My main questions relate to the experimental evidence of the broad claim.

Can it be demonstrated that ego-frame invariance, rather than other architectural differences, is responsible for the improvement?

Does the relationship hold when task breadth is increased beyond simply adding detection classes?

Is there a way to show that the encoder actually learns more invariant representations and that this explains the improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.