acceptodds
Under review as a conference paper at ICLR 2027

Event Camera Motion–Appearance Disentanglement

Abstract

An event camera observes changes in brightness, so the signal produced by an object depends jointly on its appearance and its movement. Separating these factors could let a representation recognize the same object under different motions, or compare movements across different objects. We ask how far this separation can be obtained from existing event representations without retraining their encoders. Our starting point is simple: observations that share one factor reveal which feature changes should be ignored when representing that factor. We study this idea through paired spectral analysis and lightweight learned projections, using independently pretrained encoders for feature analysis and reconstruction-pretrained autoencoders for rendering. A controlled dataset that records every object along every trajectory allows us to evaluate the factors separately and then test their recombination. Retrieval assesses whether the codes distinguish the requested factor despite changes in the other; rendering asks whether codes from different clips can produce the corresponding event observation. The experiments show that useful factor-specific retrieval does not by itself ensure successful rendering, and that the interface between extracted codes and a renderer substantially affects the latter. Captured-event probes delimit the applicability of the controlled findings. Together, these tests distinguish information that can be extracted from a frozen representation from information that can be independently used.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
event cameras, neuromorphic vision, disentanglement, appearance–motion factorization, frozen representations, weak paired supervision, factor-specific retrieval

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_togwOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a method to disentangle motion and appearance information from pretrained encoders for event camera data. Authors find that although these dimensions can be identified, they cannot necessarily be successfully recombined. To address this they introduce PRISMA, a method that trains two small networks to extract appearance and motion codes and a predictor that can learn how to combine them into a target latent representation. PRISMA achieves event-data recombination accuracy of 79% compared to 57% in the benchmark approach. Object recognition is also improved by ~20%. The results are strong, but performance on real event camera recordings is much weaker.

Critical strengths
  • One question that springs to mind is “how do we know the recombination network is actually using the latent codes and not hallucinating something plausible?”. The authors do several things to address this concern, leaving me convinced that this works:
    • Check whether codes actually contain that semantic information
    • Test what happens when the incorrect codes are combined
    • Compare PRISMA with self-construction
  • Very interesting theoretical question about event sensor data, so I enjoyed reading this.
  • Convincing ablations and useful baselines. The paper compares agains invairance based learning, contrastive approaches, training free methods. I liked the controls showing that training only the predictor or predicting a sample from it’s own codes performs worse.
  • Did show some downstream utility in object recognition tasks. So not just theoretically interesting.
Critical weaknesses
  • The paper does not really motivate well why you would want to do this. Is it just to learn more about how event data encoders organize information? Or is it to make a new generate method for synthetic event creation? Or something else? Like, why should I care about this?
  • Why would I not just reconstruct objects and then apply trajectories to them, if this is meant as a generative method?
  • PRISMA only reports working on rigid objects - what about deformable motion and non-rigid objects? Do they still separate this way in the latent space? e.g would a person running and a dog running cluster differently? I feel this paper is somewhat limited in utility since it only works on this very narrow range of objects
  • Doesn’t seem that this would generalize to real event sensor data, at least I do not see any evidence for that. So not sure how useful this is beyond synthetic settings. Paper would be much stronger if we at least understood why that is - is it just a domain adaption problem?
  • Only one encoder is evaluated (Voxel-AE). I would like to see this ideally across a few different encoders and see if it generalizes.
Questions for authors
  • Please do a better job explaining what exactly whitening is. The first sentence is “The two maps are applied to whitened representations”. Took me a while to figure out what that is exactly. Maybe just a sentence like “Whitening is a method to normalize latent features to facilitate fair comparison” or something like that.
  • Some other literature to consider:
    • Compositional Generalization via Forced Rendering of Disentangled Latents also deals with the distinction between readable and composable representations.
    • EventNeRF: Neural Radiance Fields From a Single Colour Event Camera: NERFS also disntnabgle scene appearance from viewpoint changes. Maybe relevant?
2 comments
AuthorsOpenReview ↗

Response to Reviewer togw

We thank the reviewer for their positive evaluation and constructive feedback. We appreciate the in-depth analysis of the paper’s strengths, particularly the soundness in the method approach, the originality of the paper’s central question, the convincing ablation and baselines, and the downstream tasks presented. We address the specific questions below.

Clarifying the Whitening Procedure (Q1)

We thank the reviewer for this suggestion. As suggested, we reworked the whitening introduction to better explain the technique. The new text will appear in the revised version of the paper, which we will upload during the discussion period.

Additional Related Literature (Q2)

We thank the reviewer for these suggestions. As requested, we will expand the discussion about related works and alternative approaches, and how they differ from ours. For Compositional Generalization via Forced Rendering of Disentangled Latents, we agree it is relevant, since it also separates readable from composable representations. Our related work focused on the event-based literature, and we will extend it with this paper. We especially thank the reviewer for the connection with the NeRF literature, which we had not considered: EventNeRF: Neural Radiance Fields From a Single Colour Event Camera separates scene appearance from viewpoint through per-scene optimisation (the weights differ per scene), while we study feed-forward encoders that generalize across scenes. We will discuss this difference in the related works.

We will be happy to answer any other question that you may have.

Reviewer_togwOpenReview ↗

Response

Thank you and best of luck!

Reviewer_wx2LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a synthetic benchmark in which each object (appearance) is captured across multiple trajectories (motion), complemented by real event-camera recordings. They show that existing event encoders already exhibit an implicit separation of appearance and motion, and that this structure can be extracted—even without additional training—by using pairs of event streams that share one factor. Building on this, they propose Predictive Inference of Separable Motion and Appearance, which combines the appearance code from one stream with the motion code from another to predict the representation of the corresponding composed recording.

Overall, the paper addresses a meaningful question—whether appearance and motion can be disentangled and recombined within frozen event-based—and supports its claims through a controlled benchmark, a staged evaluation protocol, and a simple yet effective predictive procedure. However, the key claims about the physical interpretation of event imaging are not backed by direct evidence. Meanwhile, multiple experimental settings and reported quantities (e.g., loss weights, latent dimensionality, figure values) appear inconsistent or insufficiently specified and require clarification. The conclusions regarding generation/data augmentation rely primarily on synthetic experiments, with only limited validation on real event data.

Critical strengths
  1. The paper focuses on two distinct questions in frozen event representations: whether objects and motion can be recognized separately, and whether the encodings of the two can be swapped and recombined. Given that appearance and motion are inherently coupled in the event signal, this distinction helps judge whether the factors learned by the model are truly usable, and complements information that classification accuracy and cluster visualization cannot easily reveal.

  2. EventPaths covers the complete object–trajectory combinations, so each factor swap can be compared against its corresponding target sample. By separately examining the factor encodings, the composed features and the decoding results, it shows at which stage performance degrades. The generalized eigenvalue separator provides a valuable simple baseline, whose results indicate that the frozen features already contain a factor structure that can be exploited.

  3. By using sample pairs that share one factor, PRISMA can directly use the features of already-observed samples as prediction targets, without the need to construct additional counterfactual training samples. Under the Voxel-AE setting, generative recombination accuracy improves from 56.8% to 79.2%. After adding the generated data, object recognition accuracy improves from 46.8% to 77.8%, whereas adding an equal amount of perturbed duplicate samples reaches only 47.2%. This comparison indicates that the combinatorial diversity introduced by recombination indeed helps downstream recognition.

Critical weaknesses
  1. The introduction attributes the difficulty of disentanglement to physical factors and mentions that feature decomposition shows the object–motion interaction term dominates, but neither the main text nor the appendix provides the explicit form of this decomposition or the contribution proportion of each term. Section 4.3 further explains the degradation of later-generation quality as motion accumulation leading to loss of appearance information. Based solely on the generation results in Figure 4, it is difficult to judge whether the degradation stems from insufficient sensor observation, feature compression or recombination prediction error. I hope the authors can supplement the decomposition results, and compare the target events, the reconstructions of the true latent variables, and the recombination generation results over the same time segments.

  2. Freezing the invariance head and training only the predictor supports the claim that the prediction target needs to act on the encoding at the same time. However, the self-prediction results in Table 3 uses weight λ=100, while Appendix B gives PRISMA's default weight as λ=1000. If the two groups of experiments use different weights, it is difficult to attribute the entire gap between 20.2% and 79.2% to the change in the prediction target. Please clarify the actual configuration of each group, and explain whether the two targets still show a similar gap under the same loss weight or the same tuning budget.

  3. The Both accuracy of the composed features is 99.3%, which drops to 79.2% after decoding, while the true latent variables decoded by the decoder yield 100%. These results show the gap between retrieval and generation, but they are not sufficient to rule out the influence of the decoder. That the decoder can handle true latent variables does not mean it is equally stable for combined latent variables that deviate from the training distribution. It is necessary to further separate the combination prediction error from the decoder's sensitivity, by examining the relationship between latent-variable error and generation error, or by testing the decoder's response to local perturbations.

  4. The paper already has experiments with real sensors on N-ImageNet, and the authors explain the difference between camera saccade and real object motion. However, the core results of generation and data augmentation still come from Voxel-AE on synthetic EventPaths. In Table 5, PRISMA's combined retrieval mean on GEP-small is lower than that of the generalized eigenvalue baseline, which also indicates that the gains depend on the specific encoder. I hope the authors can state more clearly the applicable scope of these conclusions. If the value of generation in real-event applications is to be emphasized, corresponding validation is preferred.

  5. Appendix A explains the whitening and the reason for retaining low-variance directions. However, the main text 4.1 writes Voxel-AE's latent variables as 64-dimensional, whereas Appendix A discusses its first 128 principal components. Do the two places refer to the same representation? Please clarify the original latent-variable dimension and the whitening rank actually used. In addition, ω₁–ω₄ in Equation (1) are not fully listed in Appendix B, and the curve values in Figure 4 are inconsistent with the caption and the main-text description. I suggest completing the relevant configurations and unifying the values to facilitate verification.

Questions for authors

Q1. Is the accuracy gap caused by recombination-prediction error or by the decoder's sensitivity to off-distribution latents? Does the target comparison hold when both are trained at the same λ (100 vs. 1000)?

Q2. Do the generation and augmentation gains extend beyond Voxel-AE on synthetic EventPaths, given that PRISMA's combined retrieval on GEP-small falls below the generalized-eigenvalue baseline?

Other questions please refer to the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wx2L (Part 1/2)

We thank the reviewer for their constructive feedback. We appreciate their recognition of the distinction between recognising and recombining the factors, the complete object–trajectory grid of EventPaths, the training-free eigenproblem baseline, and the augmentation gain from combinatorial diversity. We address each concern below. Since Q1 contains two questions, we answer them separately as Q1a and Q1b.

Accuracy gap cause (Q1a)

We thank the reviewer for this point. Following their suggestions, we ran two additional analyses to better separate recombination error from decoder sensitivity.

Latent error vs. generation error. We sorted queries by the distance between the composed and true latents and report decoded accuracy per quartile (3 seeds, %):

Latent errorBoth ↑
Smallest quartile99.2
Second quartile87.5
Third quartile76.3
Largest quartile53.9

Accuracy drops as latent error increases. When recombination is accurate, the decoder achieves 99.2% Both accuracy, despite never seeing composed latents during training. The failures are also selective: the decoded sample keeps the correct trajectory in 97.2% of cases (Table 3), and what is lost is the object.

Decoder response to perturbations. The size of the error alone, however, does not explain the failures. To test whether the decoder is simply sensitive to off-distribution latents, we perturbed each true latent with an error of the same norm as that query's recombination error, either sampled using the covariance of the recombination errors or taken from another query (rescaled to match the norm). We then decoded the perturbed latents (3 seeds, %):

Decoded latentBoth ↑
True latent100.0
True latent + random matched-norm error96.5
True latent + another query's matched-norm error96.2
PRISMA latent (actual recombination error)79.2

Matched-norm perturbations reduce accuracy by less than 4 points, compared with 21 points for the actual recombination errors. The gap is therefore not due to a general sensitivity to off-distribution latents of the same magnitude, but depends on the specific recombination errors. This error is what the PRISMA target reduces relative to the alternatives (see Q1b). We keep the decoder frozen by design, to evaluate what the learned factors support without adaptation. We will add both analyses to the revised manuscript.

Target Comparison Under the Same Loss Weight (Q1b)

We thank the reviewer for raising this point. As noted, the self-prediction row of Table 3 (the own-sample target) used λ = 100 and PRISMA λ = 1000. We here report both targets trained at the same λ values, with 3 seeds and the same budget. Decoded recombination accuracy (%):

λ1001000
self-prediction20.2 ± 0.114.0 ± 0.4
PRISMA77.2 ± 0.279.2 ± 0.3

At every matched λ, PRISMA is ahead by at least 57 points, and comparing each target at its own best λ gives 79.2 against 20.2. The gap is therefore due to the prediction target, not the loss weight. Table 3 will also report the λ-matched rows.

AuthorsOpenReview ↗

Response to Reviewer wx2L (Part 2/2)

Generation and augmentation gains generalizability (Q2)

We thank the reviewer for this important observation, and we will make the scope explicit in the limitations.

  • Generation and augmentation require a decoder. Among the four encoders only Voxel-AE has one. The generation and augmentation conclusions are therefore stated for EventPaths with Voxel-AE, while the factorisation itself, which needs no decoder, is evaluated on all four encoders and on a real sensor.
  • Factorisation and retrieval extend to released encoders (Table 4) and to captured data (Table 5). On N-ImageNet, PRISMA’s appearance code gives the best class accuracy on all three released encoders (17.7 / 16.7 / 19.3%, against 9.2 / 11.2 / 17.5% for the best other splitter). On GEP-small, the combination retrieval of PRISMA and of our training-free eigenproblem splitter cannot be separated (13.1 ± 7.0 and 17.5 ± 7.3 over the six held-out conditions), and we will note this in the discussion of Table 5.

Comment on Insufficient Evidence for Disentanglement and Generation Degradation (W1)

We thank the reviewer for this observation, and provide the requested analyses.

Explicit decomposition. On the full EventPaths grid (512 objects × 96 trajectories), we use a balanced two-way decomposition, f(a, m) = μ + α(a) + β(m) + γ(a, m), where α is the object effect, β the trajectory effect and γ their interaction. The share of each term is its trace covariance divided by the total. We apply it to the Voxel-AE latent, the representation behind the generation results.

RepresentationObject α (%)Trajectory β (%)Interaction γ (%)
Voxel-AE latent10.528.760.9

These results show that the object–motion interaction is the dominant term of the representation. This is consistent with the sensor model (an edge parallel to the motion produces no events, so the observed appearance depends on the motion). We will revise the introduction to present it this way, and add the formula and the table to the Appendix.

Generation over time (Fig. 4). Following the reviewer's suggestion, we compared, over the same 80 ms bins, the target events, the reconstruction of the true latent and the recombination results. Each generated bin is matched against the target events of the same bin (object accuracy (%) ↑ over 3 seeds, for the bins plotted in Fig. 4):

Bin2468101214
Reconstruction of the true latent10010099.810010010099.0
Recombination, PRISMA83.183.681.481.877.573.665.5
Recombination, invariance64.263.460.561.857.553.849.0

The true latent reconstructs the object correctly in at least 99% of cases in every bin (the plotted bins have similar event counts), so the object is not lost by the sensor or by the encoder. Identity is lost only in the recombined latents, for both methods, with PRISMA 16 to 21 points ahead throughout. The decay is therefore due to recombination, and we will revise Section 4.3 to say so. A possible reason is that, later in the clip, the object looks less like what the appearance donor showed.

Comment on Inconsistent Loss Weights in Ablation Studies (W2)

Please see our response to Q1b (Response Part 1).

Comment on Decoder Sensitivity and Generation Errors (W3)

Please see our response to Q1a (Response Part 1).

Comment on Limited Real-World Validation (W4)

Please see our response to Q2.

Comment on Inconsistencies in Experimental Configurations and Reporting (W5)

We thank the reviewer for raising these points, which will be clarified in the revision:

  • Latent dimension. The Voxel-AE latent is 512-dimensional. The "64" in Section 4.1 refers to the code budget (48 appearance + 16 motion dimensions), which we will state explicitly.
  • Whitening. The EventPaths tables use rank 384; Table 5 uses rank 128 with 16-dimensional codes, the N-ImageNet protocol (Appendix B). We will state both ranks in Section 3.1.
  • Loss weights. Appendix B will list ω₁ = 25, ω₂ = 25, ω₃ = 1, ω₄ = 5, the time-reversal invariance term (weight 10) and λ = 1000.
  • Figure 4. The curves of Fig. 4 show bins 2–14 of the clip. The values quoted in the caption and in Section 4.3 (68.0% → 44.5% for PRISMA, 45.3% → 35.0% for invariance) referred to the first and the last bin, which contain about half of the events of the others and are therefore not plotted, as they are not directly comparable. The caption and the text will quote the plotted bins (83.1% → 65.5% for PRISMA, 64.2% → 49.0% for invariance), and Fig. 4 will show the 3-seed mean, matching the table in W1.

We hope these analyses address the reviewer's concerns, and we are happy to answer any further questions.

Thank you for the reply.

Most of my concerns have been addressed. Although I have no direct experience in this field, after I take this and the other reviews' comments into account, I would like to raise the score.

Reviewer_3DjKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the task of motion-appearance disentanglement in frozen event camera representations. In particular, the paper investigates whether the pretrained event camera representations already contain separable motion and appearance information and how much of that information can be separated. The paper further investigates whether they can be recombined to represent new object-motion combinations. Finally, a new method is proposed to improve the recombination of appearance and motion representations from different event recordings. Experiments on synthetic and real event-camera datasets demonstrate the effectiveness of the proposed approach.

Critical strengths

Interesting Research Question:

  • The paper addresses an interesting research question of whether event-camera encoders trained for other objectives already contain appearance and motion information in a separable form, without requiring retraining. The distinction between factor readability and compositionality is particularly interesting, as the paper shows that being able to identify appearance and motion independently does not necessarily mean that they can be successfully recombined. This motivates the proposed predictive learning approach and provides a useful perspective for understanding and evaluating pretrained event-camera representations.

Technical Contribution:

  • The proposed PRISMA framework is simple and well motivated. Unlike earlier approaches that explicitly train encoders to separate appearance and motion, PRISMA keeps the pretrained encoder frozen and learns separate appearance and motion representations using lightweight projection heads and a predictive learning objective. The key idea is to combine the appearance representation from one recording with the motion representation from another to predict the frozen feature representation of the corresponding object–motion combination. This provides a simple approach to learning compositional representations without retraining the underlying encoder.

New Dataset :

  • The introduction of EventPaths is a useful contribution. The dataset contains different objects moving along multiple trajectories, allowing systematic evaluation of appearance and motion disentanglement. In particular, having ground-truth recordings for different object–motion combinations enables a more direct evaluation of the recombination capabilities of the proposed method and the future research.

Experimental Evaluation:

  • The paper provides a detailed experimental evaluation across multiple frozen event encoders with different pretraining objectives. The results show that appearance and motion information can be extracted from pretrained representations using training-free approaches like generalized eigenproblem, highlighting that the underlying representations already contain meaningful factor information.

  • The proposed PRISMA further improves recombination accuracy from 56.8% to 79.2% on Voxel-AE. The ablation studies also demonstrate the importance of jointly learning the factor representations and predictor, as well as using cross-sample prediction.

  • Finally, the experiments show that the generated event recordings can improve object recognition when used as additional training data, demonstrating the practical usefulness of the proposed approach.

Minor Writing Error

  • line 161 " aboe-mentioned cues" should be " above-mentioned cues
Critical weaknesses

I did not find any major technical weakness in the approach. However, the following points need to be addressed.

Dependence on Paired Training Data:

  • The proposed approach requires pairs of event recordings sharing either the same appearance or motion. While such pairs are easy to obtain from the synthetic EventPaths dataset, collecting them in real-world scenarios may be challenging. The current real-world evaluation on N-ImageNet is limited to static images with camera-induced motion. How could the required training pairs be obtained in more realistic scenarios involving complex object and camera motion? Furthermore, how robust is PRISMA to imperfect factor correspondences, and to what extent does this requirement limit its applicability to real-world event-camera recordings?

Encoder Dependency and Generalization of PRISMA:

PRISMA achieves substantially higher exact-sample retrieval accuracy on Voxel-AE (79.8%) compared to the other pretrained encoders (1.0%, 8.1%, and 13.3%) in Table 4. This is particularly interesting for E2FAI, where PRISMA achieves high factor identification accuracy (95.2%) and reasonable factor composition accuracy (67.4%), but only 8.1% exact-sample retrieval. Similarly, on GEP-small, PRISMA performs best, but the improvement over the training-free generalized eigenproblem is relatively small (13.3% vs. 12.3%).

The authors discuss these differences in Appendix D.2, suggesting that the improvements depend on how much information the frozen representations retain for prediction. However, this explanation is not investigated further. Given that Voxel-AE is trained on EventPaths while the other encoders are independently pretrained, it would be useful to understand whether these differences arise from the pretraining objectives, the representations themselves, or limitations of PRISMA.

Furthermore, Appendix B indicates that the training configuration was selected on Voxel-AE and applied unchanged to the other encoders. Could this also contribute to the observed performance differences?

Ablations:

  • The ablation studies in Section 4.3 demonstrate the importance of jointly training the factor heads and predictor, as well as using cross-sample prediction. However, these experiments are primarily conducted on Voxel-AE, where PRISMA achieves its strongest results.. Do the same conclusions hold for EventPretrain, E2FAI, and GEP-small? It would be useful to investigate whether joint training and cross-sample prediction provide similar benefits across different pretrained representations.
Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3DjK (Part 1/2)

We thank the reviewer for their thorough and detailed review of our work. We appreciate their recognition of the proposed approach strengths, mainly the original and interesting research question, the PRISMA positive technical contribution, the usefulness of the proposed new EventPaths dataset, and the detailed and expansive evaluation protocol, able to clearly demonstrate the improvements given by the proposed approach and its practical usefulness. We also appreciate the reviewer's explicit observation of the lack of any major technical weaknesses.

Dependence on Paired Training Data (Q1)

We thank the reviewer for raising this important point. PRISMA requires only pairs that share one factor (appearance or motion), with no per-sample factor values and no counterfactual samples. In real recordings, such pairs can be obtained in several ways:

  • Same appearance, different motion: the same object or scene in different time segments of a recording, or under a different camera motion. Moreover, a same-appearance pair can be generated from any recording by time reversal (reversed timestamps and flipped polarities give the same scene under the reversed motion), which PRISMA already uses as an invariance term (Appendix B).
  • Same motion, different appearance: different scenes recorded under the same ego-motion, available from an IMU or odometry.

Notably, this is already how PRISMA is trained on a real sensor in Table 5: on N-ImageNet, the pairs come only from recording metadata (the same image under different recording conditions, or different images under the same condition).

Robustness to imperfect pairs. To answer the reviewer's question on imperfect correspondences, we retrained PRISMA and the invariance baseline (Inv.) (Voxel-AE, 3 seeds) after replacing a fraction of the training pairs, in every loss term, with pairs sharing neither factor, i.e. label noise on the pairs (%, ↑):

Wrong pairsSame-object R@1 (PRISMA)Same-object R@1 (Inv.)Exact-sample R@1 (PRISMA)Exact-sample R@1 (Inv.)Composed "both" (PRISMA)Composed "both" (Inv.)
0%99.897.779.829.799.389.6
10%99.792.358.418.596.583.0
20%99.380.829.212.286.379.0

These results show that, even with 20% wrong pairs, the appearance code still identifies the object (99.3% vs 80.8% for invariance), and PRISMA stays ahead on every metric. Composition shows to be more sensitive: exact-sample R@1 drops from 79.8% to 29.2%. Together with Table 5, this indicates that PRISMA tolerates a minority of imperfect correspondences.

We agree that complex scenes with several independently moving objects require more elaborate pair mining, and will add this to the limitations.

AuthorsOpenReview ↗

Response to Reviewer 3DjK (Part 2/2)

Encoder Dependency and Generalization of PRISMA (Q2)

We thank the reviewer for this question, which we address point by point.

Pretraining objective or data. Following the reviewer’s suggestion, we performed an additional experiment on the first stage only, the pretraining of the encoder: we pretrained the Voxel-AE encoder, with the same architecture and the same reconstruction objective, on N-ImageNet and on DSEC instead of EventPaths. The encoder is then frozen, and the second stage is unchanged: all splitters, including PRISMA, are trained on the EventPaths pairs and evaluated on EventPaths with the same protocol as in the paper (3 seeds). Exact-sample R@1 (%) ↑:

Voxel-AE pretrained onGen. eigenproblemInvariancePRISMA
EventPaths64.129.7 ± 0.779.8 ± 0.6
N-ImageNet8.23.5 ± 0.316.0 ± 0.8
DSEC13.36.0 ± 0.619.9 ± 1.4

These results show that the difference observed in Table 4 is reproduced by changing only the pretraining data of the frozen encoder, with the architecture, the objective and the training of PRISMA unchanged. This indicates that it originates in the match between the data seen by the encoder during pretraining and the evaluated data, rather than in PRISMA, which remains the best splitter in every setting here. The held-out trajectories are excluded from the pretraining of the encoder in all cases.

Representation vs. PRISMA. In every row above, PRISMA is the best splitter and improves over the training-free generalised eigenproblem by 15.7, 7.8 and 6.6 points. When the pretraining data changes, all three splitters decrease together, including the generalised eigenproblem, which involves no training. The decrease therefore originates in the frozen representation, before any splitter is applied, while the gain of PRISMA is preserved in all three pretraining settings.

Identification vs. exact-sample retrieval on E2FAI. The metrics ask for increasingly fine information. Identification (95.2%) and composition (67.4%) require the composed feature to carry the right object and the right motion. Exact-sample retrieval asks for the single recording among 49,152, which includes the same object under the 95 other trajectories, with query trajectories never seen in training. It therefore depends on sample-level detail of the frozen feature, which the experiment above shows to depend on the pretraining data, while the factors themselves are largely recovered.

Training configuration. Following the reviewer's observation, we varied the configuration separately for each released encoder, changing one setting at a time: the loss weight λ (100, 300), the code sizes (24 + 8, 96 + 32), the whitening rank (128, 256), the predictor width (1024) and the head width (768). We report the best result obtained for each encoder. Exact-sample R@1 (%) ↑:

ConfigurationEventPretrainE2FAIGEP-small
Paper (λ = 1000, codes 48 + 16, rank 384, predictor 512, head 384)1.08.113.3
Best configuration per encoder1.29.114.7

These results show that selecting the configuration per encoder improves the results by at most 1.4 points, against a difference of more than 65 points with Voxel-AE. Therefore, the configuration is not the principal cause of the difference. We will also report the full sweep table in the Appendix.

Ablations (Q3)

We thank the reviewer for this suggestion. As requested, we ran the controls of Section 4.3 on the other encoders. The table below reports composed-feature both accuracy (%, mean over 3 seeds), with PRISMA at λ = 1000 and self-prediction (the own-sample target of Table 3) at both λ = 100 and λ = 1000:

Encoderinvariancefrozen heads + predictorself-prediction (λ = 100)self-prediction (λ = 1000)PRISMA
Voxel-AE89.690.642.627.599.3
EventPretrain27.129.722.120.828.5
E2FAI44.845.348.042.167.4
GEP-small69.171.446.240.479.2

Both conclusions of Section 4.3 carry over beyond Voxel-AE:

  • Cross-sample prediction is necessary. PRISMA outperforms self-prediction on all four encoders at either λ, by +56.7 / +6.4 / +19.4 / +33.0 points over its best setting.
  • Joint training is necessary. Training only the predictor on frozen invariance heads recovers little (+0.5 / +2.3 on E2FAI / GEP-small), while joint training gives +22.6 / +10.1, the same pattern as on Voxel-AE. On EventPretrain, invariance, frozen heads and PRISMA lie within 2.6 points of each other (27.1–29.7), close to what the training-free splitters reach on this representation (31.4, Table 4): there the composition is limited by the frozen representation and the trained splitters cannot be distinguished, consistent with Section 4.3.

Comment on the Minor Writing Error

We thank the reviewer for the suggestion and fixed the typo as requested.

Reviewer_3DjKOpenReview ↗

Acknowledgement of the Response

Thanks for the detailed response. Most of the concerns have been addressed now. I will consider it and the other reviews for the final ratings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of the Revised Paper

We thank all three reviewers for their careful reviews and for the time spent on our paper.

We appreciate the reviewers’ recognition of the interest of our main question, in particular the difference between factors that can be read from a frozen representation and factors that can actually be recombined (3DjK, wx2L, togw). They also highlighted PRISMA as a simple and well-motivated method (3DjK), EventPaths and its complete object–trajectory grid (3DjK, wx2L), the training-free eigenproblem baseline (wx2L), the controls showing that recombination really uses the codes (togw), and the gain from recombined samples on recognition (3DjK, wx2L, togw).

We have answered each review in detail in the individual responses, with additional experiments and clarifications where requested, and we have now uploaded a revised version of the paper. All changes are marked in blue so they are easy to find; also, as in the original submission, the appendix is in the supplementary material zip.

Main text

  • Introduction: the practical uses of the separation (togw), and the share of each term of the feature decomposition (wx2L).
  • Related work: Liang et al. (2025) and EventNeRF (togw); typo fixed (3DjK).
  • Section 3.1: expanded and clearer definition of whitening (togw), the latent size and the whitening ranks used (wx2L).
  • Section 4.2: the GEP-small comparison on N-ImageNet (wx2L).
  • Section 4.3: the role of the encoder's pretraining data, with a new Table 6, and the per-encoder configuration (3DjK); the controls at matched λ and on all four encoders, and training with wrong pairs (3DjK, wx2L); the temporal decay and the analysis of recombination error versus decoder sensitivity (wx2L).
  • Section 4.4: reconstruction and re-simulation as an alternative to recombination (togw).
  • Section 5: a new Limitations paragraph (3DjK, wx2L, togw).

Appendix

  • A, B: whitening ranks and all weights of the objective (wx2L).
  • D.3: recombination error versus decoder sensitivity (wx2L).
  • D.5: training with wrong pairs (3DjK).
  • D.6: per-setting configuration sweep on the released encoders (3DjK).
  • D.7: prediction target at matched λ (wx2L).
  • D.8: controls on all four encoders (3DjK).
  • D.9: object accuracy per temporal bin, with the reconstruction of the true latent (wx2L).
  • D.10: the explicit form of the feature decomposition (wx2L).

Since the discussion period ends soon, we would be grateful if the reviewers could tell us as early as possible whether any point is still unclear or not fully addressed, so that we can still answer in time. We also kindly ask the reviewers to take the new results and the revised paper into account in their final assessment. Thank you again for your time and for the valuable and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.