- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a method to disentangle motion and appearance information from pretrained encoders for event camera data. Authors find that although these dimensions can be identified, they cannot necessarily be successfully recombined. To address this they introduce PRISMA, a method that trains two small networks to extract appearance and motion codes and a predictor that can learn how to combine them into a target latent representation. PRISMA achieves event-data recombination accuracy of 79% compared to 57% in the benchmark approach. Object recognition is also improved by ~20%. The results are strong, but performance on real event camera recordings is much weaker.
Critical strengths
- One question that springs to mind is “how do we know the recombination network is actually using the latent codes and not hallucinating something plausible?”. The authors do several things to address this concern, leaving me convinced that this works:
- Check whether codes actually contain that semantic information
- Test what happens when the incorrect codes are combined
- Compare PRISMA with self-construction
- Very interesting theoretical question about event sensor data, so I enjoyed reading this.
- Convincing ablations and useful baselines. The paper compares agains invairance based learning, contrastive approaches, training free methods. I liked the controls showing that training only the predictor or predicting a sample from it’s own codes performs worse.
- Did show some downstream utility in object recognition tasks. So not just theoretically interesting.
Critical weaknesses
- The paper does not really motivate well why you would want to do this. Is it just to learn more about how event data encoders organize information? Or is it to make a new generate method for synthetic event creation? Or something else? Like, why should I care about this?
- Why would I not just reconstruct objects and then apply trajectories to them, if this is meant as a generative method?
- PRISMA only reports working on rigid objects - what about deformable motion and non-rigid objects? Do they still separate this way in the latent space? e.g would a person running and a dog running cluster differently? I feel this paper is somewhat limited in utility since it only works on this very narrow range of objects
- Doesn’t seem that this would generalize to real event sensor data, at least I do not see any evidence for that. So not sure how useful this is beyond synthetic settings. Paper would be much stronger if we at least understood why that is - is it just a domain adaption problem?
- Only one encoder is evaluated (Voxel-AE). I would like to see this ideally across a few different encoders and see if it generalizes.
Questions for authors
- Please do a better job explaining what exactly whitening is. The first sentence is “The two maps are applied to whitened representations”. Took me a while to figure out what that is exactly. Maybe just a sentence like “Whitening is a method to normalize latent features to facilitate fair comparison” or something like that.
- Some other literature to consider:
- Compositional Generalization via Forced Rendering of Disentangled Latents also deals with the distinction between readable and composable representations.
- EventNeRF: Neural Radiance Fields From a Single Colour Event Camera: NERFS also disntnabgle scene appearance from viewpoint changes. Maybe relevant?
2 comments
Response
Thank you and best of luck!
Response to Reviewer togw
We thank the reviewer for their positive evaluation and constructive feedback. We appreciate the in-depth analysis of the paper’s strengths, particularly the soundness in the method approach, the originality of the paper’s central question, the convincing ablation and baselines, and the downstream tasks presented. We address the specific questions below.
Clarifying the Whitening Procedure (Q1)
We thank the reviewer for this suggestion. As suggested, we reworked the whitening introduction to better explain the technique. The new text will appear in the revised version of the paper, which we will upload during the discussion period.
Additional Related Literature (Q2)
We thank the reviewer for these suggestions. As requested, we will expand the discussion about related works and alternative approaches, and how they differ from ours. For Compositional Generalization via Forced Rendering of Disentangled Latents, we agree it is relevant, since it also separates readable from composable representations. Our related work focused on the event-based literature, and we will extend it with this paper. We especially thank the reviewer for the connection with the NeRF literature, which we had not considered: EventNeRF: Neural Radiance Fields From a Single Colour Event Camera separates scene appearance from viewpoint through per-scene optimisation (the weights differ per scene), while we study feed-forward encoders that generalize across scenes. We will discuss this difference in the related works.
We will be happy to answer any other question that you may have.