- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I think it has to do with encoders in 3d representation, but I could not tell, the paper is very hard to parse.
Critical strengths
It has 9 pages.
Critical weaknesses
This paper is too hard to parse for me to make a review. The problem is never introduced clearly, the proposed solution is never proposed clearly. The paper is poorly organized.
Questions for authors
Why did you submit a paper this hard to read? Why submit a paper in this state, this is irrespecutful of the reviewers' time.
Here are my notes while "reading":
Reading the abstract I have not idea what the problem is. 4 sentences in we talk about lidar. This abstract is near impossible to parse.
Introduction: it is not improving, first paragraph and I still dont understand the problem. wow third paragraph is quite something.
5 comments
Maybe it is me, but that first sentence to me is confusing. I do not know much about this problem of comparing encoders. The first sentence feels like 2 ideas being put together (in the introduction). Overall the new version of the paper does not clarify my confusion. Do you have a related work I could look at for me to make sense of the work? So far I am not inclined to change my score since I believe research should be accessible even if you are not an expert.
Thank you for reading the revision and for pointing to exactly where it loses you. You are right: the first sentence tried to say two things at once, that we are comparing two encoders and that we are matching what they are given. Here is the problem with one concrete example.
Suppose we want to compare two neural-network encoders fairly. As an example, take two encoders that read LiDAR data. We give both encoders exactly the same information, namely which cells of a top-down grid contain a LiDAR point. If one encoder does better, it is natural to credit its architecture. We found that “the same information” can still be handed to an encoder in two ways. In one, a cell’s position is only implied by where it sits in the grid. In the other, the cell’s coordinates are also written out as numbers in the encoder’s input. Both ways carry the same information, yet which encoder appears better can depend on which way was used.
That is the problem our paper studies. Many encoder comparisons describe what information each model receives, but not always how that same information is encoded for each model. If that choice changes the result, a reader cannot tell whether the winner has the better architecture or simply received the information in a form that is easier for that architecture to use. Our paper measures this effect on two datasets, gives evidence that much of the effect comes from how easily each encoder can use position from its input, and proposes that this presentation choice be stated and controlled when encoders are compared.
For background, two papers from our Related Work may help. Goyal et al., “Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline” (ICML 2021), show that conclusions about which 3D architecture is better can change when details of the comparison are controlled. Liu et al., “An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution” (NeurIPS 2018), show that a convolutional network can have position implicit in the spatial layout and still find it much easier to use when coordinates are supplied explicitly.
Our paper connects those two ideas: even after the information itself is matched, can the way it is presented to different encoders change the conclusion about the architecture? We have rewritten the opening of §1 in the revised PDF(uploaded now) so that it begins with this example, in short sentences, with each idea stated on its own.
We are sorry and we take the point that the paper was hard to read, and we have rewritten the abstract and opening of the introduction so that the problem is stated in plain language before any notation appears. In short:
Two encoders are given exactly the same information: the same set of occupied cells on a map, verified cell by cell. The only thing that differs is the format in which that set is handed to each encoder, for example whether a cell’s coordinates are written into the encoder’s input features or left for the encoder to infer from where the cell sits. That formatting choice alone changes which encoder wins, by a wide margin, on two different datasets. Comparisons between encoders usually report what information each model received, not how it was formatted, so the effect is invisible in how results are normally written up. The paper measures this effect, identifies what causes it, and proposes a simple reporting rule that makes the formatting choice explicit.
The revised first page is written at this level, and the formal definitions now start in §3, which is also shorter. We would be grateful if the reviewer could look at the revised first page and tell us whether any specific passage remains unclear.