- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a learnable algorithm for aerial exploration. Their method is based on creating a compressed map representation in order to reduce computational requirements without sacrificing exploration performance. This compressed representation is built in a way that makes it similar to a map built purely from raw observations according to a learnable value function. In other words, it retains only the information that is most relevant to future actions. The proposed method is evaluated quantitively on a simulated benchmark, where its performance seems somewhat comparable to other works in terms of planning time and latency. Real world experiments are also provided for qualitative purposes.
Though there are some potentially valuables ideas, the paper seems very poorly written. What I could piece together was only after undue effort to decode and reverse engineering its content based on much better written previous work. This was the case for essentially all parts of the submission, including figures and tables (e.g., Table I and Figure 4 should have the same data, but are clearly distinct).
I believe that the paper would require extensive revision before publication, so I do not recommend acceptance at this stage.
Critical strengths
The paper brings some interesting ideas and the resulting algorithm is comparable to related work. In particular, they consider an objective that apparently tackles a potential issue in entropy based objectives: the fact that reducing map entropy may not lead to changes in future actions, so it may not be an ideal objective. This seem well motivated and effective for the problem
The submission also includes some some real world experiments, although these are only qualitative and for demonstration purposes. Depending on which reported data is corrected (Table I or Figure 4), one could also consider the performance of the method to be comparable to similar work.
Critical weaknesses
The main issue with the paper is that it is particularly difficult to read and understand, even after multiple passes at the submission and related work. This is not because the algorithms are inherently complex, but because throughout the paper, among other things, there are very strange word choices and sentences are constructed in unnatural ways. The most closely related paper according to the authors, PACE, has no such issues and could be understood with a small fraction of the effort spent in this submission.
To illustrate, consider this sentence from line 228: "We charge shadow construction and retention to compute and memory totals". This use of the word "charge" is perhaps natural on financial contexts, (e.g., a customer was charged for a product), but is not typically used in the technical context of the paper, even though in principle compute and memory requirements also incur a sort of cost. Perhaps this is why the authors used the term "compression audits", with "audit" being a similar strange word choice in the context, but not in finance.
The previous example might not be so significant in isolation, but similar issues occurred in many other parts of the writeup and I ended up compiling a sort of decoding table to approximately map what was written to what a typical reader would be able to understand if they were familiar with the foundational literature (such as the works of Kaelbling and Sutton, both prominent authors cited by the submission) and based on other sections of the paper. The "outcome law" is the "state-transition function"; "entropy bonus" may refer to "maximizing entropy reduction" as an objective, which was mentioned earlier in the text; "continuation" might be used for the "cost-to-go" or "value function" or for "rollouts" depending on the context; "observation-contingent" might be a different wording for "observation-conditioned"; "limited-FoV" is the expression for experiments using a depth camera (PACE used "Depth Cam" for example).
Structurally, the writeup might benefit from adding more conjunctions, otherwise it becomes harder to understand how one idea evolves to the next as the text starts to resembles a collection of disjoint points rather than a cohesive narrative. That would also shift the focus from describing "what was done" to explaining the "why it was done", which would further improve clarity. As another example, consider that both this submission and PACE use RMS jerk as a metric, but PACE makes sure to explain, even if briefly, that it is used to quantify the smoothness of the trajectory. A reader familiar with trajectory planning probably already knows what jerk is, but being explicit about it facilitates reading and also makes the paper more accessible to a broader audience.
As a last point about readability, figure 1 also seems constructed in a strange and confusing way. Each scene inset seems to have a slightly different scene layout in previously observed regions and it's not clear if this is an artifact or a result of the periodic scene recompression (or "compression audits" as they may be called). The description mentions a blue and an orange feedback, but I can only find a green or perhaps teal one and it's not clear what it is connected to. It is also confusing that the shadow model has the same number of edges and nodes as the compressed, when, to my understanding from the text, the shadow one should be larger. Perhaps that is the state of the compressed model before compression?
To highlight a second more technical weakness, it would seem that the main motivation of the paper is summarized in this sentence from line 87: "The central falsifiable claim is that this representation preserves decision-relevant history at lower memory cost than geometry reconstruction or a cached visit order". Yet, the authors seemingly never report the algorithm's memory footprint, only the planning latency. Upon reading PACE, it became clear that the problem relates the Traveling Salesman Problem (TSP), so my impression is that the memory-footprint is not actually the main issue, but rather the fact that large graphs lead to expensive solutions, since the TSP is NP-hard. This might explain why memory footprint was never reported, but ultimately it's the author's job to clearly motivate their work and experiments.
Questions for authors
A question important to my assessment: why is the data from Figure 4 seemingly different from Table I? For example, CPM appears to have the best time in table I, but is surpassed by many methods according to Figure 4? The caption says the data is the same
General questions and comments (not critical for the assessment)
- Does the replanning latency affect the exploration time? In other words, is the drone effectively waiting for a decision during that time or still flying, so the latency cost gets absorbed into the flying time?
- Related to the previous question, I think it would be interesting to include CPU/GPU utilization as another metric.
- It is generally not recommended to use 3D bar graphs when 2D ones can convey the same information. The extra dimension adds visual noise and makes it harder to read.