- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an architecture for training behavioral cloning policies for aerial vehicles. The key contributions are:
- Adaptive depth of prediction. Traditional world models output a fixed number of trajectories, while this paper proposes a mechanism to stop early if the current action is deemed sufficient (by another module).
- A dataset built in Airsim to evaluate the proposed architecture.
The validation relies mainly on open-loop trajectory prediction, since the closed-loop prediction fail (both for the proposed method and the selected baselines). There are a lot of ablation of the approach, showing that the key technical contributions (resampling, and adaptive rollout) bring about relatively small improvements in open loop trajectory prediction.
Critical strengths
The main strength is the idea that the sampling should not be fixed but adaptive according to the scene and task complexity. This concept is well understood and used in model-based RL (see for example "Adaptive Complexity Model Predictive Control", Norby et al.). Applying this concept to a vision-based navigation policy is something I have not seen before. It is however very similar to work in manipulation ( see for instance "ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies"), but more specific to future state prediction.
Critical weaknesses
The most critical limitation is that the proposed approach is not capable of completing the tasks in closed loop. Since open-loop prediction is barely within the scope of this paper (the claims are specifically about control of resource-constrained platform), it seems that more work is necessary to accomplish this goal.
It is unclear why all the selected baselines fail: image-goal navigation is a well-studied topic and a lot of specifically designed baselines exist (see for instance "GOAT: Go to Anything"). Could it just be that the proposed dataset is not diverse enough, and all methods overfit? This potentially justifies why the prediction errors are so low (centimeter-level), but the closed-loop performance is poor. Overall, I did not understand why creating a new dataset, since so many large datasets already exist (see for example "OpenFly: A comprehensive platform for aerial vision-language navigation").
There are also a few parts in the method that are not fully justified. As far as I understood, several trajectories are predicted from the same starting state , and the predicted trajectories are then used to generate actions. The trajectories are not scored against each other (as typical in planning methods), but rather always used to predict the ground-truth action. Since the learning signal is the same for all trajectories (See eq. 5), what is stopping them to collapse into a single prediction ()? There is no visualization or analysis that the trajectories are different. Looking at the error heatmap in Fig. 4, the actions are also barely different.
While I have previously alluded to it, another limitation worth mentioning is that there are no baselines representative of the state of the art in image based navigation, in particular for drones. This is a mature topic and therefore requires a better placement in the context of prior work.
Questions for authors
- There are very little technical details on the definition of states and control. Is the model predicting velocities, waypoints, or torques? What does it observe beyond the image (if anything)? At what frequency is each module updated?
- Similarly, there are very sparse details about the dataset: What is the average speed the drone moves? is it planar motion or 3D? How is the data generated? Why is this better than previous datasets?
- In table 3, why adding planning makes the model so much faster? Shouldn't it be faster if you only predict one set of actions?
- In table 4, the results show that stopping computation at random has a very similar performance than the proposed DPO-style approach (difference is on the second decimal digit). It should be clarified why that is the case.