- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper considers adaptive perception under a compute budget (here, a tradeoff between latency/energy and accuracy). Unlike much prior work, the tradeoff is measured not on the perception task itself but on a downstream task (here, planning and control). The intuition is that a more expensive perception output may improve perception performance but still hurt the downstream task.
For such an “allocator” or controller, it is important to consider the input, the cheap output and the available context when making the tradeoff decision. The paper frames this as an escalation problem: deciding whether the more expensive perception branch should be called at a given timestep. They do so by considering allocators, some of which are trained to predict the per-frame value of the decision between the cheap and expensive perception branches on the downstream tasks (planning and control). The paper introduces DEEP, a benchmark that scores allocators of this kind against a budget-constrained “decision oracle.”
Critical strengths
-
The core idea, valuing perception compute by its effect on a downstream decision rather than by perception quality, is a good one. The larger problem space and motivation make sense to me, and this is an underexplored and novel space.
-
A benchmark and metric for evaluating adaptive perception with respect to a downstream task are useful for future progress in this space.
-
Studying adaptive perception and planning together is very difficult because of the engineering effort needed to set up such pipelines. The paper explores many critical design decisions and runs ablations that illuminate aspects of this difficult problem.
Critical weaknesses
-
The paper defines a value function, but then trains several simple “allocator” functions to approximate the decision-making procedure. This seems like a weakness: given a defined value function and a downstream metric, why not train an allocator that directly optimizes downstream performance? More concretely, the allocators regress the per-frame value and then select the top-k frames, instead of directly optimizing the budgeted allocation objective. Even as a benchmark, the paper would be stronger with at least one baseline trained on the true objective directly.
-
Training is per-frame supervised learning, not sequential decision-making, which might offer additional gains. The value function is defined with the previous action fixed, so temporal effects such as scheduling consistency and action persistence are excluded by construction.
-
The cheap model and the allocator run on every input, and an escalated input additionally runs its selected higher-fidelity mode. This seems suboptimal, because the cheap model must always run regardless of the current perception state. The allocator may use previous frames, but in my view the decision could rely on the previous output state, the input context, or a latent controller memory before running any perception on the current frame. This design space is largely unexplored in the paper.
-
The main manuscript does not explain clearly how the downstream planner is connected to the perception module. In general, I found the paper very hard to read, and the core idea is hidden behind a lot of paper-specific terminology.
Questions for authors
Please reflect on the weaknesses mentioned above, and consider revising the manuscript to explain the core idea more clearly. The paper seems to be a cross between a benchmark and a methods paper: the benchmark formulation appears to be tied to a particular solution approach. Its evaluation protocol assumes a cascaded architecture, per-frame ranking, and a fixed action history, which constrains the space of admissible allocators. As a result, concerns about the methods also become concerns about the benchmark itself. The paper would also benefit from connecting to prior work on adaptive perception under compute constraints (e.g., https://arxiv.org/html/2005.10420v2), and from contrasting its setting with that earlier work, where perception performance itself was the objective.
Reviewer feedback request [by AC]
Dear Reviewer,
Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.
Best,
AC