acceptodds
Under review as a conference paper at ICLR 2027

The Decision Value of Perception Compute

Abstract

Adaptive perception spends extra computation on inputs where perception is expected to improve. When perception feeds a downstream decision system, a better perception output need not produce a better decision. We define the decision value of perception compute as the change in downstream loss from escalating an input from a cheap to an expensive perception mode. Because this value can be negative, the allocation of perception compute should be judged against a budget-constrained decision oracle, with uniform full-fidelity inference as a baseline rather than an upper bound. We introduce DEEP (Decision Evaluation for Escalated Perception), a benchmark that scores pre-escalation allocators against this oracle under selection, latency and energy budgets, charging each allocator for its own computation. With deployed monocular geometry on KITTI and nuScenes, we find that 34-54% of the escalations that change downstream loss make it worse; harmful escalations also occur for the published PDM-Closed planner, evaluated open-loop on nuPlan with real detector outcomes. On nuScenes, perception-level gain frequently disagrees in sign with decision value. This mismatch has practical consequences: choosing among fixed deployable signals by missed-object perception gain rather than by decision value reduces realized test decision gain by 7.4% of the all-cheap loss on average. Learned allocators recover part of the oracle's value by finding beneficial escalations but select nearly as much harm as random, and once their own computation is charged at a 20% latency budget, only the lightweight routers, at about 3.5% of a full detector pass, still beat random.

Primary area:
datasets and benchmarks
Keywords:
adaptive perception, adaptive inference, resource-aware perception, compute allocation, selective routing, decision value, autonomous driving, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SPKnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers adaptive perception under a compute budget (here, a tradeoff between latency/energy and accuracy). Unlike much prior work, the tradeoff is measured not on the perception task itself but on a downstream task (here, planning and control). The intuition is that a more expensive perception output may improve perception performance but still hurt the downstream task.

For such an “allocator” or controller, it is important to consider the input, the cheap output and the available context when making the tradeoff decision. The paper frames this as an escalation problem: deciding whether the more expensive perception branch should be called at a given timestep. They do so by considering allocators, some of which are trained to predict the per-frame value of the decision between the cheap and expensive perception branches on the downstream tasks (planning and control). The paper introduces DEEP, a benchmark that scores allocators of this kind against a budget-constrained “decision oracle.”

Critical strengths
  1. The core idea, valuing perception compute by its effect on a downstream decision rather than by perception quality, is a good one. The larger problem space and motivation make sense to me, and this is an underexplored and novel space.

  2. A benchmark and metric for evaluating adaptive perception with respect to a downstream task are useful for future progress in this space.

  3. Studying adaptive perception and planning together is very difficult because of the engineering effort needed to set up such pipelines. The paper explores many critical design decisions and runs ablations that illuminate aspects of this difficult problem.

Critical weaknesses
  1. The paper defines a value function, but then trains several simple “allocator” functions to approximate the decision-making procedure. This seems like a weakness: given a defined value function and a downstream metric, why not train an allocator that directly optimizes downstream performance? More concretely, the allocators regress the per-frame value and then select the top-k frames, instead of directly optimizing the budgeted allocation objective. Even as a benchmark, the paper would be stronger with at least one baseline trained on the true objective directly.

  2. Training is per-frame supervised learning, not sequential decision-making, which might offer additional gains. The value function is defined with the previous action fixed, so temporal effects such as scheduling consistency and action persistence are excluded by construction.

  3. The cheap model and the allocator run on every input, and an escalated input additionally runs its selected higher-fidelity mode. This seems suboptimal, because the cheap model must always run regardless of the current perception state. The allocator may use previous frames, but in my view the decision could rely on the previous output state, the input context, or a latent controller memory before running any perception on the current frame. This design space is largely unexplored in the paper.

  4. The main manuscript does not explain clearly how the downstream planner is connected to the perception module. In general, I found the paper very hard to read, and the core idea is hidden behind a lot of paper-specific terminology.

Questions for authors

Please reflect on the weaknesses mentioned above, and consider revising the manuscript to explain the core idea more clearly. The paper seems to be a cross between a benchmark and a methods paper: the benchmark formulation appears to be tied to a particular solution approach. Its evaluation protocol assumes a cascaded architecture, per-frame ranking, and a fixed action history, which constrains the space of admissible allocators. As a result, concerns about the methods also become concerns about the benchmark itself. The paper would also benefit from connecting to prior work on adaptive perception under compute constraints (e.g., https://arxiv.org/html/2005.10420v2), and from contrasting its setting with that earlier work, where perception performance itself was the objective.

Reviewer_iby3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DEEP, a benchmark for evaluating adaptive perception based on its downstream decision value rather than perception accuracy alone. It defines decision value as the reduction in downstream loss when escalating an input from a cheap to an expensive perception mode, allowing for both beneficial and harmful outcomes. DEEP evaluates pre-escalation allocators against a budget-constrained oracle under selection, latency, and energy budgets, accounting for the allocators’ computational costs. Experiments on KITTI, nuScenes, and nuPlan show that 34–54% of escalations that change downstream loss worsen it in monocular-geometry settings. The results demonstrate that perception-level improvements frequently disagree with downstream decision value, making perception metrics unreliable proxies for compute allocation. Learned allocators recover some oracle value, but their practical advantages are often limited once their computational overhead is included.

Critical strengths
  1. The paper identifies an important gap in adaptive inference: the value of additional perception compute should be measured by its effect on downstream decisions, not just changes in perception accuracy. This distinction is clearly motivated and relevant to autonomous driving and other task-oriented systems.

  2. The signed decision-value formulation provides a clean way to represent both beneficial and harmful escalations. The budget-constrained oracle is a useful reference because it exposes the opportunity cost of allocating computation to inputs where the expensive mode worsens the downstream loss. Proposition 1 also gives a precise characterization of optimal rankings under equal escalation costs.

  3. DEEP is more comprehensive than a simple benchmark of perception accuracy versus inference latency. It evaluates pre-escalation allocators, includes the allocator's own computational cost, considers both latency and energy budgets, and provides cached outputs and evaluation scripts. This design could support reproducible comparisons among future allocation methods.

  4. The experiments cover multiple datasets, detector configurations, downstream systems, and perception geometries. The evaluation includes KITTI, nuScenes, and an external nuPlan track with published planners. The paper also includes controls for geometry, detector operating points, transient detections, and loss parameters, which strengthen the evidence that the reported sign variation is not limited to one narrowly defined configuration.

  5. The paper provides useful evidence that perception metrics are imperfect proxies for decision value. In the nuScenes experiments, the signs of perception gain and decision value disagree on approximately 25–51% of the relevant frames, depending on the metric and downstream system. The comparison between downstream systems further illustrates that the same perception change can have different consequences for different consumers.

Critical weaknesses
  1. The distinction between the proposed formulation and established value-of-information or task-aware computation ideas needs sharper substantiation. The paper itself connects its formulation to classical information value theory, rational metareasoning, and active perception. Its contribution appears to be a concrete operationalization for perception escalation, together with a benchmark and empirical analysis. That may be a valuable contribution, but the paper should establish more clearly what is conceptually new beyond applying a signed marginal-utility objective to a particular adaptive-perception setting.

  2. The counterfactual evaluation does not establish the benefit of adaptive perception in closed-loop driving. Both perception modes are evaluated against a fixed reference scene state, and the paper explicitly fixes the all-cheap action history when evaluating sequential decisions. This isolates the immediate effect of an escalation, but it cannot capture how a changed action affects subsequent observations, vehicle state, or future decisions. The findings support claims about the defined single-step loss, not necessarily about realized driving safety or long-horizon control performance.

  3. The empirical evidence for the external planner track is limited by sparse decision-changing events. The paper reports only five states with nonzero safety-loss change for IDM, and the PDM-Closed results are based on 45 affected states for the safety loss. The authors acknowledge that the external track corroborates rather than replaces the core evaluation. Consequently, the evidence for generalization beyond the benchmark's principal detection-based pipelines remains limited.

  4. The benchmark's conclusions depend on the chosen downstream losses and perception-to-control pipeline. The core evaluation includes a braking loss, a trajectory-controller loss, and a learned planner's displacement from the logged future trajectory. These objectives are not equivalent to actual collision risk or overall driving quality. Although the paper reports robustness sweeps, it does not establish that the magnitude of the reported harmful-escalation rates generalizes to other reasonable safety objectives, perception stacks, or deployed decision systems.

  5. The learned allocators demonstrate only modest and inconsistent practical gains over simple baselines. Several learned models outperform random selection under selection budgets, but their performance varies across cells and architectures. Under measured latency budgets, most learned allocators lose their advantage because of their overhead, while lightweight detection-list routers retain it more consistently. This limits the evidence that the benchmark currently enables a broadly effective adaptive-perception solution, even if the benchmark itself is useful.

  6. The normalized decision gain metric can be difficult to interpret in sparse settings. The paper acknowledges that nDG becomes unstable when the oracle's total decision value is concentrated in a small number of inputs. Its bootstrap procedure drops draws whose oracle value is below a threshold, with this affecting a substantial fraction of draws in the two PDM-Closed cells. Reporting realized gain alongside nDG is appropriate, but the filtered intervals and sparse external-track results warrant particular caution when interpreting apparent improvements.

Questions for authors
  1. Could the authors clarify the precise methodological novelty of DEEP relative to rational metareasoning, value-of-information-based active perception, and existing task-aware routing methods? In particular, which aspects of the formulation or benchmark cannot be obtained by adapting existing decision-aware allocation frameworks?

  2. The benchmark evaluates each escalation against a fixed reference scene state and, for sequential control, fixes the history to that of all-cheap operation. Can the authors clarify which conclusions they believe remain valid for closed-loop driving, where escalation-induced actions may change subsequent states and observations?

  3. How sensitive are the reported harmful-escalation rates to the choice of loss function, particularly the use of trajectory displacement rather than a direct safety measure for the learned planner? Can the authors identify which findings remain consistent across the existing loss-weight and geometry sensitivity analyses?

4.Given that IDM has only five states with nonzero safety-loss changes and PDM-Closed has relatively few affected states, how should readers interpret the statistical strength of the nuPlan evidence? Which conclusions are supported independently by the external track rather than primarily by the core benchmark?

  1. At the reported 20% latency budget, only lightweight routers consistently retain an advantage over random among the learned allocators. Could the authors summarize the realized decision-gain improvements over random after charging actual computational costs, and clarify how consistently these improvements hold across held-out evaluation units?

  2. Can the authors clarify how they distinguish the limitations of the available pre-escalation signals from the limitations of the allocator architectures? In particular, what evidence supports the conclusion that the remaining gap to the oracle is primarily due to ranking quality rather than insufficient predictive information?

Reviewer_VuKsOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

∙\bullet This work studies when it is actually useful to spend extra compute on better perception.

∙\bullet The main point is that better perception does not always mean a better driving decision.

∙\bullet The authors define the signed decision value of perception compute as the change in downstream decision loss when using a cheap detector versus a stronger detector.

∙\bullet They show that escalation can sometimes hurt, for example when the stronger detector adds false positives that make the planner brake unnecessarily.

∙\bullet They introduce DEEP, a benchmark for testing whether a routing method can choose which frames deserve extra perception compute.

∙\bullet Validation is done on KITTI, nuScenes, and nuPlan settings with braking, trajectory, and planner losses, including latency and energy costs on edge hardware.

Critical strengths

∙\bullet The work makes a clear case that perception should be evaluated by its effect on decisions, not only by detection accuracy. This is consequential because adaptive perception systems are usually motivated by saving compute, but the paper shows that spending more compute can even make decisions worse.

∙\bullet This work gives a clear benchmark, oracle definition, metrics, and several downstream decision tasks. Rather than just showing one example, it gives a way to compare future compute-allocation methods.

Critical weaknesses

∙\bullet Downstream decision models may limit generality. The conclusions depend a lot on the chosen braking/planning losses and pipelines. This matters because a different planner or loss design might react differently to false positives, so the exact amount of harmful escalation may not transfer.

∙\bullet The learned routers improve over random in some settings, but the gains shrink once latency and energy overhead are counted. This makes the benchmark convincing, but the proposed practical solution is less strong than the problem formulation. Can you include a short discussion regarding this analysis?

Questions for authors

∙\bullet How sensitive are the harmful escalation rates to the exact downstream loss weights, especially the penalty for unnecessary braking?

∙\bullet It would be useful to test DEEP with more closed-loop or log-replay evaluations, since small perception differences may have larger long-term effects. If feasible in time constrained please show a small experiment. If time does not permit, it is also fair to include a short discussion regarding the same, instead of experiments.

1 comment
Area_Chair_6ScfOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.