acceptodds
Under review as a conference paper at ICLR 2027

Opportunity, Recoverability, and Hardware Realization in Conditional Neural Computation

Abstract

Conditional computation assigns different amounts of computation to different inputs. Poor performance can have several causes. The candidate models may not make complementary errors, the router may choose poorly, or dynamic execution may erase the expected computational savings. We study these effects under a common expected-compute budget. Our framework measures oracle opportunity relative to a matched static model and compares it with an input-independent mixture control. It also measures how much of that opportunity a learned router recovers and provides an exact finite-sample accuracy oracle. We test the framework in three settings: a nested CNN on CIFAR-10, independently trained ResNet-18 candidates on CIFAR-100, and an MLP bank on CoverType. Candidate and router training use disjoint data splits. All three candidate banks show substantial oracle opportunity. Recovering it is harder. The prespecified marginal-utility router falls below the static reference on both image tasks and gives only a small gain on CoverType. In a secondary exploratory analysis, a ResNet margin policy frozen on validation data recovers about \(10%\) of the available opportunity. The hardware results reveal a separate limitation. On an RTX 3090, matched MACs do not imply matched latency or energy. The dynamic policy is slower and more energy-consuming than the static medium model, while its comparison with the static large model changes with batch size. Our results prove that we should therefore report candidate opportunity, routing recovery, and hardware realization as separate quantities.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
conditional computation, dynamic neural networks, adaptive inference, neural network routing, efficient deep learning, hardware-aware machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7cWwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a measurement framework for conditional computation, where a router allocates different amounts of compute to different inputs under an expected-compute budget. It separates (i) oracle opportunity O, the gain of a label-informed oracle over a compute-matched static model (with an input-independent mixture control and an exact solver for the budgeted accuracy oracle), (ii) routing regret G, which the theory splits into feature-information restriction and fitting error, and (iii) hardware realization, the latency/energy ratio to a static reference. On three small settings (nested CNN/CIFAR-10, ResNet-18 bank/CIFAR-100, MLP bank/CoverType), oracle opportunity is large (6.8–11.8 points), but a prespecified marginal-utility router is below the static baseline on both image tasks and recovers 3.3% of it on CoverType. A post-specified, validation-frozen ResNet margin policy recovers about 10%, yet on an RTX 3090 it is slower and more energy-consuming than static medium.

Critical strengths

S1 - The formalism is clean and clear. The authors decomposes conditional computation into interesting concepts (oracle opportunity, routing regret, hardware realization) that make it easier to understand and evaluate.

S2 - The experimental protocol is careful and transparent about what is confirmatory and what is exploratory, and it supports a useful message: matched MACs do not imply matched latency or energy.

Critical weaknesses

W1 - The study covers three small settings with three-model banks. A larger experimental framework would help better assess the validity of the claims in the paper (more/larger architectures, different tasks). The hardware results come from one GPU, one network (ResNet) and an explicit-dispatch implementation without compilation, CUDA graphs or fused routing. The prespecified router gains −1.29, −0.83 and +0.23 points (CIFAR-10, CIFAR-100, CoverType). The positive image results come only from small-model uncertainty controls on ResNet (margin: +1.19 points, ≈10% recovery), which are exploratory: the margin policy was chosen after inspecting the initial three-seed results, in which the intervals of all three controls include zero, and the same controls hurt on CIFAR-10. "Opportunity ≠ recoverability" is thus shown only for generic routers on shallow features.

W2 - Unclear positioning makes the results hard to interpret. The paper combines a theoretical framework, a router benchmark and a hardware study without saying which is its main contribution. Only Sec. 4 states that the evaluated routers are “diagnostic policies rather than proposed routing architectures”, yet the prespecified marginal-utility router comes with its own losses and a confirmatory protocol. Read as a measurement paper, it does not show that the framework can separate the causes of poor routing (information vs. fitting errors are not identified, as the authors note). Read as a method paper, its router falls below the static baseline on both image tasks. The reader cannot tell whether “router below static” is a finding about conditional computation or the failure of one particular router.

Questions for authors

Q1 - Can O be broken down by route type (inputs only the small model gets right vs. inputs only a larger one gets right), and can the oracle be reported at tighter budgets where the constraint binds? This would show how much of O is plausibly learnable.

Q2 - Unlike ResNet and CoverType (Tables 9 and 14), no candidate/union table is given for CIFAR-10. What are the accuracies and MACs of the three nested candidates, which one defines the 9.73% target, and how was the "separately trained static CNN" built?

Reviewer_e46iOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose to evaluate conditional computation, in which each input is routed to one of several candidate models of different sizes, through three separate quantities: (1) the gain that an oracle with access to the labels could obtain over a static model of the same cost (2) the share of this gain that a learned router recovers, and (3) the measured latency and energy on hardware.

Critical strengths

I had a very hard time following this paper as this is well outside of my area of expertise and the writing did not help, i.e, it is not written to be understood by a typical ML person. Therefore, I do not have anything positive to say.

Critical weaknesses
  • I remain unconvinced that the problem this paper addresses arises in practice. The authors state that the three quantities are often conflated, but they do not point to an evaluation that reached a wrong conclusion because of it.

  • It is unclear what the proposed quantities add to a direct comparison of accuracy against measured latency.

  • The paper is hard to read. Many terms are introduced through dense notation with little intuition, so a reader outside this subfield cannot assess the contribution.

Questions for authors
  • Could the authors give an example where these quantities lead to a different design decision than a measured accuracy-latency comparison?
Reviewer_VrQAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies conditional computation, in which different amounts of computation are allocated to different inputs. It proposes an evaluation framework that separates three questions under a common expected-compute budget: how much accuracy opportunity a candidate model bank offers relative to a matched static model, how much of that opportunity a learned router recovers, and whether the resulting allocation translates into actual hardware savings. The framework is evaluated in three settings: a nested CNN on CIFAR-10, independently trained ResNet-18 candidates on CIFAR-100, and an MLP bank on CoverType. The results show substantial oracle opportunity (6.80~11.81 percentage points), but the prespecified router falls below the static reference on both image tasks, and a post-hoc margin policy recovers only about 10% of the opportunity. On an RTX 3090, the dynamic policy is slower and more energy-consuming than the static medium model despite matched MACs.

Critical strengths

none

Critical weaknesses
  1. Limited experimental scope relative to the conclusions. Experiments use small datasets, small models, a single prespecified router, and one GPU, with no established conditional computation methods (e.g., early exit or mixture-of-experts) evaluated. The paper draws general conclusions about evaluating conditional computation, but the weak routing and hardware results may reflect this specific router and implementation rather than conditional computation in general.

  2. Limited novelty. That oracle routing exceeds learned routing, and that matched MACs do not imply matched latency or energy, are both well known. As an evaluation framework rather than a new method, the paper’s value depends on revealing new insights, which it does not clearly provide.

Questions for authors

Q1. Concepts in the second paragraph of the Introduction, such as the router, are introduced without definitions or references. Please define these terms and cite relevant prior work.

Q2. Would the conclusions hold for established conditional computation methods (e.g., early exit or mixture-of-experts), larger models and datasets, or other hardware platforms? Since the hardware results depend on the implementation, how much of the latency and energy overhead is due to the specific dynamic execution used here?

Q3. The prespecified marginal-utility router performs below the static reference on both image tasks. Why was this router chosen, and could stronger routing methods recover more of the opportunity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.