- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a measurement framework for conditional computation, where a router allocates different amounts of compute to different inputs under an expected-compute budget. It separates (i) oracle opportunity O, the gain of a label-informed oracle over a compute-matched static model (with an input-independent mixture control and an exact solver for the budgeted accuracy oracle), (ii) routing regret G, which the theory splits into feature-information restriction and fitting error, and (iii) hardware realization, the latency/energy ratio to a static reference. On three small settings (nested CNN/CIFAR-10, ResNet-18 bank/CIFAR-100, MLP bank/CoverType), oracle opportunity is large (6.8–11.8 points), but a prespecified marginal-utility router is below the static baseline on both image tasks and recovers 3.3% of it on CoverType. A post-specified, validation-frozen ResNet margin policy recovers about 10%, yet on an RTX 3090 it is slower and more energy-consuming than static medium.
Critical strengths
S1 - The formalism is clean and clear. The authors decomposes conditional computation into interesting concepts (oracle opportunity, routing regret, hardware realization) that make it easier to understand and evaluate.
S2 - The experimental protocol is careful and transparent about what is confirmatory and what is exploratory, and it supports a useful message: matched MACs do not imply matched latency or energy.
Critical weaknesses
W1 - The study covers three small settings with three-model banks. A larger experimental framework would help better assess the validity of the claims in the paper (more/larger architectures, different tasks). The hardware results come from one GPU, one network (ResNet) and an explicit-dispatch implementation without compilation, CUDA graphs or fused routing. The prespecified router gains −1.29, −0.83 and +0.23 points (CIFAR-10, CIFAR-100, CoverType). The positive image results come only from small-model uncertainty controls on ResNet (margin: +1.19 points, ≈10% recovery), which are exploratory: the margin policy was chosen after inspecting the initial three-seed results, in which the intervals of all three controls include zero, and the same controls hurt on CIFAR-10. "Opportunity ≠ recoverability" is thus shown only for generic routers on shallow features.
W2 - Unclear positioning makes the results hard to interpret. The paper combines a theoretical framework, a router benchmark and a hardware study without saying which is its main contribution. Only Sec. 4 states that the evaluated routers are “diagnostic policies rather than proposed routing architectures”, yet the prespecified marginal-utility router comes with its own losses and a confirmatory protocol. Read as a measurement paper, it does not show that the framework can separate the causes of poor routing (information vs. fitting errors are not identified, as the authors note). Read as a method paper, its router falls below the static baseline on both image tasks. The reader cannot tell whether “router below static” is a finding about conditional computation or the failure of one particular router.
Questions for authors
Q1 - Can O be broken down by route type (inputs only the small model gets right vs. inputs only a larger one gets right), and can the oracle be reported at tighter budgets where the constraint binds? This would show how much of O is plausibly learnable.
Q2 - Unlike ResNet and CoverType (Tables 9 and 14), no candidate/union table is given for CIFAR-10. What are the accuracies and MACs of the three nested candidates, which one defines the 9.73% target, and how was the "separately trained static CNN" built?