- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies learning to defer under budget constraints. The authors provide a new framework for this problem, show that the budget is respected, develop a lagrangian of the problem that they solve, and provide experiments that show that their algorithm outperforms the baseline models.
Critical strengths
I could not identify strengths.
Critical weaknesses
I had trouble reading this paper. This made understanding the results somewhat impossible for me. I will provide some examples in hopes that the authors may improve the writing such that the paper becomes more readable. As it stands right now I can not recommend accepting the paper.
The first improvement lies in the first paragraph, the second sentence. Here, a pool is mentioned. But it is completely unclear what kind of pool this is. It can be guessed from the text, but it could be a pool of classification systems, a pool of inference times, a pool of human experts, a pool of pre-trained models, or any combination of these. This type of unclarity is present throughout the text. For example, on the first page alone there are several highly unclear phrases: routing each input, decide which members each input needs, instance-dependent routing, per-entity penalties whose coefficients are tuned before training, aggregation (line 44, what are we aggregating?), fusion weights, induce the committee size per input, price-adjusted utility, dual price, and realised average cost. New concepts are continuously used without introduction. I strongly recommend the authors to carefully read the paper and see where new concepts are used without introduction, and then write the paper in such a way that every new concept that is needed is carefully introduced.
The problem is also highly unclear. The learning objective as stated is to minimize the expected loss of some distribution q as a function of and such that the expected cost of training adheres to some budget. There are several problems. Nowhere in the problem setting the authors specify how , , and are related. The loss function ell has an unspecified parameter and I have no idea what role this parameter serves, is introduced, but it does not seem to appear in the objective. But most egregious of all: I never learned what we can do to minimize the expected loss. Do we have access to a data set of feature and output pairs? Do we have access to features, but not outcomes and do we have to pay for these outcomes? Do we have access to outcomes, but not features and do we pay for the features? This makes it impossible for me to read the paper, as I really do not understand what the precise goal is.
In section 3.2, several parameters are introduced, but never defined nor are their roles explained. This is true for , , , , , , and . On line 151 and are introduced and on line 155 the authors have decided to use and . Are these the same objects? Are they different? It is completely unclear to me.
Questions for authors
.
3 comments
Response to Reviewer TnKf (2/3)
W2. Available data, parameter dependence and optimization
We jointly learn the image encoder, predictor selector, and fusion-weight network from labeled data over a fixed predictor pool, using the final classification loss and an average consultation-cost constraint.
Training uses a labeled dataset , where is an image and its ground-truth class. Images, labels, and full-pool responses are available during training. CIFAR-100's simulated experts follow a prescribed label-conditioned response mechanism. The selector and fusion-weight network take image features and known source information as inputs.
The parameters comprise the image encoder and selector parameters, and parameterizes the fusion network. We have placed these definitions before the learning objective and written their effect on prediction explicitly at the fixed current price:
Here, is a candidate class, is expert 's fixed forecast for that class, is its strict forward fusion weight, and is the final fused probability. Updating and changes the selection and weights, hence the probability assigned to the observed class. Each minibatch supplies labels for the fused classification loss. The forward pass uses strict fusion, while backpropagation retains a smooth fusion surrogate. The selector receives approximate gradients through straight-through estimation (ST), which uses binary selections in the forward pass and the derivatives of continuous gates in the backward pass. The image encoder, selector, and fusion network all receive classification gradients through this surrogate path. The complete computation appears in Appendix A.3. Source predictors and simulated-response mechanisms remain fixed.
Equation (1) defines the classification loss as
Here, abbreviates the fused distribution above, is the strict forward fused probability of the observed class , and is a fixed small offset that keeps the logarithm finite at zero. Section 3.3 separately explains the forward fused probability, the backward surrogate, and the roles of their numerical constants.
Equation (2) distinguishes the cost units used for training from those used for reporting. Expert 's costs in these units are denoted by and , respectively, giving
Both charge the same set actually consulted, including the expert activated when the raw selection is empty. Accounting and cost-unit transfer are given in Appendix A.5. This budget accounts for consultation cost at prediction time. Training obtains full-pool responses to compute approximate gradients, and their acquisition cost is recorded separately. Dataset construction and total training computation are training expenses.
The budget in training units is denoted by . The price is fixed during each minibatch network update and enters both the selection score and the cost penalty in Eq. (7). After each complete pass over the training data, Eq. (8) updates the price using the difference between the sample-weighted mean effective consultation cost and . Sections 3.1 and 3.4 now state the cost accounting and update schedule explicitly.
Response to Reviewer TnKf (3/3)
W3. Symbol definitions and changes in notation
Equation (3) specifies the feature, utility, and score computations. We have expanded the accompanying definitions: is the image feature, the image encoder, a learned projection, a learned embedding identifying expert , the scalar utility network, and the normalized training-unit cost. Brackets denote vector concatenation, and is the fixed offset in the cost-normalization denominator. Adjusting the utility by price gives the score
For a fixed input, abbreviates . The same convention applies to the soft selection gate and raw binary selection . We now state this before Eq. (4). In that equation, is the sigmoid function, the selector temperature, and the threshold, with default . The soft gate is , and the binary selection is , where is the indicator function, equal to one when its condition holds and zero otherwise. The ST variable is . The stop-gradient operator preserves its forward value and has zero derivative in backpropagation.
In Section 3.3, we further distinguish the source forecast , selection soft gate , and strict forward fusion weight . The symbol specifically denotes a smooth surrogate weight used in classification backpropagation. smooths the raw ST selection values in this surrogate, and provides its additional weight normalization. Raw selection and effective consultation selection also use distinct notation, making selection, fusion, and cost accounting easier to compare.
Figure 1 summarizes the modules and their computational relationships. The notation guide at the end of the appendix collects these symbols in computation order for reference alongside Sections 3.1–3.4.
Please let us know if any part of the problem setup or notation is still unclear.
Response to Reviewer TnKf (1/3)
Thank you for pointing out where the presentation was unclear. We have revised the introduction and Sections 3.1–3.4 to clarify the learning task, available training data, learnable parameters, and notation. Below, we address each concern and indicate the corresponding definitions and revisions.
W1. Predictor pool and routing/fusion terminology
The predictor pool is a fixed collection of prediction sources with different capabilities and consultation costs. These sources can include pre-trained AI classifiers, human experts, and simulated experts. We also refer to them collectively as experts. Classifiers provide class-probability vectors, and categorical expert responses are represented as one-hot vectors, with one at the returned class and zero elsewhere. Our experiments use trained classifiers and simulated experts.
Routing assigns an input image x a binary selection vector m(x), where mi(x)=1 indicates that the raw rule selects expert i, and zero indicates that it does not. The shared cost price is denoted by λ≥0, and the raw selected set by Sλ(x)=i:mi(x)=1. Selection uses the price-adjusted score si(x), obtained by subtracting the shared price times the training-unit cost from learned utility.
The set actually consulted is denoted by Seff(x): it retains every nonempty raw selection and activates the highest-scoring expert, indexed by i⋆(x), when the raw selection is empty; ties are resolved first by lower training-unit cost and then by lower expert index. The effective selection vector meff(x) records this consultation set. Committee size is the number of experts actually consulted. Fusion combines their forecasts using nonnegative weights that sum to one, with strictly zero forward weights for unconsulted experts. CATABC jointly learns selection and fusion from the final classification loss and allocates consultation resources under an average-cost constraint.
The introduction opens with pre-trained models and human experts, defines the predictor pool at its first occurrence, and introduces utility, price adjustment, selection, fusion, and cost feedback in order. Realised average cost is computed using the effective selection vector, and budget deviations update the shared price. These concepts precede the subsequent discussions that depend on them.