- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This work proposes RouteCP a method that aims to dynamically reuse parts of source models (denoted as “experts”) for a given target task at the instance level. The authors construct that routing mechanism without needing to evaluate all experts at test time by introducing expert embeddings that are used together with the representations of the target datapoints from the shared backbone to predict what experts to use for that specific datapoint. They then enforce sparsity on the expert selection by manually keeping only the top- experts, renormalize the routing logits over that subset and construct datapoint representations by incorporating a weighted sum of these experts to the base backbone representation. The authors then create a specific non-conformity score involving the classification probability of the label under the model that incorporates the shared experts along with a variance term that corresponds to the spread of the representations given by the selected experts. They then introduce the overall loss to train the system that involves 5 terms, a standard CE loss, a conformal coverage loss that aims to reduce the set size, an alignment term between the class-conditional representations of the experts and two terms, which are relatively standard in mixture-of-experts literature, that encourage reduction in entropy of the expert selection mechanism along with load balancing across the experts. Finally, the authors discuss some theoretical properties of the algorithm and evaluate it on both toy tasks and domain adaptation tasks showing improved accuracy and smaller conformal prediction set sizes.
Critical strengths
I unfortunately couldn’t pinpoint something.
Critical weaknesses
I believe that this work has several critical weaknesses which make me recommend rejection. More specifically
- The manuscript is quite hard to parse and follow, primarily due to a quite aggressive AI use in writing. There are several areas that do not immediately follow from what was discussed or are generally not clear. For example
- At section 1
- What is a “cell” and its “cell-average”? Discussed at line 74
- What “safety certification” do you talk about at line 71?
- At section 2
- Line 99 talks about a “transfer unit” being “auditable”. What does this mean? What is the “audit” that it goes through?
- At line 135, there is a paragraph about an “efficiency analysis” but, as far as I am concerned, it doesn’t connect with the text above.
- There are frequent mentions about “certification” but the concept was not introduced so it is unclear what it refers to
- The authors talk about a “safety stage” and a “target only counterfactual”. Again none of these were introduced and it is unclear what they even mean.
- At section 3
- What are the “cells”? What are you actually certifying?
- How is equation 3.4 obtained? It seems to be dropped out of the blue.
- At section 1
- Limited novelty: overall, it seems that the authors apply classical methods from the Mixture of Experts literature (expert embeddings for routing, load balancing and sparse expert selection) while making them aware to the conformal prediction procedure that happens at test time. This has also been introduced in, e.g., Stutz et al. (which the authors do not seem to cite)
Stutz et al., Learning Optimal Conformal Classifiers, ICLR 2022
Questions for authors
Some questions for the authors are the following
- The amount of is fixed and is learned per expert and enters only via an inner product with . Why do you even need ? You can just learn an expert specific bias to shift .
- Why is and not when you mention in the text that you “select the largest logits "? Furthermore, in the text that follows the authors argue about a “no-transfer” action which means that no expert was selected. How does this happen if you “select the largest logits”? Is there some threshold also involved that allows for no expert to be selected? If yes, it should be mentioned in the text.
- Do the conditions for the results in section 4 (the ones discussed in lines 244-246) hold in practice?
- How big does the target dataset need to be in order to learn a reliable routing for RouteCP?