- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark to evaluate whether or not an LLM is good at choosing its model and computation budget. They do this by considering prompt pairs and feeding them separately into the LLM. They then observe the computation budget and model that the LLM chooses in each case. The LLM either uses the same setting, or switches. The question is whether this decision was justified or not. To answer this, the authors propose forcing the chosen setup and its alternative to the LLM and analyzing its reponse to the prompt pair many times. This allows them to experimentally observe which option gives them the best trade off between cost and action quality. Based on this, they can know whether or not the original choice of the LLM was justified.
Additionally, the authors introduce a method for choosing the model and computation option. This is done by training a NN on the offline data used for the benchmark to learn the quality-cost trade-off given the setting, and then choosing the best one.
The authors test PO-CTRLBench and PORC on a number of experiments, and compare them to some benchmarks. PO-CTRLBench shows that route-based metrics for choosing a model might be misleading and not choose the option that has the best quality-cost tradeoff, while PORC is seen to decrease token cost while also increasing quality compared to existing benchmarks.
Critical strengths
I think the paper's claim about how adaptive inference systems are currently evaluated is very interesting and well discussed. Essentially, only looking at consistency of the route is not an enough metric, and the authors do a good job justifying this both in the text and via the experiments.
Critical weaknesses
1- One of the main lacks of the paper is not clearly mentioning the actual consistency rates in the experiments. The authors only mention the ranks of the methods obtained using route concsitency, but not the actual rates. This makes the claims difficult to verify.
2- I have a concern regarding how cost is compared in the experiment with three models (0.5B, 1.5B, and 7B). It seems like the authors use the number of output tokens as the main measure of cost. However, producing a token using the 7B model, for example, is going to be much more expensive than the 0.5B model. This way of measuring cost ignores many aspects such as model size, latency, and computational cost.
3- It seems like the main comparison of the five different routing protocols only uses the PPFR and CRMR metrics, which to my understanding, only calculate the quality and correctness of the edited prompt.
Questions for authors
1- Can the authors report the actual consistency rates for the experiments? (regarding W1) 2- Can the authors discuss how cost can be calculated well in situations like those explained in W2? It would be beneficial for the paper if the authors could re-evaluate the result using a more realistic cost measure. 3- It would be great if table 3 could also report metrics that report the quality of the answer to the original prompt as well, for the sake of completeness of the reported results (Regarding W3).