- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers problems related to model routing, basically determining which memories should be retrieved and which models should be invoked based on those memories. The goal is basically to try and avoid invoking models at every step and to achieve some cost savings, e.g. by effectively routing to smaller models when possible.
Critical strengths
The problem of memory-conditioned model routing is interesting, and the problem formulation here makes sense.
Reasonably detailed analysis, qualitative stuff, ablations, etc. Overall the analysis is reasonably thorough.
Easier to follow and more direct in its contribution compared to most papers in my batch.
Critical weaknesses
The biggest limitation here is just that the results seem really mixed. Even as presented in the abstract, the savings seem to be not very much (to the point that it would impact the likelihood of anyone reproducing the work here). And in the main experiments section the results do not seem all that compelling, as the comparison points are more likely model ablations etc. rather than a more convincing comparison against e.g. SotA models. Based on what's reported in the experiments I think it's doubtful that the paper would inspire enough confidence that others would be likely to follow up on this work.
The technical approach seems fine though perhaps is not a particularly large intellectual contribution. The paper is perhaps more a variation of existing work on model routing with some smaller specific technical innovations, but it's hard to spot a "big idea" here.
Questions for authors
see above.