- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper under review presents a new framework (called BitRouter) for learning when a cheaper routing policy can replace a reference policy. The authors consider complete routing policies as “bandit arms”, using the session’s terminal quality and cost as evidence about the policy. Policies are organised as edits to a reference configuration, allowing evidence to be reused and shared among related policies. A new policy is only adopted when the evidence supports the required quality and cost requirements and the expected future savings can recover the extra cost of exploration. The proposed approach has been evaluated by experiments against baselines.
Critical strengths
-
The paper addresses a practical problem in the field of agentic workflows.
-
The edit representation for data exchange between related policies is promising.
-
Different types of costs are used for learning.
-
The discussion of limitations is quite solid.
Critical weaknesses
-
The empirical evaluation only uses one benchmark and a single strong-model baseline (plus the random-routing method).
-
The cost metric in the experiments only covers accepted-path nominal API cost, while there might be other relevant costs in practice.
-
There is no ablation study to analyse the impact of the different components of the overall framework.
-
A theoretical analysis of the approximations used within the proposed framework would be useful.
Questions for authors
-
Did you analyse the impact of your structural evidence sharing on the other components of your framework?
-
How well does the proposed approach generalise to different workloads or different domains?