- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that centralized realized context gains do not imply routable value. The paper presents a theoretical counterexample and proposes a policy evaluation method based on matching budgets. Across three domains, the router outperformed random selection in 4 out of 6 zero-shot pairing sets and outperformed uncertainty-based selection in 3 sets. These results are exploratory in nature.
Critical strengths
- This paper explains why realized return concentration cannot establish routable value. The counterexample holds, and the proposed evaluation metric is appropriate.
- This paper employs a budget-matched baseline, time-segmentation, and paired confidence intervals. It also reports negative results and research limitations.
Critical weaknesses
- The data used in this evaluation also served as the basis for the development of the method. There was no completely independent test set, nor was any multiplicity correction performed. The return of the retail strategy was 2/4 that of the random selection strategy and 2/3 that of the uncertainty-based selection strategy. However, since the strategy included only three time blocks, its confidence intervals were unreliable.
- The theoretical results are based on established principles of conditional expectation. The router itself employs a conventional design. The main evaluation relies on fully folded, time-specific decisions. Online evaluations and shared-time evaluations are reported separately. The combined approach was not tested against competitive routing baselines.
Questions for authors
- Can the authors validate the routing benefits using independent data that includes a sufficient number of time intervals?
- Given the same budget, does the online time-window router outperform uncertainty-based methods and the variable-point selection method?
- Table 10 reports 4200 recalculated configurations, but the combined total of the "Completed" and "Derived" categories is only 1800. Could the authors please explain this discrepancy?
- How were the "one-third of the time" estimates in Table 2 calculated?