- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates cost-sensitive provider routing and stopping for distributed retrieval, where observations from searched providers incur costs. It proposes a framework combining neural maximum-score prediction, residual updates from paid feedback, adaptive provider ordering, and learned stopping. Experiments on large-scale MS MARCO retrieval demonstrate a 6.59% reduction in mean search work under the reported semantic partition setting while satisfying an independently evaluated omission-risk criterion. Further analyses separate work before and after result completion and examine practical execution costs.
Critical strengths
- Careful risk-aware experimental design. The use of matched predictors and stopping heads, frozen policies, and independent finite-pool risk evaluation strengthens the credibility of the measured efficiency improvements.
- Insightful decomposition of retrieval costs. Distinguishing work incurred before and after result completion reveals why adaptive routing may reduce costs in some provider configurations but not others. The additional execution measurements also clarify the distinction between search work, throughput, and latency.
Critical weaknesses
- Modest and setting-dependent efficiency gains. The main improvement over matched fixed ordering is 6.59%, while alternative covariance settings and source-based partitions show substantially smaller or negative benefits. One-shot cost ordering also explains much of the improvement, limiting evidence for the incremental value of repeated paid-feedback updates.
- Restricted deployment generality. Experiments primarily concern partitioned dense retrieval with a fixed top-k objective and cost proxies based on searched corpus size. The behavior under heterogeneous real-world provider latency, network overhead, and distribution shifts remains uncertain.
Questions for authors
- Can the authors quantify the incremental benefit of dynamic residual updates over one-shot cost ordering under matched risk constraints on an independent test set?
- How robust are the improvements when provider costs are measured using actual service latency rather than document-count proxies?
2 comments
Thanks for the responses! The authors have made good revisions based on my comments.
Thank you for the assessment of the risk protocol and replay analysis. We added an independently evaluated one-shot baseline and expanded the actual-execution benchmark to address both questions.
Response to Question 1: Increment over one-shot ordering under independent risk evaluation.
We freeze one-shot and dynamic actors with identical seeds and admission rules, fitting each actor's stopping head on its own TRAIN trajectories. Thresholds are selected on historical TUNE only. A separate 32-slot family uses CERT-2 and TEST-2, each with 8,192 queries per condition and per-slot failure budget 0.05/32.
Both pass the same 5% target. At these frozen operating points, dynamic uses 0.62% more mean work [0.26%, 0.98%, paired query bootstrap] and makes fewer observed omissions. The independent comparison therefore shows a work–risk tradeoff: passing a common target still permits different realized risks.
We also report the empirical work–risk curves. Descriptive interpolation at 4% risk gives one-shot/dynamic work of 2.248M/2.065M, and at 5% gives 1.939M/1.843M: dynamic reductions of 8.1% and 4.9%. These descriptive curve comparisons show lower dynamic work than one-shot in the displayed range. Fixed order remains competitive. The interpolated points do not carry independent risk certificates.
The original paired-head policy is kept distinct: without refitting, it saves 6.38% [6.08%, 6.68%] versus its original fixed counterpart on TEST-2, consistent with the submitted 6.59%. Its CERT-2 bound is 3.64%. The own-head comparison answers your stronger baseline question without conflating it with replication of the submitted policy.
Response to Question 2: Actual service latency and the cost model.
The new real-search benchmark measures 1,024 original TEST queries per condition across three randomized blocks, including GPU prediction, controller computation and Faiss searches. Under two concurrent queries and the same two-core allowance, semantic dynamic/fixed/static throughput is 5.18/4.84/3.57 queries/s, with CPU time 385.5/412.3/559.6 ms per query. Dynamic improves throughput over fixed by 7.0%.
For a single concurrent query, dynamic/fixed/parallel-static mean service times are 383.0/397.2/283.7 ms; P99 values are 701.5/683.2/321.2 ms. Thus the work reduction has an operational benefit under resource sharing, while parallel static retains the single-query latency advantage. On source providers, static also remains faster in throughput (15.51 versus 12.29 queries/s for dynamic).
These measurements establish an actual throughput benefit for policies trained with document costs. Training with heterogeneous measured provider latencies as action prices remains untested. In the semantic single-query benchmark, work and service time are strongly correlated (Pearson 0.996 for fixed and dynamic); this supports the proxy within that benchmark, while the effect of changing the training cost remains open. Timing excludes encoding, network, cold loading and arrival queueing. Sequential feedback dependencies would also contribute to latency when network RTT is introduced. Section 4.7 and Appendix M state this measurement scope.