acceptodds
Under review as a conference paper at ICLR 2027

The Cost of Safe Skipping: Adaptive Routing with Paid Feedback

Abstract

Retrieval across providers with different search costs must decide both where to search and when to stop. Public routing and feedback-based stopping already address parts of this problem; we ask what reordering from paid returns adds once both are available. The key distinction is between finding the complete top-k and deciding to stop, since routing can affect these costs differently. We study this interaction with a neural maximum predictor, residual updates from paid returns, cost-sensitive acquisition and a learned stopping head. On 8.84 million MS MARCO passages, a frozen rank-eight policy reduces mean work by 6.59% relative to fixed order with the same predictor and stopping head. Its independent finite-pool omission-risk upper bound is 3.84% at a 5% target. Replay of 8,192 test queries locates the saving after result completion. Matched acquisition and threshold controls connect the gain to cost prioritization and stopping calibration, with one-shot cost ordering accounting for much of the reduction. On source-pure providers, earlier completion is offset by later searches. Warm neural execution retains the semantic throughput gain, while parallel static search has lower single-query latency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
distributed dense retrieval, adaptive routing, sequential decision making, cost-aware retrieval, risk control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_6oKqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates cost-sensitive provider routing and stopping for distributed retrieval, where observations from searched providers incur costs. It proposes a framework combining neural maximum-score prediction, residual updates from paid feedback, adaptive provider ordering, and learned stopping. Experiments on large-scale MS MARCO retrieval demonstrate a 6.59% reduction in mean search work under the reported semantic partition setting while satisfying an independently evaluated omission-risk criterion. Further analyses separate work before and after result completion and examine practical execution costs.

Critical strengths
  1. Careful risk-aware experimental design. The use of matched predictors and stopping heads, frozen policies, and independent finite-pool risk evaluation strengthens the credibility of the measured efficiency improvements.
  2. Insightful decomposition of retrieval costs. Distinguishing work incurred before and after result completion reveals why adaptive routing may reduce costs in some provider configurations but not others. The additional execution measurements also clarify the distinction between search work, throughput, and latency.
Critical weaknesses
  1. Modest and setting-dependent efficiency gains. The main improvement over matched fixed ordering is 6.59%, while alternative covariance settings and source-based partitions show substantially smaller or negative benefits. One-shot cost ordering also explains much of the improvement, limiting evidence for the incremental value of repeated paid-feedback updates.
  2. Restricted deployment generality. Experiments primarily concern partitioned dense retrieval with a fixed top-k objective and cost proxies based on searched corpus size. The behavior under heterogeneous real-world provider latency, network overhead, and distribution shifts remains uncertain.
Questions for authors
  1. Can the authors quantify the incremental benefit of dynamic residual updates over one-shot cost ordering under matched risk constraints on an independent test set?
  2. How robust are the improvements when provider costs are measured using actual service latency rather than document-count proxies?
2 comments

Thank you for the assessment of the risk protocol and replay analysis. We added an independently evaluated one-shot baseline and expanded the actual-execution benchmark to address both questions.

Response to Question 1: Increment over one-shot ordering under independent risk evaluation.

We freeze one-shot and dynamic actors with identical seeds and admission rules, fitting each actor's stopping head on its own TRAIN trajectories. Thresholds are selected on historical TUNE only. A separate 32-slot family uses CERT-2 and TEST-2, each with 8,192 queries per condition and per-slot failure budget 0.05/32.

Semantic policyMean work (M)P99 (M)CERT-2 upper boundTEST-2 omissions
One-shot cost2.3873.7584.17%307
Dynamic2.4023.7573.49%252

Both pass the same 5% target. At these frozen operating points, dynamic uses 0.62% more mean work [0.26%, 0.98%, paired query bootstrap] and makes fewer observed omissions. The independent comparison therefore shows a work–risk tradeoff: passing a common target still permits different realized risks.

We also report the empirical work–risk curves. Descriptive interpolation at 4% risk gives one-shot/dynamic work of 2.248M/2.065M, and at 5% gives 1.939M/1.843M: dynamic reductions of 8.1% and 4.9%. These descriptive curve comparisons show lower dynamic work than one-shot in the displayed range. Fixed order remains competitive. The interpolated points do not carry independent risk certificates.

The original paired-head policy is kept distinct: without refitting, it saves 6.38% [6.08%, 6.68%] versus its original fixed counterpart on TEST-2, consistent with the submitted 6.59%. Its CERT-2 bound is 3.64%. The own-head comparison answers your stronger baseline question without conflating it with replication of the submitted policy.

Response to Question 2: Actual service latency and the cost model.

The new real-search benchmark measures 1,024 original TEST queries per condition across three randomized blocks, including GPU prediction, controller computation and Faiss searches. Under two concurrent queries and the same two-core allowance, semantic dynamic/fixed/static throughput is 5.18/4.84/3.57 queries/s, with CPU time 385.5/412.3/559.6 ms per query. Dynamic improves throughput over fixed by 7.0%.

For a single concurrent query, dynamic/fixed/parallel-static mean service times are 383.0/397.2/283.7 ms; P99 values are 701.5/683.2/321.2 ms. Thus the work reduction has an operational benefit under resource sharing, while parallel static retains the single-query latency advantage. On source providers, static also remains faster in throughput (15.51 versus 12.29 queries/s for dynamic).

These measurements establish an actual throughput benefit for policies trained with document costs. Training with heterogeneous measured provider latencies as action prices remains untested. In the semantic single-query benchmark, work and service time are strongly correlated (Pearson 0.996 for fixed and dynamic); this supports the proxy within that benchmark, while the effect of changing the training cost remains open. Timing excludes encoding, network, cold loading and arrival queueing. Sequential feedback dependencies would also contribute to latency when network RTT is introduced. Section 4.7 and Appendix M state this measurement scope.

Thanks for the responses! The authors have made good revisions based on my comments.

Reviewer_xZNFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of retrieval across many different search providers, each with potentially different costs. The main contribution of the paper is a new learned search policy to determine how to reduce costs in this setting of paid search over multiple providers. The authors evaluate their policy on the MS MARCO dataset where they report a reduction in "mean work" of 6.59%.

Critical strengths
  1. The paper studies an interesting problem of cost-effective search over multiple different paid providers. This problem is becoming increasingly relevant with the recent rise in new commercial search services targeting applications such as LLM-based search and retrieval augmented generation. Understanding cost-effective search policies in this model is timely and potentially practically relevant and, to my knowledge, has not been studied in depth previously.
Critical weaknesses
  1. The writing of the paper is completely incomprehensible and I had a very difficult time understanding the main points the authors were making. In particular it is unclear to me 1) what precisely is the problem the authors are addressing and why it is important and 2) how their proposed solution works, and 3) how the experimental evaluation backs up their claims. I would encourage the authors to aim to revise the paper with a focus on readability and clarity

  2. The paper evaluates the proposed method on a single dataset and it is unclear how prior routing methods mentioned in the paper like those of Bruch, et al. and Olausson, et al. relate to this method and whether it would be appropriate to consider them as relevant baselines in this setting.

Questions for authors
  1. Can the authors explain, in simple intuitive terms, how their proposed algorithm solves the problem they introduce in the opening of the paper? I had a difficult time understanding what problem the proposed policy is trying to solve.
1 comment

Thank you for pointing out where the explanation was unclear. We have rewritten the abstract and introduction around the task, the algorithm and the evidence. A concrete search example and the redesigned Figure 1 explain the decisions, while the experiment opening states what each comparison tests.

Response to Weakness 1 and Question 1: The problem and how the algorithm works.

A query is answered by searching several providers. Searching a provider costs work and returns its local top-10 documents. We want the global top-10 across all providers, while searching as little as possible and keeping the probability of an incomplete result below a specified target. A provider may be expensive even if its expected relevance is high, so simply sorting providers by predicted relevance is not enough to represent the objective.

The algorithm first predicts each provider's maximum score and searches a common public-score seed. It then repeats three steps: use the observed maximum to update predictions for unopened providers and merge returned documents; ask the stopping head whether to continue; if continuing, choose an affordable provider by predicted competitiveness divided by cost and pay for its search. Every search is charged against a per-query budget. Budget stopping returns the current result and counts as an omission if that result is incomplete.

The key subtlety is that the system cannot directly observe when the complete answer has been found. Suppose all ten required documents have already come from A and B, but unopened C still appears promising. The system may need another paid search before stopping. Consequently, a route that finds the answer earlier can still cost more overall. This is why we evaluate the route and stopping rule together and separate work before and after actual completion in replay.

Independent queries are used to test the omission risk of the complete frozen policy. The stopping head supplies a decision score; the independent test supplies the risk statement. Individual queries may still return incomplete results.

Experimental support for this explanation.

The new semantic TEST-2 has 8,192 queries. Fixed order with a stop head that cannot see returned scores uses 3.277M mean work. Giving its stopping head those paid observations reduces work to 2.426M, a 25.98% reduction. Both pass the independent 5% omission-risk test. Dynamic selection with cost information uses 2.402M, versus 2.702M for cost-agnostic selection and 2.501M for an Optimist-style scoring control; both differences also compare independently accepted policies. These experiments separately test the roles of stopping information and cost-sensitive selection.

The original frozen dynamic/fixed pair also reproduces its work reduction on fresh queries: 6.38% versus the submitted 6.59%. We report one-shot cost ordering alongside dynamic, including both work and omissions, to distinguish the benefit of initial cost ordering from the additional effect of repeated updates.

Response to Weakness 2: Relation to Bruch et al. and Olausson et al.

Bruch et al.'s Optimist motivates a score-moment ordering rule. Our added Optimist-style control ranks providers by initial mean plus a TUNE-selected multiple of standard deviation, using the shared predictor and feedback stopping. Olausson et al.'s AMIPS supplies the SupportNet architecture used by both our fixed and dynamic actors. Thus the shared-predictor comparison asks what paid adaptation adds beyond the initial support prediction. These comparisons isolate controller choices at a common training budget. Native-system performance under each method's full training and optimization procedure is outside that comparison.

The independent certificates apply within MS MARCO. The same-corpus control compares hostname, k-means and random ownership of identical documents, separating repartitioning from corpus identity. FiQA provides a separate development diagnostic, where the dynamic TUNE guard is infeasible. The revision labels these evidence roles explicitly. The introduction, Figure 1, Section 4.4 and Appendix M connect the problem, decisions and supporting results.

Reviewer_DgDbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether feedback-driven reordering improves retrieval when prediction and adaptive stopping are already available. It combines cost-aware provider selection with a learned stopping rule and reports a 6.59% reduction in average search work on the semantic MS MARCO collection. Replay analysis locates the saving mainly after result completion. Further experiments show that the benefit depends on cost ordering, stopping thresholds, and the provider setting, and does not necessarily reduce single-query latency.

Critical strengths
  1. The experimental design and analysis are comprehensive. In particular:
    • The controls examine several possible explanations for the saving, including cost ordering, stopping thresholds, and training choices.
    • The evaluation separates independent risk confirmation from exploratory analyses, making the scope of the evidence clear.
    • The paper reports settings where the method offers little benefit or increases work, and measures both throughput and single-query latency rather than assuming that fewer searches always mean faster execution.
  2. The distinction between finding the complete result and deciding to stop is a useful insight. The paper shows that finding all required documents earlier does not necessarily reduce total search work, because the policy may continue searching before it decides to stop. This helps explain why retrieval order and stopping rules should be evaluated together.
Critical weaknesses
  1. The methodological contribution is not sufficiently established. The paper asks a reasonable but relatively narrow question: what adaptive ordering adds when prediction and feedback-based stopping are already available. The strongest contribution appears to be the empirical analysis of how routing and stopping interact. That analysis is useful, but the case for the controller itself as a substantial methodological advance remains limited.

  2. The motivation and contribution statements need clearer presentation. The introduction gives little concrete explanation of the practical difficulty left unresolved by existing routing and stopping methods. The distinction between finding all required results and deciding to stop is potentially compelling, but it is introduced too briefly. A concrete search example would help explain why this distinction matters. The three stated contributions also mix method design, evaluation procedures, and empirical findings without clearly stating what each item adds.

  3. The evidence for benefits on source-based collections is limited. The clearest improvement comes from the setting where the authors partition a large corpus into semantic clusters. When documents are grouped by their actual source hostnames, the rank-eight policy provides essentially no reduction in search work. This leaves uncertainty about the method’s usefulness when providers are determined by existing data sources rather than constructed semantic clusters.

Questions for authors
  1. What additional benefit requires repeated reordering? In Table 7, one-shot cost ordering uses nearly the same work as the dynamic policy but makes more omissions. Could you compare them at comparable omission risk, using the same budget-admission rule and thresholds selected without test labels? An independently evaluated one-shot baseline would make this comparison more convincing.
  2. How important is information value in the actual experiments? Appendix K shows that Equation 5 can overlook providers that offer useful information without contributing a final top-k document. Can you provide a targeted diagnostic of whether this behavior occurs in the evaluated collections, and explain why the chosen acquisition rule is appropriate despite this limitation?
  3. What explains the difference between the semantic and source-based settings? Since both the corpus and the partitioning change, the current comparison cannot separate their effects. Could you provide a controlled comparison using the same corpus under both partitioning schemes, or other evidence identifying which properties are associated with the observed benefit?
2 comments

Thank you for the detailed comments. We have made the motivation and contribution statements more concrete and added the requested one-shot, information-value and same-corpus partition comparisons.

Response to Weaknesses 1–2: Motivation and contributions.

Consider a query whose global top-10 documents are already among the returns from A and B. The service cannot observe that completion: C still appears competitive, so it may pay for C before stopping. An earlier complete result therefore need not mean a cheaper executed search. This illustrative example now precedes the method, and the replay analysis measures the effect on actual queries.

The contribution statements now specify the paid-feedback controller, a work decomposition that explains where savings arise, and evidence for adaptation's incremental value and operating conditions. The controller couples paid residual updates, competitiveness per cost and learned stopping under the complete-result omission criterion and per-query cap. The added evidence identifies substantial benefits from stopping feedback and cost prioritization, with a more conditional increment from repeated reordering. In particular, fixed order with/without returned-score information uses 2.426M/3.277M work on TEST-2, a 25.98% reduction, with both policies independently accepted at 5%. This isolates the value of return-informed stopping; the following comparison measures the added effect of reordering.

Response to Question 1: An independently evaluated one-shot baseline.

One-shot and dynamic now share the public seed and budget admission rule; each receives its own stop head trained on its TRAIN trajectories. All thresholds are selected on historical TUNE, before new CERT-2/TEST-2 outcomes are opened.

Own-head policyTEST-2 work (M)CERT-2 boundTEST-2 errors / 8,192
Fixed2.4263.65%247
One-shot cost2.3874.17%307
Dynamic2.4023.49%252

All pass the 5% target. At these frozen operating points, dynamic uses 0.62% more work than one-shot and makes fewer observed omissions. Equal acceptance targets leave room for different empirical risks. Descriptive interpolation at 4% empirical risk gives one-shot/dynamic work of 2.248M/2.065M (8.1% less for dynamic), and at 5% gives 1.939M/1.843M (4.9% less). These curve comparisons indicate an operating-point-dependent advantage over one-shot; fixed order is cheaper at the same interpolated risks. The interpolated points are descriptive and carry no new independent certificates.

Response to Question 2: Information from providers that contribute no final result.

The replay shows that these returns change subsequent search decisions. Replaying the original EVAL paths exactly, returns from providers containing no global top-10 document change the next choice in 21.7% of 42,342 eligible semantic steps and 23.1% of 100,581 source steps. Among changed choices, the updated versus pre-update candidate contains a top-10 item in 3.3%/2.9% of semantic cases and 8.2%/6.0% of source cases. The comparison uses the identical next-step affordable set.

The diagnostic measures the influence of paid information on selection, including residual and boundary updates. Measuring the causal benefit of deliberately selecting an information-only provider would require a separate intervention. Equation 5 is a tractable direct-competitiveness heuristic; the retained counterexample in Appendix K explains why it can miss such information-seeking actions.

Response to Weakness 3 and Question 3: A same-corpus partition control.

On the identical 992,502-document source corpus, hostname fixed/dynamic work is 777,339/777,387, while 100 k-means providers give 356,721/327,410, an 8.22% reduction at the common 3.5% TUNE guard. EVAL errors are 63/51 and 96/102, respectively, out of 2,048. The cap remains 866,297; the reference rank and models are refitted per partition. The mean number of necessary providers changes from 5.94 to 3.11, and fixed-order after-completion work share from 26.4% to 69.6%.

The gain therefore also appears on this corpus after repartitioning. Provider sizes, reference rank and answer concentration change together, so the experiment identifies their joint effect. Balanced-random providers fail the guard for all tested policies. Section 4.6 and Appendix M report these outcomes alongside the adverse FiQA and full-covariance controls, making the conditions of the measured benefit explicit.

Thank you for the response. We’re keeping our original rating. The paper’s strongest contribution is its analysis of why finding the complete result earlier may still leave total search work unchanged. The controller builds on existing ideas for predicting provider scores and deciding when to stop, adding a rule that uses search results and costs to reorder the remaining providers. In our view, this is a modest methodological advance.

Reviewer_KEpJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates cost-aware adaptive routing and stopping for multi-provider retrieval. It combines neural maximum-score prediction, residual updates from paid feedback, cost-sensitive routing, and learned stopping. Experiments on MS MARCO report a 6.59% reduction in retrieval work under the rank-eight configuration.

Critical strengths
  1. The paper addresses the challenge of minimizing retrieval costs while controlling omission risk in multi-provider retrieval systems.
  2. The paper employs independent risk certification and extensive experiments to evaluate retrieval efficiency under explicit omission-risk constraints.
  3. The decomposition of retrieval work before and after result completion provides useful insights into the interaction between routing and stopping.
Critical weaknesses
  1. The method relies on predefined provider partitions but does not evaluate the effects of partition granularity, semantic heterogeneity, or distribution shifts. Its effectiveness under alternative or imperfect partitioning strategies remains unclear.
  2. The framework largely combines existing techniques, including source routing in RAGRoute and maximum-score prediction in AMIPS. Moreover, Table 2 shows that dynamic routing incurs higher work than fixed routing under both shared stopping thresholds (2.518M vs. 2.465M at θF; 2.303M vs. 2.257M at θD), although with fewer omissions.
  3. Section 4.4 shows that one-shot cost prioritization already explains much of the improvement, questioning the necessity of repeated feedback. Furthermore, Table 1 reports a 6.59% work reduction for semantic rank-eight routing but a 6.00% increase under full covariance and almost no improvement for source-pure rank-eight providers.
  4. The document-count cost metric does not necessarily reflect actual retrieval latency or system overhead. In Section 4.6, dynamic retrieval exhibits higher single-query service time than parallel static search (398 ms vs. 284 ms), despite reducing document work.
Questions for authors

Q1. How sensitive is the method to provider partitioning quality, granularity, and distribution shifts? Q2. What is the key methodological contribution beyond RAGRoute and AMIPS? Q3. How much of the improvement comes specifically from repeated feedback rather than one-shot cost prioritization? Why do the efficiency gains vary substantially across covariance models and provider conditions? Q4. How well does the document-count-based metric reflect practical retrieval efficiency? Could the authors provide more comprehensive end-to-end latency comparisons with conventional retrieval approaches?

1 comment

Thank you for these questions. We added a same-corpus partition control, component comparisons and a larger execution benchmark. The results address the four concerns below.

Response to Question 1: Sensitivity to provider partitioning.

We now hold the corpus fixed: the same 992,502 documents, query roles and embedding/scoring pipeline are partitioned into 100 hostname, spherical k-means or balanced-random providers. The total cap remains 866,297 documents; the predictor and stopping head are refitted and the reference rank is recalibrated per partition.

PartitionFixed / dynamic workFixed / dynamic EVAL errorsDynamic saving
Hostname777,339 / 777,38763 / 51−0.01%
k-means356,721 / 327,41096 / 1028.22%
Random863,321 / 863,2151,432 / 1,494TUNE guard infeasible

EVAL contains 2,048 queries, and the first two pairs meet the common 3.5% TUNE guard. The gain therefore also appears on the source corpus when its provider partition changes. This is a joint partition effect: provider-size skew, reference rank and the number of necessary providers change together. Provider count is held fixed, and the experiment evaluates repartitioning of a known corpus; sensitivity to provider count and unseen distribution shift remains untested.

Response to Question 2: Contribution beyond the public router and maximum predictor.

RAGRoute-style public scores and AMIPS's SupportNet supply the initial information. Our controller specifies how paid returns update the residual state, affect competitiveness per cost and inform stopping, while every executed path respects the budget and the complete frozen policy receives an omission-risk test. The evaluation then separates the value of these decisions. At the independent 5% target, dynamic saves 11.12% over cost-agnostic selection and 3.98% over an Optimist-style score control with the same predictor and own-trajectory stop-head training.

The threshold-exchange results in Table 2 are retained. They show that applying the same numerical cutoff changes both work and omissions: a shared score threshold does not impose equal risk. The original 6.59% is a comparison of complete policies, each with its TUNE-selected stopping point. That frozen-policy result replicates at 6.38% on TEST-2, with independent bounds of 3.67%/3.64% for fixed/dynamic.

Response to Question 3: Incremental value of repeated feedback.

We now independently test one-shot and dynamic actors with the same admission rule and a separately fitted stop head for each. At the 5% target, work is 2.387M/2.402M and TEST-2 omissions are 307/252; both pass. Thus repeated updates trade a small amount of work for fewer observed omissions at this operating point. Descriptive interpolation at 4% empirical risk gives 2.248M/2.065M, an 8.1% dynamic reduction relative to one-shot; these interpolations are not separately certified operating points. Fixed order is also competitive on the empirical frontier.

The source result has a concrete explanation in replay: among common successes, dynamic saves 75,235 document units before completion but spends 75,266 more afterwards, leaving little net gain. The cause of the full-covariance reversal remains unresolved. We retain that result and use the partition and training controls to show how the benefit varies across conditions.

Response to Question 4: Actual execution efficiency and latency.

We expanded real Faiss/GPU execution to 1,024 original TEST queries per condition and three randomized blocks. With two concurrent queries and the same two CPU cores, semantic dynamic reaches 5.18 queries/s versus 4.84 fixed and 3.57 static; CPU time is 385.5/412.3/559.6 ms per query. At one concurrent query, mean service time is 383.0 ms dynamic, 397.2 ms fixed and 283.7 ms parallel static; P99 is 701.5/683.2/321.2 ms. These results establish a throughput benefit under shared resource limits while preserving the single-query latency advantage of parallel static.

The benchmark includes prediction, controller work and actual searches, starting from embeddings. Encoding, network, cold loading and queueing are excluded, and no new policy is trained on heterogeneous latency prices. Section 4.7 and Appendix M make that scope explicit.

Reviewer_KZTZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper deals with an interesting problem. When you perform retrieval across various providers with varying search costs, there are critical decisions of searching and stopping. When you do exhaustive search, you can get the top k optimally, but how do you Balance cost.

The paper proposes a controller which is a predictor of provider maximum scores. The author also presents experimental results on MS Marco.

Critical strengths
  • The paper clearly addresses a relevant and important problem, which is becoming even more important for agentic search.
  • The cost saving in the experiment seemed real.
  • I enjoy reading some parts of the appendix.
Critical weaknesses
  • Table 1 is quite hard to read. Important results are not bolded. Similarly, for Figure 2, for the right-hand side figure, it's not clear which one is the clear winner.
  • This is a minor weakness, but the quality of the figures is not great, including Figure 1. Even for Figure 3, the length of the y-axis is unnecessarily high, and there is no consistency in the number of decimal places used for the numbers.
  • The critical issue in the paper is that the novelty of the algorithm is quite limited. There is nothing substantially better, and I am wondering: did the authors try to compare with constraint optimization formulations?
  • The experiments also seem quite limited. The neural training runs are quite few in number.
Questions for authors

Please read weaknesses.

1 comment

Thank you for the comments. We have revised the figures and tables, added the requested constrained-optimization baselines, and expanded the training repeats. We address each concern below.

Response to Weaknesses 1–2: Readability of the tables and figures.

Table 1 now marks the lowest mean work and lowest P99 separately within each condition. Mean work and P99 are treated as separate objectives. Figure 2 uses consistent numerical precision and distinguishes mean bars from P99 ticks; the source rank-eight pair is effectively tied, while full-covariance dynamic has the lowest source mean. We redrew Figure 1 around the online predict–select–pay–update loop, and Figure 3 uses horizontal stacked bars with shorter labels and consistent units. The underlying submitted results are retained.

Response to Weakness 3: Constrained-optimization baselines and methodological contribution.

The new comparison shows lower work for feedback-stopped search than for constrained one-shot selection at the same independently tested risk target. We use the same SupportNet prior, common paid seed, provider costs and work caps. After the seed, the baseline selects the minimum-cost provider set whose excluded predicted competitiveness mass is at most epsilon. Its discretized 0/1-knapsack solver matches exhaustive enumeration on 2,400 small instances. Epsilon is selected on TUNE using the same mean-work-plus-0.25-P99 objective and 3.5% omission guard as the controllers.

We also evaluated seed-conditioned, budget-aware, learned-probability-calibrated and calibrated-plus-budget-aware variants. The best TUNE-feasible secondary variant was selected before independent evaluation: seed-conditioned on semantic providers and calibrated on source providers. Appendix M now reports all five variants' development outcomes, including infeasible ones.

On the fresh semantic TEST-2 set, both constrained baselines and both feedback-stopped controllers pass the independent 5% target:

PolicyTEST-2 mean work (M)CERT-2 boundTEST-2 omissions / 8,192
Constrained one-shot set3.5983.81%262
Seed-conditioned constrained set3.5553.85%265
Fixed order, feedback stopping2.4263.65%247
Dynamic, feedback stopping2.4023.49%252

Feedback-stopped fixed order uses about 32.6% and 31.8% less work than the two constrained baselines, with fewer observed omissions (247 versus 262/265). Dynamic uses 2.402M with 252 omissions. This comparison evaluates each executable policy, including cap truncation, against the same independently tested risk target. The predicted mass is a surrogate; acceptance comes from the complete-policy test.

The controller addresses the sequential decision that remains after the initial search. The constrained optimizer chooses a provider set from the seed observation. Our controller uses each later paid maximum to update the residual state, rank the remaining providers by competitiveness per unit cost, and decide whether to stop. Finding the complete top-10 can precede the point at which the available evidence supports stopping, so later feedback can affect both search order and executed cost. The budget rule and independent omission test apply to the full policy.

The component controls support this mechanism directly. Keeping fixed order, allowing the stop head to use paid-return features reduces work from 3.277M to 2.426M, a 25.98% saving [25.27%, 26.72%], with both policies accepted at 5%. Section 4.4 separately evaluates the additional effect of repeated reordering through one-shot and boundary-only controls. This separates the improvement over constrained set selection from the additional effect of dynamic ordering.

Response to Weakness 4: Repeated training and variation across runs.

We now report five neural initialization/minibatch seeds, each trained for 100,000 steps, with TUNE-selected checkpoints, refitted rank-eight residuals and a refitted paired stop head. Their dynamic savings on the same development EVAL set are 6.79%, 8.94%, 5.80%, −6.99% and 10.01%. Four of five runs save work; the mean is 4.91%, with sample SD 6.86 percentage points. All five declared runs are reported.

Separately, the original frozen model reproduces a 6.38% saving [6.08%, 6.68%] on the new 8,192-query TEST-2 set. The new test measures query generalization for that model; the five runs measure training variation. The replication is reported in Section 4.2; the five training runs are in Section 4.5, Table 12 and Appendix M.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.