acceptodds
Under review as a conference paper at ICLR 2027

PACO-PO: Policy-Adaptive Cost-Constrained Policy Optimization for Joint Reasoning Effort and Retrieval Depth in Budgeted Single-Turn RAG

Abstract

Single-turn retrieval-augmented question answering must decide, for every query, how hard to reason and how deeply to retrieve, and providers charge both the generated reasoning and the retrieved passages to one token budget. However, existing controllers tune the two levers apart, length control for reasoning effort and retrieval control for depth, so they miss that strong retrieval substitutes for reasoning while weak retrieval demands more of it, and cost-aware training bills reasoning alone, through a reward weight that must be swept or a per-query target length. We propose PACO-PO, a controller of roughly 0.2M parameters that, with the base LLM and retriever frozen, treats reasoning effort and retrieval depth as one coupled budgeted decision and learns it as a factorized joint effort–depth policy whose depth head is conditioned on the selected effort. The policy reads a zero-cost retrieval preview, the retriever's score profile and the passage lengths, to judge whether retrieval already suffices before any passage enters the context, so the choice itself spends no LLM tokens. Training casts the choice as a one-step constrained Markov decision process optimized by group-relative policy optimization, where a single Lagrangian dual variable prices the token budget, so one training run replaces a reward-weight sweep. On HotpotQA, Natural Questions, and MATH-500 with retrieval, PACO-PO comes within one point of the best fixed configuration (46.9% versus 47.6% macro accuracy) at 64% of its token cost, exceeds the best budget-matched single-axis controller by 2.9 points, and keeps the macro training-time expected cost within 3% of the budget.

Primary area:
optimization
Keywords:
retrieval-augmented generation, adaptive reasoning effort, retrieval depth selection, token budget allocation, constrained Markov decision process, group relative policy optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a5RxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies per-query allocation of reasoning effort and retrieval depth in single-turn RAG under a single expected token budget. PACO-PO is a ~0.2M-parameter controller (base LLM Llama-3.1-8B-Instruct and BGE retriever both frozen) that picks one of 12 (effort, depth) cells: reasoning caps of 128/1024/4096 tokens x k ∈\in {0, 3, 5, 10} passages. It reads a MiniLM query embedding plus a "retrieval preview" (top-10 retriever score statistics and passage lengths) that costs no LLM tokens. The policy is factorized as p(e)·p(k|e), framed as a one-step constrained MDP, and trained with GRPO, where a single Lagrangian dual variable prices the budget (B = 0.65x the cost of the best fixed cell). On HotpotQA, NQ and MATH-500 (with retrieval over MATH training solutions), PACO-PO reaches 46.9% macro accuracy at 0.64 of the best fixed cell's cost (47.6%). It also beats a confidence cascade, a difficulty router and a LinUCB bandit (44.4–45.5%), which spend more (0.71–0.76). Ablations cover single-axis controllers, a flat 12-way head, removing the preview, a frozen price, a tuned scalarization weight, and transfer to Qwen2.5-7B.

Critical strengths
  • The problem is of interest to atleast one sub-field of researchers as reasoning tokens and retrieved context draw on the same bill, and letting the deployer set a budget instead of an exchange-rate weight is a sensible interface.
  • Each component (factorized head, score-profile features, primal–dual pricing on top of GRPO) is simple, clearly motivated and cheap with the controller having ~0.2M parameters and the base LLM and retriever staying frozen.
  • PACO-PO uses ~36% fewer tokens than the best fixed cell for a 0.7-point macro accuracy drop. On HotpotQA it is 4.8 points above the best budget-matched random mix of fixed cells (Table 2). The savings over the simplest adaptive baseline are smaller, though (see Weakness 3).
  • Well organized ablations. Table 3 changes one thing per row at the same budget. The frozen-λ row shows that budget tracking depends on the price moving during training, not only on where it ends up.
Critical weaknesses
  • The methodological contribution is incremental. Primal–dual Lagrangian relaxation for constrained RL is standard. GRPO is used off the shelf. The novelty therefore lies mainly in applying these tools to a two-dimensional action grid.
  • The experiments are too narrow. Three datasets seem too few for claims about general per-query allocation, and two of them (HotpotQA, NQ) are Wikipedia QA over the same index.
  • Each baseline is shown at a single operating point that costs more than PACO-PO (0.71–0.76 vs. 0.64). Sweeping each baseline's knob (cascade threshold, router mapping) and comparing Pareto curves, or comparing at matched cost, would be fairer. It is also unclear what the cascade threshold and router mapping were tuned for on the probe split.
  • PACO-PO adds components to the pipeline including a retriever call on every query (including queries that end up at k = 0), a MiniLM encoder and the controller. End-to-end serving latency should therefore be measured, and I could not find it in the paper.
  • No measure of statistical uncertainty. The ablations are also single-seed.
  • The paper also overstates the claim that prior work treats the two axes separately. Yue et al. (Inference Scaling for Long-Context Retrieval Augmented Generation) jointly allocate the number of retrieved documents and the inference iterations under a compute budget, using a computation-allocation model.
Questions for authors
  • The controller trains on MATH training problems, and the retrieval corpus contains MATH training solutions. Unless self-matches are removed, retrieval during training returns each query's own solution, which cannot happen on MATH-500 at evaluation. Please clarify.
  • Normalization, budget and training pool all appear to be set per benchmark. Real deployments see a mixed query stream. Training a single controller on the union and evaluating on a mixture, or across datasets, would test the routing claim much better. More multi-hop or knowledge-intensive sets (e.g., 2WikiMultiHopQA, MuSiQue, PopQA) and a non-math reasoning set will also help.
  • Every experiment uses B = 0.65. The method's interface is "the deployer sets a budget", so it needs accuracy–cost curves across several budgets, with baselines at matched cost. All experiments use 7–8B instruct models, with reasoning effort set by a prompt instruction plus a hard stop. Experiments with broader set of reasoning models with reasoning effort controls will help.
  • Please give the split of savings into input vs. output tokens, plus p50/p95 latency and throughput under batched serving (vLLM is already in the stack). This would considerably strengthen the practical claim, and it could favor PACO-PO over the cascade, which needs several LLM calls.
5 comments
AuthorsOpenReview ↗

Response to Reviewer a5Rx

Dear Reviewer a5Rx:

Thank you for recognizing the task value, the method design, and the method effectiveness. We have responded point by point to your concerns regarding the contribution, more datasets, baselines at matched cost, latency, statistical uncertainty, positioning, self-matches, and multiple budgets. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer a5Rx. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Incremental methodology (Weakness 1)

A1: We respectfully clarify that the contribution is the problem and its formulation, while primal-dual relaxation and GRPO [1] are standard tools that PACO-PO runs on a one-step constrained problem. First, one expected token budget charges generated reasoning and retrieved context together, which couples effort and depth through cost. Second, a featurization is read from the retriever's scores and the index metadata before any passage enters the context. Third, the constraint is written relative to B, which lets one dimensionless price and one step size serve benchmarks with different token scales. Proposition 1 shows that the dual function is finite and convex without concavity of the inner problem, which a neural controller does not have.


Summarized W2: Three datasets, two over one index (Weakness 2, Question 2)

A2: Following your suggestion, we further evaluate PACO-PO and the difficulty router on 2WikiMultiHopQA, MuSiQue and PopQA, and on a mixed stream of the three benchmarks with one controller trained on the union of their training splits under one normalization and one budget. Table R1 shows that PACO-PO is more accurate than the router on the mixed stream at a lower cost, with 46.2% at 0.66 against 44.5% at 0.71. HotpotQA and NQ both retrieve from a Wikipedia index, which holds the retrieval stack fixed while the query type changes from multi-hop to single-hop, and the revised version contains no non-math reasoning set.

Table R1: Accuracy (%) and cost fraction beyond the three benchmarks, with PACO-PO and router accuracy as three-seed means and the recalibrated Qwen2.5-7B controller as a one-seed macro.

SettingBest fixed acc.PACO-PO acc.PACO-PO costRouter acc.Router cost
2WikiMultiHopQA45.745.50.6643.00.73
MuSiQue38.337.90.6537.20.72
PopQA44.643.30.6539.00.68
Mixed stream (three benchmarks)47.746.20.6644.50.71
MATH-500 without self-matches52.051.10.6850.20.79
Qwen2.5-7B, recalibrated47.745.80.67(n/a)(n/a)

Following your suggestion, we have added these results to Appendix A.9 as Table 17 (Lines 1703–1714), and summarized them in Section 5.3 (Lines 454–457).


Summarized W3: Baselines at a single operating point (Weakness 3)

A3: Following your suggestion, we further read the cascade and the router at the cost of PACO-PO for three budgets on HotpotQA, each at its setting whose probe cost lies nearest that cost, beside the best mix of all 12 fixed cells. Table R2 shows that PACO-PO is above both baselines at each budget, with 47.6% against 45.8% for the router at B = 0.65 and 48.1% against 48.0% for the router and 47.7% for the cascade at B = 0.8, where the evaluation cost of each baseline is not printed and need not equal that of PACO-PO, and by our computation for this response each entry carries a binomial standard error of at most 1.58 points. On the probe split, the cascade threshold and the router mapping are tuned for the highest probe accuracy among the settings whose probe cost fraction is at most 0.85, and none of the three baselines reads B.

Table R2: HotpotQA accuracy (%) at three budget fractions, with the cascade and the router read at the setting whose probe cost lies nearest the cost of PACO-PO.

BudgetPACO-PO acc.PACO-PO costCascade at that costRouter at that costBest 12-cell mix at that cost
B = 0.547.10.5241.342.943.2
B = 0.6547.60.6545.145.844.9
B = 0.848.10.7947.748.046.5

Following your suggestion, we have added these results to Appendix A.9 as Table 16 (Lines 1690–1698), and described the baseline implementations and how the cascade threshold and the router mapping are tuned in Appendix A.4 (Lines 1191–1205 and 1207–1216).

(Continued)

Summarized W4: Serving latency and the input/output split (Weakness 4, Question 4)

A4: As you suggested, we further report the input and output tokens per query and a modeled token-processing latency at batch 1. Table R3 shows that the savings come from both input and output tokens, as PACO-PO saves 474 input and 87 output tokens per query against the best fixed cell. The modeled latency covers token processing and the controller overhead, while the retriever call, which is part of every action with k > 0 and of every retrieving baseline and adds a call for the queries routed to k = 0, the MiniLM encoding and the cascade round trips come on top of it, and end-to-end latency, latency percentiles and batched throughput are not reported.

Table R3: HotpotQA mean input and output tokens per query and modeled token-processing latency per query at batch 1.

MethodInput tokensOutput tokensModeled latency (s)
Best fixed cell (HIGH, 10)13013148.5
Confidence cascade76444811.9
PACO-PO (Ours)827 (-474)227 (-87)6.0

Following your suggestion, we have added these results to Appendix A.9 as Table 21 (Lines 1747–1753).


Summarized W5: Statistical uncertainty (Weakness 5)

A5: We respectfully clarify that an upper bound on the binomial standard error of a single accuracy estimate, which we compute for this response, is 1.58 points per QA benchmark, 2.24 points on MATH-500 and 1.05 points for the macro. Following your suggestion, we further report the across-seed standard deviations of the main comparison in Table R4, where the macro accuracy of PACO-PO is 47.1 ± 0.8, and of the ablation rows in Table R5, whose learned rows are three-seed means in the revised version.

Table R4: Accuracy (%) of the five methods of Table 1 over three seeds as mean ± standard deviation.

MethodHotpotQANQMATH-500MacroMacro diff. to PACO-PO
PACO-PO (Ours)47.6 ± 2.042.5 ± 2.251.1 ± 3.947.1 ± 0.8(ref.)
Best fixed cell48.4 ± 0.042.7 ± 0.052.0 ± 0.047.7 ± 0.00.6
Confidence cascade47.4 ± 2.141.6 ± 1.145.9 ± 1.645.0 ± 1.2-2.1
Difficulty router46.5 ± 1.239.2 ± 1.750.8 ± 1.245.5 ± 0.8-1.6
Bandit (LinUCB [2])46.8 ± 1.137.0 ± 2.249.4 ± 1.844.4 ± 0.8-2.7

Table R5: Macro accuracy (%) of the ablation rows of Table 3 over three seeds as mean ± standard deviation, where the parameter-matched flat head is printed without its spread, and after 10 and 20 of the 40 outer iterations.

ConfigurationMacro acc. (40 it.)After 10 it.After 20 it.
PACO-PO (Ours, full)47.1 ± 0.844.945.5
Flat 12-way head45.4 ± 0.4 (-1.7)43.043.8
Flat 12-way head, parameter-matched45.6 (-1.5)42.143.4
Without retrieval preview45.0 ± 1.541.943.6
Learned depth only43.0 ± 2.139.942.5
Learned effort only44.0 ± 1.041.743.3
lambda frozen44.9 ± 0.342.143.2

Following your suggestion, we have added these results to Appendix A.9 as Tables 13 and 15 (Lines 1649–1658 and 1674–1684), summarized the seed spread in Section 5.3 (Lines 451–453), and updated the Limitations in Appendix A.0 (Lines 841–854).


Summarized W6: Positioning against Yue et al. (Weakness 6)

A6: We respectfully clarify that our setting differs from that of Yue et al. [3] in the per-query choice. Yue et al. allocate retrieved documents and generation iterations under a compute budget at the level of a task, whereas PACO-PO chooses for each query from a token-free retrieval preview under a budget that charges the generated reasoning tokens together with the retrieved context. The separation our paper describes is between per-query controllers of reasoning length and of retrieval depth.

Following your suggestion, we have limited the claimed separation to per-query controllers in the abstract (Lines 14–33), the caption of Figure 1 (Lines 46–50) and the first contribution (Lines 79–81), revised the related work on adaptive retrieval and reasoning effort (Lines 460–466), and added Yue et al. [3] to the related work of Appendix A.1 (Lines 884–901).

(Continued)

Summarized W7: Self-matches in the MATH reference corpus (Question 1)

A7: We respectfully clarify that with each training problem's own solution excluded from retrieval, PACO-PO keeps its MATH-500 accuracy of 51.1%. The training, probe and evaluation pools are disjoint, and the MATH-500 retrieval corpus holds the MATH training solutions. In the added run that removes each training problem's own solution from retrieval, Table R1 above shows 51.1% at a cost of 0.68 for PACO-PO, against 51.1% at 0.67 in Table 1, and 50.2% for the router.

Following your suggestion, we have added this result to Appendix A.9 as part of Table 17 (Lines 1703–1714), and added the self-match paragraph to the Limitations in Appendix A.0 (Lines 831–834).


Summarized W8: Several budgets and other reasoning models (Question 3)

A8: We respectfully clarify that a new budget needs a new training run under our training procedure, because the controller does not read B, and budget conditioning is the direction that removes this. Table R2 above gives HotpotQA at B = 0.5, 0.65 and 0.8 with the cascade and the router read near the cost of PACO-PO, and Table R1 above gives a recalibrated Qwen2.5-7B controller. The tables contain no model family with native effort control.

These results are in Appendix A.9 as Tables 16 and 17 (Lines 1690–1698 and 1703–1714).


References

[1] Zhihong Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. 2024.

[2] Lihong Li et al. A contextual-bandit approach to personalized news article recommendation. WWW, 2010.

[3] Zhenrui Yue et al. Inference scaling for long-context retrieval augmented generation. ICLR, 2025.


Best Wishes,

The Authors of Paper 52793

Reviewer_zDAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACO-PO, a lightweight policy that jointly allocates reasoning effort and retrieval depth for budget-constrained single-turn RAG. It combines retrieval-score previews, a factorized action policy, and Lagrangian budget optimization. Experiments on HotpotQA, Natural Questions, and MATH-500 report near-fixed-policy accuracy with substantially reduced token consumption.

Critical strengths
  1. Joint resource allocation is practically meaningful. PACO-PO jointly selects reasoning effort and retrieval depth under a shared token budget, addressing an important limitation of independently configured inference pipelines.

  2. The empirical accuracy-cost trade-off is promising. Table 1 reports 46.9% macro accuracy at 64% of the best fixed configuration's token cost, while Table 3 provides evidence that joint control improves over single-axis alternatives.

Critical weaknesses
  1. The claimed budget compliance is not established for the deployed policy. Training uses stochastic action sampling, whereas deployment uses greedy selection. The reported budget violations are measured differently across methods, undermining a central comparison.

  2. The experimental evidence does not sufficiently isolate the proposed method's advantages. The budget-matched static mixture covers only six representative cells, while key ablations use single seeds without uncertainty estimates. Consequently, the significance of the reported improvements remains uncertain.

Questions for authors

(1) In Eq. (6), deployment uses greedy action selection, whereas Eq. (14) constrains the expected cost under stochastic training policies. Appendix A.4 further reports training-time violations for PACO-PO but evaluation-time violations for baselines. How can these values be directly compared? Could the authors report deployment-time budget violations for all methods under an identical protocol?

(2) Table 2 claims that PACO-PO outperforms the best budget-matched static mixture, but the mixture is optimized over only six representative configurations rather than all 12 available actions. This restriction may underestimate the achievable static accuracy-cost frontier. What is the performance of the optimal mixture over all 12 cells under the same budget?

(3) Table 3 attributes a 1.3-point improvement to the factorized policy over a flat 12-way head, although Eq. (4) establishes equivalent expressiveness. The improvement may reflect optimization variance rather than the proposed conditional structure. Could the authors provide multi-seed comparisons and analyze whether effort-conditioned factorization consistently improves learning efficiency or final accuracy?

(4) The retrieval preview in Eq. (8) relies on benchmark-specific score normalization and assumes that score sharpness indicates retrieval sufficiency. Yet score concentration does not necessarily imply passage relevance or answerability. Could the authors quantify the correlation between preview features and retrieval success, and evaluate robustness under different retrievers or retrieval-score distributions?

(5) Table 5 presents the dual approach as preferable to scalarization, but its accuracy is lower than the closest fixed-weight baseline (44.8% versus 45.2%), with slightly different costs. Moreover, Table 4 shows substantial sensitivity of budget violations to the dual step size. Can the authors provide matched-cost Pareto comparisons and explain whether dual optimization genuinely eliminates tuning rather than shifting it to the step size?

(6) Appendix A.6 uses only 1,000 evaluation queries for each QA benchmark, while most ablations and the backbone-transfer experiment use a single seed. Several reported improvements are below two percentage points. Could the authors provide confidence intervals, paired significance tests, and multi-seed ablation results to establish which gains are statistically reliable?

5 comments
AuthorsOpenReview ↗

Response to Reviewer zDAY

Dear Reviewer zDAY:

Thank you for recognizing the task value and the method effectiveness. We have responded point by point to your concerns regarding budget compliance, mixtures of fixed configurations, more seeds, other retrievers, the comparison with weighted objectives, and statistical reliability. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer zDAY. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Budget compliance of the deployed greedy policy (Weakness 1, Question 1)

A1: Following your suggestion, we further report the deployed deviation from B for PACO-PO, greedy and sampled, and for the three adaptive controllers under one definition. Table R1 shows that the deployed greedy policy of PACO-PO stays below B on all three benchmarks, with a macro deviation of -0.06, while the three adaptive controllers spend above B in the macro. In Table 1 the Viol. column is a training-time estimate for PACO-PO and, for the baselines, a mean over seeds of the absolute per-seed deployed deviation, and Table R1 puts all methods on the deployed one, although the feasibility of the greedy controller is not certified.

Table R1: Signed relative deviation of deployed spend from B, (cost - B)/B, under one definition for all methods.

MethodHotpotQANQMATH-500Macro
PACO-PO (Ours) greedy-0.07-0.03-0.08-0.06
PACO-PO (Ours) sampled-0.08-0.01-0.06-0.05
Confidence cascade0.070.050.130.08
Difficulty router0.030.020.090.05
Bandit (LinUCB [1])0.010.11-0.050.02

Following your suggestion, we have added these results to Appendix A.9 as Table 18 (Lines 1720–1727), revised the definition of the violation in Section 5.1 (Lines 321–323) and Appendix A.4 (Lines 1218–1246), revised the main comparison in Section 5.2 (Lines 333–358), and summarized the deployed spend in Section 5.3 (Lines 454–457).


Summarized W2: The static mixture over six cells (Weakness 2, Question 2)

A2: As you suggested, we quantify the best static mixture over all 12 cells at the same budget. Table R2 gives the remaining six fixed cells on HotpotQA, and Table R3 shows that PACO-PO stays above the static mixture over all 12 cells, with 47.6% against 44.9% on HotpotQA at its cost of 0.65. As stated in Appendix A.5 (Lines 1295–1296), a mix of any number of cells at a fixed expected cost attains its maximum on a mix of at most two.

Table R2: The remaining six fixed cells on HotpotQA.

CellAcc. (%)Cost
(LOW, 5)43.50.53
(LOW, 10)42.00.89
(MED, 0)36.70.62
(MED, 10)47.40.94
(HIGH, 0)36.90.81
(HIGH, 3)45.10.87

Table R3: HotpotQA accuracy (%) at three budget fractions, with the best 12-cell mix read at the cost of PACO-PO and the cascade and the router at the setting whose probe cost lies nearest that cost.

BudgetPACO-PO acc.PACO-PO costCascade at that costRouter at that costBest 12-cell mix at that cost
B = 0.547.10.5241.342.943.2
B = 0.6547.60.6545.145.844.9
B = 0.848.10.7947.748.046.5

Following your suggestions, we have added these results to Appendix A.9 as Tables 19 and 16 (Lines 1728–1736 and 1690–1698), and revised the static-mixture paragraph in Section 5.2 (Lines 360–368) and Appendix A.5 (Lines 1289–1299).

(Continued)

Summarized W3: Factorized versus flat head (Question 3)

A3: Following your suggestion, we further report the ablation rows of Table 3 as three-seed means with their spread, a parameter-matched flat head, and the accuracy after 10 and 20 of the 40 outer iterations. Table R4 shows that the full method lies above the flat head in the three-seed mean, and the parameter-matched flat head reaches 45.6%, 1.5 points below the full method, while, by our computation for this response, the two heads, which Eq. 4 makes equally expressive, also differ in sample sharing, as under a uniform policy with G = 6 actions per query each effort level is selected 2 times per group in expectation against 0.5 times for each flat cell.

Table R4: Macro accuracy (%) of the ablation rows of Table 3 over three seeds as mean ± standard deviation, where the parameter-matched flat head is printed without its spread, and after 10 and 20 of the 40 outer iterations.

ConfigurationMacro acc. (40 it.)After 10 it.After 20 it.
PACO-PO (Ours, full)47.1 ± 0.844.945.5
Flat 12-way head45.4 ± 0.4 (-1.7)43.043.8
Flat 12-way head, parameter-matched45.6 (-1.5)42.143.4
Without retrieval preview45.0 ± 1.541.943.6
Learned depth only43.0 ± 2.139.942.5
Learned effort only44.0 ± 1.041.743.3
lambda frozen44.9 ± 0.342.143.2

Following your suggestion, we have added these results to Appendix A.9 as Table 15 (Lines 1674–1684), described the parameter-matched flat head in Appendix A.4 (Lines 1176–1189), and revised the factorized-head paragraph in Section 5.3 (Lines 371–392) and Appendix A.5 (Lines 1301–1314).


Summarized W4: What the retrieval preview measures (Question 4)

A4: We respectfully clarify that the controller learns its own mapping from the preview features to cells from the rewards of sampled actions, and the per-benchmark normalization serves to align score scales. Following your suggestion, we further evaluate PACO-PO on HotpotQA under two other retrievers (i.e., BM25 and a hybrid BM25 + BGE), each with its own retriever's probe, score normalization and training run. Table R5 shows that the controller reaches 47.6% under BM25 and 46.1% under the hybrid, against 47.6% under the default retriever in Table 1, although the score scales differ. The revised version reports the value of the preview to the decision rather than a correlation with retrieval success, as removing the preview lowers macro accuracy from 47.1% to 45.0% at the same budget.

Table R5: HotpotQA results of a non-learned entropy-threshold rule and of PACO-PO with two other retrievers, accuracy (%) as three-seed means.

ControllerAcc. (%)CostDeviation from B
Entropy-threshold rule (e = MED)43.70.770.10
PACO-PO (Ours), BM2547.60.64-0.09
PACO-PO (Ours), hybrid BM25 + BGE46.10.660.01

Following your suggestion, we have added these results to Appendix A.9 as Table 14 (Lines 1666–1673), and stated the reading of a sharp or flat score profile as a design assumption in Section 2 (Lines 104–122) and in the preview paragraph of Section 5.3 (Lines 394–400).


Summarized W5: Dual versus scalarization, and the step size (Question 5)

A5: We respectfully clarify that the step size needs no search for accuracy, because accuracy stays between 46.7% and 47.4% across the step sizes of Table 4 and one value serves all three benchmarks, while the violation changes from 0.03 at the adopted 0.1 to 0.11 at 0.02 and 0.10 at 0.5. Following your suggestion, we further compare with the scalarization weights 0.1 and 2.0 beside the three of Table 5, where the dual reaches 45.0% at a cost of 0.65 against 45.0% at 0.68 for the weight 1.0. Table R6 shows that accuracy follows spend across the five weights, from 45.4% at a cost of 0.89 for the weight 0.1 to 40.8% at 0.61 for the weight 2.0, with each added weight run with one seed.

Table R6: Two more scalarization weights, macro over HotpotQA and NQ, one seed.

WeightAcc. (%)Cost
alpha = 0.145.40.89
alpha = 2.040.80.61

Following your suggestion, we have added these results to Appendix A.9 as Table 20 (Lines 1739–1744), revised the analyses of the step size and of matched scalarization in Section 5.3 (Lines 411–423 and 424–435), and revised the paragraph on budget pricing via a single dual variable in Section 5.3 (Lines 402–409).

(Continued)

Summarized W6: Confidence intervals and paired tests (Question 6, Weakness 2)

A6: We respectfully clarify that an upper bound on the binomial standard error of a single accuracy estimate, which we compute for this response, is 1.58 points on each QA benchmark, 2.24 points on MATH-500 and 1.05 points for the macro, which bounds a normal-approximation 95% interval around a macro estimate at about 2.07 points on each side. Following your suggestion, we further report the across-seed standard deviations of the main comparison in Table R7, where the macro accuracy of PACO-PO is 47.1 ± 0.8, and of the ablation rows in Table R4 above. The revised version reports no paired test on per-query outcomes.

Table R7: Accuracy (%) of the five methods of Table 1 over three seeds as mean ± standard deviation.

MethodHotpotQANQMATH-500MacroMacro diff. to PACO-PO
PACO-PO (Ours)47.6 ± 2.042.5 ± 2.251.1 ± 3.947.1 ± 0.8(ref.)
Best fixed cell48.4 ± 0.042.7 ± 0.052.0 ± 0.047.7 ± 0.00.6
Confidence cascade47.4 ± 2.141.6 ± 1.145.9 ± 1.645.0 ± 1.2-2.1
Difficulty router46.5 ± 1.239.2 ± 1.750.8 ± 1.245.5 ± 0.8-1.6
Bandit (LinUCB [1])46.8 ± 1.137.0 ± 2.249.4 ± 1.844.4 ± 0.8-2.7

Following your suggestion, we have added these results to Appendix A.9 as Table 13 (Lines 1649–1658), summarized the seed spread in Section 5.3 (Lines 451–453), and updated the Limitations in Appendix A.0 (Lines 841–854).


References

[1] Lihong Li et al. A contextual-bandit approach to personalized news article recommendation. WWW, 2010.


Best Wishes,

The Authors of Paper 52793

Reviewer_MEmNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACO-PO, a lightweight controller that allocates reasoning effort and retrieval depth per query as one coupled decision under a single expected token budget. The authors argue that these two axes are not independent dials but two coordinates of one budgeted decision, since reasoning tokens and retrieved-context tokens are both charged to the same per-query budget. PACO-PO has three components: (1) a factorized joint policy that first samples an effort level e∈{LOW,MED,HIGH}e\in\{\text{LOW},\text{MED},\text{HIGH}\} and then samples a retrieval depth k∈{0,3,5,10}k\in\{0,3,5,10\} conditioned on ee, keeping the two axes coupled; (2) a zero-cost retrieval preview computed from retriever score statistics and index metadata, which informs the allocation before any passage enters the context and spends no LLM tokens; and (3) a single Lagrangian dual variable that converts the user's token budget BB into a price, updated by projected dual descent during training so that one training run replaces a reward-weight sweep. The base LLM and retriever are frozen; only a ~0.2M-parameter controller is trained with group-relative policy optimization.

Critical strengths
  1. The paper identifies a real deployment issue: reasoning effort and retrieval depth draw on the same token budget, yet existing methods typically tune them as separate axes, hand-set rules, or reward weights.

  2. The experiments provide meaningful evidence of effectiveness in several settings.

Critical weaknesses

The main comparison uses only three benchmarks and essentially one base backbone, with a brief same-class transfer experiment; most ablations and the transfer study are single-seed, and the main comparison reports point estimates without standard errors.

Please see more in the Questions below.

Questions for authors
  1. Is the probe cost (running all 12 cells on 200 queries per benchmark) included in the reported cost fractions or GPU-hour totals? If not, how would including it change the effective cost of PACO-PO relative to the best fixed cell, and does the "one training run replaces a weight sweep" claim still hold once the probe is charged?

  2. The paper identifies the entropy of the softmax-normalized score profile as operationalizing the sharp-versus-flat distinction. Could a simple non-learned rule based on this entropy (e.g., a threshold on H(σ(s1:10))H(\sigma(s_{1:10})) combined with a fixed effort level) achieve similar accuracy at similar cost? This would test whether the learned controller is necessary for the preview's benefit, or whether the gain comes primarily from the feature itself.

  3. The flat head reaches 45.6% versus 46.9% for the factorized policy despite having a more flexible parameterization over the same 12 cells. Is this improvement due to the inductive bias of effort-conditioned depth, or to optimization effects (fewer effective parameters, better gradient flow)? A principled explanation would strengthen the design justification beyond the post-hoc "substitution pattern" argument.

  4. The budget is instantiated as 0.65× the best-cell probe cost. How sensitive is the method to this fraction? If a deployer targets 0.5 or 0.8, does the method degrade gracefully, or would retraining or recalibration be needed? Appendix A.8 mentions budget-conditioning as future work, but a sensitivity analysis at the current setting would clarify how much of the result depends on this choice.

  5. The transfer experiment shows cost drifting from 0.64 to 0.69 after swapping Llama-3.1-8B for Qwen2.5-7B. What recalibration procedure would the authors recommend for a practitioner deploying on a new backbone without retraining the controller — for example, re-running the probe to refit the price, or rescaling BB by the observed cost drift?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MEmN

Dear Reviewer MEmN:

Thank you for recognizing the task value and the method effectiveness. We have responded point by point to your concerns regarding more datasets, more seeds, the probe cost, a non-learned rule, the model design, budget sensitivity, and backbone recalibration. In the marked copy appended to the revised PDF (pages 34–67), the revisions made in response to your comments are tagged with @Reviewer MEmN. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Three benchmarks, one backbone, single seeds, no standard errors (Critical Weakness)

A1: As you suggested, we further report the across-seed standard deviations of Table 1, and for this response we compute upper bounds on the binomial standard error of 1.58 points on each QA benchmark, 2.24 points on MATH-500 and 1.05 points for the macro accuracy. Table R1 shows that the macro accuracy of PACO-PO is 47.1 ± 0.8 over three seeds. Table R3 below gives the seed spread of the ablation rows, Table R5 below gives further datasets and a recalibrated Qwen2.5-7B controller, and the three benchmarks place the difficulty in different places, namely multi-hop retrieval and reasoning on HotpotQA, single-hop retrieval on NQ, and reasoning on MATH-500.

Table R1: Accuracy (%) of the five methods of Table 1 over three seeds as mean ± standard deviation.

MethodHotpotQANQMATH-500MacroMacro diff. to PACO-PO
PACO-PO (Ours)47.6 ± 2.042.5 ± 2.251.1 ± 3.947.1 ± 0.8(ref.)
Best fixed cell48.4 ± 0.042.7 ± 0.052.0 ± 0.047.7 ± 0.00.6
Confidence cascade47.4 ± 2.141.6 ± 1.145.9 ± 1.645.0 ± 1.2-2.1
Difficulty router46.5 ± 1.239.2 ± 1.750.8 ± 1.245.5 ± 0.8-1.6
Bandit (LinUCB [1])46.8 ± 1.137.0 ± 2.249.4 ± 1.844.4 ± 0.8-2.7

Following your suggestion, we have added these results to Appendix A.9 as Table 13 (Lines 1649–1658), summarized the seed spread in Section 5.3 (Lines 451–453), and updated the Limitations in Appendix A.0 (Lines 841–854).


Summarized W2: Whether the probe is charged (Question 1)

A2: We respectfully clarify that the compute footprint of Appendix A.6 counts the probe, one per benchmark, whereas the cost fractions are per-query deployment costs. Charged to both sides, the episode counts of Appendix A.6 give, by our computation for this response, 7,680 + 2,400 = 10,080 episodes for the dual against 3 x 7,680 + 2,400 = 25,440 for the three-weight sweep, a ratio of 2.52 in episodes. Against the best fixed cell, whose setup is the probe alone, the probe and the training run are a one-time cost, and, by our argument for this response, as the number of deployed queries grows the token ratio of PACO-PO to the best fixed cell tends to its cost fraction on each benchmark (0.65, 0.65 and 0.67). The revised version prints no training tokens, and it gives no value of this ratio at a finite number of queries.

The compute footprint, with the probe counted, is given in Appendix A.6 (Lines 1483–1497).


Summarized W3: A non-learned entropy rule (Question 2)

A3: Following your suggestion, we further compare with a non-learned entropy-threshold rule at a fixed effort (e = MED), which retrieves k = 10 when the entropy lies above its threshold and k = 3 otherwise, with the threshold set on the probe split. Table R2 shows that the non-learned rule reaches a lower accuracy at a higher cost than the learned controller, with 43.7% at a cost of 0.77 on HotpotQA against 47.6% at 0.65 for PACO-PO in Table 1. Table R2 also gives PACO-PO under two other retrievers.

Table R2: HotpotQA results of a non-learned entropy-threshold rule and of PACO-PO with two other retrievers, accuracy (%) as three-seed means.

ControllerAcc. (%)CostDeviation from B
Entropy-threshold rule (e = MED)43.70.770.10
PACO-PO (Ours), BM2547.60.64-0.09
PACO-PO (Ours), hybrid BM25 + BGE46.10.660.01

Following your suggestion, we have added these results to Appendix A.9 as Table 14 (Lines 1666–1673).

(Continued)

Summarized W4: Factorized versus flat head (Question 3)

A4: We respectfully clarify that a parameter-matched flat head (199,203 against 199,543 trainable parameters) reaches 45.6%, 1.5 points below the full method and 0.2 points above the unmatched flat head, as shown in Table R3. By our computation for this response, the two heads, which Eq. 4 makes equally expressive, also differ in sample sharing, as under a uniform policy with G = 6 actions per query each effort level is selected 2 times per group in expectation against 0.5 times for each of the 12 flat cells. Table R3 also gives the seed spread of the ablation rows and the accuracy after 10 and 20 of the 40 outer iterations.

Table R3: Macro accuracy (%) of the ablation rows of Table 3 over three seeds as mean ± standard deviation, where the parameter-matched flat head is printed without its spread, and after 10 and 20 of the 40 outer iterations.

ConfigurationMacro acc. (40 it.)After 10 it.After 20 it.
PACO-PO (Ours, full)47.1 ± 0.844.945.5
Flat 12-way head45.4 ± 0.4 (-1.7)43.043.8
Flat 12-way head, parameter-matched45.6 (-1.5)42.143.4
Without retrieval preview45.0 ± 1.541.943.6
Learned depth only43.0 ± 2.139.942.5
Learned effort only44.0 ± 1.041.743.3
lambda frozen44.9 ± 0.342.143.2

Following your suggestion, we have added these results to Appendix A.9 as Table 15 (Lines 1674–1684), described the parameter-matched flat head in Appendix A.4 (Lines 1176–1189), and revised the factorized-head paragraph in Section 5.3 (Lines 371–392) and Appendix A.5 (Lines 1301–1314).


Summarized W5: Sensitivity to the budget fraction (Question 4)

A5: We respectfully clarify that another budget fraction needs another training run with the same procedure, because the controller does not read B, and budget conditioning is the direction that removes this. Following your suggestion, we further evaluate PACO-PO on HotpotQA at the budget fractions 0.5 and 0.8 beside 0.65, with the cascade and the router read at their settings nearest its cost and the best mix of all 12 cells read at its cost. Table R4 shows that the accuracy of PACO-PO rises with B, at 47.1%, 47.6% and 48.1% for B = 0.5, 0.65 and 0.8, against 42.9%, 45.8% and 48.0% for the router and 41.3%, 45.1% and 47.7% for the cascade at their settings nearest its cost, and by our computation for this response each entry carries a binomial standard error of at most 1.58 points and a difference of two entries at most 3.16 points.

Table R4: HotpotQA accuracy (%) at three budget fractions, with the cascade and the router read at the setting whose probe cost lies nearest the cost of PACO-PO.

BudgetPACO-PO acc.PACO-PO costCascade at that costRouter at that costBest 12-cell mix at that cost
B = 0.547.10.5241.342.943.2
B = 0.6547.60.6545.145.844.9
B = 0.848.10.7947.748.046.5

Following your suggestion, we have added these results to Appendix A.9 as Table 16 (Lines 1690–1698).


Summarized W6: Recalibration on a new backbone (Question 5)

A6: We respectfully clarify that refitting the price alone cannot change deployed spend, because the price λ is not used at inference, and the recalibration acts on the deployed decision instead. Without retraining, the practitioner re-runs the probe on the new backbone, predicts the cost of each action from the passage lengths and the probe's token counts, and deploys argmax_a log pi(a | q) - mu c_hat(q, a)/B with a scalar mu set on the probe split such that the mean predicted cost is at or below B where that is feasible. Table R5 shows that the recalibrated Qwen2.5-7B controller has a cost fraction of 0.67, at or below the 0.69 of the unrecalibrated swap of Table 6, at an accuracy of 45.8% against 46.7%, and the realized cost need not meet B. Table R5 also gives the further datasets.

Table R5: Accuracy (%) and cost fraction beyond the three benchmarks, with PACO-PO and router accuracy as three-seed means and the recalibrated Qwen2.5-7B controller as a one-seed macro.

SettingBest fixed acc.PACO-PO acc.PACO-PO costRouter acc.Router cost
2WikiMultiHopQA45.745.50.6643.00.73
MuSiQue38.337.90.6537.20.72
PopQA44.643.30.6539.00.68
Mixed stream (three benchmarks)47.746.20.6644.50.71
MATH-500 without self-matches52.051.10.6850.20.79
Qwen2.5-7B, recalibrated47.745.80.67(n/a)(n/a)

Following your suggestion, we have added these results to Appendix A.9 (Lines 1703–1714), and revised the cross-backbone paragraph in Section 5.3 (Lines 437–449) and Appendix A.5 (Lines 1348–1354).

(Continued)

References

[1] Lihong Li et al. A contextual-bandit approach to personalized news article recommendation. WWW, 2010.


Best Wishes,

The Authors of Paper 52793

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Line references refer to the clean copy, PDF pages 1–33.

Summary

Single-turn retrieval-augmented question answering charges the generated reasoning and the retrieved passages to one token budget, yet existing per-query controllers tune reasoning effort and retrieval depth apart. PACO-PO learns a factorized effort-depth policy that reads a zero-cost retrieval preview and is priced by a single dual variable. On HotpotQA, Natural Questions and MATH-500, it comes within one point of the best fixed configuration (47.1% versus 47.7% macro accuracy) at 66% of the token cost of that configuration. The revision adds three benchmarks and a mixed stream, three-seed results, the deployed budget compliance, the full 12-cell static mixture, and a token accounting with a modeled latency.

Strengths

  • Important problem. Reasoning effort and retrieval depth share one token budget yet are tuned apart (MEmN, zDAY, a5Rx).
  • Practical design. The deployer sets a budget instead of an exchange-rate weight, and each component is simple, clearly motivated and cheap (a5Rx).
  • Evidence and ablations. Meaningful evidence of effectiveness in several settings (MEmN); joint control improves over single-axis alternatives (zDAY); ablations change one thing per row at the same budget (a5Rx).

Weaknesses and Revisions

  1. Benchmarks beyond the three (MEmN, a5Rx). Added 2WikiMultiHopQA, MuSiQue, PopQA, a mixed stream served by one controller, and MATH-500 without self-matches in retrieval in Table 17 (Lines 1703–1714).
  2. Seeds and statistical uncertainty (MEmN, zDAY, a5Rx). Reported three-seed means with standard deviations for the main comparison and the ablations in Tables 13 and 15 (Lines 1649–1658 and 1674–1684).
  3. Probe cost, entropy rule and retrieval preview (MEmN, zDAY). Clarified that the compute footprint counts the probe (Appendix A.6, Lines 1483–1497), and added an entropy-threshold rule and the BM25 and hybrid retrievers on HotpotQA in Table 14 (Lines 1666–1673).
  4. Factorized head (MEmN, zDAY). Added a parameter-matched flat head and the accuracy after 10 and 20 outer iterations in Table 15 (Lines 1674–1684).
  5. Budget fractions (MEmN, a5Rx). Evaluated B = 0.5 and 0.8 beside 0.65 on HotpotQA in Table 16 (Lines 1690–1698).
  6. Backbone recalibration (MEmN, a5Rx). Added a recalibrated Qwen2.5-7B controller without retraining in Table 17 (Lines 1703–1714).
  7. Deployed budget compliance and 12-cell mixture (zDAY). Reported the deployed deviation from B for PACO-PO and the three adaptive controllers in Table 18 (Lines 1720–1727), and all 12 fixed cells with their best mix in Tables 19 and 16 (Lines 1728–1736 and 1690–1698).
  8. Dual pricing versus scalarization (zDAY). Added the scalarization weights 0.1 and 2.0 in Table 20 (Lines 1739–1744).
  9. Contribution and positioning (a5Rx). Clarified that the contribution is the problem and its formulation (Response A1 to a5Rx), and positioned the work against Yue et al. in Appendix A.1 (Lines 884–901).
  10. Serving latency (a5Rx). Counted input and output tokens per query and modeled a token-processing latency in Table 21 (Lines 1747–1753).

In the marked copy (PDF pages 34–67), the revisions made in response to the reviewers are tagged with the reviewer IDs. The reviews of zDAY and a5Rx quote the reviewed version (46.9% at a cost fraction of 0.64); the revised version reports 47.1% at 0.66.

All experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52793

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.