acceptodds
Under review as a conference paper at ICLR 2027

PIVOT: Risk-Controlled Routing of Verifier Feedback for Reliable Policy Improvement

Abstract

Post-training relies on verifiers with different reliability, coverage, and cost. Existing routers principally estimate label accuracy, although learning depends on the update induced by feedback. We introduce PIVOT, a sequential training-time router that treats verifier feedback as a policy intervention. It predicts reliability adjusted directional alignment before acquisition, maps heterogeneous outputs to preference graphs, and gates the resulting optimizer-native update. Across Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct in four domains under a common cost cap, PIVOT reaches 68.4 trusted utility. It improves on common-pool BayesianRouter-H by 1.6 points and uncertainty routing by 2.3 points, with worst slice utility of 61.1. At the pre-specified α = 0.05 budget, held-out gold-graph inconsistency (GG-risk) is 0.047 at 82% coverage. Setup cost amortizes after 37.6K prompts relative to BayesianRouter-H and 67.6K relative to uncertainty routing. Fully stale reuse after policy shift raises GG-risk from 0.048 to 0.082, delimiting the phase-wise guarantee.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Post-Training, Verifier Routing, Heterogeneous Feedback, Preference Learning, Reward Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_54ctOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PIVOT selects verifier feedback during policy training using predicted reliability, update alignment, and acquisition cost. It converts heterogeneous feedback into preference graphs, then accepts, escalates, or skips each group. A phase-wise certificate controls gold-graph inconsistency among admitted feedback, subject to a coverage requirement. Experiments on two policy models and four domains report a 1.6-point average utility improvement over BayesianRouter-H using the same verifier pool, with GG-risk of 0.047 at 82% coverage.

Critical strengths
  1. The distinction between correct feedback and useful policy updates is well motivated. Using update alignment to guide verifier selection addresses this distinction directly. The separation between acquisition and admission also makes the roles of cost and reliability reasonably clear.
  2. The experiments include several useful controls. The common-pool comparison reduces differences in verifier access, while Table S16 shows that routing remains beneficial when loss weighting is held fixed. Results across two models, four domains, five seeds per model, and an additional DPO setting provide encouraging support for the approach.
Critical weaknesses
  1. Some reported diagnostics need clarification. Table S25 defines edge density relative to K(K−1)K(K-1), but reports values above the 0.5 limit for acyclic graphs. Table S39’s GPU-price sensitivity also does not match the resource accounting in Tables S7–S8: halving the GPU rate gives approximately 0.316 per prompt, rather than 0.498. The reference cost itself is reproducible. These may be reporting errors, but the authors should establish whether they affect the implementation or conclusions.
  2. The practical benefit depends on amortizing repeated setup costs. The paper accounts for these costs and reports break-even points, but the experiments use four frozen-policy phases. It remains unclear whether the advantage holds when policies are updated more frequently and fewer examples are processed between certification rounds. This limits how broadly the cost results can be interpreted.
Questions for authors
  1. What normalization was used for Table S25’s edge densities, and does correcting this discrepancy affect admission decisions or GG-risk?
  2. Can the authors reconcile Table S39 with Tables S7–S8 and clarify whether the corrected calculations change the break-even estimates?
  3. How would more frequent policy updates affect setup amortization and the cost advantage reported in the four-phase experiments?
6 comments

Response to Reviewer 54ct

We thank the reviewer for the numerical checks and the question about repeated setup costs.

W1 and Q1 Graph density normalization

The density columns in Tables S25 and S26 report unordered-pair coverage. The S25 caption incorrectly names the directed-edge denominator. For a graph with KK responses and preference-edge set EE, let cpairc_{\mathrm{pair}} be unordered-pair coverage and ddird_{\mathrm{dir}} directed-edge density. With ∣E∣|E| denoting the edge count, the two normalizations are

\qquad d_{\mathrm{dir}}=\frac{\lvert E\rvert}{K(K-1)}. \tag{R17}$$ Thus the two normalizations have DAG upper bounds of 1 and 0.5, respectively, and use denominators 28 and 56 at $K=8$. The captions should name unordered-pair coverage for the displayed values. For audited group $i$, let $A_i\in\\{0,1\\}$ indicate final admission, $C_i\in\\{0,1\\}$ strict gold-graph consistency, and $\rho_i$ its prespecified consequence weight. Empirical GG-risk $\widehat R_{\mathrm{GG}}$ is $$\widehat R_{\mathrm{GG}} =\frac{\sum_i\rho_iA_i(1-C_i)}{\sum_i\rho_iA_i}. \tag{R18}$$ In the local experiment, the consistency model uses $d_{\mathrm{dir}}$ before isotonic probability calibration, while the displayed diagnostic is $c_{\mathrm{pair}}$. Their factor-of-two relation resolves the apparent violation of the DAG limit without changing the graph's edges. Correcting the caption leaves this calibrated input unchanged, and the GG-risk calculated from the same recorded admissions and strict-consistency labels in Eq. (R18) is unchanged.

W1 and Q2 Resource accounting and price sensitivity

Original rate-card reconciliation. We hold S8 resource usage and logged call overhead fixed and apply the S7 prices of USD 3/15 per million input/output tokens, USD 10 per device-hour and USD 0.006 per logged call. For method mm, let TmT_m denote the USD total recomputed from these resources. Tables R21 and R22 give the reference totals and one-factor price changes under this rate card.

Table R21. Original-study resource use and reference-cost reconciliation.

MethodInput MOutput MGPU hCallsTmT_mS37 totalResidual
PIVOT1,9602405,228800,60066,563.6066,560−3.60-3.60
BayesianRouter-H2,1502705,520786,50070,419.0070,400−19.00-19.00
Uncertainty Router2,5803406,035175,40074,242.4074,242−0.40-0.40

Notes. Input/output M denotes millions of tokens; GPU h denotes cumulative device-hours. Calls include setup and online calls. Residual equals the displayed S37 total minus TmT_m.

Table R22. Original-rate price sensitivity at the 128,000-prompt horizon.

Rate multiplierPIVOTBayesianRouter-HUncertainty Router
Reference0.520030.550150.58002
Hosted ×0.5\times0.50.483000.509130.52986
Hosted ×1.5\times1.50.557060.591160.63018
GPU ×0.5\times0.50.315810.334520.34428
GPU ×1.5\times1.50.724250.765770.81576

Notes. Entries are USD per prompt with fixed resource usage and call overhead. Hosted multipliers change both input- and output-token prices; GPU multipliers change the device-hour rate. Values are rounded to five decimals.

Separate new local-GPU cadence study. The current resource ledger uses local inference, with zero hosted-token charges. GPU time is cumulative device-hours, charged at USD 3.99 per device-hour.

For method mm at a fixed number FF of phases per policy, let hmh_m be GPU-hours per run, bmb_m CPU fitting cost in USD, and aa the multiplier of the reference GPU price. The full cost Cm(a)C_m(a) in USD per 32,768-candidate run is

\tag{R19}$$ **Table R23.** Local-GPU full-cost sensitivity at a fixed candidate horizon. | $F$ | Method | GPU-hours | CPU USD | GPU $\times0.5$ | Reference USD | GPU $\times1.5$ | | --- | --- | --- | --- | --- | --- | --- | | 4 | H-proxy | 36.383409 | 0.64 | 73.22 | 145.81 | 218.39 | | 4 | UQ-proxy | 30.597384 | 0.16 | 61.20 | 122.24 | 183.29 | | 4 | PIVOT | 431.452638 | 0.36 | 861.11 | 1,721.86 | 2,582.60 | | 8 | H-proxy | 56.209105 | 1.28 | 113.42 | 225.55 | 337.69 | | 8 | UQ-proxy | 34.864260 | 0.32 | 69.87 | 139.43 | 208.98 | | 8 | PIVOT | 809.654675 | 0.72 | 1,615.98 | 3,231.24 | 4,846.50 | | 16 | H-proxy | 95.671751 | 2.56 | 193.43 | 384.29 | 575.16 | | 16 | UQ-proxy | 42.550299 | 0.64 | 85.53 | 170.42 | 255.30 | | 16 | PIVOT | 1,562.199552 | 1.44 | 3,118.03 | 6,234.62 | 9,351.20 | *Notes.* $F$ is the number of phases per policy. Costs are USD per 32,768-candidate run, averaged over five seeds. The reference GPU rate is USD 3.99 per cumulative device-hour, with the listed CPU cost added. Values are repriced from unrounded components and rounded to two decimals; hosted-token charges are zero. Hosted-token price multipliers leave this local ledger unchanged. Shared CPU/storage cost is added once at study level. The fixed-horizon full costs and original mixed hosted/local incremental costs have different resource scopes. For a fixed renewal schedule, let $H$ be the number of candidate prompts, $S_m$ the setup cost in USD, and $c_m$ the online cost per prompt for method $m$. Total cost is $B_m(H)$. The break-even horizon $H^*_{P,b}$ compares PIVOT (index $P$) with baseline $b$, $$B_m(H)=S_m+c_mH,\qquad H^*_{P,b}=\frac{S_P-S_b}{c_b-c_P},\quad c_b > c_P. \tag{R20}$$ With resource usage fixed, Table R21 reproduces the reference totals at S37's displayed precision, and Table R22 gives 0.31581 USD/prompt for PIVOT at half GPU price, correcting S39's 0.498. This corrects the counterfactual cost calculation; at the unchanged reference prices, the S37 setup and online profiles retain break-even horizons of 37,647.06 prompts against BayesianRouter-H and 67,610.06 against Uncertainty Router. A different GPU price changes both setup and online charges, so its break-even must use the repriced $S_m$ and $c_m$ in Eq. (R20).

W2 and Q3 Policy-update frequency and setup amortization

The added cadence study fixes the total candidate horizon at H0=32,768H_0=32{,}768 groups and varies the number of phases per policy over F∈4,8,16F\in\\{4,8,16\\}. The policies are Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, trained with rank-16 LoRA, GRPO, eight responses per task item (one per prompt formulation), and AdamW at 5×10−65\times10^{-6}. A phase collects feedback at a frozen policy/optimizer state, then optimizes the buffered losses.

Table R24. Phase-renewal configurations at a fixed candidate budget.

ConfigurationF=4F=4F=8F=8F=16F=16
Candidate groups per model-phase4,0962,0481,024
Total candidate groups per run32,76832,76832,768
Candidate responses per run262,144262,144262,144
Seeds per configuration555
Route fit per model-phase384384384
Route calibration per model-phase128128128
Audit groups per domain per model-phase128128128
Certificate groups per model-phase8,1928,1928,192

Notes. FF counts phases per policy, so each run has 2F2F model-phases across both policies. A candidate group contains one complete response for each of eight prompt formulations of the same item. Route-fit, calibration, audit and certificate groups are separate from the update candidates.

In this configuration, qq is calibrated strict-consistency probability and r=[2q−1]+r=[2q-1]_+ its reliability margin; [,⋅,]+[\\,\cdot\\,]_+ truncates negative values to zero. H-proxy uses Gaussian-perturbed competence scores; UQ-proxy uses acquired-RM uncertainty 1−q1-q, escalating the highest 12%. These score proxies are distinct from the fitted matched-information BayesianRouter-H and Uncertainty-H learners.

For queried group ii, let rir_i be its reliability margin and τi\tau_i its directional alignment with the trusted audit gradient. With QQ the number of queried groups and AiA_i their admission indicators, the mean feedback score is zˉ=Q−1∑iAiriτi\bar z=Q^{-1}\sum_i A_ir_i\tau_i.

Table R25. Routing, admission and full cost across phase-renewal schedules.

FFMethodQueriesAdmittedFeedback scoreGG-riskFull cost USD/run
4H-proxy32,76832,369.80.16880.03181145.81
4UQ-proxy32,76832,700.40.24840.00800122.24
4PIVOT32,76832,504.40.31260.002681,721.86
8H-proxy32,76832,382.40.16880.03168225.55
8UQ-proxy32,76832,707.40.24830.00779139.43
8PIVOT32,76832,491.80.31340.002573,231.24
16H-proxy32,76832,361.60.16920.03221384.29
16UQ-proxy32,76832,705.20.24900.00813170.42
16PIVOT32,76832,515.00.31310.002416,234.62

Notes. Values are means across five seeds. Queries and admissions count groups; fractional admission counts reflect averaging. Feedback score averages admission-weighted rτr\tau over all queried groups. Full cost is USD per run, including generation, optimization, setup, certification, acquired feedback, audit/virtual steps, fitting, evaluation and runtime.

Reviewer_7jm2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PIVOT, which chooses which grader to use for each training example by predicting whether its feedback will help the model learn. It then decides whether to use that feedback, ask a stronger grader, or skip the update. Across two models and four task domains, it reports better task performance than competing grading methods at lower grading cost.

Critical strengths
  • Choosing a grader based on whether its feedback helps training makes sense. The paper shows that its score predicts improvement better than grader accuracy alone, and the reported performance gains hold across all 10 training seeds.

  • The comparisons account for the cost of setting up and running the method, including extra grading calls. The paper also tests when the method stops working, such as when the model changes but the grader-selection rules are not updated.

Critical weaknesses
  • The paper does not show how well the models performed before training. It reports science accuracy of 61–65%, compared with public scores of roughly 30–36% for these models, which would be a large improvement. Report their starting scores under the same evaluation procedure and explain how the results compare with the public figures.
Questions for authors

See weakness

5 comments

Response to Reviewer 7jm2

We thank the reviewer for requesting the initial-policy scores and a clearer evaluation denominator.

Weakness and question Initial policy performance

We report initial and final accuracy on the complete benchmarks and fixed gold-eligible subsets. Full-benchmark accuracy is the primary endpoint; the fixed subsets provide diagnostic comparisons.

For each benchmark item, the Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct checkpoints each generate eight complete responses using eight distinct prompt formulations. These prompts follow the instructions specified by the public dataset or the prompts used in published evaluations of that dataset, together with variants of those prompts. Each prompt elicits one complete response, which is scored using the domain-specific evaluation rules. We construct each policy's preference graph from its eight scored responses. The initialization reference graphs determine the eligibility masks, which are fixed before terminal scoring and shared across methods.

Initialization uses meta-llama/Llama-3.1-8B-Instruct and Qwen/Qwen2.5-7B-Instruct. Responses are generated at temperature 0.8 and top-p 0.95; the first complete response supplies the cached initial score. The terminal comparison uses BayesianRouter-H and PIVOT, with seeds 17, 29, 43, 61 and 89 and the prescribed phase-four endpoint. Methods share item order, policy-specific prompt/chat formatting, decoding, extraction and scoring rules; terminal labels do not select checkpoints or thresholds.

For item ii, let Ei⋆E_i^\star be its gold-supported edge set and Gi=1[Ei⋆≠∅]G_i=\mathbf1[E_i^\star\ne\varnothing] its eligibility indicator; 1[⋅]\mathbf1[\cdot] is a binary indicator. Gold eligibility is distinct from structural adapter pass, verifier-specific consistency Ci,vC_{i,v} and online admission AiA_i. Let NN be the full benchmark size, nGn_G the gold-nonempty count, cc the fully correct responses among an item's eight responses, and nn the size of an evaluated population. Code gold compares hidden-test pass fractions; IFEval uses componentwise constraint dominance. Partial-only groups have c=0c=0 with finer-grained comparison edges.

Table R17. Initial-policy first-response accuracy by evaluation population.

Initial policyBenchmarkFull set correct/nnGold-nonempty subset, correct/nnExcluded subset correct/nn
Llama-3.1-8B-InstructMATH-500246/500 (49.20%)167/277 (60.29%)79/223 (35.43%)
Llama-3.1-8B-InstructLiveCodeBench release-v465/713 (9.12%)62/108 (57.41%)3/605 (0.50%)
Llama-3.1-8B-InstructIFEval416/541 (76.89%)157/275 (57.09%)259/266 (97.37%)
Llama-3.1-8B-InstructGPQA Diamond55/198 (27.78%)54/105 (51.43%)1/93 (1.08%)
Qwen2.5-7B-InstructMATH-500379/500 (75.80%)161/275 (58.55%)218/225 (96.89%)
Qwen2.5-7B-InstructLiveCodeBench release-v4190/713 (26.65%)180/329 (54.71%)10/384 (2.60%)
Qwen2.5-7B-InstructIFEval356/541 (65.80%)175/293 (59.73%)181/248 (72.98%)
Qwen2.5-7B-InstructGPQA Diamond73/198 (36.87%)62/105 (59.05%)11/93 (11.83%)

Notes. Cells report correct/nn (accuracy) for the first generated response on the full set and its eligible/excluded partition. Gold-nonempty means Ei⋆≠∅E_i^\star\ne\varnothing, separately from verifier-specific consistency and final admission.

Table R18. Gold-graph eligibility of eight-response benchmark groups.

Policy / domainFull NNc=0c=0c=8c=8Mixed, 1≤c≤71\le c\le7Partial-only within c=0c=0Gold-nonempty nGn_GnG/Nn_G/N
Llama / MATH50014479277027755.40%
Llama / Code7136023108010815.15%
Llama / IFEval5417259275027550.83%
Llama / GPQA198921105010553.03%
Qwen / MATH5007218275027555.00%
Qwen / Code71337410329032946.14%
Qwen / IFEval54167181293029354.16%
Qwen / GPQA1988211105010553.03%

Notes. cc counts fully correct responses. The c=0c=0, c=8c=8 and mixed columns partition the full set; partial-only is a subset of c=0c=0. Gold-nonempty nGn_G equals mixed plus partial-only. Llama and Qwen denote the two Instruct policies named above.

Fixed-population before-and-after comparison. For policy pp, the initialization reference group freezes Sp=i:Ei,p⋆≠∅S_p=\\{i:E^\star_{i,p}\ne\varnothing\\}. All methods and seeds reuse this list. Every full GPQA evaluation retains 198 items; each policy's eligible list contains 105 items, or 53.03% of the benchmark. The two lists have different item indices. Initial answers are cached once, while final accuracies use the five seeds' integer correct counts on the same full set and fixed mask.

Let Q∈full,eligibleQ\in\\{\mathrm{full},\mathrm{eligible}\\} index the population, with np,full=Nn_{p,\mathrm{full}}=N and np,eligible=∣Sp∣n_{p,\mathrm{eligible}}=|S_p|. Write kp,Q(0)k_{p,Q}^{(0)} for the cached initial correct count. For method m∈B,Pm\in\\{\mathrm{B},\mathrm{P}\\}, denoting BayesianRouter-H and PIVOT, let kp,s,Q(m)k_{p,s,Q}^{(m)} be the terminal correct count for seed ss in S=17,29,43,61,89\mathcal S=\\{17,29,43,61,89\\}. Initial accuracy ap,Q(0)a_{p,Q}^{(0)} and five-seed terminal mean aˉp,Q(m)\bar a_{p,Q}^{(m)}, in percent, are

\bar a_{p,Q}^{(m)}=\frac{100}{5n_{p,Q}}\sum_{s\in\mathcal S}k_{p,s,Q}^{(m)}. \tag{R15}$$ **Table R19.** GPQA initial and final accuracy on full and fixed eligible populations. **(a) Initial scores and five-seed final accuracy (%)** | Policy | Population | Initial | BayesianRouter-H final | PIVOT final | | --- | --- | --- | --- | --- | | Llama8B | Full (198) | 27.78 | $31.92\pm3.74$ | $32.93\pm4.38$ | | Llama8B | Eligible (105) | 51.43 | $59.24\pm7.04$ | $61.14\pm8.26$ | | Qwen7B | Full (198) | 36.87 | $37.78\pm2.03$ | $39.70\pm1.66$ | | Qwen7B | Eligible (105) | 59.05 | $61.52\pm4.02$ | $64.95\pm3.89$ | **(b) Final correct answers by seed, full/eligible** | Policy | Seed | BayesianRouter-H | PIVOT | | --- | --- | --- | --- | | Llama8B | 17 | 59/58 | 60/59 | | Llama8B | 29 | 58/57 | 58/57 | | Llama8B | 43 | 76/75 | 80/79 | | Llama8B | 61 | 60/59 | 64/63 | | Llama8B | 89 | 63/62 | 64/63 | | Llama8B | Total | 316/311 | 326/321 | | Qwen7B | 17 | 79/68 | 78/67 | | Qwen7B | 29 | 77/67 | 80/70 | | Qwen7B | 43 | 71/60 | 78/67 | | Qwen7B | 61 | 77/68 | 83/74 | | Qwen7B | 89 | 70/60 | 74/63 | | Qwen7B | Total | 374/323 | 393/341 | *Notes.* Initial scores use the cached first generated response for each item; final entries are mean $\pm$ sample SD across seeds 17, 29, 43, 61 and 89. Each full-set seed scores 198 items; both policy-specific eligible lists contain 105 items. Panel (b) gives full/eligible correct counts, with no terminal-answer reselection. Pooling the five seeds gives PIVOT GPQA counts of 326/990 and 321/525 for Llama (full/eligible), and 393/990 and 341/525 for Qwen. The full-set accuracies are 32.93% and 39.70%, and the eligible accuracies are 61.14% and 64.95% (Table R19). Initial counts are 55/198 and 54/105 for Llama, and 73/198 and 62/105 for Qwen. Each comparison retains its own fixed population.

Full and conditional macro utility. Let dd index math, code, instruction and science, and aˉp,d,Q(m)\bar a_{p,d,Q}^{(m)} be the mean score in that model--domain cell. Write Um,QU_{m,Q} for the equal-weight eight-cell macro utility and Wm,QW_{m,Q} for the lowest cell mean; for initialization, each cell mean equals its cached score. The PIVOT-minus-BayesianRouter-H contrast is ΔP,B,Q\Delta_{\mathrm{P},\mathrm{B},Q}:

U_{m,Q}&=\frac18\sum_{p=1}^{2}\sum_{d=1}^{4}\bar a_{p,d,Q}^{(m)},\qquad W_{m,Q}=\min_{p,d}\bar a_{p,d,Q}^{(m)},\\\\ \Delta_{\mathrm{P},\mathrm{B},Q} &=U_{\mathrm{P},Q}-U_{\mathrm{B},Q}. \end{aligned} \tag{R16}$$ **Table R20.** Full and eligible final accuracy for BayesianRouter-H and PIVOT. **(a) Eight model--domain cells, full/eligible accuracy (%)** | Policy | Domain | Initial | BayesianRouter-H final | PIVOT final | | --- | --- | --- | --- | --- | | Llama8B | Math | 49.20/60.29 | 54.80/70.83 | 55.56/72.20 | | Llama8B | Code | 9.12/57.41 | 10.04/63.70 | 10.24/65.00 | | Llama8B | Instruction | 76.89/57.09 | 83.84/71.20 | 84.36/72.22 | | Llama8B | Science | 27.78/51.43 | 31.92/59.24 | 32.93/61.14 | | Qwen7B | Math | 75.80/58.55 | 82.80/71.56 | 83.60/73.02 | | Qwen7B | Code | 26.65/54.71 | 31.19/64.68 | 31.81/66.02 | | Qwen7B | Instruction | 65.80/59.73 | 72.05/71.60 | 72.83/72.97 | | Qwen7B | Science | 36.87/59.05 | 37.78/61.52 | 39.70/64.95 | **(b) Equal-weight macro utility and worst-cell scores (points)** | Population | Initial | BayesianRouter-H | PIVOT | $\Delta_{\mathrm{P},\mathrm{B},Q}$ | $W_{\mathrm{B},Q}/W_{\mathrm{P},Q}$ | | --- | --- | --- | --- | --- | --- | | Full | 46.01 | $50.55\pm0.74$ | $51.38\pm0.64$ | 0.82 | 10.04/10.24 | | Eligible | 57.28 | $66.79\pm1.45$ | $68.44\pm1.26$ | 1.65 | 59.24/61.14 | *Notes.* Macro means, SDs and method contrasts use unrounded seed-level scores, separately for full and eligible populations. $W$ is the minimum of the eight cell means. Full and eligible results are computed from the same terminal scoring vectors; the information-controlled routing comparison uses its own declared conditions. The before-and-after comparisons use the same scored items within each population. PIVOT's full-set GPQA accuracy increases from 27.78% to 32.93% for Llama and from 36.87% to 39.70% for Qwen; the corresponding eligible scores increase from 51.43% to 61.14% and from 59.05% to 64.95% (Table R19). Table R20 gives full macro utility of 51.38 for PIVOT and 50.55 for BayesianRouter-H, and eligible utility of 68.44 and 66.79. Method differences computed from unrounded scores are 0.82 and 1.65 points, respectively. The full-set endpoints provide the appropriate population scope for comparison with public full-benchmark figures. The 61--65% science scores describe the fixed 105-item eligible lists rather than the complete 198-item benchmark.
Reviewer_ZcXfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

this paper study which verifier to query, and whether to admit its output, while training a policy with GRPO or DPO. It treats each verifier output as an intervention on the policy update not only as a label that is correct or wrong. PIVOT has 4 parts consisting of determinsistic adapter, pre-query router , post-query check. The alignment tau is the cosine between the optimizer-native virtual LoRA step that a verifier graph induces and the gradient of a gold-preference objective computed on a separate audit fold. The experiments + authors claim 3 contributions: 1) a policy-conditional verifier score in LoRA space, 2) a common graph interface for heterogeneous feedback, and 3) a certified sequential acquisition policy.

Critical strengths

I think the question is good and the design is careful. Training-time verifier choice should depend on the current policy, and the paper keeps it separate from inference-time routing. The paper is open about the scope of its guarantee. It says that GG-risk is not downstream harm (L225-228). Proposition 1 / S1 is correct as stated.

Critical weaknesses

It's mostly about claim soundness.

  1. Adapter rule doesn’t fit its numbers. according to the paper with 8 responses per prompt and yes/no exact answers, the stated rule lets at most 39% of math groups through, but the paper reports 91.9%. 4 reported graph densities are also above 0.5, which no valid graph can reach.
  2. Data access is not fair: the whole gain depends on a signal built from gold labeled audit data that no baseline gets. Without that signal, PIVOT scores 66.5, below BayesianRouter-H’s 66.8.
  3. In Fig 1A, the incomplete but useful response should beat the lucky guess. Under Eq. 2, that edge contradicts a supported gold edge, so a graph that contains it counts as a GG-risk failure. In Fig. 1B, the useful edge is unverified which is also a failure under the strict C. The paper says that it keeps strict consistency for admission (L46). But I could not find any analysis of how often admitted graphs contain edges between outcome-tied responses, or whether those graphs help.
Questions for authors

How did you compute Table S20(a)? Is each row one phase or a pool of phases? For each of the 8 phases, please give n, the weighted sums for N and D, and the raw admits on the certificate fold and on the held-out fold. Please also define the held-out fold, which Table S4 does not list.

13 comments

Response to Reviewer ZcXf

We thank the reviewer for the comments on group feedback, audit information access, strict graph consistency and certificate reporting.

W1 Group feedback and graph density

Training uses the complete graph over eight full responses to distinct prompts for one task, retaining multiple maxima. Structural validity requires a nonempty acyclic graph with no vetoed maximal response. GRPO uses graph-derived group advantages; DPO uses directed preference edges.

For a group of K=8K=8 responses, let EE be its preference-edge set and cc the number of correct responses. With binary outcomes, EE contains c(8−c)c(8-c) edges from correct to incorrect responses and is nonempty for 1≤c≤71\le c\le7. For an independent Bernoulli model with per-response success probability pp,

\qquad \max_p\Pr(E\ne\varnothing\mid p)=0.9921875. \tag{R1}$$ The unique-winner rule instead requires exactly one correct response, $$\Pr(c=1\mid p)=8p(1-p)^7, \qquad \max_p8p(1-p)^7=(7/8)^7\approx0.392696. \tag{R2}$$ The group rule therefore also accepts multiple correct responses, whereas all-correct and all-incorrect groups have no binary preference edge and abstain. **Table R1.** Unique and multiple maxima in binary gold-reference graphs. | Policy / benchmark | Empty | Unique maximum | Multiple maxima | Eligible groups | | --- | --- | --- | --- | --- | | Llama-3.1-8B / MATH | 223 | 0 | 277 | 277 | | Llama-3.1-8B / GPQA | 93 | 1 | 104 | 105 | | Qwen2.5-7B / MATH | 225 | 2 | 273 | 275 | | Qwen2.5-7B / GPQA | 93 | 1 | 104 | 105 | *Notes.* Counts refer to eight-response gold-reference groups. Eligible groups include unique and multiple maxima; empty graphs abstain. These counts are separate from verifier-specific adapter pass. MATH uses MATH-500, and GPQA uses GPQA Diamond. Table R1 shows that most retained MATH reference groups have multiple maxima. These benchmark-reference counts describe a different population from the source-adapter groups underlying the reviewer's 91.9% rate. Retaining multiple maxima gives the group rule in Eq. (R1), so the unique-winner ceiling in Eq. (R2) does not apply to that rate. The four values above 0.5 are 0.61 for exact answer/tests and 0.54 for the scalar reward model in Table S25, and 0.58 for math and 0.55 for code in Table S26. These entries report unordered-pair coverage, whereas the S25 caption incorrectly specifies a directed-edge denominator. Let $c_{\mathrm{pair}}$ denote the fraction of unordered response pairs with an oriented edge and $d_{\mathrm{dir}}$ the fraction of the $K(K-1)$ possible directed edges. With $|E|$ the edge count, $$c_{\mathrm{pair}}=\frac{\lvert E\rvert}{\binom K2} =2\\,\frac{\lvert E\rvert}{K(K-1)} =2d_{\mathrm{dir}}. \tag{R3}$$ At $K=8$, the two denominators are 28 and 56. The four entries correspond to directed densities 0.305, 0.270, 0.290 and 0.275, all below the directed acyclic graph (DAG) limit of 0.5. Their reported statistic is unordered-pair coverage, with upper bound 1.

W2 Audit information access

We compare six routing learners using a common verifier pool and matched access to gold-audit information.

For response group ii and verifier vv, let di,vd_{i,v} be the optimizer-native virtual policy update induced by that verifier's feedback. Let gaudg^{\mathrm{aud}} be the trusted-objective gradient from the disjoint gold-audit fold and SS the fixed 256-dimensional Johnson--Lindenstrauss (JL) projection applied to both vectors in rank-16 trainable LoRA space. Their cosine similarity is directional alignment τi,v\tau_{i,v}.

The calibrated strict-consistency probability qi,vq_{i,v} determines the reliability margin r(qi,v)r(q_{i,v}). The +D+D condition supervises the router with the reliability-adjusted target zi,v=r(qi,v)τi,vz_{i,v}=r(q_{i,v})\tau_{i,v}.

The comparison has three information conditions. R0 uses 3,072 route-fit groups with graph-consistency supervision. +A adds 512 audit groups with their gold graphs and consistency labels. +D uses the same 3,584 groups and supplies gold, consistency, alignment and update summaries, with routing target z=rτz=r\tau. Thus +A to +D holds the gold-group count fixed and changes the policy-conditional target.

All arms share the post-query consistency model, audit direction, projection, gate, loss weight and single-repair rule. Information levels vary pre-query training exposure. The reliability margin uses an inconsistency penalty of 1.5; [,⋅,]+[\\,\cdot\\,]_+ truncates negative values to zero. The post-query quantities satisfy

r(q)&=[q-1.5(1-q)]_+,\\\\ \tau_{i,v}&=\cos(Sg^{\mathrm{aud}},Sd_{i,v}),\\\\ z_{i,v}&=r(q_{i,v})\tau_{i,v}. \end{aligned} \tag{R4}$$ Direction targets for route-fit groups use all 128 audit groups per domain. Audit fitting groups are split into four blocks of 32; each block uses the other 96 groups as its gradient reference. Four-fold out-of-fold consistency predictions construct the fitting targets before the final post-query head is refitted and independently calibrated. The shared gate requires structural validity, $q\ge0.75$, $r(q) > 0$ and $\tau\ge0.10$, with one repair query after initial failure. All learners use the same 0.05 acquisition-cost coefficient. The twenty configurations, five seeds, two policies and four phases give 800 certification points, each with one fixed candidate and 16,384 groups. Their joint failure allocation is 0.05 and concentration radius is 0.01779252. For methods $a$ and $b$, let $U_a$ and $U_b$ be their conditional macro utilities in score points and $\Delta_{a,b}=U_a-U_b$ their difference. Method labels $\mathrm{P}$ and $\mathrm{B}$ denote PIVOT and BayesianRouter-H. Table R2 compares them at each common information level.

Table R2. Utility across matched pre-query information conditions.

ConditionBayesianRouter-HPIVOTΔP,B\Delta_{\mathrm{P},\mathrm{B}} (points)
R065.2865.470.19
+A66.4166.840.43
+D67.6968.380.69

Notes. Values are five-seed conditional macro utilities from the matched-information comparison. Both learners use the same groups and target within each condition; differences in this table are calculated from the displayed means. The paired +D+D estimate in Table R4 uses unrounded seed-level differences.

Table R3. Focal results with matched directional-alignment targets.

LearnerUtilityGG-riskCoverageCost USDUSD/executed
PIVOT68.380.019840.775264,471.610.5318
BayesianRouter-H67.690.019290.708162,329.360.5595
VI-H67.840.019770.727963,380.180.5526

Notes. All learners use the same +D+D target z=rτz=r\tau. Utility is conditional macro utility; GG-risk is consequence-weighted. Coverage includes certificate skips. Cost is incremental USD per seed, including failed setup; USD/executed divides this cost by the number of executed candidate groups. Values are means over five seeds.

For a fixed comparator bb, let s∈1,…,5s\in\\{1,\ldots,5\\} index the paired seeds and UP,sU_{\mathrm{P},s} and Ub,sU_{b,s} their method-specific utilities. Define the seed-wise difference as Δs=UP,s−Ub,s\Delta_s=U_{\mathrm{P},s}-U_{b,s}, with sample mean Δˉ\bar\Delta and sample SD sΔs_\Delta. Its pointwise 95% interval is Δˉ±2.776445sΔ/5\bar\Delta\pm2.776445s_\Delta/\sqrt5, where 2.776445 is the Student-tt critical value with four degrees of freedom. These intervals summarize variation across the five paired seeds.

Table R4. Paired utility differences with matched directional-alignment targets.

Comparator bbΔP,b\Delta_{\mathrm{P},b}Paired SD95% lower95% upper
BayesianRouter-H0.68940.45550.12381.2549
VI-H0.54490.22260.26850.8213

Notes. ΔP,b\Delta_{\mathrm{P},b} and the SD of the five paired-seed differences are in utility points. Intervals are pointwise 95% Student-tt intervals with four degrees of freedom, unadjusted for multiple comparisons. Statistics are computed before display rounding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.