- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PIVOT selects verifier feedback during policy training using predicted reliability, update alignment, and acquisition cost. It converts heterogeneous feedback into preference graphs, then accepts, escalates, or skips each group. A phase-wise certificate controls gold-graph inconsistency among admitted feedback, subject to a coverage requirement. Experiments on two policy models and four domains report a 1.6-point average utility improvement over BayesianRouter-H using the same verifier pool, with GG-risk of 0.047 at 82% coverage.
Critical strengths
- The distinction between correct feedback and useful policy updates is well motivated. Using update alignment to guide verifier selection addresses this distinction directly. The separation between acquisition and admission also makes the roles of cost and reliability reasonably clear.
- The experiments include several useful controls. The common-pool comparison reduces differences in verifier access, while Table S16 shows that routing remains beneficial when loss weighting is held fixed. Results across two models, four domains, five seeds per model, and an additional DPO setting provide encouraging support for the approach.
Critical weaknesses
- Some reported diagnostics need clarification. Table S25 defines edge density relative to , but reports values above the 0.5 limit for acyclic graphs. Table S39’s GPU-price sensitivity also does not match the resource accounting in Tables S7–S8: halving the GPU rate gives approximately 0.316 per prompt, rather than 0.498. The reference cost itself is reproducible. These may be reporting errors, but the authors should establish whether they affect the implementation or conclusions.
- The practical benefit depends on amortizing repeated setup costs. The paper accounts for these costs and reports break-even points, but the experiments use four frozen-policy phases. It remains unclear whether the advantage holds when policies are updated more frequently and fewer examples are processed between certification rounds. This limits how broadly the cost results can be interpreted.
Questions for authors
- What normalization was used for Table S25’s edge densities, and does correcting this discrepancy affect admission decisions or GG-risk?
- Can the authors reconcile Table S39 with Tables S7–S8 and clarify whether the corrected calculations change the break-even estimates?
- How would more frequent policy updates affect setup amortization and the cost advantage reported in the four-phase experiments?
6 comments
W1 and Q2 Resource accounting and price sensitivity
Original rate-card reconciliation. We hold S8 resource usage and logged call overhead fixed and apply the S7 prices of USD 3/15 per million input/output tokens, USD 10 per device-hour and USD 0.006 per logged call. For method , let denote the USD total recomputed from these resources. Tables R21 and R22 give the reference totals and one-factor price changes under this rate card.
Table R21. Original-study resource use and reference-cost reconciliation.
| Method | Input M | Output M | GPU h | Calls | S37 total | Residual | |
|---|---|---|---|---|---|---|---|
| PIVOT | 1,960 | 240 | 5,228 | 800,600 | 66,563.60 | 66,560 | |
| BayesianRouter-H | 2,150 | 270 | 5,520 | 786,500 | 70,419.00 | 70,400 | |
| Uncertainty Router | 2,580 | 340 | 6,035 | 175,400 | 74,242.40 | 74,242 |
Notes. Input/output M denotes millions of tokens; GPU h denotes cumulative device-hours. Calls include setup and online calls. Residual equals the displayed S37 total minus .
Table R22. Original-rate price sensitivity at the 128,000-prompt horizon.
| Rate multiplier | PIVOT | BayesianRouter-H | Uncertainty Router |
|---|---|---|---|
| Reference | 0.52003 | 0.55015 | 0.58002 |
| Hosted | 0.48300 | 0.50913 | 0.52986 |
| Hosted | 0.55706 | 0.59116 | 0.63018 |
| GPU | 0.31581 | 0.33452 | 0.34428 |
| GPU | 0.72425 | 0.76577 | 0.81576 |
Notes. Entries are USD per prompt with fixed resource usage and call overhead. Hosted multipliers change both input- and output-token prices; GPU multipliers change the device-hour rate. Values are rounded to five decimals.
Separate new local-GPU cadence study. The current resource ledger uses local inference, with zero hosted-token charges. GPU time is cumulative device-hours, charged at USD 3.99 per device-hour.
For method at a fixed number of phases per policy, let be GPU-hours per run, CPU fitting cost in USD, and the multiplier of the reference GPU price. The full cost in USD per 32,768-candidate run is
\tag{R19}$$ **Table R23.** Local-GPU full-cost sensitivity at a fixed candidate horizon. | $F$ | Method | GPU-hours | CPU USD | GPU $\times0.5$ | Reference USD | GPU $\times1.5$ | | --- | --- | --- | --- | --- | --- | --- | | 4 | H-proxy | 36.383409 | 0.64 | 73.22 | 145.81 | 218.39 | | 4 | UQ-proxy | 30.597384 | 0.16 | 61.20 | 122.24 | 183.29 | | 4 | PIVOT | 431.452638 | 0.36 | 861.11 | 1,721.86 | 2,582.60 | | 8 | H-proxy | 56.209105 | 1.28 | 113.42 | 225.55 | 337.69 | | 8 | UQ-proxy | 34.864260 | 0.32 | 69.87 | 139.43 | 208.98 | | 8 | PIVOT | 809.654675 | 0.72 | 1,615.98 | 3,231.24 | 4,846.50 | | 16 | H-proxy | 95.671751 | 2.56 | 193.43 | 384.29 | 575.16 | | 16 | UQ-proxy | 42.550299 | 0.64 | 85.53 | 170.42 | 255.30 | | 16 | PIVOT | 1,562.199552 | 1.44 | 3,118.03 | 6,234.62 | 9,351.20 | *Notes.* $F$ is the number of phases per policy. Costs are USD per 32,768-candidate run, averaged over five seeds. The reference GPU rate is USD 3.99 per cumulative device-hour, with the listed CPU cost added. Values are repriced from unrounded components and rounded to two decimals; hosted-token charges are zero. Hosted-token price multipliers leave this local ledger unchanged. Shared CPU/storage cost is added once at study level. The fixed-horizon full costs and original mixed hosted/local incremental costs have different resource scopes. For a fixed renewal schedule, let $H$ be the number of candidate prompts, $S_m$ the setup cost in USD, and $c_m$ the online cost per prompt for method $m$. Total cost is $B_m(H)$. The break-even horizon $H^*_{P,b}$ compares PIVOT (index $P$) with baseline $b$, $$B_m(H)=S_m+c_mH,\qquad H^*_{P,b}=\frac{S_P-S_b}{c_b-c_P},\quad c_b > c_P. \tag{R20}$$ With resource usage fixed, Table R21 reproduces the reference totals at S37's displayed precision, and Table R22 gives 0.31581 USD/prompt for PIVOT at half GPU price, correcting S39's 0.498. This corrects the counterfactual cost calculation; at the unchanged reference prices, the S37 setup and online profiles retain break-even horizons of 37,647.06 prompts against BayesianRouter-H and 67,610.06 against Uncertainty Router. A different GPU price changes both setup and online charges, so its break-even must use the repriced $S_m$ and $c_m$ in Eq. (R20).W2 and Q3 Policy-update frequency and setup amortization
The added cadence study fixes the total candidate horizon at groups and varies the number of phases per policy over . The policies are Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, trained with rank-16 LoRA, GRPO, eight responses per task item (one per prompt formulation), and AdamW at . A phase collects feedback at a frozen policy/optimizer state, then optimizes the buffered losses.
Table R24. Phase-renewal configurations at a fixed candidate budget.
| Configuration | |||
|---|---|---|---|
| Candidate groups per model-phase | 4,096 | 2,048 | 1,024 |
| Total candidate groups per run | 32,768 | 32,768 | 32,768 |
| Candidate responses per run | 262,144 | 262,144 | 262,144 |
| Seeds per configuration | 5 | 5 | 5 |
| Route fit per model-phase | 384 | 384 | 384 |
| Route calibration per model-phase | 128 | 128 | 128 |
| Audit groups per domain per model-phase | 128 | 128 | 128 |
| Certificate groups per model-phase | 8,192 | 8,192 | 8,192 |
Notes. counts phases per policy, so each run has model-phases across both policies. A candidate group contains one complete response for each of eight prompt formulations of the same item. Route-fit, calibration, audit and certificate groups are separate from the update candidates.
In this configuration, is calibrated strict-consistency probability and its reliability margin; truncates negative values to zero. H-proxy uses Gaussian-perturbed competence scores; UQ-proxy uses acquired-RM uncertainty , escalating the highest 12%. These score proxies are distinct from the fitted matched-information BayesianRouter-H and Uncertainty-H learners.
For queried group , let be its reliability margin and its directional alignment with the trusted audit gradient. With the number of queried groups and their admission indicators, the mean feedback score is .
Table R25. Routing, admission and full cost across phase-renewal schedules.
| Method | Queries | Admitted | Feedback score | GG-risk | Full cost USD/run | |
|---|---|---|---|---|---|---|
| 4 | H-proxy | 32,768 | 32,369.8 | 0.1688 | 0.03181 | 145.81 |
| 4 | UQ-proxy | 32,768 | 32,700.4 | 0.2484 | 0.00800 | 122.24 |
| 4 | PIVOT | 32,768 | 32,504.4 | 0.3126 | 0.00268 | 1,721.86 |
| 8 | H-proxy | 32,768 | 32,382.4 | 0.1688 | 0.03168 | 225.55 |
| 8 | UQ-proxy | 32,768 | 32,707.4 | 0.2483 | 0.00779 | 139.43 |
| 8 | PIVOT | 32,768 | 32,491.8 | 0.3134 | 0.00257 | 3,231.24 |
| 16 | H-proxy | 32,768 | 32,361.6 | 0.1692 | 0.03221 | 384.29 |
| 16 | UQ-proxy | 32,768 | 32,705.2 | 0.2490 | 0.00813 | 170.42 |
| 16 | PIVOT | 32,768 | 32,515.0 | 0.3131 | 0.00241 | 6,234.62 |
Notes. Values are means across five seeds. Queries and admissions count groups; fractional admission counts reflect averaging. Feedback score averages admission-weighted over all queried groups. Full cost is USD per run, including generation, optimization, setup, certification, acquired feedback, audit/virtual steps, fitting, evaluation and runtime.
Response to Reviewer 54ct
We thank the reviewer for the numerical checks and the question about repeated setup costs.
W1 and Q1 Graph density normalization
The density columns in Tables S25 and S26 report unordered-pair coverage. The S25 caption incorrectly names the directed-edge denominator. For a graph with K responses and preference-edge set E, let cpair be unordered-pair coverage and ddir directed-edge density. With ∣E∣ denoting the edge count, the two normalizations are
\qquad d_{\mathrm{dir}}=\frac{\lvert E\rvert}{K(K-1)}. \tag{R17}$$ Thus the two normalizations have DAG upper bounds of 1 and 0.5, respectively, and use denominators 28 and 56 at $K=8$. The captions should name unordered-pair coverage for the displayed values. For audited group $i$, let $A_i\in\\{0,1\\}$ indicate final admission, $C_i\in\\{0,1\\}$ strict gold-graph consistency, and $\rho_i$ its prespecified consequence weight. Empirical GG-risk $\widehat R_{\mathrm{GG}}$ is $$\widehat R_{\mathrm{GG}} =\frac{\sum_i\rho_iA_i(1-C_i)}{\sum_i\rho_iA_i}. \tag{R18}$$ In the local experiment, the consistency model uses $d_{\mathrm{dir}}$ before isotonic probability calibration, while the displayed diagnostic is $c_{\mathrm{pair}}$. Their factor-of-two relation resolves the apparent violation of the DAG limit without changing the graph's edges. Correcting the caption leaves this calibrated input unchanged, and the GG-risk calculated from the same recorded admissions and strict-consistency labels in Eq. (R18) is unchanged.