- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether improvements measured by a cheap proxy verifier remain improvements after deployment, when the deployed policy can change the environment or other agents’ responses. It introduces operator-relative Improvement Fidelity, distinguishing aggregate policy-evaluation accuracy from the reliability of the particular updates an improvement process proposes. The theoretical results relate update errors to uniform value accuracy, operator shift, and candidate margins. PIVOT-KG models proxy-to-deployment corrections with a calibrated Gaussian belief and allocates paired high-fidelity evaluations according to expected selection-regret reduction per unit cost. Experiments cover controlled response environments, poker, Melting Pot, and driving simulators. The reported results demonstrate substantial proxy–deployment disagreement and benefits over Uniform in several settings, particularly Leduc and redesigned HighwayEnv. The supplementary artifact provides executable selectors, saved observations, and analysis records that support offline verification of the principal reported results.
Critical strengths
-
The paper articulates a useful evaluation target for policy improvement. Measuring fidelity over the transitions actually proposed by an improvement operator makes explicit why strong aggregate evaluation metrics may fail to protect replacement decisions. Connecting this issue to deployment response and candidate margins gives the formulation practical relevance: it distinguishes evaluation errors that merely affect predicted values from errors that change the selected update.
-
The principal allocation comparisons are controlled, and the supplementary artifact substantially improves their verifiability. The Table 1 experiments and redesigned HighwayEnv comparison share posterior updating and terminal selection between PIVOT-KG and Uniform, supporting attribution of their differences to query allocation. The supplementary source and saved observations permit replay of the primary decisions, while redesigned HighwayEnv uses an exact expectation over Uniform query subsets. These features strengthen the positive Leduc and HighwayEnv results. The manuscript also reports several inconclusive and negative comparisons, making the limits of the evidence more transparent.
Critical weaknesses
-
The methodological advance beyond established knowledge-gradient and simulation-selection methods remains limited. Under the stated posterior-mean terminal decision, Eq. (6) reduces to the standard one-step knowledge-gradient objective, with cost normalization. The supplied implementation explicitly uses correlated Gaussian KG, and correlated beliefs together with paired simulation have direct precedents in Frazier et al. (2009) and Xie et al. (2016). The paper acknowledges these connections, but it remains unclear what additional methodological capability is required by, or developed specifically for, the adaptive-world setting. The theoretical results provide useful organization and counterexamples, but largely follow elementary error and distribution-shift arguments.
-
The existing stronger-baseline comparisons do not yet establish a compelling incremental benefit of the KG-specific allocation rule. In the first-cohort Leduc primary setting, PIVOT-KG, IVR, and LUCB select the same policy on all 30 test roots and attain the same mean ISR of approximately 0.00927. Thus, this positive result against Uniform does not distinguish the proposed acquisition objective from simpler targeted allocation rules. In the second Leduc cohort, the primary comparison is positive against expected Uniform, but the secondary comparisons against LUCB and top-proxy do not establish a clear advantage; the IVR-menu contrast is also based on few differing selections. The strongest redesigned HighwayEnv result emphasizes Uniform rather than these matched targeted alternatives.
Questions for authors
-
Clarification of contribution: Which part of PIVOT-KG constitutes a methodological advance beyond applying established correlated Gaussian knowledge gradient to proxy-to-deployment corrections? In particular, does the adaptive-world formulation introduce additional estimation or decision structure beyond the calibrated correction prior, or should the contribution primarily be understood as the evaluation formulation and empirical study?
-
Evidence that could change the assessment: The supplementary first-cohort Leduc results show identical final selections for PIVOT-KG, IVR, and LUCB on all 30 primary test roots. Are matched IVR, LUCB, and top-proxy results available for the eight-candidate HighwayEnv redesign, using the same prior, observation model, query budget, and terminal selection rule?
-
Calibration robustness: The exact-versus-noisy calibration comparison is useful, but it does not quantify variability from drawing a different calibration cohort. How stable are the principal contrasts when the correction model is refitted across calibration-root resamples or independent calibration sets? For the first poker cohorts, how sensitive are the conclusions to the observation-variance floor and covariance regularization?
2 comments
Thank you for the clarifications and additional experiments. They address most of my previous concerns about the baseline comparisons and calibration sensitivity, and they strengthen the empirical evidence. However, the methodological novelty remains limited, and neither of the two new HighwayEnv cohorts shows a clear advantage for PIVOT-KG over IVR. These additions do not substantially change my assessment of the paper’s overall contribution. I therefore maintain my original rating of 2 (Weak Rejection).
RESPONSE TO REVIEWER 2o9X
Advance beyond KG. Equation 6 represents an established one-step correlated-Gaussian KG with cost normalization. We introduce Improvement Fidelity as an operator-relative target for the sign and ordering of proxy versus deployment gains for proposed updates and link differential errors to decision margins. PIVOT is the paired validation framework. PIVOT-KG applies existing KG to this structure. It allows evaluation of whether calibration helps, whether new HF observations help, and whether allocating the same HF budget differently helps.
HighwayEnv baselines. The saved-truth replay favors PIVOT-KG over exact Uniform. At K=8 and B=4, mean ISR falls from 0.0435 to 0.0055. This results in a difference of 0.0380 with a paired 95 percent CI of [0.0300, 0.0474]. This replay uses saved deployment values and simulated observations. Sec. 6.5 reports fresh simulator runs where PIVOT-KG, integrated variance reduction (IVR), and Top-Proxy share the panel, prior, observation model, posterior update, query budget, and terminal rule.
Baseline minus PIVOT-KG ISR contrasts are shown with paired 95 percent CIs. Positive values favor PIVOT-KG.
Both IVR intervals include zero. The PIVOT-KG advantage over Top-Proxy in development was not established on the independent cohort. Standard LUCB cannot complete its eight initial pulls at B=4. Strong baselines can make the same decisions. In the first Leduc cohort, KG, IVR, and LUCB chose the same action at all 30 roots. In the exploratory MetaDrive PGMA, KG and IVR agree in all 24 roots and all three disagreements vs Top-Proxy favor Top-Proxy. Across all 24 roots the KG-minus-Top-Proxy utility contrast is -8.396448. The +26.8376 gain for Calibrated-No-HF versus Proxy-Only measures calibration value (Appendix C).