- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when an adaptive scientific imaging policy should replace a conservative fallback under a fixed resource budget. The authors propose Q-Safe, a framework that combines feasibility checks, objective and proxy validation, and bounded-loss calibration to control the risk of policy replacement. Its core replacement mechanism uses a counterfactual representation of the proposed and fallback actions to predict replacement risk, followed by threshold calibration over spatial blocks.
The main experiments on ConnectomeBench2 evaluate two loss contracts, binary harm and fallback regret, using prospectively frozen test sets. Additional experiments on other imaging datasets examine schedule feasibility, objective alignment, and proxy validity. The results demonstrate that risk-controlled replacement can improve task utility and that different loss definitions can lead to substantially different deployment decisions.
Critical strengths
-
The paper addresses a meaningful problem: when should an adaptive policy be allowed to replace an existing policy under a fixed budget. Rather than assuming that learned adaptation is always beneficial, the authors explicitly consider the potential harm of replacement and the availability of a conservative fallback. This is a useful perspective for scientific imaging, where incorrect decisions can affect downstream scientific analyses.
-
The authors make a substantial effort to evaluate their approach through frozen protocols, independent test shards, and comparisons across different species and loss definitions. The second confirmation experiment is particularly informative in demonstrating how changing the loss contract affects coverage, utility, and harmful actions. The external diagnostic experiments also provide useful negative results and help clarify the limitations of adaptive acquisition.
Critical weaknesses
-
The methodological novelty of Q-Safe is not sufficiently established. The core approach builds on existing work in risk-controlled policy switching and bounded-loss calibration (e.g., Joshi et al., 2026; Bi et al., 2026; Angelopoulos et al., 2024). While the application setting and decision granularity differ, the paper does not clearly establish what methodological advances Q-Safe introduces beyond these approaches. The limited utility changes in the representation ablations and the competitive performance of simpler encoders further leave the source of its contribution unclear.
-
The experimental evidence does not fully support the proposed integrated framework. The paper presents a unified diagnose–then–replace framework, but its components are largely evaluated independently. The main experiments support the effectiveness of risk-controlled policy replacement, while the external studies separately examine schedule feasibility, objective alignment, and proxy validity. How these components interact and what additional value their integration provides remain insufficiently demonstrated.
Questions for authors
-
What are the specific methodological capabilities of Q-Safe that distinguish it from existing risk-controlled policy switching and adaptive-policy authorization methods, particularly Joshi et al. (2026) and Bi et al. (2026), beyond differences in application setting and decision granularity?
-
Can the authors demonstrate how the schedule-feasibility, objective-alignment, and proxy-validation modules interact with the policy-replacement mechanism within a single end-to-end task? In particular, how do the outcomes of these checks affect the subsequent calibration and final deployment decisions?
-
What role does the predicted utility difference (\hat{\Delta}\theta) play in the final replacement decision, given that the calibration procedure appears to rely on the harm score (s\theta)? Would removing the utility-prediction head affect the final risk–coverage–utility trade-off?
-
How much of the observed improvement can be attributed specifically to Q-Safe rather than applying an existing bounded-loss risk-control procedure to the same proposal–fallback pairs and learned scores? Could a matched comparison help isolate this contribution?
References
[1] Joshi, S., Wang, T., Hassani, H., & Dobriban, E. (2026). Risk-Controlled Post-Processing of Decision Policies. arXiv:2605.06479.
[2] Bi, J., Pinilla, S., & Zhu, C. (2026). Certifying When Decision-Time Information Justifies Adaptive Experimentation. arXiv:2607.27651.
3 comments
Thank you for the detailed response and additional experiments. I consider Q2 and Q3 adequately addressed.
My remaining concerns are Q1 and Q4. Could you clarify what methodological capabilities distinguish Q-Safe from Joshi et al. (2026) and Bi et al. (2026), beyond differences in application setting and decision granularity?
Relatedly, given that standard CRC produces an identical gate under matched conditions, what additional contributions or measurable benefits arise from Q-Safe's proposed design and framework integration?
This would help me better assess the novelty of the proposed method.
Thank you for the follow-up.
Q1. Methodological distinction
Joshi et al. choose when to depart from an incumbent policy using the estimated change in the probability of exceeding a specified loss cutoff. Q-Safe defines loss relative to an executable, budget-matched fallback. It calibrates either the incidence of harmful replacements or their mean utility shortfall, with fallback executions contributing zero loss. On ConnectomeBench2, the harm scorer pools the two complete top- sets separately and uses both embeddings and their signed difference (). Shared items cancel in the realized utility difference under additive utility (Eq. 7), motivating this paired representation.
OPAL uses held-out evidence to authorize adaptive measurement and distinguishes oracle opportunity from decision-time information. Q-Safe applies this distinction to the complete acquisition and processing pipeline. For additive tile losses, the equal-cost oracle test upper-bounds the gain of any rescan ranking after accounting for preview cost. On validation data, we check the acquisition objective against the target endpoint, compare the processed proposal with the processed uniform reference, and test whether the preview-derived score ranks their realized utility differences. The processor comparison measures how mixing preview and rescan inputs affects the task outcome. These checks determine which configurations advance to calibration.
Q4. What changes the replacement decision
With the action pairs, scores, loss, spatial blocks, and domain rule fixed, applying add-one CRC within each calibration group and intersecting the resulting thresholds yields the same gate. The following comparisons show how the representation, loss, validation criteria, and calibration grouping affect the resulting decisions.
- Paired representation (Tables 1 and 2). Removing the explicit embedding difference lowers harm AP from 0.249 to 0.224 and raises worst-shard harm on reused development shards from 1.579% to 1.835%. Utility decreases by 0.015 pp. Under the same calibration procedure, the full counterfactual scorer adds 1.157 pp utility over ExtraTrees on
test25–28(paired 95% CI [0.812, 1.512], primarily in mouse). Ontest29–32, under the same regret calibration procedure, it achieves 2.61 pp higher coverage, 1.01 pp higher utility, and 0.048 pp lower regret. - Replacement loss (Table 3). With the scorer and calibration procedure fixed, switching the 2% target from harm incidence to fallback regret on
test29–32increases coverage by 13.99 pp and utility by 3.30 pp, primarily in fly. Pooled regret is 0.191%, and harm incidence rises from 0.539% to 1.018%. The loss definition determines which frequency–severity trade-off the gate implements. - Validation before calibration (3D PFIB replay). At matched nominal dwell, the earlier mitochondrial processor with a gap veto reached 0.7943 Dice and 0.3723 VOI on validation. With component consolidation, the selective configuration reached 0.8268 Dice and 0.3040 VOI, versus 0.7997 and 0.3112 for uniform with the same consolidation. The context-level selector passed validation (ρ = 0.371), and the pipeline advanced to calibration. On the one-shot spatial holdout, the gate achieved 62.50% coverage and 1.089% regret. Relative to uniform fallback, mean Dice rose from 0.3804 to 0.6208 and mean VOI fell from 0.3646 to 0.3591. For synapses, selective processing trailed uniform on validation and the selector had ρ = −0.019. The workflow retained uniform acquisition before calibration.
- Calibration across fields (2D Liver replay). With the same scorer and 2% binary-harm target, pooled calibration yielded 7.86% coverage and 2.143% observed harm. One validation field had no admissible threshold, so per-field intersection selected uniform fallback throughout, with 0% coverage and 0% harm.
We thank Reviewer Z2tq for the specific questions.
Q1 & W1. Relationship to prior risk-controlled switching
Joshi et al. (2026) preserve agreement with an existing policy subject to a constraint on loss violations. Bi et al.'s OPAL (2026) uses held-out evidence on activation risk, coverage, and net value to authorize an adaptive acquisition rule against a fixed plan. Q-Safe combines the following capabilities for budget-matched acquisition.
Q2 & W2. How the checks affect deployment
Our registered multi-dwell EM replays (Global Response §2) show how validation determines whether to calibrate a gate or retain uniform acquisition.
Q3. Role of the utility-prediction head
For supported groups, acceptance is determined by sθ(q,k)≤tc. The utility head contributes through the auxiliary Smooth L1 training loss (weight 0.15, Eq. 6). In a single-seed post-confirmation ablation, we set that weight to zero, retrained, and reapplied the same calibration procedure.
Auxiliary utility supervision improved validation harm AP and increased coverage in this ablation. Deployed utility was slightly higher with its weight set to zero. We will clarify the head's training role in §4.3.
Q4. What is attributable to Q-Safe beyond CRC?
With the same action pairs, scorer, loss, spatial blocks, and threshold rules, applying standard add-one CRC within each calibration group and intersecting the resulting thresholds yields the Q-Safe gate. On test25–28, holding the scorer fixed, calibration changed observed harm from 1.881% to 0.190% and utility from 21.847 to 12.816 pp. Holding calibration fixed, the counterfactual scorer gained 1.157 pp utility over ExtraTrees.
With the scorer fixed on test29–32, changing the replacement loss from incidence to regret increased coverage by 13.99 pp and utility by 3.30 pp, with 0.191% pooled regret and higher harm incidence (1.018% vs. 0.539%). This comparison isolates the effect of the declared loss on deployment under the same calibration procedure.