- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies when an independent counterfactual audit can certify that a frozen fine-tuned model improves over a factual-training baseline under a declared class of nuisance shifts. It develops paired, cluster-aware certificates and an independent pilot–plan–certify procedure that selects audit size, sibling count, estimator, and confidence allocation while accounting for uncertainty in the planning moments.
Critical strengths
- The paper addresses a substantive statistical design problem. Beyond constructing valid certificates, it asks how to collect enough independent evidence to achieve a specified certification power when the relevant moments are unknown.
- The dependence structure is handled carefully. It clearly distinguishes variance cancellation from pairing and observation savings from anchor reuse.
- The guarantees and limitations are stated precisely.
Critical weaknesses
-
The boundary of novelty could be sharper. Paired comparisons, variance decomposition, mean–variance robustness, and independent pilot planning have established foundations. The strongest contribution appears to be the selectable finite-sample power criteria and uncertainty-aware planning for clustered robust endpoints. A more explicit comparison with external-pilot planning and risk-control methods would clarify which results require genuinely new analysis.
-
ReCoRD uses finite-catalog sampling with enumerated prompt views. Feasibility using real model outputs and only global loss bounds remains unclear.
Questions for authors
Can the main planner return feasible plans on real model outputs using only global loss bounds? Please report pilot cost, feasibility, and total cost.
6 comments
Q1 / W2 (1/2) — Full-Planner Feasibility on Real Model Outputs
Thank you for identifying the need to test the full planner on real outputs.
Yes. In our additional frozen-Qwen/AG News experiment, the full 448-configuration planner returned feasible plans in 192/200 attempts; 187 independent final audits certified a robust gain. The mean total cost, including every pilot and all eight stops without a plan, was 53,534.72 logical model–view evaluations. The full protocol and results appear in Appendix K.1, K.2, and K.3; Comment 3 gives the outcomes and fixed-budget comparisons below.
Real outputs and sampled siblings. We compare frozen factual and CFPT LoRA models based on Qwen2.5-3B-Instruct on a fixed catalog of 7,600 AG News sources. Each source has four content-preserving input formats: original, field-labeled, JSON-formatted, and instruction-formatted, with reference probabilities . Sources are drawn IID with replacement from the uniform catalog law, and siblings IID with replacement from . Both models receive the same source/view/label. The loss difference is for 0–1 classification loss; the conditional Pearson radius is . A negative endpoint certifies improvement over the declared conditional view reweightings, holding the source law, labels, and four views fixed. This is a sampled audit of real model outputs on a finite catalog.
The complete search. Each attempt uses 128 design-pilot anchors and 1,024 independent estimation-pilot anchors. The design pilot fixes centering/tangency objects; it does not fix . The estimation pilot evaluates 32 budgets , two sibling counts , two confidence allocations for each of the paired and separate endpoint families, and three generic-endpoint settings: configurations. This is the paper's planner applied to this frozen experimental menu. Selection uses the finite-sample sufficient-power bound and Berry–Esseen witness, with the frozen ordering by , then , then tie-breakers. The selected plan is locked before fresh final source/sibling sampling.
Only global loss bounds enter. For the paired anchor statistics, the ranges are and , with . Unknown moments are estimated with uncertainty from the independent pilot. The planner receives no full-catalog population moments, catalog-specific extrema, or final-audit outcomes. It samples siblings rather than enumerating the four views. Global bounds provide valid ranges for inference; the pilot supplies information about sensitivity within those ranges.
Why selection preserves the guarantee. Design-frozen response definitions and tangencies incorporate the joint mean–sensitivity response. The estimation pilot constructs a simultaneous confidence set for the moments across the menu. The finite-sample sufficient-power bound (Theorem 1) or a Berry–Esseen witness with its validity conditions satisfied supports selection over that set (Theorem 2). Independent final data then evaluate the locked endpoint.
Under the stated assumptions, final false-certification probability is at most for every pilot history. With pilot probability at least , every returned feasible plan has conditional final certification power at least . End-to-end success also depends on returning a feasible plan, whose existence is not guaranteed. The observed 187/192 pools selected plans rather than estimating each plan's conditional power. The 0.05 safety level applies per audit, not simultaneously to all 200 repetitions.
This experiment answers the feasibility question directly: the complete planner obtains plans and robust-gain certificates from sampled real-model outputs using global loss bounds and pilot-estimated moments.
Q1 / W2 (2/2) — Pilot Costs, Total Costs, and Fixed-Budget Comparisons
The all-attempt result is 187/200 certificates at a mean total cost of 53,534.72 logical single-model–view evaluations. All 200 attempts completed. They redraw audit samples for the same frozen Qwen2.5-3B-Instruct factual/CFPT pair and 7,600-source AG News catalog described in Comment 2; they are not retrainings or new datasets. The loss is 0–1 error, the conditional Pearson radius is , and final safety is .
| Outcome | Count | Two-sided 95% exact binomial interval |
|---|---|---|
| Feasible plans / attempts | 192/200 (96.0%) | [92.27%, 98.26%] |
| Pilot stops without a plan | 8/200 | — |
| Certificates / executed final audits | 187/192 (97.40%) | [94.03%, 99.15%] |
| Certificates / all attempts | 187/200 (93.5%) | [89.14%, 96.49%] |
Every returned plan was audited once. Five final audits did not certify a gain. A pilot stop means no menu entry passed the planning criterion; a nonnegative final endpoint means that audit did not certify improvement. Neither outcome establishes harm. The intervals describe repeated-audit frequencies in this fixed setting.
| Selected | Plans | Certificates | Final evaluations per plan |
|---|---|---|---|
| 43 | 42 | 8,192 | |
| 71 | 70 | 16,384 | |
| 30 | 29 | 65,536 | |
| 28 | 27 | 32,768 | |
| 20 | 19 | 131,072 | |
| Total | 192 | 187 | — |
All branches enter the cost. Each pilot anchor supplies eight sampled siblings for both models; scores are reused for smaller- candidates. The design pilot costs evaluations and the estimation pilot , totaling 18,432 per attempt. A returned plan adds final evaluations. Each of the eight stops pays the full pilot cost and no final cost.
| Cost quantity | Evaluations |
|---|---|
| All 200 pilots | 3,686,400 |
| All 192 final audits | 7,020,544 |
| Total | 10,706,944 |
| Mean per attempt, including stops | 53,534.72 |
| Median / 90th percentile total per attempt | 34,816 / 83,968 |
| Total per obtained certificate | 57,256.39 |
These are logical classifier–view evaluations, not GPU-time measurements or counts of distinct uncached forward passes.
Competitive fixed budgets. Full experimental details appear in Appendix K.1, K.2, and K.3. All five comparisons use the same frozen models, loss, reference law, radius, and final safety. Fixed audits use the paired P-A endpoint and pay no pilot cost.
| Procedure | Certificates / 200 | Observed rate | Mean evaluations |
|---|---|---|---|
| Fixed | 125/200 | 62.5% | 2,048 |
| Fixed | 165/200 | 82.5% | 8,192 |
| Fixed | 177/200 | 88.5% | 16,384 |
| Fixed | 184/200 | 92.0% | 32,768 |
| Fixed | 189/200 | 94.5% | 131,072 |
| Planner, including stops | 187/200 | 93.5% | 53,534.72 |
Against fixed (184/200 certificates; 32,768 evaluations), the planner costs 63.4% more for a 1.5-percentage-point increase in observed certification frequency, without establishing cost or power superiority or statistical equivalence. Because the planner selects endpoint families, this is a procedure-level comparison under common evaluation and safety definitions, not an endpoint-by-endpoint ablation.
The overhead buys independent pilot information, collection sufficient to compare both sibling counts, uncertainty bounds on planning moments, and a prospective sufficient-power criterion. The selection rule minimizes sufficient final on the menu, with tie-breakers; it does not minimize pilot-plus-final cost. This accounting makes feasibility and resource requirements explicit: 192 plans and 187 certificates across 200 attempts, including every pilot stop and final audit.
W1 — Novelty Relative to External-Pilot Planning and Risk Control
The contribution is selectable finite-sample power criteria and uncertainty-aware planning for clustered robust endpoints. Pairing, variance decomposition, mean–variance robustness, and independent pilots are established ingredients. The statistical problem here is to justify certification power when both the mean loss difference and its shift sensitivity are unknown, while selecting audit size, sibling count, endpoint, and confidence allocation before independent final data.
Relation to external-pilot planning and risk control. Whitehead et al. (2016) study external-pilot sizing for continuous-outcome trials, accounting for uncertainty in standard-deviation estimates and pilot-plus-main sample cost. Learn Then Test formulates predictive risk calibration through multiple testing with finite-sample guarantees. Our object is the power of a robust paired-loss endpoint whose sensitivity is itself estimated from sampled siblings. The analysis must carry the joint mean–sensitivity response through finite-budget power criteria and allow subsequent selection across the planning menu.
For paired anchor statistics M and Q, the identity Var(M+aQ)=Var(M)+2aCov(M,Q)+a2Var(Q) is standard. The contribution is incorporating these unknown joint moments into sufficient-power criteria and a simultaneous pilot confidence set that remains valid for the selected design. The finite-sample sufficient-power bound and Berry–Esseen witness supply the power justification, with Theorem 1 giving the sufficient-power bound and Theorem 2 supporting pilot-based selection. The oracle-limit result concerns the same sufficient-witness menu; it is not an optimum over actual-power designs or total costs.
Why estimating sensitivity matters. In the frozen-Qwen/AG News experiment, a returned paired plan with (N,K)=(4096,2) had independent final bounds M+=−0.150 and V+=0.220. At ρ=0.05, its endpoint was U=M++ρV+=−0.150+0.05(0.220)≈−0.0451191, certifying a gain of at least 4.5119 percentage points over the declared conditional view-reweighting class. Its pilot-plus-final cost was 2,048+16,384+16,384=34,816 evaluations. Replacing the sensitivity bound by the global population cap V≤1, while holding the same mean bound fixed, gives a positive endpoint of 7.3607 percentage points. This final-endpoint calculation illustrates the information gained by estimating sensitivity within global bounds; prospective power is justified by the bound and witness above.
The value is prospective design information. A valid fixed audit already controls false certification. The planner additionally determines, before final outcomes are observed, whether a simultaneous pilot confidence set supports a sufficient power statement for the chosen design. This provides a reasoned choice when the necessary design is unknown; an observed fixed-budget success frequency answers a different question.
The contribution enables prospective power-based selection for a robust endpoint with estimated sensitivity. The frozen-Qwen/AG News experiment demonstrates that this selection operates on sampled real-model outputs under the declared finite-catalog law.