acceptodds
Under review as a conference paper at ICLR 2027

When Can Counterfactual Audits Certify a Fine-Tuning Gain?

Abstract

Counterfactual post-training expands supervision through generated edits, but neither more edits nor greater predictive consistency establishes a gain over factual training. Certifying that gain requires an independent comparison with enough power to detect it under the declared nuisance shifts. We study this evidence requirement through paired losses: common model responses can cancel, while additional edits cannot remove uncertainty across task anchors. Shared-anchor certificates and finite-sample power bounds distinguish these two sources of cost. An independent pilot–plan–certify procedure then selects the audit size, sibling count, estimator, and confidence allocation from a frozen menu, accounting for uncertainty in the planning moments themselves. Its guarantees separate false certification, pilot reliability, and final power conditional on a feasible plan. Controlled experiments isolate the savings from pairing and anchor reuse; across three planning laws, 109 of 144 pilots return feasible plans, each successful in all 256 fresh replications, but their budgets remain conservative. A prospective Qwen/ReCoRD audit illustrates covered-shift certification with a different, finite-catalog design. The practical lesson is to budget for uncertainty in the audit plan, not just in the final result: a valid certificate and an economical, adequately powered audit are distinct achievements.

Primary area:
learning theory
Keywords:
counterfactual post-training, language model post-training, learning theory, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eKwMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when an independent counterfactual audit can certify that a frozen fine-tuned model improves over a factual-training baseline under a declared class of nuisance shifts. It develops paired, cluster-aware certificates and an independent pilot–plan–certify procedure that selects audit size, sibling count, estimator, and confidence allocation while accounting for uncertainty in the planning moments.

Critical strengths
  1. The paper addresses a substantive statistical design problem. Beyond constructing valid certificates, it asks how to collect enough independent evidence to achieve a specified certification power when the relevant moments are unknown.
  2. The dependence structure is handled carefully. It clearly distinguishes variance cancellation from pairing and observation savings from anchor reuse.
  3. The guarantees and limitations are stated precisely.
Critical weaknesses
  1. The boundary of novelty could be sharper. Paired comparisons, variance decomposition, mean–variance robustness, and independent pilot planning have established foundations. The strongest contribution appears to be the selectable finite-sample power criteria and uncertainty-aware planning for clustered robust endpoints. A more explicit comparison with external-pilot planning and risk-control methods would clarify which results require genuinely new analysis.

  2. ReCoRD uses finite-catalog sampling with enumerated prompt views. Feasibility using real model outputs and only global loss bounds remains unclear.

Questions for authors

Can the main planner return feasible plans on real model outputs using only global loss bounds? Please report pilot cost, feasibility, and total cost.

6 comments
AuthorsOpenReview ↗

W1 — Novelty Relative to External-Pilot Planning and Risk Control

The contribution is selectable finite-sample power criteria and uncertainty-aware planning for clustered robust endpoints. Pairing, variance decomposition, mean–variance robustness, and independent pilots are established ingredients. The statistical problem here is to justify certification power when both the mean loss difference and its shift sensitivity are unknown, while selecting audit size, sibling count, endpoint, and confidence allocation before independent final data.

Relation to external-pilot planning and risk control. Whitehead et al. (2016) study external-pilot sizing for continuous-outcome trials, accounting for uncertainty in standard-deviation estimates and pilot-plus-main sample cost. Learn Then Test formulates predictive risk calibration through multiple testing with finite-sample guarantees. Our object is the power of a robust paired-loss endpoint whose sensitivity is itself estimated from sampled siblings. The analysis must carry the joint mean–sensitivity response through finite-budget power criteria and allow subsequent selection across the planning menu.

For paired anchor statistics MM and QQ, the identity Var⁡(M+aQ)=Var⁡(M)+2aCov⁡(M,Q)+a2Var⁡(Q)\operatorname{Var}(M+aQ)=\operatorname{Var}(M)+2a\operatorname{Cov}(M,Q)+a^2\operatorname{Var}(Q) is standard. The contribution is incorporating these unknown joint moments into sufficient-power criteria and a simultaneous pilot confidence set that remains valid for the selected design. The finite-sample sufficient-power bound and Berry–Esseen witness supply the power justification, with Theorem 1 giving the sufficient-power bound and Theorem 2 supporting pilot-based selection. The oracle-limit result concerns the same sufficient-witness menu; it is not an optimum over actual-power designs or total costs.

Why estimating sensitivity matters. In the frozen-Qwen/AG News experiment, a returned paired plan with (N,K)=(4096,2)(N,K)=(4096,2) had independent final bounds M+=−0.150M^+=-0.150 and V+=0.220V^+=0.220. At ρ=0.05\rho=0.05, its endpoint was U=M++ρV+=−0.150+0.05(0.220)≈−0.0451191,U=M^++\sqrt{\rho V^+}=-0.150+\sqrt{0.05(0.220)}\approx-0.0451191, certifying a gain of at least 4.5119 percentage points over the declared conditional view-reweighting class. Its pilot-plus-final cost was 2,048+16,384+16,384=34,8162,048+16,384+16,384=34,816 evaluations. Replacing the sensitivity bound by the global population cap V≤1V\le1, while holding the same mean bound fixed, gives a positive endpoint of 7.3607 percentage points. This final-endpoint calculation illustrates the information gained by estimating sensitivity within global bounds; prospective power is justified by the bound and witness above.

The value is prospective design information. A valid fixed audit already controls false certification. The planner additionally determines, before final outcomes are observed, whether a simultaneous pilot confidence set supports a sufficient power statement for the chosen design. This provides a reasoned choice when the necessary design is unknown; an observed fixed-budget success frequency answers a different question.

The contribution enables prospective power-based selection for a robust endpoint with estimated sensitivity. The frozen-Qwen/AG News experiment demonstrates that this selection operates on sampled real-model outputs under the declared finite-catalog law.

AuthorsOpenReview ↗

Q1 / W2 (1/2) — Full-Planner Feasibility on Real Model Outputs

Thank you for identifying the need to test the full planner on real outputs.

Yes. In our additional frozen-Qwen/AG News experiment, the full 448-configuration planner returned feasible plans in 192/200 attempts; 187 independent final audits certified a robust gain. The mean total cost, including every pilot and all eight stops without a plan, was 53,534.72 logical model–view evaluations. The full protocol and results appear in Appendix K.1, K.2, and K.3; Comment 3 gives the outcomes and fixed-budget comparisons below.

Real outputs and sampled siblings. We compare frozen factual and CFPT LoRA models based on Qwen2.5-3B-Instruct on a fixed catalog of 7,600 AG News sources. Each source has four content-preserving input formats: original, field-labeled, JSON-formatted, and instruction-formatted, with reference probabilities ν=(0.40,0.30,0.20,0.10)\nu=(0.40,0.30,0.20,0.10). Sources are drawn IID with replacement from the uniform catalog law, and siblings IID with replacement from ν\nu. Both models receive the same source/view/label. The loss difference is Z=ℓ(wh)−ℓ(wF)∈[−1,1]Z=\ell(w_h)-\ell(w_F)\in[-1,1] for 0–1 classification loss; the conditional Pearson χ2\chi^2 radius is ρ=0.05\rho=0.05. A negative endpoint certifies improvement over the declared conditional view reweightings, holding the source law, labels, and four views fixed. This is a sampled audit of real model outputs on a finite catalog.

The complete search. Each attempt uses 128 design-pilot anchors and 1,024 independent estimation-pilot anchors. The design pilot fixes centering/tangency objects; it does not fix KK. The estimation pilot evaluates 32 budgets N∈256,512,…,8192N\in\\{256,512,\ldots,8192\\}, two sibling counts K∈2,8K\in\\{2,8\\}, two confidence allocations for each of the paired and separate endpoint families, and three generic-endpoint settings: 32×2×(2+2+3)=44832\times2\times(2+2+3)=448 configurations. This is the paper's planner applied to this frozen experimental menu. Selection uses the finite-sample sufficient-power bound and Berry–Esseen witness, with the frozen ordering by NN, then NKNK, then tie-breakers. The selected plan is locked before fresh final source/sibling sampling.

Only global loss bounds enter. For the paired anchor statistics, Mi=1K∑j=1KZij,Qi=1K∑j=1K/2(Zi,2j−1−Zi,2j)2,M_i=\frac1K\sum_{j=1}^K Z_{ij},\qquad Q_i=\frac1K\sum_{j=1}^{K/2}(Z_{i,2j-1}-Z_{i,2j})^2, the ranges are Mi∈[−1,1]M_i\in[-1,1] and Qi∈[0,2]Q_i\in[0,2], with EQi=Vpair≤1\mathbb E Q_i=V_{\rm pair}\le1. Unknown moments are estimated with uncertainty from the independent pilot. The planner receives no full-catalog population moments, catalog-specific extrema, or final-audit outcomes. It samples siblings rather than enumerating the four views. Global bounds provide valid ranges for inference; the pilot supplies information about sensitivity within those ranges.

Why selection preserves the guarantee. Design-frozen response definitions and tangencies incorporate the joint mean–sensitivity response. The estimation pilot constructs a simultaneous confidence set for the moments across the menu. The finite-sample sufficient-power bound (Theorem 1) or a Berry–Esseen witness with its validity conditions satisfied supports selection over that set (Theorem 2). Independent final data then evaluate the locked endpoint.

Under the stated assumptions, final false-certification probability is at most αfinal=0.05\alpha_{\rm final}=0.05 for every pilot history. With pilot probability at least 1−δplan=0.9751-\delta_{\rm plan}=0.975, every returned feasible plan has conditional final certification power at least p=0.90p=0.90. End-to-end success also depends on returning a feasible plan, whose existence is not guaranteed. The observed 187/192 pools selected plans rather than estimating each plan's conditional power. The 0.05 safety level applies per audit, not simultaneously to all 200 repetitions.

This experiment answers the feasibility question directly: the complete planner obtains plans and robust-gain certificates from sampled real-model outputs using global loss bounds and pilot-estimated moments.

AuthorsOpenReview ↗

Q1 / W2 (2/2) — Pilot Costs, Total Costs, and Fixed-Budget Comparisons

The all-attempt result is 187/200 certificates at a mean total cost of 53,534.72 logical single-model–view evaluations. All 200 attempts completed. They redraw audit samples for the same frozen Qwen2.5-3B-Instruct factual/CFPT pair and 7,600-source AG News catalog described in Comment 2; they are not retrainings or new datasets. The loss is 0–1 error, the conditional Pearson radius is ρ=0.05\rho=0.05, and final safety is α=0.05\alpha=0.05.

OutcomeCountTwo-sided 95% exact binomial interval
Feasible plans / attempts192/200 (96.0%)[92.27%, 98.26%]
Pilot stops without a plan8/200—
Certificates / executed final audits187/192 (97.40%)[94.03%, 99.15%]
Certificates / all attempts187/200 (93.5%)[89.14%, 96.49%]

Every returned plan was audited once. Five final audits did not certify a gain. A pilot stop means no menu entry passed the planning criterion; a nonnegative final endpoint means that audit did not certify improvement. Neither outcome establishes harm. The intervals describe repeated-audit frequencies in this fixed setting.

Selected (N,K)(N,K)PlansCertificatesFinal evaluations per plan
(2048,2)(2048,2)43428,192
(4096,2)(4096,2)717016,384
(4096,8)(4096,8)302965,536
(8192,2)(8192,2)282732,768
(8192,8)(8192,8)2019131,072
Total192187—

All branches enter the cost. Each pilot anchor supplies eight sampled siblings for both models; scores are reused for smaller-KK candidates. The design pilot costs 16(128)=2,04816(128)=2,048 evaluations and the estimation pilot 16(1024)=16,38416(1024)=16,384, totaling 18,432 per attempt. A returned plan adds 2KN2KN final evaluations. Each of the eight stops pays the full pilot cost and no final cost.

Cost quantityEvaluations
All 200 pilots3,686,400
All 192 final audits7,020,544
Total10,706,944
Mean per attempt, including stops53,534.72
Median / 90th percentile total per attempt34,816 / 83,968
Total per obtained certificate57,256.39

These are logical classifier–view evaluations, not GPU-time measurements or counts of distinct uncached forward passes.

Competitive fixed budgets. Full experimental details appear in Appendix K.1, K.2, and K.3. All five comparisons use the same frozen models, loss, reference law, radius, and final safety. Fixed audits use the paired P-A endpoint and pay no pilot cost.

ProcedureCertificates / 200Observed rateMean evaluations
Fixed (512,2)(512,2)125/20062.5%2,048
Fixed (2048,2)(2048,2)165/20082.5%8,192
Fixed (4096,2)(4096,2)177/20088.5%16,384
Fixed (8192,2)(8192,2)184/20092.0%32,768
Fixed (8192,8)(8192,8)189/20094.5%131,072
Planner, including stops187/20093.5%53,534.72

Against fixed (8192,2)(8192,2) (184/200 certificates; 32,768 evaluations), the planner costs 63.4% more for a 1.5-percentage-point increase in observed certification frequency, without establishing cost or power superiority or statistical equivalence. Because the planner selects endpoint families, this is a procedure-level comparison under common evaluation and safety definitions, not an endpoint-by-endpoint ablation.

The overhead buys independent pilot information, collection sufficient to compare both sibling counts, uncertainty bounds on planning moments, and a prospective sufficient-power criterion. The selection rule minimizes sufficient final NN on the menu, with tie-breakers; it does not minimize pilot-plus-final cost. This accounting makes feasibility and resource requirements explicit: 192 plans and 187 certificates across 200 attempts, including every pilot stop and final audit.

Reviewer_TVm8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a framework for studying whether Counterfactual Post-Training (CFPT) under bounded nuisance distribution shifts improves model performance more than factual training. Results show the effectiveness of the framework under certain hypotheses; however, the amount of data required to guarantee that it works can make it impractical.

Critical strengths
  • S1: The paper presents a theoretically grounded framework.
  • S2: The paper employs carefully controlled experiments to validate its main research questions.
  • S3: The paper transparently acknowledges its limitations.
Critical weaknesses
  • W1: The paper uses Pearson χ2\chi^2 divergence to characterize allowable nuisance shifts but does not establish whether this choice adequately captures realistic shifts in LLM applications.
  • W2: The pilot-based planning procedure produces substantially larger final-audit budgets than both empirical benchmarks and theoretical sufficient bounds, raising concerns about its conservativeness and practical efficiency, as the authors also acknowledge in lines 444-445.
Questions for authors
  • Q1: What motivates the choice of Pearson χ2\chi^2 divergence for characterizing nuisance shifts?
  • Q2: To what extent is this choice appropriate for modeling realistic distribution shifts in LLM applications?
  • Q3: Can the validity of this modeling assumption be empirically assessed by comparing it against an observed distribution of shifts?
5 comments
AuthorsOpenReview ↗

Q1 — Why Conditional Pearson χ²: Proof Intuition and Risk Controls

Thank you for your thoughtful review and for recognizing the theoretical framework, controlled experiments, and transparent reporting. Your question about the divergence choice prompted us to clarify its role in the proof and its connection to the risk comparison.

Conditional Pearson divergence follows directly from the risk comparison being certified: changing the view assignment within each task pairs a density-ratio norm with the within-task sensitivity of the two models' loss difference.

Fix a CFPT model and its factual comparator. Let CC denote the task, SS its view, μ\mu the task distribution, ν(S∣C)\nu(S\mid C) the reference router, and π(S∣C)\pi(S\mid C) the target router. Define gh(C,S)g_h(C,S) as the label-averaged loss of CFPT minus that of the factual comparator. Thus Δπ,h=Eμπgh<0\Delta_{\pi,h}=\mathbb E_{\mu\pi}g_h<0 favors CFPT. Both laws use the same task distribution, label mechanism q(Y∣C)q(Y\mid C), and input map. Assume losses in [0,B][0,B] and π(⋅∣C)≪ν(⋅∣C)\pi(\cdot\mid C)\ll\nu(\cdot\mid C) with a square-integrable density ratio.

Proof idea. Write aπ=dπ/dνa_\pi=d\pi/d\nu and gˉh(C)=Eν[gh∣C]\bar g_h(C)=\mathbb E_\nu[g_h\mid C]. Since each router is a conditional probability law, Eν[aπ−1∣C]=0\mathbb E_\nu[a_\pi-1\mid C]=0. Therefore

=\mathbb E_{\mu\nu}[(a_\pi-1)g_h] =\mathbb E_{\mu\nu}[(a_\pi-1)(g_h-\bar g_h(C))].$$ The task-level mean cancels: rerouting changes the allocation among views, while preserving the task population. Cauchy–Schwarz now gives $$|\Delta_{\pi,h}-\Delta_{\nu,h}| \le\sqrt{\rho_\mu(\pi)V_h},\qquad \rho_\mu(\pi)=\mathbb E_{\mu\nu}(a_\pi-1)^2,\quad V_h=\mathbb E_\mu\operatorname{Var}_\nu(g_h\mid C).$$ Pearson $\chi^2$ is exactly the squared norm of the routing change; $V_h$ is exactly the squared norm of the centered paired loss. We use target-relative-to-reference divergence, without a factor $1/2$. The radius averages conditional divergences over tasks, rather than requiring an identical bound for every task. Pairing also matters to this choice of sensitivity. A view that increases both models' losses by the same amount leaves $g_h$ unchanged. Consequently, $V_h$ measures sensitivity of the comparison itself, retaining cancellations that separate model-wise bounds can lose. For the declared class, $$\sup_{\pi:\rho_\mu(\pi)\le\rho_0}\Delta_{\pi,h} \le \Delta_{\nu,h}+\sqrt{\rho_0V_h}.$$ An upper endpoint below zero certifies improvement throughout this class. **When is the bound sharp?** If $V_h>0$, the candidate tilt $$a_{\pi_0}=1+\sqrt{\rho_0/V_h}\\,(g_h-\bar g_h(C))$$ has conditional mean one and squared centered norm $\rho_0$. It attains the upper bound when $a_{\pi_0}\ge0$ almost surely. The upper bound remains valid when this tilt fails nonnegativity; equality requires an admissible optimizer. If $V_h=0$, every supported router has the same risk difference. The controlled simulation in synthetic catalog B illustrates the consequence. The reference gap is **−1.058 percentage points (pp)**, yet an admissible router at radius $0.05$ increases it to **+5.820 pp**. Thus an allowed rerouting can reverse the favorable reference comparison. This answers Q1: conditional Pearson divergence and paired sensitivity are the two squared norms in the same change-of-measure identity. Their product controls the target risk gap. Comments 2–3 assess class membership and risk under the stated application conditions.
AuthorsOpenReview ↗

Q2 — Testing Shift-Class Membership with Seven Controlled Routers

Thank you for emphasizing the need to connect the shift class to concrete LLM use cases. In response, we added the seven-router study below to make the covered mechanisms and the membership test explicit.

The class describes task-dependent rerouting among fixed, valid views of the same task population—for example, assigning existing prompt variants to requests. Applicability requires a common source distribution and label mechanism, unchanged concrete inputs for each view, and positive reference probability for every target view. Freeze the model pair, losses, and routers before evaluation. These conditions must be checked alongside the radius.

Seven-router controlled simulation (Appendix J.1). The reference is uniform over four views. For executed routing probabilities ps(C)=π(s∣C)p_s(C)=\pi(s\mid C), the conditional radius is ρμ=Eμ[4∑s=14ps(C)2−1].\rho_\mu= \mathbb E_\mu\left[4\sum_{s=1}^4p_s(C)^2-1\right]. Calibration and verification use independent source samples. Each seven-router batch has simultaneous failure budget 0.0050.005 (99.5% coverage); the two batches jointly have coverage at least 99%. At ρ0=0.05\rho_0=0.05, an upper limit at most 0.050.05 establishes coverage; a lower limit above 0.050.05 establishes outside status; otherwise the result is unresolved.

RouterCalibration estimate [CI]Independent verification [CI]Both decisions
S0 No change0 [0, .00738]0 [0, .00738]Covered
S1 Small global.01920 [.01182, .02658].01920 [.01182, .02658]Covered
S2 Small conditional.03115 [.02321, .03908].03108 [.02315, .03902]Covered
S3 Boundary conditional.04987 [.04161, .05814].04977 [.04150, .05804]Unresolved
S4 Strong conditional.16476 [.15445, .17507].16442 [.15408, .17475]Outside
S5 Large global.12000 [.11262, .12738].12000 [.11262, .12738]Outside
S6 Deterministic on 4% of sources.12524 [.09032, .16016].11206 [.07857, .14555]Outside

Conditional versus marginal change. For this task-independent uniform reference, with pˉs=Eμps(C)\bar p_s=\mathbb E_\mu p_s(C),

+4\sum_s\operatorname{Var}_\mu[p_s(C)].$$ S4 and S6 have essentially uniform aggregate template frequencies, yet exact catalog conditional radii **0.16518** and **0.12000**. Opposing task-group shifts cancel in S4's histogram. S6 is deterministic on 4% of sources, contributing $0.04\times3=0.12$. Aggregate frequencies miss this conditional variation. **How to measure the radius.** Correct logged propensities give $T_i=4\sum_s p_s(C_i)^2-1\in[0,3]$. They must describe the probabilities actually executed, including retries, truncation, and fallbacks; softmax scores do not suffice if serving uses argmax. Alternatively, with $L=16$ conditionally independent choices per source, use collisions: $$H_i=\binom{L}{2}^{-1}\sum_{a<b}(4\mathbf1\\{S_{ia}=S_{ib}\\}-1), \qquad \mathbb E[H_i\mid C_i]=T_i.$$ Treat each source and its choices as one independent cluster. The 120 overlapping pairs are not independent observations. Use the conservative range $H_i\in[-1,3]$, retain negative values in estimation, and intersect only the final valid radius interval with $[0,3]$. | Router | Collision estimate [simultaneous CI] | Decision | |---|---|---| | S2 | .03104 [.02525, .03684] | Covered | | S3 | .04962 [.04348, .05575] | Unresolved | | S4 | .16688 [.15882, .17493] | Outside | This three-target family has failure budget $0.005$; equal two-sided allocation uses $0.005/(2\times3)$ per target per tail. One choice per source without logged probabilities or further structure generally cannot identify the conditional radius. S3's exact radius is $0.05$: it belongs to the closed population class, while its interval cannot establish membership. In 200 repetitions per tested size, S2/S6 were always unresolved at $N=1{,}024$, and always covered/outside, respectively, at tested sizes $4{,}096$–$16{,}384$; S3 remained unresolved throughout. These are observed simulation frequencies. Changes to the source mixture, text within a view, or reference support require separate assessment. These results answer Q2: the conditional radius detects global, task-dependent, and concentrated rerouting, including shifts invisible to aggregate frequencies, and its intervals distinguish covered, outside, and unresolved policies.
AuthorsOpenReview ↗

Q3 — Linking Membership, Robust Certification, and Independent Validation

Thank you for suggesting an empirical assessment of the modeling assumption. This motivated us to connect membership calibration, robust certification, and a separately collected routed-risk measurement in one controlled study.

An observed routing policy can be tested for membership using a conditional-radius interval, and its risk consequence can be measured on an independent sample. The controlled simulation in Appendix J.2 demonstrates this assessment with synthetic loss catalog A and the frozen router π12\pi_{12} at temperature T=12T=12. It is a designed test, rather than a representative sample of naturally occurring LLM deployment shifts.

Three separate measurements. Calibration, robust audit, and direct validation share the same source law, labels, and four concrete views, but use separate samples.

MeasurementResultWhat it establishes
Membership calibrationRadius 0.00679, CI [0, 0.01435]Upper radius ρ+=0.01435<ρ0=0.05\rho^+=0.01435<\rho_0=0.05
Independent robust auditUA(0.05)=U_A(0.05)= −2.246 pp; UA(ρ+)=U_A(\rho^+)= −3.852 ppNegative worst-case upper endpoints
Independent routed-risk validationGap −7.193 pp, CI [−7.994, −6.392] pp; exact catalog gap −7.315 ppNegative policy-specific risk gap

Here pp denotes percentage points, and gaps are CFPT minus factual. The audit supplies upper bounds −0.0570583 on the reference risk gap and 0.0239404 on paired sensitivity, in risk units: UA(ρ)=−0.0570583+ρ⋅0.0239404.U_A(\rho)=-0.0570583+\sqrt{\rho\cdot0.0239404}. The same joint moment event controls every radius, so evaluating this bound at the calibrated upper radius preserves validity. On the joint calibration and audit event,

\Delta_{\pi_{12},A}\le U_A(\rho^+)<0.$$ Direct validation uses **32,768 IID source draws**, with one view sampled from the frozen router per draw. The paired 0–1 loss difference lies in $[-1,1]$. Its two-sided interval belongs to a **97.5% simultaneous family of nine routed-risk comparisons** and contains the exact catalog gap. This third sample was reserved for routed-risk estimation: it did not select the router, estimate audit moments, or produce the certificate. Calibration establishes membership; the audit bounds worst-case risk over the class; direct validation estimates this router's expected risk gap. Its strictly negative interval independently supports improvement for the measured policy. **Confidence ledger.** | Events included in the joint statement | Total failure budget | Joint coverage at least | |---|---|---| | This experiment's calibration family + audit moment family | $0.005+0.025=0.030$ | **97%** | | Those events + the nine-comparison routed-risk family | $0.005+0.025+0.025=0.055$ | **94.5%** | The union bound gives these guarantees without requiring event independence. They are conditional on the task, label, and support assumptions and cover only the listed families. Adding Comment 2's seven-router or collision results to a joint claim requires their additional failure budgets. **Assessment across observed shifts.** A deployment study should prespecify a representative sampling frame and weights for policy changes, check common tasks and labels, fixed concrete views, and reference support, then estimate each eligible policy's conditional radius with family-adjusted intervals. Report covered, outside, and unresolved outcomes, retaining both the total sampled denominator and the scope-eligible denominator. Report scope failures separately and do not count unresolved intervals as coverage. Independent routed-risk samples can then assess the consequences for the evaluated policies. That design estimates coverage for the specified population of policy changes. The A × T=12 result answers Q3 by connecting measured class membership, a negative robust endpoint, and an independently estimated negative policy-specific risk gap, with explicit confidence accounting.
Reviewer_riZAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies when a counterfactual post-training (CFPT) model can be reliably certified as better than its factual comparator under declared nuisance shifts. It formulates a paired certification interface based on shared-anchor loss differences, derives shared-anchor and generic finite-sibling certificates, provides finite-sample power conditions, and develops a pilot-plan-certify procedure that separates planning reliability from final audit safety. Experiments include controlled synthetic laws with known support, a prospective Qwen/ReCoRD audit, and diagnostic Qwen/MultiNLI and HANS/QA studies. The main message is that pairing and anchor reuse can reduce audit cost, while pilot planning and conservative bounds remain substantial practical costs.

Critical strengths

S1. A coherent paired-certification formulation.

Formulating the audit question as a shared-anchor paired loss difference and separating pairing from anchor reuse are consequential for the question of how much independent evidence is needed.

S2. Honest empirical reporting.

The controlled experiments report infeasible pilots, conservative budgets, and end-to-end success rates rather than only successful certifications, which strengthens credibility.

Critical weaknesses

W1. The main text is not sufficiently rigorous in presentation, which affects verifiability.

Here give a few examples.

  1. Figure 1 conflates operation arrows and data-flow arrows and does not show the training map from dataset D_{cf} to the candidate model;
  2. Equation numbering is inconsistent;
  3. Quite some notations are not defined in the main text.

This makes it difficult to follow the paper, confirm the compared objects, the data-use boundary, and the certificate pipeline, and therefore affects reliable assessment of the paper's central contribution.

W2. The “counterfactual” framing is not well aligned with the actual contribution.

The title and framing emphasize counterfactual post-training, but the operational object is a paired robust loss-difference certificate under covered nuisance shifts. The term “counterfactual” appears surprisingly sparingly in the main text relative to the title and framing; if this observation is correct, the paper should either justify the counterfactual terminology or reframe the contribution around paired certification. In addition, semantic validity of automatically generated counterfactuals is not verified; the Qwen/ReCoRD experiment is framed by the authors as establishing computability rather than a theorem-valid selection result.

Questions for authors

Q1. Please revise Figure 1 to clearly distinguish data nodes, operation nodes, and model/parameter nodes. And please state the meaning of every arrow. Please check the same problem throughout the paper.

Q2. Please unify equation numbering and define all symbols at first use in the main text. Could you provide a notation table?

Q3. Could you clarify the role of the word “counterfactual” in the main text? If it appears only rarely outside the title and appendix, please either justify the framing or retitle/reframe the paper around paired certification under covered shifts.

Q4. For automatically generated counterfactuals, how should one obtain or bound For automatically generated counterfactuals, how should one obtain or bound 𝜂_{sem} ? If an independent semantic audit is unavailable, is the certificate effectively vacuous for realistic CFPT settings?

6 comments
AuthorsOpenReview ↗

Q1 — Revised Figure 1: From Training to Independent Certification

Thank you for identifying the presentation issues that needed clarification. We have revised Figure 1 to make the sequence explicit: training → frozen models → independent paired audit → certification decision.

Nodes and arrows. The revised figure distinguishes data nodes, operation nodes, model/parameter nodes, and the final decision. Gray solid arrows carry data or computed quantities; blue dashed arrows carry model parameters. Specifically:

  • Factual data DFD_F enter factual training and produce the comparator wFw_F.
  • DFD_F enters counterfactual edit generation and produces DCF,hD_{\mathrm{CF},h}.
  • DFD_F and DCF,hD_{\mathrm{CF},h} enter CFPT training under policy hh and produce the candidate whw_h.
  • Shared initialization θinit\theta_{\mathrm{init}} enters both training operations. This is a matched experimental control; the certification result applies to arbitrary frozen model pairs.
  • Both frozen models enter their respective evaluation operations.
  • The same fresh audit clusters enter both evaluations.
  • The resulting paired loss differences enter construction of UhU_h, which determines the certification decision.

This explicitly supplies the dataset-to-candidate training map that was missing from the original diagram.

Observable audit inputs and independence. Each audit cluster contains (xik,yi)k=1K\\{(x_{ik},y_i)\\}_{k=1}^{K}, where ii indexes an independent anchor and kk indexes a nuisance view. The semantic task state CiC_i may be latent; the auditor observes the inputs and valid labels. Both models and the audit configuration are frozen before collecting final-audit data independently of training, model selection, and pilot planning.

The paired observation is

=\ell(w_h;x_{ik},y_i)-\ell(w_F;x_{ik},y_i).$$ Thus pairing means evaluating both models on the **same anchor, view, and label**. **Meaning of the decision.** Writing $R_\pi(w)$ for risk under the fixed task and valid-response laws and nuisance kernel $\pi$, the target is $$\Psi_h(\rho) =\sup_{\pi\in\mathcal E_\rho(\nu)} \\{R_\pi(w_h)-R_\pi(w_F)\\}.$$ With the declared family confidence allocation, $$\Pr\\{\forall h:\Psi_h(\rho)\le U_h\\}\ge1-\delta.$$ Consequently, $U_h<0$ certifies improvement throughout the covered shift class. Otherwise, the procedure retains the factual comparator. The optional independent pilot is omitted from the schematic and explained in the planning section. **Interpretation of the lower panels.** Panels (c)–(f) show exploratory MultiNLI diagnostics from 16 generation/training pools sharing an evaluation split. Their bands are pointwise $t$-intervals, and the label-disagreement panel uses cached automated judgments. These panels motivate the need for independent certification. The formal audit and the human semantic assessment reported below use their own sampling protocols. We have aligned the diagram, caption, and textual definitions so that the compared models, information flow, independence boundary, and certification decision can be checked directly.
AuthorsOpenReview ↗

Q2 — Unified Notation and Explicit Statistical Definitions

We have unified the notation throughout the main text and appendix, clarified first-use definitions, and added a notation guide in Appendix A.1. The revision distinguishes population targets, observable audit statistics, and uncertainty in estimating those statistics.

SymbolDefinition
wF,whw_F,w_h; h=1,…,Jh=1,\ldots,JFrozen factual comparator, frozen candidates, and declared comparison-family size.
C,Y,SC,Y,S; μ,q,ν\mu,q,\nuTask state, valid response, nuisance; task marginal, valid-response kernel, reference nuisance kernel.
zh(c,s,y)z_h(c,s,y)Labelled paired loss difference ℓwh(c,s,y)−ℓwF(c,s,y)\ell_{w_h}(c,s,y)-\ell_{w_F}(c,s,y).
gh(c,s)g_h(c,s)Label-averaged paired response EY∼q(⋅∣c)zh(c,s,Y)\mathbb E_{Y\sim q(\cdot\mid c)}z_h(c,s,Y).
Δν,h,Vh\Delta_{\nu,h},V_hReference mean Eμνgh\mathbb E_{\mu\nu}g_h and within-task sensitivity ECVar⁡S∣Cgh(C,S)\mathbb E_C\operatorname{Var}_{S\mid C}g_h(C,S).
Zik,h;Mi,h,Qi,hZ_{ik,h};M_{i,h},Q_{i,h}Observed paired difference; cluster mean and sensitivity statistic.
τK2,ΩK,ζK\tau_K^2,\Omega_K,\zeta_KVar⁡(M),Var⁡(Q),Cov⁡(M,Q)\operatorname{Var}(M),\operatorname{Var}(Q),\operatorname{Cov}(M,Q), respectively.
ρ,Ψh(ρ),Uh\rho,\Psi_h(\rho),U_hShift budget, worst-case population risk difference, and statistical upper endpoint.
N,K;r0,r1N,K;r_0,r_1Final-audit anchor/sibling counts and the two pilot-block sizes.
e,δ,δplan,pe,\delta,\delta_{\mathrm{plan}},pPer-event failure budget, family safety budget, planning-failure budget, and target final power.
B,η,ηsemB,\eta,\eta_{\mathrm{sem}}Loss bound, positive dual variable, and semantic contamination rate.

Consistent names. We use whw_h for the CFPT candidate and wFw_F for its factual comparator. The valid-response kernel is qq. We distinguish the conditional density aπ=dπ/dνa_\pi=d\pi/d\nu from pilot sizes and confidence radii, and the conditional paired mean mZ(c)=E[Z∣C=c]m_Z(c)=\mathbb E[Z\mid C=c] from the task law μ\mu. Policy sets, dual grids, and planning menus have distinct symbols. KL divergence is consistently written as DKLD_{\mathrm{KL}}.

The statistical interface is now explicit. For even KK, with conditionally IID nuisance siblings,

Q_i=\frac1K\sum_{k=1}^{K/2} (Z_{i,2k-1}-Z_{i,2k})^2.$$ Their global ranges are $M_i\in[-B,B]$ and $Q_i\in[0,2B^2]$. Under deterministic valid labels, $\mathbb E M=\Delta_\nu$ and $\mathbb E Q=V$. Exact view enumeration instead uses $$Q_i^{\mathrm{enum}} =\operatorname{Var}_{S\sim\nu(\cdot\mid C_i)}z(C_i,S,Y_i) \in[0,B^2].$$ The revision distinguishes these observation protocols and uses the corresponding range bounds. With one random valid label shared by siblings and $Y\perp S\mid C$, the observable sensitivity is $$V_{\mathrm{obs}} =\mathbb E_{C,Y}\operatorname{Var}_{S\mid C}z(C,S,Y).$$ By Jensen's inequality, $$V_{\mathrm{obs}}\ge V, \qquad V=\mathbb E_C\operatorname{Var}_{S\mid C}g(C,S).$$ An upper bound on $V_{\mathrm{obs}}$ therefore conservatively bounds the sensitivity in the transfer certificate. The deterministic-label variance and power identities retain their stated assumptions. **Equation references.** We use automatic equation numbering and give each separately referenced variance identity its own unique label. The notation guide covers the paired and generic endpoints, power conditions, and pilot quantities. These changes make the compared objects and sampling units explicit, including the distinction between **final-audit safety, pilot-planning reliability, and conditional final power**.
AuthorsOpenReview ↗

Q3 — Connecting Counterfactual Training to Independent Audits

The paper's contribution is paired certification and audit planning under covered nuisance shifts, with counterfactual post-training as the motivating application. We now state this distinction explicitly.

Role of “counterfactual.” A counterfactual intervention changes nuisance SS while preserving task CC and its valid-response law q(⋅∣C)q(\cdot\mid C). This produces matched task–view pairs on which the factual and CFPT models can be compared. The covered class fixes the task marginal μ\mu and response kernel qq, and changes only the conditional nuisance distribution.

For a reference kernel ν\nu, a covered kernel π\pi has conditional density aπ=dπ/dν≥0a_\pi=d\pi/d\nu\ge0 satisfying

\mathbb E_{\mu\nu}(a_\pi-1)^2\le\rho.$$ For the label-averaged paired response $g_h$, the certificate controls $$\Psi_h(\rho) =\sup_{\pi\in\mathcal E_\rho(\nu)}\mathbb E_{\mu\pi}g_h \le \Delta_{\nu,h}+\sqrt{\rho V_h}.$$ Thus the counterfactual construction specifies the matched intervention structure; the statistical contribution establishes when a gain can be certified over the declared shift class. **How training enters.** CFPT produces a candidate model. An independent audit then determines whether that frozen candidate improves upon its factual comparator. The training generator and audit intervention law may differ: training quality affects the resulting gain, while certification validity depends on the audit's sampling, coverage, and semantic conditions. **New training-to-audit evidence (Appendix I.1).** We trained Qwen2.5-0.5B-Instruct with LoRA under a controlled MultiNLI protocol. After exploratory design runs, settings were fixed for eight new seeds, 5101–5108. Four conditions share 900 training anchors, initialization, and 500 Adam updates: - F: factual training; - A: augmentation-only training; - K: augmentation with consistency regularization; - R: augmentation with loss-variance regularization. Here K names a training condition; the mathematical $K$ denotes sibling count. A/K/R share training-view probabilities $(0.82,0.10,0.06,0.02)$ and augmentation weight $0.70$. The consistency and variance weights are $0.40$ and $1.50$. Training uses half-Brier loss; the primary independent audit uses 0–1 error. R regularizes the single model's training-view loss variance. Models are frozen before auditing, and audit labels do not update them. At $\rho=0.05$, the eight-seed mean full-catalog worst-case differences relative to F are: | Training condition | Mean $\Psi$, percentage points | |---|---:| | A: augmentation only | −3.707 | | K: consistency regularization | −4.224 | | R: variance regularization | −4.681 | R has lower $\Psi$ than K in every seed, with a mean difference of approximately −0.458 pp. These are matched-control results for the declared four-view benchmark-label target. The frozen R-versus-F comparisons receive the independent certificates reported in Q4. The revised framing therefore connects **task-preserving interventions → trained candidates → independent paired certification**, while identifying paired certification, rather than a particular training algorithm, as the methodological contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.