acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Best Available Cause: Calibrated Verification and Triage under Hypothesis-Space Inadequacy

Abstract

From clinical practice to LLM reasoning, causal diagnosis commits to the top-ranked candidate from a set of explanations. When the true cause is absent from that set, even the best candidate can be wrong. We call the absence of any task-compatible candidate hypothesis-space inadequacy, and ranking alone cannot detect it. Diagnosing it requires separating it from selection failure, where an adequate space still yields a wrong pick. We propose Causal Adequacy Verification with Evidence-guided Action and Triage (CAVEAT) with a dual-null test design, one test and one repair for each failure type. The core idea is to make inadequacy testable without knowing the true cause, by defining compatibility through the consequences the task depends on and comparing every candidate's predictions with the observed evidence. The first test assesses support for the space and requests revision when support is insufficient; the second checks the selected candidate and prompts distinguishing evidence when the choice remains unresolved. Among task-incompatible selections, calibration limits how often commitment occurs, assuming past and new failures are exchangeable under the same workflow. On a clinical benchmark, retrieved candidate lists omit the recorded diagnosis in 78% of cases. Across physical, biological, and clinical tasks, CAVEAT reduces wrong commitments, with retained usefulness depending on the evidence. In controlled LLM reasoning, test-guided repair yields more correct commitments than either fixed repair, even with imperfect candidate supply.

Primary area:
causal reasoning
Keywords:
causal inference, causal reasoning, hypothesis-space adequacy, selective prediction, conformal inference, verification and triage, root-cause analysis, evidence acquisition, large language models, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KNt3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces what it calls "hypothesis-space inadequacy": A_ϵ (O,Q)=∅, meaning that none of the currently available hypotheses is sufficiently compatible with the observed consequences. The key idea is to do the following: (a) Score every candidate against observed consequences. (b) Test whether any candidate appears adequate. (c) Test whether the selected candidate is trustworthy. (d) If either test fails, refuse commitment and request either: revise the hypothesis set or get more evidence. That is essentially an open-set verification layer built on conformal calibration.

The paper thus is mainly a conformal-verification framework for detecting when the currently available hypothesis set may be fundamentally inadequate, and for routing between abstention, revision, and additional evidence gathering. That is a legitimate contribution, but much narrower than the paper's framing suggests.

Critical strengths

The paper separates Selection failure ("The correct explanation is represented but not chosen.") from Hypothesis-space inadequacy ("No represented explanation is task-compatible.")

Other literature addressing such tasks usually assumes a complete state space. For example, the medical diagnosis literature usually assumes the true fault belongs to the fault universe. This paper explicitly addresses when neither assumption holds. That is actually useful. The novelty is therefore not sensing, but is "calibrated detection that the current explanatory universe may itself be wrong."

The second contribution is the use of causality to integrate with information-theoretic concepts. Causality can bring key insights into such decision making processes.

Critical weaknesses

The primary issue with this paper is that it over-inflates what is claims to do, and in fact its contributions are quite narrow: mainly a conformal-verification framework for detecting when the currently available hypothesis set may be fundamentally inadequate.

Within the broader context of the defined task, the paper omits a great deal. The following comments examine what seems to be missing, relative to the other approaches to solving this task.

This paper makes a set of assumptions totally different to those made in similar active sensing contexts, viewed from a control perspective. Because the paper completely separates diagnosis, evidence gathering, decision making, this seems odd, since a control theory typically couples them. A control theorist would formulate something like π^*=arg⁡min┬π E["cost" ] over: sensing actions diagnosis beliefs interventions future observations simultaneously. Examples of the areas taking such a perspective include POMDP diagnosis, dual control, Bayes-adaptive control, active fault diagnosis and information-state control. In all of these formulations, information gathering is valuable only because it improves future decisions.

The paper never solves such an optimization problem. Instead it proposes a cycle consisting of: verify, perhaps query, perhaps revise, perhaps commit. The query policy is arg⁡max┬q U(q)-λc(q), but this is only a local heuristic for choosing among candidate queries, not a global optimal sensing-and-decision problem. From a control viewpoint this is extremely weak.

Why active-sensing researchers may be unimpressed: The paper repeatedly discusses acquiring additional evidence. At first glance this sounds like active sensing. However active sensing normally solves: q^=arg⁡max⁡E ["future utility" ] or q^=arg⁡max⁡"VoI" (q) under costs. The present paper does not derive any optimal acquisition policy. Instead: query only happens after verification, queries are selected from a fixed set, the utility function is delegated to the user, no optimality theorem exists. Thus from an active-sensing perspective this is almost a post-processing wrapper around an existing diagnostic system.

I think the paper's most significant conceptual weakness concerns notions of Value of Information (VoI). The authors state that seek distinguishing evidence among unresolved candidates. But this is exactly the classical Value of Information problem. Suppose we have candidate causes h_1,…,h_k and possible measurements q_1,…,q_m. A control/decision-theoretic formulation asks: "VoI" (q)=E["decision quality after " q]-E["decision quality before " q]. The present paper never computes this quantity. Instead it introduces a generic separation utility U(q) and leaves it unspecified. That means that the following issues are not addressed: (a) no optimal evidence theory, (b) no information-theoretic characterization, (c) no regret-optimal sensing strategy. Consequently the active-sensing component is much weaker than decades-old VoI formulations.

The sensing model is surprisingly unrealistic: The paper assumes candidate predictions can be compared against observed evidence through discrepancy measures. Effectively: d(P_h,P ̂^* ) drives everything. But where does P ̂^* come from? The paper largely treats it as observable or estimable from evidence. This ignores major diagnosis issues: sensor noise, sensor bias, adversarial corruption, missing observations, partial observability. A diagnosis researcher would immediately ask: What if the evidence itself is inadequate? The paper studies only: Is the hypothesis space inadequate? Those are not the same problem. In diagnosis theory there is usually a three-way ambiguity: wrong hypothesis, inadequate hypothesis space, inadequate observations. Only (1) and (2) are represented here.

The paper is surprisingly disconnected from diagnosability theory: The paper's central object is A_ϵ (O,Q). Compatibility depends on agreement with selected consequences. In classical diagnosis theory we would ask if (a) faults can be distinguished? (b) does evidence induce an equivalence class? (c) what is the residual ambiguity? (d) what observation partition results? The paper never develops anything resembling a diagnosability partition. In fact the uncertainty set S(O) is effectively an equivalence class of indistinguishable explanations. But the authors never analyze: partition structure, identifiability conditions, observability bounds, distinguishability guarantees. This is a missed opportunity.

The action-theoretic section feels bolted on: I was particularly unconvinced by the regret decomposition R_D=R_V+R_S, where: R_V: hypothesis-space inadequacy regret, and R_S: selection regret. Mathematically this is true because the authors define R_D=J_sel-J^* and add/subtract an intermediate term. But this is not a deep result. It is essentially a bookkeeping identity. A control theorist would expect the following issues to be addressed: (a) optimal policy characterization, (b) Bellman equation, (c) regret bounds, (d) information-state evolution. None of those appear.

Questions for authors
  1. Can you precisely address the VoI issues that you contribute to?

  2. What is the objective that you are trying to optimize? Can this be precisely formulated?

  3. How does causality improve the proposed framework, when it seems you solve exactly the classical Value of Information problem?

9 comments
AuthorsOpenReview ↗

Response to Reviewer KNt3 (part 1/3)

Thank you for pressing on the connection between verification and decision-focused acquisition. CAVEAT's goal is to decide whether the available explanations can support the task before committing, to separate that failure from a wrong selection, and to route the matching repair under calibrated limits. The revision states this as a constrained objective over the complete workflow (Eq. 6) and solves its terminal rule exactly (Eq. 27). A sensing planner answers a different question, which evidence to acquire next, and we do not solve that sequential problem. The two are complementary. Proposition 3 gives an information limit that binds every acquisition policy on the admissible menu, optimal or not. Table 25 shows that inside the strongest planner we built, expanded-model VoI, failure-specific calibration still lowers inadequate acceptance from 12.60% to 9.92% at unchanged yield. The planner chooses the evidence. The verifier decides what it licenses. Statements are written out below. Pages refer to the clean revised PDF.

Q2 and coupling: the objective

Let ϖ\varpi be a complete workflow (proposal, sensing, revision, selection, stopping), CϖC_\varpi a task-compatible terminal selection, ΓN\Gamma_N its verifier calibrated on NN independent historical incidents, and cϖc_\varpi the acquisition cost. Equation (6) (p. 5) is max⁡ϖ  Pr⁡(Cϖ,  ΓN=Commit)−λ E[cϖ]s.t.Pr⁡(ΓN=Commit∣H0,ϖV)≤αV,    Pr⁡(ΓN=Commit∣H0,ϖS)≤αS.\max_{\varpi}\;\Pr(C_\varpi,\;\Gamma_N=\mathrm{Commit})-\lambda\,\mathbb{E}[c_\varpi]\quad\text{s.t.}\quad \Pr(\Gamma_N=\mathrm{Commit}\mid H^V_{0,\varpi})\le\alpha_V,\;\; \Pr(\Gamma_N=\mathrm{Commit}\mid H^S_{0,\varpi})\le\alpha_S . V denotes inadequacy and S a wrong selection within an adequate space. The nulls are indexed by ϖ\varpi because repairs change the terminal failure populations as well as the useful decisions, so modular implementation does not imply mathematical independence. The guarantees are marginal within each failure population, not per-incident certificates. A constrained VoI formulation can express this objective. Our question is what evidence and calibration it requires when the space can be inadequate. The original separation utility is a heuristic acquisition rule, not a solution of the global problem, and the paper now says so (p. 5 lines 236–246).

Q3 and diagnosability: what causality adds

Proposition 3 (p. 7; construction and proof in Appendix E.1 pp. 40–41). Let Y=1{U≤σ(aX+bZ)}Y=\mathbf 1\{U\le\sigma(aX+bZ)\} with UU uniform and σ\sigma logistic. Both represented candidates have b=0b=0 and differ only in a∈{−2,2}a\in\{-2,2\}. Compare an adequate population (b=0b=0) with an inadequate one (b∈{−d,d}b\in\{-d,d\} equally likely, d>0d>0). With total-variation discrepancy and tolerance ϵ<σ(d)−1/2\epsilon<\sigma(d)-1/2 the second population is inadequate. If aa and the initial information have the same joint law in both populations, every finite-horizon adaptive policy restricted to do(X=x,Z=0)\mathrm{do}(X=x,Z=0) has identical transcript laws in the two populations. Hence Pr⁡(Commit∣H0V)≤αV\Pr(\mathrm{Commit}\mid H_0^V)\le\alpha_V forces Pr⁡(correct commitment∣(H0V)c)≤αV\Pr(\text{correct commitment}\mid (H_0^V)^c)\le\alpha_V.

In words. Interventions on XX distinguish the listed candidates but cannot reveal the ZZ effect both omit, because under Z=0Z=0 it never enters the response. Any planner on that menu sees the same data whether or not the space is adequate. Interventions involving ZZ expose the omission. With mm responses to do(X=0,Z=1)\mathrm{do}(X=0,Z=1) and KK positives, K∣(H0V)c∼Bin(m,12),K∣H0V∼12Bin(m,σ(d))+12Bin(m,σ(−d)).K\mid(H_0^V)^c\sim\mathrm{Bin}(m,\tfrac12),\qquad K\mid H_0^V\sim\tfrac12\mathrm{Bin}(m,\sigma(d))+\tfrac12\mathrm{Bin}(m,\sigma(-d)). For m=1m=1 the laws agree. For m≥2m\ge2 they differ. At m=12m=12, d=2d=2 their total variation is about .809. Equal single-response marginals do not imply equal joint transcript laws. This identifies a limit of the observation menu before any optimization over it. Lemma 2's zero-value condition requires a common conditional kernel, not equal marginals (p. 18).

Q1: beyond closed-candidate VoI

Across 96 configurations where the represented candidates predict the same response law, closed-candidate EVSI is zero analytically (max computed ∣⋅∣=3.33×10−16|\cdot|=3.33\times10^{-16}), yet an agreeing query attains the largest ideal constrained frontier in five of eight conditions, and in the heterogeneous condition querying zz raises that frontier by 26.21 points over no query (Appendix E.12 p. 53). These are model checks, not achieved gains. Part 2 gives the implemented policies.

AuthorsOpenReview ↗

Response to Reviewer KNt3 (part 2/3)

Q1 continued: implemented acquisition

Screen-first policy. The first query maximizes candidate-prediction sensitivity to Gaussian coefficient perturbations (τ=1\tau=1, 400 draws) minus between-candidate spread. If both gates pass, commit. Otherwise V failure requests the screening query and S failure the candidate-separating query. Each commitment opportunity spends half the level, with separate stage banks. Query choice uses no test labels, but calibration still needs historical failure labels. Heterogeneous condition, six repetitions:

PolicyCorrect (%)Wrong (%)Queries
Closed-candidate greedy VoI5.075.071.00
Fixed screening then discrimination39.144.952.00
Screen-first41.283.961.80
Oracle-selected fixed pair43.444.962.00

The oracle row is an evaluated fixed pair, not an LP bound. Screen-first uses a sensitivity proxy, not oracle multipliers. It loses to the fixed pair in the noisy condition (5.89% vs 12.54%). Table 35 (p. 54) has all eight conditions; the policy is specified on p. 53 lines 2847–2861.

A stronger planner acquires better and we say so. Expanded-model VoI fits a four-component product-binomial mixture for inadequate mechanisms from historical branch outcomes and solves the two-query commit/defer problem by finite-horizon dynamic programming. Heterogeneous condition, twelve repetitions:

PolicyCorrect (%)Wrong (%)Queries
Expanded-model VoI43.853.631.64
Gate-directed sensing42.484.041.78

It values screening evidence because its model represents inadequate mechanisms (Table 24 p. 45).

Protection isolated inside that planner. Hold its acquisition and learned scalar fixed and change only the calibration:

CalibrationV acceptance (%)Correct (%)
Pooled failures12.6072.37
Separate V/S banks9.9272.27

Twelve-repetition means, less-inadequacy condition; Table 25 (p. 46) has all conditions. Null-score calibration needs representative failure labels and null-specific exchangeability, not a correct planner posterior. Table 31 (p. 50) repeats this for six replacement workflows. Table 4B (p. 9) and Figure 8 (p. 51) give the real-telemetry trade-off.

Control characterization and regret

Regret is an identity. With J⋆(u)J^\star(u) the harm of action uu under the operative mechanism, uhu_h the action candidate hh induces, J⋆=min⁡uJ⋆(u)J^\star=\min_u J^\star(u), JH⋆=min⁡h∈H(O)J⋆(uh)J^\star_H=\min_{h\in H(O)}J^\star(u_h) and Jsel=J⋆(uh^)J_{\mathrm{sel}}=J^\star(u_{\hat h}), Proposition 5 (Appendix A.8 p. 18) is RD:=Jsel−J⋆=(JH⋆−J⋆)+(Jsel−JH⋆)=RV+RSR_D:=J_{\mathrm{sel}}-J^\star=(J^\star_H-J^\star)+(J_{\mathrm{sel}}-J^\star_H)=R_V+R_S. The proof adds and subtracts JH⋆J^\star_H. Nothing rests on it as a sensing-optimality theorem.

What Proposition 2 proves. Let wV=Pr⁡(H0V)w_V=\Pr(H_0^V), wA=1−wVw_A=1-w_V, βV\beta_V the routing gate's attainable level, ϕV=Pr⁡(pV>αV∣(H0V)c)\phi_V=\Pr(p_V>\alpha_V\mid(H_0^V)^c) its false-rejection rate on adequate spaces, YR,YQ,YrouteY_R,Y_Q,Y_{\mathrm{route}} correct commitment under always-revise, always-query and routing, ΔV=E[YR−YQ∣H0V]\Delta_V=\mathbb{E}[Y_R-Y_Q\mid H_0^V], ΔA=E[YQ−YR∣(H0V)c]\Delta_A=\mathbb{E}[Y_Q-Y_R\mid(H_0^V)^c]. Then (p. 6; proof Appendix A.9 pp. 18–19) E[Yroute]−max⁡{E[YR],E[YQ]}  ≥  min⁡{wVΔV,  wAΔA}−wVβV−wAϕV.\mathbb{E}[Y_{\mathrm{route}}]-\max\{\mathbb{E}[Y_R],\mathbb{E}[Y_Q]\}\;\ge\;\min\{w_V\Delta_V,\;w_A\Delta_A\}-w_V\beta_V-w_A\phi_V . The proof couples the policies on the same incident. With BB the revision verdict, Yroute=BYR+(1−B)YQY_{\mathrm{route}}=BY_R+(1-B)Y_Q and Ymatch=1H0VYR+1(H0V)cYQY_{\mathrm{match}}=\mathbf 1_{H_0^V}Y_R+\mathbf 1_{(H_0^V)^c}Y_Q. Then E[Ymatch]−E[YR]=wAΔA\mathbb{E}[Y_{\mathrm{match}}]-\mathbb{E}[Y_R]=w_A\Delta_A and E[Ymatch]−E[YQ]=wVΔV\mathbb{E}[Y_{\mathrm{match}}]-\mathbb{E}[Y_Q]=w_V\Delta_V, and routing differs from the matched outcome only on the disagreement event, whose probability is at most wVβV+wAϕVw_V\beta_V+w_A\phi_V by Theorem 1. No independence of routing errors and repair success is assumed.

AuthorsOpenReview ↗

Response to Reviewer KNt3 (part 3/3)

Control characterization continued

The terminal problem is solved exactly. Fix a query qq and finite transcript FF. Let uq(f)=Pr⁡q(F=f,Cq)u_q(f)=\Pr_q(F=f,C_q), bjq(f)=Pr⁡q(F=f,H0j)b_{jq}(f)=\Pr_q(F=f,H_0^j), νjq=∑fbjq(f)\nu_{jq}=\sum_f b_{jq}(f). The best randomized terminal rule ϕ(f)∈[0,1]\phi(f)\in[0,1] solves

Bq=max⁡0≤ϕ≤1∑fuq(f)ϕ(f)subject to∑fbjq(f)ϕ(f)≤αjνjq(j∈{V,S})B_q=\max_{0\le\phi\le1}\sum_f u_q(f)\phi(f)\qquad\text{subject to}\qquad\sum_f b_{jq}(f)\phi(f)\le\alpha_j\nu_{jq}\quad(j\in\lbrace V,S\rbrace)

and by finite LP duality

Bq=min⁡ηV,ηS≥0{∑jηjαjνjq+∑f[uq(f)−ηVbVq(f)−ηSbSq(f)]+}B_q=\min_{\eta_V,\eta_S\ge0}\Big\lbrace\sum_j\eta_j\alpha_j\nu_{jq}+\sum_f\big[u_q(f)-\eta_Vb_{Vq}(f)-\eta_Sb_{Sq}(f)\big]_+\Big\rbrace

(Eq. 27 p. 41). The multipliers price the two risks separately. In the archived extension they are (1,0)(1,0) for query xx and (.768,7.354)(.768,7.354) for zz (p. 53).

The comparator's Bellman recursion does not carry the budgets. It solves the two-query commit/defer problem by dynamic programming under its fitted model (reward +1+1, −4-4, cost .02.02). Its state is the model posterior. The two risk budgets are not part of that state. Replacing its native terminal decision by calibrated verification changes the optimization problem, so its optimality does not transfer to the combined rule (Table 24 p. 45; Appendix E.5).

Information state. Our gate statistics are ranks in the two null banks, pj(O)=(1+∑i=1nj1{Aj(Oij)≤Aj(O)})/(nj+1)p_j(O)=\big(1+\sum_{i=1}^{n_j}\mathbf 1\lbrace A_j(O_i^j)\le A_j(O)\rbrace \big)/(n_j+1), with a marginal guarantee under exchangeability within each failure population. They are verification feedback, not a sufficient predictive statistic. The predictive law is fitted separately.

Finite history

Conditional on bank sizes nV,nSn_V,n_S with IID draws and an independent test incident, define Rj(f)=Pr⁡(Aj≤Aj(f)∣H0j)R_j(f)=\Pr(A_j\le A_j(f)\mid H_0^j), kj=⌊αj(nj+1)⌋k_j=\lfloor\alpha_j(n_j+1)\rfloor, $\Psi(n,k,r)=\Pr\lbrace \mathrm{Bin}(n,r)\le k-1\rbrace $ and u(df)=Pr⁡(F∈df,C)u(df)=\Pr(F\in df,C). Expected correct two-gate acceptance is exactly

JnV,nS(2)=∫Ψ(nV,kV,RV(f))Ψ(nS,kS,RS(f))u(df)J^{(2)}_{n_V,n_S}=\int\Psi(n_V,k_V,R_V(f)) \Psi(n_S,k_S,R_S(f)) u(df)

(Eq. 28 p. 42): passing gate jj means at most kj−1k_j-1 bank scores lie below the test score, a binomial count, independent across gates given the transcript. Proposition 6 (p. 42): with mj=⌈1/αj⌉−1m_j=\lceil1/\alpha_j\rceil-1 and NN historical incidents with null probabilities νj\nu_j,

JN≤Pr⁡(C)Pr⁡(NV≥mV,NS≥mS)≤Pr⁡(C)min⁡jPr⁡{Bin(N,νj)≥mj}.J_N\le\Pr(C) \Pr(N_V\ge m_V,N_S\ge m_S)\le\Pr(C)\min_{j}\Pr\lbrace \mathrm{Bin}(N,\nu_j)\ge m_j\rbrace .

At α=.10\alpha=.10 a bank needs nine failures before any commitment. RCAEval final banks average 12.9 V and 14.4 S cases; splitting them for level selection leaves under nine V cases in 65 of 80 fold banks (Appendix E.9 p. 51). This is why a better selector can lower verified yield and why we report fixed levels.

Observation noise, bias, missingness, corruption

Section 3 (p. 4 lines 167–175) now specifies how P^⋆\widehat P^\star is obtained, with measurement and collection inside the frozen workflow.

Noise raises the evidence needed, computably. In the Proposition 3 construction, let each response be replaced by a fair bit with probability η<1\eta<1 and put δ=(1−η)(σ(d)−12)\delta=(1-\eta)(\sigma(d)-\tfrac12). With nn responses, K∣A∼Bin(n,12)K\mid A\sim\mathrm{Bin}(n,\tfrac12) and K∣V∼12Bin(n,12+δ)+12Bin(n,12−δ)K\mid V\sim\tfrac12\mathrm{Bin}(n,\tfrac12+\delta)+\tfrac12\mathrm{Bin}(n,\tfrac12-\delta). The test $\psi_n=\mathbf 1\lbrace \lvert K/n-\tfrac12\rvert\le\delta/2\rbrace $ has Pr⁡V(ψn=1)≤e−nδ2/2\Pr_V(\psi_n=1)\le e^{-n\delta^2/2} and Pr⁡A(ψn=1)≥1−2e−nδ2/2\Pr_A(\psi_n=1)\ge1-2e^{-n\delta^2/2} by Hoeffding, so n≥⌈2log⁡(1/α)/δ2⌉n\ge\lceil2\log(1/\alpha)/\delta^2\rceil suffices. At d=2d=2, α=.10\alpha=.10 this is 32 at η=0\eta=0 and 89 at η=.4\eta=.4 (Appendix E.1 p. 40).

Bounded score uncertainty gives a conservative gate. If, conditional on a jj-null test incident, all historical statuses and latent scores lie in their intervals jointly with probability at least 1−δj1-\delta_j, then the conservative rank pj=(1+dj(U∗j)+oj(U∗j))/(1+∣Dj∣+oj(U∗j))p_j=\big(1+d_j(U^j_{\ast})+o_j(U^j_{\ast})\big)/\big(1+\lvert D_j\rvert+o_j(U^j_{\ast})\big), with dj,ojd_j,o_j counting definite and optional historical lower bounds below the test upper bound, satisfies $\Pr(\mathrm{Commit}\mid H_0^j)\le\min\lbrace 1,\alpha_j+\delta_j\rbrace $ (Appendix E.4 pp. 43–44). Pointwise intervals alone do not suffice; δj\delta_j is not estimated from the discrepancy. Unknown bias or corruption can be observationally identical to omission, so a failed gate requests reconsideration of the workflow, not proof that the cause is absent.

Closing

The additions address what you asked: which observations can expose an omission (Proposition 3), when repair helps (Proposition 2), which decisions finite history can release (Eq. 28, Proposition 6), and how noisy or bounded evidence enters the guarantee (E.1, E.4). They complement a stronger planner; we do not claim a general optimal-control solution. If Eq. (6) and the isolation in Table 25 address the coupling concern, we would value knowing what remains so we can respond before the discussion closes. We would be grateful for your reassessment on that basis.

Reviewer_TttzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAVEAT, a framework for distinguishing hypothesis-space inadequacy from selection failure in causal reasoning. It employs two separately calibrated conformal tests to determine whether to accept a selected hypothesis, revise the candidate space, or acquire additional evidence. The authors provide theoretical guarantees on false commitments and evaluate the framework across synthetic causal models, physical and biological intervention datasets, clinical records, and LLM-based reasoning tasks.

Critical strengths

The paper provides a formal formulation of two distinct failure types and evaluates the proposed verification framework across several datasets. The inclusion of theoretical analysis and controlled experiments helps clarify the assumptions and limitations of the approach, although these do not establish a clear methodological advance over existing methods.

Critical weaknesses

1. The methodological novelty is limited. The main theoretical guarantees largely follow from established conformal testing and classical statistical results. In particular, Theorem 1 follows from standard conformal validity and the decomposition of incompatible selections into two failure types. It remains unclear what fundamentally new theoretical insight CAVEAT provides beyond combining existing verification and abstention mechanisms.

2. The reported safety improvements are largely achieved through excessive abstention. On LiveClin (Table 4), CAVEAT makes only 1 correct commitment out of 171 cases, abstaining on the remaining 170. Ordinary conformal prediction achieves 2 correct commitments with the same zero-error rate. Therefore, the proposed method does not demonstrate a convincing safety–utility advantage over a substantially simpler baseline.

3. The experimental evaluation relies heavily on artificially constructed hypothesis-space inadequacy. As acknowledged in Section 5.1, all benchmarks except LiveClin introduce inadequacy by deliberately removing the true mechanism or intervention family. Detecting omitted candidates from a predefined library is substantially different from identifying genuinely missing causal explanations in open-world settings. Consequently, the experiments do not sufficiently support the broader claims about practical causal diagnosis.

4. The framework relies on strong calibration assumptions that substantially limit its practical applicability. It requires historical incidents labeled by failure type, potentially demanding knowledge of the underlying causal mechanisms. Furthermore, changes to candidate generation, retrieval, or querying require recalibration. These requirements raise concerns about applicability to evolving LLM-based reasoning systems, particularly when previously unseen causes are encountered.

Questions for authors

1. Given that ordinary conformal prediction achieves the same zero-error rate with more correct commitments on LiveClin, what concrete advantage does CAVEAT provide over simpler calibrated abstention methods at matched risk and coverage levels?

2. Can the authors demonstrate that CAVEAT remains effective when hypothesis-space inadequacy arises naturally rather than through the deliberate removal of known mechanisms?

3. How can the required failure-type-specific calibration data be obtained in realistic open-world settings where the missing causal mechanisms may be previously unknown?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tttz (part 1/2)

Thank you for asking us to identify the advance beyond standard calibration and reduced commitment. We agree that the calibration machinery is established. The revision makes explicit what the framework adds, why pooled protection does not settle the task, and where that distinction improves decisions. The main entry points are Section 4 (pp. 5–7), Table 4B (p. 9), and Findings 3–4 (p. 10). All locations refer to the clean revised PDF.

W1: contribution beyond the conformal rank argument

The calibration step is Mondrian, label-conditional calibration with the failure type as the label, as stated in Appendix B.4 (p. 21). We do not claim a new conformal theorem. The gap is that overall error control does not answer whether any candidate serves the task, what evidence can expose a shared omission, or which repair can restore a useful decision. Our contribution has three parts:

  1. The verification object. Task-relative adequacy is defined through candidate predictions and task-relevant consequences, rather than ranking alone. The test uses incident evidence without knowing the new incident's cause; historical calibration still requires adjudication (Definition 1, p. 3; Section 3.1, p. 4).
  2. Evidence and repair coupled to the two failures. We characterize when revision/query routing improves recovery and when the admissible interventions cannot expose an omission (Propositions 2–3, pp. 6–7).
  3. The finite-history limit. Better selection can leave too few selection failures to calibrate productive commitment (Eq. (28), Proposition 6, p. 42).

These conclusions are not supplied by the rank-validity argument alone:

ResultInsight beyond the validity proof
Proposition 3: causal observation limitation (p. 7; Appendix E.1, pp. 40–41)Distinguishing the listed candidates can miss what they all leave out. In the construction, every finite adaptive policy using the restricted menu observes the same transcript law in adequate and inadequate populations; a different intervention exposes the omission.
Proposition 2: coupled repair comparison (p. 6)Routing helps when matching the repair to the failure gains more than routing mistakes lose. The bound compares against both fixed repairs without assuming independent routing errors and repair success; it explains favorable regimes and ties.
Eq. (28), Proposition 6: finite-history yield (p. 42)A better selector can leave less failure history to calibrate against. Under independent IID null banks and an independent test incident, exact expected two-gate yield depends on score laws and bank sizes.

The complete-workflow objective and terminal LP connect these requirements to correct commitment under separate failure limits (Eq. (6), p. 5; Eq. (27), pp. 41–42). They use classical tools to characterize when adequacy-sensitive verification and repair change decisions.

W2/Q1: an advantage beyond withholding more

RCAEval: more useful commitment at comparable inadequacy risk. Real-system telemetry under injected faults, with retrieval-produced omissions, gives the following mean results per 120 test incidents. V denotes inadequacy; S denotes wrong selection within an adequate space. Pooled is a scalar failure test, distinct from LiveClin's prediction-set CP.

Fixed ruleCorrect / 120Wrong / 120V acceptance (%)S acceptance (%)
Pooled .0516.131.288.321.82
Pooled .1027.852.3816.222.87
Dual .10/.3028.381.737.745.67

This gives more correct commitments at approximately matched V risk, or lower V acceptance at similar yield. It does not match both risks or total error. The pooled-plus-space rule is close to dual, and this is not a bank-only isolation. These are 40 overlapping three-way splits of 360 injected-fault incidents; Table 4B (p. 9), Table 32 (p. 50), and Figure 8 (p. 51) report the comparison and full curves.

AuthorsOpenReview ↗

Response to Reviewer Tttz (part 2/2)

W2/Q1 continued: pooled protection and stronger components

Why the pooled guarantee is insufficient for the stated target. A pooled guarantee controls acceptance averaged over its failure mixture; it does not impose the same limit within each failure population. On RCAEval, pooled .10 accepts 16.22% of inadequate lists versus 7.74% for dual at similar correct yield. The six-workflow study exposes the same issue for selection failures:

Learned routerWorst S acceptance: pooled / separate (%)Mean correct: pooled / separate (%)
Typed31.62 / 9.2726.97 / 25.93
Generic65.42 / 9.9826.75 / 25.38

Each comparison freezes the workflow and scalar. Worst S is each rule's largest condition-specific mean; the maxima need not occur in the same condition. Correct yield averages all eight conditions (Table 31, p. 50). Separate banks reduce failure-specific acceptance with the displayed yield cost. This illustrates the gap between the pooled target and the protection the task requests, not a failure of conformal validity.

The effect survives a stronger planner. To isolate protection from scoring and sensing, we fix expanded-model VoI acquisition and the learned scalar:

CalibrationV acceptance (%)Correct commitment (%)
Pooled failures12.6072.37
Separate V/S banks9.9272.27

These are twelve-repetition means in the less-inadequacy condition; Table 25 (p. 46) retains all eight conditions. This benefit does not require two handcrafted scores.

K562 adds outcome-held-out policy selection: at a common 10% development error budget, CAVEAT gives 12.76% correct/10.05% wrong versus 8.85%/11.95% for the baseline in the dense-neighbor setting. These are real measurements with controlled omissions, and realized risks differ (Table 30, p. 49).

The original LiveClin result remains 1/0 correct/wrong commitments versus ordinary CP's 2/0. The shared response explains the retrieval constraint and the no-singleton audit even after adding the diagnosis (Appendix D.6, pp. 31–32). The separate development follow-up finds weak selection-margin separation (AUC .563); revision gives 9/12 versus 7/8 without revision and ties a cheaper trigger. These identify component limitations, not clinical superiority (Table 4A, p. 9; Appendix E.10, pp. 51–52).

W3/Q2: omission without deliberately deleting the answer

The revision separates how candidates fail from how measurements are obtained:

SettingEvidence without deliberate candidate deletion
Original LiveClinRetrieved lists omit the recorded diagnosis in 133/171 cases
Defects4JTop-10 lists omit an exact fixed line in 135/262 bugs; a benchmark proxy
Light tunnelRetrieval omits the reference target in 40/87 sub-incidents; physical interventions remain controlled
RCAEvalMean 12.3/120 test lists omit the injected service; telemetry faults remain controlled
Model-proposed spacesQwen-14B revision raises adequacy from 79% to 92% and correct commitments from 64 to 75, with zero wrong, on 120 controlled worlds

Tables 22 (p. 38) and 34 (p. 53) distinguish these protocols. The audits demonstrate occurrence under their declared targets; the model-proposed study demonstrates controlled recovery. A cause can be known in the world yet absent from the active explanation space. This is different from claiming discovery of a previously unknown disease.

W4/Q3: realistic labels and evolving workflows

Scoring a new incident does not require its cause label; calibration requires justified historical failure information. Resolved incidents, intervention measurements, or independently adjudicated consequences can supply it. An unexplained incident alone is not an inadequacy label. Proposition 7 (p. 43) conservatively handles sound sets of possible historical statuses without inventing labels; fully unresolved history can still force withholding.

The frozen object is the complete policy, which may itself query and revise adaptively. Changing it afterward requires recalibration or a justified transfer bound. Representative historical data remain necessary, including for evolving LLM systems (Section 3, p. 4; Appendix E.3, pp. 42–43; calibration-transfer Lemma 1, p. 16).

The calibration is standard. Our contribution is the task-relative adequacy framework, its coupling to repair, and the analysis of when evidence and finite history support useful commitment.

We would appreciate reassessment of the methodological contribution in light of Propositions 2, 3 and 6 and the fixed-score protection comparison, rather than Theorem 1 alone.

Many thanks for the detailed response. I think it makes sense. I have raised my rating to 3 (Weak Accept).

Thank you for reassessing. The contribution statement in the introduction and Section 3 reflects this exchange. We appreciate the time you gave the paper.

Reviewer_nHxDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies causal diagnosis when an active hypothesis space lacks any task-compatible explanation, and distinguishes this failure from selecting the wrong candidate within an adequate space. It introduces CAVEAT, which measures agreement between candidates’ predicted consequences and incident evidence, applies separately calibrated conformal tests for hypothesis-space inadequacy and selection failure, and routes cases to commitment, candidate revision, distinguishing-evidence acquisition, or uncertainty. Under null-specific exchangeability and a fixed pipeline, the paper establishes finite-sample control of commitments to incompatible selections and analyzes the conditions favoring failure-aware repair. Experiments span structural causal models, physical interventions, biological perturbations, clinical cases, and controlled LLM causal reasoning. The main empirical finding is a large reduction in wrong commitments, although useful retention and recovery vary considerably across tasks.

Critical strengths

Meaningful problem formulation and actionable failure decomposition. The paper identifies an important limitation of ranking-based diagnosis: even the best-ranked explanation may be task-incompatible when no suitable explanation is present in the candidate space. Separating this failure from incorrect selection within an adequate space is conceptually useful and leads naturally to different corrective actions. The failure-specific calibration provides a coherent way to attach an error-control objective to these decisions.

Broad evaluation and careful characterization of limitations. The proposed framework is evaluated across synthetic mechanisms, physical and biological data, clinical cases, and several LLM reasoners. The manuscript connects its experiments to explicit theoretical assumptions and reports unsuccessful results as well as successes, including calibration mismatch, limited clinical retention, and imperfect candidate supply. This breadth and transparency strengthen the empirical characterization of the method.

Critical weaknesses

Limited practical utility at the reported operating point. Although CAVEAT substantially reduces wrong commitments, this often comes at the cost of a very low correct-commitment rate. On LiveClin, CAVEAT makes only one correct exact commitment among 171 cases, compared with two for ordinary conformal prediction, while both methods make zero wrong commitments. This result does not establish a practical advantage over simpler abstention-based methods. Comparisons at matched realized error rates or coverage levels are needed to demonstrate that CAVEAT provides a better trade-off between safety and useful decisions.

The evidence for the distinctive benefit of repair routing is not yet conclusive. The initial matched-repair experiment did not establish a significant advantage over always revising the candidate space (51 versus 50 correct commitments, with a reported confidence interval including zero). The subsequent experiment modifies the task and candidate construction to create complementary scenarios in which revision and querying address different failure types. Moreover, the paper does not evaluate an equally calibrated discrepancy-based re-ranking baseline, despite a post-hoc audit showing that the minimum-discrepancy candidate is task-compatible in 178 of 240 worlds under full candidate supply. Consequently, it remains unclear whether the reported gains demonstrate the general effectiveness of the proposed routing mechanism or depend substantially on the constructed experimental setting and straightforward evidence-based candidate selection.

Questions for authors

Q1. At matched realized wrong-commitment rates and matched evidence budgets, what are the correct-commitment or risk–coverage curves for CAVEAT versus ordinary conformal prediction and other selective baselines on the real-data benchmarks?

Q2. How sensitive are the main results to replacing label-identity compatibility with task-relevant consequence compatibility, particularly when multiple mechanisms can lead to equivalent downstream decisions?

Q3. Can the routing advantage be demonstrated on naturally occurring failure mixtures, without designing complementary revision/query regimes or separately recalibrating each candidate-supply probability?

Q4. What are the gains of CAVEAT over (i) an equally calibrated minimum-discrepancy re-ranking baseline, (ii) a simple score-based router, and (iii) random evidence acquisition, under matched inference and measurement budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nHxD (part 1/2)

Thank you for asking whether reduced commitment translates into useful decisions, and whether simpler repair policies suffice. Your comments led us to test both questions directly. The revision adds risk–yield curves, outcome-held-out policy selection, and stronger repair comparators. RCAEval shows the value of explicit inadequacy protection; controlled LLM trials identify conditions where directing repair improves recovery. Below we connect each concern to its evidence and manuscript location. All locations refer to the clean revised PDF.

W1/Q1: useful commitment at comparable risk and evidence budgets

RCAEval: more correct commitments at comparable inadequacy risk. On 360 injected-fault incidents, candidates come from retrieval without deliberate deletion. Forty system-stratified splits separate fitting, calibration, and testing; selections and telemetry are fixed across verifiers. Mean results per 120 test incidents are:

Fixed ruleCorrectWrongV acceptance (%)S acceptance (%)
Pooled .0516.1251.2758.321.82
Pooled .1027.8502.37516.222.87
Dual .10/.3028.3751.7257.745.67
Pooled .30 + space .1027.8001.9257.747.05

V acceptance is the share of inadequate-space incidents committed. S acceptance is the share of wrong selections within adequate spaces committed. At approximately matched observed V acceptance, dual releases 28.38 correct commitments against 16.13; at similar yield, it lowers V acceptance from 16.22% to 7.74%. This does not match both risks or total error. The close hybrid supports explicit inadequacy protection rather than architectural uniqueness. Figure 8 shows both risk axes across the fixed-level sweeps. These are overlapping splits, not independent cohorts; the pooled scalar test differs from LiveClin's prediction-set CP, and this is not a bank-only isolation (Table 4B, p. 9; Appendix E.9, pp. 50–51; Table 32, p. 50; Figure 8, p. 51).

K562: higher correct yield after selection at a common development budget. Family-disjoint, outcome-held-out policy selection at a common 10% development error budget gives:

PolicyCorrect (%)Wrong (%)
Baseline8.8511.95
CAVEAT12.7610.05

Both use the same measurements. Realized test risks differ; isolated-decoy results are mixed. This is archived-score policy selection on real perturbation measurements with controlled omissions (Table 30, p. 49; Finding 4, p. 10). Separately, the descriptive frontier at a 6.39% wrong-commitment allowance gives 10.11% correct against 6.15% for the best baseline/deferral mixture; those mixture weights use exposed outcomes (Table 29, p. 49).

LiveClin identifies where the tested workflow needs improvement. On the sealed test, 133 of 171 retrieved lists omit the recorded diagnosis. Without changing those lists, exact recovery is possible only among the remaining 38 cases, and still requires correct selection. CAVEAT makes 1 correct commitment and ordinary conformal makes 2, with zero wrong commitments for both (Table 4A, p. 9). Even inserting the missing diagnosis produces no exact singleton under the tested verifier (Appendix D.6, pp. 31–32).

On 283 separate, exposed development records, a new scorer raises correct selections from 19 to 29 with fixed text, and to 38 with additional recorded text; selection-margin separation remains weak (AUC .563). Bounded revision gives 9 correct/12 wrong commitments versus 7/8 without revision and ties a cheaper trigger (Appendix E.10, pp. 51–52; Table 33, p. 52). These audits identify retrieval and scoring bottlenecks and show why threshold adjustment alone does not address all the tested workflow's limitations. They do not establish a clinical recovery advantage.

AuthorsOpenReview ↗

Response to Reviewer nHxD (part 2/2)

W2/Q3: repair routing, natural mixtures, and recalibration

Routing improves recovery in the controlled imperfect-supply trials. Proposition 2 (p. 6) states when routing helps: complementary repair gains must exceed routing losses. The following comparisons test parts of this mechanism:

StudyResult
Qwen 3B–32B, .50 compatible-candidate supplyRouting adds 6–20 correct commitments over the stronger fixed repair
Qwen-7B versus random repair, same supply+12.5 percentage points; paired story-cluster interval [7.9, 17.5]
Qwen-14B generates its own spaces, 120 controlled worldsAdequacy 79% to 92%; correct commitments 64 to 75, with zero wrong in both arms

OLMo's original routing-versus-random result remains nearly tied: 51 versus 50 correct. The model-proposed-space study does not delete answers from the model's proposals, but its worlds and probes remain controlled (Table 22, p. 38). Retrieval-produced omissions are also audited in clinical, software, physical, and telemetry records (Table 34, p. 53). These establish occurrence and test verification; they do not establish natural routing superiority.

Recalibration. Freezing calibration at supply .50 leaves correct counts unchanged in all 30 model/supply cells; wrong counts can change (Appendix E.6, p. 47). The yield pattern survives this replay without supply-specific recalibration. Validity under arbitrary distribution shift does not follow.

Q2: compatibility beyond label identity

The formal target is task-relevant consequences, not mechanism identity. Qwen-14B's revised spaces yield 75 task-compatible commitments but 67 exact-graph matches (Table 22, p. 38). The action guard releases 610 of its 1,332 actions with two candidates retained, with none exceeding the stated tolerance in the SCM study (Finding 5, p. 10). Clinical code categories have not been independently adjudicated as action-equivalent, so we do not relabel clinical errors as successful actions.

Q4: stronger re-ranking, routing, and random queries

Stronger re-ranking exposes a finite-history calibration trade-off. The full-supply discrepancy re-ranker gives 178 correct and 4–6 wrong commitments per 240 worlds with pooled calibration. Its terminal S bank is empty, so the implemented strict .10 dual rule cannot release commitments; borrowing the original selector's bank would not be matched calibration (Appendix E.6, p. 47). In the separate six-workflow study, supported noisy-condition re-ranking changes V/S acceptance from 9.85%/11.86% to 6.54%/8.43%, with correct commitment falling from 24.91% to 20.23% (Appendix E.8, pp. 49–50; Table 31, p. 50). Proposition 6 (p. 42) explains this finite-history trade-off.

A learned router matches the gate router; separate calibration improves its failure-specific protection.

ReasonerGate: correct / wrongLogistic: correct / wrong
Qwen-7B86 / 3.6787.33 / 3.67
Qwen-14B117 / 4.67118.67 / 4.67

These are means per 240 worlds over three splits, with the same repair opportunity; actual costs are reported separately (Table 26, p. 47). The original router is replaceable. In the plug-in study, separate calibration reduces the typed learned router's S acceptance from 31.62% to 9.27% in the inadequacy-dominated condition (Table 31, p. 50).

Random queries remain inconclusive. CLadder gives 42/1 versus 40/1 correct/wrong, paired p=.50p=.50; Chamber's gain is 5.34 points with interval [−1.4,13.96][-1.4,13.96]. The heterogeneous-query extension tests a different mechanism and does not replace these results (Appendix E.12, pp. 52–54; Table 35, p. 54).

These additions show where CAVEAT contributes: explicit inadequacy protection improves the RCAEval risk–yield trade-off, and directed repair improves recovery in the controlled Qwen trials. The learned-router comparison also shows that this protection can accompany a different routing implementation. Do Figure 8 and the stronger comparisons in Tables 26 and 31 address W1 and Q4? We would appreciate reassessment in light of these results and clarification of any remaining decisive concern.

Thank you for the additional experiments and clarifications. However, the key concerns remain: the routing mechanism offers little advantage over simpler alternatives, directed queries do not significantly outperform random ones, and superiority on natural tasks remains unverified. Moreover, both the manuscript and rebuttal exhibit defensive writing, with extensive qualifications and repeated reframing of the contributions rather than fully addressing the underlying limitations; therefore, I maintain my original score.

AuthorsOpenReview ↗

Thanks and summary before closing

Thank you for your continued engagement. Not to further clarify anything and to persuade you. Just wanna put summarized and more direct answers to each concern for the convenience of every reader

QuestionAnswerEvidence and follow-upDoes not show
Does inadequacy protection give useful commitments beyond reduced error?YesRCAEval. Real microservice telemetry under injected faults. 28.38 correct commitments per 120 incidents versus 16.13 for pooled verification at inadequate-case acceptance 7.74% versus 8.32% (Table 4B and Figure 8).Superiority at matched total error or on LiveClin.
Does routing beat simpler alternatives?PartlyBeats the stronger fixed repair by 20 and 30 correct commitments at 7B and 14B with matched wrong counts (Table 12). A trained adequacy classifier reaches similar yield to the gate router (Appendix E.6 and Table 26).That the gate routes better than a learned classifier.
Do directed queries beat random acquisition?Not shownCLadder 42 versus 40 correct commitments with one wrong in each arm. Chamber recovery difference 5.34 points with 95% interval [−1.40, 13.96].An advantage over random acquisition.
Is the advantage shown in a real-data setting?PartlyRCAEval above. On LiveClin the sealed workflow does not outperform ordinary conformal prediction. Adding the recorded diagnosis still yields no exact singleton under the tested verifier (Appendix D.6). Scoring and revision follow-ups are reported in Appendix E.10 and Table 33.A clinical advantage.

The paper reports the condition behind each Partly or Not shown result. Proposition 2 and Table 11 for routing. Proposition 3 and Appendix E.12 for directed queries. The scorer audits in Appendix D.6 for LiveClin.

Thank you for the questions that prompted these comparisons and audits.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared response to the reviewers and Area Chair

Thank you for the thoughtful reviews. They helped us sharpen both the motivation and the evidence for CAVEAT. Before trusting the best-ranked explanation, a reasoner should ask whether its candidates can support the task at all. CAVEAT distinguishes a missing task-compatible candidate (V) from a wrong selection when one is present (S), then guides revision or distinguishing evidence. Your main concerns were novelty beyond standard calibration, the connection to decision-focused sensing, and useful commitment beyond reduced error.

What changed, why, and where

The clean manuscript is not color-marked. These locations refer to the revised 55-page PDF, with ten main-text pages.

UpdatePurposeLocation
Framework and scopeClarify what verification and repair add beyond rankingAbstract, p. 1; Introduction, pp. 1–3; conclusion, p. 10
VoI formulation and repair analysisConnect correct commitment, separate failure budgets, and repairEq. (6), p. 5; Proposition 2, p. 6; Eq. (27) and utility/frontier relation, pp. 41–42
Causal and finite-history analysesIdentify when omissions are observable and history supports commitmentProposition 3, p. 7; E.1–E.2, pp. 40–43; Eq. (28) and Proposition 6, p. 42
Observation and partial-label extensionsState requirements for noisy evidence and historical adjudicationSection 3, p. 4; E.3–E.4, pp. 43–44
Stronger planners, selectors, and routersSeparate verification from scoring and acquisition; define both VoI baselinesTables 23–26, pp. 45–48; Table 31, p. 51; greedy EVSI definition, p. 54; Table 35, p. 55
RCAEval curves and held-out K562 selectionTest useful commitment under stated risk budgetsTables 1–4, pp. 8–9; Findings 3–4, p. 10; Tables 30–32, pp. 50–51; Figure 8, p. 52
Clinical cohorts and omission auditsSeparate occurrence, protection, and recovery evidenceTable 4A, p. 9; E.7, pp. 48–50; E.10–E.11, pp. 52–54; Tables 33–34, pp. 53–54

We also standardized labels, percentage rounding, and table layout. RCAEval's pooled scalar failure test is distinct from LiveClin's prediction-set conformal baseline.

What these updates add

Useful protection on RCAEval. On real-system telemetry under injected faults, explicit inadequacy protection releases a mean 28.38 correct commitments per 120 incidents versus 16.13 for pooled verification, at comparable observed V acceptance of 7.74% versus 8.32% (Table 4B). This approximately matches V risk, not both risks or total error. The close pooled-plus-space rule supports explicit inadequacy protection; this comparison does not isolate the banks alone. Levels are fixed; forty overlapping three-way splits describe resampling variability, not independent cohorts (Table 32 and Figure 8).

A clear connection to VoI. Evidence acquisition is a VoI problem. Our original separation rule is a heuristic; the added EVSI and finite-horizon dynamic programming use standard methods. CAVEAT contributes the task-relative verification-and-repair formulation and analysis. We value evidence by correct commitments enabled under separate V/S acceptance limits. Appendix E.2 now relates this frontier to ordinary terminal utility. Our +1/-4/0 baseline fixes both failure penalties at four; the constrained frontier derives their dual penalties from the budgets. Expanded-model VoI can also value omission-revealing evidence. The LP, DP, and conformal ranks are standard.

Protection within stronger workflows. We added expanded-model and adaptive two-query VoI to test whether verification adds protection within stronger planners. With that acquisition policy and a learned scalar score fixed, separate calibration lowers V acceptance from 12.60% to 9.92%, with correct commitment at 72.37% versus 72.27% in the less-inadequacy condition (Table 25). Calibration requires exchangeable failures under a frozen workflow, not a correct planner posterior. Controlled LLM trials test repair; the causal and finite-history analyses explain when evidence and calibration support it.

What LiveClin establishes. We retained this natural-omission setting because 133 of 171 sealed lists lack the recorded diagnosis. Those fixed lists cannot yield it without revision; the remaining 38 are only potentially correct before selection and verification. Adding the diagnosis still produces no exact singleton under the tested verifier (D.6, pp. 31–32). A separate 283-record development follow-up finds weak selection-margin separation (AUC .563; E.10). These audits identify retrieval and scoring bottlenecks in the tested workflows. The sealed comparison is 1 correct/0 wrong for CAVEAT versus 2/0 for ordinary conformal. The exposed revision follow-up ties a cheaper trigger. These results do not establish clinical recovery superiority. Human decision benefit requires a human-use study.

AuthorsOpenReview ↗

Further manuscript clarification

Thank you for the continued discussion. Reviewer KNt3 suggested organizing the explanation around the two verification questions. We have used this structure to make CAVEAT’s thesis and its relationship to value of information more explicit.

CAVEAT first assesses support for a task-compatible candidate in the active space. Once the space gate passes, it assesses the selected candidate. The gate results guide candidate revision or distinguishing evidence. Both repairs return to verification before commitment.

This round focuses on the presentation of the existing method and evidence. The main changes are listed below.

  • Abstract and Introduction (pp. 1 to 3). We connect the two verification questions to their corresponding repairs. Figure 1’s visual-loss example provides a concrete walk-through. The contributions follow the same explanation.

  • VoI connection (Introduction p. 2 and Section 3.3 p. 5). We explain the closed-candidate zero-EVSI case and distinguish it from expanded-model design. We bring the existing utility/frontier relationship into the main text. The baseline uses +1/-4/0 terminal utility. The constrained frontier values correct commitment under separate V/S budgets. Separation remains the primary acquisition heuristic.

  • Gate and workflow terminology (Sections 2 and 3 on pp. 3 to 5). We identify the space gate as the inadequacy gate and map the two gates onto the existing four-stage pipeline. We also state that selection-failure calibration membership does not depend on the space-gate outcome.

  • Theory overview (Section 4 opening on p. 5). We connect the existing results to commitment control and useful commitment. We also identify which results address repair routing and omission-revealing evidence.

  • Experimental interpretation and conclusion (pp. 7 to 10). We specify the metric denominators and report the RCAEval risk trade-off directly. We state the lack of a statistically significant recovery advantage over random queries on the light-tunnel panel. We identify weak score separation as one bottleneck in the tested LiveClin workflow.

The underlying method and formal results are unchanged. The reported experimental values are unchanged. No new experiments were added in this round. The appendix content and pagination are unchanged.

The manuscript retains ten main-text pages and 55 pages in total. Table 4 remains on page 9. Finding 3 now spans pages 9 to 10. The earlier shared response continues to summarize the studies added in the previous revision.

Thank you for helping us make the manuscript easier to follow and the interpretation of its results more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.