acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty Is Not Actionability

Abstract

Two questions can carry the same perfectly calibrated error probability yet favor opposite interventions: reflection repairs one, whereas retrieval repairs the other. Uncertainty omits the oriented state–action effect that determines whether an intervention crosses a decision boundary. We prove that this information gap forces tight worst-case regret for every action-blind policy over interventions, even under perfect calibration and randomized choice. Under decision-preserving transformations, we then characterize which effects are identifiable and derive minimal-dimensional coordinates for the resulting maximal invariants: anchored margins for answer scores and anchored cycles for assignment scores. When these coordinates are hidden, black-box logs reveal only the repair, harm, and cost of the executed action. We therefore introduce the Factorized Action-Value Estimator (FAVE), which estimates conditional action value by sharing need, response, and direction across actions. Across 30,918 natural matching cycles, conditional repair rises from 16.8% in the positive-but-insufficient region to 72.7% after boundary crossing. On a 2,000-item, five-intervention portfolio, FAVE exceeds equal-information pooled prediction by .0096 (two-way 95% CI ) and by .0203 on held-out intervention types; its advantage persists across all predeclared token prices. At an eightfold harm price, validation-selected pooling attains .0768 utility while every fixed intervention has nonpositive population value. Uncertainty locates the incumbent. Actionability predicts where each intervention will move it and what that movement is worth.

Primary area:
learning theory
Keywords:
Actionability, Decision Theory, Identifiability, Uncertainty Quantification, Adaptive Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_xEc3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why calibrated uncertainty alone cannot determine which intervention will improve a model’s answer. It develops a boundary-based characterization of intervention effects and proposes FAVE, a factorized action-value estimator evaluated using randomized one-action logs across five interventions.

Critical strengths

Clear motivating distinction between uncertainty and intervention value.

Useful matched comparisons and explicit reporting of negative results.

Critical weaknesses

The distinction between needing intervention and benefiting from a particular intervention is useful, and I appreciate the controlled comparisons. However, I have several major concerns:

First, the theoretical results do not yet establish a sufficiently substantive contribution beyond their definitions. Theorem 1 follows directly by substituting the definition of the score displacement into the post-intervention margin. Theorem 2 constructs observationally identical environments with different optimal actions, so its lower bound follows from assigning probability at most 1/K1/K to one action. Theorem 3 explicitly invokes classical maximal-invariant factorization. These results organize the problem, but they do not establish when practical uncertainty-based methods lose information or when FAVE can recover it. Can you identify a nontrivial prediction or algorithmic consequence of this theory that would not follow from directly learning conditional action values?

Second, the maximal-invariant characterization depends on a narrower transformation group than the broader decision-preservation language suggests. Section 3.2 removes global translations, although positive scaling also preserves answer rankings. Anchored margins retain scale information and therefore characterize translation-invariant effects rather than precisely the information necessary to preserve decisions. The paper does declare its groups, but the choice determines both the retained information and the minimal dimension. Why is translation the appropriate observation ambiguity for the answer-score interface, and which intervention decisions require the numerical margin information that would disappear under independent positive rescaling?

Third, the practical advantage over a strong simple policy is not established in the main utility regime. At the primary token price, best fixed achieves .1447 utility versus FAVE’s .1381, while using 16.5 rather than 20.2 extra tokens per item. FAVE also underperforms pooled prediction at 10% logging coverage and has no clear advantage at 25%. The high-harm experiment is useful, but it establishes a benefit in a particular cost regime rather than a general advantage under sparse feedback.

Questions for authors

What concrete algorithmic insight follows from the boundary geometry and maximal-invariant results beyond directly estimating conditional action values?

Under what sample-size and harm-cost conditions does FAVE reliably outperform fixed retrieval, which achieves higher utility and lower token use at the primary token price?

Reviewer_yKD5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that uncertainty about an answer is not enough to decide which intervention would improve it. It develops a theoretical account of this distinction and proposes FAVE, which estimates intervention value by separating the need for correction, the probability of an output change, and whether that change is beneficial. Experiments include semantic matching and an LLM intervention benchmark with one logged action per training item. FAVE improves over direct outcome prediction, with particularly encouraging results when harming a correct answer carries a high penalty.

Critical strengths
  1. The paper addresses a practical problem: additional reasoning or retrieval can make an answer worse. The high-harm experiment is a compelling example of why conditional intervention matters, since FAVE retains positive utility while every fixed intervention has negative average utility.

  2. The comparisons control for training information and model configuration, and the results include uncertainty across logging seeds. I also appreciate that the paper reports settings where FAVE does not help, rather than presenting the gains as universal.

Critical weaknesses
  1. The practical gain over stronger baselines is not yet clear in the standard setting. FAVE improves over pooled prediction, but does not clearly outperform split-head in the main neural experiment. Fixed retrieval is also competitive at the primary token price. The random-forest and high-harm results provide stronger support, but I am still unsure how much of the overall gain comes from choosing better interventions, rather than deciding more accurately when to leave the answer alone. This distinction matters to the paper’s motivation.

  2. The evidence for generalization is limited. The held-out-action results are promising, but use hand-specified action descriptors within the same item distribution. Cross-benchmark transfer is less successful, with no positive aggregate result and a substantial drop on HotpotQA. This leaves open how much the gains depend on the current model and intervention portfolio. A small replication with another model or intervention setup would strengthen the empirical case. The exact model versions, features, and retrieval protocol should also be stated more clearly.

Questions for authors
  1. At comparable token costs or harm rates, how does FAVE compare with split-head and the fold-selected fixed baseline? The appendix mentions an action-disagreement analysis. Does it suggest that the gains mainly come from choosing better interventions, or from deciding more accurately when not to intervene?

  2. Have the authors looked into what causes the performance drop on HotpotQA under cross-benchmark transfer? Please also clarify the input features, retrieval sources, and intervention protocols, as these details would help interpret the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.