acceptodds
Under review as a conference paper at ICLR 2027

Where to Spend the Intervention Budget: Understanding and Using Preintervention Information

Abstract

An uncertain decision is worth revisiting only if an alternative action, a repair, can improve its outcome. With a limited budget, an agent must choose which decisions to evaluate before it knows what their alternatives will achieve. We study which available information helps make this choice, measured by Capture: the share of an oracle's repair value that a selection obtains under the same budget. Across four synthetic retrieval settings and seven seeds, features that describe the candidate alternatives yield 0.4266 Capture, compared with 0.2669 for retrieval features when both feed the same value regressor. Predicting whether a useful repair exists reaches 0.5078; nearest neighbor regression on similar, previously evaluated decisions reaches 0.6235. These findings motivate Cue Informed Allocation (CIA), which estimates repair value and whether a repair exists, chooses a scoring rule on previously evaluated decisions, and ranks new decisions for evaluation. At a 20% budget, it reaches 0.6316 Capture, mostly owing to the nearest neighbor estimate. We also compare eight policy learning methods adapted to the same allocation task and examine recorded outcomes from interactive tasks. Candidate features and outcomes of similar decisions account for most of the gain; combining predictions and choosing rules add smaller gains that depend on the setting.

Primary area:
causal reasoning
Keywords:
causal credit assignment, active learning, intervention budget, long-horizon agents, counterfactual feedback

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WP6qOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study a setting where an agent failed to complete some task, and an auditor (called ``allocator") tries to reveal which decisions led to the failure under a limited budget. In particular, they define repair value : the improvement that the best valid alternative gives over the executed action, divided by the effort of evaluating the candidates and Capture which is the ratio of repair value over an oracle measured improvement which has upfront access to all outcomes and operates on the same budget. They propose CIA, an algorithm to estimate repair value and do a series of experiments to evaluate its efficiency under different scoring functions (functions that represent the value of an action).

Critical strengths
  • The overall motivation of the problem is strong: failure of an agent that tries to repair it (itself or with external help) by examining its decisions and alternatives is a reasonable process.
  • There is a variety of factors considered in the designed experiments.
Critical weaknesses
  • The exposition is weak. The paper does not have a coherent order of presentation of definitions, model assumptions as well as explanatory examples. The effort spent on formalizing the setup is minimal. There is presentation of the units and domains of the quantities and functions introduced in Section 2.
    another example:
    In page 2, when basic setup is formalized there is this sentence the simulator sets gi when it constructs the alternatives, so positive score differences without a designated valid repair receive zero value. The term ``simulator" is not defined before and it is unclear what is does.
    Section 3 also comes absurdly; how does information about prediction of repair opportunities relate to the set up presented on section 2? What do we need to predict?

  • The proposed methods are rather simple ; a combination of nearest neighbors search with standard ML techniques as ridge regression. The gains on Capture as presented on Table 1 over nearest neighbors are also not substantial.

Questions for authors
  • In the setup of the problem after taking action a the allocator is revealed the scores for all actions in AiA_i. Why is that a reasonable assumption in practice? More often than not taking an action has binary feedback: we either gained the outcome we expected or not. In addition, generating the scores of an action is more often than not, a costly procedure itself. So I don't see how or why an allocator would receive such strong information. Please provide relevant examples or justify the assumption.

  • On page 2, on the denominator of capture there is a term TopK(v,k)TopK(v,k) , what is this term exactly?

  • What is the main benefit of your method vs NN?

  • Since this is about agent failure, is there an experiment you can do using an actual agent executing alternative actions, rather than relying primarily on simulator-defined outcomes? If not why?

Reviewer_4d2gOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to choose, under a fixed budget, which decision points in agent trajectories to “intervene” on, meaning that all candidate actions at that point are completed and evaluated. The choice must use only information available before intervention. Allocation quality is measured by Capture: the share of the oracle’s top-kk repair value that a selection obtains at the same budget.

The main study uses a synthetic two-hop retrieval simulator. It has four settings patterned after HotpotQA, 2Wiki, MuSiQue and Bamboogle, and seven seeds. Each setting is held out in turn, with 2,016 source and 1,512 test decision points, and four candidate actions per decision. The authors report that:

  • Candidate features beat retrieval-confidence features (0.4266 vs. 0.2669 Capture at 20%).
  • A binary “recoverability” target beats value regression (0.5078 vs. 0.4325).
  • Nearest-neighbor regression over previously evaluated source decisions reaches 0.6235.

The proposed method, CIA, chooses among 11 fixed combinations of these signals by leave-one-source-setting-out validation. It reaches 0.6316, compared with 0.3393 for the best heuristic chosen per test set.

The paper also adapts eight policy-learning and credit-assignment methods to the allocation task. These use a single seed and perform close to random. It adds many robustness analyses and exploratory results on recorded Search, WebShop and ALFWorld outcomes. Code and decision records are released.

Critical strengths

1. Interesting and relevant problem.

The paper separates acquisition (which counterfactual outcomes to pay for) from credit assignment (how to use them once observed). This framing is useful: interventions cost model calls, tool calls or human time, and existing credit-assignment methods usually take the set of observed outcomes as given.

2. Reproducibility.

The paper is well supported for reproduction. The anonymous repository includes the analysis code, configurations, intervention records and expected outputs. Seeds (47001–47007) and hyperparameters are fully specified, and comparisons are paired on identical records.

Critical weaknesses

1. Clarity is the main problem.

Section 2 introduces notation, but the formulation is minimal and scattered, and much of it is defined by the simulator rather than by the problem. As a result, the paper is hard to read and its claims are hard to assess.

  • The problem is never stated as an optimization problem. The paper does not say what is random, from which distribution source and test decisions are drawn, or what the allocator knows. It also never states the objective it optimizes. It is never made explicit that allocation is a one-shot top-kk subset selection rather than a sequential or adaptive process.

  • Key quantities are opaque. gig_i is set by the simulator when it “designates” a valid repair, and its meaning outside the simulator is unclear. The roles of eie_i (effort) and did_i (price), and the choice of τ=0.025\tau = 0.025, are not motivated.

  • The 12 features are named but never defined, even in Appendix B.3. Examples include “ambiguity pressure”, “prefix diversity” and “alternative redundancy”. Since a central claim concerns which information predicts repairs, this omission is significant.

  • The many coined terms make the text hard to follow. The paper uses intervention, repair, recoverability, Capture, purchase, source/target, phase, family, branch, Rule 1 and CIA.

  • The main text depends on the appendix. For example, Sec. 3.1 cites Table 10, which is in the appendix. The same results are repeated several times; the label-noise numbers appear in Fig. 5, Table 24 and twice in the text. Reporting four decimals suggests a precision the experiments do not support.

  • The link between the motivation and the experiments is loose. The introduction discusses LLM agents. In the main study, however, the decision points are produced by a parametric simulator, and no agent is involved.

2. The experimental evaluation is weak.

  • The main evidence is synthetic, and the domain shift is small. The four “settings” are the same generator with different noise and difficulty parameters. They share the same 7 branch families and 3 phases. Holding out one setting is therefore a weak test of transfer. The success of kk-NN may simply reflect that test points are close to source configurations. Naming the settings after HotpotQA, MuSiQue, etc. can mislead readers, even though the paper states that the tasks are not drawn from these datasets.

  • The central signal may be an artifact of the simulator. Document overlap is used both as an input feature and inside the outcome score. With overlap features alone, Rule 1 reaches 0.545 Capture. Without them it reaches 0.357, essentially the hindsight-best heuristic (0.339). The paper’s main conclusion (candidate features predict useful repairs) therefore cannot be separated from how the simulator scores outcomes. An experiment where outcomes do not depend on the features is needed.

  • The non-synthetic evidence is minimal. The recorded-task study (App. F.4) is very small: ALFWorld has 12 test points, so 3 are selected at 20%. Results have mixed signs: CIA wins on 3 of 6 test sets at 20%, with one tie and two losses. No seeds or confidence intervals are reported, and the authors state that feature availability before intervention was not verified.

  • Only an acquisition proxy is measured. Capture measures acquired value, not the quantity that motivates the paper: whether learning from the selected outcomes improves a policy or critic. The authors acknowledge that this is not measured. The secondary metrics (episode support, cosine similarity, recall@10%) are computed from the same outcome vectors and do not close this gap. Without a downstream experiment, the practical value of better Capture is unclear.

3. The method and the metric are not principled.

  • Why a new metric? With uniform costs, Capture is exactly 1−normalized regret1 - \text{normalized regret} with respect to the top-kk oracle. Equivalently, it is value-weighted recall@kk, closely related to NDCG@kk. The paper should state this connection and explain why the existing metrics are not sufficient. It should also discuss the properties of Capture:

    • It is non-monotone in the budget.
    • It is set to zero when the oracle value is zero.
    • Under unequal costs the oracle is a greedy knapsack, so it is not an upper bound, and Capture can in principle exceed 1.

    The definition of viv_i is itself ad hoc: the gate gig_i, the division by eie_i, and the threshold τ\tau. No sensitivity analysis is given for these choices.

  • The method is not derived from any objective. CIA consists of kk-NN, ridge regression and 11 hand-picked weightings (e.g. 0.8/0.1/0.1), plus validation-based selection. Several further choices are unjustified: ridge regression on a binary label instead of a classifier, 3×3\times upweighting of positives, and z-scoring over the test batch. Yet the problem has a natural principled solution. With uniform costs and one-shot selection, maximizing the expected acquired value means ranking by the posterior mean E[vi∣xi]\mathbb{E}[v_i \mid x_i]. From this view, two of the headline findings are statements about misspecification of the linear model, not about which information predicts repairs:

    • Recoverability beating value regression.
    • kk-NN beating ridge regression.

    Ranking by P(v>τ∣x)P(v > \tau \mid x) should not beat ranking by E[v∣x]\mathbb{E}[v \mid x] when the latter is well estimated.

  • There is no analysis of when the approach works. kk-NN assumes smoothness of vv in feature space and adequate source coverage. The paper does not characterize covariate shift between source and target, and gives no guarantee. It also does not position the method against established frameworks for budgeted acquisition: value of information, budgeted active learning, best-arm identification and contextual bandits. In particular, it omits the natural adaptive variant, in which each purchased outcome updates the ranker before the next purchase.

  • The analysis does not support the stated conclusions. The authors acknowledge that the “which information helps” conclusions depend on the estimator. If so, the analysis does not tell us which information is useful; it tells us which estimator works best in this simulator.

Questions for authors

Problem formulation (W1). Could you state the allocation problem formally? Please specify the distribution from which source and test decision points are drawn, what the allocator observes, and the objective being optimized. In particular, is the task a one-shot top-kk selection? If so, is ranking by E[v∣x]\mathbb{E}[v \mid x] optimal for expected Capture, and how does this square with your finding that the binary recoverability target outperforms value regression?

Experimental evidence (W2). Your main conclusion is that candidate features predict useful repairs. However, overlap features alone reach 0.545 Capture and excluding them drops performance to 0.357, close to the hindsight-best heuristic (0.339). Can you report results in a setting where the outcome score does not depend on the input features, and show that acquiring outcomes with CIA improves a downstream policy or critic compared with random or heuristic selection at the same budget?

Metric and method design (W3). With uniform costs, Capture equals one minus normalized regret with respect to the top-kk oracle, and is closely related to value-weighted recall@kk and NDCG@kk. What does Capture offer beyond these standard metrics? Relatedly, why are the 11 candidate rules and their fixed weights (e.g.\ 0.8/0.1/0.1) the right search space, given that replacing one component with a depth-3 random forest (0.641) already outperforms CIA (0.632)?

Reviewer_EvC9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the setting where an agent has limited resources to revisit its decisions and we want to identify which decisions should be prioritized. They assume access to a set of agent decisions, each with the action originally taken and a small set of alternative actions and historical decisions whose candidate actions have all been evaluated. For the new decisions, candidate descriptions and initial scores are available, but candidate outcomes are hidden. The proposed method, Cue Informed Allocation (CIA) uses a set of features to compute estimated repair value, global recoverability, and phase-specific recoverability. The paper defines 11 candidate scoring rules, which combines these three predictors in different ways. Then, historical validation chooses which rule to use on the new decisions. The chosen scoring rule is applied to rank the new decisions. The experiments use four synthetic retrieval settings and offline evaluation on recorded Search, WebShop, and ALFWorld tasks. Overall, the paper provides evidence that evaluation effort should be directed toward decisions with promising alternatives. The strongest empirical finding is the effectiveness of a simple nearest-neighbor approach; the additional benefit of full CIA is modest.

Critical strengths
  1. The paper addresses a clear and important resource-allocation problem.
  2. The method is simple and interpretable and the experiments identify which components contribute to performance. Their empirical comparisons separate the effects of candidate features, prediction targets, nearest-neighbor estimation, and scoring-rule selection. They show that similar historical outcomes account for most of the improvement.
Critical weaknesses
  1. The strongest empirical evidence comes from synthetic tasks. The additional evaluation on recorded interactive tasks has mixed results and some test sets are small. These experiments provide limited evidence that the method will work reliably in live agent settings.
  2. The method assumes substantial resources are already available. It requires candidate alternatives for every decision and a relevant historical collection whose alternatives have already been evaluated.
  3. Full CIA offers only a small improvement over nearest neighbors. At a 20% budget, Capture increases from 62.35% to 63.16%. This supports the usefulness of historical similarity more strongly than the need for the additional predictors and rule-selection procedure.
  4. The related-work discussion could better connect the paper to decision-focused prediction and resource allocation. For example, Ren et al. (2024) [1] study why predictive accuracy can fail to reflect downstream allocation quality, while Cortes-Gomez et al. (2025) [2] incorporate downstream utility into uncertainty quantification. These connections would help contextualize the paper’s use of Capture for validation and its distinction between uncertainty and repair value. [1] Decision-Focused Evaluation of Worst-Case Distribution Shift. Ren et al (2024) [2] Decision-Focused Uncertainty Quantification. Cortes-Gomez et al. (2025)
Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.