acceptodds
Desk rejected at ICLR 2027

When Should an Agent Commit? Preference-Conditioned Workflow Control

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Stateful LLM agents repeatedly face a choice between replanning and committing to a multi-step plan. Frequent replanning preserves responsiveness to new observations but incurs repeated reasoning cost, whereas longer commitments amortize planning overhead while delaying high-level plan revision. We formulate this trade-off as preference-conditioned adaptive commitment and introduce , a high-level controller that selects the next commitment horizon from the current workflow state and a quality–cost preference while keeping the executor frozen. restores intermediate execution states to compare alternative horizons under a shared frozen continuation, yielding matched downstream quality–cost profiles. Horizon-specific predictors model relative quality retention and computational saving; the requested preference combines these predictions into a commitment policy, while a trainable residual is refined through on-policy interaction. We further analyze the induced semi-Markov control problem and use a finite controlled study to characterize when feedback-dependent horizon selection has value. The benchmark protocol evaluates the learned controller on ALFWorld and WebShop against fixed, query-level, and learned commitment strategies.

Primary area:
reinforcement learning
Keywords:
Agentic workflows, Multi-objective reinforcement learning, State-local decision structure

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WtQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Preference-conditioned adaptive commitment (PACE) is a framework that learns when an LLM agent should replan versus commit to executing its current plan for multiple steps (to achieve balance in compute cost during inference and end task performance). Rather than using a fixed replanning frequency, PACE dynamically selects a commitment horizon based on the current workflow state and a user-specified preference between task quality and inference efficiency.

To train this controller, PACE restores intermediate execution states and evaluates other commitment horizons under a shared continuation policy. These are used to train separate quality-retention and cost-saving predictors, whose outputs are combined according to the specified preference. The resulting policy is further refined through PPO, while the underlying LLM executor remains frozen. Experiments on ALFworld and WebShop report improved quality-cost trade offs compared with fixed horizon and learned commitment baselines.

Critical strengths
  • Preference-conditioned control: PACE allows a single controller to adapt its planning decisions to different quality-cost preferences without retraining the underlying LLM. The reported improvements on ALFWorld and WebShop also suggest the proposed method's potential effectiveness.

  • Counterfactual-based learning: The proposed restored-state counterfactual evaluation provides a principled way to compare alternative commitment horizons under the same workflow state and continuation policy, offering useful supervision for adaptive replanning.

  • Sound motivation: I think the motivation of looking into quality-cost preference trade offs is convincing enough of a motivation.

Critical weaknesses

Missing References:

  • Paglieri et al. (2025), Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents - Studies adaptive planning frequency in LLM agents, directly overlapping with PACE's central motivation. https://arxiv.org/abs/2509.03581
  • Liu et al. (2026), When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents - Proposes BRACE, which learns when and how to replan under computational constraints, raising questions about PACE's novelty and advantages over existing adaptive replanning methods. https://proceedings.mlr.press/v306/liu26fo.html
Questions for authors

Q1) How does PACE fundamentally differ from existing adaptive replanning approaches, particularly the ones in the Missing References in Critical Weaknesses? What specific advantages does PACE offer compared to these existing formulations?

Q2) Can the authors provide direct evidence that PACE actually adjusts its commitment horizons in response to intermediate observations? For example, how does the selected horizon change when unexpected observations invalidate an existing plan?

Q3) Could the authors compare PACE against a preference-conditioned controller that uses only the initial task information and execution progress, without access to intermediate observations? This, I think, will serve as a helpful baseline that would help isolate whether the reported improvements genuinely come from feedback-dependent adaptation.

Q4) I believe this question is important since the authors claim inference cost (might overlap with Q23; Ccould the authors provide the exact executor model, hardware configuration, decoding parameters, and inference setup? Additionally, could they report the number of planning calls and token consumption to clarify where the computational savings originate? I believe this will also be necessary to isolate where the efficiency contributions actually occur.

Q5) Could the authors provide confidence intervals or statistical significance tests for the improvements over the strongest baselines, particularly CF-Direct + PPO and Multi-Scalar PPO? I know this may take some time (and I personally don't like asking these kinds of requests during review) but it does say in the Appendix A.2 that tThe benchmark evaluation uses three independent training seeds and four executions per task and preference" (which I believe should give us the confidence intervals and mean)

1 comment

Thank you for the questions regarding prior work, intermediate observations, and computational cost.

Q1. Differences from Learning When to Plan and BRACE

PACE trains a lightweight, preference-conditioned commitment policy around a frozen executor using restored-state counterfactual supervision during training. In contrast, Learning When to Plan fine-tunes a unified LLM for a fixed trade-off, while BRACE manages replanning modes and token/latency budgets.

Seed-0 results on identical splits and backbone (Qwen2.5-7B-Instruct, g_Q = 0.5):

MethodALFWorld success (%) ↑ALFWorld cost (s) ↓WebShop score ↑WebShop cost (s) ↓
Learning When to Plan44.7882.790.399536.89
BRACE (text adapter; no E-RECAP)39.5583.330.368244.29
CF-Direct + PPO23.13120.690.360442.04
PACE43.0980.770.393136.60

PACE achieves the lowest inference latency across both environments and outperforms CF-Direct + PPO in quality, while keeping the 7B LLM completely frozen and supporting continuous runtime preference control.

Q2. Horizon changes in response to intermediate observations

When an action fails mid-segment, control returns to PACE to re-evaluate the horizon from the updated history. In WebShop (seed-0, g_Q = 0.5), across 183 early-return boundaries caused by invalid actions, the mean selected horizon contracted from 3.25 to 2.10 (shortening in 66 cases; e.g., dropping from 8 to 1 after opening an unexpected product page). This confirms that PACE adaptively shortens commitments based on intermediate execution feedback.

Q3. Controller using only initial task information and execution progress

We implemented Initial+Progress, which receives only the initial task, initial observation, g_Q, and execution progress (excluding subsequent observations and action history), while keeping the executor unchanged.

ControllerALFWorld success (%) ↑ALFWorld cost (s) ↓WebShop score ↑WebShop cost (s) ↓
Initial+Progress (no-feedback)35.6096.000.397935.87
PACE43.0980.770.363136.60

At g_Q = 0.5, intermediate observation feedback drives clear gains in partially observable, multi-step ALFWorld (+7.49% success, -15.23s), whereas shallower search-and-select workflows in WebShop depend primarily on initial query choices under single-seed evaluation.

Q4. Executor configuration, planning calls, and token consumption

All runs use Qwen2.5-7B-Instruct (bfloat16, RTX 4090, Hugging Face SDPA, batch size <= 4, 4,096-token left-truncated input, greedy decoding up to 384 tokens, EOS left-padding). Per-episode means (including failed episodes) are:

Benchmark / MethodPlanning calls ↓Input tokens ↓Output tokens ↓
ALFWorld / CF-Direct + PPO43.7568,9312,845
ALFWorld / PACE34.2556,5082,474
WebShop / CF-Direct + PPO18.0322,4891,217
WebShop / PACE18.1022,5881,211

PACE reduces planning calls by 21.7% and input tokens by 18.0% in ALFWorld while maintaining compact budgets in WebShop.

Q5. Confidence intervals against CF-Direct + PPO and Multi-Scalar PPO

Using 10,000 paired task-bootstrap resamples across all 134 ALFWorld and 500 WebShop tasks at g_Q = 0.5 (with inference time as cost):

BenchmarkComparisonUtility difference95% bootstrap CI
ALFWorldPACE − (CF-Direct + PPO)+0.1444[0.0472, 0.2427]
WebShopPACE − (CF-Direct + PPO)+0.0059[-0.0275, 0.0390]

PACE shows a statistically significant utility gain over CF-Direct + PPO in ALFWorld and a positive mean advantage in WebShop. Due to the high compute cost of counterfactual rollout collection across all 634 native benchmark tasks, our statistical analysis focuses on this primary baseline; multi-seed intervals for Multi-Scalar PPO will be added in the final revision.

Reviewer_ZKwaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper under review presents a new concept of a trainable workflow controller (named PACE) for stateful LLM agents. At each workflow decision point, the controller determines how many actions the agent should execute before calling the planner again. The authors call this decision execution commitment. The proposed approach learns a single preference-conditioned policy to dynamically select between short or long commitment rather than relying on fully reactive planning or a fixed Plan-k strategy. During training, “counterfactual futures” with different commitment horizons are used. The proposed approach has been evaluated by experiments against several baselines.

Critical strengths
  • The paper addresses an interesting practical problem on LLM agents.

  • The results are presented based on a solid combination of theoretical analysis and empirical evaluation.

  • The proposed approach takes different workflow states and user-specified quality-cost preferences into consideration.

  • The separation between quality retention and cost saving is promising.

Critical weaknesses
  • It is unclear how scalable the proposed approach is for larger agents or more horizons.

  • While the proposed approach has been tested with two benchmarks, more experiments would be needed to see how well it generalises to other domains.

  • The proposed approach assumes a fixed workflow structure.

  • The paper provides relatively little qualitative analysis of when the controller chooses long / short horizons in concrete cases.

Questions for authors
  • In practise, how could users specify their quality-cost preferences?

  • Did you observe any systematic patterns when the controller chooses long / short commitment horizons?

1 comment

Thank you for highlighting the practical motivation and the separation of quality retention from cost saving.

1. How can users specify quality–cost preferences?

Users specify gQ∈[0,1]g_Q \in [0, 1] (with gC=1−gQg_C = 1-g_Q) in the utility objective gQJQ−gCJC/C0g_Q J_Q - g_C J_C/C_0. Because PACE conditions directly on gQg_Q at runtime, a single trained controller supports any operating point without retraining. Practically, users can inspect the validation quality–cost Pareto curve and select the gQg_Q that meets their latency or accuracy budget prior to deployment.

2. When does the controller choose long or short horizons?

In the WebShop seed-0 run at gQ=0.5g_Q=0.5, across 183 boundaries where the next planned action became unavailable mid-segment, the mean selected horizon contracted from 3.25 to 2.10, showing that PACE integrates updated history and failure feedback to shorten commitments when plans are disrupted.

We also compare PACE with a No-feedback controller (observing only the initial task/observation, gQg_Q, and progress) on seed-0 checkpoints at gQ=0.5g_Q=0.5:

ControllerALFWorld success (%) ↑ALFWorld cost (s) ↓WebShop score ↑WebShop cost (s) ↓
No-feedback35.6096.000.397935.87
PACE43.0980.770.393136.60

The paired task-bootstrap utility difference (PACE − No-feedback) is +0.0459 (95% CI: [-0.0510, 0.1423]) in ALFWorld and -0.0771 ([-0.1235, -0.0318]) in WebShop. Intermediate observation feedback drives clear gains in partially observable, multi-step ALFWorld (+1.49% success, -9.23s), whereas shallower search-and-select workflows in WebShop depend primarily on initial query choices under single-seed evaluation.

3. How does the approach scale to larger agents or more horizons?

Counterfactual profile collection is strictly an offline training cost. On eight WebShop states, expanding the candidate horizon set from {1, 2} to {1, 2, 4, 8} (doubling candidate count and quadrupling maximum horizon) increased branch-collection tokens by only 2.33× due to shared prefixes and early segment terminations. At deployment, PACE executes a single lightweight forward pass without counterfactual branches, so inference latency is invariant to the training horizon set and scales cleanly with the executor.

4. Generalisation to other domains

We additionally evaluated PACE on ScienceWorld across 30 held-out tasks spanning 30 task types (including failed episodes in all means):

MethodScienceWorld normalized score ↑Tokens per episode
ReAct-style baseline0.0453249,826
PACE0.0620258,704

PACE achieves a 36.9% relative score improvement (0.0620 vs. 0.0453) with only a 3.5% increase in token consumption, sustaining productive multi-step trajectories up to the step limit in 26/30 episodes (vs. 22/30 for the baseline).

5. Fixed workflow structure

Separating high-level planning from deterministic execution is a standard paradigm for reliable agent systems, and PACE’s state-and-preference interface is agnostic to the internal control flow—naturally extending to hierarchical or dynamic sub-task workflows. We will clarify this scope and discuss flexible workflow extensions in the revised manuscript.

Reviewer_YwtMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACE, a preference-conditioned controller that selects how long an agent commits before replanning. With a frozen executor, restored-state counterfactual rollouts supervise separate predictions of relative quality and cost savings. These predictions guide a controller refined through policy optimization, while supervision is refreshed as the continuation policy changes. Experiments on ALFWorld and WebShop report improved quality–cost tradeoffs and generalization to unseen preferences. Ablations and a controlled simulator further examine the proposed design and the value of adaptive commitment.

Critical strengths
  1. Explicit treatment of stale supervision. Counterfactual outcomes depend on the continuation policy, so their supervision can become outdated during training. Refreshing these targets addresses an important training issue. Separating quality-prediction, cost-prediction, and policy-residual updates also helps preserve the predictors’ intended meanings.

  2. Practical preference-conditioned control. A single controller adapts replanning frequency to different quality–cost preferences without deployment-time counterfactual branching. The reported tradeoffs and interpolation to held-out preferences support its potential usefulness under varying inference budgets.

Critical weaknesses
  1. Insufficient baseline coverage. The evaluation lacks comparisons with directly relevant adaptive replanning methods, such as Learning When to Plan. Additional comparisons under matched execution conditions and compute budgets are needed to establish the method’s empirical advantage and novelty.
  2. The contribution of the proposed certainty structure is insufficiently isolated. Equation (3) estimates continuation-dependent relative action values, rather than plan stability. This is appropriate for the stated utility objective, but leaves unclear what the proposed structure adds beyond separate quality/cost value estimation. Moreover, the unrestricted residual in Eq. (5) can override the profile-derived horizon ranking. Table 2 supports the usefulness of outcome decomposition, but does not establish how accurately the predictors rank horizons or how much the final policy relies on them. Predictor validation and controlled ablations separating outcome supervision, the profile-derived logits, and residual refinement would better substantiate the claimed mechanism.
Questions for authors
  1. During a commitment, could new observations trigger implicit replanning that appears to be only a low-level action adjustment? Could the authors provide statistical evidence that such cases are rare?

  2. Counterfactual supervision depends on a frozen continuation policy, which subsequent optimization changes. How does PACE handle cases where profile distillation opposes an improvement under the updated continuation? What determines how much policy optimization is performed before the targets are refreshed?

1 comment

Thank you for recognizing practical preference-conditioned control and the treatment of stale supervision.

1. Baseline coverage: comparisons with adaptive replanning methods

PACE trains a lightweight, preference-conditioned commitment policy around a frozen executor by decoupling horizon-specific quality retention and cost saving (Sec. 4.1). In contrast, Learning When to Plan fine-tunes an LLM for a fixed trade-off, while BRACE governs replanning modes and token/latency budgets. We will add both references and clarify these distinctions.

Comparison at gQ=0.5g_Q=0.5 (Learning When to Plan is adapted to the task; BRACE uses its public controller core with a text adapter and without E-RECAP):

MethodALFWorld success (%) ↑ALFWorld cost (s) ↓WebShop score ↑WebShop cost (s) ↓
Learning When to Plan44.7882.790.399536.89
BRACE (text adapter; no E-RECAP)39.5583.330.368244.29
CF-Direct + PPO23.13120.690.360442.04
PACE43.0980.770.393136.60

PACE achieves higher quality and lower latency than BRACE and CF-Direct + PPO across both environments. Compared with Learning When to Plan, PACE delivers comparable performance at lower inference cost while keeping the backbone frozen and supporting continuous runtime preference control. Under ALFWorld step limits, PACE also cuts premature truncations vs. CF-Direct (52/134 vs. 71/134).

2. Contribution of the certainty structure: predictor validation and controlled ablations

In Sec. 4.2, “planning certainty” denotes quality retention relative to k=1k=1 under the reference continuation (distinct from plan validity probability), paired with cost saving. We will standardize this terminology throughout.

At policy-reached boundaries in the controlled simulator, the predictors achieve 0.0380 MAE (quality retention), 0.0182 MAE (cost saving), and 74.19% pairwise horizon-ranking accuracy. Controlled ablations under identical supervision show:

LearnerHorizon regret ↓KL divergence ↓
Direct distillation0.0052660.20135
Scalar predictor + residual0.0046300.37076
Separate quality/cost predictors without residual0.0051340.57081
Separate quality/cost predictors with residual0.0048980.41829

Adding the residual improves both metrics for the separate predictors (regret: 0.005134 → 0.004898; KL: 0.57081 → 0.41829), closing the approximation gap required to retain explicit quality/cost decoupling for zero-shot preference conditioning. The final policy agrees with the predictor ranking at 84.91% of boundaries and refines it at 15.09%. Evaluated under the frozen reference continuation, these refinements yield a near-zero mean utility shift (-0.000391), confirming that the residual adapts decisions to the evolving on-policy distribution rather than overfitting the static reference policy (Eq. 5).

3. Could intermediate observations cause implicit replanning?

The executor generates an action list once per segment. Intermediate observations only check executability and bind object IDs—never triggering within-segment LLM calls—and any unexecutable action immediately returns control (Sec. 4.1).

Auditing 634 test episodes (13,642 segments, including 913 multi-action segments with ≥2\ge 2 actions) at gQ=0.5g_Q=0.5 revealed zero modifications to committed action sequences or nonnumeric templates (one-sided 95% episode upper bound: 1.36% across 219 multi-action episodes). Deterministic ALFWorld argument grounding accounted for 321 object-ID bindings across 229 segments (77/913 multi-action segments after step 1; 8.43%, trajectory-clustered 95% CI: [5.41%, 12.05%]), resolving symbolic IDs while strictly preserving the committed plan.

4. How are conflicting updates and stale supervision handled?

During PPO, the predictors remain frozen while fresh trajectories update the residual under a weighted profile-distillation regularizer (λpref=0.1\lambda_{\mathrm{pref}}=0.1); profiles are periodically refreshed under the updated controller (every 64 PPO episodes / 4 epochs, 60 fitting steps, 24 rounds).

Across three paired seeds (observation accuracy 0.9, mode-switch 0.02, planning cost 0.05) with matched environment-interaction budgets (within 8 primitive steps/seed; No-Refresh converts refresh steps into extra PPO rollouts), PACE achieves higher mean utility across gQ∈{0.1,…,0.9}g_Q \in \{0.1,\ldots,0.9\} (0.399915 vs. 0.398828, paired gain +0.001086, 95% paired t-CI: [-0.003734, 0.005906]). A gradient diagnostic at final checkpoints detected opposing PPO and distillation gradients in 1/3 checkpoints, confirming that PACE mediates temporary objective conflicts within a round before realigning supervision at the next refresh.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.