acceptodds
Under review as a conference paper at ICLR 2027

The Default Trap: Rethinking Plan Evaluation in Tool-Using LLM Agents

Abstract

An executor can respond strongly to a change in a supplied plan's priority while showing a small change in the same information-selection probability when a default-aligned whole plan is removed. We call the risk of interpreting the latter as weak responsiveness to alternative priorities the default trap. We compare paired plans that prioritize different information targets with a shared no-plan reference. An accounting identity relates these distinct behavioral contrasts. Across 3,200 decision windows on 160 selected Retail, Airline, and AgentDojo tasks, switching priorities strongly redirects two models' choices, while the two plan-versus-default contrasts differ. In 2,160 additional windows, reversing account-list order shifts default target selection by 63.3–98.3 percentage points; priority-switching effects remain 96.7–100.0 points in either order. A separate 3,240-window component study finds strong control under single priority sentences, with effects of additional text varying by group and direction. Finally, 1,080 full-task episodes yield observed success differences of to points relative to no plan. All Retail and Airline success intervals include zero; AgentDojo results describe four fixed application worlds. These findings support joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, agent planning, tool use, plan ablation, priority control, position bias, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PXK6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how a supplied plan's priority changes action selection in tool-using LLM agents and names the default trap: interpreting a small change on removing a default-aligned whole plan as weak responsiveness to alternative priorities. Paired plans replace a contiguous priority span and are compared with a shared no-plan reference, and an accounting identity, D_A = R_A + G_A, relates the two contrasts. Across 3,200 decision windows, switching priorities strongly redirects two models' choices, reversing account-list order shifts default target selection by 63.3–98.3 points, and 1,080 full-task episodes show success differences of −19.4 to +8.3 points relative to no plan. The paper concludes that responsiveness, defaults, and task outcomes should be evaluated together.

Critical strengths

Matched design with a shared no-plan reference. The paper contributes "a matched, bidirectional measurement protocol with an accounting identity". Each text condition defines three input conditions: P_A, P_B, and a shared no-plan condition ∅ that deletes the full plan field. Five executions repeat each plan twice and execute no plan once, with identical initial state, history, tools, and continuation instructions. This makes the plan-to-action contrasts internally valid. The same restriction limits what they say about plans (see W1).

Every category remains in the analysis. All eight endpoint categories remain in the denominator, and the full outcome vocabulary identifies another source of removal effects. In Airline/Mini, R_B = 91.96 combines a 68.75-point decrease in A selection with a 23.21-point gain in branch coverage. Coverage also explains differences between the two directional effects: in AgentDojo/Luna, the 7.64-point gap between D_A and D_B equals the A-plan versus B-plan coverage difference. Retaining these outcomes makes changes in selection and stopping behavior visible within the same outcome population.

Tool and token counts complement success. In Airline/Gemini, mean executor tokens rise from 90.7 thousand to 133.8 and 138.5 thousand, increases of 47.5% and 52.6%, and the two absolute token-difference intervals are positive, while success is 30.8%, 30.8%, and 33.3%. AgentDojo/Gemini uses fewer calls under both plans alongside lower observed success, illustrating why calls alone do not measure efficiency. Supplied plans can therefore raise cost without a reliable gain in success, which does not follow from the design and is useful to practitioners.

Critical weaknesses

W1. The intervention supports claims about a priority sentence, not about plans. The paper itself states that "a priority edit measures responsiveness to the supplied priority text". Each pair retains a shared body and edits one priority span, and in the complete frozen plan pair (Appendix C.2) only "ratings" versus "prices" changes. Single priority sentences produce D_A = 86.7–100.0 points in Retail and Airline, and the component study "shows why a large priority response alone cannot establish the value of the surrounding plan structure". The title, "Rethinking Plan Evaluation in Tool-Using LLM Agents", promises more than this, while later decisions and richer plans remain a direction for evaluation, beyond the systems tested here.

W2. The accounting identity adds little, and the default trap is not shown to occur in practice. Adding and subtracting the default probability gives an accounting identity (Equation 2), and the paper notes that "the empirical evidence comes from the implemented input interventions", not from the identity. That a small R_A can accompany a large D_A when default behavior already favors A follows from the positive headroom bound R_k ≤ 1 − p_k(∅), which the paper states in Section 3.2. The default trap is defined as the risk of interpreting a small removal contrast as weak responsiveness to alternative priorities, but the paper cites no work that drew this conclusion from a whole-plan removal ablation. It is unclear whether the default trap is a problem anyone has actually run into.

W3. The absence of a success effect follows largely from task selection. The paper selects states with two useful reads allowed by the benchmark rules whose order preserves business state. It notes that state preservation does not imply equal information value, cost, or downstream consequences, but the selection still targets choices where either order is permissible. The result that "strong local responsiveness accompanies uncertain completion gains" is therefore expected: supplied priorities reliably change the initial information choice, while task success additionally incorporates subsequent decisions, tool use, and termination. All 12 Retail and Airline plan-versus-default intervals include or touch zero. A setup in which one priority matches the benchmark-provided reference actions and the other does not would test whether control has practical value.

W4. The A/B assignment may produce the removal asymmetry. In the paired-priority study, default A probability ranges from 66.67% to 89.29% while default B probability is at most 18.06% (Table 3), and across the six groups, B-directed removal effects exceed A-directed effects. The paper concludes that "reporting either removal direction alone therefore yields a different impression of the same executor's responsiveness to priorities". In the list-order intervention, A and B identify the first and last records in the original order, so the default there reflects position. For the paired-priority study, each task fixes two useful information targets A and B, but the paper does not say how a target is assigned to A. Without this, the unequal removal effects may come from labeling rather than from the executor.

W5. The same model fills most evaluation roles, and model-based judgments are not validated by humans. In the follow-up studies, Mini prepares and reviews the plan pairs, simulates the user, judges required natural-language assertions, performs policy review, audits local endpoints, and is one of the executors. The paper acknowledges that Mini's role as executor, simulator, and reviewer permits shared errors, but it does not measure them. In the endpoint audit, agreement is with model review, not human ground truth, and the enriched sample does not estimate population classification accuracy. Full-task local endpoints are not audited (Table 7, 0 audited).

W6. Small, heavily filtered samples. The list-order intervention and component study use 20 tasks per domain, and the full-task evaluation uses 15 Retail and 13 Airline tasks. Success intervals are wide, for example [−22.2, 6.7] for Retail/Mini B minus A, and the paper states that these intervals leave practically important benefits and harms unresolved. Intervals are nominal, pointwise percentile intervals without multiplicity correction, and there is no power analysis. Each wording runs each plan twice and no plan once, so the default, which is the reference in every R_k, rests on half as many draws as each plan. AgentDojo results describe four fixed application worlds and have no intervals, although resampling tasks would give them. The task pool is heavily filtered: AgentDojo retains four of 40 workspace and two of 16 banking tasks, and the paper notes that task composition changes effect size, with Retail D_A for Luna ranging from 75.69 to 98.75 points across source groups (Figure 6). AgentDojo/Gemini has D_A = 33.3 with substantial OTHER probability in the full-task evaluation and 25.0 under single priority sentences, which contradicts the abstract's claim that the component study "finds strong control under single priority sentences". The paper does not discuss this group.

W7. The list-order finding is largely known. The paper states that position bias means selection changes when an item's position changes with its content fixed, and documents it through work it cites (Lu et al., 2022; Blankenstein et al., 2026; Wadi & Ma, 2026). The contribution is in showing that "an ablation report can therefore change substantially when the same records are presented in a different order", and the paper should state its contribution that way. Only a full reversal of the account list is tested, so primacy and recency effects cannot be separated.

W8. One quantity has several names. The paper introduces "priority control, also called directional control", and D_A further appears as directional effect, priority switching and priority responsiveness. R_k appears as removal contrast, plan-minus-none contrast, plan-versus-default contrast and A-directed removal effect. In Appendix E the legacy field p_AB includes BOTH, while designated-branch coverage C = p_A + p_B excludes BOTH. The reader has to rebuild these correspondences to follow the main claim. A glossary and one name per quantity would fix this.

W9. Key details are in the appendices. The abstract is dense with qualifications, while information needed to judge the main claim, such as how A and B are assigned, how endpoints are audited, and the customer-cluster intervals, appears only in the appendices.

Questions for authors
  1. A/B assignment (W4). Each task fixes two useful information targets A and B, and the selected endpoint is defined before formal execution. How was a target assigned to A in the paired-priority study? Was A chosen as the target the executor reads first without a plan, the first one mentioned in the dialogue, or by another rule? If the labels were swapped or randomized, would B-directed removal effects still exceed A-directed effects?

  2. Evidence for the default trap (W2). Can you point to published work that reported a small whole-plan removal contrast and interpreted it as weak responsiveness to alternative priorities? Without such cases, how often does this misreading occur in practice?

  3. Priorities that differ in correctness (W3). The full-task evaluation requires benchmark-provided reference actions that can be replayed. Have you tested pairs in which one priority follows the reference action order and the other does not? This would show whether priority control affects task success when the order matters.

  4. AgentDojo/Gemini (W6). This group has D_A = 33.3 with substantial OTHER probability in the full-task evaluation and 25.0 under single priority sentences. Since OTHER denotes another read, which reads does Gemini choose instead of A and B? How does this result fit the abstract's claim that the component study finds strong control under single priority sentences?

  5. Default estimation (W6). We run each plan twice and no plan once, so every R_k rests on half as many default draws as plan draws. How much do the R_A and R_B estimates and their intervals change with additional no-plan executions?

  6. Human validation (W5). The paper states that Mini's multiple roles permit correlated errors, which the deterministic native checks and retained review records help examine. Have any endpoint labels, assertion judgments or policy reviews been checked by humans? Is there a reason the Full-task local endpoint (E3) row of Table 7 has no audited windows?

  7. Primacy versus recency (W7). The list-order intervention reverses account records while fixing target identities, plans, and business state. Since A and B identify the first and last records in the original order or reservation list, can you separate primacy from recency, for example by moving A to a middle position?

  8. Statistical power (W6). The paper states that the success intervals leave practically important benefits and harms unresolved and do not establish equivalence. What success difference could the full-task evaluation detect with 15 Retail and 13 Airline tasks?

  9. Plans beyond a single priority span (W1). Extending the protocol to later decisions, richer plans, and multi-agent coordination is a direction for evaluation. Do you have any results, even preliminary ones, where paired plans differ in more than one priority span or where the measured decision is not the first read?

3 comments

In response to the Program Chairs’ concern about AI use, I used Grammarly to improve the readability of my review. I identified the paper’s strengths and weaknesses myself. I did not use any AI tools to identify those strengths or weaknesses, form my overall assessment, or determine my score.

Thank you for the review. D denotes priority switching and R plan minus none. Effects are in pp.

Q1 / W4. Target assignment

Labels were fixed before formal no-plan runs on previously used tasks.

DomainPair construction
Retail75/96 first/last account records, 20 product pairs, one requested-order pair
Airline23/28 first/last records, three requested-reservation pairs, two outbound/return searches
AgentDojoSemantic target pairs

Swapping labels reverses the R_A/R_B asymmetry. Fair random relabeling equalizes their aggregates in expectation. Label-invariant (D_A+D_B)/2 is 63.54–99.11 across the six original groups. We will report each pair type's exact assignment rule and whether prior task use informed it.

Q2 / W2. Practical occurrence

The identity is an accounting decomposition. Record reversal changes removal effects with switching fixed.

Retail/LunaOriginal recordsReversed records
R_A3.3100
D_A100100

We have not identified verified published examples or estimated prevalence. We will describe the default trap as a possible interpretation error.

Q3 / W3. Correctness

Correctness-sensitive pairs remain untested. Admissible reorderings limit conclusions about the value of correct ordering. Such tests require verified preconditions or information dependencies. Completion benefits and harms remain unresolved in these selected tasks.

Q4 / W6. Gemini OTHER

Reads within OTHERCount in each sentence condition
City-level hotel, restaurant or car-rental enumeration36/45
Channel enumeration3/45
Webpage reads3/45
Channel-message reads3/45

AgentDojo/Gemini has D_A=25.0 for single sentences and immediate D_A=33.3 in full-task runs. We will restrict the abstract's strong sentence-only control claim to Retail and Airline and report this exception.

Q5 / W6. Additional defaults

The Mini follow-up used 288 windows on 24 states, with three runs per plan and six defaults on one date. Plan runs stay fixed in the 3 → 6 comparison.

DomainR_A, 3 → 6R_B, 3 → 6D_A
Retail25.00 → 31.2575.00 → 68.7587.50
Airline16.67 → 22.92100.00 → 100.0095.83
AgentDojo29.17 → 29.1779.17 → 81.2587.50
R_A intervalFirst 3 defaultsAll 6 defaults
Retail[0,62.5][4.17,62.5]
Airline[−4.17,41.67][−4.17,52.08]

D remains fixed by construction. One-default sensitivity and R_B interval changes remain unreported.

Q6 / W5. Human validation

Endpoints, assertions and policy judgments lack independent human validation. Current validation uses deterministic native checks and model review. E3=0 records no separate full-task local-endpoint audits. Assertion sensitivity on five Retail tasks and 135 episodes permits benefits and harms. Shared simulator/reviewer errors remain unmeasured.

Q7 / W7. Position

We tested full reversal. Middle-position interventions remain untested, so primacy and recency are unresolved. We will frame the contribution as showing how record order changes removal-based evaluation.

Q8 / W6. Statistical sensitivity

DomainCustomersApproximate detectable difference
Retail156.7–26.3
Airline1319.4–30.5

Paired-t design sensitivity uses observed customer-level variance, alpha=.05 and power=.80. Reported intervals are nominal and pointwise, without multiplicity adjustment. AgentDojo estimates describe fixed application worlds. Small samples and filtering limit generalization.

Q9 / W1. Richer plans and later decisions

Luna generated plans freely. Edits reversed existing orders. Model screening admitted 3/24 states. Mini ran three repeats per condition and role.

TaskDeveloper D_AAssistant D_A
airline_330.000.00
banking_150.0033.33
travel_1866.670.00

Developer travel_18 has R_A=0. The banking assistant effect has D_B=0 and involves non-target/stopping changes. Both roles contain external planner text.

At common later histories after an additional native read, 3/8 travel states admitted edits.

Later travel_18 conditionABOTHER
Natural plan030
Reversed plan300
No plan021

Here D_A=D_B=100 and R_B=33.33. The other two states show zero switching. Edits include multi-step blocks with preserved dependencies. Plans enter at the measured checkpoint. Earlier-plan persistence remains untested. We will scope claims to supplied priorities and checkpoint decisions.

W8–W9. Presentation

We will use one name per quantity and move assignment rules, role allocation, task counts, audit coverage and clustering into the main text. C=p_A+p_B excludes BOTH. Legacy p_AB includes it.

Thank you for the detailed response. The target-assignment breakdown, explanation of Gemini’s OTHER outcomes, and additional-default runs clarify several specific questions. I also appreciate the naturally generated plans and later-checkpoint experiments, which extend the evidence beyond the original setup.

My remaining concern is the significance and scope of the contribution, as raised in W1, W2, and W7. The additions strengthen the evidence for the distinction between removal effects and priority responsiveness. However, the broader relevance to plan evaluation remains limited, and the practical occurrence of the proposed interpretation error is still unestablished. Neither limitation invalidates the methodological observation, but together they leave me unconvinced that the contribution is substantial enough for acceptance. The response has addressed several concerns, but my overall recommendation remains weak rejection.

Reviewer_qRG1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a measurement pitfall in evaluating plan influence for tool-using LLM agents, the default trap. When an agent's default behavior already aligns with a plan's priority, removing that plan shows a small effect, which may be misinterpreted as weak plan influence. The paper introduces a paired-priority protocol where two plans differing only in a priority span are compared against a shared no-plan reference. An accounting identity (D_A = R_A + G_A) relates priority-switching effects to removal contrasts. Three studies demonstrate: (1) strong priority control (67–99 pp) alongside small removal effects (6–24 pp) in 3,200 decision windows; (2) default behavior shifts by 63–98 pp when account-list order is reversed while priority control stays at 97–100 pp; (3) task success differences of −19.4 to +8.3 pp relative to no plan, with all Retail/Airline intervals including zero.

Critical strengths

The default trap is a real measurement pitfall. The paired-priority design is a rigorous way to separate directional control from removal effects. The list-order intervention, which reverses account-list presentation while fixing semantic identities and plans, provides a controlled causal test that defaults are presentation-dependent. These methodological contributions have direct practical value for researchers evaluating plan influence in agent systems.

Critical weaknesses
  1. The core insight is a valid methodological point but not a deep finding. The experimental evidence confirms an intuitively obvious prediction: if the agent already does X by default, removing an "X-first" plan won't change behavior, but switching to a "Y-first" plan will. The list-order experiment adds substance by showing that defaults are presentation-dependent, but this is a well-documented phenomenon (position bias).

  2. All 12 Retail/Airline plan-vs-no-plan success intervals include zero, with differences ranging from −6.7 to +5.1 pp. The component study shows that single priority sentences produce the same strong local control as full plans (Table 8: 86.7–100.0 pp). Together, these results suggest that, in the studied tasks, local action selection is highly controllable but plans do not detectably improve task completion.

Questions for authors

Future Work:

Have you considered extending the protocol to later decision points? The current decision window ends at the first measured read. Plans might show more value in coordinating sequences of actions rather than redirecting a single initial choice.

1 comment

Thank you for highlighting the measurement contribution and the question of later decisions. We added later-checkpoint experiments and a replication with additional default executions. Behavioral and success contrasts are in percentage points (pp). Token changes are relative percentages.

Our contribution concerns how plan influence is measured and interpreted. The accounting identity organizes the contrasts, and the presentation intervention demonstrates how their interpretation changes for the same executor. Retail/Luna gives the following results.

Account-list orderA-directed removalB-directed removalA-directed switching
Original3.3100.0100.0
Reversed100.00.0100.0

Target identities, plan text, and business state remain fixed. A-directed removal changes from 3.3 to 100.0 pp while switching remains at 100.0 pp. An assessment of weak versus strong influence based on removal therefore depends substantially on presentation, even with unchanged measured priority responsiveness.

The list-order experiment uses presentation sensitivity to expose this dependence in plan-ablation evaluations. We will sharpen the contribution around estimand interpretation and an operational protocol for measuring directional responsiveness alongside default alignment. The protocol retains OTHER, BOTH, stopping, and errors, allowing target switching and designated-branch coverage to be assessed over the full outcome population.

We agree that a single priority sentence can produce strong local control and that task-completion gains from full plans remain unresolved. The current evidence supports the following conclusions.

EvaluationEvidenceInterpretation
Retail/Airline sentence-only controls86.7–100.0 ppA single priority sentence can supply strong local control
AgentDojo/Gemini sentence-only controlD_A = 25.0 ppSentence responsiveness varies across settings
Retail/Airline task successAll 12 intervals include zero, with differences from −6.7 to +5.1 ppThe direction and magnitude of task-level effects remain uncertain
Airline/Gemini executor tokens47.5%–52.6% higher under plansPlans incur additional execution cost in this setting

The success study includes 15 Retail and 13 Airline customers. Its uncertainty intervals accommodate both benefits and harms. We will explicitly scope the 86.7–100.0 range to Retail/Airline and report AgentDojo/Gemini separately. We will also align the paper's claims with the evidence for local responsiveness and the unresolved contribution of multi-step plan structure to task utility.

For later decisions, we executed an additional native task-relevant read and intervened at the resulting common history. Three of eight prespecified travel states admitted a valid order reversal in a freely generated remaining plan. Mini executed three repeats per condition with assistant placement.

TaskNatural planReversed planNo planD_A / D_BR_natural
travel_143 A3 A3 A0 / 00.00
travel_173 OTHER3 OTHER1 A, 2 OTHER0 / 0−33.33
travel_183 B3 A2 B, 1 OTHER100 / 10033.33

D_A and D_B measure directional switching. R_natural measures removal relative to the natural plan's priority, respectively A, A, and B.

Travel_18 exhibits the switching/removal distinction at a later checkpoint. The other two states show zero switching. These exploratory results cover three eligible states in one travel world, with plans introduced at the matched checkpoint. They extend the protocol beyond the initial decision. The broader question of sequence coordination calls for following a persistent plan across multiple actions and measuring task completion.

A separate natural-plan experiment admitted 3 of 24 initial states and examined message role.

TaskMessage roleObserved result
travel_18DeveloperD_A = 66.67 pp, R_A = 0 pp
travel_18AssistantZero switching

This experiment shows that responsiveness can depend on message placement. We report all eligible cases and both roles. The external Luna planner used visible task context, with reference plans and default outcomes excluded from its inputs.

Finally, the additional-default replication gives the following results.

DomainStatesA-directed switchingA-directed removal
Retail887.5031.25
Airline895.8322.92
AgentDojo887.5029.17

Each state receives three executions per constructed plan and six default executions, all on the same date. Retail/Airline removal estimates differ by 12.50 pp between the two default halves. We report this sensitivity alongside the switching/removal distinction to make the uncertainty in the baseline estimate explicit.

Reviewer_jKWGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a paired-priority evaluation protocol that compares two opposing priorities against a no-plan baseline and varies the order in which information is presented. It shows that a small behavioral change after removing one plan does not imply that the model is unresponsive to alternative plans. The authors therefore recommend reporting local behavioral control, default choices, and task outcomes together.

Critical strengths
  1. The paper identifies a genuine source of confusion in evaluation and supports its argument with clear experiments. If a model already favors A without a plan, an A-first plan has limited room to increase A selection. This does not mean that a B-first plan cannot redirect its behavior. For Retail/Luna, the probability of selecting A is 90.89% under A-first, 84.90% without a plan, and 0.26% under B-first. The authors correctly distinguish the questions answered by these interventions, without suggesting that a small removal effect conceals a larger “true” effect.

  2. The crossed intervention on list order and plan priority is convincing. The authors reverse the account-list order while holding target identities, plan text, and business state fixed. Default A-selection probabilities shift by 63.3–98.3 percentage points, while A-directed priority control remains at 96.7–100.0 points under either ordering. This provides direct experimental evidence that presentation changes the default reference distribution, making conclusions drawn from a single direction of plan removal sensitive to the evaluation context.

  3. The statistical analysis and interpretation are appropriately cautious. The analysis retains OTHER, BOTH, errors, and stopping events rather than restricting effects to runs that select A or B. It uses paired customer-cluster bootstrapping for Retail and Airline and checks sensitivity to alternative weighting schemes. The authors also distinguish repeated executions from independent tasks, statistical nonsignificance from equivalence, zero-width empirical intervals from population invariance, and input responsiveness from plan retention. Comparisons among single priority sentences, full plans, and neutral guidance further guard against attributing the observed effects solely to multi-step plan structure.

Critical weaknesses
  1. The experiments primarily measure compliance with explicit local instructions, limiting their relevance to broader plan evaluation. The core experiments select states with two information targets that can be read in either order without changing the business state. Plans are supplied through a developer message, remain visible throughout execution, and are evaluated mainly through the first target read. Appendix E further shows that a single priority sentence yields directional effects of 86.7–100.0 percentage points on Retail and Airline.

  2. The strongest supported conclusion is therefore that models can follow persistent, high-priority instructions about local read order, producing behavior that differs from the default. The experiments do not cover maintaining subgoals across multiple decisions, handling dependencies, revising plans after tool observations, or choosing alternative strategies later in a trajectory. This limitation partly follows from the protocol itself. Appendix A.1 retains only 4 of 40 workspace tasks and 2 of 16 banking tasks from AgentDojo, excluding cases with serial dependencies, among others. Such filtering improves experimental control but also removes many tasks that make planning difficult. The authors acknowledge that they study a narrower form of input responsiveness, so this is a limitation of scope rather than a conceptual error.

  3. The framework identifies a possible misinterpretation but does not yet show how it changes practical evaluations or system choices. Equation (2) is an accounting identity obtained by adding and subtracting the same reference probability, and positional effects on selection are already established. The useful contribution is to bring these observations into a controlled protocol that distinguishes different intervention targets. However, the experiments mainly use constructed A-first/B-first comparisons. They do not establish how often default alignment affects plans produced by actual planners, or whether jointly reporting these measurements changes the assessment of a planner or executor.

  4. The full-task evaluation contains too few independent tasks to characterize the relationship between local control and task utility convincingly. The 1,080 full-task executions cover only 40 tasks: 15 Retail, 13 Airline, and 12 AgentDojo tasks. Crossing three models, three conditions, and three repeats increases the number of observations without a corresponding increase in task diversity. All success-difference intervals for Retail and Airline include or touch zero, and some still allow practically meaningful benefits or harms. These results support the conclusion that changing a local choice does not, by itself, demonstrate a task-level benefit. They do not yet establish which tasks benefit from a supplied priority, when it is harmful, or whether local control predicts downstream gains. The authors correctly avoid interpreting nonsignificance as evidence of no effect; nevertheless, the full-task results remain less informative than the core local experiments.

Questions for authors

Does the default-trap pattern persist at later decision points or in tasks with sequential dependencies? How does it change when plans are supplied as agent-generated text rather than persistent developer instructions?

1 comment

Thank you for the careful review. We added experiments with naturally generated plans, assistant placement, later decisions, and repeated defaults.

D_q is the selection probability of target q under q-first minus that under the opposite priority. R_q is its selection probability under q-first minus that without a plan. Effects are in percentage points (pp).

1. Naturally generated plans and message roles

Luna generated plans from visible task context, excluding A/B labels, reference plans, and default outcomes. We retained each original plan and reversed an existing order through a dependency-preserving edit. Model-based screening admitted 3/24 initial states. Mini executed each condition three times per role.

TaskDeveloper D_AAssistant D_A
airline_330.000.00
banking_150.0033.33
travel_1866.670.00

In developer/travel_18, the natural plan and no-plan condition select A in 3/3 runs. The reversed plan selects B in 2/3, giving R_A=0 and D_A=66.67.

Assistant placement yields zero observed switching for travel_18. Banking's assistant effect includes non-target/stopping changes, with D_B=0. Both roles use text from the external Luna planner. The experiment measures responsiveness to this text under each role.

2. Later decisions and dependencies

We intervened at common histories after an additional native task-relevant read. Three of eight prespecified travel states admitted a valid remaining-plan edit. All eligible cases appear below, with three repeats per condition and assistant placement.

TaskNatural planReversed planNo planD_A / D_BR_natural
travel_143 A3 A3 A0 / 00.00
travel_173 OTHER3 OTHER1 A, 2 OTHER0 / 0-33.33
travel_183 B3 A2 B, 1 OTHER100 / 10033.33

R_natural follows the original priority target, respectively A, A, and B. Later/travel_18 shows a 33.33 pp removal effect alongside a 100 pp priority effect. The discrepancy therefore also appears at a later checkpoint under assistant placement.

These states share one travel world. Some edits reorder multi-step subplan blocks while preserving internal dependencies. Plans are inserted at the checkpoint, where we measure responsiveness to the remaining plan. With three repeats, one outcome changes an estimate by 33.33 pp. Cross-decision plan retention and revision after new observations remain open questions.

3. Evaluation target and practical interpretation

Equation (2) is an accounting identity. The contribution is the controlled protocol separating departure from a default from responsiveness to an alternative priority.

Reorderable reads preserve feasibility and business state. Paired plans share a common body. Sentence-only controls quantify the contribution of the priority instruction. These controls identify the local behavioral response measured by the protocol.

The natural-plan example makes the evaluation consequence concrete. Removal reports zero effect for developer/travel_18, while priority switching reveals responsiveness to an alternative. Joint reporting captures both properties of the same executor.

Airline/GeminiNoneA-firstB-first
Full-task success30.8%30.8%33.3%
Executor-token increase vs. none—47.5%52.6%

Plans increase executor tokens while completion benefits remain uncertain. Joint reporting connects responsiveness, completion, and cost.

4. Replication and precision

We reran all three conditions on 24 states, with three executions per constructed plan and six defaults, all on the same date.

DomainStatesD_AR_A
Retail887.5031.25
Airline895.8322.92
AgentDojo887.5029.17

Retail/Airline R_A differs by 12.50 pp between the two three-draw default halves, quantifying baseline sensitivity.

Repeated executions characterize execution variability. The full-task study covers 40 tasks, including 15 Retail and 13 Airline customers, across 1,080 executions. Its success-difference intervals permit meaningful benefits and harms. Broader independent-task coverage is needed to characterize which tasks benefit and how local responsiveness relates to downstream utility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.