- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how a supplied plan's priority changes action selection in tool-using LLM agents and names the default trap: interpreting a small change on removing a default-aligned whole plan as weak responsiveness to alternative priorities. Paired plans replace a contiguous priority span and are compared with a shared no-plan reference, and an accounting identity, D_A = R_A + G_A, relates the two contrasts. Across 3,200 decision windows, switching priorities strongly redirects two models' choices, reversing account-list order shifts default target selection by 63.3–98.3 points, and 1,080 full-task episodes show success differences of −19.4 to +8.3 points relative to no plan. The paper concludes that responsiveness, defaults, and task outcomes should be evaluated together.
Critical strengths
Matched design with a shared no-plan reference. The paper contributes "a matched, bidirectional measurement protocol with an accounting identity". Each text condition defines three input conditions: P_A, P_B, and a shared no-plan condition ∅ that deletes the full plan field. Five executions repeat each plan twice and execute no plan once, with identical initial state, history, tools, and continuation instructions. This makes the plan-to-action contrasts internally valid. The same restriction limits what they say about plans (see W1).
Every category remains in the analysis. All eight endpoint categories remain in the denominator, and the full outcome vocabulary identifies another source of removal effects. In Airline/Mini, R_B = 91.96 combines a 68.75-point decrease in A selection with a 23.21-point gain in branch coverage. Coverage also explains differences between the two directional effects: in AgentDojo/Luna, the 7.64-point gap between D_A and D_B equals the A-plan versus B-plan coverage difference. Retaining these outcomes makes changes in selection and stopping behavior visible within the same outcome population.
Tool and token counts complement success. In Airline/Gemini, mean executor tokens rise from 90.7 thousand to 133.8 and 138.5 thousand, increases of 47.5% and 52.6%, and the two absolute token-difference intervals are positive, while success is 30.8%, 30.8%, and 33.3%. AgentDojo/Gemini uses fewer calls under both plans alongside lower observed success, illustrating why calls alone do not measure efficiency. Supplied plans can therefore raise cost without a reliable gain in success, which does not follow from the design and is useful to practitioners.
Critical weaknesses
W1. The intervention supports claims about a priority sentence, not about plans. The paper itself states that "a priority edit measures responsiveness to the supplied priority text". Each pair retains a shared body and edits one priority span, and in the complete frozen plan pair (Appendix C.2) only "ratings" versus "prices" changes. Single priority sentences produce D_A = 86.7–100.0 points in Retail and Airline, and the component study "shows why a large priority response alone cannot establish the value of the surrounding plan structure". The title, "Rethinking Plan Evaluation in Tool-Using LLM Agents", promises more than this, while later decisions and richer plans remain a direction for evaluation, beyond the systems tested here.
W2. The accounting identity adds little, and the default trap is not shown to occur in practice. Adding and subtracting the default probability gives an accounting identity (Equation 2), and the paper notes that "the empirical evidence comes from the implemented input interventions", not from the identity. That a small R_A can accompany a large D_A when default behavior already favors A follows from the positive headroom bound R_k ≤ 1 − p_k(∅), which the paper states in Section 3.2. The default trap is defined as the risk of interpreting a small removal contrast as weak responsiveness to alternative priorities, but the paper cites no work that drew this conclusion from a whole-plan removal ablation. It is unclear whether the default trap is a problem anyone has actually run into.
W3. The absence of a success effect follows largely from task selection. The paper selects states with two useful reads allowed by the benchmark rules whose order preserves business state. It notes that state preservation does not imply equal information value, cost, or downstream consequences, but the selection still targets choices where either order is permissible. The result that "strong local responsiveness accompanies uncertain completion gains" is therefore expected: supplied priorities reliably change the initial information choice, while task success additionally incorporates subsequent decisions, tool use, and termination. All 12 Retail and Airline plan-versus-default intervals include or touch zero. A setup in which one priority matches the benchmark-provided reference actions and the other does not would test whether control has practical value.
W4. The A/B assignment may produce the removal asymmetry. In the paired-priority study, default A probability ranges from 66.67% to 89.29% while default B probability is at most 18.06% (Table 3), and across the six groups, B-directed removal effects exceed A-directed effects. The paper concludes that "reporting either removal direction alone therefore yields a different impression of the same executor's responsiveness to priorities". In the list-order intervention, A and B identify the first and last records in the original order, so the default there reflects position. For the paired-priority study, each task fixes two useful information targets A and B, but the paper does not say how a target is assigned to A. Without this, the unequal removal effects may come from labeling rather than from the executor.
W5. The same model fills most evaluation roles, and model-based judgments are not validated by humans. In the follow-up studies, Mini prepares and reviews the plan pairs, simulates the user, judges required natural-language assertions, performs policy review, audits local endpoints, and is one of the executors. The paper acknowledges that Mini's role as executor, simulator, and reviewer permits shared errors, but it does not measure them. In the endpoint audit, agreement is with model review, not human ground truth, and the enriched sample does not estimate population classification accuracy. Full-task local endpoints are not audited (Table 7, 0 audited).
W6. Small, heavily filtered samples. The list-order intervention and component study use 20 tasks per domain, and the full-task evaluation uses 15 Retail and 13 Airline tasks. Success intervals are wide, for example [−22.2, 6.7] for Retail/Mini B minus A, and the paper states that these intervals leave practically important benefits and harms unresolved. Intervals are nominal, pointwise percentile intervals without multiplicity correction, and there is no power analysis. Each wording runs each plan twice and no plan once, so the default, which is the reference in every R_k, rests on half as many draws as each plan. AgentDojo results describe four fixed application worlds and have no intervals, although resampling tasks would give them. The task pool is heavily filtered: AgentDojo retains four of 40 workspace and two of 16 banking tasks, and the paper notes that task composition changes effect size, with Retail D_A for Luna ranging from 75.69 to 98.75 points across source groups (Figure 6). AgentDojo/Gemini has D_A = 33.3 with substantial OTHER probability in the full-task evaluation and 25.0 under single priority sentences, which contradicts the abstract's claim that the component study "finds strong control under single priority sentences". The paper does not discuss this group.
W7. The list-order finding is largely known. The paper states that position bias means selection changes when an item's position changes with its content fixed, and documents it through work it cites (Lu et al., 2022; Blankenstein et al., 2026; Wadi & Ma, 2026). The contribution is in showing that "an ablation report can therefore change substantially when the same records are presented in a different order", and the paper should state its contribution that way. Only a full reversal of the account list is tested, so primacy and recency effects cannot be separated.
W8. One quantity has several names. The paper introduces "priority control, also called directional control", and D_A further appears as directional effect, priority switching and priority responsiveness. R_k appears as removal contrast, plan-minus-none contrast, plan-versus-default contrast and A-directed removal effect. In Appendix E the legacy field p_AB includes BOTH, while designated-branch coverage C = p_A + p_B excludes BOTH. The reader has to rebuild these correspondences to follow the main claim. A glossary and one name per quantity would fix this.
W9. Key details are in the appendices. The abstract is dense with qualifications, while information needed to judge the main claim, such as how A and B are assigned, how endpoints are audited, and the customer-cluster intervals, appears only in the appendices.
Questions for authors
-
A/B assignment (W4). Each task fixes
two useful information targets A and B, andthe selected endpoint is defined before formal execution. How was a target assigned to A in the paired-priority study? Was A chosen as the target the executor reads first without a plan, the first one mentioned in the dialogue, or by another rule? If the labels were swapped or randomized, wouldB-directed removal effectsstillexceed A-directed effects? -
Evidence for the default trap (W2). Can you point to published work that reported a small
whole-plan removalcontrast and interpreted it asweak responsiveness to alternative priorities? Without such cases, how often does this misreading occur in practice? -
Priorities that differ in correctness (W3). The full-task evaluation requires
benchmark-provided reference actions that can be replayed. Have you tested pairs in which one priority follows the reference action order and the other does not? This would show whether priority control affects task success when the order matters. -
AgentDojo/Gemini (W6). This group has D_A = 33.3 with
substantial OTHER probabilityin the full-task evaluation and 25.0 under single priority sentences. SinceOTHER denotes another read, which reads does Gemini choose instead of A and B? How does this result fit the abstract's claim that the component studyfinds strong control under single priority sentences? -
Default estimation (W6).
We run each plan twice and no plan once, so every R_k rests on half as many default draws as plan draws. How much do the R_A and R_B estimates and their intervals change with additional no-plan executions? -
Human validation (W5). The paper states that
Mini's multiple roles permit correlated errors, which the deterministic native checks and retained review records help examine. Have any endpoint labels, assertion judgments or policy reviews been checked by humans? Is there a reason theFull-task local endpoint (E3)row of Table 7 has no audited windows? -
Primacy versus recency (W7).
The list-order intervention reverses account records while fixing target identities, plans, and business state. SinceA and B identify the first and last records in the original order or reservation list, can you separate primacy from recency, for example by moving A to a middle position? -
Statistical power (W6). The paper states that the success intervals
leave practically important benefits and harms unresolved and do not establish equivalence. What success difference could the full-task evaluation detect with 15 Retail and 13 Airline tasks? -
Plans beyond a single priority span (W1).
Extending the protocol to later decisions, richer plans, and multi-agent coordination is a direction for evaluation. Do you have any results, even preliminary ones, where paired plans differ in more than one priority span or where the measured decision is not the first read?
3 comments
Thank you for the review. D denotes priority switching and R plan minus none. Effects are in pp.
Q1 / W4. Target assignment
Labels were fixed before formal no-plan runs on previously used tasks.
| Domain | Pair construction |
|---|---|
| Retail | 75/96 first/last account records, 20 product pairs, one requested-order pair |
| Airline | 23/28 first/last records, three requested-reservation pairs, two outbound/return searches |
| AgentDojo | Semantic target pairs |
Swapping labels reverses the R_A/R_B asymmetry. Fair random relabeling equalizes their aggregates in expectation. Label-invariant (D_A+D_B)/2 is 63.54–99.11 across the six original groups. We will report each pair type's exact assignment rule and whether prior task use informed it.
Q2 / W2. Practical occurrence
The identity is an accounting decomposition. Record reversal changes removal effects with switching fixed.
| Retail/Luna | Original records | Reversed records |
|---|---|---|
| R_A | 3.3 | 100 |
| D_A | 100 | 100 |
We have not identified verified published examples or estimated prevalence. We will describe the default trap as a possible interpretation error.
Q3 / W3. Correctness
Correctness-sensitive pairs remain untested. Admissible reorderings limit conclusions about the value of correct ordering. Such tests require verified preconditions or information dependencies. Completion benefits and harms remain unresolved in these selected tasks.
Q4 / W6. Gemini OTHER
| Reads within OTHER | Count in each sentence condition |
|---|---|
| City-level hotel, restaurant or car-rental enumeration | 36/45 |
| Channel enumeration | 3/45 |
| Webpage reads | 3/45 |
| Channel-message reads | 3/45 |
AgentDojo/Gemini has D_A=25.0 for single sentences and immediate D_A=33.3 in full-task runs. We will restrict the abstract's strong sentence-only control claim to Retail and Airline and report this exception.
Q5 / W6. Additional defaults
The Mini follow-up used 288 windows on 24 states, with three runs per plan and six defaults on one date. Plan runs stay fixed in the 3 → 6 comparison.
| Domain | R_A, 3 → 6 | R_B, 3 → 6 | D_A |
|---|---|---|---|
| Retail | 25.00 → 31.25 | 75.00 → 68.75 | 87.50 |
| Airline | 16.67 → 22.92 | 100.00 → 100.00 | 95.83 |
| AgentDojo | 29.17 → 29.17 | 79.17 → 81.25 | 87.50 |
| R_A interval | First 3 defaults | All 6 defaults |
|---|---|---|
| Retail | [0,62.5] | [4.17,62.5] |
| Airline | [−4.17,41.67] | [−4.17,52.08] |
D remains fixed by construction. One-default sensitivity and R_B interval changes remain unreported.
Q6 / W5. Human validation
Endpoints, assertions and policy judgments lack independent human validation. Current validation uses deterministic native checks and model review. E3=0 records no separate full-task local-endpoint audits. Assertion sensitivity on five Retail tasks and 135 episodes permits benefits and harms. Shared simulator/reviewer errors remain unmeasured.
Q7 / W7. Position
We tested full reversal. Middle-position interventions remain untested, so primacy and recency are unresolved. We will frame the contribution as showing how record order changes removal-based evaluation.
Q8 / W6. Statistical sensitivity
| Domain | Customers | Approximate detectable difference |
|---|---|---|
| Retail | 15 | 6.7–26.3 |
| Airline | 13 | 19.4–30.5 |
Paired-t design sensitivity uses observed customer-level variance, alpha=.05 and power=.80. Reported intervals are nominal and pointwise, without multiplicity adjustment. AgentDojo estimates describe fixed application worlds. Small samples and filtering limit generalization.
Q9 / W1. Richer plans and later decisions
Luna generated plans freely. Edits reversed existing orders. Model screening admitted 3/24 states. Mini ran three repeats per condition and role.
| Task | Developer D_A | Assistant D_A |
|---|---|---|
| airline_33 | 0.00 | 0.00 |
| banking_15 | 0.00 | 33.33 |
| travel_18 | 66.67 | 0.00 |
Developer travel_18 has R_A=0. The banking assistant effect has D_B=0 and involves non-target/stopping changes. Both roles contain external planner text.
At common later histories after an additional native read, 3/8 travel states admitted edits.
| Later travel_18 condition | A | B | OTHER |
|---|---|---|---|
| Natural plan | 0 | 3 | 0 |
| Reversed plan | 3 | 0 | 0 |
| No plan | 0 | 2 | 1 |
Here D_A=D_B=100 and R_B=33.33. The other two states show zero switching. Edits include multi-step blocks with preserved dependencies. Plans enter at the measured checkpoint. Earlier-plan persistence remains untested. We will scope claims to supplied priorities and checkpoint decisions.
W8–W9. Presentation
We will use one name per quantity and move assignment rules, role allocation, task counts, audit coverage and clustering into the main text. C=p_A+p_B excludes BOTH. Legacy p_AB includes it.
Thank you for the detailed response. The target-assignment breakdown, explanation of Gemini’s OTHER outcomes, and additional-default runs clarify several specific questions. I also appreciate the naturally generated plans and later-checkpoint experiments, which extend the evidence beyond the original setup.
My remaining concern is the significance and scope of the contribution, as raised in W1, W2, and W7. The additions strengthen the evidence for the distinction between removal effects and priority responsiveness. However, the broader relevance to plan evaluation remains limited, and the practical occurrence of the proposed interpretation error is still unestablished. Neither limitation invalidates the methodological observation, but together they leave me unconvinced that the contribution is substantial enough for acceptance. The response has addressed several concerns, but my overall recommendation remains weak rejection.
In response to the Program Chairs’ concern about AI use, I used Grammarly to improve the readability of my review. I identified the paper’s strengths and weaknesses myself. I did not use any AI tools to identify those strengths or weaknesses, form my overall assessment, or determine my score.