acceptodds
Under review as a conference paper at ICLR 2027

PLOT: Compiling Policy Documents into Decision Tools for LLM Agents

Abstract

LLM agents must follow domain-specific policies when using tools to automate business workflows. Beyond retrieving policy text, they must combine rules across documents, evaluate conditions and exceptions against the current task state, and decide which actions to take and how to execute them through tools. Repeating this interpretation during execution and across tasks can lead to missed requirements and inconsistent decisions. We introduce PLOT, which enhances agent actions with reusable decision tools constructed on demand. For each request, PLOT explicitly declares the required policy-dependent decisions, such as eligibility or permissible amounts, and resolves them by retrieving previously constructed decision tools or building new ones grounded in relevant policy documents. Each tool takes task-specific facts as inputs and applies reusable policy logic to compute a policy-dependent decision. Tools that pass admission checks on source references and code structure enter a decision tool library, and a separate execution agent calls them to complete the task request. We evaluate PLOT on τ -Knowledge and AutomationBench using GPT-5.2 and DeepSeek-V4-Flash. PLOT outperforms the strongest baseline in every setting, improving passˆ1 on τ -Knowledge by 3.8% and partial credit on AutomationBench by 4.7%, on average across the models. It also uses 11.3% fewer tokens than ACE on average. Ablations on τ -Knowledge suggest benefits from slot declaration, admission checks, and cross-task decision tool reuse. Replacing executable decision tools with cited natural-language rules reduces task success from 36.1% to 20.6%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agents, Tool Use, Tool Generation, Policy Adherence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SrMaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PLOT aims to transfer pilicy documents into executable and reusable tools for decision making. Instead of re-read and re-inference in the policy contexts, PLOT treat policy-related judgement as function, and directly call the function when execute.

Critical strengths
  1. PLOT transforms policy logic from something that must be reinterpreted each time into an executable and reusable tool.
  2. The experimental design is comprehensive. Ablation studies convincingly demonstrate the contribution of most design components.
Critical weaknesses
  1. The novelty is relatively low. The core idea of ​​compiling natural language policies into executable code before invoking them is not conceptually surprising. In essence, it represents a systematic engineering integration of several existing approaches.
Questions for authors

The core of the paper is cross-task reuse. However, experiments show that it seems only true on τ-Knowledge, but the reuse rate on AutomationBench is close to 0. The authors attribute this to "dataset dependency" in discussion (sec 5). Then, is PLOT's reuse advantage essentially only applicable to scenarios where "multiple tasks share the same policy corpus"?

1 comment

Novelty & Contribution of PLOT

The design goal of PLOT is to improve how agents make decisions in policy-governed environments. Reviewer xqkW highlighted the combination of task-specific decision slots, document-grounded compilation, and source-aware reuse as PLOT’s main distinction from prior work. A user request often leaves policy judgments implicit, and PLOT makes these judgments explicit as decision slots. Each slot guides the agent to the governing documents. The agent implements their rules as a callable function of task-specific facts. Admission checks verify that the function matches its declared signature and that the cited documents were read. They also screen for unused inputs and constant returns. This gives the agent a source-grounded way to evaluate each judgment when choosing actions, setting their arguments, and deciding their order.

PLOT compiles each policy-dependent decision into a tool, written from the policy documents, before the agent acts. Tool-making methods such as LATM instead build a function from a few demonstrations of a task, so a policy check missing from the demonstrations is also missing from the function. Code-as-action agents such as CodeAct make policy decisions within the task program itself. If the program executes an action before checking whether the policy permits it, the check comes too late to prevent a potential violation. As Reviewer RJzo noted, the comparison agent in our banking example “had read both relevant policies, but handled the requests in an order that made the final request impossible.” Guardrail methods such as ToolGuard compile policies into checks that accept or reject an action the agent has proposed, while a PLOT tool returns the decision that informs which action to take.

Clarification of Reuse

We would like to clarify that in τ-Knowledge, different banking tasks share a policy corpus and can call the same tool with new facts when the same decision recurs under the same rules. However, AutomationBench is designed to test agents on a diverse set of business workflows, with each task having its own simulated business scenario. This characteristic of AutomationBench makes reuse impossible.

Reviewer_xqkWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PLOT, a framework that compiles policy documents into reusable decision tools for LLM agents. Given a user request, a compilation agent identifies the required policy-dependent decisions, declares their interfaces, and retrieves or constructs Python functions that implement the relevant rules. Functions that pass source-reference and structural checks are retained in a library. A separate execution agent calls these functions with task-specific information to complete the request. The paper evaluates PLOT on τ-Knowledge and AutomationBench using two base models, showing improvements over Vanilla, CodeAct, and ACE.

Critical strengths
  1. The paper tackles an important problem.

  2. The proposed method is reasonable. Separating reusable policy logic from task-specific facts provides a concrete interface for applying the same rules across requests. Explicitly declaring the required decisions also makes the scope of policy preparation easier to inspect. These are reasonable design choices for improving the consistency of agent behavior.

Critical weaknesses
  1. The contribution of the paper over the existing work is not clear. ToolGuard already compiles company policies into executable checks, while Large Language Models as Tool Makers separates tool construction from tool use and supports reuse through a function cache. PLOT cites these approaches, but the experiments compare only with Vanilla, CodeAct, and ACE. Its main distinction seems to be the combination of task-specific decision slots, document-grounded compilation, and source-aware reuse. It would be better to have a comparison that establishes the value of these additions over a document-grounded tool-making baseline with comparable retrieval, execution, and memory capabilities.

  2. The ablations do not cleanly isolate some of the proposed design choices. Appendix A.6 states that removing slot declaration also disables automatic handoff upon slot completion. This changes both the declaration mechanism and the allocation of effort between compilation and execution, making the resulting performance and token differences difficult to attribute to slots alone. PLOTtext is a useful control because it preserves the two-phase workflow and several other components. However, it changes both the policy representation and the code-dependent checks.

  3. The evaluation does not directly establish the improvement in procedural policy adherence emphasized by the motivation. The motivating example concerns action ordering, whereas AutomationBench evaluates final application states. Final-state success does not necessarily reveal whether an agent skipped a prerequisite or performed operations in an impermissible order. Partial credit also combines requirements to perform actions with requirements to avoid prohibited changes. A decomposition of these outcomes would make the improvements easier to interpret.

Questions for authors

See the weaknesses.

1 comment

Contribution of PLOT and comparison with a comparable tool-making baseline (W1). The contribution of PLOT is to improve how agents make decisions in policy-governed environments. PLOT declares the decision slots, compiles the rules governing each decision into an executable function, and preserves the policy sources so later agents can check whether the same rules apply. As Reviewer Y7gJ noted, these functions “bind policy logic to explicit input arguments and document provenance.” Tool-making methods such as LATM [1] instead build a function from a few demonstrations of a task, so a policy check missing from the demonstrations is also missing from the function.

Following the suggestion, we compared PLOT with the official LATM implementation [2] with policy retrieval, code execution, and a persistent function cache. On τ-Knowledge with DeepSeek-V4-Flash, we observed that PLOT substantially outperformed this baseline (36.8% vs. 27.8% in pass^1). A task-level function can pass LATM’s verification examples and still miss a condition that determines when a rule applies. For example, in one ATM fee audit, LATM’s checker passed all three verification examples and applied the out-of-network fee rule to a withdrawal at the bank’s own ATM, leaving a $3 refund shortfall. PLOT scoped its domestic fee functions to out-of-network withdrawals and flagged the in-network fee waiver in the handoff, so the execution agent issued the full refund.

Methodpass¹ (%)pass² (%)pass³ (%)
LATM with document grounding27.817.211.3
PLOT36.828.925.8

Ablation Interpretation and Complementary Controls (W2)

Slot declaration and slot-completion handoff are intentionally coupled in PLOT: declared slots define what must be compiled, and their completion determines when the compiled exploration context is integrated into the execution prompt. Thus, removing slot declaration necessarily removes this slot-based completion signal, rather than independently ablating a separate handoff component. Similarly, PLOT–PLOT_text is intended to compare the compiled decision representation as a whole: executable decision tools with their associated checks versus reusable textual policy knowledge.

In addition, we added complementary controls to improve attribution. PLOT w/o decision tools retains declared decisions, their inputs, and the same structured handoff, but removes decision functions, reducing τ-Knowledge pass^1 from 36.8% to 29.9% and AutomationBench partial credit from 57.4% to 52.3%. PLOT w/o admission checks retains executable decision tools but removes citation and function-shape checks, reducing these scores to 24.4% and 55.2%, respectively. Together, these ablations provide more targeted evidence for the contributions of compiled decision tools and admission checks within PLOT.

Additionally, CodeAct + read-only policy exploration reaches 28.9% pass^1, below standard CodeAct (33.3%) and PLOT (36.8%). Thus, the read-only preparation phase and ordered handoff alone do not reproduce PLOT’s gains in these experiments.

Report the detailed performance gain on AutomationBench (W3). We checked the pass rates for both required-outcome and prohibited-outcome checks in AutomationBench. Compared with the strongest baseline by original partial credit for each backbone, PLOT improved both rates with GPT-5.2 (+5.90 and +5.52 percentage points). With DeepSeek-V4-Flash, prohibited-outcome avoidance improved by 4.32 points and required-outcome satisfaction decreased by 0.74 points. In conclusion, in most settings, we observed that PLOT improved performance in both directions: completing more required operations and avoiding prohibited operations.

[1] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. “Large Language Models as Tool Makers.” In The Twelfth International Conference on Learning Representations (ICLR), 2024.

[2] https://github.com/ctlllll/LLM-ToolMaker

Reviewer_RJzoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

An agent can read the right policy and still take actions in the wrong order. In the paper’s banking example, a customer asks to dispute a charge, replace a card, and raise a credit limit. Filing the dispute or ordering the card first would temporarily block the increase. PLOT tries to catch such dependencies before anything is changed. One agent identifies the decisions the request needs, retrieves an existing decision function or builds a new one from the policies, and hands it to a second agent. The second agent calls the function with the customer’s current information and carries out the request. New functions are saved for later tasks.

The authors test PLOT on 97 banking tasks in τ-Knowledge and 120 business workflows in AutomationBench, using two models. Against the strongest baseline for each comparison, average banking task success rises by 4.1 percentage points with GPT-5.2 and 3.4 points with DeepSeek-V4-Flash. On AutomationBench, the gains in partial credit—the share of task checks passed—are 7.9 and 1.4 points, respectively. The paper also reports 11.3% fewer tokens than ACE, on average.

Critical strengths

The banking example makes the problem easy to see. One comparison agent had read both relevant policies, but handled the requests in an order that made the final request impossible. The failure happens after the policies are found, which makes it a good test for PLOT.

I like the split between policy rules and customer facts. The ATM-fee example shows a function built to check past charges being used later to compare account options for a different customer.

The authors inspected the functions they generated, reported incorrect ones, and examined unsuccessful reuse as well as successful reuse. Their token accounting includes both PLOT’s compilation work and ACE’s playbook updates.

Critical weaknesses

The checks do not establish that a function makes the right decision. They check that the code loads and uses its inputs, and that any cited sources were read. The authors’ manual review still finds 42 incorrect functions among 1,172 saved functions, plus two with conflicting source policies. One function, for example, accepts a payment marked “Dept Head: Approved; CFO: Denied” because it searches separately for “CFO” and “Approved.” Since a saved function may be called again later, it would help to know how often such errors affected task outcomes. An independent check of a sample of the correctness judgments would also make the reported 96.2% figure easier to assess.

The baseline comparison needs more detail. ACE, which stores written playbook advice, scores 8.6 points below Vanilla, a standard tool-using agent, on τ-Knowledge with both models. More detail on how ACE’s playbook was updated and what feedback it received would help explain this gap. The GPT-5.2 Vanilla banking result comes from released benchmark trajectories, while the other results come from the authors’ runs; the paper should make clear that task setup and token accounting are comparable.

Evidence for cross-task reuse is confined to banking. Those tasks share one policy collection. AutomationBench covers more domains, but its policies differ by task and none of its saved functions are reused. The paper should frame the claim about reusable policy tools within that evidence and explain more sharply what its combination adds to earlier methods that turn policies into code or save tools for later use.

Questions for authors

Could you compare PLOT with a version that lists the needed decisions and gives the execution agent the same ordered handoff, but does not generate decision functions? What accounts for the AutomationBench gain when there is no cross-task reuse?

Could you report paired confidence intervals or tests for the main results and repeat the τ-Knowledge ablations across all three task orders?

How often were incorrect saved functions called in later tasks, and did they cause scored failures? Could a second reviewer check a sample of the function-correctness labels?

What task feedback, if any, did ACE use to update its playbook? How were the released GPT-5.2 Vanilla trajectories matched to the other runs for evaluation and cost?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RJzo (1/2)

Function correctness analysis (W1 & Q3). As described in Section 3.5, PLOT can revise a previously saved function before applying it to the current task. We report an analysis of the reuse patterns. Across six τ-Knowledge runs using GPT and DeepSeek, there are 266 cross-task reuse function calls and 30 updates to previously saved functions. We manually compared the original and revised functions against the governing policies: 10 updates corrected a policy or calculation error, 15 extended rule coverage or adapted a function to a new task or interface, and two changed comments or formatting. For example, a Business Silver rewards calculator originally awarded 1,500 points for a $149.99 purchase. The revised function rounded down to 1,499 points, as required by the policy. However, we identified one later task in which an incorrect saved function contributed to a scored failure and two tasks in which the incorrect function was used but reuse was not the root cause. Reliably translating natural-language requirements into semantically correct code remains an open challenge for LLMs [1]. As discussed in Section 5, verifying that PLOT’s generated tools faithfully implement their source policies requires semantic validation beyond our admission checks.

To further analyze function correctness, we randomly sampled 21 functions originally labeled incorrect and 21 functions originally labeled correct. A second author reviewed these 42 functions using the same criterion. The second author agreed with all 21 original correct labels and judged 12 of the 21 functions originally marked incorrect to have correct core logic, yielding 30/42 agreement (71.4%). The results suggest that the original review applied the criterion conservatively in some cases, particularly in judging whether parsing failures or omitted policy details invalidated the function’s core logic.

[1] Zhe Ye, Zhengxu Yan, Jingxuan He, Timothe Kasriel, Kaiyu Yang, and Dawn Song. “VERINA: Benchmarking Verifiable Code Generation.” In The Fourteenth International Conference on Learning Representations (ICLR), 2026.

Implementation Details (W2 & Q4). (1) We adapted the official ACE implementation to both benchmarks. After each episode, the Reflector reviews the dialogue, retrieved policies, and tool outputs, including errors. The Curator uses the reflection and task instructions to add natural language guidance to the existing playbook for future tasks. After the Figure 1 episode, it added a reminder to complete the account-closure workflow before ordering a replacement card, including asking the customer why they wanted to close the account.

(2) We used the first three officially released trials for each of the same 97 tasks, with the same GPT-5.2 user simulator. They released token count data, and we applied the same model prices.

Cross-task reuse results interpretation (W3). Differences in benchmark design account for the cross-task reuse results in our two benchmarks. In τ-Knowledge, different banking tasks share a policy corpus and can call the same tool with new facts when the same decision recurs under the same rules. However, AutomationBench is designed to test agents on a diverse set of business workflows, with each task having its own simulated business scenario. This characteristic of AutomationBench makes reuse impossible.

AuthorsOpenReview ↗

Response to Reviewer RJzo (2/2)

Contribution of PLOT over existing methods (W3). The design goal of PLOT is to improve how agents make decisions in policy-governed environments. A user request often leaves policy judgments implicit. PLOT makes these judgments explicit as decision slots, and each slot points the agent to the policy documents that govern the decision. The agent then writes a function that applies the relevant rules to facts about the task. Admission checks verify that the function matches its declared signature and that the cited documents were read. They also screen for unused inputs and constant returns. This gives the agent a source-grounded way to evaluate each judgment when choosing actions, setting their arguments, and deciding their order.

PLOT compiles each policy-dependent decision into a tool, written from the policy documents, before the agent acts. Tool-making methods such as LATM [1] instead build a function from a few demonstrations of a task, so a policy check missing from the demonstrations is also missing from the function. CodeAct makes policy judgments within the scripts that execute the task. However, PLOT identifies the required decisions explicitly and compiles the rules governing each decision into a separate callable function. Guardrail methods such as ToolGuard compile policies into checks that accept or reject an action the agent has proposed, while a PLOT tool generates the decision that informs which action to take.

[1] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. “Large Language Models as Tool Makers.” In The Twelfth International Conference on Learning Representations (ICLR), 2024.

Ablation study results on decision functions and AutomationBench (Q1).

Contribution of decision functions

We compared PLOT with PLOT w/o decision functions, which disables generating decision functions but lists the required decisions and gives the execution agent the same structured handoff. For example, in an AutomationBench expense audit task, the preparation agent lists the decision “Which records are flagged as anomalies?” and identifies the expense amount, category, date, and threshold rules as inputs.

Methodpass^1pass^2pass^3
PLOT w/o decision functions29.9%20.6%16.5%
PLOT36.8%28.9%25.8%

Component contributions in AutomationBench

We observed that the full method outperformed the variant without decision functions by 6.9 percentage points.

We ran three ablations on AutomationBench to test how slot declaration, admission checks, and decision functions contribute to PLOT’s performance.

MethodStrict CompletionPartial Credit
Full PLOT18.3%57.37%
PLOT w/o Slot Declaration17.5%54.21%
PLOT w/o Admission Checks18.3%55.15%
PLOT w/o Decision Functions16.7%52.28%

Removing any of the three components lowered partial credit on AutomationBench, with the largest drop coming from removing decision functions.

Report confidence intervals and results from multiple runs (Q2). 1) We compute 95% task-paired percentile-bootstrap confidence intervals for PLOT–baseline differences using 1,000,000 resamples. Each resample draws task IDs with replacement, preserving the pairing between methods. PLOT improved τ success and AutomationBench partial scores with both GPT-5.2 and DeepSeek, with all four 95% confidence intervals entirely above zero. On AutomationBench Strict, the estimated gains were small. This result may reflect the difficulty of AutomationBench’s strict success criterion and the lack of opportunities for tool reuse in the benchmark.

SettingMetricBaselinePLOT − baseline (pp)95% CI (pp)
τ / GPT-5.2SuccessVanilla+4.12[2.06, 6.53]
τ / DeepSeekSuccessCodeAct+3.44[1.72, 5.50]
AutomationBench / GPT-5.2StrictACE+1.67[−1.67, 5.00]
AutomationBench / GPT-5.2PartialVanilla+7.91[6.95, 8.88]
AutomationBench / DeepSeekStrictACE+0.83[−1.67, 4.17]
AutomationBench / DeepSeekPartialACE+1.42[0.94, 1.91]
  1. We evaluated the four original τ-Knowledge ablations with DeepSeek-V4-Flash across all three task orders.
Methodpass^1pass^2pass^3
Full PLOT36.8%28.9%25.8%
PLOT w/o Slot Declaration25.8%17.2%13.4%
PLOT w/o Admission Checks24.4%16.5%13.4%
PLOT w/o Cross-Task Decision Tool Library29.2%21.3%17.5%
PLOT-text25.8%14.8%9.3%
Reviewer_Y7gJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Agents often fail at policy workflows because they modify the environment state while still retrieving and interpreting the governing rules. This paper addresses that problem by introducing a read-only compilation phase that translates text policies into parameterized Python functions before an execution agent acts. The approach combines tool-creation frameworks like LATM and CREATOR with a strict Plan-then-Execute architecture, backed by thorough trace audits, though the empirical gains are modest and partially confounded by the two-stage handoff prompt.

Critical strengths
  1. Prohibiting environment writes during the initial compilation phase prevents premature state mutations, such as filing a dispute that immediately blocks a requested credit limit increase.
  2. Representing policies as parameterized Python functions is more reliable than maintaining natural-language playbooks like ACE. Text summaries can overgeneralize across tasks and trigger out-of-scope actions, whereas functions bind policy logic to explicit input arguments and document provenance.
Critical weaknesses
  1. Credit assignment across the pipeline is unclear. The compilation agent generates a structured OWED handoff note that explicitly orders remaining actions for the execution agent, giving PLOT a 100-step read-only lookahead and action-ordering step that the baselines lack. Furthermore, PLOTtext\text{PLOT}_{\text{text}} drops to 20.6% on τ\tau-Knowledge—12 points below Vanilla—which suggests the two-phase handoff introduces substantial friction unless paired with code, rather than cleanly isolating the benefit of code over text.
  2. The cross-task reuse claim does not hold on AutomationBench, where runtime reuse is 0 out of 120 tasks. All gains on that benchmark stem from single-task decomposition, yet component ablations on AutomationBench are omitted.
Questions for authors
  1. If standard CodeAct is given the same read-only exploration phase and structured handoff prompt without slot declaration or Python tool compilation, how much of the ~4% gain remains?
  2. Given that AutomationBench has zero cross-task reuse, what are the ablation results (without slot declaration and without admission checks) on that benchmark?
1 comment

Thank you for your thoughtful reviews.

Added comparison settings for the contribution of the exploration phase and decision functions (W1 & Q1). For clarity, PLOT’s policy compilation and execution share the same 200-step episode budget as the baselines on tau-knowledge. The 100-step compilation threshold is within this budget.

To assess the contributions of the exploration phase and decision representation, we evaluated two new controls with DeepSeek-V4-Flash over three runs on τ-Knowledge, following the evaluation protocol in Sec. 4.1.

To directly examine the contribution of the exploration phase, we compare CodeAct with its variant:

  • CodeAct + read-only policy exploration. We prepend a read-only exploration phase to standard CodeAct execution. An exploration agent first retrieves relevant policies, gathers customer and account facts, and identifies the remaining actions and their required order. It then passes the conversation, relevant policy documents, and an OWED-ordered handoff note to a CodeAct execution agent, which completes the task using standard CodeAct code actions. The exploration phase uses the same step limit and structured handoff format as PLOT.

The results show that read-only exploration and ordered handoff alone do not reproduce PLOT’s gain: the augmented CodeAct scores 28.9% pass^1 versus 33.3% for standard CodeAct.

Furthermore, to clearly show the contribution of the decision representations beyond PLOT_text, we compare PLOT and PLOT_text with a new variant of PLOT:

  • PLOT w/o decision functions. This variant explicitly identifies the required policy decisions and their inputs and retains PLOT’s structured handoff, but does not provide compiled decision functions to the execution agent.

We observed that PLOT without decision functions reduced pass^1 from 36.8% to 29.9%. These comparisons support the contribution of decision functions within PLOT’s pipeline.

Methodpass^1pass^2pass^3
CodeAct33.3%23.4%18.6%
CodeAct + read only policy exploration [New]28.9%18.6%14.4%
PLOT-text25.8%14.8%9.3%
PLOT w/o decision functions [New]29.9%20.6%16.5%
PLOT36.8%28.9%25.8%

Ablation results on AutomationBench (W2 & Q2). Following the ablation study in Section 4.4, we ran three ablations on AutomationBench to test how slot declaration, admission checks, and decision functions contribute to PLOT’s performance. Here are the results:

MethodStrict CompletionPartial Credit
Full PLOT18.3%57.37%
PLOT w/o Slot Declaration17.5%54.21%
PLOT w/o Admission Checks18.3%55.15%
PLOT w/o Decision Functions16.7%52.28%

We observe that removing any of the three components lowered partial credit on AutomationBench, with the largest drop coming from removing decision functions. Furthermore, in the ablation study on the tau-Knowledge benchmark, removing slot declaration, admission checks, or decision functions lowered task success. We observed that the ablation results were consistent across these two benchmarks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. All changes are marked in blue in the revised PDF.

Strengths noted by reviewers. Reviewers found the problem important (xqkW) and the motivating banking example clear (RJzo). They highlighted the separation of reusable policy logic from task-specific facts as a concrete interface for applying the same rules across requests (RJzo, xqkW, SrMa). Furthermore, they pointed out that compiled functions “bind policy logic to explicit input arguments and document provenance” (Y7gJ), whereas natural-language playbooks can overgeneralize. Reviewers also valued the comprehensive experiments and ablations (SrMa) and our analysis of incorrect functions and unsuccessful reuse (RJzo).

Main concerns and revisions.

  1. Contribution and relation to prior work (RJzo, xqkW, SrMa). PLOT makes each policy-dependent decision the unit of tool construction: it declares the required decisions as policy slots and then compiles the related business policy documents into callable tools. Unlike guards such as ToolGuard, which accept or reject a proposed action, a PLOT tool returns the decision that informs which action to take. We added a contribution list (Section 1) and modified the related work narrative to be clearer (Section 2).

  2. Where the gains come from (Y7gJ, RJzo, xqkW). New controls with DeepSeek-V4-Flash on τ-Knowledge show that the exploration phase and handoff alone do not explain the gains: CodeAct with the same read-only exploration and structured handoff scores 28.9% pass^1 versus 33.3% for standard CodeAct, and PLOT without decision tools scores 29.9% versus 36.8%. A document-grounded LATM baseline reaches 27.8% (Appendix A.6). We now also report ablations over three task orders and on AutomationBench (Table 3).

  3. Statistical reliability and policy adherence (RJzo, xqkW). Paired bootstrap 95% confidence intervals for PLOT’s gains in τ-Knowledge pass^1 and AutomationBench partial credit all lie above zero (Appendix A.7). Splitting AutomationBench assertions shows that PLOT improves prohibited-outcome avoidance with both models (+5.5 and +4.3 points) (Section 4.2, Appendix A.8).

  4. Correctness of compiled tools (RJzo). A second author reviewed 42 sampled correctness labels; all 12 disagreements judged a function labeled incorrect to be correct (Appendix C). Of 30 accepted updates to saved tools, 10 corrected an earlier policy or calculation error (Appendix D).

  5. Scope of cross-task reuse (Y7gJ, RJzo, SrMa). Reuse arises when tasks share a policy corpus, as in τ-Knowledge. However, AutomationBench tasks have task-specific policies, so PLOT’s gains come from compiling decisions within each task, as the new AutomationBench ablations show (Section 5, Table 3).

We also clarified some implementation details, including ACE’s playbook updates and the setup of the official GPT-5.2 Vanilla trials (Section 4.1, Appendix A.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.