- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PLOT aims to transfer pilicy documents into executable and reusable tools for decision making. Instead of re-read and re-inference in the policy contexts, PLOT treat policy-related judgement as function, and directly call the function when execute.
Critical strengths
- PLOT transforms policy logic from something that must be reinterpreted each time into an executable and reusable tool.
- The experimental design is comprehensive. Ablation studies convincingly demonstrate the contribution of most design components.
Critical weaknesses
- The novelty is relatively low. The core idea of compiling natural language policies into executable code before invoking them is not conceptually surprising. In essence, it represents a systematic engineering integration of several existing approaches.
Questions for authors
The core of the paper is cross-task reuse. However, experiments show that it seems only true on τ-Knowledge, but the reuse rate on AutomationBench is close to 0. The authors attribute this to "dataset dependency" in discussion (sec 5). Then, is PLOT's reuse advantage essentially only applicable to scenarios where "multiple tasks share the same policy corpus"?
Novelty & Contribution of PLOT
The design goal of PLOT is to improve how agents make decisions in policy-governed environments. Reviewer xqkW highlighted the combination of task-specific decision slots, document-grounded compilation, and source-aware reuse as PLOT’s main distinction from prior work. A user request often leaves policy judgments implicit, and PLOT makes these judgments explicit as decision slots. Each slot guides the agent to the governing documents. The agent implements their rules as a callable function of task-specific facts. Admission checks verify that the function matches its declared signature and that the cited documents were read. They also screen for unused inputs and constant returns. This gives the agent a source-grounded way to evaluate each judgment when choosing actions, setting their arguments, and deciding their order.
PLOT compiles each policy-dependent decision into a tool, written from the policy documents, before the agent acts. Tool-making methods such as LATM instead build a function from a few demonstrations of a task, so a policy check missing from the demonstrations is also missing from the function. Code-as-action agents such as CodeAct make policy decisions within the task program itself. If the program executes an action before checking whether the policy permits it, the check comes too late to prevent a potential violation. As Reviewer RJzo noted, the comparison agent in our banking example “had read both relevant policies, but handled the requests in an order that made the final request impossible.” Guardrail methods such as ToolGuard compile policies into checks that accept or reject an action the agent has proposed, while a PLOT tool returns the decision that informs which action to take.
Clarification of Reuse
We would like to clarify that in τ-Knowledge, different banking tasks share a policy corpus and can call the same tool with new facts when the same decision recurs under the same rules. However, AutomationBench is designed to test agents on a diverse set of business workflows, with each task having its own simulated business scenario. This characteristic of AutomationBench makes reuse impossible.