- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies prospective authorization, where a tool-using LLM must wait for a later authorization signal before executing a proposed action. It introduces I24, a benchmark of 1,764 episodes across three authorization modes, three signal conditions, and four domains. State-explicit supervision reduces the reported symbolic transition error from 34.6% to 5.8% on 306 held-out episodes. PAG combines a frozen planner, learned authorization tracker, pending-action ledger, and exactly-once executor, with a reported 2.6% native error on all 1,764 episodes, including tracker-training episodes. The paper also reports transfer experiments, component ablations, comparisons across planner models, and tests of general capabilities.
Critical strengths
1. A practical problem with a clear definition and a controlled benchmark design.
Knowing when an action is authorized is a practical problem that deserves evaluation alongside choosing the right action. The paper formally defines three episode-level error types. Its paired 3×3 design varies the authorization mode and signal condition while keeping the action and dialogue context fixed. This helps identify errors in when the model acts. The problem definition and benchmark design are useful contributions even apart from the proposed training method.
2. The breakdown of baseline errors is useful.
Rule prompting reduces pre-signal errors from 14.4% to 3.9% but leaves duplicates at 23.2%. Few-shot prompting reduces duplicates from 22.2% to 10.8% while increasing missed errors to 20.9%. Reporting these errors separately shows that a prompting method can reduce one type of error while increasing another. This makes it easier to see where each method succeeds and fails.
Critical weaknesses
1. The experiments do not show whether state-explicit supervision improves on ordinary fine-tuning.
The main improvement from 34.6% to 5.8% compares C1 with a frozen model. The rule and few-shot baselines also receive no task-specific fine-tuning. What is missing is a plain/action-only SFT baseline trained on the same 1,458 episodes, with the same base model and training budget, but without the intermediate state targets.
MechDel does not answer this question. Shuffling the intermediate state labels produces 66.7–72.5% error. This shows that incorrect state labels hurt performance, but it does not show that correct state labels improve performance over ordinary SFT.
The results in Table 1 make this comparison especially important. C1 with 3B and 8B Llama models reaches 0.0% error, three of four leave-one-domain-out settings reach 0.0%, and OOD-v2 error (3.1%) is lower than the main held-out error (5.8%). C1 therefore achieves near-perfect performance in several controlled settings. Whether ordinary SFT can do the same without explicit state targets is still an open question (see Q1).
The paper also needs to explain how the full-state training objective in Section 3.3 relates to the decision and action-matching losses for PAG's tracker in Appendix C.1. These may describe different models, but the authors should state which training setup was used for each experiment. The results show that C1's training procedure works better than the tested baselines. They do not show how much the state targets add, or whether those targets are necessary. This missing comparison directly affects the paper's claim that state-explicit supervision is a useful contribution beyond ordinary fine-tuning.
2. The native evaluation does not show how PAG performs on unseen episodes, and the ablations are not fully described.
Section 4.3 evaluates PAG on all 1,764 I24 episodes, including the 1,458 episodes used to train the tracker (Appendix C.1). About 82.7% of the evaluation episodes were therefore used in tracker training. Testing native execution is useful, but the overall 2.6% error combines results from training and held-out episodes. It does not tell us how well PAG performs on unseen episodes. The component ablations and comparisons across planner models also need results on the held-out episodes to show whether their findings hold beyond the training data.
Table 3 lists which modules are removed but does not explain what the system does in their place. The ledger stores pending actions, and the executor checks whether an action has already been executed. These rules already provide part of the required behavior, so the ablations must be described clearly to show what the learned tracker adds. Held-out results and these implementation details are needed to assess PAG's performance on unseen episodes and the claim that each component is necessary.
The current experiments do not show whether state-explicit supervision improves on ordinary SFT or how well PAG performs on unseen episodes in native execution. Answering these questions requires the missing SFT comparison and separate results for the held-out native episodes. Those results could change the conclusions about both methods. The current evidence is therefore not sufficient to support acceptance.
Questions for authors
-
Comparison with ordinary SFT (additional experiment; W1). A comparison with plain/action-only SFT using the same base model, 1,458 training episodes, and training budget, but no intermediate state targets, would help show what the state targets add. If such results are available, I would appreciate the transition errors for both methods on the same 306 held-out episodes. A consistent advantage for C1 would support the claimed benefit of state-explicit supervision.
-
Native results on held-out episodes (reanalysis; W2). It would be helpful to see the results in Tables 2–4 for the 306 held-out episodes only. If PAG's advantage and the effects of removing each component hold on this set, that would address much of my concern about the native evaluation.
-
Training, evalution and ablation details (clarification; W1–W2). Please clarify the training outputs and losses used for C1 and PAG, including whether Section 3.3 describes C1 and Appendix C.1 describes PAG's tracker. A brief explanation of what the system does when each component is removed in Table 3 would also make the ablations easier to interpret. These details would clarify which training method was tested and what each component contributes. Also, I am wondering why PAG's results in Table 5 differ from Frozen at all (e.g., the 1.3-point drop on MMLU), given that the planner is frozen and the tracker is trained separately.
-
Symbolic error rates (clarification). Section 4.2 reports 5.8% symbolic held-out error for C1, while Section 4.3 refers to 0.0%. I would appreciate clarification of whether these come from different settings, with the model and evaluation split identified for each, or whether one value needs correction. This would help clarify the comparison with native execution.
-
Benchmark reproducibility and validity (clarification; Strength 1). The paper says that data construction and annotation details are in Appendices A–D, prompt templates are in Appendix C, and the complete OOD-v2 expression list is in Appendix A, but I could not find these materials. Please provide the I24 generation procedure, including the templates and how distractor signals are created, as well as the evaluation prompts, OOD-v2 expression list, and the ledger's action-matching rule. I consider the benchmark design a main strength, but these details are needed for others to reproduce and check the results.
It would also help to explain whether matching keywords or phrases in the dialogue templates could produce similarly low error rates. Examples from the benchmark or experimental results addressing this possibility would help support the interpretation of the results.
-
Model details and figure corrections (minor; requires revision).
- Please specify the exact model repository/checkpoint identifier referred to as "Qwen3-8B-Instruct" (e.g., Qwen3-8B with thinking mode enabled or disabled, Qwen3-8B-Base, or Qwen3-VL-8B-Instruct), state whether thinking mode was enabled or disabled where applicable, and use the precise model name consistently throughout the paper.
- Please translate all text in Figure 2 into English and use the same decision labels in the text and figures (e.g.,
ALLOWversusALLOW_NOW).