- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DirectAct replaces autoregressive action-string generation with a small head that scores complete executable action tuples from a supplied candidate set using a frozen backbone. The main comparison holds training data, structured inputs, candidate sets, and approximate trainable parameter counts constant against a compact constrained LoRA generator. On 1,218 states from 88 prospectively collected successful workflows, DirectAct meets a predeclared two-percentage-point non-inferiority criterion on Qwen and MAI while reducing measured action inference latency by 4.51-5.13 times across three backbones.
Critical strengths
-
The controlled compact-generator comparison is persuasive for the paper's conditional interface claim. The generator already emits only six to seven tokens and selects from the same executable set, so the result is more informative than a comparison only against verbose JSON. The exhaustive sequence-scoring analysis further separates greedy decoding quality from the underlying generator's candidate scores.
-
The prospective collection protocol, checkpoints fixed before the primary collection, and workflow-clustered non-inferiority analysis improve the credibility of the offline comparison. The paper distinguishes the exploratory Llama result, ecological JSON timing, and separate closed-loop variant rather than presenting them all as the same controlled experiment.
Critical weaknesses
-
The demonstrated benefit is conditional on an unusually structured and relatively small supplied action set, and is not yet evidence of retained closed-loop GUI task success. The test states come from successful reference workflows in seven task families also represented in training. Candidate generation already supplies control/value constraints, and its coverage under policy errors or unforeseen values is not quantified. Appendix H uses only five expense tasks and a different conditional-value checkpoint, so it does not close this gap for the main DirectAct model. This limits deployment/generalization claims, although it does not invalidate the matched offline result.
-
Pooled action-weighted non-inferiority hides potentially important heterogeneity. Appendix F reports sizeable family-specific regressions, including SMS reply, and a negative MAI difference for actions without values. A user-facing workflow can fail from a small number of such errors. The current evidence therefore supports average action accuracy on this collection more strongly than uniform replacement of the generator across workflow types.
Questions for authors
-
Please report workflow-macro differences and uncertainty, alongside the existing action-weighted result, and explain the SMS/no-value regressions. Are they concentrated in a small number of workflows or systematic action types?
-
What are candidate-set coverage and failure modes on states reached by the main checkpoint itself, including incorrect actions? Can the main-checkpoint closed-loop result be separated from the different Appendix H variant?
-
How much of the accuracy/latency result remains with a simpler flat candidate scorer or one-token candidate-index output? This would help distinguish the benefit of direct selection itself from the proposed factorized tuple head.