- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Dyad, an architecture that augments a pretrained LLM with an action encoder for direct typed decision-making over runtime-provided action sets. Dyad independently encodes candidate action descriptions and scores them against the LLM’s state representation, while retaining autoregressive language generation. The model is initialized through supervised representation alignment and then trained with agentic RL. Experiments report improvements over GRPO/GiGPO baselines across in-domain and cross-environment evaluations.
Critical strengths
- Dyad provides a conceptually clean factorization of typed decision-making into an evolving LLM state representation and independently encoded action representations. This design integrates direct action selection with autoregressive reasoning in a single policy and enables parallel, potentially reusable action encoding.
Critical weaknesses
-
The necessity of description-conditioned dynamic action encoding is not established. If the global action inventory is known in advance and only the permissible subset changes over time, a fixed action head with an -dependent mask already supports the setting. Dyad becomes necessary only when previously unseen action schemas are introduced at deployment. However, the paper neither compares against a matched fixed action head with dynamic masking nor establishes strict action-schema disjointness between training and evaluation that makes fixed action head incompatible.
-
Dyad introduces a separate pretrained Transformer action encoder and an additional GPT-5.5-generated alignment dataset, whereas the GRPO/GiGPO baselines use only the original LLM. Since alignment substantially improves performance in Figure 6(b), the reported gains conflate the proposed factorization with additional model capacity, supervision, and computation. The action encoder architecture, parameter count, and total training/inference cost are also insufficiently reported.
Questions for authors
- Does select complete executable actions including arguments, or only action/tool types while the LM autoregressively generates arguments? Please clarify how argument generation is incorporated into the policy and training, particularly for single-interface environments such as SWE-bench.
2 comments
We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback. If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.
W1: Necessity of description-conditioned action encoding
Thanks for your insightful observation. Our goal is to generalize to action schemas first encountered at deployment. A fixed head with a dynamic mask handles changing subsets of a known inventory. ExpA, for example, allocates an output row to each predefined action, initializes it from the action description, and restricts selection to actions valid in the active environment (Yue et al., 2026, §4.1 and Appendix B). A mask selects among existing rows. Dyad can score a newly supplied operation because the same learned encoder generates its row from the environment and action descriptions, without fitting new action-specific parameters (Eqs. 1–3).
W2: Additional capacity, alignment supervision, and cost
Thank you for raising this important concern. To make the comparison as fair as possible while preserving each method’s action interface, we match the pretrained policy LM, RL algorithm, training environment, and evaluation tasks for each Dyad–baseline pair (§5.1). The textual baselines generate action names through their vocabulary head. Dyad’s separate pretrained Transformer encoder generates one weight vector per candidate, and these vectors form its action head (Eq. 3).
GPT-5.5 examples initialize only Dyad’s projector, with both backbones frozen (§3.1). Matching this initialization exactly would require adding an action projector to the textual baselines. We therefore test the effect of alignment within Dyad. With the architecture held fixed, alignment improves CodeGym validation reward over RL alone (Figure 6b). For the full-policy comparison, we report encoder parameter counts in ksc8 W1/Q1 and training and inference cost estimates in D69P W2/Q2.
Q1: What πact selects and how arguments are trained
Thank you for this question. πact selects only the operation type; the LM then generates its arguments autoregressively to complete the executable call (Appendix A.1). The probability of an executable call is thus the product of πact for the operation and πLM for its argument tokens. In SWE-bench, Bash is the only candidate, so πact=1 and the LM generates open-ended command strings. With a frozen LM, encoder updates therefore cannot change command generation (see our response to Reviewer ksc8, W2 & Q2).