acceptodds
Under review as a conference paper at ICLR 2027

Dyad: Extending Large Language Models with Native Typed Decision-Making

Abstract

We study how to build more capable general-purpose agents by extending large language models (LLMs) with native typed decision-making. We introduce Dyad, an architecture that augments a pretrained LLM with an environment-conditioned action encoder that embeds each candidate action description in parallel, then scores these embeddings against the LLM's internal state to yield a distribution over typed actions. By factorizing decision-making into representations of the evolving interaction state and environment-specific action semantics, Dyad introduces an inductive bias for learning reusable representations while keeping action scoring efficient even as the action space grows. We investigate two complementary reinforcement learning settings driven by environment interaction. With the LLM frozen, training the action encoder alone achieves consistent gains across four unseen environments, enabling modular adaptation without modifying any LLM parameters. Jointly optimizing both components outperforms conventional RL post-training across diverse interactive tasks and model scales, including a 3.80% average absolute gain on ALFWorld with a 9B model, while improving general knowledge, reasoning, and coding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, typed decision-making, tool use, reinforcement learning, dynamic action spaces, action representations, agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_6DSEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dyad, an architecture that augments a pretrained LLM with an action encoder for direct typed decision-making over runtime-provided action sets. Dyad independently encodes candidate action descriptions and scores them against the LLM’s state representation, while retaining autoregressive language generation. The model is initialized through supervised representation alignment and then trained with agentic RL. Experiments report improvements over GRPO/GiGPO baselines across in-domain and cross-environment evaluations.

Critical strengths
  1. Dyad provides a conceptually clean factorization of typed decision-making into an evolving LLM state representation and independently encoded action representations. This design integrates direct action selection with autoregressive reasoning in a single policy and enables parallel, potentially reusable action encoding.
Critical weaknesses
  1. The necessity of description-conditioned dynamic action encoding is not established. If the global action inventory is known in advance and only the permissible subset changes over time, a fixed action head with an AtA_t-dependent mask already supports the setting. Dyad becomes necessary only when previously unseen action schemas are introduced at deployment. However, the paper neither compares against a matched fixed action head with dynamic masking nor establishes strict action-schema disjointness between training and evaluation that makes fixed action head incompatible.

  2. Dyad introduces a separate pretrained Transformer action encoder and an additional GPT-5.5-generated alignment dataset, whereas the GRPO/GiGPO baselines use only the original LLM. Since alignment substantially improves performance in Figure 6(b), the reported gains conflate the proposed factorization with additional model capacity, supervision, and computation. The action encoder architecture, parameter count, and total training/inference cost are also insufficiently reported.

Questions for authors
  1. Does πact\pi^{act} select complete executable actions including arguments, or only action/tool types while the LM autoregressively generates arguments? Please clarify how argument generation is incorporated into the policy and training, particularly for single-interface environments such as SWE-bench.
2 comments

W1: Necessity of description-conditioned action encoding

Thanks for your insightful observation. Our goal is to generalize to action schemas first encountered at deployment. A fixed head with a dynamic mask handles changing subsets of a known inventory. ExpA, for example, allocates an output row to each predefined action, initializes it from the action description, and restricts selection to actions valid in the active environment (Yue et al., 2026, §4.1 and Appendix B). A mask selects among existing rows. Dyad can score a newly supplied operation because the same learned encoder generates its row from the environment and action descriptions, without fitting new action-specific parameters (Eqs. 1–3).

W2: Additional capacity, alignment supervision, and cost

Thank you for raising this important concern. To make the comparison as fair as possible while preserving each method’s action interface, we match the pretrained policy LM, RL algorithm, training environment, and evaluation tasks for each Dyad–baseline pair (§5.1). The textual baselines generate action names through their vocabulary head. Dyad’s separate pretrained Transformer encoder generates one weight vector per candidate, and these vectors form its action head (Eq. 3).

GPT-5.5 examples initialize only Dyad’s projector, with both backbones frozen (§3.1). Matching this initialization exactly would require adding an action projector to the textual baselines. We therefore test the effect of alignment within Dyad. With the architecture held fixed, alignment improves CodeGym validation reward over RL alone (Figure 6b). For the full-policy comparison, we report encoder parameter counts in ksc8 W1/Q1 and training and inference cost estimates in D69P W2/Q2.

Q1: What πact\pi^{\mathrm{act}} selects and how arguments are trained

Thank you for this question. πact\pi^{\mathrm{act}} selects only the operation type; the LM then generates its arguments autoregressively to complete the executable call (Appendix A.1). The probability of an executable call is thus the product of πact\pi^{\mathrm{act}} for the operation and πLM\pi^{\mathrm{LM}} for its argument tokens. In SWE-bench, Bash is the only candidate, so πact=1\pi^{\mathrm{act}}=1 and the LM generates open-ended command strings. With a frozen LM, encoder updates therefore cannot change command generation (see our response to Reviewer ksc8, W2 & Q2).

We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback. If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.

Reviewer_D69POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a technique for LLM post-training that augments an LLM with an environment-conditioned action encoder, which embeds each permitted candidate action description and enables direct scoring over the discrete space of permitted actions. The authors conduct training runs on DIVE and CodeGym, with evaluations on ALFWorld, WebShop, τ²-bench, and SWE-bench Verified, comparing against standard GRPO and GiGPO.

Critical strengths
  1. The authors evaluate the method on several benchmarks (ALFWorld, WebShop, τ2\tau^2-bench, and SWE-bench Verified), across in-domain and cross-environment settings, using multiple models.
  2. The method section clearly explains the approach’s architecture and implementation details.
  3. The empirical results are promising, with generally consistent improvements across benchmarks and models.
Critical weaknesses
  1. The paper’s novelty appears more incremental and architectural than methodological. Direct action-set encoding and scoring have already been explored in preceding works such as Jev, Laya, and SemIf; the authors’ contribution seems primarily to lie in the state–action factorization. The authors also emphasize that, unlike preceding approaches, their action encoder does not condition on the evolving interaction state. However, this state could contain useful action-related information, and an ablation study comparing state-conditioned and state-independent action encoding would be useful.
  2. The method introduces additional computational overhead at the training phase, at least through joint training of the action encoder. However, the paper does not quantify this overhead relative to standard GRPO and GiGPO.
  3. The paper states that evaluations use three seeds, but does not clearly describe the seeding protocol or specify whether these correspond to independent training runs, evaluation randomness, or both. The benchmark tables also lack confidence intervals or other uncertainty estimates.
  4. Some concerns arise from the method’s reliance on the environment providing a valid candidate action set and action descriptions. How the method would transfer to settings where such information is unavailable remains unclear.
  5. Interpreting shorter responses as evidence of more efficient reasoning seems overstated, since Dyad’s replacement of textual action generation with direct action scoring may itself reduce response length. Separating reasoning-token counts from action-token counts and measuring computational cost would help substantiate this claim.
Questions for authors
  1. Can you perform an ablation in which Dyad’s action encoder is conditioned on the evolving interaction state?
  2. Can you report the computational overhead relative to plain GRPO and GiGPO during both training and inference, using matched hardware and accounting for action-embedding caching?
  3. Can you add confidence intervals or other uncertainty estimates to the results tables and clarify whether the three seeds correspond to independent training runs, evaluation randomness, or both?
  4. Given the evaluation on SWE-bench Verified through a Bash interface, how does Dyad handle open-ended shell commands and arguments? What does it score directly in this setting, and what advantages does it offer over generating commands as text tokens?
4 comments

W1 & Q1: Novelty and state-conditioned action encoding

We appreciate your concern regarding novelty. In comparing Dyad with prior work on direct scoring over action sets (Jev, Laya, and SemIf), we would highlight two key architectural contributions:

  • Native action scoring. Dyad integrates typed action selection with language reasoning and open-ended generation in a single LLM policy.

  • State–action factorization. Dyad separates history-conditioned LM states from environment-conditioned action representations, enabling embedding reuse across steps.

To examine state conditioning, we add interaction history to the action encoder input and evaluate Qwen3.5-4B with GRPO on ALFWorld and WebShop, both with the original Dyad weights and after retraining. Neither setting improves over Dyad, while history conditioning requires re-encoding candidate actions at every step.

Action encoder inputEncoder weightsALFWorld (%)WebShop (%)
Environment + action descriptions (Dyad)Dyad70.1164.20
+ interaction historyDyad (not retrained)63.5258.09
+ interaction historyRetrained with history66.4461.87

W2 & Q2: Computational overhead

Thank you for this important question. We measure training and inference costs on ALFWorld with Qwen3.5-4B using the same 8×H200 node, with Dyad jointly optimizing the policy LM and a separate action encoder of comparable scale (parameter counts in Reviewer ksc8, W1 & Q1). Initial action encoding is included in cold-cache inference, while warm-cache inference reuses precomputed embeddings. Dyad incurs higher training costs but lower task-level inference time through embedding reuse and fewer reasoning tokens (see W5).

MethodTraining Time (s/iter)Initial Action Encoding (ms)Cold-Cache Inference (s/task)Warm-Cache Inference (s/task)
GRPO274.66N/A29.4029.40
Dyad-GRPO360.22679.3225.3124.63
GiGPO263.92N/A27.9327.93
Dyad-GiGPO352.29679.3224.3323.65

W3 & Q3: Seeding protocol and uncertainty

Thank you for pointing out this ambiguity. The three seeds are three evaluation runs of the same trained checkpoint, capturing evaluation variance. We will clarify this protocol and report the standard deviation across the three evaluation runs in all result tables.

W4: Reliance on candidate actions and descriptions

Thank you for raising this concern. Many applications already expose the choices available to an agent, from routing a support request to one of several teams to selecting a tool from a documented API. Recent System 1 models such as Jev and Laya are designed for this kind of typed decision-making.

When a task does not expose a candidate action set, one can first abstract the task into an interactive environment with documented operations. CodeGym, one of Dyad’s training environments, does this by extracting functions from solutions to static coding problems and exposing them as callable tools. Similarly, RSI-Forge constructs executable environments from published research problems across 18 fields to support recursive self-improvement. Dyad can therefore serve a broad range of tasks, both where an action interface already exists and where one can be constructed through such an abstraction.

W5: Response length and reasoning efficiency

Thank you for raising this point. Dyad's shorter responses accompany improved task performance, suggesting more efficient problem-solving through concise reasoning and effective interaction (Fig. 4). We further measure reasoning tokens, action tokens, and interaction steps with Qwen3.5-4B on ALFWorld, accumulated over complete trajectories and averaged across tasks, with action tokens including arguments, formatting, and Dyad's inserted action text.

MethodReasoning Tokens/TaskReasoning Tokens/StepAction Tokens/TaskInteraction Steps/Task
GRPO6,026.87199.17225.9430.26
Dyad-GRPO4,988.36174.24216.6928.63
GiGPO5,711.21198.72213.7528.74
Dyad-GiGPO4,898.09186.81229.5426.22

Q4: Open-ended shell commands in SWE-bench

Thank you for this question. On SWE-bench Verified, Bash is the sole candidate operation, while the LM generates the entire shell command autoregressively, preserving open-ended command generation (Appendix A.1). Moreover, Dyad better preserves the pretrained LLM's general capabilities during agentic RL while improving coding performance (Figure 3), which is particularly relevant to solving repository-level issues in SWE-bench. The frozen-LLM comparison is discussed in our response to Reviewer ksc8, W2/Q2.

We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback. If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.

Thank you for the detailed clarifications and additional experiments. The ablations and timing results address my questions about state conditioning and computational overhead. However, I remain unconvinced that Dyad’s architectural contributions are sufficiently distinct from prior work on direct action scoring. The method’s applicability also still depends on having a suitable candidate-action interface. I therefore keep my score at 2 (Weak rejection).

We respectfully disagree with the assessment of Dyad’s architectural novelty. We regard Jev and the cited open-source alternatives as contemporaneous work.

Dyad extends a pretrained LLM with an action encoder, enabling language generation and typed decision-making within a single policy. It factorizes typed decision-making into pretrained LLM state representations and independently encoded, environment-conditioned action representations, constructing a native decision head over dynamically provided actions. Action representations can be reused while their input descriptions and encoder parameters remain unchanged. Dyad supports frozen-LLM adaptation and joint optimization through agentic RL, with joint training improving agent performance while better preserving general knowledge, reasoning, and coding.

The candidate-action interface is also part of the setting of the cited decision models. For choice selection, Jev and the cited alternatives likewise predict distributions directly over provided choices. This setting encompasses support-request routing, tool use, and navigating unfamiliar software.

Could you point us to other closely related prior work beyond Jev and the cited open-source alternatives? We would be glad to strengthen our related-work discussion accordingly.

Comments on the paper

3 comments

W1 & Q1: Isolating the benefit of the factorization

Thank you for this important comment. DRRN established factorized state–action scoring for text-based decision-making. Dyad extends a pretrained generative LLM into a unified reasoning-and-acting policy, retaining open-ended generation while supporting native typed decisions and frozen-LLM adaptation. We further compare text generation, constrained decoding, and Dyad using Qwen3.5-4B with GRPO on ALFWorld. The constrained baseline uses the same candidate set and selection granularity as Dyad under a comparable RL budget.

The action encoder uses a separate Qwen3.5 backbone with a projector replacing the vocabulary LM head, and its parameter counts are provided below. Matched-hardware training and end-to-end inference costs for Qwen3.5-4B are reported in our response to Reviewer D69P, W2/Q2. As shown below, Dyad outperforms constrained decoding by 3.31%, indicating that candidate restriction alone does not explain the improvement.

MethodCandidate-Constrained SelectionALFWorld (%)
Text GenerationNo64.11
Constrained DecodingYes66.80
DyadYes70.11
Policy LMAction Encoder BackbonePolicy Model Params (B)Action Encoder Params (B)Total Params (B)
Qwen3.5-4BQwen3.5-4B4.544.569.10
Qwen3.5-9BQwen3.5-9B9.418.4417.85

W2 & Q2: Scope of native action selection and SWE-bench under a frozen LLM

Thank you for requesting this clarification. Dyad selects operation types from AtA_t, while the LM generates their arguments autoregressively. In SWE-bench, At={Bash}A_t=\{\mathrm{Bash}\}, so Bash is selected deterministically once the LM emits vactv_{\mathrm{act}}.

Even with the LM frozen, Dyad's native typed interface can improve the reliability of Bash invocation. Similar to constrained decoding, Dyad directly supplies the valid operation type rather than requiring the LM to generate and correctly format the tool name, while retaining open-ended shell command generation. Moreover, Dyad executes one call per interaction, allowing subsequent commands to incorporate actual execution feedback.

Q3: Seeding protocol and variability of smaller gains

Thank you for this question. For each model and method, the three seeds are three evaluation runs of the same trained checkpoint, not three independent RL training runs. We will state this explicitly and report the standard deviation over these evaluation runs in all result tables. The complete Table 3 results below report mean ± SD (%) across the three evaluation runs.

Model / MethodDIVE: ALFDIVE: WebShopDIVE: τ²DIVE: SWECodeGym: ALFCodeGym: WebShopCodeGym: τ²CodeGym: SWE
Qwen3.5-4B23.45 ± 1.4328.54 ± 0.6861.05 ± 1.2343.21 ± 1.7423.45 ± 1.4328.54 ± 0.6861.05 ± 1.2343.21 ± 1.74
+ GRPO29.71 ± 2.1230.42 ± 1.8760.52 ± 0.9345.67 ± 2.2832.18 ± 2.3433.65 ± 1.8264.10 ± 0.7647.98 ± 2.53
+ Dyad-GRPO31.52 ± 1.2331.22 ± 0.6464.54 ± 1.4744.28 ± 1.4833.65 ± 1.0235.71 ± 0.5765.26 ± 1.7245.02 ± 1.63
+ GiGPO28.22 ± 2.5829.54 ± 1.5361.84 ± 0.4845.02 ± 2.1731.09 ± 2.4727.97 ± 1.6263.48 ± 0.7346.64 ± 2.42
+ Dyad-GiGPO33.07 ± 0.8732.45 ± 0.8361.86 ± 1.8647.10 ± 0.9431.72 ± 1.1331.21 ± 1.9764.78 ± 1.5849.61 ± 1.37
Qwen3.5-9B45.66 ± 2.7733.43 ± 1.7462.17 ± 0.8344.56 ± 2.1245.66 ± 2.7733.43 ± 1.7462.17 ± 0.8344.56 ± 2.12
+ GRPO52.67 ± 1.3634.52 ± 0.8262.06 ± 1.9647.22 ± 1.4351.55 ± 1.5236.23 ± 0.5861.31 ± 1.8749.40 ± 1.48
+ Dyad-GRPO53.82 ± 2.2736.31 ± 1.6364.88 ± 0.5746.20 ± 2.6455.13 ± 2.4238.07 ± 2.7663.02 ± 0.7350.62 ± 2.26
+ GiGPO50.98 ± 3.2736.23 ± 2.6863.52 ± 1.5348.83 ± 0.9252.08 ± 1.9336.58 ± 1.0464.21 ± 1.6249.78 ± 1.17
+ Dyad-GiGPO52.06 ± 1.6337.02 ± 1.0761.90 ± 0.8450.42 ± 1.3755.01 ± 2.2835.10 ± 0.9365.85 ± 0.6751.25 ± 2.34
Qwen3.5-27B68.45 ± 1.2744.56 ± 0.8368.21 ± 0.4666.78 ± 0.9368.45 ± 1.2744.56 ± 0.8368.21 ± 0.4666.78 ± 0.93
+ GRPO69.25 ± 0.9346.68 ± 1.6769.82 ± 1.4368.92 ± 1.0771.89 ± 2.0342.13 ± 1.8669.29 ± 1.4867.91 ± 1.04
+ Dyad-GRPO71.42 ± 2.0448.81 ± 2.1371.46 ± 0.6370.24 ± 2.1771.11 ± 0.7748.29 ± 0.9470.10 ± 0.5768.34 ± 2.23
+ GiGPO69.84 ± 1.4747.20 ± 0.9269.56 ± 1.1868.02 ± 1.0370.79 ± 1.1345.91 ± 1.9669.51 ± 1.4367.90 ± 0.88
+ Dyad-GiGPO72.30 ± 0.7345.44 ± 1.2771.87 ± 0.3471.53 ± 0.8373.21 ± 1.8747.72 ± 2.1472.30 ± 0.8768.98 ± 1.93

Thanks for the clarification, most of my concerns are resolved, I raise my score to 3.

We sincerely appreciate your constructive feedback and the updated rating. Your comments have been very helpful in improving our work. We welcome any further questions or discussion regarding the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.