AlienBody: Diagnosing How LLM Agents Acquire and Use Action Models

Agent benchmarks typically expose meaningful action names, conflating name retrieval with action understanding. We introduce AlienBody, a 900-instance benchmark that randomizes action mappings across six action families to separate exploration, interpretation, and planning. Controlled naming interventions reveal a model-specific prior: misleading labels lower GPT-4o's success below anonymous controls on two families while truthful labels raise it directionally, so named success need not reflect identification from interaction. Even when the mapping reaches the agent as effect names, the tested in-context agents achieve only 0–10% success on Relational environments, while symbolic breadth-first search that computes with the mapping achieves 100%. Direct probes on the two tested models show why: defining the operators behind the names lifts per-step predictions from near-blind-guess accuracy to at most 52.6%, so a description of the actions does not by itself substitute for computing with them. A frozen interface study under one matched protocol isolates how that knowledge is used: on Relational grids, a programmable interface improves success over a conversational transition oracle by 49.75 points for DeepSeek V4 Flash and 35.75 for Qwen3.5-35B-A3B. Boolean workflows reverse that ranking for one model, and inspecting successful programs separates plans computed in code from fixed plans merely submitted through code. When the mapping is unknown, schema induction with verified search and active identification provides a constructive route from observations to control. These interventions make acquisition and use separately measurable, exposing the central shortfall: action information, whether a familiar label, a correct description, or a supplied mapping, can be available without being usable for control.