Many of the paper's claims rely on knowledge that appears to be proprietary and cannot be accessed, so they cannot be verified. Because of this limitation, the study can do little more than report results on questions of interest for Jev. This is fine, but claims on the specific features of JEV that cannot be verified should not appear in the paper. The paper is also very dense: it proposes many things at once, and does not have the space and depth to investigate each of them fully.
Major
1. Unverifiable claims
- "Jev, by contrast, answers a batch of typed questions in one fixed-cost forward pass" (line 100): this claim is unsupported, as it cannot be verified: Jev is closed source, and the paper gives no details on what it does.
- "Grows with context length" (lines 99–100): it is not clear why this is stated, since there is no term of comparison. Does Jev's forward pass not grow with context length? The paper does not show it.
- "TypeSafe AI describes its training procedure, RLCD, ..." (line 104): the reference (TypeSafe AI, 2026b) is uninformative and does not contain any description of RLCD, which seems to be a closed, proprietary method. The paper should not state what cannot be verified.
2. Methods
- "The value function is the most demanding. Its target r+γmaxa′Q(s′,a′) adds a reward to a value and discounts the sum" (line 163): this is incorrect. A Q-function is a model of the return. The equation refers to a specific method to learn that function, i.e., one-step TD learning, which is not the current standard. For this reason, if Jev can approximate a reward, it should also be able to approximate a Q-function.
- Similarly, "since its update performs arithmetic on the answer" (line 58): what exactly does this mean? A value function is as static as a policy.
- "All four take a state-like input" (line 154): some take an action as well (e.g., a Q-function). Does this change what Jev can represent? If not, this should be stated explicitly.
- "Reward is the least demanding of the four. It enters the objective as a sum of terms" (line 159). Why is a reward a sum of terms?
- "A choice question supplies both whenever those sets are finite" (line 162): is being finite enough? What if the set is large (e.g., Atari)? Is the model still able to provide a distribution over such a large set?
- Line 191 suggests that Jev will be used as a ranker for the Q-values of actions in the experiments, but no experiment shows that. Also, using a ranker rather than a critic has been explored in the past (Lagoudakis & Parr, 2003), but the text gives no reference to it.
- Line 205: if "Jev as Transition" is set aside, it should not be in the paper.
3. Benchmarks and input construction
- There exist consolidated benchmarks to investigate LLMs, VLMs and decision models for games and RL (e.g., BALROG). However, the paper only tests on MiniGrid, usually considered a diagnostic task, and on three Atari games.
- Jev sounds great, but such an effort to solve MiniGrid sounds like overkill: these environments have been solved. If Jev is such a huge step forward compared to LLMs (or such a new frontier), I would rather see investigations on more complex problems. For example, how does it do on NetHack?
- Not using these benchmarks raises some questions (which could also be answered without them, but that requires further investigation and validation of the benchmarks used in the paper):
- In MiniGrid, the task information leaks information on the optimal policy, e.g. "Task information. Collect the matching key, unlock the door, then enter the goal" (line 1298).
- The same is true for Atari, where the input construction adds information such as "RGB-detected objects and estimated motion: oxygen, divers and enemies for Seaquest; vehicles and the chicken for Freeway; and the cannon, aliens, projectiles and shields for SpaceInvaders" (lines 822–824).
- It is not clear how much this additional information contributes, as there are no ablations for it, so the effects of Jev cannot be disentangled from it. In support of this, the harder tasks (Atari) show lower, no, or even negative gains with respect to the baselines, suggesting that the input construction plays a crucial role in the performance.
- How the input is constructed (line 816) is unclear, and not described in enough detail to reproduce.
- Because observations cannot be fed raw to Jev, and inputs must be constructed, it is not clear whether the gains come from the information that the construction leaks into the inputs.
- In RQ2, it is not clear what "We keep Jev fixed and remove candidate routes, path distances, and progress/priority hints on four tasks, retaining identical observational memory and task rules" (line 368) means: there are no details on how the ablation of the input construction is carried out, making it non-reproducible.
4. Experiments
- Baselines are lacking. Why only Qwen?
- Qwen disappears as a baseline in RQ3 and RQ4: why?
- RQ3 and RQ4: some tasks benefit more from one Jev role than from another, but why is not investigated, even though it is a key component of the findings.
- RQ3: how are the "fixed logged states" (line 393) sampled, and why? This is not stated, and the results are not reproducible.
5. Literature review
Much of the relevant literature is missing. This seems a consequence of the paper addressing too many problems at once (Jev as a policy, as an exploration guide and as a replay prioritiser): none of them is investigated in depth, and each lacks the work it should be positioned against. The most relevant missing works are listed under References below.
6. Conclusions
The conclusions sometimes say more than the results show, and they are much more cautious than the abstract, which therefore overclaims.
- The conclusion states "These uses can improve early learning" (line 482), but the experiments section says otherwise:
- "improve selected tasks ... no tested configuration is uniformly best" (lines 418–419);
- "improves selected tasks but does not consistently outperform PER" (lines 449–450).
- "Our findings demonstrate that a decision model need not be a reliable standalone controller to improve RL training" (lines 484–485): this is already known. Learning from imperfect teachers or priors is the premise of works the paper itself cites (Zhou et al., 2024; Yan et al., 2025) and of imitation learning from suboptimal experts.
- The conclusions should say plainly that part of the gain may come from the hand-built inputs, if so.
Minor
7. Writing and notation
- "Generates nothing" (line 40) is contradictory, as the paper then states that "it returns typed answers" (line 41).
- "Typed questions", "typed answers": these terms must be defined, since the paper presents them as a major point of difference with LLMs.
- "The objects it can fill group into the positions it can occupy, and each position admits several roles" (line 58). "Objects", "positions" and "roles" are used before being defined (Section 3), and inconsistently: "role" means an RL object in Section 3.1 and a use of a position in Section 3.2. The four positions are not a grouping of the four objects: Meta Regulator corresponds to none of them, and Score Rater merges R and Q despite their different domains. The intuition is hard to grasp.
- "An RL system asks four things of a function approximator" (line 152): it is the opposite. RL does not "ask"; a function approximator can be used to estimate some of the quantities in an RL problem.
- The MDP description is too generic and does not give useful information. For example, what exactly is the state space? A sequence of c tokens? What is the action space? A discrete set over tokens? In fact, the "Jev model" paragraph has to specify what a "situation" is, to differentiate it from a "state".
- Most environments in the experiments are POMDPs (e.g., Atari), not MDPs.
- Yu is loosely defined. What is its space?
- Xy is loosely defined, because the space of X is loosely defined. What is its space?
- Line 185: C is the action space, but that was defined as A in the preliminaries, where C is a set of options.
- Line 199: XS×A is not defined.
- Line 261: Ht has not been introduced yet. How is it constructed?
References
Direct counterparts of the paper's algorithms:
- Replay rater (§4.3): Sharony, E., Jurgenson, T., Krupnik, O., Di Castro, D., and Mannor, S. VLM-guided experience replay. arXiv:2602.01915, 2026. (A VLM scores experience; the score multiplies the TD error for prioritisation.)
- Exploration (§4.2): Hao, Q., Song, Y., Liao, Q., Yuan, J., and Li, Y. LLM-Explorer: A plug-in reinforcement learning policy exploration enhancement driven by large language models. In Advances in Neural Information Processing Systems 38, 2025. (LLM-set exploration for DQN, DDPG and TD3, on Atari.)
- Meta Regulator (lines 194–199): Hu, B., Zhao, C., Zhang, P., Zhou, Z., Yang, Y., Xu, Z., and Liu, B. Enabling intelligent interactions between an agent and an LLM: A reinforcement learning approach. Reinforcement Learning Journal (RLC), 2024. (When2Ask: learns when to query the LLM, on MiniGrid.)
- Reference policy (§4.1): Hu, H. and Sadigh, D. Language instructed reinforcement learning for human-AI coordination. In ICML, 2023. (An LLM prior policy regularises the RL objective.)
- Research questions 1–2 (lines 53–54): Klissarov, M., Hjelm, D., Toshev, A., and Mazoure, B. On the modeling capabilities of large language models for sequential decision making. In ICLR, 2025. (LLMs as direct policies vs as reward models.)
Action evaluation, reward and credit assignment (Score Rater):
- Pignatelli, E., et al. Assessing the zero-shot capabilities of LLMs for action evaluation in RL. ICML 2024 Workshop on AutoRL; arXiv:2409.12798, 2024.
- Qu, Y., et al. Latent reward: LLM-empowered credit assignment in episodic reinforcement learning. arXiv:2412.11120, 2024.
- Klissarov, M., D'Oro, P., Sodhani, S., Raileanu, R., Bacon, P.-L., Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. In ICLR, 2024.
- Zheng, Q., Henaff, M., Zhang, A., Grover, A., and Amos, B. Online intrinsic rewards for decision making agents from large language model feedback. Reinforcement Learning Journal (RLC), 2025. (Distils LLM feedback into a reward model: relevant to the cost argument, lines 97–100.)
- Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In ICLR, 2023.
- Xie, T., et al. Text2Reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
- Yu, W., et al. Language to rewards for robotic skill synthesis. In CoRL, 2023.
- Fan, L., et al. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In NeurIPS, 2022.
- Ahn, M., et al. Do as I can, not as I say: Grounding language in robotic affordances. In CoRL, 2022.
Exploration, interestingness and goal selection:
- Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In ICML, 2023.
- Zhang, J., Lehman, J., Stanley, K., and Clune, J. OMNI: Open-endedness via models of human notions of interestingness. In ICLR, 2024.
- Faldor, M., Zhang, J., Cully, A., and Clune, J. OMNI-EPIC: Open-endedness via models of human notions of interestingness with environments programmed in code. In ICLR, 2025.
- Lu, C., Hu, S., and Clune, J. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. In ICLR, 2025. (A foundation model judges which states are interesting to return to.)
- Colas, C., Teodorescu, L., Oudeyer, P.-Y., Yuan, X., and Côté, M.-A. Augmenting autotelic agents with large language models. In CoLLAs, 2023.
- Mu, J., et al. Improving intrinsic exploration with language abstractions. In NeurIPS, 2022.
- Tam, A., et al. Semantic exploration from language abstractions and pretrained representations. In NeurIPS, 2022.
- Nottingham, K., et al. Do embodied agents dream of pixelated sheep? Embodied decision making using language guided world modelling. In ICML, 2023.
- Wang, G., et al. Voyager: An open-ended embodied agent with large language models. TMLR, 2024.
LLMs as policies, executed directly (research question 1):
- Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In ICML, 2023.
- Tan, W., et al. True knowledge comes from practice: Aligning LLMs with embodied environments via reinforcement learning. In ICLR, 2024.
- Szot, A., et al. Large language models as generalizable policies for embodied tasks. In ICLR, 2024.
- Li, S., et al. Pre-trained language models for interactive decision-making. In NeurIPS, 2022.
- Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022.
- Paglieri, D., et al. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In ICLR, 2025.