acceptodds
Under review as a conference paper at ICLR 2027

Can JEV Be Your Q or Policy in Reinforcement Learning?

470 views

Abstract

Foundation models supply reinforcement learning (RL) with priors that mitigate its longstanding weaknesses in sample efficiency and transfer, but their token-by-token generation makes queries sequential and costly. Jev, a recently released decision model, generates nothing and returns calibrated, typed answers in a single forward pass. Existing work studies foundation models in RL either as models to be trained or as generators to be prompted, and Jev belongs to neither category, having so far served only as a black box in single domains. How well such a model decides on its own in RL environments, and how it can improve RL as a component of training, therefore remain unaddressed. To this end, in this paper we first examine the requirements that the objects of an RL system place on the answers they consume, and establish that Jev can fulfill all of them except the cardinal use of a value function. The remaining objects form positions that admit several roles each. We then construct algorithms that employ Jev at three of these positions, as a reference policy, an exploration judge, and a replay rater, to improve sample efficiency, exploration, and learning performance. Across nine MiniGrid tasks and three Atari games, training with Jev outperforms a standard RL learner, including where the learner makes no progress alone, while the model itself remains untrained. To our knowledge, we present the first use of Jev within the RL learning process and establish a frozen decision model as a usable component of RL training, inviting further exploration of how Jev and other advanced decision models can improve RL.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Jev Model, Learning Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2AX1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper looks at Jev, a recently released closed decision model that returns typed answers in a single forward pass. The paper asks how it can be used inside RL. First it is mapped onto the objects an RL system needs; it is argued that it can fill all of policy, value, reward, transiion except the cardinal use of a value function. The paper actually plug Jev into three slots: reference policy, exploration judge, and replay rater; experiments are then analyzed.

Critical strengths
  • analysis on Jev, which is a very recent model, so overall full of new contents
  • honest about negative results
Critical weaknesses
  • Novelty is thin beyond we used Jev. Each method is an existing algorithm with Jev dropped into some slot. Importantly, the properties the paper uses to set Jev apart are never actually used or measured: cheap single-pass cost, calibration, multiple questions per call, etc. are never utilized or measured. As they are not utilized or measured, all the experimented settings could be built with any open LLMs, so I don't see any unique usage of Jev, while these usages using LLMs have been studied quite a bit already.
  • Q is excluded because a rubric score is ordinal and can't be added or discounted. But reward is also summed and discounted into return, so not sure why Q should have been excluded. The implementation also takes an expectation over (0, 0.25, 0.5, 0.75, 1) and adds the result to the reward and normalizes it into replay probabilities; this kind of usage already treats rubric score as cardinal.
  • The information the Jev used: route candidate, path distances, etc. should have been input to other baselines as DQN or scripted teacher baselines to be fair.
  • There are not enough proper baselines; No LLM teacher/judge in the same setup, no RND or count-based exploration. The Qwen comparison is zero-shot only, and non-thinking 8B with 48 token cap feels like very weak setup.
  • with current budgets, DQN is near zero on most tasks, which makes me wonder if the budget is insufficient or the tuning was insufficient. standard errors are large and many of the claimed improvements sit within the error bars (it is not even 1.96se error bar I suppose)
  • conclusions are phrased about 'decision model' in general based on a single model. As we have insufficient understandings on this model (since being disclosed), I personally think most of the claims with this insufficient statistical evidence are overstatements.
Questions for authors
  • Is there any setup that Jev can clearly show advantage over other methods, that cannot be replaced? With current form it feels like all applications proposed can be done with LLMs instead.
Reviewer_aTAZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a frozen Jev decision model, which returns typed answers (choices, rubric scores, or probabilities) in a single forward pass, can be used in reinforcement learning. It first analyzes which RL functions Jev's answer forms can represent, arguing that Jev can serve as a policy, score rater, meta-regulator, or transition model, but not as a cardinal value function. The empirical study then proceeds in two parts. First, Jev is evaluated directly as a zero-shot policy on nine MiniGrid tasks and three Atari games and compared with non-thinking Qwen3-8B under a matched structured-input interface; an ablation on four navigation tasks shows that its performance depends substantially on engineered route features. Second, Jev is used as a component of training: as a reference policy (Jev-BC, Jev-PEX, and Jev-guided exploratory actions in DQN), as an exploration judge (a gate replacing epsilon-greedy and novelty-based intrinsic bonuses), and as a replay rater that sets sampling priorities. These methods are compared mainly against DQN and, for replay, PER, and are supplemented by fixed-experience analyses that separate data quality, reward shaping, and sampling effects, as well as negative results for subgoal guidance in the appendix. The reported results show task-dependent improvements in early learning, including some tasks where DQN alone makes no progress, with students evaluated without Jev or auxiliary rewards.

Critical strengths
  1. The targeted control experiments provide useful insight into the sources of improvement. Figure 9 retrains learners on the collected experience without auxiliary rewards, examining whether benefits persist through improved data collection. Figure 10 holds experience fixed and replaces Jev's state scores with averages within visitation bins and collection windows, testing the value of distinctions beyond the coarse visitation summary. Figure 11 preserves the total replay probability assigned to each reward class while removing within-class differences, examining whether the ratings contribute beyond favoring rewarded transitions. These controls make the study more informative by investigating how the guidance affects learning.

  2. The empirical reporting is appropriately qualified and provides detailed documentation for inspection. The paper acknowledges task-dependent performance, the substantial contribution of engineered navigation features on some tasks, and the lack of consistent improvements over PER or from subgoal guidance. It also carefully limits the Qwen comparison to the tested configuration. Appendices C and F document evaluation seeds, query instructions and excerpts, answer options, input examples, and memory contents and reset rules. Together, these qualifications and implementation details help readers interpret the results and understand the tested system, which is particularly valuable for a study involving a closed-source model.

Critical weaknesses
  1. The motivation for choosing Jev is not adequately validated. A central motivation is its lower cost and latency relative to generative foundation models, yet neither advantage is measured in the reported experiments. Section 2.2 refers readers to Appendix A for latency and cost information, but this information is absent. More importantly, the comparison with an LLM is restricted to direct control in RQ1. In RQ3-RQ5, which evaluate the paper's main contributions-using Jev as a reference policy, exploration judge, and replay rater-the main baselines are DQN and, for replay, PER; there are no corresponding LLM-based components. These experiments therefore demonstrate benefits from adding Jev on selected tasks, but do not establish its advantages over LLM-based guidance or distinguish its contribution from that of other reasonable priors. Since Qwen3-8B is already deployed with an aligned input interface, using option log-likelihoods to construct a reference distribution and prompting it with the same rating questions would provide natural component-level comparisons. Results under comparable query-cost budgets, together with measured latency and total cost, are needed to substantiate the motivation.

  2. The broader significance and generalizability remain insufficiently established. RQ4 lacks standard exploration baselines such as count-based exploration or RND, making it difficult to assess the value of Jev's judgments relative to established alternatives. The empirical findings are also tied to a single closed-source model and a specific API version. Meanwhile, the role analysis in Section 3-for example, the distinction between ordinal scores and cardinal value estimates-applies more broadly to scoring models and does not by itself establish a contribution specific to this model class. Combined with the limited training budgets, small number of training seeds, and task-dependent results, the current evidence leaves the significance and generalizability of the approach unclear.

  3. The attribution of gains to Jev is confounded by engineered inputs and information asymmetry. Jev receives structured scene descriptions, task rules, and, for FourRooms and MultiRoom-N4, observed-map memory with candidate routes and path distances computed by a hand-designed module, whereas the student learns from raw local observations. RQ2 shows that removing these navigation features reduces pure-Jev success from 96% to 40% on FourRooms and from 100% to 38% on MultiRoom-N4, indicating that much of the direct-control performance stems from the supplied route structure. Moreover, DoorKey-8 x8 rises from 62% (RQ1, local input) to 100% (RQ2, even without navigation features), suggesting that the memory component itself also has a large effect that is not separately analyzed. A rule-based policy that follows the supplied routes without Jev would clarify how much of the reference-guided gains in RQ3 is attributable to Jev rather than to the input pipeline. The details of the route construction (search procedure, form of progress/priority hints) should also be specified.

Questions for authors

1.Which of Jev's claimed properties (low cost/latency, calibration, typed outputs) is responsible for the gains in RQ3-RQ5? Could the authors report results with an LLM-based counterpart for each channel, e.g., Qwen3-8B option log-likelihoods as the reference distribution and the same rating questions for bonuses and replay priorities, ideally under comparable query budgets?

2.Could the authors report per-query latency and cost, and the total query cost of each method? Section 2.2 defers this information to Appendix A, but it does not appear there.

3.How much of the reference-guided gain in RQ3 would remain if the teacher were a rule-based policy that follows the supplied routes without Jev? Relatedly, could the authors report the same actor-critic learners used in Jev-BC and Jev-PEX without Jev, so that gains from the learner change can be separated from gains from Jev?

4.How are the candidate routes, path distances, and progress/priority hints constructed (search procedure, tie-breaking, exact fields shown to Jev)?

5.DoorKey-8x8 rises from 62% in RQ1 to 100% in RQ2 without navigation features, and MultiRoom-N2 from 58% to 90%. Is this due to the observed-map memory? Could the authors ablate memory separately?

Reviewer_L9Y3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Jev, a closed "decision model" that answers typed questions (a choice, a point on an ordered rubric, or a probability) about a text description in one forward pass. It first argues which RL objects such answers can fill (a policy, an ordinal value, a reward and a transition model, but not a cardinal value), then uses Jev, frozen, in three roles: as a reference policy, as an exploration judge, and as a replay rater. It compares Jev zero-shot with Qwen3-8B, and evaluates the learning methods against DQN and prioritised replay on nine MiniGrid tasks and three Atari games.

Critical strengths

The paper is well written, and the question it asks is clear: what a decision model that generates no text can do inside an RL learner, both on its own and as a component of training (lines 53–54). The experiments are well thought out, and designed with particular care to leave little space for confounders: Jev and Qwen3-8B are compared under a matched information interface (lines 68–71); an ablation removes the engineered navigation features (Table 2); retraining identical learners on each method's recorded experience separates the effect of the collected data from that of the auxiliary reward (Figure 9); and replacing Jev's scores with visitation-bin means separates them from a count summary (Figure 10).

Critical weaknesses

Many of the paper's claims rely on knowledge that appears to be proprietary and cannot be accessed, so they cannot be verified. Because of this limitation, the study can do little more than report results on questions of interest for Jev. This is fine, but claims on the specific features of JEV that cannot be verified should not appear in the paper. The paper is also very dense: it proposes many things at once, and does not have the space and depth to investigate each of them fully.

Major

1. Unverifiable claims

  • "Jev, by contrast, answers a batch of typed questions in one fixed-cost forward pass" (line 100): this claim is unsupported, as it cannot be verified: Jev is closed source, and the paper gives no details on what it does.
  • "Grows with context length" (lines 99–100): it is not clear why this is stated, since there is no term of comparison. Does Jev's forward pass not grow with context length? The paper does not show it.
  • "TypeSafe AI describes its training procedure, RLCD, ..." (line 104): the reference (TypeSafe AI, 2026b) is uninformative and does not contain any description of RLCD, which seems to be a closed, proprietary method. The paper should not state what cannot be verified.

2. Methods

  • "The value function is the most demanding. Its target r+γmax⁡a′Q(s′,a′)r + \gamma \max_{a'} Q(s', a') adds a reward to a value and discounts the sum" (line 163): this is incorrect. A Q-function is a model of the return. The equation refers to a specific method to learn that function, i.e., one-step TD learning, which is not the current standard. For this reason, if Jev can approximate a reward, it should also be able to approximate a Q-function.
  • Similarly, "since its update performs arithmetic on the answer" (line 58): what exactly does this mean? A value function is as static as a policy.
  • "All four take a state-like input" (line 154): some take an action as well (e.g., a Q-function). Does this change what Jev can represent? If not, this should be stated explicitly.
  • "Reward is the least demanding of the four. It enters the objective as a sum of terms" (line 159). Why is a reward a sum of terms?
  • "A choice question supplies both whenever those sets are finite" (line 162): is being finite enough? What if the set is large (e.g., Atari)? Is the model still able to provide a distribution over such a large set?
  • Line 191 suggests that Jev will be used as a ranker for the Q-values of actions in the experiments, but no experiment shows that. Also, using a ranker rather than a critic has been explored in the past (Lagoudakis & Parr, 2003), but the text gives no reference to it.
  • Line 205: if "Jev as Transition" is set aside, it should not be in the paper.

3. Benchmarks and input construction

  • There exist consolidated benchmarks to investigate LLMs, VLMs and decision models for games and RL (e.g., BALROG). However, the paper only tests on MiniGrid, usually considered a diagnostic task, and on three Atari games.
  • Jev sounds great, but such an effort to solve MiniGrid sounds like overkill: these environments have been solved. If Jev is such a huge step forward compared to LLMs (or such a new frontier), I would rather see investigations on more complex problems. For example, how does it do on NetHack?
  • Not using these benchmarks raises some questions (which could also be answered without them, but that requires further investigation and validation of the benchmarks used in the paper):
    • In MiniGrid, the task information leaks information on the optimal policy, e.g. "Task information. Collect the matching key, unlock the door, then enter the goal" (line 1298).
    • The same is true for Atari, where the input construction adds information such as "RGB-detected objects and estimated motion: oxygen, divers and enemies for Seaquest; vehicles and the chicken for Freeway; and the cannon, aliens, projectiles and shields for SpaceInvaders" (lines 822–824).
    • It is not clear how much this additional information contributes, as there are no ablations for it, so the effects of Jev cannot be disentangled from it. In support of this, the harder tasks (Atari) show lower, no, or even negative gains with respect to the baselines, suggesting that the input construction plays a crucial role in the performance.
  • How the input is constructed (line 816) is unclear, and not described in enough detail to reproduce.
  • Because observations cannot be fed raw to Jev, and inputs must be constructed, it is not clear whether the gains come from the information that the construction leaks into the inputs.
  • In RQ2, it is not clear what "We keep Jev fixed and remove candidate routes, path distances, and progress/priority hints on four tasks, retaining identical observational memory and task rules" (line 368) means: there are no details on how the ablation of the input construction is carried out, making it non-reproducible.

4. Experiments

  • Baselines are lacking. Why only Qwen?
  • Qwen disappears as a baseline in RQ3 and RQ4: why?
  • RQ3 and RQ4: some tasks benefit more from one Jev role than from another, but why is not investigated, even though it is a key component of the findings.
  • RQ3: how are the "fixed logged states" (line 393) sampled, and why? This is not stated, and the results are not reproducible.

5. Literature review Much of the relevant literature is missing. This seems a consequence of the paper addressing too many problems at once (Jev as a policy, as an exploration guide and as a replay prioritiser): none of them is investigated in depth, and each lacks the work it should be positioned against. The most relevant missing works are listed under References below.

6. Conclusions The conclusions sometimes say more than the results show, and they are much more cautious than the abstract, which therefore overclaims.

  • The conclusion states "These uses can improve early learning" (line 482), but the experiments section says otherwise:
    • "improve selected tasks ... no tested configuration is uniformly best" (lines 418–419);
    • "improves selected tasks but does not consistently outperform PER" (lines 449–450).
  • "Our findings demonstrate that a decision model need not be a reliable standalone controller to improve RL training" (lines 484–485): this is already known. Learning from imperfect teachers or priors is the premise of works the paper itself cites (Zhou et al., 2024; Yan et al., 2025) and of imitation learning from suboptimal experts.
  • The conclusions should say plainly that part of the gain may come from the hand-built inputs, if so.

Minor

7. Writing and notation

  • "Generates nothing" (line 40) is contradictory, as the paper then states that "it returns typed answers" (line 41).
  • "Typed questions", "typed answers": these terms must be defined, since the paper presents them as a major point of difference with LLMs.
  • "The objects it can fill group into the positions it can occupy, and each position admits several roles" (line 58). "Objects", "positions" and "roles" are used before being defined (Section 3), and inconsistently: "role" means an RL object in Section 3.1 and a use of a position in Section 3.2. The four positions are not a grouping of the four objects: Meta Regulator corresponds to none of them, and Score Rater merges R and Q despite their different domains. The intuition is hard to grasp.
  • "An RL system asks four things of a function approximator" (line 152): it is the opposite. RL does not "ask"; a function approximator can be used to estimate some of the quantities in an RL problem.
  • The MDP description is too generic and does not give useful information. For example, what exactly is the state space? A sequence of cc tokens? What is the action space? A discrete set over tokens? In fact, the "Jev model" paragraph has to specify what a "situation" is, to differentiate it from a "state".
  • Most environments in the experiments are POMDPs (e.g., Atari), not MDPs.
  • Yu\mathcal{Y}_u is loosely defined. What is its space?
  • Xy\mathcal{X}_y is loosely defined, because the space of X\mathcal{X} is loosely defined. What is its space?
  • Line 185: CC is the action space, but that was defined as A\mathcal{A} in the preliminaries, where CC is a set of options.
  • Line 199: XS×A\mathcal{X}_{S \times A} is not defined.
  • Line 261: HtH_t has not been introduced yet. How is it constructed?

References

Direct counterparts of the paper's algorithms:

  • Replay rater (§4.3): Sharony, E., Jurgenson, T., Krupnik, O., Di Castro, D., and Mannor, S. VLM-guided experience replay. arXiv:2602.01915, 2026. (A VLM scores experience; the score multiplies the TD error for prioritisation.)
  • Exploration (§4.2): Hao, Q., Song, Y., Liao, Q., Yuan, J., and Li, Y. LLM-Explorer: A plug-in reinforcement learning policy exploration enhancement driven by large language models. In Advances in Neural Information Processing Systems 38, 2025. (LLM-set exploration for DQN, DDPG and TD3, on Atari.)
  • Meta Regulator (lines 194–199): Hu, B., Zhao, C., Zhang, P., Zhou, Z., Yang, Y., Xu, Z., and Liu, B. Enabling intelligent interactions between an agent and an LLM: A reinforcement learning approach. Reinforcement Learning Journal (RLC), 2024. (When2Ask: learns when to query the LLM, on MiniGrid.)
  • Reference policy (§4.1): Hu, H. and Sadigh, D. Language instructed reinforcement learning for human-AI coordination. In ICML, 2023. (An LLM prior policy regularises the RL objective.)
  • Research questions 1–2 (lines 53–54): Klissarov, M., Hjelm, D., Toshev, A., and Mazoure, B. On the modeling capabilities of large language models for sequential decision making. In ICLR, 2025. (LLMs as direct policies vs as reward models.)

Action evaluation, reward and credit assignment (Score Rater):

  • Pignatelli, E., et al. Assessing the zero-shot capabilities of LLMs for action evaluation in RL. ICML 2024 Workshop on AutoRL; arXiv:2409.12798, 2024.
  • Qu, Y., et al. Latent reward: LLM-empowered credit assignment in episodic reinforcement learning. arXiv:2412.11120, 2024.
  • Klissarov, M., D'Oro, P., Sodhani, S., Raileanu, R., Bacon, P.-L., Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. In ICLR, 2024.
  • Zheng, Q., Henaff, M., Zhang, A., Grover, A., and Amos, B. Online intrinsic rewards for decision making agents from large language model feedback. Reinforcement Learning Journal (RLC), 2025. (Distils LLM feedback into a reward model: relevant to the cost argument, lines 97–100.)
  • Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In ICLR, 2023.
  • Xie, T., et al. Text2Reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
  • Yu, W., et al. Language to rewards for robotic skill synthesis. In CoRL, 2023.
  • Fan, L., et al. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In NeurIPS, 2022.
  • Ahn, M., et al. Do as I can, not as I say: Grounding language in robotic affordances. In CoRL, 2022.

Exploration, interestingness and goal selection:

  • Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In ICML, 2023.
  • Zhang, J., Lehman, J., Stanley, K., and Clune, J. OMNI: Open-endedness via models of human notions of interestingness. In ICLR, 2024.
  • Faldor, M., Zhang, J., Cully, A., and Clune, J. OMNI-EPIC: Open-endedness via models of human notions of interestingness with environments programmed in code. In ICLR, 2025.
  • Lu, C., Hu, S., and Clune, J. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. In ICLR, 2025. (A foundation model judges which states are interesting to return to.)
  • Colas, C., Teodorescu, L., Oudeyer, P.-Y., Yuan, X., and Côté, M.-A. Augmenting autotelic agents with large language models. In CoLLAs, 2023.
  • Mu, J., et al. Improving intrinsic exploration with language abstractions. In NeurIPS, 2022.
  • Tam, A., et al. Semantic exploration from language abstractions and pretrained representations. In NeurIPS, 2022.
  • Nottingham, K., et al. Do embodied agents dream of pixelated sheep? Embodied decision making using language guided world modelling. In ICML, 2023.
  • Wang, G., et al. Voyager: An open-ended embodied agent with large language models. TMLR, 2024.

LLMs as policies, executed directly (research question 1):

  • Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In ICML, 2023.
  • Tan, W., et al. True knowledge comes from practice: Aligning LLMs with embodied environments via reinforcement learning. In ICLR, 2024.
  • Szot, A., et al. Large language models as generalizable policies for embodied tasks. In ICLR, 2024.
  • Li, S., et al. Pre-trained language models for interactive decision-making. In NeurIPS, 2022.
  • Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022.
  • Paglieri, D., et al. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In ICLR, 2025.
Questions for authors

The questions are raised with each point in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.