acceptodds
Under review as a conference paper at ICLR 2027

LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games

Abstract

Large language models can intervene in reinforcement learning through both reward design and action selection, yet aggregate performance offers an incomplete account of what these interventions actually do. Similar returns can conceal different learning mechanisms, while plausible rewards can induce undesirable behavior. We introduce LocusRL, a diagnostic framework that connects controlled reward–policy comparisons with audits of reward judgments, signal delivery, optimization objectives, and executed actions. The framework traces performance differences to testable explanations and checks targeted corrections through executable rules and counterfactual replay. Across two evaluation batches covering ten Connect Four training seeds, we uncover seed-dependent reversals in intervention effects and show how tracing actual updates changes their interpretation: historical Qwen training operates through reward-weighted teacher-action likelihood. A separate matched three-seed reward-direction experiment distinguishes sensitivity to a learning signal from its usefulness. With terminal rewards held fixed, a sign-reversed dense oracle yields a 2.8% aggregate win rate, compared with 57.2% for terminal-only training and 46.7% for the positive dense oracle. Thus, a reward can strongly influence learning without improving performance. At the decision level, counterfactual replay verifies a winning correction to a diagnosed action error. Complementary experiments in Leduc and reward-validation studies in Goofspiel extend the analysis to imperfect-information settings, revealing how reference-label definitions and validation-data exposure affect intervention assessment. Together, these findings show why evaluating LLM interventions requires tracing how their outputs become learning signals and actions. LocusRL turns aggregate outcomes into actionable diagnoses and verifiable corrections.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, LLM interventions, reward shaping, diagnostic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pMfsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary: The paper proposes LocusRL, a checklist-style procedure for diagnosing how LLM-provided rewards and actions are used in RL training, tested mainly on the authors' own Connect Four pipeline. My subjective opinion: My overall impression is very negative. The paper shows heavy traces of AI auto research:

  1. The main text figures look AI-generated, even the experimental results are drawn as infographics instead of being reported in tables, and the "findings" do not read like real insights from a human researcher.
  2. The appendix is filled with hashes, commit ids and python commands, which no proper technical paper would include.
  3. This does not read like a paper written by a qualified researcher, and its quality is far below the ICLR standard. I really suggest the authors first take responsibility for the quality of their own paper and complete basic academic training before submitting.
Critical strengths

None.

Critical weaknesses

W1: The presentation is far below the ICLR bar. All five figures in the main text are infographic-style images with decorative icons and "Key takeaways" boxes, not normal plots. Even the experimental data charts are AI-generated. Fig. 3(b) is missing one of its seven seed points, although the text says there are three ties and so on. The writing reads like a experimental log, with version tags such as "v0.4 delivery" and script file names. The appendix is filled with SHA-256 hashes, commit IDs, file paths, and python commands.

W2: The research question is unclear, and the findings are not real contributions. Most of the "findings" are problems in the authors' own pipeline

  1. The loss was implemented differently from how it was described.
  2. The reward cache covers only about a third of the lookups.
  3. A parsing fallback misses a winning move.
Questions for authors
  1. Were the figures, especially Figs. 2–5, generated directly from the experimental data? Please explain the missing S5 point in Fig. 3(b) and the uneven axis in Fig. 2(b).
  2. Obviously, this is not a paper that meets the standards of ICLR; it is merely a report on the engineering process.
Reviewer_ZhYSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers ways to use LLMs to provide additional feedback during RL training, and how to evaluate such methods. One of these is to use previously collected offline judgements in a "cache" collected from an LLM, similar to a hybrid online-offline RL setting. The authors then consider a way to evaluate this and other methods by considering multiple rubrics ( agreement to a reference, density, ...) and evaluate them in post-training LLMs for three simple games.

Critical strengths

None

Critical weaknesses
  • The presentation of the paper is bad. There are many nonstandard and undefined terms. The claims do not map clearly to contributions in the paper. Overall the paper is difficult to read and understad, and seemingly AI generated in large parts.

  • The claims that are made are not sufficiently supported as far as they can be understood. This is partially because the experimental setup is not clear, and the methods are not clear, but even where it is clear the experiments are limited to Qwen 3B models trained on three very simple games.

  • The methods evaluated may not scale to more complex settings.

  • The proposed LocusRL evaluation framework is more subjective than easily quantifiable. (what is "agreement", what is "evidence validity"). An evaluation framework ideally should be more objective and provide clear actionable advice.

Questions for authors

None

Reviewer_pQBYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

LocusRL proposes a diagnostic workflow for examining how LLM-generated rewards and actions are used in training and execution, with case studies in competitive games.

Critical strengths
  1. The paper identifies concrete implementation issues through loss checks and trajectory replay.

  2. It acknowledges limitations in the evidence, including repeated trajectories and differences between training setups.

Critical weaknesses
  1. The problem is not clearly defined. The introduction discusses reward delivery and execution fallbacks before explaining the underlying setting. I struggled to understand which existing problem the framework is intended to solve and how it relates to prior work.

  2. The technical contribution is unclear. The proposed checks appear individually reasonable, but it is difficult to identify what the combined workflow adds beyond standard debugging and validation. The examples show that implementation problems exist, but do not clearly establish the value of the proposed method.

  3. The experiments are difficult to connect to the conclusions. Historical teacher-action training, on-policy learning, rule-based reward controls, and trajectory repairs use different setups. I found it hard to follow which hypothesis each experiment tests and how the results jointly support the framework’s contribution.

Questions for authors
  1. What does LocusRL enable beyond a conventional diagnostic checklist?

Comments on the paper

5 comments
AuthorsOpenReview ↗

Overall response and summary of revisions

We thank Reviewer 1 for recognizing the concrete findings from loss checks and trajectory replay, and our disclosure of repeated trajectories and differences between training setups. The reviews identify a shared problem with clarity. The revision now defines the setting and evidence-to-decision procedure in Sections 1–3 and maps the experiments to their separate hypotheses.

Research question and contribution

When an LLM provides a reward judgment or action, how can we establish what intervention actually reaches the learner or environment? The environment supplies terminal rewards; the LLM may add dense feedback or propose actions. Between proposal and effect lie a reward-delivery mechanism, an optimization objective, and an action-execution mechanism. LocusRL connects outcome comparisons to executable tests at these interfaces, then checks whether the resulting interpretation or correction survives a targeted follow-up. Its contribution is this operational diagnosis-and-validation protocol and the empirical distinctions it establishes.

The experiments address four different questions:

  1. Is judgment quality model-dependent? On the same 120-board test, the stronger API configuration and Qwen base achieve action success of 116/120 versus 69/120, and reward-direction success of 106/120 versus 16/120.

  2. Does sensitivity to reward imply benefit? A three-seed MLP control using a corrected one-step tactical reference yields terminal/positive/reverse wins of 95/180, 95/180, and 3/180.

  3. Does a correct local repair improve new games? Six fixed checkpoints evaluated in 360 original/corrected pairs produce 64 replacements at three boards. Two pairs change from loss to win, both in one training seed.

  4. Are historical records independent evidence? Input and trajectory audits show that the added-batch mean Hybrid-minus-Policy contrast changes from -0.2381 to +0.1134 percentage points after input deduplication.

These studies separate judgment correctness, learning sensitivity, correction coverage, and terminal utility. A strong judgment score does not establish an RL gain; a valid repair does not imply that its trigger occurs often or changes many outcomes. The new paired action study makes the latter distinction explicit: 181 pairs remain wins, 176 remain losses, one remains a draw, and two change from loss to win.

We have regenerated the historical seed and terminal-replay figures (revised Figures 1 and 3) from numeric inputs with programmatic coordinates, restored every seed marker, and checked the historical replay board against the original episode logs. The historical replay certificates and the new 720-game evaluation are separate evidence. Training seeds remain the replication blocks; repeated paths and multiple checks within a game are not additional replications.

The stronger-model comparison uses different inference budgets, the reward-direction control uses a rule-based MLP setup, and the action study uses fixed Qwen adapters. We report their respective questions rather than combining them into one estimate of LLM training benefit. Our reviewer-specific responses provide the definitions, numerical corrections, and scope needed to assess each result.

Revision locations: Sections 4.1, 4.3–4.5; Tables 4–6; Appendices T, V–X.

AuthorsOpenReview ↗

Response to Reviewer pQBY: Problem Definition, Contribution, and Experimental Connections

Thank you for recognizing the concrete loss-check and trajectory-replay findings and our disclosure of evidence limitations. Sections 1–3 now introduce the setting before implementation-specific terms.

  1. Problem and setting

We study systems in which an LLM supplies dense reward judgments, proposed actions, or both, while the environment supplies terminal outcomes. A proposed judgment may never reach an update, and a proposed action may differ from the executed action. The research question is which observable tests distinguish these mechanisms when aggregate return alone cannot. The historical four-arm comparison crosses system configurations whose policy families also differ in architecture, data, and update rule; it is therefore a descriptive comparison, not an identified single-factor causal decomposition.

Its role is complementary to reward-synthesis and policy-training methods: it asks which interface-level evidence justifies interpreting an observed return as the intended intervention.

  1. What does LocusRL enable beyond a conventional checklist?

The contribution is a traceable connection from an interpretation to a falsifiable check and a follow-up decision. Consider the execution case:

• Final-action legality initially appears satisfactory, but state reconstruction reveals a legal fallback that discards an immediate win.

• Preserving the recorded prefix and executing a winning alternative certifies the local correction: 20 Hybrid loss records share one first-repair board under two checkpoints. They are not 20 independent tactical cases.

• In a separate prospective evaluation, the fallback-only rule makes no changes. Inspection identifies valid parsed actions outside that trigger as another source of missed wins.

• Expanding the check to every agent decision produces 64 replacements at three boards in 360 paired games. Only two pairs change from loss to win; both occur in the same training seed, one per arm. Thus the executable correction works when applicable, while its measured terminal benefit is small and concentrated.

This chain distinguishes an execution defect, trigger coverage, and actual outcome benefit. A legality check or aggregate return alone does not supply that distinction. The same structure applies to objective identity and reward validation: checking the logged numerical loss changes the interpretation of historical teacher-action training; matching validation states against prior exposure changes whether a reward candidate qualifies as independently confirmed. The individual checks are familiar. The proposed contribution is their explicit hypothesis-to-evidence-to-decision organization, demonstrated through these consequential findings. We have not measured diagnostic superiority over expert debugging.

  1. How the experiments connect

Historical runs motivate competing explanations. Shared-runtime reevaluation tests fixed-checkpoint behavior under one execution contract. Separate on-policy runs test a different, explicitly sampled update mechanism. The rule-based reward-sign control tests whether a learning signal changes behavior and whether its positive direction helps. Historical terminal replay tests a sufficient local action correction; the new paired games test its activation and terminal effect. These are complementary tests, not pooled estimates of one treatment. In the separate on-policy follow-up, all six Policy/Hybrid runs completed 100 updates with observed parameter changes, checkpoint reloads, and a maximum pre-update sampled/recomputed log-probability difference of 0.0. The SFT-only reference is not strictly budget-matched to historical runs whose actual update count and demonstration exposure remain unconfirmed.

For example, the corrected-reference MLP control gives terminal/positive/reverse wins of 95/180, 95/180, and 3/180. Positive-minus-terminal seed differences are +10, 0, and -10 wins. This separates strong directional sensitivity from a consistent positive advantage, without attributing a historical Qwen result to that rule-based reward.

  1. Replication and presentation

The added-seven-seed audit changes the small Hybrid-minus-Policy mean from -0.2381 to +0.1134 percentage points after deduplicating reconstructed rollout inputs. The seed signs remain two positive, two negative, and three tied. We therefore interpret the result as unstable across seeds, rather than a reliable negative average effect. The paired training seed is the replication block; neither unique-path counts nor nonzero signs estimate a general effective sample size. The corrected plots show the original two batches separately, including every tie. The revision makes these explanation-to-check links explicit.

Locations: Sections 1–3 (Table 1), 4.1 and 4.5 (Table 6); Appendices T, V and X.

AuthorsOpenReview ↗

Response to Reviewer ZhYS: Operational Metrics, Model Dependence, and Generality

Thank you for raising clarity, quantification, model dependence, and transfer.

  1. Setting and terminology

The cache is a lookup table of previously collected LLM reward judgments. A hit supplies a dense term; a miss masks that term while preserving the environment reward. It is not, by itself, an offline-RL dataset or a hybrid online/offline RL algorithm. Historical Qwen teacher-action adaptation, separately verified on-policy learning, and MLP reward-direction controls have distinct objectives and evidence roles. The underlying question is how a nominal LLM output becomes an actual learning signal or executed action.

  1. The quantities are operational, not a subjective composite score

Each diagnostic has a specified input, denominator, and interpretation:

• Reference agreement: outputs satisfying a fixed task reference divided by all scored cases, retaining invalid outputs in the denominator. This checks whether judgments satisfy that reference.

• Signal delivery: exact cache hits divided by lookups; separately, nonzero dense-reward optimization records divided by all optimization records. These measure whether and how often the signal reaches the relevant interface.

• Objective identity: logged loss checked against the stated formula; for on-policy learning, sampled versus recomputed pre-update action log probability. These identify the update mechanism actually tested.

• Execution: executed actions checked against legal actions and executable winning-action sets. This identifies legal actions that miss certified alternatives.

• Evidence validity: reconstructed input/path identity and validation-state exposure matching. These determine which records support replication or independent confirmation.

Cache access and nonzero optimizer exposure have different denominators and are not interchangeable. Human interpretation selects the hypothesis and reference; once specified, counting, matching, and state replay are executable. There is no overall subjective “evidence validity” score. The supplied independent grid checker reproduces all 480 test-task grades directly from the recorded boards and raw model outputs. System-level fault-detection recall and human-time savings have not been measured.

  1. Stronger-model diagnostic control

On the same 120 test boards, balanced across five tactical categories, the GLM-5.3-flash configuration achieves 116/120 action-task successes (96.7%) and 106/120 reward-direction successes (88.3%). The Qwen2.5-3B-Instruct base achieves 69/120 (57.5%) and 16/120 (13.3%). GLM has 3 invalid action outputs and 12 invalid reward outputs; all remain in the denominator. The action criterion uses the case-specific accepted action set, including cases with multiple acceptable moves, rather than equating every success with a unique best move.

GLM uses thinking with a 32,768-token output cap; Qwen uses greedy decoding with a 1,024-token cap. This compares model/inference configurations, not matched compute or isolated parameter scaling. It establishes that the diagnosis distinguishes materially different judgment capabilities. It does not establish a training gain from replacing the reward model.

  1. Generality and scalability

Connect Four supplies the principal training and execution studies. Leduc contributes a separate learning comparison and validation-exposure audit; Goofspiel contributes reward-validation studies that stop before formal training. These are different evidence roles, not three equivalent full training benchmarks.

The transferable part is the interface-level protocol: reward access and masking, update identity, proposed versus executed actions, and evaluation/exposure provenance. Those checks can use logs without enumerating a game's state space. Exact winning-action certificates additionally require a suitable simulator or verifier; in Connect Four they are obtained by testing at most seven legal columns. General complex environments may require different references, and their coverage must be measured. We do not extrapolate the present games into a validated large-domain performance claim. The stronger-model comparison and the 360-pair execution follow-up strengthen specific tests while keeping that distinction explicit.

Locations: Sections 3 and 4.4 (Table 5), Discussion; Appendices C, V–X.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.