- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Summary: The paper proposes LocusRL, a checklist-style procedure for diagnosing how LLM-provided rewards and actions are used in RL training, tested mainly on the authors' own Connect Four pipeline. My subjective opinion: My overall impression is very negative. The paper shows heavy traces of AI auto research:
- The main text figures look AI-generated, even the experimental results are drawn as infographics instead of being reported in tables, and the "findings" do not read like real insights from a human researcher.
- The appendix is filled with hashes, commit ids and python commands, which no proper technical paper would include.
- This does not read like a paper written by a qualified researcher, and its quality is far below the ICLR standard. I really suggest the authors first take responsibility for the quality of their own paper and complete basic academic training before submitting.
Critical strengths
None.
Critical weaknesses
W1: The presentation is far below the ICLR bar. All five figures in the main text are infographic-style images with decorative icons and "Key takeaways" boxes, not normal plots. Even the experimental data charts are AI-generated. Fig. 3(b) is missing one of its seven seed points, although the text says there are three ties and so on. The writing reads like a experimental log, with version tags such as "v0.4 delivery" and script file names. The appendix is filled with SHA-256 hashes, commit IDs, file paths, and python commands.
W2: The research question is unclear, and the findings are not real contributions. Most of the "findings" are problems in the authors' own pipeline
- The loss was implemented differently from how it was described.
- The reward cache covers only about a third of the lookups.
- A parsing fallback misses a winning move.
Questions for authors
- Were the figures, especially Figs. 2–5, generated directly from the experimental data? Please explain the missing S5 point in Fig. 3(b) and the uneven axis in Fig. 2(b).
- Obviously, this is not a paper that meets the standards of ICLR; it is merely a report on the engineering process.
Overall response and summary of revisions
We thank Reviewer 1 for recognizing the concrete findings from loss checks and trajectory replay, and our disclosure of repeated trajectories and differences between training setups. The reviews identify a shared problem with clarity. The revision now defines the setting and evidence-to-decision procedure in Sections 1–3 and maps the experiments to their separate hypotheses.
Research question and contribution
When an LLM provides a reward judgment or action, how can we establish what intervention actually reaches the learner or environment? The environment supplies terminal rewards; the LLM may add dense feedback or propose actions. Between proposal and effect lie a reward-delivery mechanism, an optimization objective, and an action-execution mechanism. LocusRL connects outcome comparisons to executable tests at these interfaces, then checks whether the resulting interpretation or correction survives a targeted follow-up. Its contribution is this operational diagnosis-and-validation protocol and the empirical distinctions it establishes.
The experiments address four different questions:
Is judgment quality model-dependent? On the same 120-board test, the stronger API configuration and Qwen base achieve action success of 116/120 versus 69/120, and reward-direction success of 106/120 versus 16/120.
Does sensitivity to reward imply benefit? A three-seed MLP control using a corrected one-step tactical reference yields terminal/positive/reverse wins of 95/180, 95/180, and 3/180.
Does a correct local repair improve new games? Six fixed checkpoints evaluated in 360 original/corrected pairs produce 64 replacements at three boards. Two pairs change from loss to win, both in one training seed.
Are historical records independent evidence? Input and trajectory audits show that the added-batch mean Hybrid-minus-Policy contrast changes from -0.2381 to +0.1134 percentage points after input deduplication.
These studies separate judgment correctness, learning sensitivity, correction coverage, and terminal utility. A strong judgment score does not establish an RL gain; a valid repair does not imply that its trigger occurs often or changes many outcomes. The new paired action study makes the latter distinction explicit: 181 pairs remain wins, 176 remain losses, one remains a draw, and two change from loss to win.
We have regenerated the historical seed and terminal-replay figures (revised Figures 1 and 3) from numeric inputs with programmatic coordinates, restored every seed marker, and checked the historical replay board against the original episode logs. The historical replay certificates and the new 720-game evaluation are separate evidence. Training seeds remain the replication blocks; repeated paths and multiple checks within a game are not additional replications.
The stronger-model comparison uses different inference budgets, the reward-direction control uses a rule-based MLP setup, and the action study uses fixed Qwen adapters. We report their respective questions rather than combining them into one estimate of LLM training benefit. Our reviewer-specific responses provide the definitions, numerical corrections, and scope needed to assess each result.
Revision locations: Sections 4.1, 4.3–4.5; Tables 4–6; Appendices T, V–X.