- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Bad paper. Probably almost fully written by AI even if experiment ideas are good. There is no structure in that paper, no background material, no clear methodology sections. We have to guess what the setting and the contributions are. I will not read more of past section 3.
Critical strengths
Important topic of replication.
Critical weaknesses
First sentence is incomprehensible: Agent RL reports a credit-assignment comparison as “A beats B by X over three seeds” . This is not what credit-assignment means; especially in an RL context. "TRL and verl-agent configuration" neither TRL and verl-agent are defined. Many sentences are unnecessarily long and cant be understood e.g." The shrinkage belongs to the reading: the method arm sits at the ceiling of the success scale in five of six depths; at a fixed update count the gap does not shrink; the rate at which runs first reach the success threshold shows no resolvable trend; and off the ceiling, at constant anchor coverage, the endpoint gap grows, factor 0.44 [0.26, 0.70]" What is a method arm ? " On the field’s own stack the endpoint and hazard readings differ in size, log-difference 0.7507 [0.2259, 1.9358]" what the heck is a field's own stack ?????
The main premise of the paper is false: ``One clause of that sentence is “± standard deviation over three seeds”. Its implicit promise is that fixing the seed fixes the run''
Indeed, unless stated otherwise, running multiple seeds of a random process, such as RL training, is just a way to get an estimate of the score of an agent. Presenting mean +- std has never been a claim on bitwise reproducibility: see e.g. Is Deep RL reproducible? State of the art and new tools https://inria.hal.science/hal-05645819/file/RR-9617.pdf
Questions for authors
Please write for humans.
3 comments
Thank you for correcting some text. I am still very confident that most of the writing was generated by AI which makes the original submission near impossible to understand. The paper seems to have changed quite a bit and I will not review a new manuscript. Please feel free to prove me wrong.
Thank you for reading our reply. You are right that the original submission was hard to read, and that is our responsibility.
The revision is the same study. It reports the same three findings. What changed is mainly the writing and the structure, and the evidence grew: fifteen preregistered experiments, with over 1,200 runs, finished after the submission (listed in our general comment). Our replies to Reviewer KkEe list the two analyses that reviewer asked for (App. C.7 and App. K) and the one claim we withdrew (App. D.2).
AI use was disclosed. The submission form and the paper's Use of Generative AI statement (p. 11) say, by task, where AI tools were used, as ICLR asks. We reviewed all of it and are responsible for every sentence and every number. Each number in the main text is traced to its source and checked by programs in the supplementary code.
We understand that you may not return to the paper. Thank you for the time you gave it.
Thank you for the time you spent on the submission, and for calling replication an important topic. You were right on both main points. The paper was hard to read, and mean ± std over seeds is not a claim of bit-for-bit reproduction. We rewrote the whole paper with one aim, that a reader new to agent RL can follow it on a first reading. Pages 1–6 now give the background, a setup section (§2), every definition, our three contributions (p. 2) and a table of the main results.
Your points, in order:
"First sentence is incomprehensible." The abstract now opens "Papers proposing a credit-assignment method for language-model agents report the comparison in one sentence". §2.1 says what credit assignment means here. GRPO gives every turn of a trajectory the same score, while GiGPO also scores each turn against the turns of other trajectories that share its observation.
"neither TRL and verl-agent are defined." Both are now defined where they first appear (p. 2).
"Many sentences are unnecessarily long." We split them. The sentence you quote is now three labelled paragraphs of §4.2 (budget rule, estimand, evaluation step).
"What is a method arm?" The term is gone. The two methods are now called the method (GiGPO) and the baseline (GRPO).
"the field's own stack". This term is gone too, and each stack is now named (verl-agent, TRL, our own loop). We no longer claim the result in that sentence: our ALFWorld runs share most of their validation games, and once this is counted its interval no longer excludes zero (App. D.2).
"The main premise of the paper is false." We agree with you and with the Inria report you pointed us to, which we now cite in §1 and §7. Mean ± std over seeds estimates a method's expected score and the spread between runs. It was never a claim that a seed reproduces a run bit for bit. The sentence you quote implied otherwise, and we removed it. None of the measurements in §3 depends on it. Your report shows that one seed trains different agents on different machines, and §3 asks the same of the code that trains language-model agents. Re-running a seed reproduces a run in 1 of 15 default configurations, and among KnobChain runs launched together the seed explains at most about three tenths of the variance between runs.
Beyond §3. §4 holds the second finding. When the better method is near full success, a gap in success rates is set by how often the baseline still fails. Whether a benefit grows with task length therefore depends on the training budget and the evaluation step (Figure 5). This accounts for much of the growth in two of three published ALFWorld tables (Figure 3). §5 shows that three runs per method are too few to detect such a gap or its trend.
"Please write for humans." That was the aim of the rewrite. If any sentence, anywhere in the paper, is still unclear, tell us which and we will rewrite it.
AI use. The paper's Use of Generative AI statement (p. 11) discloses AI use by task. We have put a great deal of effort into revising the paper by hand, with the aim of making it clearer to read. The writing is our responsibility.
The submission cost you time, and we are sorry. Thank you for considering the revision.