acceptodds
Under review as a conference paper at ICLR 2027

Seeds Do Not Replicate Runs: The Unit of Replication, the Estimand and the Budget Rule in Agent RL

Abstract

Agent RL reports a credit-assignment comparison as “A beats B by over three seeds”. Three choices hide in that sentence, and on one pool of runs we measure what each decides. First, a seed argument is not a unit of replication. Whether re-running it reproduces the run is decided below the library: of the TRL and verl-agent configurations we census, only in-process generation replays to the bit, on one model and task, and our own loop replays only with its nondeterministic kernels off. Within one launch, our registered designs leave the seed's share of run-to-run variance unresolved, with upper ends near a quarter. Across launches, a re-executed seed argument moves the endpoint as far as changing it. On the released verl-agent stack, sixteen seed arguments put the seed's share at , at most , at the final evaluation. Second, the size of a horizon effect is a property of the estimand and the budget rule. On KnobChain, a corridor whose length we set, 384 paired runs of Qwen2.5-7B-Instruct with a turn-level credit rule against an episode-level one show the endpoint gap shrinking nearly threefold with the horizon under a budget that grows with it. The shrinkage belongs to the reading: the method arm sits at the ceiling of the success scale in five of six depths; at a fixed update count the gap does not shrink; the rate at which runs first reach the success threshold shows no resolvable trend; and off the ceiling, at constant anchor coverage, the endpoint gap grows, factor . On the field's own stack the endpoint and hazard readings differ in size, log-difference . That experiment reproduces the published growth under both estimands. The separation holds at the registered threshold and not below it. Third, three runs per arm resolve neither the effect nor the horizon trend, and the effect itself is not in question: under this dense shaped reward the method multiplies the takeoff hazard by two to three at every horizon. Report replications in runs rather than seed arguments, name the estimand and the budget rule, and fix the success threshold before training. The environment and the code accompany this submission; the run pool and the registration record will be released with the paper.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM agents, credit assignment, GRPO, GiGPO, replication, seeds, nondeterminism, experimental methodology, horizon scaling, pre-registration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Js8bOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Bad paper. Probably almost fully written by AI even if experiment ideas are good. There is no structure in that paper, no background material, no clear methodology sections. We have to guess what the setting and the contributions are. I will not read more of past section 3.

Critical strengths

Important topic of replication.

Critical weaknesses

First sentence is incomprehensible: Agent RL reports a credit-assignment comparison as “A beats B by X over three seeds” . This is not what credit-assignment means; especially in an RL context. "TRL and verl-agent configuration" neither TRL and verl-agent are defined. Many sentences are unnecessarily long and cant be understood e.g." The shrinkage belongs to the reading: the method arm sits at the ceiling of the success scale in five of six depths; at a fixed update count the gap does not shrink; the rate at which runs first reach the success threshold shows no resolvable trend; and off the ceiling, at constant anchor coverage, the endpoint gap grows, factor 0.44 [0.26, 0.70]" What is a method arm ? " On the field’s own stack the endpoint and hazard readings differ in size, log-difference 0.7507 [0.2259, 1.9358]" what the heck is a field's own stack ?????

The main premise of the paper is false: ``One clause of that sentence is “± standard deviation over three seeds”. Its implicit promise is that fixing the seed fixes the run''

Indeed, unless stated otherwise, running multiple seeds of a random process, such as RL training, is just a way to get an estimate of the score of an agent. Presenting mean +- std has never been a claim on bitwise reproducibility: see e.g. Is Deep RL reproducible? State of the art and new tools https://inria.hal.science/hal-05645819/file/RR-9617.pdf

Questions for authors

Please write for humans.

3 comments

Thank you for the time you spent on the submission, and for calling replication an important topic. You were right on both main points. The paper was hard to read, and mean ± std over seeds is not a claim of bit-for-bit reproduction. We rewrote the whole paper with one aim, that a reader new to agent RL can follow it on a first reading. Pages 1–6 now give the background, a setup section (§2), every definition, our three contributions (p. 2) and a table of the main results.

Your points, in order:

"First sentence is incomprehensible." The abstract now opens "Papers proposing a credit-assignment method for language-model agents report the comparison in one sentence". §2.1 says what credit assignment means here. GRPO gives every turn of a trajectory the same score, while GiGPO also scores each turn against the turns of other trajectories that share its observation.

"neither TRL and verl-agent are defined." Both are now defined where they first appear (p. 2).

"Many sentences are unnecessarily long." We split them. The sentence you quote is now three labelled paragraphs of §4.2 (budget rule, estimand, evaluation step).

"What is a method arm?" The term is gone. The two methods are now called the method (GiGPO) and the baseline (GRPO).

"the field's own stack". This term is gone too, and each stack is now named (verl-agent, TRL, our own loop). We no longer claim the result in that sentence: our ALFWorld runs share most of their validation games, and once this is counted its interval no longer excludes zero (App. D.2).

"The main premise of the paper is false." We agree with you and with the Inria report you pointed us to, which we now cite in §1 and §7. Mean ± std over seeds estimates a method's expected score and the spread between runs. It was never a claim that a seed reproduces a run bit for bit. The sentence you quote implied otherwise, and we removed it. None of the measurements in §3 depends on it. Your report shows that one seed trains different agents on different machines, and §3 asks the same of the code that trains language-model agents. Re-running a seed reproduces a run in 1 of 15 default configurations, and among KnobChain runs launched together the seed explains at most about three tenths of the variance between runs.

Beyond §3. §4 holds the second finding. When the better method is near full success, a gap in success rates is set by how often the baseline still fails. Whether a benefit grows with task length therefore depends on the training budget and the evaluation step (Figure 5). This accounts for much of the growth in two of three published ALFWorld tables (Figure 3). §5 shows that three runs per method are too few to detect such a gap or its trend.

"Please write for humans." That was the aim of the rewrite. If any sentence, anywhere in the paper, is still unclear, tell us which and we will rewrite it.

AI use. The paper's Use of Generative AI statement (p. 11) discloses AI use by task. We have put a great deal of effort into revising the paper by hand, with the aim of making it clearer to read. The writing is our responsibility.

The submission cost you time, and we are sorry. Thank you for considering the revision.

Thank you for correcting some text. I am still very confident that most of the writing was generated by AI which makes the original submission near impossible to understand. The paper seems to have changed quite a bit and I will not review a new manuscript. Please feel free to prove me wrong.

Thank you for reading our reply. You are right that the original submission was hard to read, and that is our responsibility.

The revision is the same study. It reports the same three findings. What changed is mainly the writing and the structure, and the evidence grew: fifteen preregistered experiments, with over 1,200 runs, finished after the submission (listed in our general comment). Our replies to Reviewer KkEe list the two analyses that reviewer asked for (App. C.7 and App. K) and the one claim we withdrew (App. D.2).

AI use was disclosed. The submission form and the paper's Use of Generative AI statement (p. 11) say, by task, where AI tools were used, as ICLR asks. We reviewed all of it and are responsible for every sentence and every number. Each number in the main text is traced to its source and checked by programs in the supplementary code.

We understand that you may not return to the paper. Thank you for the time you gave it.

Reviewer_pdANOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates several important methodological issues in Agentic RL, including the reproducibility of random seeds, the sensitivity of horizon-dependent credit assignment conclusions to evaluation metrics and training budgets, and the reliability of few-seed evaluations. However, my background is primarily in traditional deep RL, and I am less familiar with recent developments in Agentic RL. From the perspective of classical deep RL, many of these issues have been studied extensively, so the conceptual novelty is somewhat limited. Nevertheless, I believe revisiting and systematically quantifying these problems in modern LLM-based Agent RL is valuable, particularly given the growing interest in long-horizon credit assignment.

Critical strengths
  1. In my view, this work has tried contributes to a broader question that Under what conditions can evaluation systematically mischaracterize the horizon dependence of credit assignment methods? (which I think is important for agentic rl) Although the paper does not fully address this question, I believe it makes meaningful progress toward understanding it.

  2. The experiments demonstrate how ceiling effects, training budget choices, and different estimands can lead to substantially different interpretations of horizon-dependent performance trends. The reproducibility analysis on contemporary Agent RL training stacks also provides useful evidence.

Critical weaknesses
  1. Several central observations, particularly regarding seed reproducibility, statistical reliability, and evaluation sensitivity, are already well established in traditional deep RL. The contribution is therefore primarily empirical and methodological. (Just in my view as an research in classical deep RL)

  2. Much of the analysis relies on a synthetic environment and specific model, reward, and training configurations. It remains unclear to what extent these findings generalize to more realistic Agentic RL tasks.

Questions for authors
  1. How do the authors expect these findings to generalize to more realistic long-horizon Agentic RL tasks, particularly those with sparse terminal rewards and more complex interaction dynamics?
1 comment

Thank you for the thoughtful and encouraging review. Your framing, under what conditions evaluation can mischaracterise the horizon dependence of credit-assignment methods, is the question §4 takes up within the settings §8 lists, and we agree that the paper does not settle it in general. You also found parts of the paper hard to follow. We rewrote it, and pages 1–6 now give the background, every definition and a table of the main results.

On novelty relative to classical deep RL. We agree that few runs mislead, and that nondeterminism beyond the seed changes results, are established in classical deep RL. §1 and §7 now say so, and Table 27 sets each known result beside what we add. What is new is, first, the horizon claim. Near the success ceiling a gap follows the baseline's remaining failures. This accounts for much of the reported growth with task length in two of three published per-type tables (in the ReBel table, about 61% of the growth in the log of the gap), and the growth recurs on GiGPO's own stack with the method at the ceiling. Second, which configurations of the stacks that train language-model agents reproduce a run bit for bit (1 of 15 at default settings). Third, how much of the variance between runs the seed explains there. Among runs launched together on KnobChain, with both methods pooled, three estimates of it range from 0.02 to 0.13, with one-sided upper bounds of at most 0.28. On ALFWorld it is 0.15 in the preregistered experiment and 0.31 with two experiments pooled in an exploratory analysis (0.25, with an interval containing zero, in the pooled reading its preregistration declared).

On generalisation to realistic long-horizon tasks with sparse terminal rewards. ALFWorld is a text household environment whose reward, under verl-agent's published recipe, is paid only on success, apart from a small penalty for invalid actions, and its task types need two to four sub-goals. §4.3 reports it, now with an experiment that runs the recipe to twice its budget. The growth in the published table recurs at the published budget (0.56×) and at twice it (0.63×), with the method at the ceiling (above 0.90) on one short task type, so by our own rule we do not read it as an effect of length. Where every success rate lies between 0.15 and 0.85, the growth is smaller and mostly not detected (exploratory). Around KnobChain we changed the credit-assignment method, the model family and the numerical precision, and the shrinkage recurred each time with the method under test at the ceiling, where §4.1 expects a gap to follow the baseline (§4.2). Under a purely terminal reward the runs learned only at three times the budget. There one preregistered prediction held and one missed its threshold by one seed (Appendix G.3).

As for what we expect to transfer, the identity behind the second finding holds for any pair of success rates. Wherever a method nears full success, a horizon trend in the gap can therefore depend on the budget rule and the evaluation step, whatever the reward. The three-run arithmetic is general. The census and the seed's share are numbers about particular stacks, but the way of measuring them transfers. We have not tested tool use, web environments or settings with several agents, and §8 names these as untested.

If ALFWorld's per-type results or the terminal-reward runs would help, we can post them here. Thank you again for your review.

Reviewer_KkEeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This is a measurement paper about what the standard sentence "method A beats B by X over three seeds" means in RL with LLM agents. It argues that the sentence hides two choices. The first is the unit of replication. Re-executing a seed argument replays a run bitwise only in specific configurations. The second choice is the estimand and the budget rule. It points out a number of problems and then proposes three reporting prescriptions, and every prediction is preregistered in a hash-stamped log, with 74 scored predictions and 11 misses reported.

Critical strengths
  1. Exceptional methodological care and transparency. The append-only, hash-stamped register; the eleven misses printed as scored (Table 39); failures of the authors' own machinery disclosed (Appendix M); and verification scripts that check printed numbers against their source records set a standard few empirical ML papers approach.
  2. An actionable census. Table 2 tells practitioners which configurations replay bitwise and what restores replay (e.g., keeping the generation engine in the serving process). This is directly useful to anyone releasing seeds as a reproducibility device.
  3. A clear demonstration that horizon conclusions depend on the reading. The ceiling decomposition, the 24-specification curve, the fixed-count versus proportional-budget contrast and the ALFWorld replication together show that "the benefit shrinks with horizon" can be a property of the scale and budget rather than of the method.
Critical weaknesses
  1. The presentation prevents the results from being used. This goes beyond style. Key claims are qualified so heavily within each sentence that their status is unclear: the abstract states that the seed's share is "unresolved, with upper ends near a quarter". For an ICLR audience the paper needs a rewrite organized around three or four primary claims, each with its estimate, interval and status (confirmatory, exploratory, bound). It feels like hedged AI writing.
  2. The title claims more than the resolved evidence. That seeds fail to replay runs bitwise under nondeterministic CUDA kernels is expected and documented. The substantive claim, that the seed accounts for a small share of run-to-run variance, rests on cross-batch experiments that confound launch batch with code revision; the run that would separate them "was never launched" (§8). The registered same-launch designs do not resolve the share. What is established is non-replay and an in-launch seed share of at most about a quarter, with wide uncertainty; the title and abstract should say so.
  3. Multiplicity remains despite registration. The paper reports dozens of readings across four pooling rules, five scales, 24 specifications and swept thresholds, and several results hold only at the registered threshold. A short, pre-specified list of primary hypotheses, with a multiplicity correction, would make the evidential weight clear.
Questions for authors
  1. Can you state, in a few sentences each, the paper's primary confirmatory hypotheses and their registered outcomes?
  2. Do you plan to run the same-launch experiment on the pool's own code revision, which would separate the launch-batch and code-revision components of the cross-batch shares?
  3. Does the replication reading hold for KnobChain under a standard stack (single-precision master weights, AdamW, clipping, a larger batch)?
  4. Can you add a human-coded subsample of the survey and report human–LLM agreement?
  5. Given a fixed compute budget, how should a practitioner divide runs between distinct seed arguments and re-executions? What does the recommendation to re-execute one configuration per arm twice buy, quantitatively?
2 comments

Thank you for the careful review and for the strengths you named. The census is now Table 2 (in full, Table 15), and the eleven misses are Table 31. The weaknesses and your first question are answered here, the other questions in a second comment.

Critical Weaknesses

"The presentation prevents the results from being used." We agree. We rewrote the paper around three findings, one section each, and Tables 2 and 3 (pp. 4 and 6) give the key results with their estimates, 95% intervals and status. §2.4 sets one rule for the text: an estimate in neither table is exploratory unless the text says it was preregistered. The abstract's "unresolved, with upper ends near a quarter" is now three estimates, from 0.02 to 0.13, with one-sided 95% upper bounds of at most 0.28. We have put a great deal of effort into revising the paper by hand, with the aim of making it clearer to read. If a sentence still reads as hedged, tell us which and we will rewrite it.

"The title claims more than the resolved evidence." We agree. Non-replay under nondeterministic kernels is expected. What the census adds is which configurations do replay, the part you called actionable. The claim that the seed's share is small now rests on a bound, on KnobChain with both methods pooled. The one-sided 95% upper bound of the ICC within a launch is 0.20 in bfloat16, 0.28 under a single-precision master copy and 0.22 on the main experiment's code revision (second comment). The bound grew from "a quarter" to 0.28 because the 0.28 comes from launches analysed after the submission. The new title asks a question and claims no result, and the cross-launch comparisons, which confound the launch with the code revision, are no longer used as evidence.

"Multiplicity remains despite registration." We agree. No hypothesis was registered as primary, and a list chosen now would not be pre-specified, so we corrected over the 77 registered tests that carried a statistic at the time (the launch described in the second comment was registered later and is not among them). Of the 35 rejections, 16 survive Holm–Bonferroni and 30 Benjamini–Hochberg. Three key results in Table 3 survive Holm (4.64×, ALFWorld's 0.56× and 0.44×). Two are measured with the method at the ceiling. The third lies close to both floor and ceiling, and the same runs show shrinkage once the method reaches the ceiling (§4). Other readings across pooling rules, scales, specifications and thresholds are exploratory unless marked. The ALFWorld difference D, which held only at the registered threshold, is withdrawn: our ALFWorld runs share most of their validation games, and once this is counted its interval no longer excludes zero (App. D.2; Figure 15). We can post the list of every test with its p-values here. One correction to the submission: it named two predictions that compare new runs with runs analysed before the prediction was filed. Counting every prediction whose statistic uses any such run, 19 of its 74 do, and §2.4 now reports the same count over the 154 predictions of the tally, which is 44.

Questions For Authors

"Can you state, in a few sentences each, the paper's primary confirmatory hypotheses and their registered outcomes?" None was registered as primary. For each finding, here is the claim and the preregistered predictions closest to it.

Finding 1: re-running a seed rarely reproduces a run, and the seed's share is bounded. The preregistered tests of deterministic kernels held: two runs of a seed were identical in 16 of 16 pairs on one host, and again across two hosts. Five predictions about re-runs failed (Table 31), among them that the ALFWorld ICC's interval would exclude zero (ICC 0.15). The bound comes from runs launched together on KnobChain: the ICC is 0.13 under the master copy (upper bound 0.28; registered after the first of its two launches had been analysed) and 0.02 on the main experiment's revision (second comment).

Finding 2: whether the benefit grows with the horizon depends on the estimand, the budget rule and the evaluation step. The preregistered estimates point in all three directions, depending on how the runs are read. Under the master copy the gap shrinks at the proportional budget (4.64×), with the method at the ceiling. At matched baseline progress, or with both horizons censored alike, no trend is detected. In the variant the gap grows at the preregistered evaluation step (0.44×); at the later step, where the method has reached the ceiling, the same runs show shrinkage (printed, not scored). On ALFWorld the published growth recurs (0.56×).

Finding 3: three runs per method are too few. This needs no test, because it is arithmetic. An exact paired test on three pairs cannot give a p-value below 0.25, so no three-run comparison can reach p < 0.05, however large the gap. With the spread we measured, a paired t-test on three runs detects our gap at H = 30 with power only 0.23.

Questions For Authors (continued)

"Do you plan to run the same-launch experiment on the pool's own code revision, which would separate the launch-batch and code-revision components of the cross-batch shares?" We have now run it, after your review, as a preregistered experiment on the main experiment's code revision (the submission's pool). It has the same sixteen seeds, both methods and three runs of each, 96 runs in one launch with the main experiment's own arguments, and all of them completed. By its preregistered rule it does not separate the launch from the code revision.

Both methods pooledEstimate95% interval
ICC within the launch0.02[−0.17, 0.26]
Variance between two runs of a seed, across launches over within one: against the main experiment's own runs (same revision)1.28×[0.58, 4.22]
The same ratio against the earlier re-run (later revision)1.07×[0.94, 1.25]

The preregistered rule states a bound on the seed's share only if the upper end of the ICC's interval is below 0.25, so this launch does not resolve it; its one-sided bound, computed afterwards, is 0.22. The first ratio is the comparison your question asks for. Its interval contains both one and the 1.76 of the earlier cross-launch comparisons, so it leaves the question open. The second lies wholly below 1.76, so that excess is not reproduced. §3.3 now says that what another launch adds is not settled. It is one launch at one horizon, twelve weeks after the main experiment and on other machines, with about 30 degrees of freedom per ratio (App. C.7).

"Does the replication reading hold for KnobChain under a standard stack (single-precision master weights, AdamW, clipping, a larger batch)?" In part. The loop already clips gradients. Under a single-precision master copy the reading holds: runs still do not reproduce (Table 15), the seed's share stays bounded (first comment) and the horizon shrinkage recurs (4.64× and 2.73×), each time with the method at the ceiling. At 32 and 64 episodes per update the gap stays positive in at least 14 of 16 seeds, but the ICC there was not measured (§8). We did not run AdamW on KnobChain (§8). The public stacks in the paper do use it, with single-precision weights: verl-agent reproduces a run in none of its configurations and TRL in 1 of 6 at default settings (Table 2). On verl-agent the ICC is at most 0.16 on Sokoban, and on ALFWorld it is 0.15 in the preregistered experiment (0.31 with two experiments pooled in an exploratory analysis; the pooled reading its preregistration declared, 0.25, has an interval containing zero).

"Can you add a human-coded subsample of the survey and report human–LLM agreement?" Yes, for all 78 papers. One author coded every paper by hand, knowing only the aggregate counts and not the per-paper answers of either language-model coding (one item, the list of credit-assignment papers, is not blind). The paper now prints the author's counts. 59 of the 78 papers state no number of runs (56 under both language-model codings), and 63 give no uncertainty for the headline result (61). Cohen's κ on the run count is 0.82 and 0.91. It is lowest, 0.42 and 0.37, on what the seed controls, an item on which no claim rests (App. K).

"Given a fixed compute budget, how should a practitioner divide runs between distinct seed arguments and re-executions? What does the recommendation to re-execute one configuration per arm twice buy, quantitatively?" Distinct seeds, each run once. Within a launch a second run of each seed costs between 1% (H = 100) and 22% (ALFWorld) of the precision at equal budget, in point estimate. The re-run of one seed per method costs two runs and buys a check of bit-for-bit reproduction. In a configuration like the one of §3.2, whose pairs of runs all differed, it detects non-reproduction with probability at least 0.994, which tells a reader whether an interval is over seeds or over runs. It cannot measure the seed's share, since its power to detect an ICC of 0.3 is 0.06 (App. I).

Thank you again for the detailed questions. If another analysis or table would help, we would be glad to add it. If the revision resolves your concern about presentation, we would be grateful if you would reconsider that part of your assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers for their detailed reviews; several of KkEe's questions led to new work. Below we summarise, for the reviewers and the Area Chair, how the revised version responds. The next comment lists the experiments completed since the submission, and our reply to each reviewer gives the details.

1. Language and structure (all reviewers). We thoroughly revised the language and the structure of the paper. The revised version is built around three findings, one section each (§3–§5), with a setup section that gives the background (§2), and the terms of §3 and §4 are defined at the start of each. Tables 2 and 3 give the key results with their intervals and status, and the title now asks a question.

2. Claims and status (KkEe). We narrowed what the title and abstract claim: the seed's share is stated with a bound, and every statement of status (preregistered or exploratory) was checked against the time-stamped register. We corrected for multiplicity over the 77 registered tests with a statistic; 16 of the 35 rejections survive Holm–Bonferroni and 30 Benjamini–Hochberg (App. L).

3. New experiments (KkEe, pdAN). Fifteen preregistered experiments, with over 1,200 training runs, finished after the submission (next comment). They include the same-launch experiment on the main experiment's code revision that KkEe asked for (App. C.7), experiments under a single-precision master copy (App. E.5) and ALFWorld trained to twice the published budget (App. D.4). One of the authors also coded all 78 surveyed papers by hand (App. K).

4. Premise and novelty (Js8b, pdAN). We agree with Js8b that mean ± std over seeds was never a claim of bit-for-bit reproduction, and we removed the sentence that implied it (§1, §3). Table 27 sets each result known from classical deep RL beside what we add (§7).

If anything is still unclear, we would be glad to clarify.

Experiments completed since the submission. The revision reports all fifteen preregistered experiments that finished after the submission, whatever their outcome.

Runs of one seed within a launch (App. C)

ExperimentRunsResultApp.
Re-run on the main experiment's code revision (KkEe's question)96ICC 0.02 [−0.17, 0.26]; against the later-revision re-run, the earlier excess between launches is not reproduced (1.07×)C.7
Runs assigned to hosts by design96assigning hosts does not detectably change how closely runs agreeC.4

The ceiling and the replications (App. E)

ExperimentRunsResultApp.
First master-copy experiment, H = 30 and H = 100192the gap shrinks with the horizon (4.64×), with the H = 100 method at the ceilingE.5
Second master-copy experiment, both horizons to 540 updates128the gap shrinks (2.73×), with the H = 100 method at the ceiling (0.94)E.5
Master copy at a third horizon, H = 5532the method at the ceiling; no conclusion drawnE.5
H = 30 runs trained to the H = 100 budget, in both regimes64with the horizons compared alike, no interval excludes oneE.5
A second model family, Llama-3.1-8B64the method at the ceiling at both horizonsE.5
Mixed-horizon design (one policy on all horizons)32the gap grows with the horizon (0.56×) but shrinks on the logit scale, with the method at the ceiling at both horizonsE.4
Mixed-horizon design under the master copy32gap ratio 0.62×, interval contains oneE.4
Pilots for a configuration between floor and ceiling–none had both methods between 0.15 and 0.85 at both horizonsE.2

The public stacks: ALFWorld (App. D)

ExperimentRunsResultApp.
ALFWorld to twice the published budget (300 updates)32the published growth recurs (0.56× and 0.63×), with the method at the ceiling on one short task typeD.4
ALFWorld task types trained separately, two budget rules64the gap grows with the sub-goal count under the proportional budget (0.56×), with the long type's method at the ceilingD.3

The training loop and re-running a seed (App. A, B)

ExperimentRunsResultApp.
Keyed sampler96re-runs of one seed still first differ at update 2B.5
bfloat16 loop at the master copy's stable rate (5 × 10⁻⁷)96the bfloat16 loop does not learn at that rateA.4
bfloat16 with stochastic rounding, two learning rates192rounding reverses which rate learnsA.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.