acceptodds
Under review as a conference paper at ICLR 2027

Preserving Every Target Row Does Not Preserve Propagated Uncertainty in Recursive Fitted Evaluation

Abstract

Bootstrap procedures use ensembles of value functions to estimate uncertainty in a policy's value through recursive fitted evaluation. Fixing the target vectors leaves unspecified which model fits each vector, a choice implementations make implicitly. We show that reassigning those vectors among models can collapse ensemble spread by changing their dependence on the models' bootstrap resamples and fitting histories. We prove that reassigning target vectors can change the limiting distribution as ensemble size grows at fixed recursion depth and derive exact finite-ensemble moment recursions for affine fits. In Bootstrap-FQE experiments across six environments, reassignment at every update reduces spread on all 105 datasets and lowers observed coverage of nominal 90% intervals from 88–100% to 4–30% against the evaluation references. Indexing choices that determine which model fits each vector therefore belong to the ensemble's statistical specification.

Primary area:
reinforcement learning
Keywords:
Off-policy evaluation, uncertainty quantification, bootstrap inference, fitted Q evaluation, joint predictive distributions, offline reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GDqHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of uncertainty propagation of Bootstrap-FQE. The authors prove that reassigning target vectors can collapse ensemble spread, even when all target rows are fully preserved. They establish a weak convergence theory under mild conditions. They also empirically demonstrate that whole-row reassignment reduces spread on all 105 datasets in 6 environemnts, with converage of nominal 90% intervals drops from 88–100% to 4–30%.

Critical strengths
  • I like the intervention design. It is clean. The whole-row permutation exactly preserves the empirical distribution of complete rows and changes only the recipient. So they can separate assignment from the confound of altered target values. This distinguishes this work with prior works like MSG and REDQ.

  • The empirical effect is extremely consistent. In all the 6 environments and 105 datasets, the results all point in the same direction with a ten-fold sweep.

  • The authors desinged a validation chain to rule out alternative explanations. The validation chain includes: 1) theoretical precdiction, 2) exact tabular fitting without optimization which is further extended to fixed-feature ridge, 3) neural networks. It directly addresses the most natural objection: that the reduction in spread is an artifact of neural network optimization. The γ\gamma sweep in Table 2 is convincing.

  • I think the two counterintuitive findings have direct practical value. First, point estimates improve while the interval score deteriorates. Second, permuting only within a single update actually increases spread, whereas permuting across the full recursion reduces it. Both show that common evaluation shortcuts can lead to conclusions with the wrong sign.

Critical weaknesses
  • The major issue is that no prescription is offered. We can learn from this paper that assignment must be declared. However, we still don't know which one to choose, how large the spread should be, or how to calibrate. I think this limits the paper's impact, though it is acceptable for a diagnostic paper.

  • This paper treats the drop in converage as a failure. The wording in the abstract (e.g. collapse, lowers observed coverage) and the conclusion both suggest that reassignment is a bad outcome. However, the author did not explain why this is True.

  • The datasets in the main experiments are very small. Each dataset has only N=10 episodes, and Table 1 reports all results at this size. It becomes suspicious whether the theory is really practical or the experiments are really informative.

Questions for authors

See weakness part.

4 comments

Thank you for your review. We appreciate your positive assessment of the intervention design and the validation chain. Your main question is which assignment to choose, how large the spread should be, and how to calibrate. You also ask why reduced coverage is a failure and whether the results hold beyond datasets of 10 episodes. We address these with a new theorem and four new experiments and analyses. We ask that your final assessment reflect these additions.

W1: Which assignment to use

Under own-row assignment, each member fits its own target vector at every update. Whole-row reassignment keeps every target vector, changing only which member fits it, yet the spread falls below the own-row spread on all 105 datasets, as reported in the original submission. The two assignments therefore give two different spreads. A confidence interval needs the spread that matches the estimator's sampling variability.

A new theorem, Proposition A.13 (lines 1097–1166), proves which of the two spreads matches that variability in the one-state tabular model of Section 4.4. Under own-row assignment, the bootstrap distribution has the same limit as the sampling distribution of the estimator. The own-row spread therefore matches the estimator's sampling variability, so own-row intervals attain their nominal coverage asymptotically with no calibration. Whole-row reassignment leaves the point estimate unchanged but multiplies the limiting variance of the bootstrap distribution by:

ϱT=(1−γ)(1+γT)(1+γ)(1−γT),\varrho_T=\frac{(1-\gamma)(1+\gamma^T)}{(1+\gamma)(1-\gamma^T)},

which is below one for every discount factor 0<γ<10<\gamma<1 and depth T≥2T\ge2, so whole-row intervals cover less than their nominal level. At γ=0.95\gamma=0.95 and depth 200, the nominal 90% whole-row interval has limiting coverage 20.8%. The Introduction now states (lines 68–70):

In this setting, we prove that own-row bootstrap intervals are asymptotically valid, whereas whole-row intervals undercover because their spread omits the covariance across backups that the estimator's sampling distribution contains.

The theorem thus answers, in the one-state model, your three questions: which assignment to choose, how large the spread should be, and how to calibrate. Each member must fit its own target vector, the own-row spread then matches the estimator's sampling variability, and the own-row interval needs no calibration. The standard bootstrap assigns target vectors this way. The theorem proves that preserving every target vector cannot replace own-row assignment: whole-row reassignment alters no target vector at any update, yet the whole-row variance is too small by the factor ϱT\varrho_T. In the neural experiments, whole-row intervals at nominal 90% cover 12 of the 105 datasets where own-row intervals cover 99, as reported in the original submission. Preserving every target vector therefore does not certify the spread.

Nor does the value estimate. In a new experiment in the official NIMI ensemble-FQE implementation (Li et al., 2026), we kept every target vector, changing only which critic (ensemble member) fits it. On the full Walker2d-medium-v2 dataset, the spread, measured as the SD across critics, falls by a factor of 11.38 ±\pm 1.36 over five paired seeds while the value estimate moves by less than 0.3% (Section 4.3, lines 389–403; new Table 14, lines 1674–1683). Point accuracy and a single update do not certify the spread either. In the Bootstrap-FQE experiments on Hopper, whole-row reassignment lowers the RMSE of the ensemble mean from 8.769 to 6.809 while coverage falls from 10/10 to 3/10. One reassigned update increases the spread at 19 of 21 checkpoints, whereas reassignment at every update reduces it on all 105 datasets. Both results are reported in the original submission.

The prescription is therefore twofold. For bootstrap confidence intervals, each member must fit its own target vector. Every method must also state which member fits each target vector, because the target vectors, the value estimate, the point accuracy, and a single update do not certify the spread. The Discussion now states what depends on this assignment, namely any confidence interval, exploration bonus, or pessimistic penalty computed from ensemble spread (lines 475–485).

W2: Why reduced coverage is a failure

Proposition A.13 (W1) also proves why reduced coverage is a failure (lines 1097–1166). In the one-state model, own-row and whole-row assignment give identical point estimates, so the own-row and whole-row intervals differ only in width. The own-row interval is asymptotically valid, so the narrower whole-row interval excludes the true value more often than its nominal level allows. A new simulation over 1,000 datasets confirms the own-row and whole-row coverage that Proposition A.13 predicts at three discount factors (Appendix C.1, lines 1310–1316; new Table 3, lines 1318–1325). For your convenience, the results are below:

Discount factorOwn-row coverageWhole-row coverageWhole-row limit from Proposition A.13
0.5882/1000630/10000.658
0.8882/1000408/10000.417
0.95882/1000201/10000.208

Coverage of the true value by nominal 90% intervals. The two assignments give identical point estimates on every dataset.

For both assignments and every discount factor, the predicted limit lies within the 95% binomial interval of the observed coverage, with an own-row limit of 0.900.

The whole-row spread is smaller because it measures a different quantity. Each update, called a backup in the paper, adds an intercept, the mean reward in one member's bootstrap resample: own-row assignment adds the same member's intercept at every backup, whereas whole-row reassignment adds a different member's intercept at each backup. We added the following to Section 3.3 (lines 229–233):

The reduced spread is not a sharper estimate of the same uncertainty. A member that drew a new resample before every backup would likewise have an independent intercept at each backup, so its output would have the same distribution as a whole-row output in the limit (Corollary A.10, for members that retain only their resample). Whole-row spread thus measures how much estimates vary under a new resample at every backup, whereas own-row spread measures how much they vary under one resample throughout, as the bootstrap prescribes.

In the neural experiments, the two assignments give different point estimates, so a new analysis separates width from point estimate. We moved every whole-row interval so that it is centered at the own-row ensemble mean while keeping its width, so the moved interval differs from the own-row interval only in width. Coverage at nominal 90% still falls in every environment, from 99 of the 105 datasets for the own-row intervals to 38 for the moved intervals (Section 4.2, lines 366–387; new column of Table 1, lines 324–337). The whole-row intervals themselves are narrower, yet the interval score, which penalizes both width and failure to contain the reference policy value, is worse for whole-row intervals than for own-row intervals in every environment, as reported in the original submission.

Reduced coverage is therefore a failure of the whole-row interval rather than a gain in precision. The abstract and the Conclusion now state that, in the one-state model, own-row intervals are asymptotically valid whereas whole-row intervals undercover, and how far the narrower intervals alone lower coverage in the neural experiments (lines 18–25, 529–533).

W3: Dataset size

The main experiments use 10 episodes for the reason that Section 4.1 now states (lines 343–345):

We use small datasets because uncertainty estimates matter most when data are scarce, as with retrospective clinical data (Li et al., 2026).

To test whether whole-row reassignment still reduces spread and coverage when data are less scarce, we ran the own-row and whole-row comparison with ten times the data. In a new experiment, we extended each Hopper and Walker2d dataset from 10 to 100 episodes with all other settings unchanged (Appendix C.3, lines 1416–1425). For CartPole, the original submission already varied the dataset size from 10 to 100 episodes. For your convenience, the results with 100 episodes are below:

EnvironmentDatasets on which own-row SD exceeds whole-row SDOwn-row coverageWhole-row coverage
CartPole (original submission)25/2524/255/25
Hopper (new)10/1010/106/10
Walker2d (new)10/1010/104/10

Coverage of nominal 90% intervals with 100 episodes per dataset (new Table 6, lines 1404–1413; CartPole from Table 4 of the original submission, now Table 5).

In CartPole, the original submission shows a median reduction in spread of at least 4.35×4.35\times and whole-row coverage of at most 6/25 at every dataset size from 10 to 100 episodes. Section 4.1 now reports the results with 100 episodes in the main text (lines 352–356).

The reduction in spread also appears on a full dataset in a published method. The new experiment in the NIMI implementation (W1) trains on the full Walker2d-medium-v2 dataset and changes only the assignment (Section 4.3, lines 389–403). For your convenience, the results are below:

AssignmentValue estimateSD across criticsOwn-row SD / whole-row SD
Own-row116.35 ±\pm 0.185.428 ±\pm 0.1931
Whole-row116.67 ±\pm 0.080.496 ±\pm 0.04411.38 ±\pm 1.36

Means and standard errors over five paired training seeds at 25,000 updates (new Table 14, lines 1674–1683). The whole-row SD is below the own-row SD on all five seeds.

Unlimited data does not restore whole-row coverage either. Proposition A.13 takes the dataset size to infinity, so the whole-row coverage of 20.8% at γ=0.95\gamma=0.95 and depth 200 is the coverage that remains with unlimited data (W1).

The reduced spread and the reduced coverage are therefore not effects of small datasets.

Dear Reviewer GDqH,

We hope you have had a chance to read our response. It addresses all three weaknesses through an explicit prescription, experiments with larger datasets, and a proof that, in the one-state model, own-row intervals are asymptotically valid whereas reassigned intervals undercover.

Please let us know if any concerns remain so that we can address them before the discussion ends.

Reviewer_9FRKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at bootstrap ensembles for recursive fitted evaluation, mainly Bootstrap-FQE. Each member fits Bellman targets computed from its own Q-function on its own resample, and the spread across members is used as a confidence interval. The authors point out that implementations implicitly decide which member fits which target vector, and that this choice can change the uncertainty even when the set of target vectors stays the same. They compare three assignment rules: own-row (each member fits its own targets), whole-row (whole target vectors are randomly permuted among members at each update), and coordinatewise (each input's targets are permuted independently).

They show that each assignment rule converges to its own limit as the ensemble grows at fixed depth. In a scalar example the limits are: τ2(∑s<Tγs)2\tau^2\big(\sum_{s<T}\gamma^s\big)^2 for own-row vs. τ2∑s<Tγ2s\tau^2\sum_{s<T}\gamma^{2s} for whole-row. They also derive exact finite-ensemble moments for affine fits. In their experiments, whole-row reassignment lowers spread on every dataset and drops coverage, even when RMSE improves.

Overall, the paper studies this implicit choice and its effect on the reported uncertainty in detail, both theoretically and empirically.

Critical strengths

The paper identifies a relevant choice that is usually overlooked. Many ensemble methods use member spread for confidence intervals, exploration, or pessimism, and implementations decide which member fits which target vector implicitly. The paper separates this choice from the target values and shows that it alone can change coverage substantially (from 88–100% to 4–30%).

Critical weaknesses

1. Whole-row reassignment may be too specific to matter in practice.
It isolates assignment, but none of the implementations in D.2 use it. Methods that do change assignment, like shared-minimum targets or REDQ-style broadcasting, also change or duplicate the targets. The paper tests only one of these (shared minimum in NIMI ensemble FQE), with a single seed, and that run cannot separate the two effects. Also, since own-row is the bootstrap itself, we would expect whole-row to behave differently. So it is unclear how much assignment alone matters in methods people use. The dropout ablation (C.4) does test a choice used in practice, but only on CartPole.

2. Much of the formalism in the main text is not needed for the results.
The results the experiments test come from elementary calculations: the two-member example (3.2), the scalar variance formula (Eq. 3), and the affine moment recursion (3.4). With a common operator, the affine recursion already gives the scalar result exactly at any MM, so no limit is needed. Yet most of 3.3 builds general machinery (Polish spaces, kernels, retained-state dynamics) for Theorem 3.1, which only shows that the M→∞M \to \infty limit exists. It gives no prediction that the experiments use, and Remark A.4 says its bound is not useful at the depths studied. This space would be better used for intuition. I found the clearest explanation of the effect in Cor. A.8: without fitting history, whole-row is equivalent to drawing a fresh resample at every backup. This is currently only in the appendix and should be moved to the main text.

Questions for authors
  1. Methods used in practice, such as shared-minimum targets, change both which member fits each target and the target values themselves. Could you separate these two effects on the same datasets? This would show how much of the effect in practice comes from assignment alone.

  2. A single whole-row update usually increases spread (19 of 21 checkpoints), but applying it at every update reduces spread on all 105 neural datasets. Why? Is there a general condition, beyond the scalar and ridge cases, under which whole-row reduces spread? Is the one-step increase caused by warm-started networks suddenly fitting another member's targets?

4 comments

Thank you for your review. We appreciate your recognition that the paper identifies a relevant choice that is usually overlooked. Your review primarily concerns how much assignment alone matters in methods people use. You also ask why a single reassigned update increases spread when repeated reassignment reduces it. We address these with three new theoretical results and six new experiments. We ran the separation you asked for, extended the dropout comparison to three environments, and restructured Section 3.3 as you suggested. We ask that you update your score accordingly.

W1/Q1: How much assignment alone matters in practice

We ran the separation you asked for in Q1. We repeated the NIMI comparison over five paired seeds, where the original submission used one. We also added two target rules, so that one arm changes the assignment and nothing else. Whole-row reassignment keeps every target vector and changes only which critic fits it, whereas the shared mean gives every critic the same mean target. All arms use the official NIMI ensemble-FQE implementation (Li et al., 2026) with its published settings and 20 critics, on the full Walker2d-medium-v2 dataset of 999,999 transitions. Within each seed, they share initial parameters and minibatches. We have added a new subsection, Section 4.3 (lines 389–403), and a new appendix section, Appendix D.3 (lines 1690–1716), detailing these new experiments. For your convenience, the results are below:

Target ruleWhat changes from own-rowValue estimateSD across criticsOwn-row SD / rule SD
Own-row (NIMI default)Nothing116.35±0.18116.35\pm0.185.428±0.1935.428\pm0.19311
Whole-row (new)Which critic fits each vector116.67±0.08116.67\pm0.080.496±0.044\mathbf{0.496\pm0.044}11.38±1.36\mathbf{11.38\pm1.36}
Shared mean (new)Which critic fits each vector, and the target values116.22±0.08116.22\pm0.080.460±0.0250.460\pm0.02512.02±0.9712.02\pm0.97
Shared minimum (NIMI option)Both, plus a pessimistic penalty−161.27±2.97-161.27\pm2.971.037±0.0651.037\pm0.0655.30±0.355.30\pm0.35

Means ±\pm standard errors over five paired seeds after 25,000 updates (new Table 14, lines 1674–1683).

Changing only which critic fits each vector reduces the SD on every seed while moving the value estimate by less than 0.3%, so the value estimate gives no sign that the uncertainty has changed.

NIMI's shared-minimum option also subtracts a pessimistic penalty from every target, the gap between the mean and the minimum target across critics, which moves its value estimate to −161.27-161.27. A second new experiment applies that same penalty under every rule. Changing only the assignment then reduces the SD by a factor of 20.43 ±\pm 1.52, against 19.86 ±\pm 1.53 for sharing the mean target, each on all five seeds (new Table 15, lines 1728–1741). With the penalty, the shared mean is NIMI's shared-minimum option with the penalty taken from a recorded run, so this comparison sets a change of assignment alone against the target rule that the published code provides.

In both experiments, changing only the assignment reduces the SD as much as sharing the targets does, to within one standard error. A reduction as large as the one that a shared target produces therefore requires no change to any target value.

You note that own-row assignment is the bootstrap itself, so whole-row reassignment is expected to behave differently. That expectation fixes neither the direction nor the size of the difference. For affine fits with a shared transition matrix, the direction depends on the covariances between backups. Reassignment lowers the expected member variance when they sum to a positive value and raises it when they sum to a negative value (new Equation 2, lines 291–302; W3). In the one-state model of the exact tabular experiment, where the sum is positive, the revision proves the size of the difference and its consequence for the confidence interval. A new theorem, Proposition A.13 (lines 1097–1166), shows that the own-row interval is asymptotically valid, whereas the whole-row interval, around the same point estimate, has limiting coverage 20.8% at a nominal 90% (γ=0.95\gamma=0.95, depth 200).

These results show that, in a published method, which member fits each target vector changes the reported uncertainty by an order of magnitude while the value estimate stays within 0.3%. In the one-state model, the same change invalidates the confidence interval.

W2: The dropout comparison beyond CartPole

We extended the dropout comparison to three environments. A new experiment runs it on CartPole, Cliff Walking, and Hopper (Appendix C.4, lines 1440–1459). It also removes a second difference between the two arms of the original comparison, in which each arm computed its final readout with its own masks. Both arms now use the same masks at the final readout, so they differ only in whether masks were shared across inputs when targets were computed (locked and batched masking in the submission). For your convenience, the results are below:

EnvironmentDatasets with larger spread under shared masksMedian ratio of spread, shared masks to independent masks
CartPole25/253.82
Cliff Walking25/251.87
Hopper10/105.77

Spread is the SD of the members' value estimates. A ratio above 1 means that the ensemble trained with shared masks has the larger spread (new Table 7, lines 1448–1457).

Shared masks give the larger spread on all 60 datasets, so the effect of sharing masks is not specific to CartPole. Section 4.1 now reports this result (lines 361–363).

W3: Formalism and intuition in Section 3.3

We restructured Section 3.3 as you suggested (lines 205–250). The general formalism, including the retained-state dynamics, fitting kernels, and measure-theoretic assumptions, is now in Appendix A.2 (lines 678–809), which leaves an informal statement of Theorem 3.1 in Section 3.3. The section now opens with the scalar recursion, which shows that the own-row and whole-row variances differ by exactly the sum of the covariances between backups (lines 207–227). It then states the interpretation that you identified in Corollary A.8 (now A.10). We added the following to Section 3.3 (lines 229–233):

The reduced spread is not a sharper estimate of the same uncertainty. A member that drew a new resample before every backup would likewise have an independent intercept at each backup, so its output would have the same distribution as a whole-row output in the limit (Corollary A.10, for members that retain only their resample). Whole-row spread thus measures how much estimates vary under a new resample at every backup, whereas own-row spread measures how much they vary under one resample throughout, as the bootstrap prescribes.

Following your observation that a common operator gives the result exactly at any ensemble size, we added a new theoretical result, Proposition A.9 (lines 937–976), stated in Section 3.4 as Equation 2 of the revised paper (lines 291–302). For any shared transition matrix, it gives the change in expected member variance under repeated reassignment in closed form, at every ensemble size and depth.

We kept an informal statement of Theorem 3.1 because the exact recursions require affine fits, and the theorem is the result that covers fits that retain their fitting history, including neural networks. The theorem shows that the gap between own-row and whole-row spread does not vanish as the ensemble grows when the two limiting variances differ (lines 247–250). The ensemble-size sweep of the original submission tests this consequence. The gap persists from 10 to 100 members, with a median reduction in spread of at least 6.08×6.08\times at every size (Table 4 of the original submission, now Table 5).

Q2: One reassigned update versus reassignment at every update

One reassigned update and reassignment at every update are different comparisons. Section 4.5 now explains the difference with a new identity and three new experiments (lines 463–469; Appendix C.5, lines 1462–1537).

For affine fits with a shared transition matrix, one reassigned update of an own-row ensemble removes only the covariances between the newest backup and each earlier backup, whereas reassignment at every update removes the covariances between all pairs of backups (new Equation 13, lines 963–971). One update therefore increases expected variance exactly when a member's reward fit and its continuation value are negatively correlated across members. Repeated reassignment reduces expected variance exactly when the covariances between backups sum to a positive value (new Equation 2, lines 291–302; W3). Equation 2 holds for any shared transition matrix, so it extends the condition beyond the scalar case. Outside the shared-matrix case, the experiments below show whether reassignment raises or lowers the spread, exactly for ridge FQE with member-specific matrices and by measurement for neural networks.

We tested the warm-start explanation that you propose. In a new experiment, we repeated the single-update comparison at the 21 checkpoints of the original submission, either retaining each member's Adam state or resetting it while keeping the learned weights:

EnvironmentRetained AdamReset Adam, learned weights kept
CartPole7/71/7
Cliff Walking6/70/7
Sepsis6/70/7
All checkpoints19/211/21

Checkpoints at which one reassigned update increases the median spread, with the default eight fitting epochs (new Table 8, lines 1464–1487).

At the eight fitting epochs per backup used in the main experiments, resetting only Adam while keeping the warm-started weights reverses 18 of the 19 increases and shrinks the remaining one by 98%. Those 18 increases therefore require the retained optimizer state.

An increase after one update also occurs in ridge FQE, which involves no optimizer. In a new experiment with member-specific matrices, for which the expected variance can be computed exactly, one reassigned update increases expected variance on 10 of the 25 CartPole datasets, whereas 400 reassigned updates leave it below the own-row variance on all 25 (new Table 9, lines 1503–1511). A further new experiment continues the neural CartPole ensembles from depth 100. Switching to whole-row reassignment increases spread after one update on 20 of the 25 datasets, whereas keeping the switch for 400 updates reduces it on 24 of the 25. Switching whole-row ensembles back to own-row assignment for 400 updates increases spread on all 25 (new Table 10, lines 1526–1537).

A single update therefore does not predict the effect of reassignment at every update, which leaves expected variance below its own-row value on all 25 ridge datasets after 400 updates and reduces spread on all 105 neural datasets of the original submission.

Dear Reviewer 9FRK,

We hope you have had a chance to read our response. It addresses each point in your review through the five-seed assignment-only NIMI experiment, dropout experiments in three environments, and a restructured Section 3.3. It also provides a variance-reduction condition for affine fits with a shared transition matrix, along with theory and experiments, including an optimizer reset, explaining the single-update increases.

Please let us know if any concerns remain so that we can address them before the discussion ends.

Reviewer_x1TuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the assignment of regression targets to bootstrap ensemble members affects propagated uncertainty in recursive fitted policy evaluation. The authors separate two dependences that the collection of target vectors leaves unspecified: coupling among targets within a vector, and the association between each target vector and the member that fits it. Standard Bootstrap-FQE keeps each member's resample, fitting history, and target vector together (own-row assignment). The authors test what is lost when this association is broken while every target vector is preserved, through random permutation of complete target vectors among members at each update (whole-row reassignment).

The main contributions are: (1) a formal distinction between coupling and assignment in recursive fitting; (2) a large-ensemble convergence theorem under permutation-induced dependence, together with a scalar example showing that own-row and whole-row assignment can converge to different limiting laws at fixed depth; and (3) exact finite-ensemble moment recursions for affine fits.

Validation has three parts. In an exact tabular setting, whole-row reassignment reduces spread by the predicted factor √((1+γ)/(1−γ)), with identical point estimates. For fixed-feature ridge FQE, the affine recursions match sampled moments closely. In neural Bootstrap-FQE on six environments and 105 datasets, whole-row reassignment reduces spread on every dataset by median factors of 2.3 to 6.1, and coverage of nominal 90% intervals falls from 88–100% to 4–30%. Further experiments vary ensemble and dataset size, compare single-update with repeated reassignment, test coupling through dropout masks, and survey how published implementations construct targets.

Critical strengths
  1. Clean isolation of a specification choice. Whole-row reassignment preserves every target vector, so any change in spread is attributable to assignment alone. The exact tabular experiment removes optimization as an explanation: the measured spread ratios match the predicted values to within about 1% at every discount, and point estimates are identical. The affine validation in Table 3 likewise agrees with Proposition A.6 to within sampling error. This weighs heavily in my assessment because it establishes the effect as a property of the recursion, not an artifact of neural training.
  2. Useful theory and a concrete link to practice. Theorem 3.1 gives large-ensemble limits for ensembles whose members interact through permuted targets, without continuity assumptions on the update or readout maps. The exact finite-ensemble recursions are a practical tool for affine fits. The audit of published implementations in Appendix D.2 shows that related target constructions are common: shared-minimum broadcasts, single-critic broadcasts in REDQ-style targets, and independent per-input dropout in UWAC. The UWAC choice corresponds to the batched masking of Appendix C.4, which reduced spread by a median factor of 5.89. This matters for my assessment because it shows that the coupling result, at least, applies to a choice that practitioners actually make.
Critical weaknesses
  1. The main assignment result is closer to quantification than to a new principle. The paper itself notes (Remark A.5) that standard Bootstrap-FQE implements own-row assignment, so whole-row reassignment departs from the bootstrap construction by design. The mechanism in the tabular case is intuitive, i.e., own-row assignment accumulates each member's resampling error coherently across backups, whereas whole-row assignment averages it across members. Prior work, notably Ghasemipour et al. (2022), has already argued that the independence structure of ensemble targets matters for uncertainty. The paper's distinction is that MSG changes target values together with their assignment, which is valid but narrow. The theory shows that assignment can change the limiting law, through a scalar example, but gives no general condition for the direction or size of the effect. Corollary A.8 states that no variance ordering holds in general, and Section 4.3 shows that a single update can increase spread. Finally, none of the published implementations in Appendix D.2 changes assignment alone, and the one direct test on a published method (NIMI, Table 6) uses a single seed and also changes target values. This is consequential because the paper's significance rests on practical relevance, and for the assignment result, as distinct from coupling, that relevance remains indirect.
  2. The coverage results conflate reduced spread with shifts in the point estimate. The headline claim is that assignment alone reduces spread and coverage. In the neural experiments, however, whole-row reassignment also moves the ensemble mean substantially in some environments: in CartPole from 66.88 to 48.15 against a reference of 67.42, and in Walker2d from 185.74 to 222.39 against a reference of 183.42 (Table 5). Once the interval center moves away from the reference, coverage falls regardless of width. Only the exact tabular setting holds point estimates fixed. The authors acknowledge in the Limitations that own-row intervals are not shown to be calibrated, that four of the six references are truncated, and that the intervals omit initial-state variation. The coupling experiment points the same way: even locked masks, the construction that preserves coupling, give only 8/25 coverage (Appendix C.4). This is consequential because the coverage collapse is the paper's main evidence of practical harm, and in its current form that evidence does not separate the spread effect from the mean-shift effect.
Questions for authors
  1. Can the authors decompose the coverage loss in the neural experiments into the contribution of narrower intervals and the contribution of the shifted ensemble mean? For example, they could report coverage of whole-row-width intervals centered at the own-row mean, or the bias of each ensemble mean relative to the reference. If most of the loss comes from the mean shift, the conclusion that assignment collapses uncertainty would need qualification.
  2. Can the authors show the assignment effect, separately from changes in target values, in a construction that practitioners use? For example, they could compare a REDQ-style single-critic broadcast with a variant that preserves the collection of target vectors, or repeat the NIMI comparison over several seeds. This would determine whether the assignment result has practical consequences beyond the deliberate intervention.
  3. Proposition A.7 attributes the own-row excess to positive covariance across backup contributions. Can the authors state a sufficient condition, even for affine fits, under which repeated whole-row reassignment reduces spread? This would turn the existence result into a prediction.
6 comments

Thank you for your thoughtful and careful review. We appreciate your positive assessment of the intervention design and the supporting theory. Your main concerns focus on (1) whether the assignment result is a new principle or a quantification and (2) whether the coverage results separate reduced spread from shifts in the point estimate. We address both with three new theoretical results and ten new experiments and analyses, including the analyses you proposed in Q1, the multi-seed NIMI comparison you proposed in Q2, and the condition you asked for in Q3. We ask that you revisit your soundness, presentation, and overall assessment in light of these additions.

W1: Novelty of the assignment result

Prior work shows that sharing targets changes the uncertainty, but sharing changes two things at once. Ghasemipour et al. (2022) compare independent targets with shared targets, which differ from independent targets both in their values and in which member fits them. That comparison therefore cannot show whether an ensemble that keeps every target vector keeps its uncertainty. Whole-row reassignment departs from the bootstrap in exactly one respect, which member fits each target vector, so it measures what that comparison cannot.

Our new experiment makes that measurement in the official NIMI implementation (Section 4.3, lines 389–403; W3). We kept every target vector and changed only which critic fits it. For comparison, we also gave every critic the same mean target, which changes the target values as well as their assignment (new Table 14, lines 1674–1683):

  • Keeping every target vector and changing only the assignment reduces the SD across critics by a factor of 11.38 ±\pm 1.36
  • Sharing the mean target reduces it by 12.02 ±\pm 0.97

Changing the assignment alone is therefore enough to reduce the SD by an order of magnitude, with no change to any target value. The reduction is as large as the one that sharing produces, to within one standard error. The same holds when the pessimistic penalty of NIMI's shared-minimum option is subtracted from the targets under every rule, where the two factors are 20.43 ±\pm 1.52 and 19.86 ±\pm 1.53 (new Table 15, lines 1728–1741; W3).

The revision also proves, in the one-state model of the exact tabular experiment, that the reduced spread makes the confidence interval invalid. A new theorem, Proposition A.13 (lines 1097–1166), shows that the own-row interval is asymptotically valid, whereas the whole-row interval, around the same point estimate, has limiting coverage 20.8% at a nominal 90% (γ=0.95\gamma=0.95, depth 200). The whole-row interval undercovers because its spread measures a different quantity. For members that retain only their resample, a whole-row member has, in the limit, the same output distribution as a member that draws a new resample before every backup (Corollary A.10, numbered A.8 in the submission). We added the following to Section 3.3 (lines 229–233):

Whole-row spread thus measures how much estimates vary under a new resample at every backup, whereas own-row spread measures how much they vary under one resample throughout, as the bootstrap prescribes.

The direction of the effect does not follow from the intuition of the tabular case. For affine fits with a shared transition matrix, reassignment removes the covariance between backups, which lowers the expected member variance when that covariance is positive, as in the tabular case, and raises it when it is negative (new Equation 2, lines 291–302; W2).

The assignment result is therefore a new principle, and the distinction between target values and their assignment is not narrow. Keeping every target vector does not keep the uncertainty, which prior comparisons could not show, and in NIMI the assignment alone reduces the SD as much as sharing the targets does. The revision proves that the reduced spread invalidates the confidence interval in the one-state model. It also gives the direction and size of the change in expected member variance exactly for affine fits with a shared transition matrix.

W2/Q3: A condition for the direction and size of the effect

We proved the condition you asked for. For affine fits with a shared transition matrix AA, intercepts cbc_b, and readout aa, the new Proposition A.9 (lines 937–976) gives the expected change in member variance under repeated reassignment exactly, at every ensemble size M≥2M\ge2 and depth TT. We added the result to Section 3.4 as Equation 2 of the revised paper (lines 291–302). With Cc=M−1∑b(cb−cˉ)(cb−cˉ)⊤C_c=M^{-1}\sum_b(c_b-\bar c)(c_b-\bar c)^\top and wk=(Ak)⊤aw_k=(A^k)^\top a,

Var⁡M(a⊤θb,Town)−EσVar⁡M(a⊤θb,Twhole)=2MM−1∑0≤j<k≤Twj⊤Ccwk.(2)\operatorname{Var}_M(a^\top\theta_{b,T}^{\mathrm{own}})-\mathbb E_\sigma\operatorname{Var}_M(a^\top\theta_{b,T}^{\mathrm{whole}})=\frac{2M}{M-1}\sum_{0\le j \lt k\le T}w_j^\top C_cw_k. \qquad (2)

Here θb,T\theta_{b,T} is the parameter vector of member bb after TT updates from its reward fit θb,0=cb\theta_{b,0}=c_b, Var⁡M\operatorname{Var}_M is the sample variance across the MM members, and Eσ\mathbb E_\sigma averages over the reassignment permutations. Each term of the sum is the covariance between the contributions of two different backups. Repeated reassignment reduces the expected member variance exactly when the sum is positive and increases it exactly when the sum is negative, so the condition is necessary as well as sufficient. The formula also gives the size of the change. The sum is nonnegative whenever AA, aa, and CcC_c are entrywise nonnegative, which includes the one-state tabular model.

Beyond a shared transition matrix, exact computation and experiment establish the direction. In a new experiment on ridge FQE with member-specific matrices, for which the expected variance can be computed exactly, 400 reassigned updates leave that variance below the own-row variance on all 25 CartPole datasets (new Table 9, lines 1503–1511). For neural networks, reassignment at every update reduces spread on all 105 datasets, as reported in the original submission.

A single reassigned update can increase the spread even where reassignment at every update reduces it. The new Equation 13 gives the exact condition for affine fits with a shared transition matrix (lines 963–971). One reassigned update increases expected variance exactly when a member's reward fit and its continuation value are negatively correlated across members. With member-specific matrices, the same ridge experiment shows an increase after one update on 10 of the 25 datasets (new Table 9). In the neural ensembles, the original submission reported an increase at 19 of 21 checkpoints. A new experiment shows that resetting Adam while keeping the learned weights reverses 18 of these 19 increases and shrinks the remaining one by 98% (new Table 8, lines 1464–1487).

Equation 2 therefore turns the existence result into a prediction for affine fits with a shared transition matrix. With member-specific matrices, the ridge experiment shows that repeated reassignment reduces expected variance on all 25 CartPole datasets.

W3/Q2: Assignment alone in a published method

We ran the experiment you proposed. We repeated the NIMI comparison over five paired seeds. We also added two target rules, so that one arm changes the assignment and nothing else. All arms use the official NIMI ensemble-FQE implementation (Li et al., 2026) with its published settings and 20 critics, on the full Walker2d-medium-v2 dataset of 999,999 transitions. Within each seed, they share initial parameters and minibatches. We have added a new subsection, Section 4.3 (lines 389–403), and a new appendix section, Appendix D.3 (lines 1690–1716), detailing these new experiments. For your convenience, the results are below:

Target ruleValue estimateSD across criticsOwn-row SD / rule SD
Own-row (NIMI default)116.35±0.18116.35\pm0.185.428±0.1935.428\pm0.19311
Whole-row (assignment only, new)116.67±0.08116.67\pm0.080.496±0.044\mathbf{0.496\pm0.044}11.38±1.36\mathbf{11.38\pm1.36}
Shared mean (new)116.22±0.08116.22\pm0.080.460±0.0250.460\pm0.02512.02±0.9712.02\pm0.97
Shared minimum (NIMI option)−161.27±2.97-161.27\pm2.971.037±0.0651.037\pm0.0655.30±0.355.30\pm0.35

Means ±\pm standard errors over five paired seeds after 25,000 updates (new Table 14, lines 1674–1683).

Changing only the assignment lowers the SD on every seed while moving the value estimate by less than 0.3%, so the value estimate gives no sign that the uncertainty has changed.

NIMI's own option, the shared minimum, also subtracts a pessimistic penalty from every target, the gap between the mean and the minimum target across critics, which moves its value estimate to −161.27-161.27. A second new experiment applies that same penalty under every rule. Changing only the assignment then reduces the SD by a factor of 20.43 ±\pm 1.52, against 19.86 ±\pm 1.53 for the shared mean, each on all five seeds (new Table 15, lines 1728–1741). With the penalty, the shared mean is NIMI's shared-minimum option with the penalty taken from a recorded run, so this comparison sets a change of assignment alone against the target rule that the published code provides.

These experiments show that, in a published method, which member fits each target vector changes the reported uncertainty by an order of magnitude. With or without NIMI's pessimistic penalty, changing only the assignment reduces the SD as much as sharing the targets does, to within one standard error.

Comments on the paper

2 comments

General response

We thank all reviewers for their careful and constructive feedback. We are encouraged that they recognized the merits of the work. Reviewers described the intervention design as "clean" [GDqH] and a "clean isolation of a specification choice" [x1Tu], and noted that the paper "identifies a relevant choice that is usually overlooked" [9FRK]. They found the empirical effect "extremely consistent" [GDqH] and credited the paper with "useful theory and a concrete link to practice" [x1Tu]. They also noted that the validation chain "directly addresses the most natural objection" [GDqH] and establishes the effect "as a property of the recursion, not an artifact of neural training" [x1Tu], and that the counterintuitive findings "have direct practical value" [GDqH].

The reviewers' main points are whether reduced coverage is a failure caused by assignment alone [GDqH, x1Tu], whether assignment alone matters in a published method [9FRK, x1Tu], and when reassignment reduces spread [x1Tu, 9FRK]. Based on the reviewers' feedback, we proved three new theoretical results and ran eleven new experiments and analyses to:

  • Establish which assignment gives valid intervals by proving, in a tabular model, that own-row intervals are asymptotically valid whereas whole-row intervals undercover around the same point estimate, with limiting coverage of 20.8% at nominal 90% [GDqH, x1Tu].
  • Separate narrower intervals from the shifted ensemble mean by recentering whole-row intervals at the own-row mean, where coverage still falls from 99 to 38 of the 105 datasets [x1Tu, GDqH].
  • Isolate assignment in a published method by changing only which critic fits each target vector, which reduces the SD across critics by a factor of 11.38, as much as replacing the target vectors with their mean, while the value estimate moves by less than 0.3% [9FRK, x1Tu].
  • Give a condition for when reassignment reduces spread, and by how much, in closed form for affine fits with a shared transition matrix [x1Tu, 9FRK].
  • Explain why a single reassigned update can increase spread with a new identity and an optimizer-reset experiment that reverses 18 of the 19 increases [9FRK].
  • Test the results beyond the original settings with ten times the data, more environments, and fivefold longer reference rollouts [GDqH, 9FRK, x1Tu].

We also restructured Section 3.3 around a scalar recursion that explains why reassignment reduces spread, moving the general formalism to the appendix and the fresh-resampling explanation into the main text, as Reviewer 9FRK suggested [9FRK, x1Tu].

Notes to reviewers

We use W# for weaknesses and Q# for questions in our individual responses. All line, section, table, and result numbers refer to the revised manuscript uploaded with the rebuttal, in which changes are shown in blue. Where the revision renumbers a result or renames a term, the individual responses give both (for example, Corollary A.8 of the submission is now Corollary A.10).

We thank the reviewers again for their comments, which have strengthened both the results and the presentation of the paper. If any concerns remain, we would be glad to address them.

Dear Area Chair and Reviewers,

We thank the Area Chair for their service and the reviewers for their constructive feedback. For convenience, the table below maps each concern to our action and where it appears in the revision.

ConcernReviewer(s)Our Action
Does assignment alone matter in a published method?9FRK, x1TuNew experiments: In the official NIMI code, changing only which critic fits each target vector cuts the SD across critics on all five seeds, by a mean factor of 11.38×\mathbf{11.38\times}, while the value estimate moves by less than 0.3%. Giving every critic the same mean target, which also changes the target values, cuts it by the same factor to within one standard error (12.02×12.02\times). With NIMI's pessimistic penalty held identical across assignment rules, assignment alone cuts the SD by 20.43×\mathbf{20.43\times}, against 19.86×19.86\times for the shared target (Sec. 4.3; App. D.3).
Does coverage fall because intervals narrow or because the mean shifts?x1Tu, GDqHNew analysis: With the mean held fixed, coverage falls in all six environments, from 99/105 to 38/105. Shifting the mean alone, at own-row width, lowers it only to 82/105 (Sec. 4.2).
How does the assignment result differ from prior comparisons?x1TuNew experiments and theory: Prior comparisons change target values together with their assignment. Ours keeps every target vector and the reported uncertainty still falls, by an order of magnitude in a published method (Sec. 4.3) and provably in the one-state model (Sec. 4.4).
Why is reduced coverage a failure, and which assignment should practitioners use?GDqH, x1TuNew theory: Prop. A.13 proves that, in the one-state model, own-row intervals are asymptotically valid and whole-row intervals cover 20.8% at nominal 90%, around the identical point estimate. Simulation confirms it: 882/1000 vs. 201/1000 (Sec. 4.4). For bootstrap intervals, each member must therefore fit its own target vector (Sec. 3.3; Prop. A.13), and every method must state its assignment (Sec. 6).
When does reassignment reduce spread, and why does a single reassigned update increase it?x1Tu, 9FRKNew theory and experiments: For affine fits with a shared transition matrix, Prop. A.9 gives the exact change in variance: reassignment reduces it exactly when the covariances between backups sum to a positive value (Sec. 3.4). One update removes only the covariances with the newest backup, so it does not predict the effect of reassigning at every update (Eq. 13). In the neural ensembles, resetting Adam reverses 18 of the 19 single-update increases (Sec. 4.5).
Are the results confined to 10 episodes, truncated references, or CartPole?GDqH, x1Tu, 9FRKNew experiments: With 100 episodes, whole-row reassignment reduces spread on all 45 datasets and lowers coverage in every environment, from 44/45 to 15/45 overall (Sec. 4.1; App. C.3). Fivefold longer MuJoCo rollouts leave every coverage count unchanged, and Cliff Walking, with an exact reference and a fixed initial state, falls 24/25 →\to 3/25, and to 7/25 with the mean held fixed (App. B.3). Shared dropout masks give larger spread on all 60 datasets in three environments (App. C.4).
Can the main-text theory be made more accessible?9FRK, x1TuRevision: Each result is now stated plainly before it is formalized. Sec. 3.3 is rewritten around a scalar recursion, with Theorem 3.1 stated informally, the fresh-resampling explanation in the main text, and the general formalism in App. A.2.

These additions are in the revised manuscript, with changes shown in blue. The central claim and every result of the original submission are unchanged. Each addition, made in response to a reviewer's comment, supports that claim.

We remain available through the end of the discussion period and would be glad to clarify anything further. We would be grateful if reviewers could indicate whether the revision addresses their concerns and update their ratings accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.