- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of uncertainty propagation of Bootstrap-FQE. The authors prove that reassigning target vectors can collapse ensemble spread, even when all target rows are fully preserved. They establish a weak convergence theory under mild conditions. They also empirically demonstrate that whole-row reassignment reduces spread on all 105 datasets in 6 environemnts, with converage of nominal 90% intervals drops from 88–100% to 4–30%.
Critical strengths
-
I like the intervention design. It is clean. The whole-row permutation exactly preserves the empirical distribution of complete rows and changes only the recipient. So they can separate assignment from the confound of altered target values. This distinguishes this work with prior works like MSG and REDQ.
-
The empirical effect is extremely consistent. In all the 6 environments and 105 datasets, the results all point in the same direction with a ten-fold sweep.
-
The authors desinged a validation chain to rule out alternative explanations. The validation chain includes: 1) theoretical precdiction, 2) exact tabular fitting without optimization which is further extended to fixed-feature ridge, 3) neural networks. It directly addresses the most natural objection: that the reduction in spread is an artifact of neural network optimization. The sweep in Table 2 is convincing.
-
I think the two counterintuitive findings have direct practical value. First, point estimates improve while the interval score deteriorates. Second, permuting only within a single update actually increases spread, whereas permuting across the full recursion reduces it. Both show that common evaluation shortcuts can lead to conclusions with the wrong sign.
Critical weaknesses
-
The major issue is that no prescription is offered. We can learn from this paper that assignment must be declared. However, we still don't know which one to choose, how large the spread should be, or how to calibrate. I think this limits the paper's impact, though it is acceptable for a diagnostic paper.
-
This paper treats the drop in converage as a failure. The wording in the abstract (e.g. collapse, lowers observed coverage) and the conclusion both suggest that reassignment is a bad outcome. However, the author did not explain why this is True.
-
The datasets in the main experiments are very small. Each dataset has only N=10 episodes, and Table 1 reports all results at this size. It becomes suspicious whether the theory is really practical or the experiments are really informative.
Questions for authors
See weakness part.
4 comments
W2: Why reduced coverage is a failure
Proposition A.13 (W1) also proves why reduced coverage is a failure (lines 1097–1166). In the one-state model, own-row and whole-row assignment give identical point estimates, so the own-row and whole-row intervals differ only in width. The own-row interval is asymptotically valid, so the narrower whole-row interval excludes the true value more often than its nominal level allows. A new simulation over 1,000 datasets confirms the own-row and whole-row coverage that Proposition A.13 predicts at three discount factors (Appendix C.1, lines 1310–1316; new Table 3, lines 1318–1325). For your convenience, the results are below:
| Discount factor | Own-row coverage | Whole-row coverage | Whole-row limit from Proposition A.13 |
|---|---|---|---|
| 0.5 | 882/1000 | 630/1000 | 0.658 |
| 0.8 | 882/1000 | 408/1000 | 0.417 |
| 0.95 | 882/1000 | 201/1000 | 0.208 |
Coverage of the true value by nominal 90% intervals. The two assignments give identical point estimates on every dataset.
For both assignments and every discount factor, the predicted limit lies within the 95% binomial interval of the observed coverage, with an own-row limit of 0.900.
The whole-row spread is smaller because it measures a different quantity. Each update, called a backup in the paper, adds an intercept, the mean reward in one member's bootstrap resample: own-row assignment adds the same member's intercept at every backup, whereas whole-row reassignment adds a different member's intercept at each backup. We added the following to Section 3.3 (lines 229–233):
The reduced spread is not a sharper estimate of the same uncertainty. A member that drew a new resample before every backup would likewise have an independent intercept at each backup, so its output would have the same distribution as a whole-row output in the limit (Corollary A.10, for members that retain only their resample). Whole-row spread thus measures how much estimates vary under a new resample at every backup, whereas own-row spread measures how much they vary under one resample throughout, as the bootstrap prescribes.
In the neural experiments, the two assignments give different point estimates, so a new analysis separates width from point estimate. We moved every whole-row interval so that it is centered at the own-row ensemble mean while keeping its width, so the moved interval differs from the own-row interval only in width. Coverage at nominal 90% still falls in every environment, from 99 of the 105 datasets for the own-row intervals to 38 for the moved intervals (Section 4.2, lines 366–387; new column of Table 1, lines 324–337). The whole-row intervals themselves are narrower, yet the interval score, which penalizes both width and failure to contain the reference policy value, is worse for whole-row intervals than for own-row intervals in every environment, as reported in the original submission.
Reduced coverage is therefore a failure of the whole-row interval rather than a gain in precision. The abstract and the Conclusion now state that, in the one-state model, own-row intervals are asymptotically valid whereas whole-row intervals undercover, and how far the narrower intervals alone lower coverage in the neural experiments (lines 18–25, 529–533).
W3: Dataset size
The main experiments use 10 episodes for the reason that Section 4.1 now states (lines 343–345):
We use small datasets because uncertainty estimates matter most when data are scarce, as with retrospective clinical data (Li et al., 2026).
To test whether whole-row reassignment still reduces spread and coverage when data are less scarce, we ran the own-row and whole-row comparison with ten times the data. In a new experiment, we extended each Hopper and Walker2d dataset from 10 to 100 episodes with all other settings unchanged (Appendix C.3, lines 1416–1425). For CartPole, the original submission already varied the dataset size from 10 to 100 episodes. For your convenience, the results with 100 episodes are below:
| Environment | Datasets on which own-row SD exceeds whole-row SD | Own-row coverage | Whole-row coverage |
|---|---|---|---|
| CartPole (original submission) | 25/25 | 24/25 | 5/25 |
| Hopper (new) | 10/10 | 10/10 | 6/10 |
| Walker2d (new) | 10/10 | 10/10 | 4/10 |
Coverage of nominal 90% intervals with 100 episodes per dataset (new Table 6, lines 1404–1413; CartPole from Table 4 of the original submission, now Table 5).
In CartPole, the original submission shows a median reduction in spread of at least and whole-row coverage of at most 6/25 at every dataset size from 10 to 100 episodes. Section 4.1 now reports the results with 100 episodes in the main text (lines 352–356).
The reduction in spread also appears on a full dataset in a published method. The new experiment in the NIMI implementation (W1) trains on the full Walker2d-medium-v2 dataset and changes only the assignment (Section 4.3, lines 389–403). For your convenience, the results are below:
| Assignment | Value estimate | SD across critics | Own-row SD / whole-row SD |
|---|---|---|---|
| Own-row | 116.35 0.18 | 5.428 0.193 | 1 |
| Whole-row | 116.67 0.08 | 0.496 0.044 | 11.38 1.36 |
Means and standard errors over five paired training seeds at 25,000 updates (new Table 14, lines 1674–1683). The whole-row SD is below the own-row SD on all five seeds.
Unlimited data does not restore whole-row coverage either. Proposition A.13 takes the dataset size to infinity, so the whole-row coverage of 20.8% at and depth 200 is the coverage that remains with unlimited data (W1).
The reduced spread and the reduced coverage are therefore not effects of small datasets.
Dear Reviewer GDqH,
We hope you have had a chance to read our response. It addresses all three weaknesses through an explicit prescription, experiments with larger datasets, and a proof that, in the one-state model, own-row intervals are asymptotically valid whereas reassigned intervals undercover.
Please let us know if any concerns remain so that we can address them before the discussion ends.
Thank you for your review. We appreciate your positive assessment of the intervention design and the validation chain. Your main question is which assignment to choose, how large the spread should be, and how to calibrate. You also ask why reduced coverage is a failure and whether the results hold beyond datasets of 10 episodes. We address these with a new theorem and four new experiments and analyses. We ask that your final assessment reflect these additions.
W1: Which assignment to use
Under own-row assignment, each member fits its own target vector at every update. Whole-row reassignment keeps every target vector, changing only which member fits it, yet the spread falls below the own-row spread on all 105 datasets, as reported in the original submission. The two assignments therefore give two different spreads. A confidence interval needs the spread that matches the estimator's sampling variability.
A new theorem, Proposition A.13 (lines 1097–1166), proves which of the two spreads matches that variability in the one-state tabular model of Section 4.4. Under own-row assignment, the bootstrap distribution has the same limit as the sampling distribution of the estimator. The own-row spread therefore matches the estimator's sampling variability, so own-row intervals attain their nominal coverage asymptotically with no calibration. Whole-row reassignment leaves the point estimate unchanged but multiplies the limiting variance of the bootstrap distribution by:
ϱT=(1+γ)(1−γT)(1−γ)(1+γT),
which is below one for every discount factor 0<γ<1 and depth T≥2, so whole-row intervals cover less than their nominal level. At γ=0.95 and depth 200, the nominal 90% whole-row interval has limiting coverage 20.8%. The Introduction now states (lines 68–70):
The theorem thus answers, in the one-state model, your three questions: which assignment to choose, how large the spread should be, and how to calibrate. Each member must fit its own target vector, the own-row spread then matches the estimator's sampling variability, and the own-row interval needs no calibration. The standard bootstrap assigns target vectors this way. The theorem proves that preserving every target vector cannot replace own-row assignment: whole-row reassignment alters no target vector at any update, yet the whole-row variance is too small by the factor ϱT. In the neural experiments, whole-row intervals at nominal 90% cover 12 of the 105 datasets where own-row intervals cover 99, as reported in the original submission. Preserving every target vector therefore does not certify the spread.
Nor does the value estimate. In a new experiment in the official NIMI ensemble-FQE implementation (Li et al., 2026), we kept every target vector, changing only which critic (ensemble member) fits it. On the full Walker2d-medium-v2 dataset, the spread, measured as the SD across critics, falls by a factor of 11.38 ± 1.36 over five paired seeds while the value estimate moves by less than 0.3% (Section 4.3, lines 389–403; new Table 14, lines 1674–1683). Point accuracy and a single update do not certify the spread either. In the Bootstrap-FQE experiments on Hopper, whole-row reassignment lowers the RMSE of the ensemble mean from 8.769 to 6.809 while coverage falls from 10/10 to 3/10. One reassigned update increases the spread at 19 of 21 checkpoints, whereas reassignment at every update reduces it on all 105 datasets. Both results are reported in the original submission.
The prescription is therefore twofold. For bootstrap confidence intervals, each member must fit its own target vector. Every method must also state which member fits each target vector, because the target vectors, the value estimate, the point accuracy, and a single update do not certify the spread. The Discussion now states what depends on this assignment, namely any confidence interval, exploration bonus, or pessimistic penalty computed from ensemble spread (lines 475–485).