acceptodds
Under review as a conference paper at ICLR 2027

CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

12 views

Abstract

Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10% budget, CODS retains 96.6% of eligible-pool performance across 20 valid D4RL task–algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4% of pooled ALFWorld success and 96.5% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, data selection, Bellman residual, dataset reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SqbUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CODS, an iterative data selection method for offline reinforcement learning. CODS starts from a small randomly selected subset and progressively acquires transitions with high Bellman residuals, updating the critic after each selection round. The resulting subset is then fixed and reused for downstream training. Experiments on D4RL demonstrate that CODS can retain most of the full-data performance using only 10% of the data, outperforming several existing data selection baselines. The authors also provide additional analyses on selection budgets, robustness, computational costs, and extensions to language-based tasks.

Critical strengths

The paper addresses a practically relevant problem of reducing redundant data in offline RL. The experiments cover multiple offline RL algorithms and include comparisons with several baselines, demonstrating the potential effectiveness of the proposed selection procedure.

Critical weaknesses
  1. The presentation lacks clarity. The theoretical bound does not have a clear connection with the empirical findings. Several experiments in Section 5 lack sufficiently clear explanations of their setups and underlying mechanisms. For instance, in Section 5.5, the authors show that their method tends to retain the reward-bearing, bottleneck, goal-proximal transitions, but do not provide an explanation on why this happens based on the ranking of the score in CODS.

  2. The core idea of iteratively selecting informative data while updating a critic is not fundamentally new. Compared with existing approaches such as ReDOR, the main contribution appears to be the use of Bellman residuals as a relatively simple selection criterion within an iterative procedure. Though the iterative nature of the algorithm might be essential, as partly indicated in Table 3, more experiments on challenging tasks are needed to strengthen this claim.

Questions for authors
  1. Although the results on the D4RL benchmark in Table 1 support the effectiveness on the data selection process to a certain extent, it still seems a bit limited since the D4RL tasks are generally not challenging enough. Could you further provide results on tasks with longer horizons (e.g., OGBench) to show that your method can be applied to harder tasks?
  2. In Figure 2, it is shown that selecting 15% of the data yields better results than 10%. Why not choose 15% as your main results instead of the currently used 10%? Would it be possible to provide some justification on your choice?
  3. Could the remaining performance gap between CODS and Pool be partly explained by reduced state-action coverage? Could you provide some metrics on the data coverage before and after you conduct the data selection?
  4. Could you give a reasonable explanation on why your data selection tends to retain the reward-bearing, bottleneck, goal-proximal transitions? This could be either theoretical or empirical.
4 comments
AuthorsOpenReview ↗

Response to Reviewer SqbU – Part 1

Thank you for the four questions. The table summarizes our answers, and the sections after it give the details.

Your pointOur answerWhere in the revision
First question: longer horizonsThree OGBench antmaze-large datasets. CODS beats ReDOR by 7.1, 6.6, and 6.2 points and keeps 73.7% of Pool success.Section 5.5; Appendix R, Table 35; abstract
Second question: the 10% budget15% retains 98.5% against 95.9% but needs 50% more data, so 10% is the compression operating point. At 2%, native CODS leads a mature one-shot selector in every cell.Table 17; Figure 2; Appendix Q
Third question: does reduced coverage explain the gap to Pool?In part. Measured directly, k-center covers the most held-out transitions and CODS is second in every cell, yet k-center trails CODS by 4.9–19.6 points.Appendix R, Table 34; Limitations
Fourth question: why does residual ranking enrich these categories?Large residuals start where reward enters the backup and move back along the value frontier. A new diagnostic shows the leading category changing from round to round.Section 5.5; Appendix R, Table 33
Novelty and the role of iterationThe score separates CODS from ReDOR. Iteration adds 0.48 points at 10% and 0.7–1.2 points at 2%, and it allows stale transitions to be evicted.Appendix Q
Table 19's denominatorOriginal-category retention divided by the 10% budget, now backed by the category counts.Table 19 caption; Appendix R, Table 32

Direct answers

To the first question: we added three long-horizon OGBench datasets, antmaze-large navigate, stitch, and explore, with IQL. CODS beats ReDOR on all three, by 7.1, 6.6, and 6.2 points, and keeps 73.7% of Pool success over the three. To the second question: 15% retains more, 98.5% against 95.9% of Pool on the four representative cells, but needs 50% more data, and 10% is the compression operating point. To the third question: in part. k-center covers more held-out transitions than CODS in every cell, so lower coverage than k-center may contribute to the gap from Pool, although CODS covers more than late one-shot selection and Random in every cell. To the fourth question: residuals first appear where reward enters the backup and then spread back along the value frontier through the bottleneck cells. A new residual-rank diagnostic shows the leading category changing across rounds, and the removal interventions of Table 19 show that these categories matter to the downstream learner. The sections below give the details.

Longer horizons: OGBench

The extension uses three AntMaze datasets from ogbench==1.2.1, each in its default single-task form: antmaze-large-navigate-singletask-task1-v0, antmaze-large-stitch-singletask-task1-v0, and antmaze-large-explore-singletask-task1-v0, with the supplied training and validation split and a 1,000-step evaluation horizon. Data are loaded with ogbench.make_env_and_datasets, bootstrap targets use masks rather than the trajectory-ending terminals, and no D4RL reward preprocessing is applied. IQL uses expectile 0.9, inverse temperature 10, and two hidden layers of 256 units. Subsets contain 10% of the training transitions and receive a 100,000-update selector budget. Selection seeds 0, 1, and 2 and downstream seeds 100 and 101 give six policies per method and dataset, and comparisons are paired by seed. Every policy receives 1,000,000 updates and 100 evaluation episodes at its final checkpoint.

Methodnavigatestitchexplore
Pool0.590 ± 0.0600.644 ± 0.0460.531 ± 0.051
CODS0.431 ± 0.0570.477 ± 0.0470.393 ± 0.044
Late one-shot0.392 ± 0.0520.439 ± 0.0450.356 ± 0.042
ReDOR0.360 ± 0.0490.411 ± 0.0430.331 ± 0.040
Random0.251 ± 0.0430.283 ± 0.0380.207 ± 0.033

Entries are success rates, mean ± SD over the six policies, which are the training replicates, not over episodes. CODS beats ReDOR by 7.1, 6.6, and 6.2 points, above the 4.72-point margin of Table 2, and keeps 73.7% of the summed Pool success. Its margins over late one-shot selection, 3.9, 3.8, and 3.7 points, are less than one SD. This matches the shared-critic control: on long horizons too, the mature algorithm-matched residual carries most of the gain over ReDOR. The revision reports these datasets in Appendix R, Section 5.5, and the abstract.

The 10% budget

The 10% budget is a compression operating point, not a claim that it maximizes return. Figure 2's stronger performance at 15% is consistent with the resource–quality tradeoff. The full budget curve is therefore important, and the higher-retention setting does not invalidate the result at the smaller budget. Table 17 already reports the 15% budget on the four representative cells. Retention rises from 95.9% at 10% to 98.5% at 15% for 50% more data, and the marginal gain per budget point falls from 0.87 between 5% and 10% to 0.23 between 10% and 20%.

AuthorsOpenReview ↗

Response to Reviewer SqbU – Part 2

Novelty and the role of iteration

Iterative selection with a critic is shared with ReDOR, so the new controls locate the contribution precisely. The shared-critic control in the reply to Reviewer 1EKb shows that 10.70 of the 11.18 points come from critic maturity, and native CODS adds 0.48 points over a mature one-shot selector. What separates CODS from ReDOR is therefore mainly the score: an unpenalized Bellman residual matched to the downstream learner's backup and computed with a mature critic. That score beats ReDOR's gradient matching by 4.72 points on 19 of 20 D4RL cells and by 6.2–7.1 points on three OGBench datasets, and its selection takes two-thirds of ReDOR's time. Iteration adds a small, consistent gain on top, which grows to 0.7–1.2 points at a 2% budget, and under noisy early targets it allows stale transitions to be evicted, which a one-shot selector cannot do.

Theory and the enrichment mechanism

The local bound certifies only 0.02% of sampled pairs, so the paper uses it to delimit the heuristic rather than to explain the results (Section 4). For the enrichment in Table 19, a simple chain gives the explanation we offer. Take a deterministic chain whose only nonzero reward occurs on the transition into the goal, with a critic initialized at zero. Every residual is zero except on that reward-bearing transition, so the first acquisition selects it. Once the critic fits it, the predecessor's value becomes inconsistent, and the largest residual moves one step back. In a maze, every path from the start region to the goal crosses the bottleneck cells, so their transitions carry large residuals while the value frontier passes. Reward-bearing, goal-proximal, and bottleneck transitions are thus where large residuals first appear. The sketch assumes exact fitting and does not predict the order of the three enrichments.

A new diagnostic tests the sketch on AntMaze-medium-play. At each acquisition round it reports the mean residual percentile of each category, against 0.50 for the eligible pool:

Learner and categoryRound 1Round 2Round 3Round 4Round 5
CQL, reward-bearing0.860.820.780.740.71
CQL, bottleneck0.790.840.810.760.72
CQL, goal-proximal0.720.760.830.790.75
IQL, reward-bearing0.830.800.760.720.69
IQL, bottleneck0.770.820.790.750.70
IQL, goal-proximal0.700.740.800.770.73

All three categories rank well above the pool in every round, and the leading category changes. Reward-bearing transitions lead in round 1, as the sketch predicts, bottleneck transitions in round 2, and goal-proximal transitions from round 3 on, for both learners. The order of the last two depends on where the bottleneck cells lie relative to the goal. Rescoring therefore moves each acquisition toward a different category, which a single ranking cannot do. The revision states the mechanism in Section 5.5 and adds the sketch and this diagnostic to Appendix R.

Coverage

For each cell, state and action are concatenated, and each coordinate is standardized with eligible-pool means and standard deviations. The radius ρ\rho is the median, over eligible transitions, of the distance to the nearest other eligible transition. A held-out transition is covered when some selected transition lies within ρ\rho of it. Each cell uses a fixed set of 20,000 held-out transitions, and the whole eligible pool is evaluated at the same radius, which caps every selector's coverage.

Coverage of held-out transitions (%):

Cellρ\rhoEligible poolCODSLate one-shotRandomk-centerReDOR
Walker2d/TD3+BC0.9248.6240.6339.5229.3943.7735.20
HalfCheetah/CQL1.0547.2139.7838.7428.2042.9134.51
Hopper/IQL0.8845.9738.9337.6027.0941.7333.81
AntMaze/CQL1.3442.7936.2434.9524.5639.4731.14

The whole pool covers 42.8–48.6% of held-out transitions, and every selector covers less. k-center covers the most in every cell, and CODS is second, ahead of late one-shot selection, ReDOR, and Random. Coverage does not decide subset quality: k-center has the highest coverage yet trails CODS by 4.9 to 19.6 points on these cells (Table 1). Lower coverage than k-center remains a possible contributor to the remaining gap from Pool. The revision adds these results to Appendix R and the Limitations.

AuthorsOpenReview ↗

Response to Reviewer SqbU – Part 3

What Table 19 shows

Table 19 provides intervention evidence beyond enrichment counts. Replacing selected reward-bearing, bottleneck, and goal-proximal transitions produces reported CQL return drops of 10.7, 8.3, and 4.8 points, compared with 1.1 for random replacement. These categories contribute to downstream performance. Excluding reward-bearing transitions also removes direct reward, as Appendix P.6 notes, because their replacements cannot preserve reward sign. The bottleneck and goal-proximal removals are the cleaner tests, since their replacements match the raw reward indicator, position, and density, and they still cost 8.3 and 4.8 points against 1.1 for random replacement.

Category denominators

Table 19's enrichment depends on the category denominator. For category CC, selected set SS, eligible pool EE, and original dataset DD, three quantities differ:

\text{Eligible-category retention}&=\frac{|S\cap C|}{|E\cap C|},\\ \text{Original-category retention}&=\frac{|S\cap C|}{|D\cap C|},\\ \text{Selected-set composition}&=\frac{|S\cap C|}{|S|}. \end{aligned}$$ Relative to the eligible pool, $$\text{Enrichment} =\frac{|S\cap C|/|S|}{|E\cap C|/|E|} =\frac{\text{Eligible-category retention}}{|S|/|E|}.$$ When $|S|=0.10|D|$ and $|E|=0.90|D|$, uniform selection from the eligible pool retains 11.11% of each eligible category in expectation. Table 19 reports original-category retention, $|S\cap C|/|D\cap C|$, and divides it by the 10% original-data budget. The reported 31.8%, 27.4%, and 24.9% therefore give the enrichments 3.18×, 2.74×, and 2.49× for CQL. The counts settle the other denominators. The masks are fixed on the original data, which contains 18,497 reward-bearing, 89,631 bottleneck, and 121,844 goal-proximal transitions. Of these, 16,642, 80,607, and 109,719 are eligible. Over the five selection seeds, the CQL subsets hold on average 5,889.4, 24,532.2, and 30,290.4 of them. Original-category retention is therefore 31.84%, 27.37%, and 24.86%, which Table 19 rounds to 31.8%, 27.4%, and 24.9%. Eligible-category retention is 35.39%, 30.43%, and 27.61%. The eligible pool keeps each category's share of the data within 0.01 points, so eligible-pool enrichment, 3.18×, 2.74×, and 2.48×, agrees with Table 19 to within 0.01. The IQL subsets hold 5,301.0, 22,192.6, and 26,878.8, for retention of 28.66%, 24.76%, and 22.06%, which Table 19 rounds to 28.7%, 24.8%, and 22.1%. The revision states the definition in the caption of Table 19 and reports these counts in Appendix R. This accounting question does not affect the category-removal return differences or the Eligible-Pool performance retention. Please let us know whether these results answer your questions.

Thank you for the detailed response and additional experiments. The authors have addressed most of my concerns by providing evaluations on more challenging OGBench tasks and clarifying the choice of the 10% data budget. Besides, they also added coverage analysis and further explanations of the data selection mechanism. I now have a clearer view of why the reward-bearing, bottleneck, and goal-proximal data are selected and turn out to be helpful for the performance.

While I still have some reservations about the methodological novelty and the overall significance of this work, I believe the rebuttal has adequately addressed most of my concerns. Therefore, I will raise my rating to 3.

Reviewer_cm3FOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

I'm not very familiar with this field, but I found that this paper is mainly compared to a paper called ReDOR from ICLR 2025, and ReDOR is much better written. The core idea of CODS is to repeatedly use a trained value estimator to identify informative transitions, then freeze the selected subset for reuse across offline reinforcement learning runs.

Critical strengths

I think the point of this paper is to use a fixed, small subset of data that enables the downstream policies to retain most of the full dataset’s performance, the novelty lies in repeatedly scoring using Bellman residuals and forming reusable subsets using lighter rules. However, as I'm not very familiar with this field, I cannot judge the importance of this solution, but I think the problem itself is sound.

Critical weaknesses
  1. I'm not sure whether a large Bellman residual can necessarily mean a transition is useful, or whether it possibly also reflects noise or a mistaken critic estimate.

  2. I found that ReDOR already attempts to select data across multiple critic stages and freezes a subset. CODS’s distinction is its simpler, algorithm-matched residual score, but as I mentioned above, I cannot judge how important it is. If this point is indeed important, then I suggest the author better explain why this issue is important in the related work section, so that all readers can accept it.

Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer cm3F

Thank you for the review. The table summarizes our answers.

Your pointOur answerWhere in the revision
Relation to ReDORThe difference is the score: an unpenalized Bellman residual matched to the downstream learner's backup. It beats ReDOR by 4.72 points on 19 of 20 D4RL cells and by 6.2–7.1 points on three OGBench datasets, and its selection takes 3.0 rather than 4.5 hours on Walker2d-medium.Related Work; Tables 2 and 6; Appendix R, Table 35
Can a large residual reflect noise or a mistaken critic?Yes. New controls measure corrupted bootstrap targets and a smaller selection critic, and the reply to Reviewer 1EKb lists when the selector is reliable.Appendix Q, Tables 29–31
What does the local theory support?A pairwise condition under which residual order implies local-proxy order, and a noise margin for estimated residuals. It certifies 0.02% of sampled pairs, so it delimits the heuristic rather than explaining the results.Section 4; Appendix Q

Relation to ReDOR

ReDOR is relevant prior work, and iteration plus subset reuse is not unique to CODS. The distinction is the selection objective. CODS uses algorithm-matched, unpenalized Bellman residuals rather than the compared gradient-matching objective. Appendix P.1 and Table 14 report implementation-fidelity checks, and Table 15 includes OPER across the main cells. These comparisons support the empirical value of the simpler score under a common data budget and fully charged selector cost.

The revision explains why the simpler score matters, and it rewrites Sections 3 and 5 in plainer language. Under equal budgets, CODS exceeds ReDOR by 4.72 points on 19 of 20 cells (Table 2). On Walker2d-medium, its one-time selection takes 3.0 rather than 4.5 hours (Table 6). Its residual also follows the backup of the learner that reuses the subset, and mismatched selectors lose 1.4–3.9 retention points (Appendix P.8). On three new long-horizon OGBench datasets, CODS beats ReDOR by 7.1, 6.6, and 6.2 points. The new shared-critic control (reply to Reviewer 1EKb) shows that repeated rescoring adds 0.48 points at equal critic training. The score, not the iteration, is therefore the main difference from ReDOR, and a one-shot selector with a mature critic captures most of its benefit. The revision adds these reasons to the ReDOR paragraph of the Related Work section.

What the local theory supports

The local theory explains when residual ranking agrees with a quadratic influence measure. It does not identify every high-residual transition as useful, and large residuals can indeed reflect noise or a mistaken critic. The reply to Reviewer 1EKb summarizes when the selector is reliable. For a fixed critic, let I≻0I\succ0 have eigenvalues in [λmin⁡,λmax⁡][\lambda_{\min},\lambda_{\max}], let 0<m≤∥gτ∥≤M0<m\le\|g_\tau\|\le M, and define Jτ=12δτ2gτ⊤I−1gτ.J_\tau=\tfrac12\delta_\tau^2g_\tau^\top I^{-1}g_\tau. Then Rayleigh-quotient bounds give

\le\sqrt{J_\tau}\le \frac{M|\delta_\tau|}{\sqrt{2\lambda_{\min}}}.$$ Thus $|\delta_i|>(M/m)\sqrt{\lambda_{\max}/\lambda_{\min}}\,|\delta_j|$ guarantees $J_i>J_j$. If the operator is restricted to the gradient span, its inverse must be defined on that span. The submitted worst-case factor is approximately $2\times10^3$, and only 0.02% of sampled pairs satisfy the sufficient separation condition. This is a limited local comparison, not the explanation of all observed gains. The new Appendix Q states this pairwise condition explicitly, together with the noise statement below. For bounded residual error $|\widehat\delta_\tau-\delta_\tau|\le\nu$, the reverse triangle inequality gives $\big||\widehat\delta_\tau|-|\delta_\tau|\big|\le\nu$. A true residual gap exceeding $2\nu$ therefore preserves the residual ordering. Establishing that noise bound, or connecting residual order to policy return, requires additional assumptions. Please let us know whether these changes address your concerns.
Reviewer_1EKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CODS constructs reusable offline RL subsets by alternating critic training with acquisition of transitions having large absolute Bellman residuals. After selection, the subset is frozen and downstream learners are trained from scratch. Experiments compare subset quality, acquisition schedules, robustness, and the amortization of selection costs across repeated training runs.

Critical strengths
  1. Substantial empirical evaluation. The study includes multiple learners, nested selection and training seeds, competitive subset-selection baselines, and corruption and duplication tests. These provide useful evidence about both effectiveness and failure modes.
  2. Careful qualification of efficiency claims. The paper distinguishes equal-update, equal-pass, and equal-hour comparisons, and explicitly acknowledges that selection is slower for a single run and provides no formal coreset guarantee.
Critical weaknesses
  1. High residuals do not necessarily identify useful data. They can reflect learnable errors, incorrect bootstrap targets, noise, or critic misspecification. The burn-in and corruption tests address part of this concern, but leave unclear when residual-based selection is reliable. Since selected transitions cannot be removed, early mistakes can permanently consume the subset budget.
  2. The rescoring comparison uses different amounts of critic training before selection. The one-shot method fixes its subset after 25,000 updates, while the five-round method makes its final selection after about 85,000. Later training cannot change the one-shot subset. The comparison therefore does not separate repeated selection from the effect of additional critic training before later choices.
Questions for authors
  1. Can the authors test selection under noisy bootstrap targets or critic misspecification, and measure whether later rounds recover from misleading early selections?
  2. Does iterative selection still outperform a one-shot selector when both receive the same amount of critic training before their final selection?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1EKb – Part 1

Thank you for the review. The table summarizes our answers, and the sections after it give the designs and every result.

Your pointOur answerWhere in the revision
First question: can high residuals reflect noisy bootstrap targets or a misspecified critic?Yes, and both are now measured. Corrupted early targets leave native CODS at 43.1, against 56.1 for a late one-shot selector. Evicting stale transitions recovers 10.4 of those 13.0 points and leaves clean cells unchanged within their 95% intervals. A smaller selection critic costs CODS 4.2 points and late one-shot selection 5.3.Appendix Q, Tables 29–31
Second question: does iteration matter once a one-shot selector gets the same critic training?Critic maturity carries 10.70 of the 11.18 points. Native CODS adds 0.48 points at 10%, with intervals excluding zero in three of four cells, and 0.7–1.2 points in every cell at 2%.Section 5.3; Appendix Q, Tables 26–28
The round ablation also changes critic maturity at the last acquisitionCorrect. Section 5.3 and the Limitations now say so, and Appendix Q lists the last acquisition time for every KK.Section 5.3; Limitations; Appendix Q
When is residual selection reliable?On clean data with a burn-in of at least 2%. The section below lists every failure we measured and its size.Tables 20–22; Appendix Q

Direct answers

To the first question: yes, we tested both noisy bootstrap targets and a misspecified critic. When bootstrap targets are corrupted during burn-in and the first acquisition, 27.4% of the first acquired batch is corrupted. Later batches add fewer corrupted transitions, 9.4% by the fifth, close to the 10% base rate, but native CODS never removes the first batch. It therefore ends at 14.84% contamination and a return of 43.1, against 56.1 for a late one-shot selector. Later rounds can repair part of this, which a one-shot selector cannot attempt: evicting transitions whose residual has become small raises native CODS to 53.5, and it leaves all four clean cells unchanged within their 95% intervals. A smaller selection critic lowers CODS by 4.2 points and late one-shot selection by 5.3 points.

To the second question: once the one-shot selector receives the same critic training before its final selection, iterative selection adds a small but consistent gain. Late one-shot selection recovers 10.70 of the 11.18 points by which native CODS beats early one-shot selection in this control. Native CODS adds 0.48 points, and its interval excludes zero in three of the four cells. At a 2% budget its lead over late one-shot selection grows to 0.7–1.2 points in every cell. Staged acquisition on the shared critic adds 0.10 points, with an interval that excludes zero only on Hopper. The sections below give the designs and the full results.

Critic maturity at the last acquisition

The round ablation holds total critic training fixed, but not critic maturity at the last acquisition. Algorithm 1 acquires before each post-acquisition update, so the final subset decision occurs after 25,000+K−1K75,00025{,}000+\frac{K-1}{K}75{,}000 updates: 25,000 for one round and 85,000 for five rounds. Both finish 100,000 updates, but later training cannot change the one-shot indices. The 11.23-point average improvement in Table 3 is thus an equal-total-update schedule result, not an isolated effect of rescoring. The revision corrects the sentence in Section 5.3 that attributed the gain to rescoring alone and adds this point to the Limitations. A new final Appendix Q lists the last acquisition time for every KK: 25,000, 75,000, 85,000, and 92,500 updates for K=1,3,5,10K=1,3,5,10.

AuthorsOpenReview ↗

Response to Reviewer 1EKb – Part 2

Shared-critic control: design

A shared critic trajectory separates critic maturity from staged acquisition. The control uses the four cells of Table 3: Walker2d-medium with TD3+BC, HalfCheetah-medium with CQL, Hopper-medium with IQL, and AntMaze-medium-play with CQL. The original dataset, trajectory holdout, eligible pool, burn-in indices, and preprocessing are shared across arms. Within each cell and selection replicate, a reference learner trains on a common 2% burn-in for 25,000 updates. Training then continues on a fixed, uniformly sampled minibatch stream from the entire eligible pool, independently of any acquisition. We save the complete scoring state at 25k, 40k, 55k, 70k, and 85k updates: the critic, the target networks, any actor or value network used in the backup, and the state of the target draws.

SelectorAcquisition ruleFinal subset
Early one-shotAcquire the remaining 8% at 25k updates10% of original data
Late one-shotAcquire the remaining 8% at 85k updates10%
Five stages, shared trajectoryAcquire 1.6% at each saved checkpoint10%
Native CODSUse its own critic trained on the growing selected subset10%

Every arm is charged 100,000 critic updates, including the shared reference training. Fresh downstream policies receive 1,000,000 updates and are evaluated as in the paper: every 5,000 updates over 10 deterministic episodes, averaging the final 10 checkpoints. The first three selectors share one reference trajectory within each cell and replicate. Late minus early one-shot therefore measures additional critic training, and staged minus late one-shot tests staged acquisition at equal final maturity. Native CODS also changes the critic's training data, because its critic sees only the growing subset while the shared critic sees the whole pool. That comparison therefore does not isolate a single mechanism. Early one-shot selection and native CODS run the K=1K=1 and K=5K=5 procedures. Native CODS reproduces the K=5K=5 means of Table 3, and the early one-shot means, from a fresh run, lie within 0.4 points of its K=1K=1 means.

The control uses selection seeds 0–4 and downstream seeds 100, 101, and 102, with separate random streams for burn-in sampling, minibatch order, stochastic targets, corruption, and evaluation. We compute method differences at matching selection and downstream seeds, so the selection artifact is the primary independent replicate. Paired 95% percentile intervals use 100,000 hierarchical bootstrap replicates: selection artifacts are resampled first and their downstream runs second, with the same indices for every method. Cells are fixed, and the intervals are not adjusted for multiplicity.

Shared-critic control: results

Retention on the four cells is 100∑cRm,c/∑cRPool,c100\sum_cR_{m,c}/\sum_cR_{\mathrm{Pool},c}, with the submitted Pool sum of 255.2 as the common reference:

SelectorFinal choice after critic updatesPool-performance retention (%)
Early one-shot, shared critic25,00078.37
Late one-shot, shared critic85,00095.14
Five stages, shared critic85,00095.30
Native CODS, subset-dependent critic85,00095.89

Paired differences in normalized return:

CellLate − earlyFive stages − lateNative − late
Walker2d-medium/TD3+BC12.6 [11.5, 13.8]0.2 [−0.3, 0.7]0.7 [0.2, 1.2]
HalfCheetah-medium/CQL3.7 [3.1, 4.4]−0.2 [−0.5, 0.1]0.0 [−0.3, 0.3]
Hopper-medium/IQL10.7 [9.8, 11.7]0.5 [0.1, 0.9]0.6 [0.1, 1.1]
AntMaze-medium-play/CQL15.8 [14.1, 17.6]−0.1 [−0.7, 0.5]0.6 [0.1, 1.2]
Four-cell mean of the differences10.700.100.48

The per-cell means are 65.2, 39.4, 53.8, and 41.6 for early one-shot selection, 77.8, 43.1, 64.5, and 57.4 for late one-shot selection, 78.0, 42.9, 65.0, and 57.3 for five stages, and 78.5, 43.1, 65.1, and 58.0 for native CODS. Appendix Q reports them with SDs.

Selecting once with the more mature critic raises the four-cell mean by 10.70 points, and the gain is positive in every cell. At equal final maturity, staging the same budget on the shared critic adds 0.10 points, and its interval excludes zero only on Hopper. Native CODS adds 0.48 points, and its interval excludes zero on Walker2d, Hopper, and AntMaze. Of the 11.18-point gain of native CODS over early one-shot selection, 10.70 points therefore come from critic maturity at the final acquisition. The critic's maturity at selection time is what a practitioner must budget for first, and native staged acquisition then adds a small, consistent gain. Native CODS also trains its critic on the growing subset rather than the whole pool, so this comparison does not isolate staging alone.

AuthorsOpenReview ↗

Response to Reviewer 1EKb – Part 3

The same comparison at a 2% budget asks whether iteration matters more when data are scarce. Late one-shot selection, run under the submitted budget sweep's schedule and acquiring once at the final critic maturity, scores 53.9 ± 2.7, 31.9 ± 1.8, 47.0 ± 3.1, and 24.5 ± 4.2 on the four cells, against 55.0, 32.6, 48.2, and 25.6 for native CODS. Native CODS leads in every cell, by 1.1, 0.7, 1.2, and 1.1 points, a mean of 1.03 points against 0.48 at 10%. A rerun of Random at 2% on Walker2d scores 18.6 ± 2.3, matching Figure 2. The revision adds this comparison to Appendix Q. Scoring work still differs, because one-shot selection scans the pool once and staged selection five times, so the comparison matches critic updates rather than wall time.

When residual selection is reliable

High residuals can reflect learnable error, wrong bootstrap targets, noise, or a misspecified critic, and the evidence now separates several of these cases. On clean D4RL data with a burn-in of at least 2%, residual selection retains 96.6% of Pool over 20 cells. Burn-ins of 1% and 0.5% retain 95.2% and 93.1% (Table 21). The selector is unreliable under outliers and redundancy. With 10% reward corruption, raw CODS falls from 58.0 to 27.8 on CQL AntMaze, and clipped-ensemble scoring recovers only to 45.4 (Table 20). Fivefold duplication of common transitions lowers it to 48.3, and the residual–coverage hybrid recovers to 54.9 (Table 22). The two new controls below add misleading early bootstrap targets and a misspecified selection critic to this list. Under misleading early targets, a late one-shot selector exceeds native CODS by 13.0 points, and evicting stale transitions recovers 10.4 of them. Under a misspecified critic, the two selectors are within 1.7 points.

Corrupted bootstrap targets: design

Reward corruption and duplication tests expose useful failure modes, but they do not substitute for noisy bootstrap targets. Since the selected set grows monotonically, later rounds cannot evict early mistakes. Recovery can therefore mean improved return or lower contamination among newly acquired transitions, not removal of previously acquired indices. The new control corrupts the bootstrap target instead of the stored reward:

FieldSpecification
CellAntMaze-medium-play-v2 with CQL
Corruption setExactly 10% of eligible nonterminal transitions, chosen independently of residual scores
Perturbed quantityBootstrap term of critic targets and residual scores
PerturbationFixed balanced offsets ξi∈{−10,+10}\xi_i\in\{-10,+10\} in the critic's preprocessed reward units
DurationThe first 25,000 critic updates and the acquisition scoring right after update 25,000, ending at the same update for every arm
ReplicationFive selection replicates × three downstream seeds
Final downstream trainingClean rewards and clean target computation

For corruption set CC and fixed offsets ξi\xi_i, the target is

=r_i+\gamma(1-d_i)\left[ V^-_t(s_i')+ \mathbf 1\{i\in C\}\,\mathbf 1\{t\text{ in the early phase}\}\,\xi_i \right].$$ Stored rewards, observations, actions, and terminal flags are unchanged. The corrupted indices are recorded for evaluation only and are never given to a selector. Late one-shot selection acquires at 85,000 updates, after 60,000 clean updates. The control asks whether misleading early targets consume an irreversible part of the subset budget. ### Corrupted bootstrap targets: results Retention divides the return summaries by the clean Pool score of 61.2: | Procedure | Return, mean ± SD | Pool-performance retention (%) | |---|---:|---:| | Native CODS | 43.1 ± 4.5 | 70.42 ± 7.35 | | Persistence filter | 50.8 ± 3.9 | 83.01 ± 6.37 | | Eviction variant | 53.5 ± 3.3 | 87.42 ± 5.39 | | Late one-shot, shared critic | 56.1 ± 3.5 | 91.67 ± 5.72 | Share of corrupted transitions among the transitions native CODS adds at each stage (%), where the burn-in makes up 20% of the final subset and each acquisition 16%: | Arm | Burn-in | Batch 1 | Batch 2 | Batch 3 | Batch 4 | Batch 5 | Final | |---|---:|---:|---:|---:|---:|---:|---:| | Native CODS | 9.8 | 27.4 | 17.9 | 14.6 | 11.2 | 9.4 | 14.84 | Later rounds do recover. The first acquisition scores the pool while targets are corrupted, and 27.4% of its transitions are corrupted. The corrupted share of the next batches falls to 17.9%, 14.6%, 11.2%, and 9.4%, close to the 10% base rate. Because native CODS never evicts, the first batch stays, and it ends at 14.84% contamination and a return of 43.1. Late one-shot selection, which scores after 60,000 clean updates, reaches 56.1.
AuthorsOpenReview ↗

Response to Reviewer 1EKb – Part 4

Two variants of native CODS measure what later rounds can repair, which a one-shot selector cannot attempt. The persistence filter changes how each acquisition is chosen. At each round it scores the remaining eligible transitions at two critic checkpoints 2,000 updates apart, with no acquisition between them. It converts both residual vectors to percentile ranks and admits transitions in the top 15% at both, ranked by the smaller percentile, with ties broken by transition ID. Each acquisition still adds exactly 1.6% of the original data, and the current residual rank fills any remaining slots. The rule was fixed before downstream returns were evaluated, and it raises the return from 43.1 to 50.8.

The eviction variant also removes stale transitions. Before each round from the second on, it rescores the selected transitions outside the burn-in and marks as stale those below the 50th percentile of eligible-pool residuals at both of the two most recent scoring checkpoints. It evicts at most 25% of one acquisition batch per round, starting with the lowest-ranked stale transitions, and refills by the persistence-filter ranking, so the subset still reaches exactly 10%. Eviction raises the return to 53.5 and recovers 10.4 of the 13.0 points by which native CODS trails late one-shot selection. Neither variant uses the corruption labels, and neither result is credited to native CODS. On the four clean cells of Table 3, eviction changes the return by −0.2, +0.1, −0.2, and +0.2 points, with paired 95% intervals [−0.55, 0.18], [−0.14, 0.34], [−0.51, 0.13], and [−0.43, 0.79] that all include zero. Eviction therefore repairs corrupted early batches without measurably changing clean performance. The revision adds this control to Appendix Q.

Selector-critic misspecification

On AntMaze-medium-play with CQL, CODS and late one-shot selection use either the standard selection critic, with two hidden layers of 256 units, or a single hidden layer of 64 units. Update budgets, backups, the actor, the burn-in, and the paired 5 × 3 seed design are fixed. Every downstream learner is the original two-layer, 256-unit CQL model, initialized afresh and trained on clean data.

Selection criticSelectorReturn, mean ± SDRetention against Pool 61.2 (%)
Two hidden layers, 256 units eachCODS58.0 ± 3.894.77 ± 6.21
Two hidden layers, 256 units eachLate one-shot57.4 ± 4.293.79 ± 6.86
One hidden layer, 64 unitsCODS53.8 ± 3.987.9 ± 6.4
One hidden layer, 64 unitsLate one-shot52.1 ± 3.785.1 ± 6.0

The smaller critic lowers CODS by 4.2 points and late one-shot selection by 5.3 points. Under this misspecification CODS leads by 1.7 points, less than half of either SD, so the test does not show that iterative acquisition protects against a misspecified selection critic. The two-layer rows are the native CODS score of Table 3 and the late one-shot score of the shared-critic control. The one-layer rows were measured as retention against the Pool score of 61.2, and their returns are retention times 61.2. Bootstrap noise and reduced critic capacity were tested with separate interventions, and the revision adds both to Appendix Q.

Please let us know whether these results answer your questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for requests that sharpened the paper. We ran every control they asked for, and three results change how CODS should be read.

  • Iteration has a measured role. When a one-shot selector receives the same critic training, native CODS still leads by 0.48 points at the 10% budget, with per-cell intervals that exclude zero in three of the four cells. At a 2% budget it leads in every cell, by 0.7 to 1.2 points.
  • Later rounds can repair early mistakes. When early bootstrap targets are corrupted, evicting transitions whose residual has become small recovers 10.4 of the 13.0 points by which native CODS trails a late one-shot selector. On the four clean cells, eviction changes no return outside its 95% interval.
  • The gain over ReDOR holds on long horizons. On three OGBench antmaze-large datasets, CODS beats ReDOR by 6.2 to 7.1 points, more than its 4.72-point margin on D4RL.

The shared-critic control requested by Reviewer 1EKb also identifies the active ingredient: the residual must come from a mature, algorithm-matched critic. In this control, native CODS gains 11.18 points over early one-shot selection, close to the 11.23 points of Table 3, and a single selection with an equally trained critic recovers 10.70 of them. The revision states this mechanism in the abstract, the Introduction, Sections 3.2, 5.3, and 5.5, the Discussion, and the Limitations. All new results sit in two new final appendices, Q and R, so the submitted section, table, and figure numbers are unchanged.

Revisions at a glance

New resultRequested byWhere in the revision
Shared-critic control: critic maturity versus staged acquisition1EKbSection 5.3; Appendix Q, Tables 26–28
2% budget against a mature one-shot selector1EKb, SqbUSection 5.3; Appendix Q
Corrupted early bootstrap targets, two repair variants, and eviction on clean cells1EKb, cm3FAppendix Q, Tables 29–30
Smaller selection critic1EKb, cm3FAppendix Q, Table 31
Pairwise condition for residual order, with a noise margincm3FAppendix Q
Category counts behind Table 19SqbUAppendix R, Table 32
Residual ranks by category and roundSqbUAppendix R, Table 33
Held-out coverage, including the whole eligible poolSqbUAppendix R, Table 34
Three long-horizon OGBench datasetsSqbUSection 5.5; Appendix R, Table 35; abstract
Why the simpler score matters, and plainer Sections 3 and 5cm3FRelated Work; Sections 3 and 5

What the paper claims

CODS selects a static, reusable 10% subset with algorithm-matched Bellman residuals. Across 20 D4RL task–algorithm cells, it retains 96.6% of eligible-pool performance. It beats ReDOR and OPER on 19 of 20 cells and every other subset baseline on all 20, and all six advantages remain significant after Holm correction. The contribution is a simple selector whose full selection cost is charged and recovered after two reuses. Its guarantees are empirical, and Section 4 states exactly what the local theory supports.

The compute advantage comes from reuse, and the paper charges selection in full. One selection plus one run costs 3.18 hours, against 1.8 hours for one pool run, so CODS breaks even at the second run and cuts total time 5.45× at 20 runs under equal dataset passes. Budget and performance denominators also differ. The budget B=0.10NB=0.10N counts original transitions, whereas retention compares against the eligible pool. If that pool contains 0.90N0.90N transitions, the same subset is 11.1% of it. Enrichment claims need one consistent category denominator, and the reply to Reviewer SqbU states the one Table 19 uses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.