- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CODS, an iterative data selection method for offline reinforcement learning. CODS starts from a small randomly selected subset and progressively acquires transitions with high Bellman residuals, updating the critic after each selection round. The resulting subset is then fixed and reused for downstream training. Experiments on D4RL demonstrate that CODS can retain most of the full-data performance using only 10% of the data, outperforming several existing data selection baselines. The authors also provide additional analyses on selection budgets, robustness, computational costs, and extensions to language-based tasks.
Critical strengths
The paper addresses a practically relevant problem of reducing redundant data in offline RL. The experiments cover multiple offline RL algorithms and include comparisons with several baselines, demonstrating the potential effectiveness of the proposed selection procedure.
Critical weaknesses
-
The presentation lacks clarity. The theoretical bound does not have a clear connection with the empirical findings. Several experiments in Section 5 lack sufficiently clear explanations of their setups and underlying mechanisms. For instance, in Section 5.5, the authors show that their method tends to retain the reward-bearing, bottleneck, goal-proximal transitions, but do not provide an explanation on why this happens based on the ranking of the score in CODS.
-
The core idea of iteratively selecting informative data while updating a critic is not fundamentally new. Compared with existing approaches such as ReDOR, the main contribution appears to be the use of Bellman residuals as a relatively simple selection criterion within an iterative procedure. Though the iterative nature of the algorithm might be essential, as partly indicated in Table 3, more experiments on challenging tasks are needed to strengthen this claim.
Questions for authors
- Although the results on the D4RL benchmark in Table 1 support the effectiveness on the data selection process to a certain extent, it still seems a bit limited since the D4RL tasks are generally not challenging enough. Could you further provide results on tasks with longer horizons (e.g., OGBench) to show that your method can be applied to harder tasks?
- In Figure 2, it is shown that selecting 15% of the data yields better results than 10%. Why not choose 15% as your main results instead of the currently used 10%? Would it be possible to provide some justification on your choice?
- Could the remaining performance gap between CODS and Pool be partly explained by reduced state-action coverage? Could you provide some metrics on the data coverage before and after you conduct the data selection?
- Could you give a reasonable explanation on why your data selection tends to retain the reward-bearing, bottleneck, goal-proximal transitions? This could be either theoretical or empirical.
4 comments
Response to Reviewer SqbU – Part 2
Novelty and the role of iteration
Iterative selection with a critic is shared with ReDOR, so the new controls locate the contribution precisely. The shared-critic control in the reply to Reviewer 1EKb shows that 10.70 of the 11.18 points come from critic maturity, and native CODS adds 0.48 points over a mature one-shot selector. What separates CODS from ReDOR is therefore mainly the score: an unpenalized Bellman residual matched to the downstream learner's backup and computed with a mature critic. That score beats ReDOR's gradient matching by 4.72 points on 19 of 20 D4RL cells and by 6.2–7.1 points on three OGBench datasets, and its selection takes two-thirds of ReDOR's time. Iteration adds a small, consistent gain on top, which grows to 0.7–1.2 points at a 2% budget, and under noisy early targets it allows stale transitions to be evicted, which a one-shot selector cannot do.
Theory and the enrichment mechanism
The local bound certifies only 0.02% of sampled pairs, so the paper uses it to delimit the heuristic rather than to explain the results (Section 4). For the enrichment in Table 19, a simple chain gives the explanation we offer. Take a deterministic chain whose only nonzero reward occurs on the transition into the goal, with a critic initialized at zero. Every residual is zero except on that reward-bearing transition, so the first acquisition selects it. Once the critic fits it, the predecessor's value becomes inconsistent, and the largest residual moves one step back. In a maze, every path from the start region to the goal crosses the bottleneck cells, so their transitions carry large residuals while the value frontier passes. Reward-bearing, goal-proximal, and bottleneck transitions are thus where large residuals first appear. The sketch assumes exact fitting and does not predict the order of the three enrichments.
A new diagnostic tests the sketch on AntMaze-medium-play. At each acquisition round it reports the mean residual percentile of each category, against 0.50 for the eligible pool:
| Learner and category | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 |
|---|---|---|---|---|---|
| CQL, reward-bearing | 0.86 | 0.82 | 0.78 | 0.74 | 0.71 |
| CQL, bottleneck | 0.79 | 0.84 | 0.81 | 0.76 | 0.72 |
| CQL, goal-proximal | 0.72 | 0.76 | 0.83 | 0.79 | 0.75 |
| IQL, reward-bearing | 0.83 | 0.80 | 0.76 | 0.72 | 0.69 |
| IQL, bottleneck | 0.77 | 0.82 | 0.79 | 0.75 | 0.70 |
| IQL, goal-proximal | 0.70 | 0.74 | 0.80 | 0.77 | 0.73 |
All three categories rank well above the pool in every round, and the leading category changes. Reward-bearing transitions lead in round 1, as the sketch predicts, bottleneck transitions in round 2, and goal-proximal transitions from round 3 on, for both learners. The order of the last two depends on where the bottleneck cells lie relative to the goal. Rescoring therefore moves each acquisition toward a different category, which a single ranking cannot do. The revision states the mechanism in Section 5.5 and adds the sketch and this diagnostic to Appendix R.
Coverage
For each cell, state and action are concatenated, and each coordinate is standardized with eligible-pool means and standard deviations. The radius is the median, over eligible transitions, of the distance to the nearest other eligible transition. A held-out transition is covered when some selected transition lies within of it. Each cell uses a fixed set of 20,000 held-out transitions, and the whole eligible pool is evaluated at the same radius, which caps every selector's coverage.
Coverage of held-out transitions (%):
| Cell | Eligible pool | CODS | Late one-shot | Random | k-center | ReDOR | |
|---|---|---|---|---|---|---|---|
| Walker2d/TD3+BC | 0.92 | 48.62 | 40.63 | 39.52 | 29.39 | 43.77 | 35.20 |
| HalfCheetah/CQL | 1.05 | 47.21 | 39.78 | 38.74 | 28.20 | 42.91 | 34.51 |
| Hopper/IQL | 0.88 | 45.97 | 38.93 | 37.60 | 27.09 | 41.73 | 33.81 |
| AntMaze/CQL | 1.34 | 42.79 | 36.24 | 34.95 | 24.56 | 39.47 | 31.14 |
The whole pool covers 42.8–48.6% of held-out transitions, and every selector covers less. k-center covers the most in every cell, and CODS is second, ahead of late one-shot selection, ReDOR, and Random. Coverage does not decide subset quality: k-center has the highest coverage yet trails CODS by 4.9 to 19.6 points on these cells (Table 1). Lower coverage than k-center remains a possible contributor to the remaining gap from Pool. The revision adds these results to Appendix R and the Limitations.
Response to Reviewer SqbU – Part 3
What Table 19 shows
Table 19 provides intervention evidence beyond enrichment counts. Replacing selected reward-bearing, bottleneck, and goal-proximal transitions produces reported CQL return drops of 10.7, 8.3, and 4.8 points, compared with 1.1 for random replacement. These categories contribute to downstream performance. Excluding reward-bearing transitions also removes direct reward, as Appendix P.6 notes, because their replacements cannot preserve reward sign. The bottleneck and goal-proximal removals are the cleaner tests, since their replacements match the raw reward indicator, position, and density, and they still cost 8.3 and 4.8 points against 1.1 for random replacement.
Category denominators
Table 19's enrichment depends on the category denominator. For category , selected set , eligible pool , and original dataset , three quantities differ:
\text{Eligible-category retention}&=\frac{|S\cap C|}{|E\cap C|},\\ \text{Original-category retention}&=\frac{|S\cap C|}{|D\cap C|},\\ \text{Selected-set composition}&=\frac{|S\cap C|}{|S|}. \end{aligned}$$ Relative to the eligible pool, $$\text{Enrichment} =\frac{|S\cap C|/|S|}{|E\cap C|/|E|} =\frac{\text{Eligible-category retention}}{|S|/|E|}.$$ When $|S|=0.10|D|$ and $|E|=0.90|D|$, uniform selection from the eligible pool retains 11.11% of each eligible category in expectation. Table 19 reports original-category retention, $|S\cap C|/|D\cap C|$, and divides it by the 10% original-data budget. The reported 31.8%, 27.4%, and 24.9% therefore give the enrichments 3.18×, 2.74×, and 2.49× for CQL. The counts settle the other denominators. The masks are fixed on the original data, which contains 18,497 reward-bearing, 89,631 bottleneck, and 121,844 goal-proximal transitions. Of these, 16,642, 80,607, and 109,719 are eligible. Over the five selection seeds, the CQL subsets hold on average 5,889.4, 24,532.2, and 30,290.4 of them. Original-category retention is therefore 31.84%, 27.37%, and 24.86%, which Table 19 rounds to 31.8%, 27.4%, and 24.9%. Eligible-category retention is 35.39%, 30.43%, and 27.61%. The eligible pool keeps each category's share of the data within 0.01 points, so eligible-pool enrichment, 3.18×, 2.74×, and 2.48×, agrees with Table 19 to within 0.01. The IQL subsets hold 5,301.0, 22,192.6, and 26,878.8, for retention of 28.66%, 24.76%, and 22.06%, which Table 19 rounds to 28.7%, 24.8%, and 22.1%. The revision states the definition in the caption of Table 19 and reports these counts in Appendix R. This accounting question does not affect the category-removal return differences or the Eligible-Pool performance retention. Please let us know whether these results answer your questions.Thank you for the detailed response and additional experiments. The authors have addressed most of my concerns by providing evaluations on more challenging OGBench tasks and clarifying the choice of the 10% data budget. Besides, they also added coverage analysis and further explanations of the data selection mechanism. I now have a clearer view of why the reward-bearing, bottleneck, and goal-proximal data are selected and turn out to be helpful for the performance.
While I still have some reservations about the methodological novelty and the overall significance of this work, I believe the rebuttal has adequately addressed most of my concerns. Therefore, I will raise my rating to 3.
Response to Reviewer SqbU – Part 1
Thank you for the four questions. The table summarizes our answers, and the sections after it give the details.
Direct answers
To the first question: we added three long-horizon OGBench datasets, antmaze-large navigate, stitch, and explore, with IQL. CODS beats ReDOR on all three, by 7.1, 6.6, and 6.2 points, and keeps 73.7% of Pool success over the three. To the second question: 15% retains more, 98.5% against 95.9% of Pool on the four representative cells, but needs 50% more data, and 10% is the compression operating point. To the third question: in part. k-center covers more held-out transitions than CODS in every cell, so lower coverage than k-center may contribute to the gap from Pool, although CODS covers more than late one-shot selection and Random in every cell. To the fourth question: residuals first appear where reward enters the backup and then spread back along the value frontier through the bottleneck cells. A new residual-rank diagnostic shows the leading category changing across rounds, and the removal interventions of Table 19 show that these categories matter to the downstream learner. The sections below give the details.
Longer horizons: OGBench
The extension uses three AntMaze datasets from
ogbench==1.2.1, each in its default single-task form:antmaze-large-navigate-singletask-task1-v0,antmaze-large-stitch-singletask-task1-v0, andantmaze-large-explore-singletask-task1-v0, with the supplied training and validation split and a 1,000-step evaluation horizon. Data are loaded withogbench.make_env_and_datasets, bootstrap targets usemasksrather than the trajectory-endingterminals, and no D4RL reward preprocessing is applied. IQL uses expectile 0.9, inverse temperature 10, and two hidden layers of 256 units. Subsets contain 10% of the training transitions and receive a 100,000-update selector budget. Selection seeds 0, 1, and 2 and downstream seeds 100 and 101 give six policies per method and dataset, and comparisons are paired by seed. Every policy receives 1,000,000 updates and 100 evaluation episodes at its final checkpoint.Entries are success rates, mean ± SD over the six policies, which are the training replicates, not over episodes. CODS beats ReDOR by 7.1, 6.6, and 6.2 points, above the 4.72-point margin of Table 2, and keeps 73.7% of the summed Pool success. Its margins over late one-shot selection, 3.9, 3.8, and 3.7 points, are less than one SD. This matches the shared-critic control: on long horizons too, the mature algorithm-matched residual carries most of the gain over ReDOR. The revision reports these datasets in Appendix R, Section 5.5, and the abstract.
The 10% budget
The 10% budget is a compression operating point, not a claim that it maximizes return. Figure 2's stronger performance at 15% is consistent with the resource–quality tradeoff. The full budget curve is therefore important, and the higher-retention setting does not invalidate the result at the smaller budget. Table 17 already reports the 15% budget on the four representative cells. Retention rises from 95.9% at 10% to 98.5% at 15% for 50% more data, and the marginal gain per budget point falls from 0.87 between 5% and 10% to 0.23 between 10% and 20%.