acceptodds
Under review as a conference paper at ICLR 2027

Role-Dependent Use of Mixed-Quality Data in Offline RL

Abstract

Offline reinforcement learning learns from fixed datasets whose transitions may vary substantially in behavioral quality. Data generated by poor behavior, how- ever, may still provide useful transitions for critic learning or useful states for policy improvement even when their recorded actions are undesirable targets for imitation. We study this distinction by separating three roles of offline data: critic learning, Q-guided policy improvement at recorded states, and behavioral super- vision from recorded actions. Controlled routing experiments with expert–random mixtures show that the effect of the same additional data can differ substantially depending on the learning operation in which it is used, and can even change sign across environments and data regimes. Motivated by these findings, we in- troduce Trust-Mix, which uses the full dataset for critic learning and Q-guided policy improvement while selectively weighting recorded actions for behavioral supervision, without requiring source-policy labels. Experiments across three continuous-control environments and four expert-data budgets show that the bene- fit of additional random data varies across environments, data regimes, and offline RL algorithms.

Primary area:
reinforcement learning
Keywords:
Role-Dependent Use of Mixed-Quality Data in Offline RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zRShOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies three uses of offline data: transitions for critic learning, states for Q-guided policy improvement, and recorded actions for behavioral supervision. Source-aware controls progressively route the same random-policy component into these operations. Trust-Mix implements one source-label-free configuration: it uses the full dataset for critic learning and Q-guided updates, selectively weights imitation targets using learned advantages and critic disagreement, and adds proximal regularization and Q/BC gradient capping. Experiments use HalfCheetah, Hopper, and Walker2d, four expert-transition budgets, and a fixed one-million-transition random component. The results show environment- and regime-dependent effects rather than uniform gains from random data or uniform superiority of Trust-Mix.

Critical strengths
  1. The distinction between learning from a transition, improving a policy at its state, and imitating its recorded action is useful and operationalized through explicit routing controls. The paper appropriately acknowledges that selective imitation is not new and that broader critic/state access is not always beneficial. For example, the HalfCheetah and Walker2d contrasts in Figure 3 and Table 12 have different signs, which is more informative than a claim that low-quality data are uniformly helpful.

  2. The experimental reporting is unusually explicit about scope and limitations. Main paired comparisons use ten learner seeds, a prespecified one-million-update endpoint, and paired bootstrap intervals. The paper reports absolute returns, adverse component effects, and full learning histories, and clearly distinguishes the CQL+DW-AW adaptation from a verified reproduction. It also identifies the simultaneous changes in C3-to-C4 and avoids presenting sequential contrasts as an additive decomposition.

Critical weaknesses

1. The expert-only baselines perform poorly, and the cause is unclear.

On HalfCheetah with 100k expert transitions, expert-only BC scores -149.6 (Table 18), and expert-only Trust-Mix scores -226.4 (Table 9). The recorded expert segments have a mean return of 16,284.92 (Table 7). Appendix C.6 also shows negative returns for all 800 saved expert-only checkpoint evaluations. The authors need to explain why training on these expert data produces such poor policies. The paper uses improvements over expert-only training to argue that adding random data is useful. Interpreting that improvement requires an explanation of the poor starting point. The authors should report how well BC predicts the expert actions and check observation preprocessing, action scaling, and compatibility between the dataset and the evaluation environment.

2. The key experiment changes several parts of training at once.

The C3-to-C4 comparison changes which actions the policy imitates, which states are used for policy regularization, and the 10k-step initialization procedure (Section 5.2 and Table 10). It can also change how often the learner sees expert examples. Under uniform sampling from the full mixture, expert transitions would account for only 1/10011/1001 of samples at the 1k expert budget and 1/111/11 at 100k. The result therefore leaves several possible explanations: imitating random actions, changing initialization, changing the regularization data, or sampling expert examples less often. To identify the effect of imitating random actions, the authors should vary their inclusion in the imitation loss while keeping initialization, regularization, and expert-data exposure fixed. They should also report the actual sampling proportions used in each comparison.

Questions for authors
  1. What checks have been made on the expert-only HalfCheetah data-to-policy-to-environment interface, including action prediction, observation/action transformations, and evaluation compatibility? If available, please provide existing BC-fit or post-initialization diagnostics. The saved return histories alone do not distinguish these possibilities; explaining this discrepancy would materially affect my assessment.

  2. Please specify the actual source-sampling proportions for each C1–C4 role, and whether any existing contrasts keep initialization and expert exposure fixed when adding random data to BC or proximal supervision. If such controls are unavailable, what conclusion about the behavioral branch remains supported beyond the joint BC/proximal/initialization intervention?

  3. Is the severe expert-only weakness also present in the source-aware control backbone, and is it explained by the same mechanism as in Trust-Mix? Table 11 and Table 9 use different behavioral weighting and initialization rules. Please separate conclusions supported by the diagnostic backbone from those supported by the deployable source-label-free method.

2 comments
  1. Question - reply:

Thank you for highlighting this important concern.

As you correctly pointed out, the return histories alone are insufficient to identify the underlying causes of the poor expert-only performance in HalfCheetah. In our current study, we have conducted only a post-hoc analysis, presented in Table 24. However, we acknowledge that this analysis does not provide sufficient evidence to determine whether the observed performance degradation arises from behavioral cloning difficulties, observation preprocessing, action scaling, or potential incompatibilities between the dataset and the evaluation environment.

We have not conducted additional diagnostics specifically examining expert-action prediction accuracy or the data-to-policy-to-environment interface. Therefore, we cannot confidently attribute the negative returns to a particular mechanism.

We agree that a more systematic diagnostic investigation is necessary to understand this discrepancy and its implications for interpreting the benefits of mixed-quality datasets. We recognize this as an important limitation of our current experimental analysis and a priority for future investigation.

  1. Question - reply:

Thank you for raising this important methodological concern.

In our role-dependency experiments, we used fixed expert-data budgets ranging from 1k to 100k transitions, together with the random-data component. However, because we employed uniform sampling from the mixed dataset, we did not explicitly enforce a fixed proportion of expert transitions within each training minibatch. Consequently, particularly under smaller expert-data budgets, some minibatches may contain very few or no expert transitions.

Although the expert-data budgets were fixed for each experimental configuration, this sampling strategy does not guarantee equal expert-data exposure across learning components when random transitions are introduced.

For our component-level investigations, we aimed to maintain consistent experimental settings when examining how the inclusion of random data affects individual learning roles. However, we acknowledge that the C3-to-C4 comparison involves simultaneous changes to behavioral supervision, proximal regularization, and initialization. Therefore, this comparison cannot isolate the effect of including random actions in behavioral cloning independently of these other changes.

Our results consequently support the observation that changing how mixed-quality data are allocated across learning components can influence overall policy performance, but they do not establish that the observed differences arise exclusively from the inclusion or exclusion of random actions in the imitation objective.

We agree that a more controlled experiment, maintaining identical initialization, regularization, and expert-data exposure while modifying only the imitation targets, would provide stronger evidence for this specific effect.

  1. Question - reply:

Thank you for this question. We would like to clarify the distinction between the experimental configurations presented in Tables 9 and 11.

Table 9 reports the performance of Trust-Mix under the conventional source-label-free mixed-data setting, whereas Table 11 presents the source-aware, role-controlled experiments designed to investigate the effects of allocating different data sources to individual learning components.

In the source-aware experiments, dataset source information is explicitly used to control how expert and random transitions contribute to different learning roles. In contrast, the main Trust-Mix algorithm operates without source labels and relies on uniform sampling from the mixed dataset. As a result, particularly when expert transitions are scarce, individual training minibatches may contain no expert samples.

This distinction may contribute to the observed differences between the C4 configuration in Table 11 and the main Trust-Mix results in Table 9. However, we acknowledge that the two configurations also differ in behavioral weighting and initialization procedures. Therefore, we cannot attribute their performance differences solely to expert-data exposure. Furthermore, without additional diagnostics, we cannot establish whether the expert-only performance limitations in the source-aware backbone and Trust-Mix arise from the same underlying mechanism.

Our source-aware experiments are intended primarily to investigate the role-dependent effects of mixed-quality data under controlled data-routing configurations, whereas the source-label-free Trust-Mix experiments evaluate the proposed method under a more conventional mixed-data training setting. We recognize that these two sets of experiments support different conclusions and that this distinction should be communicated more clearly in the manuscript.

Thank you again. We appreciate the opportunity to clarify these aspects of our experimental methodology and acknowledge the limitations of our current analysis.

Thank you for your detailed response. While I appreciate the discussion, my main concerns remain unresolved. I therefore tend to keep my current score.

Reviewer_o2yNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Trust-Mix, which routes datasets based on the behavioral optimality of collected data for downstream offline reinforcement learning (RL). Trust-Mix assigns three different roles to each dataset, given the mixed sources of data: critic learning, Q-guided policy improvement, and behavior regularization. By comparing the quantity of different data sources and controlling the dataset source for downstream training, the authors show that the offline data source cannot be decoupled from the learning objective where the dataset is used.

Critical strengths

Problem formulation: Fundamental research in offline RL is somewhat saturated since there exists a large number of papers that present a "vanilla" offline RL algorithm. In contrast, this paper focuses on different perspectives in offline RL, i.e., the data source problem, which can arise when we train an offline RL algorithm in real-world problems like robotics, where collecting a large amount of "expert" data is always welcome but costs substantial human effort. Hence, the question the authors ask- how should we use the additional data more usefully- seems reasonable and relevant to our community. Compared to prior literature, where other researchers have focused on the data weighting and sampling strategies, this paper rather touches on the algorithmic part: how should the mixed data sources enter the algorithm?

Critical weaknesses
  1. Lack of evidence to support the novelty: According to the authors, experimental results suggest that the way to consume an offline data source cannot be characterized independently of the learning objective (L373) and varies across environments and data regimes (L281, L312). While it is understandable that revealing a general dominating rule within small-scale empirical observations is hard, the current paper does not provide an insightful outcome. Each shard from the experimental results cannot be gathered into one unified insight. The main results do not explain a single takeaway, e.g., mixing an expert dataset as large as possible is beneficial, or mixed critics play a more significant role than the Q-guided policy improvement in a limited expert budget, naively listing the empirical observations without insights. Considering the experimental results, I think this paper lacks novelty and does not deliver scientific insights to the offline RL community.
  2. Weak organization of the main text: Reading the appendix is not mandatory for neither the reviewers or the future readers. A well-written scientific paper should provide enough context and knowledge to understand the contribution (e.g., the algorithm) within the main text. However, the authors burden the readers with understanding Trust-Mix in Section 4 by requiring readers to read a 4.5-page Appendix. Without referring to the Appendix, it is really hard to understand the algorithm; for instance, what does tt stand for inside each equation? Why should the critics be separated for trust estimation and policy improvement? Furthermore, the notations, defined in the 4.5-page-long Appendix, are widely used across Section 4, which exacerbates the legibility of the paper. Since the manuscript still has a remaining page budget without violating the ICLR page limit. it cannot be attributed to the page limitation.
Questions for authors
  1. What is the takeaway of this paper? I do not think listing empirical observations can deliver significant insights to our community. How should we control the mixed sources of offline behavior for downstream value and policy learning?
  2. Is Trust-Mix only used for the experiments in Figures 2 and 3? If so, how do other offline RL algorithms show different behaviors when controlling the source of offline data?
1 comment
  1. Question - reply:

Thank you for this important question. We appreciate your perspective, as it also highlights an important direction for extending our research.

The primary takeaway of our work is that the usefulness of mixed-quality offline data depends not only on its quality or quantity, but also on the learning objective for which it is used. In particular, our empirical observations suggest that low-quality transitions may still provide useful information for value learning, even when directly imitating the corresponding actions is less beneficial for policy improvement. Therefore, we argue that data selection should be considered in relation to the roles of individual learning components rather than applying a uniform data-filtering strategy across the entire algorithm.

However, we acknowledge that determining how to optimally control mixed data sources across different environments and data regimes remains challenging. One possible direction is to introduce adaptive mechanisms or hyperparameters that dynamically regulate data utilization for individual learning components. Nevertheless, this would introduce additional architectural complexity and tuning requirements, particularly given the sensitivity of offline RL algorithms to hyperparameter choices.

Our current work focuses on investigating and empirically demonstrating the importance of role-dependent data utilization rather than establishing a universally optimal data allocation strategy. Developing more adaptive and theoretically grounded mechanisms for controlling data utilization is an important direction for our ongoing research.

We appreciate your feedback and recognize the need to communicate this central takeaway more explicitly in the manuscript.

  1. Question - reply:

Thank you for raising this question. We would like to clarify the purpose of the experiments presented in Figures 2 and 3.

In Figure 2, we evaluate the baseline offline RL algorithms using mixed-quality datasets in the conventional source-agnostic setting, where transitions are not explicitly differentiated based on their source labels. We compare their overall performance with Trust-Mix to examine how different approaches respond to variations in dataset composition.

However, we acknowledge that we did not conduct component-level data allocation experiments on the baseline algorithms, as we did for Trust-Mix. Therefore, our current experiments do not establish whether the same role-dependent behavior would emerge across other offline RL architectures.

Figure 3 provides a more controlled empirical investigation of how different learning components respond to data quality and coverage. Specifically, we examine the influence of random transitions on individual components and their contributions to overall policy performance. For this analysis, we explicitly use dataset source labels to isolate the effects of different data sources on each learning role.

These observations motivated our hypothesis that different learning components may benefit from different data utilization strategies. Rather than assuming that all available transitions should contribute equally to every learning objective, our framework explores the potential benefits of treating critic learning, Q-guided policy improvement, and behavior regularization differently.

We agree that extending these component-level investigations to other offline RL algorithms would help determine whether these observations generalize beyond Trust-Mix. We consider this an important direction for future work.

Thank you again for your constructive feedback.

Reviewer_e5sZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper study low-quality offline RL data effects on critic learning. It proposes method which uses all data for critic and Q-guided updates while selectively weighting imitation targets. Experiments cover three MuJoCo environments.

Critical strengths

The main strengths is that the method is using the full dataset low-quality random transitions still feed the critics. And only the behavioral-cloning and proximal terms are selectively down-weighted. This can be useful because it simplifies data filtration and useful exploration information in bad data isn't thrown away.

Critical weaknesses

The authors ask: how does the effect of additional data depend on what the learner uses that data for? To be honest, this sentence is unclear from the first read. And introduction is not leading to this question.

  1. The evaluation is poor. and the proposed Trust-Mix shows inconsistent performance, leading in only 3 of 12 mixed-data settings in Table 1 Removing its Q-gradient cap improves HalfCheetah-100k by +2003.3 (Table 14). All 800 expert-only HalfCheetah checkpoint evaluations are negative despite high-quality demonstrations. This unexplained failure weakens the evidence that the mixed-data improvements reflect effective data weighting rather than optimization artifacts.

  2. The weight that decides which recorded actions Trust-Mix imitates comes from critics trained on the mixed data, including a random part, which makes which method unclear. That makes it a relative ranking under a value function that may well be wrong> And in my understanding there is no proof that an action is actually good. A proposed method cant separate a good action the critic undervalues from a bad one the critic rightly rejects. And the proposed "trust" is just a soft heuristic that's only as good as the critic. The paper itself concedes the critic isn't reliable enough, and still proposed this method as a solution.

Questions for authors

See weaknesses above.

1 comment
  1. Question - reply:

Thank you for your detailed feedback and for highlighting these important concerns.

We acknowledge that Trust-Mix does not achieve state-of-the-art performance across all mixed-data budgets and environments. However, our results demonstrate promising performance across the evaluated environments, while the baseline methods also exhibit varying levels of performance depending on the environment and dataset composition.

We also agree with your observation regarding the ablation study. As you correctly pointed out, removing the Q-gradient cap leads to a substantial performance improvement in the HalfCheetah-100k setting. However, this modification does not consistently improve performance across all environments. This suggests that the effectiveness of individual components is highly dependent on the environment and their interactions with other components of the proposed framework.

We recognize that a more detailed justification and analysis of each architectural component would strengthen our work, particularly in explaining how these components contribute to stable and effective policy learning under different data-quality conditions. We consider a more systematic investigation of these interactions an important direction for future research.

Thank you again for your constructive comments and valuable suggestions.

  1. Question reply:

Thank you for highlighting this important concern.

We agree that the trust weights are derived from critics trained on mixed-quality datasets and are therefore subject to value-estimation errors. As you correctly pointed out, the critic may underestimate a good action or overestimate a poor one, particularly when dealing with out-of-distribution (OOD) actions. We acknowledge that our current approach does not provide theoretical guarantees regarding the reliability of these weights or their ability to accurately distinguish between good and bad actions.

However, we would like to clarify that the primary objective of our study is to investigate the role-dependent use of mixed-quality offline data, rather than to propose a definitive solution to critic estimation errors or guarantee optimal action selection. Trust-Mix serves as an experimental framework for exploring how data quality affects different components of offline RL, particularly critic learning and behavior-regularized policy updates.

We recognize that a more thorough theoretical and empirical justification of the proposed architectural components would strengthen our work. In particular, understanding how critics estimation errors influence the trust weights, how different components interact, and how these mechanisms could mitigate the effects of OOD actions remains an important direction for future research.

We appreciate your constructive feedback, which helps us identify the limitations of our current approach and areas requiring further investigation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.