- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies three uses of offline data: transitions for critic learning, states for Q-guided policy improvement, and recorded actions for behavioral supervision. Source-aware controls progressively route the same random-policy component into these operations. Trust-Mix implements one source-label-free configuration: it uses the full dataset for critic learning and Q-guided updates, selectively weights imitation targets using learned advantages and critic disagreement, and adds proximal regularization and Q/BC gradient capping. Experiments use HalfCheetah, Hopper, and Walker2d, four expert-transition budgets, and a fixed one-million-transition random component. The results show environment- and regime-dependent effects rather than uniform gains from random data or uniform superiority of Trust-Mix.
Critical strengths
-
The distinction between learning from a transition, improving a policy at its state, and imitating its recorded action is useful and operationalized through explicit routing controls. The paper appropriately acknowledges that selective imitation is not new and that broader critic/state access is not always beneficial. For example, the HalfCheetah and Walker2d contrasts in Figure 3 and Table 12 have different signs, which is more informative than a claim that low-quality data are uniformly helpful.
-
The experimental reporting is unusually explicit about scope and limitations. Main paired comparisons use ten learner seeds, a prespecified one-million-update endpoint, and paired bootstrap intervals. The paper reports absolute returns, adverse component effects, and full learning histories, and clearly distinguishes the CQL+DW-AW adaptation from a verified reproduction. It also identifies the simultaneous changes in C3-to-C4 and avoids presenting sequential contrasts as an additive decomposition.
Critical weaknesses
1. The expert-only baselines perform poorly, and the cause is unclear.
On HalfCheetah with 100k expert transitions, expert-only BC scores -149.6 (Table 18), and expert-only Trust-Mix scores -226.4 (Table 9). The recorded expert segments have a mean return of 16,284.92 (Table 7). Appendix C.6 also shows negative returns for all 800 saved expert-only checkpoint evaluations. The authors need to explain why training on these expert data produces such poor policies. The paper uses improvements over expert-only training to argue that adding random data is useful. Interpreting that improvement requires an explanation of the poor starting point. The authors should report how well BC predicts the expert actions and check observation preprocessing, action scaling, and compatibility between the dataset and the evaluation environment.
2. The key experiment changes several parts of training at once.
The C3-to-C4 comparison changes which actions the policy imitates, which states are used for policy regularization, and the 10k-step initialization procedure (Section 5.2 and Table 10). It can also change how often the learner sees expert examples. Under uniform sampling from the full mixture, expert transitions would account for only of samples at the 1k expert budget and at 100k. The result therefore leaves several possible explanations: imitating random actions, changing initialization, changing the regularization data, or sampling expert examples less often. To identify the effect of imitating random actions, the authors should vary their inclusion in the imitation loss while keeping initialization, regularization, and expert-data exposure fixed. They should also report the actual sampling proportions used in each comparison.
Questions for authors
-
What checks have been made on the expert-only HalfCheetah data-to-policy-to-environment interface, including action prediction, observation/action transformations, and evaluation compatibility? If available, please provide existing BC-fit or post-initialization diagnostics. The saved return histories alone do not distinguish these possibilities; explaining this discrepancy would materially affect my assessment.
-
Please specify the actual source-sampling proportions for each C1–C4 role, and whether any existing contrasts keep initialization and expert exposure fixed when adding random data to BC or proximal supervision. If such controls are unavailable, what conclusion about the behavioral branch remains supported beyond the joint BC/proximal/initialization intervention?
-
Is the severe expert-only weakness also present in the source-aware control backbone, and is it explained by the same mechanism as in Trust-Mix? Table 11 and Table 9 use different behavioral weighting and initialization rules. Please separate conclusions supported by the diagnostic backbone from those supported by the deployable source-label-free method.
2 comments
Thank you for your detailed response. While I appreciate the discussion, my main concerns remain unresolved. I therefore tend to keep my current score.
Thank you for highlighting this important concern.
As you correctly pointed out, the return histories alone are insufficient to identify the underlying causes of the poor expert-only performance in HalfCheetah. In our current study, we have conducted only a post-hoc analysis, presented in Table 24. However, we acknowledge that this analysis does not provide sufficient evidence to determine whether the observed performance degradation arises from behavioral cloning difficulties, observation preprocessing, action scaling, or potential incompatibilities between the dataset and the evaluation environment.
We have not conducted additional diagnostics specifically examining expert-action prediction accuracy or the data-to-policy-to-environment interface. Therefore, we cannot confidently attribute the negative returns to a particular mechanism.
We agree that a more systematic diagnostic investigation is necessary to understand this discrepancy and its implications for interpreting the benefits of mixed-quality datasets. We recognize this as an important limitation of our current experimental analysis and a priority for future investigation.
Thank you for raising this important methodological concern.
In our role-dependency experiments, we used fixed expert-data budgets ranging from 1k to 100k transitions, together with the random-data component. However, because we employed uniform sampling from the mixed dataset, we did not explicitly enforce a fixed proportion of expert transitions within each training minibatch. Consequently, particularly under smaller expert-data budgets, some minibatches may contain very few or no expert transitions.
Although the expert-data budgets were fixed for each experimental configuration, this sampling strategy does not guarantee equal expert-data exposure across learning components when random transitions are introduced.
For our component-level investigations, we aimed to maintain consistent experimental settings when examining how the inclusion of random data affects individual learning roles. However, we acknowledge that the C3-to-C4 comparison involves simultaneous changes to behavioral supervision, proximal regularization, and initialization. Therefore, this comparison cannot isolate the effect of including random actions in behavioral cloning independently of these other changes.
Our results consequently support the observation that changing how mixed-quality data are allocated across learning components can influence overall policy performance, but they do not establish that the observed differences arise exclusively from the inclusion or exclusion of random actions in the imitation objective.
We agree that a more controlled experiment, maintaining identical initialization, regularization, and expert-data exposure while modifying only the imitation targets, would provide stronger evidence for this specific effect.
Thank you for this question. We would like to clarify the distinction between the experimental configurations presented in Tables 9 and 11.
Table 9 reports the performance of Trust-Mix under the conventional source-label-free mixed-data setting, whereas Table 11 presents the source-aware, role-controlled experiments designed to investigate the effects of allocating different data sources to individual learning components.
In the source-aware experiments, dataset source information is explicitly used to control how expert and random transitions contribute to different learning roles. In contrast, the main Trust-Mix algorithm operates without source labels and relies on uniform sampling from the mixed dataset. As a result, particularly when expert transitions are scarce, individual training minibatches may contain no expert samples.
This distinction may contribute to the observed differences between the C4 configuration in Table 11 and the main Trust-Mix results in Table 9. However, we acknowledge that the two configurations also differ in behavioral weighting and initialization procedures. Therefore, we cannot attribute their performance differences solely to expert-data exposure. Furthermore, without additional diagnostics, we cannot establish whether the expert-only performance limitations in the source-aware backbone and Trust-Mix arise from the same underlying mechanism.
Our source-aware experiments are intended primarily to investigate the role-dependent effects of mixed-quality data under controlled data-routing configurations, whereas the source-label-free Trust-Mix experiments evaluate the proposed method under a more conventional mixed-data training setting. We recognize that these two sets of experiments support different conclusions and that this distinction should be communicated more clearly in the manuscript.
Thank you again. We appreciate the opportunity to clarify these aspects of our experimental methodology and acknowledge the limitations of our current analysis.