- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CAFRL, an online reinforcement learning method that builds on FLAC by incorporating action-to-reward causal effects, estimated from replay data, into the reference diffusion process of a flow-based policy, yielding an anisotropic kinetic-energy regularization that allocates more exploration to task-relevant action dimensions.
Critical strengths
Using causal effect information for effective exploration in reinforcement learning is sensible, especially as generative policies are increasingly adopted as policy parameterizations, and this work provides a useful direction toward utilizing such policies within an efficient reinforcement learning framework. Incorporating causal relevance into the reference process, rather than as a heuristic action weighting, is a principled choice that separates the roles of the critic and the exploration geometry while remaining a lightweight modification of FLAC. The empirical protocol is also transparent, with shared interaction budgets, five seeds, late-training averages, and disclosed configuration differences, and CAFRL achieves the highest mean return on all eight HumanoidBench tasks.
Critical weaknesses
1. Presentation of the causal effect profile: The building block of CAFRL is the causal effect profile , however, how it is obtained and how it affects exploration are not clearly explained in the main text. Given that Appendix B contains these details, I suggest referring to it at the start of Section 4 and elaborating on what this profile looks like, for completeness of the presentation.
2. Lack of analysis supporting the main claim: The experiments report only returns and a sensitivity study of on a single task. However, the central claim of the paper is that causal relevance redistributes exploration toward task-relevant action dimensions, which the returns alone cannot verify. I would expect analyses such as:
- the obtained profile for each task, mapped to body parts (e.g., whether Walk emphasizes lower-body dimensions, while Crawl, which requires whole-body coordination, yields a more distributed profile)
- whether per-dimension exploration (e.g., action variance) actually changes as prescribed by
3. Limited evaluation scope: All experiments are conducted on HumanoidBench with a single robot (H1Hand). Evaluation on additional continuous-control benchmarks would strengthen the claim of general applicability.
Overall, I found the idea promising, and I believe that proper analysis and evaluation on more embodiments would strengthen the paper. In particular, I would be happy to raise my score if the analysis of the causal profile and the resulting exploration behavior is provided during the rebuttal. Additional embodiments may not be feasible within the rebuttal period, so I do not strictly require them but highly recommend including them in a future version.
Questions for authors
See Weaknesses please
7 comments
## Response to Weakness 2: Analysis of Causal Profiles and Their Effects on Exploration
We agree that returns alone cannot establish whether causal relevance redistributes exploration as intended. We therefore provide body-part relevance profiles across tasks and analyze their influence on the reference diffusion variance during training.
Task-specific causal profiles. We map the 61 H1Hand action dimensions to five body groups using the actuator ordering: legs/feet, torso, shoulders/elbows, wrists, and fingers. Within each seed, we average relevance across the action dimensions belonging to each group, then report the mean and sample standard deviation across five training seeds. This per-dimension averaging avoids favoring groups simply because they contain more actuators.
The following table summarizes the profiles at 1M environment steps. Numbers in parentheses indicate the number of action dimensions in each group.
| Task | Legs/feet (10) | Torso (1) | Shoulders/elbows (8) | Wrists (6) | Fingers (36) |
|---|---|---|---|---|---|
| Walk | 1.334 ± 0.175 | 0.856 ± 0.816 | 0.871 ± 0.237 | 1.092 ± 0.161 | 0.926 ± 0.058 |
| Stair | 1.355 ± 0.249 | 3.273 ± 1.654 | 1.288 ± 0.369 | 0.763 ± 0.250 | 0.813 ± 0.112 |
| Balance-hard | 1.749 ± 0.383 | 1.743 ± 1.302 | 0.898 ± 0.108 | 0.843 ± 0.202 | 0.820 ± 0.092 |
| Crawl | 1.189 ± 0.216 | 0.930 ± 0.424 | 1.277 ± 0.328 | 1.081 ± 0.102 | 0.875 ± 0.107 |
| Slide | 1.494 ± 0.163 | 0.556 ± 0.796 | 1.143 ± 0.251 | 0.880 ± 0.378 | 0.864 ± 0.082 |
| Pole | 1.124 ± 0.211 | 0.712 ± 0.368 | 1.220 ± 0.334 | 0.867 ± 0.198 | 0.947 ± 0.125 |
Task-specific causal relevance (Walk vs. Crawl). Figure 11 illustrates the evolution of causal relevance and reference diffusion variance across body regions. The following results summarize the task-specific profiles and their changes during training.
| Body region | Walk (1M) | Crawl (1M) |
|---|---|---|
| Legs/feet | 1.334 | 1.189 |
| Shoulders/elbows | 0.871 | 1.277 |
Walk emphasizes lower-body control, whereas Crawl exhibits a more distributed profile across lower- and upper-body dimensions, consistent with its whole-body coordination requirements.
The profiles also evolve during training:
| Task / Region | Relevance | Reference variance |
|---|---|---|
| Walk / Torso | 2.048 (200k) → 0.856 (1M) | 0.02880 → 0.01597 |
| Crawl / Torso | 1.269 (200k) → 0.447 (600k) → 0.929 (1M) | 0.02880 → 0.01174 → 0.02215 |
Crawl's torso relevance and variance reach their minimum at 600k steps before recovering, while Walk exhibits an overall decline. These observations demonstrate that estimated causal relevance dynamically shapes the reference exploration geometry, rather than directly determining the variance of final executed actions.
We have incorporated this analysis into Appendix D.6 (Task-Specific Causal Relevance and Exploration Geometry) of the revised manuscript.
How exploration changes with the profile. Using the recorded causal-relevance profiles from Walk and Crawl, we reconstruct the bounded precision mapping and the corresponding reference diffusion variance rate, . An integration step of size injects noise variance . These runs use , , and . The reconstructed precision and covariance moments match 11,935 training records aligned by recorded step, with a maximum absolute error below .
For example:
| Task/body part | Training step | Mean relevance | Mean reference variance rate |
|---|---|---|---|
| Walk / torso | 200k | 2.048 | 0.02880 |
| Walk / torso | 1M | 0.856 | 0.01597 |
| Crawl / torso | 600k | 0.447 | 0.01174 |
As shown in Figure 13, reference diffusion variance adapts to estimated causal relevance throughout training. Walk's torso relevance and variance decrease together, while Crawl's torso values reach a minimum at 600k steps before recovering. The five-seed trajectories of relevance, precision, and variance confirm that causal estimates dynamically shape the reference exploration geometry, accounting for normalization and clipping. These results characterize reference diffusion rather than final action variance, which also depends on the learned drift and action transformation.
## Response to Weakness 3: Evaluation Beyond HumanoidBench
Thank you for suggesting broader evaluation. We have extended the comparison with FLAC to three MetaWorld tasks (Reach, Push, and Pick-place) and three dm_control tasks (Dog Walk, Dog Trot, and Humanoid Stand), covering robotic manipulation, locomotion, and posture control beyond HumanoidBench.
Both methods are evaluated using five independent training seeds per task, with results reported as mean ± standard deviation. MetaWorld performance is measured by success rate, and dm_control performance by episodic return.
| Benchmark | Task | Metric ↑ | FLAC | CAFRL |
|---|---|---|---|---|
| MetaWorld | Reach | Success rate (%) | 94.4 ± 2.7 | 97.0 ± 1.6 |
| MetaWorld | Push | Success rate (%) | 82.2 ± 3.8 | 96.0 ± 2.9 |
| MetaWorld | Pick-place | Success rate (%) | 74.8 ± 4.6 | 95.8 ± 3.5 |
| dm_control | Dog Walk | Return | 774.0 ± 24.4 | 821.6 ± 20.1 |
| dm_control | Dog Trot | Return | 663.6 ± 33.1 | 715.4 ± 24.5 |
| dm_control | Humanoid Stand | Return | 918.0 ± 12.4 | 938.2 ± 11.1 |
CAFRL achieves higher mean performance than FLAC across all six additional tasks. These results provide further evidence of CAFRL's applicability across the evaluated continuous-control tasks.
The complete experimental results and analysis are included in Appendix E, Table 12 of the revised manuscript. We appreciate the reviewer's suggestion, which helped us broaden the empirical evaluation.
## Response to Weakness 1: Presentation of the Causal Effect Profile
Thank you for this helpful suggestion. We agree that the construction and role of the causal-effect profile should be clearer when first introduced. We have clarified its estimation procedure and role in shaping exploration in Section 4.1, with further details provided in Appendix B.1.
Specifically, the profile contains one estimated relevance weight per action dimension. We apply DirectLiNGAM to state-adjusted replay data to estimate action-to-immediate-reward effects, then normalize, clip, and smooth their magnitudes into positive relevance weights. The identification assumptions are discussed in Appendix B.2.
These weights determine directional precision as follows, with the normalization factor chosen to obtain unit mean precision:
pi=clip(bCcˉi−η,pmin,pmax).
The resulting diagonal precision matrix is
PC=diag(p1,…,pd).
This matrix determines both the reference diffusion covariance rate,
σ2PC−1,
and the directional kinetic cost,
21uθ⊤PCuθ.
Within the unclipped range, higher relevance corresponds to lower precision, which increases reference variance and reduces the kinetic penalty for a given drift component. The critic continues to provide value-based guidance.
We have also included a detailed explanation and an estimation-to-exploration pipeline diagram in Appendix D.1, Figure 4, illustrating how the causal-effect profile influences flow-policy sampling and regularization.