acceptodds
Under review as a conference paper at ICLR 2027

CAFRL: Causality-Aware Flow Based Policy For Online Reinforcement Learning

Abstract

Flow-based policies enhance maximum-entropy reinforcement learning by representing complex and multimodal behaviors through learned transport dynamics. However, entropy evaluation of implicit flow policies is generally difficult, motivating path-space regularization as an alternative to explicit density-based constraints. Existing approaches typically adopt isotropic exploration geometries, assigning identical kinetic costs to different action dimensions despite their heterogeneous relevance to tasks.We introduce CAFRL, a causality-aware regularization framework that incorporates action-to-reward effects into the exploration geometry of flow policies. CAFRL derives action relevance from interaction data through causal analysis and maps the resulting effects into a reference process, inducing anisotropic path kinetic regularization. This enables task-relevant directions to deviate from the reference process with adaptive path costs while maintaining a consistent regularization scale, thereby inducing a causal anisotropic exploration geometry. Across eight HumanoidBench tasks with five random seeds for each method, CAFRL achieves the highest average return compared with FLAC, ACE, and SimbaV2 after one million environment steps. CAFRL improves over the strongest baseline on each task by a median margin of 80.3%. These results support the empirical performance of the CAFRL configuration evaluated.

Primary area:
reinforcement learning
Keywords:
Flow-based Reinforcement Learning, Causal Reinforcement Learning, Schrödinger Bridge

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_maVROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CAFRL, an online reinforcement learning method that builds on FLAC by incorporating action-to-reward causal effects, estimated from replay data, into the reference diffusion process of a flow-based policy, yielding an anisotropic kinetic-energy regularization that allocates more exploration to task-relevant action dimensions.

Critical strengths

Using causal effect information for effective exploration in reinforcement learning is sensible, especially as generative policies are increasingly adopted as policy parameterizations, and this work provides a useful direction toward utilizing such policies within an efficient reinforcement learning framework. Incorporating causal relevance into the reference process, rather than as a heuristic action weighting, is a principled choice that separates the roles of the critic and the exploration geometry while remaining a lightweight modification of FLAC. The empirical protocol is also transparent, with shared interaction budgets, five seeds, late-training averages, and disclosed configuration differences, and CAFRL achieves the highest mean return on all eight HumanoidBench tasks.

Critical weaknesses

1. Presentation of the causal effect profile: The building block of CAFRL is the causal effect profile cc, however, how it is obtained and how it affects exploration are not clearly explained in the main text. Given that Appendix B contains these details, I suggest referring to it at the start of Section 4 and elaborating on what this profile cc looks like, for completeness of the presentation.

2. Lack of analysis supporting the main claim: The experiments report only returns and a sensitivity study of η\eta on a single task. However, the central claim of the paper is that causal relevance redistributes exploration toward task-relevant action dimensions, which the returns alone cannot verify. I would expect analyses such as:

  • the obtained profile cc for each task, mapped to body parts (e.g., whether Walk emphasizes lower-body dimensions, while Crawl, which requires whole-body coordination, yields a more distributed profile)
  • whether per-dimension exploration (e.g., action variance) actually changes as prescribed by cc

3. Limited evaluation scope: All experiments are conducted on HumanoidBench with a single robot (H1Hand). Evaluation on additional continuous-control benchmarks would strengthen the claim of general applicability.

Overall, I found the idea promising, and I believe that proper analysis and evaluation on more embodiments would strengthen the paper. In particular, I would be happy to raise my score if the analysis of the causal profile and the resulting exploration behavior is provided during the rebuttal. Additional embodiments may not be feasible within the rebuttal period, so I do not strictly require them but highly recommend including them in a future version.

Questions for authors

See Weaknesses please

7 comments
AuthorsOpenReview ↗

## Response to Weakness 1: Presentation of the Causal Effect Profile

Thank you for this helpful suggestion. We agree that the construction and role of the causal-effect profile should be clearer when first introduced. We have clarified its estimation procedure and role in shaping exploration in Section 4.1, with further details provided in Appendix B.1.

Specifically, the profile contains one estimated relevance weight per action dimension. We apply DirectLiNGAM to state-adjusted replay data to estimate action-to-immediate-reward effects, then normalize, clip, and smooth their magnitudes into positive relevance weights. The identification assumptions are discussed in Appendix B.2.

These weights determine directional precision as follows, with the normalization factor chosen to obtain unit mean precision:

pi=clip⁡(bCcˉi−η,pmin⁡,pmax⁡).p_{i} = \operatorname{clip}(b_{C} \bar{c}_{i}^{-\eta}, p_{\min}, p_{\max}).

The resulting diagonal precision matrix is

PC=diag⁡(p1,…,pd).P_C = \operatorname{diag}(p_1,\ldots,p_d).

This matrix determines both the reference diffusion covariance rate,

σ2PC−1,\sigma^2 P_C^{-1},

and the directional kinetic cost,

12uθ⊤PCuθ.\frac{1}{2} u_\theta^\top P_C u_\theta.

Within the unclipped range, higher relevance corresponds to lower precision, which increases reference variance and reduces the kinetic penalty for a given drift component. The critic continues to provide value-based guidance.

We have also included a detailed explanation and an estimation-to-exploration pipeline diagram in Appendix D.1, Figure 4, illustrating how the causal-effect profile influences flow-policy sampling and regularization.

AuthorsOpenReview ↗

## Response to Weakness 2: Analysis of Causal Profiles and Their Effects on Exploration

We agree that returns alone cannot establish whether causal relevance redistributes exploration as intended. We therefore provide body-part relevance profiles across tasks and analyze their influence on the reference diffusion variance during training.

Task-specific causal profiles. We map the 61 H1Hand action dimensions to five body groups using the actuator ordering: legs/feet, torso, shoulders/elbows, wrists, and fingers. Within each seed, we average relevance across the action dimensions belonging to each group, then report the mean and sample standard deviation across five training seeds. This per-dimension averaging avoids favoring groups simply because they contain more actuators.

The following table summarizes the profiles at 1M environment steps. Numbers in parentheses indicate the number of action dimensions in each group.

TaskLegs/feet (10)Torso (1)Shoulders/elbows (8)Wrists (6)Fingers (36)
Walk1.334 ± 0.1750.856 ± 0.8160.871 ± 0.2371.092 ± 0.1610.926 ± 0.058
Stair1.355 ± 0.2493.273 ± 1.6541.288 ± 0.3690.763 ± 0.2500.813 ± 0.112
Balance-hard1.749 ± 0.3831.743 ± 1.3020.898 ± 0.1080.843 ± 0.2020.820 ± 0.092
Crawl1.189 ± 0.2160.930 ± 0.4241.277 ± 0.3281.081 ± 0.1020.875 ± 0.107
Slide1.494 ± 0.1630.556 ± 0.7961.143 ± 0.2510.880 ± 0.3780.864 ± 0.082
Pole1.124 ± 0.2110.712 ± 0.3681.220 ± 0.3340.867 ± 0.1980.947 ± 0.125

Task-specific causal relevance (Walk vs. Crawl). Figure 11 illustrates the evolution of causal relevance and reference diffusion variance across body regions. The following results summarize the task-specific profiles and their changes during training.

Body regionWalk (1M)Crawl (1M)
Legs/feet1.3341.189
Shoulders/elbows0.8711.277

Walk emphasizes lower-body control, whereas Crawl exhibits a more distributed profile across lower- and upper-body dimensions, consistent with its whole-body coordination requirements.

The profiles also evolve during training:

Task / RegionRelevanceReference variance
Walk / Torso2.048 (200k) → 0.856 (1M)0.02880 → 0.01597
Crawl / Torso1.269 (200k) → 0.447 (600k) → 0.929 (1M)0.02880 → 0.01174 → 0.02215

Crawl's torso relevance and variance reach their minimum at 600k steps before recovering, while Walk exhibits an overall decline. These observations demonstrate that estimated causal relevance dynamically shapes the reference exploration geometry, rather than directly determining the variance of final executed actions.

We have incorporated this analysis into Appendix D.6 (Task-Specific Causal Relevance and Exploration Geometry) of the revised manuscript.

How exploration changes with the profile. Using the recorded causal-relevance profiles from Walk and Crawl, we reconstruct the bounded precision mapping and the corresponding reference diffusion variance rate, vi=σ2/piv_i = \sigma^2 / p_i. An integration step of size hh injects noise variance hvih v_i. These runs use η=1.25\eta = 1.25, pi∈[0.5,2]p_i \in [0.5, 2], and σ=0.12\sigma = 0.12. The reconstructed precision and covariance moments match 11,935 training records aligned by recorded step, with a maximum absolute error below 4×10−74 \times 10^{-7}.

For example:

Task/body partTraining stepMean relevanceMean reference variance rate
Walk / torso200k2.0480.02880
Walk / torso1M0.8560.01597
Crawl / torso600k0.4470.01174

As shown in Figure 13, reference diffusion variance adapts to estimated causal relevance throughout training. Walk's torso relevance and variance decrease together, while Crawl's torso values reach a minimum at 600k steps before recovering. The five-seed trajectories of relevance, precision, and variance confirm that causal estimates dynamically shape the reference exploration geometry, accounting for normalization and clipping. These results characterize reference diffusion rather than final action variance, which also depends on the learned drift and action transformation.

AuthorsOpenReview ↗

## Response to Weakness 3: Evaluation Beyond HumanoidBench

Thank you for suggesting broader evaluation. We have extended the comparison with FLAC to three MetaWorld tasks (Reach, Push, and Pick-place) and three dm_control tasks (Dog Walk, Dog Trot, and Humanoid Stand), covering robotic manipulation, locomotion, and posture control beyond HumanoidBench.

Both methods are evaluated using five independent training seeds per task, with results reported as mean ± standard deviation. MetaWorld performance is measured by success rate, and dm_control performance by episodic return.

BenchmarkTaskMetric ↑FLACCAFRL
MetaWorldReachSuccess rate (%)94.4 ± 2.797.0 ± 1.6
MetaWorldPushSuccess rate (%)82.2 ± 3.896.0 ± 2.9
MetaWorldPick-placeSuccess rate (%)74.8 ± 4.695.8 ± 3.5
dm_controlDog WalkReturn774.0 ± 24.4821.6 ± 20.1
dm_controlDog TrotReturn663.6 ± 33.1715.4 ± 24.5
dm_controlHumanoid StandReturn918.0 ± 12.4938.2 ± 11.1

CAFRL achieves higher mean performance than FLAC across all six additional tasks. These results provide further evidence of CAFRL's applicability across the evaluated continuous-control tasks.

The complete experimental results and analysis are included in Appendix E, Table 12 of the revised manuscript. We appreciate the reviewer's suggestion, which helped us broaden the empirical evaluation.

Reviewer_azp5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAFRL uses estimated action–reward causal effects to shape the exploration geometry of a flow policy. It assigns dimension-dependent diffusion scales and corresponding kinetic-energy costs. Experiments report substantial improvements over FLAC, ACE, and SimbaV2 on eight HumanoidBench tasks.

Critical strengths
  • The motivation of allocating exploration differently across action dimensions is reasonable.
  • The formulation consistently incorporates anisotropy into both diffusion covariance and path regularization.
  • The reported improvements are promising.
Critical weaknesses
  1. The main comparison does not isolate the proposed contribution. Appendix C reports substantially different kinetic-energy targets for FLAC and CAFRL. Consequently, the gains cannot be attributed solely to causal geometry. The matched isotropic ablation on Walk is useful but insufficient to explain the broader results.
  2. The benefit of causal information is not established. The experiments lack comparisons with shuffled causal weights, random anisotropy, or non-causal action-importance estimates. The current evidence supports anisotropic exploration more directly than causality-aware exploration.
  3. Exploration scale is not fully controlled. Normalizing mean precision does not preserve total diffusion variance. Anisotropy may therefore change both the allocation and overall magnitude of exploration.
Questions for authors
  • Does CAFRL retain its advantage over FLAC with matched energy targets and tuning budgets?
  • How does it compare with shuffled weights and simple non-causal relevance estimates?
  • Do the gains persist when total diffusion variance is held fixed?
6 comments
AuthorsOpenReview ↗

Author Response 1/3 | W1 & Q1: Matched energy targets across all eight tasks

We thank the reviewer for identifying the controls needed to assess the contribution of causal geometry. We address the energy-target comparison, the information used for directional allocation, and the overall diffusion scale in three responses.

We agree that the original comparison with different kinetic-energy targets does not, by itself, isolate the contribution of the proposed geometry. We therefore extended the matched-target comparison from Walk to all eight HumanoidBench tasks.

1. Matching the target coefficient preserves the advantage across the full benchmark.

For each task, we match FLAC's kinetic-energy-target coefficient to CAFRL's. The resulting comparison is:

TaskCAFRLFLAC
Walk699.8 ± 152.0204.5 ± 46.7
Run157.0 ± 59.379.1 ± 19.0
Hurdle75.2 ± 6.944.9 ± 11.1
Stair218.3 ± 80.270.4 ± 5.8
Balance-hard96.9 ± 8.678.6 ± 1.5
Crawl929.8 ± 17.0760.8 ± 63.7
Slide185.9 ± 51.9127.1 ± 23.4
Pole709.7 ± 101.6160.0 ± 20.6

Entries report mean ± sample standard deviation across five training seeds. Each seed contributes its average return over the final ten evaluations (910k–1M environment steps). Both methods use a common 1M-step training horizon. The corresponding learning curves are provided in Appendix D.2, Figure 5 of the revised manuscript.

2. The controlled evidence extends beyond the original Walk ablation.

CAFRL achieves higher mean returns on all eight tasks, covering locomotion, balance, and whole-body coordination. The advantage therefore persists when the energy-target coefficient is matched, providing broader evidence that the improvement is not explained solely by the different target settings in the original comparison.

The matched quantity is the target coefficient. The isotropic and anisotropic kinetic metrics remain different because the geometry is the proposed change being evaluated. Matching the coefficient does not imply identical realized kinetic energy.

3. CAFRL retains its advantage under matched energy-target coefficients.

Although FLAC and CAFRL use the same kinetic-energy-target coefficients in this comparison, their realized kinetic energies may still fluctuate during training due to adaptive optimization and differences in exploration geometry. Such fluctuations are inherent to the learning process and do not invalidate the matched-target comparison. Importantly, CAFRL consistently achieves higher mean returns across all eight tasks under matched target coefficients, demonstrating that its performance advantage cannot be explained solely by differences in the prescribed energy targets.

Both methods follow the same 1M-step training and five-seed evaluation protocol. While identical target coefficients do not imply identical realized kinetic energies or hyperparameter-search effort, the results provide strong empirical evidence that CAFRL's advantage persists after controlling for the energy-target coefficient.

AuthorsOpenReview ↗

Author Response 2/3 | W2 & Q2: Informative allocation versus random anisotropy

1. The estimated relevance profile improves mean late-training return by 51.6% over random allocation.

We conducted a controlled experiment on Walk to test whether relevance-informed allocation improves learning beyond random anisotropy. We compared causal allocation with random refresh under the reported fixed-total-diffusion protocol.

Random refresh also assigns unequal exploration scales across action dimensions, but does not use estimated action–reward relevance to determine the allocation. The results are:

TaskCausal allocationRandom refreshMean improvement
Walk608.8 ± 282.3401.5 ± 37.2+51.6%

Entries report mean ± sample standard deviation across three training seeds. For each seed, we average the final ten evaluations at 910k–1M environment steps, then compute the mean and standard deviation across the three seed-level averages. All three seeds are retained.

Causal allocation achieves a 51.6% higher mean late-training return, supporting the value of the estimated relevance profile over the tested random-allocation baseline. Fig. S2this anonymous link shows the learning curves and individual-seed summaries. Response 3 reports the complementary normalized learning-curve AUC and explains the diffusion-scale control.

2. The relevance profile shapes both reference sampling and path regularization.

CAFRL maps causal relevance to directional precision through pi∝cˉi−ηp_i \propto \bar{c}_i^{-\eta}, followed by normalization and clipping. The resulting precision determines both the reference covariance and kinetic-energy regularization:

PC=diag⁡(pi),VC=PC−1.P_C = \operatorname{diag}(p_i), \qquad V_C = P_C^{-1}.

The corresponding kinetic cost is

EC=12E[∫01uθ⊤PCuθ,dτ].\mathcal{E}_{C} = \frac{1}{2}\mathbb{E}\left[\int_{0}^{1} u_{\theta}^{\top} P_{C} u_{\theta}\\,\mathrm{d}\tau\right].

Within the unclipped range, higher causal relevance induces lower directional precision, greater reference diffusion variance, and a lower kinetic cost along the corresponding action dimension. The critic provides value guidance, while causal relevance shapes the exploration geometry and directional regularization. The random-allocation control further supports the benefit of this informed allocation.

3. Shuffled weights and non-causal relevance estimates test additional aspects of causal specificity.

Shuffling preserves the estimated weight values while changing their correspondence to action dimensions. A non-causal relevance estimator instead tests whether another informative allocation rule can achieve comparable performance. These are distinct from randomly generating an anisotropic allocation.

The reported experiments provide a direct comparison with random refresh, but do not provide head-to-head results against shuffled weights or non-causal relevance estimates. Accordingly, the supported empirical claim is that the estimated relevance profile improves performance over the tested random anisotropy.

AuthorsOpenReview ↗

Author Response 3/3 | W3 & Q3: Gains under fixed total reference diffusion

1. Mean-precision normalization does not guarantee constant total diffusion variance.

We agree with the reviewer. CAFRL normalizes and clips causal relevance before mapping it to directional precision. Although the mean precision is normalized to one, this does not guarantee constant total diffusion variance.

The reference diffusion covariance rate is

P_C=\operatorname{diag}(p_1,\ldots,p_{d_A}).$$ Its total variance satisfies $$\operatorname{tr}(D_C) =\sigma^2\sum_{i=1}^{d_A}\frac{1}{p_i}.$$ Since $p_i\in[p_{\min},p_{\max}]$, we obtain $$\frac{d_A\sigma^2}{p_{\max}} \leq \operatorname{tr}(D_C) \leq \frac{d_A\sigma^2}{p_{\min}}.$$ Moreover, under unit mean precision, $\sum_i p_i^{-1}\geq d_A$, with equality only for isotropic precision. Thus, **normalization and clipping bound the total diffusion variance but do not keep it constant**. **2. Fixed-total-diffusion experiment and results.** To isolate the effect of directional allocation from the overall diffusion scale, we introduce a **fixed total diffusion-variance budget of $B=80$** across all $d_A=61$ action dimensions. For positive directional weights $u_i$, we normalize the injected diffusion-variance rates as $$q_i=B\frac{u_i}{\sum_{j=1}^{d_A}u_j}, \qquad \sum_{i=1}^{d_A}q_i=B=80.$$ Causal allocation uses $u_i=1/p_i$, where $p_i$ is derived from estimated causal relevance. Random allocation instead samples independent positive directional weights. Both methods therefore have **identical total injected diffusion variance**, differing in how that variance is distributed across action dimensions. We evaluate both allocations on Walk using three independent seeds under the same 1M-step training budget. | Walk metric | Causal allocation | Random allocation | Improvement | |---|---:|---:|---:| | Late-training return | **608.8 ± 282.3** | 401.5 ± 37.2 | **+51.6%** | | Normalized learning-curve AUC | **228.4 ± 77.7** | 143.4 ± 13.1 | **+59.3%** | Results are mean ± sample standard deviation across three seeds. Late-training return averages the final ten evaluations (910k–1M steps), while normalized AUC summarizes performance over the recorded training horizon. The learning curves and late-training comparisons are shown in [Appendix D.4, Figure 7 and Table 7](https://openreview.net/pdf?id=anPjp7LuMN#page=21). **3. Performance gains persist under fixed total diffusion variance.** **CAFRL retains a substantial performance advantage even when the total injected diffusion variance is held constant.** Specifically, causal allocation improves mean late-training return by **51.6%** and normalized AUC by **59.3%** over random anisotropic allocation. These results demonstrate that the improvement cannot be attributed solely to increased total diffusion variance. Instead, **the estimated causal relevance provides useful directional information for allocating exploration**, improving learning efficiency under the same diffusion budget. This control directly supports the effectiveness of causal-guided exploration geometry.
Reviewer_Yo33OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CAFRL, which incorporates causal information into a flow-based reinforcement learning framework to improve exploration by accounting for the varying importance of action dimensions. Based on the SAC framework, CAFRL estimates the causal effects of action dimensions on rewards using DirectLiNGAM, and then uses a generalized Schrodinger Bridge formulation to incorporate the causal information into a reference diffusion process and a corresponding kinetic energy regularizer. The paper provides theoretical analyses of the proposed formulation and conducts experiments showing that CAFRL outperforms various baselines.

Critical strengths
  1. Incorporating causal information into the exploration geometry of flow-based policies is intuitive and well-motivated, providing a novel extension to existing flow-based reinforcement learning methods.

  2. The method is well-grounded in a principled theoretical framework. The connection between causality-aware reference diffusion and kinetic-energy regularization is well-established.

  3. The experiments cover eight challenging HumanoidBench tasks and compare against strong baselines. The results and ablation study provide convincing evidence of the method's effectiveness.

Critical weaknesses
  1. The causal structural model and its underlying assumptions could be better explained. The practical validity of these assumptions and the stability of periodically updating the causal matrix also deserve further discussion.

  2. Some learning curves do not appear to have fully converged after 1M environment steps. Additional details on the choice and sensitivity of key hyperparameters would also be helpful.

Questions for authors
  1. (Line 131) I would appreciate a bit more background on the causal structural model. I am not very familiar with this literature, and I found the current presentation somewhat difficult to follow, particularly the discussion of the reward-generating mechanism. A more intuitive explanation would help improve clarity.
  2. (Line 144) How restrictive are the causal identification assumptions? It would be helpful to discuss how likely these assumptions are to hold in practical reinforcement learning settings and how violations might affect the proposed method.
  3. (Line 350) I wonder whether 1M environment steps are sufficient for a fair comparison. Some learning curves do not appear to have fully converged by the end of training. Would longer training change the relative performance of the methods?
  4. (Section 4) From my understanding, the causal matrix is periodically updated alongside the actor-critic training. How stable is this alternating optimization process? How sensitive is the performance to the causal update frequency, and how difficult is it to tune this frequency in practice?
  5. (Section 4.2) Is λ\lambda a hyperparameter that needs to be tuned? It seems to also control the trade-off between exploration and exploitation. Is it automatically tuned, as in the original SAC? Also, could the authors clarify the different roles of λ\lambda and η\eta in controlling exploration?
  6. (Line 428) The results suggest that a larger η\eta can improve exploration along causally relevant directions and accelerate learning. Would it be beneficial to start with a larger η\eta and gradually decrease it during training? Such a schedule might encourage more targeted exploration early on while allowing the policy to refine its behavior in later stages.
  7. (Section 5.2) The current ablation study mainly examines the strength of causal anisotropy, but it remains unclear whether the improvements stem from meaningful causal information or anisotropic exploration alone. An additional ablation with randomly assigned weights would help clarify this.
8 comments
AuthorsOpenReview ↗

Author Response 1/4 | W1 & Q1–Q2: Structural model and identification assumptions

We thank the reviewer for the constructive review. We address both weaknesses and all seven questions in four responses, with the experimental results included directly in the relevant responses.

W1 / Q1–Q2: Clarify the structural causal model, its reward mechanism, and the practical validity of its identification assumptions.

  1. Q1 — Structural model and reward mechanism: Thank you for this suggestion. We have added a new paragraph, Causal Information for Reinforcement Learning, in Section 2 (Related Work) to provide background on causal modeling in reinforcement learning. We have also expanded Appendix D.1 to clarify the causal structural model, the reward-generating mechanism, and the causal-effect estimation procedure.

    Rt=fR(St,At,ϵtR)R_t=f_R(S_t,A_t,\epsilon_t^R) describes the environment's existing reward mechanism. We ask which action components affect reward after accounting for the observed state. DirectLiNGAM is applied to state-adjusted replay data, and the estimated effect magnitudes form relevance weights for reference diffusion and directional path costs. The critic supplies long-term value guidance. Figure 4 (Appendix D.1) summarizes this procedure.

  2. Q2 — Identification assumptions and their practical implications: Thank you for highlighting the need to explain these assumptions and their consequences. We have clarified the structural assumptions in Section 3.2 and expanded their practical discussion in Appendix B.2. A brief explanation follows.

    The causal interpretation assumes causal Markovness and no unobserved confounding; faithfulness is a standard assumption in constraint-based discovery but is not separately required by DirectLiNGAM. DirectLiNGAM requires a linear acyclic model with independent non-Gaussian disturbances. Observed-state adjustment helps account for measured common causes, while nonlinear contacts, omitted variables, or dependent disturbances can bias the estimated effects and misallocate exploration. Periodic estimation follows changes in the replay distribution, and normalization and bounded precision limit extreme sampling configurations. These mechanisms address adaptation and numerical scale; the estimates remain approximate relevance signals under the stated identification assumptions.

AuthorsOpenReview ↗

Author Response 2/4 | W2 & Q3: Training horizon and sample efficiency

W2 / Q3: Are 1M environment steps sufficient for a fair comparison, and could longer training change the relative performance?

We thank the reviewer for raising this important question. Our experiments are designed to evaluate sample efficiency under a common interaction budget, rather than establish convergence or asymptotic superiority.

1. Fair comparison under a shared 1M-step budget.

All four methods are trained for 1M environment steps with five independent seeds, using the same evaluation frequency and number of evaluation episodes. Performance is measured by averaging the final ten evaluations (910k–1M steps) within each seed and reporting the mean ± sample standard deviation across seeds.

As shown in Table 1 and Figure 2, CAFRL achieves the highest mean return on all eight HumanoidBench tasks, with a median improvement of 80.3% over the strongest baseline. These results demonstrate CAFRL's advantage in learning efficiency under the evaluated interaction budget.

2. Could longer training change the relative performance?

Yes. A shared 1M-step budget ensures comparability in environment interactions but does not imply that all methods have converged. Longer training may improve baseline performance and change the relative ranking.

To provide additional context, we compare CAFRL with the published longer-horizon results of DreamerV3 (10M steps), TD-MPC2 (2M steps), and SAC (10M steps) in Appendix D.3, Table 6. CAFRL achieves higher reported mean returns on Stair, Balance-hard, and Pole with only 1M steps, while DreamerV3 performs better on several other tasks with longer training. Because these results use different interaction budgets, they are contextual rather than directly controlled comparisons.

Therefore, our results indicate that causal-guided action sampling improves exploration and sample efficiency within the evaluated 1M-step interaction budget. However, the relationship between the learned terminal action distribution and the environment's underlying causal structure under longer training horizons remains insufficiently explored. We regard this as an important direction for future research.

AuthorsOpenReview ↗

Author Response 3/4 | W1–W2 & Q4–Q5: Update stability and hyperparameter roles

W1–W2 / Q4–Q5: Stability of the periodically updated causal geometry, sensitivity and tuning of its update interval, and the roles of exploration-related coefficients.

1.Q4 — Numerical stability and causal-update frequency.

1.1. Numerical stability. CAFRL uses causal-effect normalization, clipping, and bounded precision to limit sampling perturbations. With pi∈[pmin⁡,pmax⁡]p_i\in[p_{\min},p_{\max}], the injected diffusion noise variance satisfies

\leq \operatorname{Var}(\varepsilon_{n,i}\mid p_i) \leq \frac{h\sigma^2}{p_{\min}}.$$ For Walk/Crawl, $p_i\in[0.5,2]$, $\sigma=0.12$, and $h=1$, yielding a variance range of **$[0.0072,0.0288]$**. This prevents vanishing or unbounded diffusion variance. Five-seed learning curves show continued improvement without sustained performance collapse, supporting numerical robustness without implying convergence of the full actor–critic optimization. **1.2. Update-frequency sensitivity.** We estimate causal relevance every **50,000 gradient updates** using 4,096 replay samples. Across 190 recorded updates from Walk and Crawl (five seeds each), the mean relative changes in reference variance are **45.5%** and **51.2%**, respectively ([Appendix D.6, Figure 14](https://openreview.net/pdf?id=anPjp7LuMN#page=28)). An offline one-cycle lag analysis yields deviations of **21.1%** and **23.8%**, indicating that outdated relevance estimates can alter exploration geometry. These diagnostics do not directly establish return sensitivity to alternative update intervals. **1.3. Practical tuning.** The same 50k-update interval is used across all eight tasks without task-specific frequency tuning. It balances relevance freshness, estimation noise, and computational cost. **Overall, bounded diffusion provides numerical robustness, while periodic causal updates enable adaptive exploration at manageable computational overhead.** 2.**Q5 — Preset targets and automatic adaptation:** Thank you for pointing out that the parameter roles need clarification. We clarify this distinction in Appendix D.1, Table 5, which summarizes the roles and configurations of the principal algorithmic parameters. Let $\kappa_E$ denote the energy-target coefficient, distinct from the clipping constant in Appendix B. **The target $\mathcal{E}_{\mathrm{target}}=\kappa_E d_A$ is preset, while $\alpha$ is automatically adapted.** | Quantity | Treatment | Role | |---|---|---| | $\kappa_E$ | Preset; task-dependent | Specifies the target path energy | | $\alpha$ | Automatically adapted | Adjusts energy regularization against the target | | $\lambda=\alpha\sigma^2$ | Derived | Weights path-space KL regularization | | $\eta$ | Preset; 1.25 in main experiments | Controls directional anisotropy | | $\sigma$ | Preset | Sets reference-diffusion amplitude | As in SAC, a preset target accompanies an adaptive multiplier; here, the regulated quantity is path energy. The target remains fixed within each run, while $\eta$ controls the relevance-to-precision mapping. Task-specific targets and precision bounds are reported in [Appendix C](https://openreview.net/pdf?id=anPjp7LuMN#page=16), with additional parameter explanations in [Appendix D.1, Table 5](https://openreview.net/pdf?id=anPjp7LuMN#page=19). Sensitivity results are discussed in Response 4/4 and [Appendix D.5](https://openreview.net/pdf?id=anPjp7LuMN#page=22).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Request for Reviewer Follow-up

Dear Area Chair,

We sincerely appreciate your efforts in facilitating the review process. We respectfully seek your assistance regarding Reviewers azp5 and maVR, as we have not yet received their follow-up responses.

We have carefully addressed their concerns through additional experiments and analyses:

1.Reviewer azp5: Added matched kinetic-energy comparisons and controlled experiments on random versus causal anisotropic exploration under fixed diffusion variance.

2.Reviewer maVR: Clarified key variables, analyzed task-specific causal effects, and added evaluations on MetaWorld and DMControl.

These additions directly address concerns about comparison fairness, causal relevance, and experimental validity.

As the discussion deadline approaches, we would greatly appreciate it if you could kindly encourage the reviewers to revisit our responses and supplementary results.

Thank you for your time and consideration.

Best regards, Authors of Submission 58747

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.