- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OrbitBench, a benchmark that can be used to evaluate whether causal discovery models can accurately still report the same causal relationships after the variables in the observational data undergo strictly monotonic, invertible numerical transformations. The paper mainly uses two types of transformations, asinh and cubic, then fixes the preprocessing rules and graph decoding process, and then compares the changes.
Critical strengths
-
The problem studied in the paper is relatively important and has practical value.
-
The transformatio processes are well controlled, ensuring randomness and equivalence.
Critical weaknesses
1.The paper’s testing and analysis cannot fully prove that the selected relationships are more correct or valuable. They can only guarantee that the stability observed under the same type of transformations can, to some extent, predict that this stability still exists under new transformations.
2.The data coverage of the benchmark proposed in the paper is slightly narrow.
3.The issue that causal relationships should remain unchanged under appropriate variable reparameterization is not a new issue, and there is already related research. The paper’s novelty is somewhat weak.
Questions for authors
N/A
3 comments
Response to rpr9 (2/3): Model rankings changed across generators, but F1 still masked changes in predicted edges
Changing the nonlinear data generator changed which pretrained model had the highest reference F1. It did not remove the finding that F1 can stay the same or improve while the model changes which variables it connects. This distinction is why we agree that adding seeds from the submitted generator alone would not have resolved your coverage concern.
New generator comparison. The submitted nonlinear panel used RFF generators released with AVICI. After submission, we tested two independently implemented mechanisms: a nonlinear additive-noise model with Laplace errors and a post-nonlinear model with heteroscedastic errors. We used the same 32-system panel as the two-view precision experiment in Part 1; here we evaluated the predicted graphs before screening.
AVICI no longer had the highest reference F1 among the pretrained systems on either new mechanism. Nevertheless, skeleton F1 did not decrease while the predicted skeleton changed in 55/128 comparisons for CDFM, 52/128 for Arrow, and 38/128 for AVICI. Each denominator represents four coordinate changes on each of 32 systems, not 128 independent datasets. We therefore do not generalize the original model ranking. What reproduced across generators was the gap between the accuracy summary and the individual relationships reported.
New graph-size and sample-size comparisons. The submitted panels used 20 variables and 800 observations. We varied graph size (10, 20, 40 variables) and sample size (200, 800, 1,600 observations) in separate controlled comparisons. At 800 observations, Arrow’s median number of changed pairs rose from 6.1 to 9.8 to 25.8 as graph size increased. Yet the fractions of all possible pairs fell from 13.6% to 5.1% to 3.3%. Larger graphs had more changed relationships in absolute terms, but a smaller proportion of possible relationships changed. We will report counts alongside their denominators.
Additional analysis of the original physical predictions. The 57 light-tunnel environments in the submission are settings of one apparatus. We compared the stored predictions with the published standard-configuration graph restricted to our 20 selected variables, after verifying that omitted recorded variables were constant in the evaluated windows. This provides a configuration-specific reference for interpreting the existing predictions. It is not a new physical dataset or an independently validated ground truth. The additional experiments broaden the simulated mechanisms and scales; generalization across physical domains remains untested.
Response to rpr9 (3/3): A higher F1 can hide an incorrect answer about a directed path
We agree that an invertible change of measurement coordinates does not create a new causal system, and that reparameterization sensitivity itself is not new. Jørgensen and Hauberg (2020) studied causal decisions under reparameterization; Yi et al. (2025) evaluated graph recovery under assumption violations including changes of scale. OrbitBench evaluates something these principles alone cannot tell us: how often the same model remains a good choice by F1 while its graph gives a different answer about a specific relationship.
Evidence in the submission. On the 16-system linear panel in Table 2, Arrow remained within 0.01 F1 of the best of the three pretrained models in both views while changing its skeleton in 63/64 comparisons. That is the frequency of disagreement, not its size: the median change was 10.5 of 190 possible pairs (5.5%). On a separate 16-system panel, Table 4 showed that 18.0% of Arrow’s native directed-path answers changed on average. Even when the decoder was required to return the same number of edges in both views, 15.7% changed.
A concrete example from the original predictions; no new model runs. On the first system in the path panel, Arrow ranked first in skeleton F1 among the three pretrained models in both views. Its F1 actually increased from 0.8955 to 0.9014. Yet its answer to “Is there a directed path from variable 1 to variable 6?” went from correct to incorrect. The generating DAG contains no such path, and Arrow’s reference graph correctly contained none. After a mild asinh re-expression of the same observations, its graph contained the path 1 → 5 → 6. Neither prediction contained a direct edge 1 → 6. Thus the model choice stayed the same and its score improved, but its answer about whether variable 1 was upstream of variable 6 became wrong.
Post-hoc check across the full path panel. We checked all 64 Arrow comparisons, not just the illustrative example. Skeleton F1 did not decrease in 11 comparisons spanning eight systems. In every one of those 11, at least one previously correct path answer became incorrect. “At least one” does not mean that most answers changed: across the full panel, using all ordered pairs of distinct variables as the denominator, 13.75% of answers went from correct to incorrect against the generating DAG and 4.24% went from incorrect to correct. Coordinate changes can introduce errors and correct them; we do not assume that the reference prediction is right.
This is the distinction we will foreground in the revision. Without ground truth, the paired audit can flag a specific conclusion that depends on the measurement coordinates even when the model’s standing gives little reason to reconsider it. With a known generating graph, we can additionally determine which answer was wrong. We will report both the frequency and the magnitude of disagreement in the abstract so that frequent changes are not mistaken for wholesale graph instability.
Response to rpr9 (1/3): At fixed retention, auditing improved edge precision on new nonlinear systems
Thank you for the time and care you put into reviewing our paper, and for recognizing the value of the controlled comparisons. Your distinction between persistence and correctness motivated a direct follow-up. On 32 new nonlinear systems with known graphs, the submitted two-view rule increased retained-edge precision from 91.04% to 92.59% relative to model scores alone, while keeping the same number of candidate edges.
Evidence in the submission. Table 5 showed that audit-assisted selection reduced the proportion of retained edges that disappeared in a held-out view from 8.27% to 6.71%. However, the audit and test views came from the same two transformation families, and the precision gain was uncertain (+0.41 percentage points, 95% CI [−0.06, 0.88]). We agree that the original experiment supported greater persistence, not a demonstrated gain in correctness.
New experiment with the submitted rule. We applied the original rule without retuning it. For each model and system, both selectors kept 80% of the candidate adjacencies, rounded up. We locked the selections without using the true graphs and before generating the held-out predictions. We then evaluated the same retained edges in two ways:
New comparisons with stability-first selection. In a separate confirmation on 32 fresh systems, an eight-view rule reduced disappearance on two non-audit families from 13.59% to 11.99% relative to score-only selection. Yet simply prioritizing edges that survived the audit produced exactly the same retained set in 95 of 96 system–model comparisons. This result supports checking additional views, not the necessity of our weighting formula.
We also tested a separately developed rule that predicts disappearance from edge scores and their variation across audit views. We froze it before evaluating two independent cohorts of 32 new systems each. Relative to stability-first, precision improved in both cohorts (+0.37 and +0.64 percentage points; pooled gain +0.50, adjusted 97.5% CI [0.22, 0.82]). Its risk advantage was less general: disappearance was 0.31 points lower on two families represented during calibration but absent from the audit views, and 0.15 points higher on two families absent even from calibration. This finding belongs to the new rule, not to the submitted formula.
For a researcher prioritizing a fixed number of candidate relationships, the practical result is that audit views can add information beyond model confidence alone. A simple stability-first ranking captures much of the persistence benefit. The accuracy evidence concerns whether two variables are adjacent; it does not establish the direction of an edge or the benefit of an intervention.