acceptodds
Under review as a conference paper at ICLR 2027

Identifying the Predictable Drift of a Semimartingale from Marginal Laws

Abstract

A special semimartingale admits a unique decomposition into a local martingale and a predictable finite-variation part . We consider the identification of when is observed only through its marginal laws at a succession of times, so that the pairing between observations, and with it the likelihood of the path, is lost. The estimand is then the projection of the sampled predictable compensator onto the observable feature filtration, namely the current state together with whatever randomness is shared across the population, so that at a fixed diffusion coefficient the marginal flow identifies the drift only up to a Markovian projection. We estimate it jointly with couplings between adjacent empirical marginals, constrained so that their conditional first moment agrees with a parametric drift, the parameter being identified by a rank condition upon the mean features. If the drift is an affine functional of an observed lag window, the joint problem is a convex quadratic programme whose solution is the pseudo-panel regression of econometrics. Our principal concern is the case, which we believe not to have been treated before, in which the drift is the output of a hidden linear dynamical system whose dynamics are themselves to be identified from the marginals. The joint problem is then a bilinear quadratically constrained programme, which we solve to certified global optimality by spatial branch and bound; with unpenalised state disturbances and a drift basis growing with the grid it is NP-hard already in latent dimension one, by reduction from rank-one matrix approximation, whereas the complexity of the deterministic system at fixed latent dimension remains open. A block-coordinate decomposition offers a cheaper alternative. Its fixed point is available in closed form; it converges locally -linearly under either an exact global solution of the identification sub-problem or the acquiescence condition of Hardt, Ma and Recht; and it shows that the couplings inform the parameter through the mean displacements between marginals and through nothing else. For the estimator itself we obtain rates at a fixed mesh, separated into Monte-Carlo, estimation and grid contributions.

Primary area:
learning on time series and dynamical systems
Keywords:
special semimartingale, predictable drift, identification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DimxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the problem of estimating the drift of a semimartingale having access to its marginal flow. It provides two different formulations of the problem. In the first the drift is parametrized as a linear function of the position of the process and some auxiliary variables s1,…,sks_1,\ldots,s_k. In the second. the drift is seen as a function of the current state of the process and a hidden variable.

The first formulation leads to a convex optimization problem, whereas the second leads to a quadratically constrained programme. These problems can be solved using classical algorithms for non-linear optimization.

Critical strengths

The problem is interesting and potentially has applications in flow matching and beyond. The proposed formulations have the advantage of leading to simple optimization problems.

Critical weaknesses
  • I am sorry to say this, but the paper is extremely challenging to be read by a human with background in stochastic calculus. There are plenty of meaningless convoluted sentences obscuring the simple messages they aim at conveying. Many key concepts on the contrary are not explained.
  • Despite this abundance of text, the two optimization problems (9) and (10) are not derived rigorously or justified with a convincing heuristic calculation.
  • The main Theorems in section 4 is about NP-hardness of the proposed models which is in my view not at all the natural results one looks for in this context.

The list of issues is not complete but frankly I have the strong feeling that the authors do not put enough effort in reviewing what LLMs produce. If this is the case, reviewers should not do this part of the job for them.

Questions for authors
  • Can you justify the objective functions in (9) and (10)? What does it represent in terms of the underlying stochastic process?
3 comments

We accept the criticism of the writing. The revision rewrites the abstract and introduction, removes the convoluted passages you objected to in Sections 1–3, 6 and 7, and adds the overview figure. On the use of AI tools: the paper discloses that they were used to aid writing and drafting, and that all AI-assisted work was reviewed by the authors. Your review shows the prose was not reviewed hard enough, and we have now done so.

Can you justify the objective functions in (9) and (10)? What do they represent in terms of the underlying stochastic process? This was missing from the main text. New Section 4.3 (proofs in Appendix G) derives it from one property of the process. Let π_k be the joint law of (X_{t_k}, X_{t_{k+1}}). It is a coupling of the two observed marginals, and by the Doob decomposition its conditional mean increment is the drift b_k(·; θ⋆). So the true joint law satisfies the constraint of (9) at θ⋆ with zero slack. Three facts follow.

  1. Feasibility is a convex-order condition. A coupling with conditional mean increments b_k(·; θ) exists if and only if the law of X + b_k(X; θ), X ∼ μ_k, is dominated by ν_k in convex order (Strassen's theorem). This defines the identified set Θ_I, which contains θ⋆. In particular, taking f = ±x gives the tower identity E_{μ_k}[b_k(X; θ)] = E_{ν_k}[X] − E_{μ_k}[X], one linear equation per pair. Under the rank condition these equations identify θ⋆.
  2. The slack is a distance to Θ_I. At fixed θ, the minimal slack is the μ-weighted ℓ¹ violation of the conditional-mean restriction by the best joint law consistent with the data. It vanishes on Θ_I and is bounded below by the pseudo-panel residual |γ_k − E_{μ_k}[b_k(X; θ)]|, with equality when the shifted target is attainable. For large λ_drift, (9) is therefore a minimum-distance estimator for the moment conditions of the Doob decomposition, in the sense of Chernozhukov, Hong and Tamer (2007), with the coupling as a nuisance parameter ranging over the Fréchet class.
  3. The transport cost is the second moment of the increment. For any coupling, E[(X_{t_{k+1}} − X_{t_k})²] = E_ν[X²] − E_μ[X²] − 2Σ_i x_i d_k[i]. This is the mean squared drift plus the expected conditional variance of the martingale increment, and both are fixed by the marginals and the drift. So the cost selects nothing among couplings with the same conditional means; the decomposition of Section 5 selects by entropy instead (the Schrödinger bridge). In (9) the cost acts only as a small linear tie-breaker in θ.
  4. The penalty in (10). λ_w Σ‖w_k‖² is the negative log-density of Gaussian state innovations with variance (2λ_w)⁻¹, so (10) is a penalised maximum-likelihood fit of the latent path, as in Shumway and Stoffer (1982).

NP-hardness is not the natural result in this context. We agree. The natural statistical results are identification, consistency and rates; in the paper these are Proposition 6.1, Corollary Q.1 and Theorem R.10 (Eq. 17). The hardness result shows that the hidden state, not the transport, makes the problem hard, and so justifies a certified global solver (Section 7) and the cheaper decomposition. The theorem has moved to Appendix S. Section 4.3 keeps one paragraph that states this purpose and the result's scope: the experiments use a fixed affine basis and are not in the hard family.

Key concepts not explained. Section 1 now explains the three nested σ-fields in words: everything at t_k, what marginal-law data can reveal, and its rounding to the grid. Section 3 explains why a coupling must be posited. Appendix A collects the stochastic-calculus definitions. If there are specific concepts you found undefined, we would be grateful for a list, and we will address each.

I thank the author for the detailed answer. Overall, the paper has improved and I am inclined to (slightly) increase my score.

AuthorsOpenReview ↗

Thank you!

Thank you! We genuinely appreciate your +1.

If there were any further details we could add, please let us know before the deadline in circa 20 hours.

Reviewer_5JneOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(−1)
Read review
Paper summary

The authors proposed a principled way to estimate the predictable drift from uncoupled marginal distributions at successive times. In their two specific settings (convex affine and non-convex linear), they reduced the estimation problems to clean optimization problems. In addition, they showed NP-hardness for the general formulation. They also seem to provide enough convergence analysis, but I did not have time to go through all 50 pages of the appendix.

Critical strengths

It an interesting and practical problem, and they provide several essential mathematical ideas which allowed them to turn the estimation problem into a well-formulated optimization problem. They also showed NP-hardness for the general formulation, which was interesting. In addition, they provided convergence guarantees for the proposed algorithm.

Critical weaknesses

They have relatively limited experimental support. The only experiment concerns a lower-dimensional setting, and I'm curious to see how robust their method is in other settings.

Questions for authors

Thank you for your submission. This was an interesting problem, and you provided a clean approach to the estimation problem. I do have one question:

You mentioned non-identifiability of the parameters from the means in Remark 2.2. Then, in line 460, you claimed "...upon the interchanged allocation of poles of Remark 2.2, which the longer record resolves". I'm not sure I quite follow the arguments made in these two places. Why does a longer record resolves the non-identifiability issue?

2 comments

Thank you for the positive assessment and for the sharp question, which exposed an imprecise sentence in the paper.

Why does a longer record resolve the non-identifiability of Remark 2.2? You are right that the remark, as submitted, did not support the sentence. Remark 2.2 is exact for the noiseless means with zero initial conditions; there no record length removes the ambiguity. The revised remark explains what breaks it in the estimator. Two mechanisms are at work, and both gain force with the number of pairs K.

  1. Monte-Carlo error in the observed means. The M-step regresses γ_k on the observed empirical means x̄_k, not on model-implied means. Their Monte-Carlo error n_k, of order σM^{-1/2}, follows n_{k+1} = s·n_k + ε_k. It passes through the observed pole s only, because the latent state is driven by u alone. Under the interchanged allocation the residual gains a term (s − r)·n_k, whose sum of squares grows like K while its fluctuation grows like √K.
  2. The acquiescence constraint. a ∈ B_ρ excludes the exact interchanges for the generating model of Section 7. What remains is an approximate interchange on the boundary of B_ρ, whose misfit also grows like K.

To check this we profiled the globally solved M-step over B_ρ on 20 seeds (code/poleswap_profile.py, NumPy only). At the setting of Section 7 (M = 600, N = 14), the true allocation is selected on 10, 17 and 18 seeds out of 20 at K = 24, 96 and 384. The others land near β̂ = −0.49, the boundary point reported in the paper. The median parameter error falls from 0.086 to 0.017 to 0.002.

We also report a limitation the check revealed. At M = 50 on the same coarse grid, the boundary point is selected at every K (0 of 20 seeds at K = 384), whereas the unrounded means of the same samples select the truth on 18 of 20. There the grid limits the estimate, not the record; its estimand is the compensator of the rounded process (Lemma R.3). This is consistent with the flat M = 50 row of Table 6, and the remark now says so.

Robustness and other settings. The submission already varied more than the headline experiment suggests. We now summarise this in Section 7 rather than leaving it in the appendices:

  • the number of copies M ∈ {50, 100, 200, 600} and pairs K ∈ {24, 48, 96} (Table 6);
  • grids up to N = 400 on a window fixed before the data are drawn (Appendix T);
  • latent dimension d = 2, 3, 4 (Appendix Y, Figure 7);
  • a non-acquiescent system;
  • a misspecified dynamics prior for the strongest competitor;
  • two harder variants (latent disturbance, slow factor), where we do not win.

The state X is real-valued by design, and the transport layer needs a grid on the state space. The parameter step, however, uses only the means (Proposition 6.1), so it extends to vector-valued X without a grid. We now list the real-valued state among the limitations in the Conclusion.

New Appendix Z adds four real data sets: bank deposit rates, reservoir inflows and the volatilities of 580 crypto-assets, all with the pairing hidden, and survey expectations of inflation, which are unpaired by design. It shows where the method helps and where it does not. With a weakly informative input it reduces to the pseudo-panel regression, which still forecasts a mean-reverting drift about 40% better than per-step methods. With an informative input it improves on that regression by about a third; on the deposit rates it identifies the pass-through of the policy rate, with poles fixed by the data rather than the constraint. A short finite impulse response of the input does as well (summary in our response to Bz1S, part 2).

Length of the appendix. The main-text claims rest on Proposition 6.1 (Appendix Q), Theorems L.1, O.2 and P.3 (convergence) and Theorem R.10 (rates). The reproducibility statement maps every numerical claim to its script.

Thank you for the response.

Reviewer_Bz1SOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies predictable drift of a special semimartingale from marginal laws observed at successive times, where the pairing between observations is unavailable. The authors consider two settings: a convex setting based on observed historical features, and a non-convex setting in which the drift is driven by a shared latent state whose dynamics must also be identified. The paper evaluate the proposed framework on synthetic data, comparing it with surrogate implementations of baseline methods such as Waddington-OT, along with internal comparisons. The experiments demonstrate the effectiveness of the alternating estimation approach in the non-convex setting.

Critical strengths
  1. The paper studies a well-motivated and practically relevant problem: learning dynamics from unpaired population snapshots in the presence of shared latent factors. In real-world applications, individuals may be difficult to track over time, while unobserved common influences can affect the entire population—for example, claims inflation in insurance.
  2. The authors provide a clear mathematical formulation of the problem, carefully explaining the observation scheme, modelling assumptions, and estimation target, as well as the limitiations imposed by the avalable information.
Critical weaknesses
  1. The experimental evaluation relies entirely on synthetic data generated under assumptions closely matching the proposed model. It therefore remains unclear how well the method generalises to real-world data, where the latent dynamics may be more complex or misspecified. 2.Unclear presentation and positioning contribution. The paper is difficult to follow, as it introduces multiple observation regimes, two model settings, and two estimation approaches with different objectives, without a clear overview of how these components fit together.The main text contains no figures, with the experimental figures relegated to the appendix. An overview diagram would help clarify the framework and the relationship between the proposed approaches. The paper would also benefit from a sharper distinction between its novel contributions and its connections to established pseudo-panel regression and linear system identification.
Questions for authors
  1. Could the authors compare the proposed method with baselines on real-world datasets? Does the method remain effective when the shared latent dynamics are substantially more complex? The current experiments do not establish whether the observed gains arise from the proposed estimation method or from a correctly specified model and access to additional inputs.
  2. Please clarify the core contributions of this paper and distinguish them from existing work. I would be willing to reconsider my rating if the authors could clearly articulate the paper’s core contributions.
  3. Can the proposed method identify the true drift, or does it only identify the evolution of the population mean?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Bz1S (part 1 of 2)

Thank you for the careful summary and for the insurance example. Claims inflation is exactly the shared latent driver the Non-Convex Setting models.

Core contributions, and how they differ from pseudo-panel regression and linear system identification. The introduction now separates the new from the known (paragraph Contributions, Figure 1).

  • Known. (a) When the drift depends on observed features only, our estimator's fixed point is Deaton–Moffitt pseudo-panel regression of mean displacements on mean features (Proposition 6.1, Appendix Q). We present this as a consistency check, not a contribution. (b) Once the mean displacements γ_k are formed, the hidden-state step is an output-error identification problem, which we solve with existing tools (Ho–Kalman; Hardt–Ma–Recht; branch and bound).
  • New. (i) The estimand: which drift is identifiable from unpaired cross-sections, namely the projection of the predictable compensator onto the state and the shared randomness (Section 1). (ii) An estimator that identifies a hidden linear system with unknown dynamics from unpaired cross-sections. Pseudo-panel methods admit no latent state with dynamics of its own. System identification presupposes an observed output time series, but here no individual is followed, so no such series is observed. Proposition 6.1 shows which output to identify (the mean displacement), and Remark 2.2 what is lost by doing so. (iii) The analysis: the closed-form fixed point, local linear convergence, rates split into Monte-Carlo, estimation and grid error, and the hardness of the general programme. (iv) The consequence that matters in practice: the identified dynamics let the estimator forecast the drift from the observed input and pool information across dates (Tables 1 and 6), which per-step trajectory-inference methods cannot do.

Gains from the method, or from correct specification and extra inputs? A new paragraph in Section 7, The source of the gains, answers this directly.

  • Inputs. Every competitor receives the same snapshots on the same grid, and those that can use it the same input u.
  • Specification. The FIR model of u differs from the true model only by truncating the latent impulse response. As the training record quadruples, its forecast error moves only from 0.050 to 0.042, while ours falls to 0.012 (Table 1). The PO-MFL surrogate is given a dynamics prior close to the truth. It is close to us in sample (0.026 vs 0.020 at N = 100) but loses that advantage once its innovation scale is wrong by a factor of two (Appendix U).
  • Where we do not win. We now say this in the main text. With an unobserved disturbance on the latent state, per-step methods tie or win in sample. With a slowly mixing factor (poles 0.9 and 0.8), neither approach forecasts well at N = 14. The real data (part 2) show the same dependence on an informative input.

True drift, or only the evolution of the population mean? (New Remark 6.2.) The estimator returns the drift as a function of the state, x ↦ α + βx + cᵀz_k, not only the mean path. However, the information it uses comes through the means (Proposition 6.1). The slope β, which says how the drift varies across individuals at one date, is identified from how the mean displacement co-moves with the mean level across dates, not from the spread within a date.

  • Within the model this suffices: the estimator is Fisher-consistent (Corollary Q.1).
  • If the true drift is not affine in x, the limit is the pseudo-true parameter of the mean regression (Appendix R). This need not be the best affine approximation in L²(μ_k).
  • Within-marginal information, such as the variance recursion, is unused; using it would also resolve the ambiguity of Remark 2.2.
  • The couplings do carry within-marginal information. As transition kernels they are three to eight times closer in W₁ to the true conditional laws than unconstrained transport (Table 7).

Presentation. Besides Figure 1 and the contributions paragraph, we rewrote the abstract and the passages in Sections 1–3, 6 and 7 that stacked qualifications. Section 4.3 now says what each programme estimates before the algorithms are introduced. The tenth page now holds an overview of the real-data results.

AuthorsOpenReview ↗

Response to Reviewer Bz1S (part 2 of 2): real-world data

Can the method be compared with baselines on real-world datasets? The new Appendix Z, summarised in Section 7, does this on four data sets, each with a protocol fixed before any estimator was run. In three the individuals are tracked, so we hide the pairing from every method and use it only for a reference: the paired regression ΔX_{i,k} = τ_k + β_ref·X_{i,k} + e_{i,k}, whose free time effect absorbs all shared randomness (the paper's estimand within affine drifts).

  • Bank deposit rates, policy rate as input. 3,187 US banks, 1987–2020.
    • The funds rate explains 90% of the shared drift. From the cross-sections alone we identify the pass-through: poles 0.24 ± 0.24i, strictly inside the constraint set, which does not bind.
    • Ours removes 38% of the pseudo-panel regression's drift error in sample (0.110 vs 0.177). From the policy rate alone it forecasts the drift 28% better, at all five origins. A four-lag filter of the input does as well.
  • Crypto-asset volatilities. 580 coins on Binance, 2019–2026, with Bitcoin's volatility as input. Up to the week a transition ends, it explains 58% of the shared shock, and ours removes a third of the pseudo-panel error (0.140 vs 0.211). A two-lag filter does as well. Lagged a week, Bitcoin explains 7%, and ours is the pseudo-panel regression.
  • Reservoir inflows. 95 Brazilian reservoirs, El Niño index as input. It explains 4% of the shared shock, so, as Proposition 6.1 predicts, ours is the pseudo-panel regression. That still removes 36% of the error of zero drift and forecasts the drift about 40% better than the last per-step (Waddington-OT) field. Per-step methods win in sample (0.10 vs 0.13).
  • Inflation expectations, unpaired by design. The Michigan Surveys of Consumers, 185 quarters, CPI inflation as input; six-month re-interviews (103,070 pairs) give a reference that no method sees.
    • Inflation explains 25% of the shared shock. Ours scores 2.14, against 2.24 for the pseudo-panel regression and 2.88 for zero drift; a finite impulse response (FIR) of the input does better (1.81).
    • In forecasts of the mean, ours (0.65) and the FIR (0.62–0.63) beat the pseudo-panel regression (0.66) and zero drift (0.73).
  • More complex latent dynamics, and where the gains come from. On the inflows and the survey most of the shared shock is driven by something other than the input; there our method degrades to the pseudo-panel regression, or close to it, rather than failing. Where it gains, on the banks and the crypto-assets, a FIR of the same input gains as much: on real data the gain comes from the input, which our method exploits from marginals alone, not from the hidden state.
  • Two limits for every unpaired method. Individual reversion is not identified: −0.67 over two quarters in the re-interviews, at most −0.43 from the marginals. Scores on the means, the only ones unpaired data allow, rank zero drift first on the inflows, where the pairing ranks it last but one.

So, to your question, the gains require an observed input that drives the shared randomness. With a weak input the method is a sound pseudo-panel estimator. With a strong one it improves on that regression by about a third and identifies the dynamics, but on these data a short filter of the input does as well, because the response is brief. Section 7 and the Conclusion state the first point, and Appendix Z.4 the second.

One computational limitation: on these data (51 to 406 transitions), branch and bound does not reach its tolerance (a relative gap of 10⁻⁶) within 180 s per M-step; the remaining gaps are 75–100%, and an hour does not help. The bound is weak, but the solutions are not: for d = 2 a scan of the poles (step 0.025, refined by Nelder–Mead), with the other parameters solved exactly, finds the global optimum, and Gurobi's solutions lie within 0.4% of it. Our claim of a certificate is now restricted to the small instances of Section 7, solved to a relative gap of 10⁻⁴, and the paper states the tolerance wherever it reports a global solve.

Thanks for your response. The additional experiments and clarifications have led me to raise my score. However, given the poor writing quality of the original manuscript, I cannot give a further increase. My final rating is 2.

Many thanks for your +1. We genuinely appreciate this.

We believe that the style, while perhaps overly technical in the original submission, should not be a key criterion for the acceptance. We stand behind the results in the original submission, as well as the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.