- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduced DynaLDI, a linear-Gaussian dynamic measurement model for estimating shared longitudinal state from multiple weak linguistic measurements. The model integrates Kalman filtering, dynamic factors, and heteroskedastic measurement models, and is validated via simulations and applications on external dataset.
Critical strengths
The paper addresses a real and important problem of how to estimate a shared longitudinal state from heterogeneous weak linguistic measurements, with rigorous validations.
Critical weaknesses
The model integrates various existing methods in longitudinal data analysis and signal processing. While this integration is reasonable, the authors do not clearly demonstrate that it provides new theoretical insights beyond existing approaches, which limits the novelty of this work.
The reliability of the stability diagnostics remains a concern. For example, only 4 of 8 prespecified diagnostic-performance criteria pass. The known-truth evaluation shows that has median Spearman correlation of only 0.036 with absolute state error, and identification RMS only 0.068.
Questions for authors
Compared to existing state-space and dynamic factor models (or other models), what new theoretical or methodological insights does DynaLDI provide for longitudinal weak linguistic measurements?
Response to Reviewer MR3o: Methodological Novelty and Diagnostic Reliability
We thank the reviewer for the valuable insights. We clarify both through additional matched experiments and explicit negative results.
Methodological contribution beyond existing models: DynaLDI combines (i) source-specific nuisance means, (ii) reliability-conditioned measurement variances with fixed, normalized loadings and (iii) an audit separating conditional state uncertainty from changes caused by re-estimating the measurement mapping. Fixed loadings constrain ambiguity between changing measurement precision and changing loading strength, without guaranteeing identification.
Precise inference conditional on fitted parameters need not be stable to changing measurement assumptions. In the Reddit audit, restricting the series to 2016 onward gives correlation 1.000 with the reference under fixed parameters but 0.001 after full refitting (Table 9). The audit operationalizes this distinction when no source is ground truth.
New matched ablations and independent baselines: In response to review, we evaluated 240 previously unused synthetic sequences (10 regimes × 24 replicates) with the fixed estimator. The ablation RMSE progression is 0.6390 (constant variance, no nuisance adjustment) →0.4859 (+nuisance) →0.4653 (+reliability conditioning) →0.4585 (+robust inflation), a 28.2% aggregate reduction.
Paired incremental improvements and bootstrap 95% intervals are 0.1531 [0.1164,0.1925], 0.0206 [0.0173,0.0243] and 0.0068 [0.0048,0.0089]. Nuisance adjustment gives the largest mean benefit but wins only 36.3% of pairs; the later components win 75.4% and 74.6%, and the full configuration wins 80.4% versus the unadjusted estimator. Thus the component gains are heterogeneous.
On the same datasets, DynaLDI (0.4585 RMSE) is compared with independent statsmodels dynamic-factor fits (nuisance regressors, constant diagonal/full covariance), approximate Student-t IRLS/RTS and adaptive innovation-EWMA Kalman/RTS:
On the same 240 matched synthetic sequences, DynaLDI achieves lower aggregate RMSE than four independently fitted dynamic-factor, robust, and adaptive state-space comparators. These exploratory results support its empirical utility without establishing universal superiority. Section 5.1 and Appendix H of the revised manuscript provide the experimental details: Table 14 reports overall RMSE and convergence, Table 15 presents scenario-level results, and Table 16 provides paired improvements, 95% bootstrap confidence intervals, win counts, and optimizer-convergence sensitivity analyses.
Positive differences favor DynaLDI; intervals are conditional on the fixed scenario suite. Student-t is approximate, not exact likelihood optimization; adaptive noise uses a single pass followed by retrospective smoothing. All 240 finite DynaLDI fits are retained; excluding four optimizer-stopping failures preserves all four mean advantages (236 pairs). The full-covariance DFM wins under correlated noise (0.4542 vs 0.5986) and has near-zero state correlation under block missingness and combined stress despite convergence. Diagonal DFM wins under clean, nuisance-drift and correlated-noise conditions. These are additional exploratory results, separate from the original 480-sequence confirmatory benchmark, and do not imply universal superiority (Section 5.1; Appendix H, Tables 14--16).
Diagnostic reliability: After correcting the robust-inflated covariance used in fixed-parameter re-evaluation, the same independent 384-sequence benchmark still passes only 4/8 diagnostic criteria (4/4 integrity checks). Corrected median Spearman correlations with instantaneous absolute error are 0.0345 for 1−ρt and 0.0676 for identification RMS, both below the predeclared 0.150 threshold; posterior SD gives 0.112. No criteria were changed.
Our additional aggregation analysis gives identification-RMS versus recovery-RMSE Spearman correlations of 0.434 across datasets, 0.759 across 16 scenario means and 0.214 (median) within scenarios. Leave-one-scenario-out correlations are 0.732−0.821, but six scenarios have negative within-scenario associations; posterior SD has median within-scenario correlation 0.651. The full envelope increases severe-misspecification coverage from 0.7157 to 0.8435 and matched coverage from 0.9347 to 0.9811, indicating conservatism, not calibrated local confidence. These analyses use the same diagnostic benchmark, not new holdout data (Section 5.3; Appendix D, Tables 10 - 11).
The contribution is methodological and empirical: a weak-measurement specification and identification audit with evidence of where recovery improves and where assumptions and diagnostics fail.