- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes predictive rank martingales for sequential detection problem, using a fixed calibration sample. The paper derives thte conditional null distribution of ranks, and constructs testing-by-betting procedures that is anytime-valid and control false alarm risk. The paper also shows detection guarantees and fixed-calibration power limits. Experiments are conducted on synthetic dataset and CIFAR-10C.
Critical strengths
- The paper presents a predictive-rank construction with finite sample to achieve anytime-valid change detection.
- The paper sheds light on the limitations that is imposed by a finite calibration sample.
Critical weaknesses
Note that the following represents my initial impression of the paper, and I am open to discussion. I welcome any corrections to my potential misunderstandings.
Critical presentation issue
I found the abstract and introduction unusually difficult to follow. The introduction moves into specialized methodological comparisons before adequately explaining what is the established problems, what is the challenge, who else are suffering from it, and why do we have to care. The opening of Introduction states a classical change detection problem to control error, but it does not tell what is new about the task (at this point, I thought the task is quickest changepoint detection. Only afterwards I realized that there is restriction like unknown distribution, different from the classic QCD).
The second and third paragraphs of Intoduction introduces technical terms in rapid succession. In hindsight after reading the paper fully, I noitce that the authors identify reference contaminatino and dependence from calibration reuses as challenges, but at a first glance these challenges are explained through a comparison between existing techniques, requiring readers to already understand the methodological distinction, whose importance the introduction should establish.
The introduction bluntly ends without clear insights that readers should take home. The contribution list follows, but their significance is not brought forward effectively. Thus, the paper including Introduction needs substantial restructuring around the established problem, unresolved obstacle, proposed insight, and results.
Limited novelty and insufficient experimental support
The paper's contribution is narrower than the collection of method names and guarantees initially suggests. Shaer et al. 2026 already study fixed-reference distribution shift monitoring, identify the calibration reusing problem, use adaptive ONS betting, and provide stopping time guarantees. Thus, keeping reference fixed, using an order-based score and adapting bets carry limited novelty.
The novelty resides in the predictive null law of the reused reference, but the derivations relies on classical uniform spacing and Dirichlet-conjugacy arguments, if I understand correctly. The paper therefore needs to establish more clearly what practical or theoretical capability it adds.
A further limitation is the strong i.i.d. assumption. Real-world time series data are rarely independent stream, so the proposed approach has a substantial practical limitation. If I understand correctly, temporally dependent stream does not necessarily have to obey the proposed predictive rank law: the anytime false-alarm guarantee does not automatically extend to real-world settings.
The experimental evaluation is also limited. The synthetic experiments reflect the directional and dispersion features being studied. A dispersion sensitive, confidence band-corrected baseline is needed to isolate the benefit of predictive centering. The CIFAR with one pretrained ResNet is also restrictive: they are tested on corrupted monitoring stream shifted from its first observation. Together with the i.i.d. assumption issue, the paper does not fully showcase it's effectiveness and practical relevance.
Questions for authors
Please provide point-by-point to my concerns above.
Dear Area Chair and Reviewers,
We have uploaded the revised manuscript. The tables below summarize the changes made in response to each reviewer. Comment numbers follow our earlier replies. All locations refer to the revised manuscript.
Reviewer 4gjc
Reviewer Rrnm
Reviewer MayF
Reviewer 8zvj
The PRM construction and main theoretical guarantees remain unchanged. The revision adds experimental comparisons, clarifies the connection to prior work, and improves the presentation.
Thank you for your time and feedback,
The authors