- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper tackles the stream-level certification problem. It proposes an averaging of the conditional risks faced along the deployment stream. This allows for martingale fluctuations and enables finite-sampela dn variance-adaptive policy tests.
Critical strengths
-
The paper is equipped with sound theoretical results. The assumptions are outlined, and the theoretical results are listed. However, there are no visible key insights from those results.
-
The paper proposes a detailed algorithm, and the results discuss it thoroughly.
Critical weaknesses
-
The paper is very difficult to read and follow. The terms used are very niche and not recognized by the general ML reader. An illustrative figure at the beginning can help a lot. Furthermore, I recommend a human-written write-up that allows for better understanding, storytelling and flow.
-
Despite providing theoretical results, we would prefer to have some key insights outlined.
-
The results only tackle forecasting-based domains.
Questions for authors
-
How is this work related to the policy design of risk adaptation at test-time in [1]?
-
The method seems to apply to any type of policy at test time, but the experiments tackle time-series forecasting. Can we then say that the emthod is restricted to such applications?
2 comments
Revised PDF uploaded
Dear Reviewer hMkq,
Thank you again for your review. We have uploaded the revised PDF, with changes marked in blue. The points you raised are addressed in the following places: the overview figure in Sec. 1 (Figure 1), the two key insights in Sec. 1 and Sec. 3.1, the plain-language explanations of specialized terms and the takeaway after each theorem in Secs. 1–3, the notation table in the appendix, the relation to [1] in Appendix A.1, and the streaming classification experiment beyond forecasting in Appendix D.6. Since generative AI was used to assist in drawing the new Figure 1, we have updated the AI Use Statement accordingly. We would appreciate it if you could let us know whether these address your concerns.
Response to Reviewer hMkq: Key insights, presentation, and generality
We thank the reviewer for the feedback. In summary, we (1) state the two key insights of the paper (W2); (2) revise the presentation, including an overview figure in the introduction (W1); (3) clarify the relation to [1] (Q1); and (4) explain that the framework is not restricted to forecasting, with a new streaming classification experiment (W3, Q2).
W2: key insights. The paper answers two questions that arise when LTT is taken from IID calibration data to a deployment stream.
We will state these explicitly in the introduction and after the corresponding results.
W1: readability. We will revise the presentation for a general ML audience. As the reviewer suggests, we will add an overview figure to the introduction that shows the full pipeline (deployment stream → what to certify → why it is testable → policy-level tests → certified region). We will also add a notation table, plain-language explanations of the specialized terms where they first appear, and a one-sentence takeaway after each theorem. The paper was written by the authors; as disclosed in the AI Use Statement, generative AI was used only for language polishing.
Q1: relation to [1]. The two works sit at different stages of deployment and are complementary. TRAM [1] composes a fixed library of risk-neutral source policies under a reward–risk tradeoff specified at deployment; it constructs a policy in a reinforcement-learning setting. StreamLTT does not construct policies. Given any finite set of candidate policies, it certifies which of them satisfy a risk constraint over a dependent stream, with family-wise error control (Eq. (11)). Policies produced by a method such as TRAM under different risk weights could serve as the candidate family Θ for StreamLTT. We will add this discussion to the related work.
W3 / Q2: is the method restricted to forecasting? No. The framework is defined for any policy applied to a frozen model on an ordered stream with bounded losses revealed after each decision (Section 2); nothing in Definition 3.1 or Theorems 3.2–3.7 is specific to forecasting. We chose forecasting for the main experiments because temporal dependence arises naturally there.
To confirm this empirically, we added a streaming classification task with a reject option on the Airlines dataset (flight-delay prediction, ordered by time). We use a histogram-based gradient-boosted tree classifier (100 boosting iterations, at most 20 leaves per tree), trained on the initial 20% of the stream and then frozen. The classifier predicts when its confidence exceeds a threshold and abstains otherwise; the candidate policies are the thresholds. We certify two risks: selective error (a prediction is issued and is wrong, α=0.10) and false alarm (the positive class is predicted and the label is negative, α=0.05). The rest of the protocol is identical to the main experiments. Each entry reports Rˉmax / empirical FWER (mean maximum certified risk / family-wise error rate).
The pattern matches the forecasting results: StreamLTT-A certifies up to risk 0.092–0.093 against a target of 0.10 and 0.047–0.048 against a target of 0.05, with empirical FWER below δ in all four settings. We will add this experiment to the appendix.