- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a new problem of forecast risk classification. The goal is to generate a risk label for each forecast. They used validation set to construct groundtruth labels and built dedicated neural network architecture for these risk classification tasks. Experiments demonstrated the proposed method can have better risk classification accuracy than other general time series classification methods.
Critical strengths
- It’s a new problem framing for time series forecasting, instead of predicting distribution or confidence interval, the paper proposes a classification setting for risks.
- The experiments demonstrate good risk classification performance from different angles: classification accuracy, error reduction when removing risk classifications; robustness when defining error on different percentiles; and ablations on different components.
Critical weaknesses
- The definition of risks is simplified. 80 percentile of very small errors is still very small errors, may not be a risk; 20 percentile of big errors is still big errors, should not be neglected.
- I found the presentation of technical details lacks clarity. I have many detailed questions about Section 4. See more in the question section.
=== Post rebuttal updates ===
The authors improved clarity in the new revision and I increased my score to weak accept.
Questions for authors
- What positional embedding in (4) is used?
- How to get E in (5), it seems is a concatenation of e? What roles do the and play? LN is LayerNormalization?
- What is forecast-prefix in Section 4.4? Are there definitions of all used features?
- In (8), , why compare a future time step to a history step L? Which L should one use?
- In (8), how to exactly compute , average of slope of last k steps and over D variables? They need to be precise.
- In (9), What are the motivations for that definition? Why are the terms defined in that way? One includes more and more forecasts as h increases, thus larger std; should it be a moving window of size k on the forecasts because you are comparing with the last k steps in the history.
- For query embedding in (11), it seems to just map an integer h to a vector; is the embedding learnt or is it positional embedding for future timesteps?
- How is fused memory M defined in (12)?
- How are the validation datasets (that used to construct the labels) derived?
- How do you compute Recall@Top20%? How is the Top selected? Based on risk probability? If so, it needs to be clear.
4 comments
Follow-up on Our Response Before the Discussion Window Closes
Dear Reviewer,
Thank you for your constructive review and time. As there are fewer than 12 hours left before the discussion phase ends, we wanted to gently follow up in case you have a brief moment to check our response and additional experiments. We are happy to answer any remaining questions before the deadline.
Warm regards,
The Authors
Quick Reminder: 2 Hours Left in Discussion
Dear Reviewers,
Just a quick note that there are about 2 hours remaining in the discussion window. We are standing by and ready to respond right away if you have any brief follow-up questions.
Best regards,
The Authors
Final Reminder: <30 Minutes Left in Discussion
Dear Reviewers,
As the discussion window wraps up in under 30 minutes, we just wanted to check in one last time. We are actively monitoring this page and ready to respond immediately should anything else come to mind.
We truly appreciate your time and constructive insights throughout this process.
Best regards,
The Authors
Gentle Reminder: Follow-up on Our Rebuttal and Additional Experiments
Dear Reviewer,
We appreciate your time and insightful comments. Since the discussion period wraps up in about a day, we kindly invite you to take a look at our rebuttal and additional experiments when you have a moment. We remain available to address any further questions ahead of the deadline.
Thank you for your consideration,
The Authors