- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper is situated in a forecasting context, where the goal is to make accurate predictions about outcomes up until time based on some history up until time . Over time the ideal prediction function may change, meaning that the prediction model might need adjustments. In the context of this paper, these adjustments are not implemented by adjusting/refitting the model, but rather by producing two alternatives: the original prediction and a corrected variant. Then a router chooses whether to use the original or the corrected prediction. The central research question of this paper is then how well such routers work. To measure success, the paper introduces a range of metrics that, for example, compare the perfect routing (in hindsight) with the chosen routing (given only the past). The finding is that router performance really depends on the context, for example, it can be the case that always choosing the correction would be harmful but still sometimes the correction is more accurate than the original.
Critical strengths
In principle it makes sense to come up with evaluation metrics for the routing and to quantify the performnace of different routers on a representative set of datasets.
Critical weaknesses
- The central issue of the paper is the clarity of the writing. Even with considerable effort I had trouble to even understand the introduction; I only understood it after having parsed the whole paper.
- I am not yet convinced by the significance of the research question in the first place. Why this complicated setup with original model, correction, and router? Why not directly update the model? The router/correction have no signal that a refit would not have? I would love to see a comparison of the routing approach against a more direct approach.
- I am not yet convinced that all the introduced metrics are really interesting. For example, why would we care about how well routing could have worked if we could have chosen in hindsight? A realistic 'cealing' would instead measure how well the routing could ideally be given the available signal? Also, in hindsight even random noise can be turned into a useful signal?
- I am not convinced yet that the datasets and settings in the empirical study really justify more general conclusions, also taking into account the limited theoretical contribution.
Questions for authors
Can you try to rewrite the conclusion paragraph in Section 6 in simple words that help someone who is not directly from the field to understand what the central contribution and benefit of your methodology is? As a practicioner who considers using a router approach to correct forecasting, what can I take from the study?