- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I do not understand the paper at all.
Critical strengths
NONE.
Only if I could undersatnd the paper can I even judge it. I see no critical strengths as the entire paper was completely incomprehensible.
Critical weaknesses
The paper is incomprehensible as written, so it cannot be evaluated
The examples below are taken from the first few pages; the same pattern continues through Section 3, after which I could no longer follow the paper. I list them so the authors and the ACs can see the problem concretely. Specific examples:
ABSTRACT
At equal additional realistic evaluation counts, retaining runner-up checkpoints and replaying fewer dates improves mean error over complete replay of one winner per group in six episodes, with agency-average gains of 4.48m at SEPTA and 0.10m at VTA.
What is 4.48m and 0.10m? What is "m" here? What is an episode?
Restricting the library to origin-available fits remains competitive
Unclear what this means
The abstract then ends with
Finite-candidate analysis separates exclusion, replay-score error, and changes in candidate performance between validation and testing. Full-replay controls distinguish the benefit of wider retention from the cost of reduced date coverage. Accurate rescoring cannot recover a discarded alternative, while retaining more candidates need not improve deployment loss. Retention and final selection must therefore be evaluated together
I have no idea what any of this means. There is no context given. If the authors would like to convey a point, the abstract should be kept high-level instead of filling it with jargons and terms that have not been defined.
INTRO
Data vintages affect estimates and model rankings
What is a data vintage? The term has not been defined.
does realistic evaluation still have access to the candidates needed for a good deployment decision?
Who has access here? What is exactly meant by "Evaluation has access to candidates"?
Replaying validation with origin-available inputs can correct the ranking of survivors, but cannot recover a discarded checkpoint
Unclear what this means
Restricting the library to origin-available fits is another response, although it may remove useful models trained on more complete historical records. We compare these choices by the selected forecaster’s deployment loss.
What is an origin-available fit (not defined earlier)? What is a forecaster's deployment loss? Is it the loss in future windows not used in the test set at model selection time? All this is unclear.
We compare origin-available (A) and retrospective (F) information at two stages: inner validation retains candidates, and outer validation selects one for deployment. In AA, AF, FA, and FF, the first letter specifies retention; the second specifies final selection. All four procedures use fixed fitted trajectories and common A test inputs (Figure 1). Matched libraries help distinguish which candidates are available from how many survive and how well they score on inner validation
It is completely unclear what this means. What is "origin-available" and what is retrospective? The whole papers hinges on these terms and they are not defined clearly.
- Figure 1 is confusing. I don't undersatnd a thing.
Our contribution is an empirical account of candidate survival, supported by finite-candidate analysis. The crossed design distinguishes retention from final ranking; matched libraries separate composition from candidate count and inner-validation quality. Full-replay controls then separate the benefit of retaining more candidates from the cost of evaluating fewer dates: at SEPTA, widening gains 6.20 m, while reduced coverage gives back 1.73 m. The analysis explains these outcomes through exclusion, replay-score error, and validation-to-test transfer. These are diagnostics computed after testing; the evaluated policies select using only pre-test information.
The motivation itself is unclear from what's written so far. The last paragraph of the intro (quoted above) discusses the authors' contributions and it is even more unclear. What is SEPTA / VTA? The abstract says they are "transit agencies" but doesn't provide references. What are they? Datasets?
The paragraph doesn't give any information on what the paper does - is it an analysis? A method? What are the takeaways? What are the results?
ON RELATED WORK:
- There is a lot of incomprehensible text in the related work as well.
Paired replay uses difference estimation (Särndal et al., 1992), related to proxy-plus-correction inference (Angelopoulos et al., 2023). Whereas proxy-assisted LLM ranking studies preference uncertainty (Chatzi et al., 2024), our endpoint is selected forecasting loss
What does this mean?
matched subset-only controls isolate the value of auxiliary scores.
What does this mean? None of these terms are defined.
Section 3
- The entire section goes on to use jargon and terms that are not defined at all. It has propositions and terms that contain no meaning (with all the context given before). I can pick any sentence at random to show this.
Section 4 on wards
- The experimental section is also incomprehensible; I have no idea what is being said here.
- Only meaningful thing I got from this section is that they used the Monash time series somehow with a train-val-test split but it is unclear so far: what the experimental setting is, what the goals are, what models they are evaluating
- Due to that, I am unable to undersatnd any of the findings of the paper until the conclusion.
Questions for authors
See weaknesses. I have nothing more to ask/clarify as I did not understand anything.