- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors use asynchronously arriving multimodal evidence to assign weights to a set of candidate futures along candidate*horizon, claiming to solve the problem that previous fusion methods reuse a single reliability score across prediction horizons. But the data source is not clearly explained, the data is not reproducible, and all the comparison experiments use self-made weak baselines, so I think the quality is not up to standard.
Critical strengths
- Expanding candidate reliability from a fixed score into a candidate*horizon matrix, with separate weights per horizon, fits the way time series are modeled and has practical value.
- It integrates satellite imagery, AIS, environmental forcing, and simulators on a real oil spill scenario, which is a meaningful contribution in constructing a real-world setting.
Critical weaknesses
- A lot of information about the data is missing, so I cannot judge whether it is real. The source of is never explained, the data is private and not reproducible, and there are no event counts, no splits, and no variance.
- The baselines the paper compares against are really bad. None of them come from existing literature; they are all self-made comparison groups. And the existing literature shows this is not because no methods exist, but because the authors did not compare with them. None of the 5 baselines is a method from the literature, they do not even compare with the work they themselves cite, and they report ECE/Brier without comparing to any calibration method, so the gains do not prove anything.
- The abstract mainly advertises asynchronous and four types of controlled experiments, but none of them were actually done. Asynchrony in the method shows up only as a single time decay factor, so the title is just decoration.
Questions for authors
I find this research very unreliable, and nothing in the paper convinced me otherwise. That is the overall impression behind my three questions.
- How many events are in the training/validation/test sets? Where does actually come from? Please provide a complete description that makes the dataset reproducible, otherwise I cannot judge whether these results are real.
- Why is none of the 5 baselines a method from existing literature? There are plenty of ensemble weighting and calibration methods out there. Did you not look for them, or did you not want to compare with them? Please explain how these baselines were chosen.
- The abstract says experiments about timestamp perturbations, missing modalities, evidence-quality variation, and horizon-dependent candidate changes. Where are those experiments? If asynchrony is just one time decay factor in a formula, then what does "Asynchronous" in the title actually mean?