- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a Colosseum of Mechanisms (CoM) framework, which is a conquer-then-divide approach. They introduce Quasi-Gradient Descent algorithm that is doing non-convex optimization to fit the parameters of the model, given that the forms of (possible functions) are specified. They that their method is faster than competitors and is better than ‘divide-and-conquer’ (DaC) approach unless the segmentation form is assumed correctly for the ‘divide-and-conquer’ (DaC) approach
Critical strengths
- Strong empirical improvements and diverse evaluations - section 5.1 evaluates a toy setup, section 5.2 evaluates the forecasting setup, section 5.3 provides a real data evaluation example (EEG)
- Rigorous theoretical background: Appendix provided step-by-step proves and the paper also clearly references textbook facts for a more naive reader
- Decently clear writing: for example, I really liked Section 2 with the toy explanation of the problem and solution - it is very nice and clear
- Reproducibility: Code is attached for submission
Critical weaknesses
- Need of a simpler into and occasional lack of notations:
- A paper systematically skips rigorously telling what letter means what (for example what is in eq. 3 - is it region-wise loss objective? what is in the same equation? is it a specific assignment or data partition? same could be pointed through a majority of equations throughout the text).
- The paper introduces some terms / jargons without explaining them properly or explaining them too late - for example, Clarke subdifferential is mentioned several times before being introduced even on a high level. The abstract also mentions "multi-mechanism model" and "segmentation form", which are field-domain jargons, which a naive reader might be unaware of.
- Some simple schematics could improve the readability - for example, in section 5.2 it is not completely obvious without looking in the appendix that the current rule for is decided by the previous predictions with steps window, a small schematics could help to understand it
- Sensitivity analysis is not provided: - non-convex optimization methods, performance remains sensitive to initializations but no analysis on this is provided. Some practical recommendations on initializations and amount of needed refits would also help the practitioners.
- Empirical EEG analysis could be improved: - see questions 1-3.
- All of the evaluations are based on the linear forms: I am interested in evaluations where candidate mechanisms are non-linear in their parameters, since there the non-convex optimization might become even harder and sensitivity to initialization even stronger, so its interesting how well CoM could still perform
Questions for authors
- In section 5.3 - was or not? It could have been more straightforward in the writing as well that is a set of seizure / no-seizure modes and is the EEG rhythms bands
- Why do you report RMSE in section 5.3 if in fact you are not doing forecasting but state classification? Could you please report the classification numbers there and also add some baselines (like supervised classification based on extracted features from the random forest)? I would also inspire you to do the forecasting setup for the EEG data (same way as you did section 5.2) since its predicting seizures is a more relevant real life problem than classifying them post-hoc.
- Could you please provide error-bars for table 3?
- The noise in section 5.1 is not really heteroscedastic - it is homoroscedastic depending on the branch. Do you think the differences in noise amplitudes could help CoM to distinguish the regimes? Could you please provide the experiment with the same noise level and with more complex heteroscedastic noise, when noise depends on x value? What about different signal-to-noise ratios? How depended it CoM on the SNR?
- Could you please show how the GT classification looks like for Fig2? Also for CoM in both Fig1 and Fig2 there is a mix of blue and red points on the intersection of the lines - could you please comment why does it happen and how could this be solved / imporved?
Clarity:
6. what does TAR stays here for?