- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GALACTIC, a method that explains time series classifiers with counterfactuals at two levels. GALACTIC-L creates a counterfactual for each series and pushes it clearly past the decision boundary, so that the same change can also work for similar series. GALACTIC-G then picks a few changes per class pair that flip as many series as possible, using an MDL score. On 10 UCR datasets and 3 classifiers, GALACTIC covers more series with shorter explanations than 8 local and 6 global baselines.
Critical strengths
- Clear motivation. The paper explains why a small counterfactual change for one series may not work for other series. Its margin requirement directly addresses this issue and improves the results of all tested generators (Table 1).
- Thorough evaluation. The experiments include 14 baselines, 3 classifiers, 3 seeds, and sensitivity studies. A held-out test (Table 8) shows that the method still performs better when selection and scoring use different series.
- Flexible global selection. GALACTIC-G can select explanations from any local generator, making it useful beyond GALACTIC-L. Its MDL formulation balances coverage and explanation complexity without manually choosing weights between them.
Critical weaknesses
- The main metric is the method's own objective. GALACTIC optimizes MDL and is also judged mainly by MDL. This MDL score charges very little for large changes, and GALACTIC makes larger changes than the closest competitor, CELS (16.7% vs 9.8% of timesteps changed). The paper should show that the gain is not just from making bigger changes.
- The changes may be unrealistic. The paper does not measure plausibility. In Figures 5 and 6, the counterfactuals contain sharp spikes, and some changes (about +4 and -6) are larger than the series itself (about -2 to 3). Such changes may exploit the classifier rather than explain it. Also, most global baselines are tabular methods that barely work on time series, so the global comparison is weak. The theory in the appendix is correct but simple, and the main guarantee relies on an assumption (Assumption 1) that is never checked.
Questions for authors
- If the baselines' changes are scaled to the same size, does GALACTIC still win?
- Can you report a plausibility measure for all methods?
- How do the results change with a different MDL design (e.g., a linear instead of log cost for change size)?
- Why do the changes in Figures 5 and 6 contain spikes, and how large are the changes compared to the series?
- Does Assumption 1 hold in your experiments?
- How does the method work when the target is not the second most likely class?
2 comments
Thank you for the detailed comments, which address most of my concerns. I will keep my score.
Thank you for the positive assessment.
W1 - Q1: The original comparison controlled action count, not edit size. We now match the ℓ2 budget per transition across all nine local pools and rerun selection (Table 12). Galactic-L remains first (Eff 0.864 vs. 0.783), with the second-smallest original median ℓ2 norm. It edits more timesteps than CELS (16.7% vs. 9.8%), but has higher Eff (0.924 vs. 0.793). Both are non-dominated on Eff, flipping cost and edit extent (Table 17). Sparsity is a trade-off.
W2 - Q2, Q4: Tables 15-16 now measure nearest-target distance, warped-distance shape and edit amplitude. Galactic-G has distance 0.169 of the dataset range vs. 0.183 for GLANCE and 0.209 for GLOBE-CE. GLANCE has better warped-distance shape (0.853 vs. 0.905), but lower Eff (0.657 vs. 0.924). FACEGroup is nearer by construction, returning observed target-class series, but has Eff 0.757 and 0.602. Figure 6 shows edits with frequent sign changes on SyntheticControl. Galactic-L has the smallest local peak change (3.0% of the value range vs. up to 7.7%, Table 16). Figure 7 also illustrates smoother edits on Coffee and transition-focused edits on Trace. The revised Ethics Statement clarifies that these diagnostics do not guarantee physical plausibility.
W2 - Q5: We revised the proof (Appendix A): MDL reduction is unconditionally submodular, while the (1−1/e) bound holds under the stated monotonicity condition for ∣S∣<2μ (Proposition 4). Of 115,918 examined candidates, 52,250 covered nothing new and would violate this condition, yet all 2,084 accepted additions satisfied it. Greedy accepts only MDL-reducing additions and yields comparable MDL to Flat Optimal across 86 pools (p=0.59, Sec.5.2). Baseline comparisons use exact Flat Optimal MILP selection.
W2, Global baselines: Since existing global methods target tabular data, we added two controls. Applying the same global selector to eight published time-series generators yields Galactic-L Eff 0.924 at 2.29 bits vs. 0.793 at 3.95 for CELS (Table 19). A gradient-based universal perturbation per transition, under a matched ℓ2 budget but without support selection or a reference margin, achieves 0.894 at 3.31 bits and edits all timesteps, vs. our 0.924 at 2.29 bits with 16.7% edited (Appendix C.6). Table 18 and Figure 11 additionally assess global methods independently of MDL.
Q3: We reran selection under the alternative linear penalty cδ=∥δ∥0+∥δ∥1, rather than merely rescoring solutions (Table 14). CELS then has lower cost (8.011 vs. 10.692), but our Eff is higher (0.927 vs. 0.833). The other seven sources cost 23.2-182.5. We clarify in the Abstract that coding choices define an exchange rate even without tuned coefficients. The linear penalty is not a description length in bits, unlike the logarithmic code-length surrogate (Appendix A).
Q6: Runner-up targeting is a default, not a restriction on t=y. Across 6,421 instances per arm, random alternative targets succeed in 82.9% of attempts vs. 89.0% for runner-up targets, with TwoPatterns and SyntheticControl hardest (33.4% and 33.9%, Appendix C.1.6). Mean ℓ2 among successes is 0.565 vs. 0.646, but these are different successful subsets, so this does not establish that alternative targets require smaller edits.