- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a generator that modifies a time series input to make a forecasting model produce a prediction within a desired range. ConTex separates the changes into a timestep selection mask and a modification strength , while GenCF directly predicts the changes. Experiments on four forecasting models and four datasets show that both methods achieve higher validity than ForecastCF in most settings, modify fewer timesteps, and run faster. A deletion test suggests that the mask identifies important timesteps.
Critical strengths
- Efficiency and validity. The generators achieve higher validity than ForecastCF in most settings while requiring much less computation.
- Comprehensive evaluation. The paper evaluates four forecasting models and four datasets using five random seeds, multiple metrics, and sensitivity studies. The authors also released the code.
Critical weaknesses
- Unclear benefit of decomposition. GenCF matches or outperforms ConTex in validity in most settings. Although ConTex produces more compact changes, it also uses an additional sparsity penalty. Testing both methods with the same penalty would clarify the benefit of decomposition.
- Limited evidence of relevance and realism. The deletion test does not clearly show that the mask identifies important timesteps, and no ground-truth evaluation is provided. ConTex also makes large changes, with proximity values about four times those of FCF-Tuned on NN5 and Electricity using N-HiTS. For realistic targets, NN5 compactness drops to 0.05–0.10, contrary to the Figure 4 caption.
Questions for authors
- Is GenCF the same as the “w/o Decomposition” ablation, and does it use a sparsity penalty?
- Can ConTex and GenCF be compared across different sparsity weights using validity, compactness, and proximity?
- Can the mask be tested on a synthetic predictor with known important timesteps and compared with gradient-based methods?
- How realistic are the counterfactuals? For example, how close are they to the nearest training sample?
- How does IER rank methods when none reach 50% validity, such as PatchTST on NN5 with realistic targets?
3 comments
Response Part 2
Response to Question 4: Counterfactual Plausibility
Thank you for this question. We would like to clarify that our study focuses on model-level counterfactual explanations, including the ability to reveal potentially implausible dependencies learned by the forecaster itself. ConTex, like the evaluated baselines, does not explicitly optimize for data-manifold realism. We therefore did not include a plausibility assessment in our evaluation, as it lies outside our objective. Importantly, enforcing plausibility during generation could conceal precisely the implausible model dependencies that our explanations aim to reveal.
We do not claim that the generated counterfactuals necessarily follow the real data distribution, as their primary purpose is to explain the forecasting model's behavior. We consider incorporating plausibility and actionability requirements an important next step, for which measuring data-manifold proximity would provide a useful complementary assessment. We have clarified this distinction and discussed how practical constraints can be incorporated in the revised manuscript (Appendix, “Actionability Constraints”).
Response to Question 5: IER Ranking Clarification
If no method reaches the 50% validity threshold, all methods receive the same failure rank (1) for that dataset–backbone combination. Thus, no method is rewarded for favorable proximity or compactness despite insufficient target validity. The overall IER is then computed by averaging the ranks across datasets, so differences arise from configurations where methods achieve sufficient validity.
We have revised the IER explanation in Appendix A.4 to clarify this case.
Clarification Regarding Apparent Contradiction in Weakness 2
Regarding the apparent contradiction in Weakness 2, the reported results are correct and do not contradict the claims made in Figure 4. The caption refers to ConTex achieving the highest compactness relative to competing methods, not uniformly high absolute compactness. As specified in Table 24, these comparisons include only methods reaching at least 50% target validity, since high compactness without sufficient target satisfaction does not indicate an effective counterfactual. We acknowledge that absolute compactness can be low across methods, particularly in certain realistic-pattern settings on NN5. We explicitley adress this as limitation in our analysis.
Thank you for the detailed comments. My concerns about ground-truth relevance and plausibility remain, so I will keep my score.
We thank the reviewer for taking the time to evaluate our paper and for the helpful comments. We have carefully considered the points raised, performed additional experiments, and made corresponding revisions to the manuscript. The updated version has been uploaded. We address the reviewer's questions and concerns below.
Response to Question 1: Is GenCF the same as w/o Decomposition, and does it use a sparsity penalty?
Yes, GenCF corresponds to the w/o Decomposition ablation and uses the same sparsity weight as ConTex. Since GenCF has no explicit mask, it instead employs a difference-based sparsity penalty:
LsparseGenCF=TF1∑t,fσ(50(∣xt,fCF−xt,f∣−τ)),
which penalizes changes exceeding the threshold τ. Thus, both variants are sparsity-regularized with identical weighting, but different formulations (see released repository
loss_torch.py, “sparsity_loss_difference_based”).Response to Question 2: Different Sparsity Weights
Our primary objective is to evaluate the benefit of the mask–strength decomposition under controlled experimental conditions. Accordingly, we compare ConTex and GenCF using the same training settings and sparsity weights. We do not claim that different hyperparameter choices could not yield different validity–compactness–proximity trade-offs. Rather, our conclusions concern the benefits of the decomposition within the evaluated setting, particularly its ability to provide explicit temporal selection scores for interpreting the forecaster, which GenCF cannot provide regardless of the sparsity weight.
Response to Question 3: Comparison with Gradient Based Attribution
Thank you for this valuable suggestion. We have expanded our experiments to include a comparison with gradient-based attribution. However, we would like to clarify an important distinction: A synthetic predictor with known temporal relevance would provide a useful test of attribution accuracy, whereas our study investigates whether meaningful temporal selection patterns can be learned from trained black-box forecasters, where the underlying relevance structure is not directly observable. The challenge is therefore to uncover such patterns rather than recover predefined relevance labels.
We nevertheless recognize that our original experiments did not sufficiently investigate whether the learned selection scores reveal meaningful underlying temporal relevance patterns. In particular, it remained unclear whether they capture recurring, target-dependent patterns across inputs beyond what conventional gradient-based attribution can identify. To address this, we extended our evaluation beyond mask faithfulness with a mask-consistency analysis, comparing the learned selection scores against input gradients of the target-violation loss. Across all 16 dataset–backbone combinations and Top-k budgets of 5%, 10%, and 20%, the learned masks exhibit greater within-target versus between-target overlap differences than gradient-based attribution (48/48 comparisons; revised manuscript, p. 9, “Selection Patterns and Stability,” Appendix A.14). Our cross-seed analysis further shows substantial agreement in selection rankings across independently trained generators sharing the same forecaster (Appendix A.13.1, Table 31).
These findings provide additional evidence that the learned scores capture recurring, target-dependent temporal patterns in black-box forecasters, beyond the functional relevance established through mask deletion. They do not establish recovery of ground-truth relevance, which remains an important question for future evaluation with synthetic predictors.