- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this research the authors present a new method (SHIR) for leveraging time series explanations to improve the quality of explanations for forecasting predictors. They combine reference perturbation scores with a small set of measurements on each new input. Then they have an LLM propose comparisons between predefined input units while an evidence graph shares compatible execution records across comparisons.They provide empirical results on forecasting tasks using WeatherBench2 and VitalDB datasets and show performance gains against other common methods of feature importance selection.
Critical strengths
S1. Good empirical results. The performance shown against the external baselines is strong. The authors put together a thorough evaluation with carefully constructed tasks. The results support the proposed combination of methods that make up SHIR including the graph-only comparison showing that current measurements improve explanation quality and the additional ablations focused on evidence organization and query selection.
S2. I think the core idea of adapting reference information to a particular prediction is interesting - as well as the combination of frameworks. The authors considered efficiency and system design in constructing the final method.
S3. The idea of handling evidence by having the LLM proposes comparisons, while the predictor’s measured responses determine the effects used to update scores is also interesting. The authors explicitly distinguish current measurements from reference defaults and avoid counting reused observations multiple times. This grounds score updates in observed model behavior and makes the resulting explanation easier to audit.
Critical weaknesses
W1. It is unclear exactly how much the LLM step adds independently. The results show that taking three new measurements helps, but a simpler heuristic rule or even a smaller model/less reasoning might choose equally useful measurements. Table 2 also shows a tradeoff in that the complementary selection improves sufficiency error but the reference-score Top-3 has higher comprehensiveness on both tasks. It would be helpful to have a clearer explanation of when comparison completion helps.
W2. It’s also a bit unclear how reliable the improvements are in the empirical setting. The main results seem to only be run once with no variation because there doesn’t seem to have been multiple runs with different seeds. It would be great to see more averages over 5 runs in the final results.
Questions for authors
Q1. How often do the three current measurements change the selected units compared with graph-only?
Q2. What’s the authors intuition for why SHIR can better preserve the prediction when its chosen inputs are kept, while T3 causes slightly larger changes when its choices are removed? Since T3 also uses LLM suggestions, have you already tested a simpler rule for choosing three queries with the same reference scores and score update?
Q3. What’s the e2e latency/cost and scalability in particular the measured latency of llm calls relative to the rest of the computation and how things scale with longer time series?
We sincerely thank the reviewer for the effort in reviewing our submission. Our detailed replies are as follows:
Response to hhax-W1/Q2:
We tested simpler rules on 50 WeatherBench2 test inputs with the same reference, perturbation, three-query budget, R0, and evaluation. Global Top 3 and Global Random 3 select from the full catalog without an online LLM; LLM Direct 3 asks the same proposer for three units; T3 selects within SHIR's LLM-derived pool.
SHIR improves all four metrics over both no-LLM rules, reducing CPP error by 16.4% relative to Global Top 3. LLM Direct 3 improves three metrics over Graph-only but lowers Comp, and SHIR beats it on all four with the same proposer and budget, supporting the use of the LLM to propose comparisons for joint batch selection rather than to pick units directly.
To show when completion helps, we compare SHIR with T3 at each size: paired differences (SHIR − T3) over the 50 inputs, Wilcoxon signed-rank tests, Holm-corrected over eight tests.
SHIR has lower retention error at every size (significant at 7.5% and 15%) and a significantly larger deletion effect at 10%; T3's higher Comp comes from the two smallest sizes. Our intuition: T3 spends the budget on the highest-scoring candidates, refining the top of the ranking, which matters most when only the first few units are deleted. SHIR instead completes comparisons, prioritizing those that cross top-k boundaries; this corrects set membership, which retention evaluates at every size. Completion therefore helps most when set membership, rather than the first few units, is uncertain.
Regarding a smaller model or less reasoning, we replaced only the online proposer, keeping Grounding, selector, R0, and budget fixed. All proposers, including the original, run in non-thinking mode. A local Qwen3.5 4B proposer improves all four metrics over the original (e.g., Comp 0.1733 vs. 0.1634; CPP error 0.0238 vs. 0.0254), and Qwen3.8 27B attains the lowest retention errors (Suff 0.0291, CPP 0.0231) at 1.40 s per input. SHIR thus works with a small, non-thinking proposer, since the proposer only nominates comparisons while predictor responses set every score.
Response to hhax-W2:
The ablation variants already average three repetitions, and SHIR remains best on Suff error and both 10% endpoints on both tasks. The per-input paired tests above add input-level evidence: against the closest comparator, SHIR's retention gains at 7.5% and 15% are significant after Holm correction. We will report five-run averages for the main results in the revision.
Response to hhax-Q1:
We compared SHIR's top-k set with Graph-only under the same reference. Replacements count k−∣SkSHIR∩SkGraph∣, averaged over all 50 inputs; Δ is SHIR − Graph-only.
Three measurements change the explanation for 50%–72% of inputs, and SHIR improves both mean deletion and mean retention error over Graph-only at every size.
Response to hhax-Q3:
Online explanation takes 3.33 s per input, 3.12 s (93.7%) of it in LLM calls, and uses 16,088 input and 353 output tokens in non-thinking mode. Reference construction runs once per task and is reused: 23.4 min, 1,300 predictor calls (200 original predictions, 1,100 perturbations), and 1.97M/2.16M input/output tokens including reasoning at maximum effort, which we will clarify in §4.1 and A.6.4 applies only to this stage.