acceptodds
Under review as a conference paper at ICLR 2027

How Complex Should an Explanation Be? A Distortion-Rate View of Time Series Attribution

Abstract

Feature attribution for time series is usually posed as a choice among solution concepts - Shapley, Owen, Banzhaf - estimated as accurately as possible. We argue, and measure, that this framing is mis-specified. First, the attributed object is not a property of the model: the coalition value function depends on a masking baseline and an output scale, and every summary of it on a coalition measure. Across seven trained models these choices move the additive share of the value function by up to 32 points, and on one real model the top-10% attributed cells agree only 32% of the time between a zero and a first-observation baseline; neither values nor rankings survive a change of convention. Second, at a fixed convention the answerable question is how much of the value function an explanation of a given size can carry. Treating an explanation as a lossy code, we measure its distortion against its rate, the numbers it emits per input. On five real models, additive attribution is inefficient even at its own rate, locality is the wrong structure - a lag-1 band buys 3-9 points where a rank-one factorisation buys 21-30 at 2-9x fewer numbers - and the optimal complexity is finite and set by the query budget. Perturbation-based faithfulness metrics do not resolve these distinctions on a game with exact ground truth. Finally, an amortized estimator that emits the coefficients of a structured surrogate ships each explanation with a measured fidelity: it reaches 99% of a 200,000-query per-input fit on one model and fails on two of six, which its audit, whose query cost we price, reports.

Primary area:
interpretability and explainable AI
Keywords:
explainability, feature attribution, Shapley values, time series, evaluation, rate-distortion, interaction effects, amortized inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j5vzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that research on time series attribution asks the wrong question. Most work asks which method (Shapley, Owen, Banzhaf) is correct and how to compute it fast. The authors show that the result depends on setup choices: what replaces a masked value, which model output is tracked, and how coalitions are sampled. They then propose a different question: how much of the model’s behaviour can an explanation of a given size capture? They measure this as error against the number of values the explanation contains. On several UEA and MIMIC-III models, they find that low-rank explanations do better than additive or local ones, and that perturbation metrics such as keep loss cannot tell good attributions from random ones. Finally, they propose a fast learned explainer that also reports how well it fits each input.

Critical strengths

A clear new way to compare explanations:
Plotting error against explanation size puts all explanation types on one scale. The result holds on new inputs and new coalitions. A rank-1 explanation beats a local (lag-1) one on four of five models while using 1.7–9× fewer values.

It shows that rankings change with the setup, not only values:
Changing the masking baseline from zero to first observation gives a Kendall τ of only 0.267. Earlier time series work shows that the perturbation choice changes evaluation scores. This paper goes further and shows that it also changes the attributions themselves.

Honest work:
The paper reports hypotheses that failed (Kronecker structure, stable rankings, the Owen estimator), uses oracle-vs-random controls for the metric study, and reports that audit-guided repair does not beat random selection.

Critical weaknesses

Only two model types, and the TCN appears on one dataset only:
All seven UEA black boxes are GRUs. The only TCN is on MIMIC-III, and the main rate–distortion study uses only GRUs. The paper does not explain this choice. Transformers and CNN-based classifiers such as InceptionTime or ResNet are standard in time series classification and likely create different interaction patterns. The one TCN result already looks different: its rank-1 energy share is 0.245, compared with 0.53–0.77 for the GRUs. The claims “locality is the wrong structure” and “low rank is the right one” may therefore describe GRUs, not time series models in general.

The main result may also depend on the setup:
All error measurements use one setup: zero baseline, logit output and p = 1/2. But §7 shows that changing the setup moves the additive share by up to 32 points. The paper does not show that “low rank beats local beats additive” holds under other setups. Also, “rate” counts values, not how easy an explanation is to use. A rank-1 factorization is not clearly easier to read than a saliency map. The authors admit this, but there is no user study. Finally, the learned explainer fails on 2 of 6 models, and the paper has no way to predict where it will fail.

Key time series evaluation work is missing, so the novelty of §7 and §8 is unclear:
The paper says it makes the baseline problem and the weakness of perturbation metrics “operational for time series” (§2). But several papers already study perturbation-based evaluation of time series attributions: Turbé et al. (2023) evaluate post-hoc methods for time series classification against known ground truth; Schlegel & Keim (2023) study how the choice of perturbation affects time series XAI evaluation; Šimić et al. (2025) analyze many perturbation methods for validating attributions of neural time series classifiers; and Baer et al. (2025) show that perturbation effects differ by class. None of these is cited or discussed. The paper should explain what §7 and §8 add beyond this work. The §8 study is also small (one synthetic benchmark, five seeds), and the main keep-loss result is p = 0.058. Testing on the ground-truth benchmarks from these papers would make the metric claims much stronger.

Turbé, H., Bjelogrlic, M., Lovis, C., & Mengaldo, G. (2023). Evaluation of post-hoc interpretability methods in time-series classification. Nature Machine Intelligence, 5(3), 250–260.

Schlegel, U., & Keim, D. A. (2023). A deep dive into perturbations as evaluation technique for time series XAI. In World Conference on Explainable Artificial Intelligence (pp. 165–180). Springer.

Šimić, I., Veas, E., & Sabol, V. (2025). A comprehensive analysis of perturbation methods in explainable AI feature attribution validation for neural time series classifiers. Scientific Reports, 15(1), 26607.

Baer, G., Grau, I., Zhang, C., & Van Gorp, P. (2025). Class-dependent perturbation effects in evaluating time series attributions. In World Conference on Explainable Artificial Intelligence (pp. 292–314). Springer.

Questions for authors

Why are all UEA black boxes GRUs, and why is the TCN used only on MIMIC-III? Was this a compute limit or a deliberate choice?

Do the results in Tables 1 and 2 hold for a TCN or transformer black box on a UEA dataset?

Does the ordering “low rank > local > additive” stay the same under p = 0.25/0.75 or the first-observation baseline?

How do the findings in §7 and §8 differ from the perturbation-evaluation studies above (Turbé et al., 2023; Schlegel & Keim, 2023; Šimić et al., 2025; Baer et al., 2025)? Is the class-dependent effect reported by Baer et al. also visible in your gauge results?

Why does the learned explainer fail on Libras and JapaneseVowels, even though a per-input rank-4 fit reaches distortion 0.18 on JapaneseVowels (§9)?

The rate–distortion study is limited to P ≤ 300 (§11). What does a full pairwise fit cost in time and memory at that size, and can the method scale further?

4 comments

Thank you for the careful review. You asked whether the structural results describe GRUs or time series models in general, and that was the right thing to ask. They describe GRUs. We tested this directly and narrowed the paper.

One change affects several numbers below, so we mention it first. Five of the UEA test splits are stored sorted by class, so our "first N inputs" covered only one to three classes. The general comment explains this, and every per-input number here has been re-measured on a fixed class-stratified sample. Both findings you singled out still hold. On the four non-PenDigits GRUs, rank-1 beats the lag-1 band while using 1.7–9× fewer numbers: it gains 10–21 points over additive, and the band gains 3–7 (Table 1). Kendall τ between the zero and first-observation baselines is 0.25 over eight models.

W1 / Q1–Q2: does the structure result hold for a TCN or a transformer? It does not. We had used the TCN only as a comparator on the clinical task, so the submission never tested this. We trained a TCN and a transformer on the same five archives (P ≤ 300, test accuracy 0.80–0.99) and repeated the held-out comparison on the same inputs (Table 2; per model in Table A.6):

rank-1 beats bandlow rank beats band at matched rate*
GRU36/4039/40 (sign test p < 1e-10)
TCN8/4020/40 (p = 1.00)
Transformer5/4018/40 (p = 0.64)

*The largest rank whose rate does not exceed the band's.

The low rank-1 share on the TCN that you pointed to (0.245) was already a hint. TCN and transformer games are close to additive (additive distortion 0.03–0.17), and on them rank-1 loses to the band even though the band uses 1.6–9× more numbers. We checked whether a short receptive field could explain this. The TCN's 61 steps cover every sequence except ERing's 65, so it cannot. §5 is now titled "Which structure? It depends on the architecture". The abstract, contributions, related work and conclusion now claim the result for recurrent models only. We did not add InceptionTime or ResNet.

W2 / Q3: does the ordering survive other conventions? First, a correction to a number you quoted. On the stratified inputs the coalition measure moves the additive share by at most 14.0 points; we had reported 32. The bootstrap interval for this swing excludes zero on six of eight models (Table 3).

For the GRUs we reran the comparison under p = 0.25, under p = 0.75 and with the first-observation baseline (20 model × convention cells). The band beats additive on all 160 inputs. Low rank at matched rate beats the band on 157/160. Rank-1 alone beats it on 123/128 inputs of the four non-PenDigits GRUs (Table A.7). PenDigits (12/32) has only 8 steps, so a lag-1 band there is already almost global. TCNs and transformers were run under a single convention, and the Limitations say so.

On usability, we agree that rate counts numbers and says nothing about how easy an explanation is to read. We ran no user study, and the Limitations now state this too.

Q5: why does the estimator fail on Libras and JapaneseVowels when a per-input fit works? Your comparison points to the answer. The estimator can represent these games, but it does not generalize across inputs. On JapaneseVowels a per-input rank-4 fit reaches median distortion 0.11. The estimator's median R² is 0.85 on training inputs and 0.16 on test inputs (Libras: 0.73 and 0.07). The drop has two parts.

The first part is an offset of the whole coalition function on test inputs: 0.46–0.58 SD, against 0.01 on training inputs. This offset is the game's constant. No attribution contains it, so it lowers the fidelity score and leaves the attributions alone. Refitting only the intercept on 32 black-box queries per input brings median R² to 0.37 on Libras and 0.49 on JapaneseVowels. One extra scale takes JapaneseVowels to 0.61. The best any intercept and scale can do is 0.45 and 0.64, so 32 queries get most of the way there. The scale is positive on 93–100% of inputs, so on those inputs no attribution changes sign or rank.

The second part is that predicted and true values become less correlated (0.92 to 0.80 on JapaneseVowels, 0.86 to 0.67 on Libras). Recalibration does not help with this part, because it sits in the coefficients themselves.

The same analysis shows how to tell in advance whether the estimator will work. We trained it on 80% of the training inputs and scored the remaining 20%. Median R² was −0.32 to −0.01 on JapaneseVowels and 0.04–0.18 on Libras, against 0.55–0.57 on CharacterTrajectories and 0.72–0.89 on MIMIC-III. One RacketSports seed is a miss: 0.02 on held-out training inputs, but 0.24 on test. All of this is in §9 and App. J.7 (Table A.21).

The rest of our reply (W3/Q4 and Q6) is in the next comment.

W3 / Q4: perturbation-evaluation literature, Baer et al., and the size of §8. Thank you for the references. All four are now cited, and Related Work keeps two questions apart. Turbé et al., Schlegel & Keim and Šimić et al. use perturbation scores to evaluate attribution methods. Our §7 is about something else: the masking convention changes the attributions themselves, both their values and their rankings. §8 then gives a case with exact ground truth in which a perturbation score does not track whether an attribution is correct.

On Baer et al., we looked in two places. First, per-input τ between the two baselines does vary by class (class means −0.11 to 0.35 on Libras, 0.18 to 0.44 on ERing). With 32 inputs spread over up to 19 classes, however, a Kruskal–Wallis test reaches p < 0.05 only on ERing and RacketSports. We read this as consistent with Baer et al., but underpowered.

Second, the class sorting itself had a larger effect. On CharacterTrajectories the correlation between the certificate and convention sensitivity was 0.39–0.58 on class-0 inputs, and it is about zero on the stratified sample. On stratified inputs, an input's own additive share predicts its convention sensitivity on five of six models (ρ = 0.38–0.66). The certificate does so only on MIMIC-III and RacketSports. We have removed that claim, and §9 now presents the certificate only as a fidelity measure.

§8 now uses 20 seeds instead of 5. Paired over seeds:

  • At additive weight 0, keep loss does not separate the oracle from a random attribution (mean difference +0.04, p = 0.77). The p = 0.058 we reported before does not hold up. We fixed no equivalence margin, so we do not take this as evidence of no effect. At weight 0.3 keep loss does separate them (p = 5e-8).
  • Insertion and deletion AUC, AUPD and AUPP separate oracle from random at both weights (p ≤ 3e-4).
  • The ground truth separates the Owen and flat estimators at both weights (Spearman +0.066, p = 2e-5, and −0.046, p = 4e-7).
  • Only one perturbation score also separates them at each weight: deletion AUC at weight 0 (p = 0.025) and drop loss at weight 0.3 (p = 1e-5). Drop loss ranks them in the opposite order to the ground truth.

The revised §8 makes this narrower claim. We could not run the ground-truth benchmarks of Turbé et al. during the discussion period, and we agree they would make §8 stronger.

Q6: cost of the full pairwise fit, and scaling. For the held-out comparison (8 inputs, 10 decoders, one 12 GB Titan Xp plus CPU):

  • ERing (P = 260): a run takes 50 min to 2 h 04 min, with 26.6 GB peak host memory.
  • JapaneseVowels (P = 300): a run takes 1 h 08 min to 2 h 11 min, with 46.5 GB.

These figures are now in App. B.2. Memory is dominated by the Gram matrix of the degree-2 basis, which has about P²/2 columns (45,151 at P = 300). It therefore grows as P⁴, which is why the full-basis measurements stop at P ≤ 300.

The full basis is only needed to fit and certify every decoder in the comparison. A rank-r explanation itself has P·r numbers, and the amortized estimator never builds the basis; it runs at P = 768 on MIMIC-III. Fitting rank-r structure directly from coalitions would remove the limit. We have not done that yet, and the paper states the P ≤ 300 limit and the measured costs.

If something is still not convincing, please tell us, and we will work on it before the discussion closes.

Thank you for the detailed response and the new experiments, which address most of my concerns. I appreciate that the authors narrowed the claims to GRU models instead of presenting the GRU result as general, and that they reported the class-sorting issue openly. The Limitations should still state that the evaluation uses only five UEA datasets and MIMIC-III, includes no CNN-based classifiers, and tests non-recurrent models under a single setup. I keep my score.

Thank you for reading the revision and for the follow-up. We added the three points to the Limitations and uploaded the revised PDF. It now says that every model is a classifier on one of seven UEA archives or MIMIC-III, that the TCN is our only convolutional classifier (no InceptionTime or ResNet), and that each TCN and transformer is a single trained model measured under one convention. To be precise about the data, the GRUs cover seven UEA archives and the TCN and transformer comparison uses five of them. Nothing else changed in this revision.

Reviewer_yjPSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the issue of framing of time-series feature attribution dependent strongly on how the explanation problem is defined. The authors highlight a trade-off between the explanation fidelity and representation size from the distortion-rate perspective.

Critical strengths

(1) This paper is well-motivated with an overlooked fundamental questions of time series attribution. (2) This authors propose a distortion-rate formulation of explanation, which makes the evaluation and comparison clear. (3) Table 1 shows the in-sample energy share. Table 2 rebuilds classes as fitted decoders and scores on 20,000 fresh coalitions of eight disjoint inputs, and confirms the ordering.

Critical weaknesses

(1) The estimator performs well on some models and fails on others. The paper does not analyze which settings will lead to failure. (2) The experiments show an interior optimum, but the mapping from the query budget to representation rate is not clear for how to do the selection. (3) The presentation of the paper could be clearer.

Questions for authors

Whether optimal complexity can be selected with a rule give a query budget?

1 comment

Thank you for the review. Both of your questions now have a direct answer in the paper. As the general comment explains, five UEA test splits turned out to be sorted by class, so every per-input number below comes from a fixed class-stratified sample.

Q / W2: can the complexity be chosen from the query budget? Yes, using only the coalitions already queried. For each input we fit the full pairwise expansion on n coalitions, rank its C coefficients by estimated energy, and keep those above a threshold. The resulting explanation is scored on 20,000 fresh coalitions. We compared three standard thresholds, each computed from the same n queries (σ̂² is the full fit's residual variance). The test covers all five GRUs, eight inputs each, at budgets from 2 to 147 coalitions per coefficient (17 settings). Regret is the held-out distortion above the best truncation, in points of explained variance:

ruleworst setting (mean)worst single input
keep every coefficient8.420.9
noise floor, b̂² > Var(v)/n3.714.7
Mallows' Cp, b̂² > 2σ̂²/n6.616.5
universal threshold, b̂² > 2 ln(C) σ̂²/n1.64.8
validation draw (20,000 extra queries)0.090.45

The universal threshold (Donoho & Johnstone, 1994) has the smallest worst case, so we recommend it when a validation draw is not affordable. It is conservative at large budgets but costs at most 1.1 points there. The best rate grows with the budget; on RacketSports it goes from 1,030 to 3,832 numbers as n/C goes from 2 to 20. This is the budget-to-rate mapping you asked about. One limit: the rule needs the full pairwise fit, so it applies from about n = 2C queries, and below that a structured decoder has to be chosen without it. §6 states the rule, and Table A.8b gives the chosen rate and regret per dataset and budget. The submission reported only the noise-floor count; on this larger sweep it is not the best rule.

W1: when does the estimator fail? It fits MIMIC-III, CharacterTrajectories and RacketSports. It fails on Libras and JapaneseVowels, where median R² drops from 0.73 and 0.85 on training inputs to 0.07 and 0.16 on test inputs. Capacity is not the issue: a rank-4 fit per JapaneseVowels input reaches median distortion 0.11. The estimator fails to generalize across inputs, in two ways.

First, the whole coalition function is offset on test inputs (0.46–0.58 SD, against 0.01 on training inputs). The offset is the game's constant, which no attribution contains. It lowers the fidelity score without changing the attributions, and it is cheap to remove. Refitting the intercept on 32 queries per input raises median R² to 0.37 on Libras and 0.49 on JapaneseVowels, and adding one scale lifts JapaneseVowels to 0.61. That closes about half the gap to training inputs and comes close to the best any intercept and scale can reach (0.45, 0.64). The scale is positive on 93–100% of inputs, so on those inputs no sign or ranking changes.

Second, predicted and true coalition values become less correlated (0.86 to 0.67 on Libras, 0.92 to 0.80 on JapaneseVowels). This part sits in the coefficients and remains open. On Libras, an additive fit from scratch on 750 queries (0.55) beats every amortized variant.

The failure can also be detected before deployment, from training data alone. Training on 80% of the training inputs and scoring the other 20% gives median R² of −0.32 to −0.01 on JapaneseVowels and 0.04–0.18 on Libras, against 0.55–0.89 on CharacterTrajectories and MIMIC-III. There is one miss, on a RacketSports seed. We treat this as a diagnostic; we have not calibrated it as a predictor (§9, App. J.7, Table A.21).

W3: presentation. We agree, and Reviewer KWCS raised the same point. The introduction now frames the paper as one question: how much of a declared value function can an explanation of a given size carry? It works through an example and says which section answers which part. A new figure (Fig. A.1) shows what lower distortion adds on real inputs. A rank-one term predicts held-out long-range interactions that Shapley values cannot express: a median 92% of their variance on RacketSports, 52% on Libras and 48% on PenDigits.

We hope this answers both questions. If it does, we would be grateful if you would reconsider your score, and if something is still missing, please let us know.

Reviewer_KWCSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies how complex a time-series explanation needs to be to capture the behavior of a model. It proposes measuring the trade-off between explanation complexity and how well it reconstructs the model's coalition value function. The authors find that low-rank explanations work better than additive or local ones, and that attribution results depend heavily on the masking choices. The paper also question the reliability of common evaluation metrics and propose a method (an amortized estimator)to generate explanations with a measure of their fidelity.

Critical strengths
  • Interesting conceptual perspective. The paper proposes viewing feature attribution as a lossy compression problem, analyzing the trade-off between explanation complexity and fidelity.

  • The paper provides evidence that low-rank representations can capture feature interactions more efficiently than additive or temporally local explanations.

Critical weaknesses
  • Clarity: While the paper is technically precise, I found it very hard to follow, and I can't overstate this enough. It's very hard for me to asses postively a paper when I need to read a paragraph multiple times to understand it. The central idea gets obscured by the large number of experimental protocols, metrics and theoretical details. In particular, the paper moves between several related but distinct questions, making it difficult to identify the main contribution and follow a coherent storyline. I would encourage the authors to simplify the presentation, provide more intuition and move some of the experimental details and ablations to the appendix.

  • Unclear connection between coalition-function and what is a good explanation. A major concern for me is that the paper assumes that explanation fidelity is the same the ability to reconstruct the coalition value function. However, it is unclear whether a lower reconstruction error translates into better / more useful explanations. Similarly, measuring explanation complexity through the number of coefficients does not necessarily reflect interpretability.

  • Most conclusions rely on relatively small experiments. The main distortion–rate analysis covers only five models with at most 300 input cells, while the coalition-measure analysis uses just eight inputs per model and a single trained-model seed, without quantifying uncertainty. Moreover, all real-world experiments focus on classification tasks, primarily using UEA and MIMIC-III datasets, and its unclear wether the rsults generalize.

Questions for authors
  • Could the authors clarify what they consider the central contribution of the paper and provide a simpler, more intuitive explanation of the distortion–rate framework? In particular, how do the different experiments connect to the main research question?

  • To what extent does accurately reconstructing the coalition value function translate into better feature attributions? Could the authors provide empirical evidence that lower reconstruction error leads to more accurate or useful explanations?

3 comments

Thank you for a frank review. We agree that clarity was the main problem, and we organized the revision around it.

Q1: what is the central contribution? An attribution method hides subsets of an input's cells and records how the model's output changes. The output for every subset is the value function v, and a Shapley map summarizes v with one number per cell. We ask how much of v an explanation of a given size can reproduce. Distortion is the share of v it fails to reproduce, and rate is how many numbers it uses. Every kind of explanation then becomes a point on one plot, the way lossy codecs are compared.

The introduction now says which section answers which part of this question:

  • §4: how to measure it.
  • §5: which explanations of a given size reproduce the most (it depends on the architecture).
  • §6: how large an explanation should be for a given query budget.
  • §7: why v must be declared first (the masking choice changes the attributions).
  • §8: why perturbation scores cannot answer the question instead.
  • §9: whether one network can answer it for every input, and what checking costs.

The introduction also gives a worked example. Suppose an event at time 3 matters only when an event at time 40 is present. An additive map cannot express this. A full pairwise explanation can, but it spends a number on every pair. A rank-one term can express it with two numbers per cell.

Q2: does lower reconstruction error give better explanations? Yes, in the sense that it gives more accurate interactions, which we can check against ground truth.

PenDigits is small enough to evaluate all 65,536 coalitions exactly. On it, ordering explanations by distortion orders them by the error of their pairwise interactions (Spearman 0.97 within each of 32 inputs; App. G). Their order-1 Shapley values are identical, so accuracy on Shapley values alone could not tell them apart.

On real inputs (RacketSports, 180 cells; Fig. A.1), we took 200 random pairs of distant cells. For each pair we measured how much hiding both changes the output beyond hiding each one, on contexts the fit never saw. Shapley values predict zero for all of these interactions. A rank-one term explains a median 92% of them (52% on Libras, 48% on PenDigits).

What this does not show is usefulness to a person. Counting numbers is not the same as interpretability, and we ran no user study. The Limitations now say both.

W3: small experiments. The structure comparison grew from 5 GRUs to 15 models: GRU, TCN and transformer on five archives, eight inputs each, with sign tests. Low rank beats locality at matched rate on 39/40 GRU inputs (p < 1e-10), but not on TCNs or transformers (20/40, 18/40). So the summary in your review, that low-rank explanations work better, holds for recurrent models only, and the abstract now says so. On GRUs the result also holds under four masking conventions (157/160 inputs, Table A.7).

We also found that five UEA test splits are sorted by class (see the general comment), so per-input results now use a fixed class-stratified sample. The coalition-measure sweep uses 32 inputs per model with bootstrap intervals (Table 3). Its largest swing is 14.0 points, not 32, and the interval excludes zero on six of eight models. §8 went from 5 to 20 seeds.

Some limits remain: one trained model per dataset in the sweep, full pairwise measurements only up to 300 cells, and classification tasks only. The framework needs only a scalar output, but we have not tested regression. The Limitations state all three.

If any section is still hard to follow, please tell us which one and we will revise it during the discussion. If the revision answers your two questions, we would be grateful if you would reconsider your score.

Thank you for taking the time to improve the clarity of the paper and additional experimetns, specially the inclusion of other architecures beyond GRUs and transparency about the genrelaization of the original findings.

However, despite this, I still find the paper difficult to follow an understand, and I would recommend the authors to work on presentation and clarity. I am also concerned about the general relevance (e.g. only classification tasks) of the findings, and limited statistical evidence (e.g. still one trained seed per model)

Overall, I appreciate the authors' efforts and recognize that the paper has improved, but I still cannot recommend acceptance.

Thank you for reading the revision and for saying clearly where it still falls short. We think all three points are fair. Two of them are now stated in the Limitations: every real model is a classifier, and the architecture comparison and the coalition-measure sweep each use one trained model per dataset and architecture. We will keep working on the presentation.

Comments on the paper

1 comment

We thank the three reviewers. They asked for a clearer paper (KWCS, yjPS), evidence that lower distortion gives a better explanation (KWCS), and tests beyond our original setup (j5vz, KWCS). The revised PDF and supplement are uploaded, and each reviewer has a separate reply. This comment covers one problem we found ourselves, because it affects numbers in all three replies.

Class-sorted test splits. Five UEA archives (JapaneseVowels, CharacterTrajectories, RacketSports, Libras, BasicMotions) store their test split sorted by class. Our per-input experiments used the first N test inputs, which on these archives covered only one to three classes. PenDigits, ERing and MIMIC-III are not affected. The amortized estimator on CharacterTrajectories (§9) had a related problem: it was trained on the first 256 training inputs, which cover 4 of the 20 classes. It now trains on all 1,422.

Each archive now has a fixed, seeded, class-balanced evaluation sample, and we re-measured every per-input result on it. The supplement contains the sample indices, the new results, and the submitted results, so the two can be compared.

What held. Rank-1 still beats the lag-1 band on four of five GRUs (the same exception, PenDigits), at 1.7–9× fewer numbers (2–9× before). Kendall τ between the zero and first-observation baselines is 0.25 over eight models (0.267 over five). A 500-query audit still reproduces the 8,000-query certificate (Spearman 0.988–0.993). The estimator still needs a receptive field that covers the series.

What narrowed. Over additive, the band gains 3–7 points and rank-1 gains 10–21 (we had 3–9 and 21–30). The largest coalition-measure swing in the additive share is 14.0 points, not 32. Top-10% agreement between the two baselines is 20%, not 32%.

What changed, all on CharacterTrajectories in §9:

  1. The certificate no longer predicts convention sensitivity; an input's own additive share does better (5 of 6 models). §9 now presents the certificate as a fidelity measure only.
  2. The interaction head's gain depends on the encoder (+0.09 convolutional, +0.16 attention, about 0 for a GRU), rather than +0.19 to +0.25 for every encoder.
  3. A per-input fit already beats the amortized estimator at 30k queries (0.54 vs 0.42). We dropped the claim that amortization matches it.

Separately, the TCN and transformer experiments that j5vz asked for narrowed the main claim. Low rank beats locality on GRUs (39/40 inputs at matched rate) but not on TCNs or transformers (20/40, 18/40). The abstract, contributions and conclusion now say "recurrent models".

Two results are still on the submitted inputs and are labelled as such: the training-pool cut (App. J.5) and the per-input budget crossover (App. J.4, Table A.17). No claim in the abstract depends on them.

Where the main changes are.

  • §1: one-question framing, a worked example, and a map of which section answers what (KWCS, yjPS).
  • App. G and Fig. A.1: what lower distortion buys, checked against ground truth (KWCS).
  • §5, Tables 2, A.6 and A.7: TCN and transformer, and four masking conventions (j5vz, KWCS).
  • §6 and App. F: a rule from query budget to rate (yjPS).
  • §8: 20 seeds instead of 5 (j5vz).
  • §9 and App. J.7: why the estimator fails, and a 32-query recalibration (yjPS, j5vz).
  • App. B.2: measured cost (j5vz).
  • Limitations: classification only, no user study, full pairwise fits only up to 300 cells.

One script cited in Appendix A (measure_per_input_aer.py) was missing from the submitted supplement. We reconstructed it; it reproduces the submitted control file exactly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.