acceptodds
Under review as a conference paper at ICLR 2027

PROVENANCE NETWORKS: END-TO-END EXEMPLARBASED EXPLAINABILITY

Abstract

We introduce provenance networks: a shared backbone feeds a task branch (e.g. classification) and a jointly trained index branch that predicts each input’s exemplar index within the training set, making the network behave like a learned, end-to-end k-nearest-neighbor classifier. Training-set index accuracy alone is a poor measure of usefulness: a model reaching 96–98% index accuracy on MNIST can still fail almost completely at recovering a known source for a perturbed test input. We introduce a test-time, ground-truth-backed source-recovery protocol (MNIST, FashionMNIST, CIFAR-10) that exposes this gap and show it closes once the index branch is trained with augmentation matched to the evaluation perturbations – at which point it decisively beats an embedding-space kNN baseline, most dramatically on CIFAR-10 (70.8% vs. 2.2% top-1 under the strongest perturbation). A ProtoPNet-style baseline is structurally incapable of this task, capped near 0% by construction despite comparable classification accuracy. We further stress-test a proposed use case, dataset debugging via index-branch entropy, against synthetic label noise, and find the opposite result: entropy performs at chance for mislabeled-data detection. On scalability, a subset of training exemplars preserves task accuracy with little loss, and a factorized index head cuts the index layer’s parameters by 41–500× but converges substantially more slowly. Together, these results argue for evaluating exemplar-attribution methods against test-time, ground-truth-backed protocols rather than training-set accuracy alone.

Primary area:
interpretability and explainable AI
Keywords:
provenance networks, exemplar attribution, source recovery, kNN, interpretability, training data attribution, dataset debugging, MNIST, CIFAR-10, retrieval, index prediction, explainable AI, prototype learning, data provenance.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gtiEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Provenance Networks, which augment a standard model with an index branch for retrieving specific training examples as exemplars at test time. The authors consider single-branch and two-branch variants. Their evaluation tests whether memorized training indices can be recovered from perturbed inputs, with augmentation-invariant training substantially improving source recovery. Comparisons are provided against embedding-based kNN and prototype-based methods. Scalability is further examined through exemplar subset selection and factorized index heads.

Critical strengths
  1. The paper presents a simple and intuitive architecture that directly links model predictions to retrievable training exemplars.

  2. The paper is generally clearly written and includes a broad empirical evaluation across multiple datasets and settings.

Critical weaknesses
  1. It is unclear why retrieving every individual training sample is more useful for explanation than using representative prototypes, especially given prototype-based methods with tree-structured designs that can already provide fine-grained explanations. This finer granularity of this work comes with substantially worse scalability and performance. Moreover, although every training instance is theoretically retrievable, the paper does not show that test-time explanations actually make use of this broad coverage. No retrieval-coverage or diversity analysis is provided.

  2. The task and index branches are jointly trained, but the paper does not show that the task prediction is strictly based on the retrieved sample. The only support is empirical retrieval performance rather than a theoretical or causal guarantee.

  3. Treating every training sample as a separate prediction target is also questionable from a learning perspective. Modern models are expected to learn patterns that generalize beyond discrete training instances, while the proposed index objective explicitly encourages instance-level memorization. The paper does not convincingly show that this formulation can preserve generalization on more complex tasks.

Questions for authors
  1. What evidence shows that the retrieved training sample actually explains the task prediction, rather than simply being correlated with it through the shared representation?

  2. Why is it necessary to make every training sample individually retrievable, especially when the paper does not show that such broad coverage is used in practice?

Reviewer_wBwROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Provenance Networks, an architectural style that attaches a Training Set Index Predicting Branch to the backbone of standard neural networks. The Task-Specific and Index Predicting branch are trained simultaneously, along with the shared backbone. No data augmentation is used by default. At inference time, the Index Predicting branch retrieves training samples which act as \textit{Provenance} for the current input.

For the dual branch architecture, the Authors test both class-conditional and class-independent index predicting. They also propose schemes for reducing the number of parameters on the index predicting branch (data subsampling, factorized index head). The Authors also test a single branch variant that is trained only with the index predicting task. At inference, they use the top-k retrieved samples to assign a class to the input in a similar manner to a kNN classifier.

To test whether the index predicting branch can be used for training data \textit{attribution}, the Authors forward perturbed training samples through the index predicting branch and measure its accuracy. In the default setting, the index predicting branch performs worse than a simple embedding-space kNN retriever when rotations or translations are used. However, using these transformations as data augmentations in the training phase reverses the ordering.

In Sec. 7.2, the Authors find that the maximum post-softmax confidence of the index predicting branch performs well for membership inference, achieving near-perfect AUROC. In 7.3 they attach a class and index predicting branches to a VAE, finding on a qualitative analysis that images retrieved by the latter closely resemble the generated image.

Critical strengths

None

Critical weaknesses

Experiments are conducted on small datasets, mostly using LeNet-like networks. The conclusions need not translate to more practical settings, where the scalability of the index predicting branch is also a major problem. While the Authors test a factorized index head to address the practical limitations of this branch, the results are well below the flat head baseline. Also, simple data re-ordering can greatly affect the performance of such a head architecture. This factorized head is sharing weights for randomly matched samples. Thus, the poor results may have been expected. The architecture choice makes the backbone parameter sharing experiment in Sec. 5.4 rather uninformative. Less than 5% of the total parameters are ever shared (only those in the conv layers), while the vast majority remain in the task-specific linear layers. This setting does not seem solid enough for generalizing conclusions.

Weakly supported findings. Sec. 7.3 only presents the conclusions of a qualitative analysis. In Sec. 7.1 the Authors simply state that “intermediate memorization (∼20–30%) appears to improve robustness for some distortions (occlusion, blur). We have not isolated why [...]”. In Sec. 7.2 the Authors state that they “have not compared against dedicated membership-inference attacks from that literature.“, leaving the results presented there uncontextualized. The evaluation of \textbf{attribution} in Sec. 6 also does not seem that well-designed. Simple accuracy may not be a good proxy for proper attribution, since the rotation/translation on input samples on datasets such as MNIST can arguably make the current sample more similar to a different one, rather than its original state. Applying perturbations to the training inputs implies that what the experiment tests is simply the perturbation invariance. It is only natural for the results to improve when explicitly training for invariance to those perturbations.

Questions for authors

No specific questions.

To my knowledge, [Balestriero, 2023] is the earliest work using the index predicting training objective. I think that the Authors should cite this source (or an earlier one if it exists) when introducing the index predicting branch.

Balestriero, Randall. "Unsupervised learning on a diet: Datum index as target free of self-supervision, reconstruction, projector head." arXiv preprint arXiv:2302.10260 (2023).

Reviewer_o8MBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper jointly trains a classifier and an index-prediction branch to link inputs to training examples. Perturbed-image retrieval experiments show that augmentation is important for outperforming embedding-space kNN. The paper also explores scalability, membership inference, robustness, and generation, mainly on small image datasets.

Critical strengths

S1. The paper addresses a relevant problem: linking model predictions to specific training examples.

S2. Within the proposed evaluation protocol, the method shows some practical value, e.g., outperforming embedding-space kNN after augmentation-invariant training.

S3. The paper clearly distinguishes provenance, attribution, and influence, and avoids overclaiming causal effects.

Critical weaknesses

W1. Predicting training-sample identities under augmentation is closely related to Exemplar-CNN (Dosovitskiy et al., 2014) and instance discrimination (Wu et al., 2018). These approaches are not discussed, leaving the methodological novelty unclear.

W2. Recovering perturbed training images mainly evaluates source retrieval, rather than whether retrieved examples explain predictions on unseen inputs. Although both methods use matched augmentation, the index branch receives instance-level supervision while the kNN baseline uses classification-trained embeddings.

W3. Important experimental details remain insufficiently documented. Table 4 marks the CIFAR-100 and Stanford Dogs backbones as “not independently verified,” while the main post-augmentation retrieval results are reported for a single seed. This limits reproducibility and confidence in the reported gains.

Minor weaknesses

m1. The appendix has minor structural issues: Section 9.15.1 is empty, and “9.17 Decoder” appears to belong under Section 9.16 alongside the encoder and reparameterization subsections.

m2. Table 4 references “dependence 1 cifar10 better.py” without an access location, and Appendix 9.9 mentions “ogDP” without explanation. Both references need clarification.

Questions for authors

Please refer to weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.