acceptodds
Under review as a conference paper at ICLR 2027

Audio Modeling Through the Lens of Epiplexity

14 views

Abstract

Sound contains rich information about the surrounding environment and has become an increasingly important modality for machine learning systems that aim to understand, detect and respond to real-world events. While considerable progress has been made in audio modeling, generalization across domains remains difficult because of limited yet highly diverse training datasets, variation in recording conditions, and other factors. This motivates the search for datasets and processing techniques that support generalization across domains. We hypothesize that the recently introduced concept of epiplexity, the amount of structural information in a dataset which can be learned by a computationally bound observer, offers a way to assess how these datasets and processing techniques contribute to model generalization. Prior work on text, images, and video has found that models trained on datasets with higher epiplexity seems to generalize better to unseen downstream tasks (Finzi et al., 2026). This opens the door for understanding how to select datasets that support training more generalizable models. However, the behavior of epiplexity in the audio domain remains unexplored. We aim to use epiplexity to better understand what kinds of audio datasets can lead to improvement generalization in models. We measure the epiplexity of over 40 different audio datasets and the correlations between the epiplexity of 12 training datasets and downstream model performances over three tokenizers: DAC, EnCodec, and X-Codec. We find that while tokenizers historically known to produce better performance yield lower epiplexity, for a fixed tokenizer, epiplexity is usually weakly correlated to improved model performance. We contribute a method for computing epiplexity and conditional epiplexity for audio datasets using PyTorch. Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, J. Zico Kolter, and Andrew Gordon Wilson. From entropy to epiplexity: Rethinking information for computationally bounded intelligence, 2026. URL https://arxiv.org/abs/2601.03220.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Machine Learning, Epiplexity, Audio, Audio Datasets

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GqQhOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the applicability of epiplexity to audio modeling by estimating unconditional and conditional epiplexity across various audio datasets tokenized with DAC, EnCodec, and X-Codec. The authors report indicate generally weak relationships between epiplexity and model performance that depend on the tokenizer and evaluation setting, with conditional epiplexity showing more consistent positive associations with classification performance.

Critical strengths

Introducing epiplexity to audio domain

This work provides an empirical starting point for understanding how epiplexity behaves on a diverse collection of audio datasets and how it relates to model generalization.

Critical weaknesses

I assigned the lowest confidence score because of my limited familiarity with epiplexity. However, drawing on my background in audio machine learning, I identified several concerns regarding the paper’s experiments.

1. Insufficient justification of re-purposing dataset tasks and evaluation protocols

In Figure 4, the authors investigate aggregated correlations between conditional epiplexity and classification performance across datasets originally designed for substantially different tasks. For example, STARSS23 and LOCATA were developed for sound event localization and detection (SELD) and acoustic source localization/tracking, respectively. Furthermore, SynTheory datasets (Chords, Intervals, Notes, and Scales) were designed around specific music-theoretic concepts.

However, Appendix A (Table 2) and Appendix B.2 (Table 10) specify origin for STARSS23, task for LOCATA, and midi program name for the SynTheory subsets as classification targets. According to the provided implementation, STARSS23's origin refers to the recording institution (Sony or TAU: Tampere University), LOCATA's task refers to one of its six recording scenarios, and SynTheory's midi program name represents instrument identity. These classification tasks differ substantially from the datasets' original research objectives, and the manuscript does not adequately justify their use as alternative evaluation tasks.

For example, STARSS23 conventionally evaluates sound event detection and localization using metrics such as ER20, F20, localization error (LE), and localization recall (LR). In contrast, Tables 11–12 report weighted F1 and classification accuracy for recording-origin classification, which cannot be interpreted as performance on the original SELD task. Similarly, classifying LOCATA's recording scenarios does not measure acoustic source localization or tracking performance. Moreover, the provided implementation selects only the first audio channel for DAC, EnCodec, and X-Codec, thereby discarding the multichannel directional information contained in FOA recordings.

The manuscript does not adequately justify these alternative classification targets or establish whether Figure 4’s correlations are robust to task selection, limiting the support this central experiment provides for its broader conclusions about audio-model performance and generalization.

2. Limited practical implications for audio dataset selection

In the Introduction, the authors aim to investigate "What audio datasets might be the best to train on from the perspective of epiplexity and generalization?" (page 2, L61-62). While the empirical findings may provide useful insights into the behavior of epiplexity in the audio domain, their practical implications for dataset selection remain unclear. In particular, the manuscript does not establish the practical value of measured epiplexity as a criterion for selecting training datasets to improve downstream generalization.

Questions for authors

Unclear connection between spatial reasoning and the experimental evaluation

In the Introduction (page 1, L47–51), the authors suggest that improving the generalization capabilities of audio models could lead to better spatial reasoning, contrasting this with the limitations of text models. However, although the manuscript includes several spatial-audio datasets, it does not evaluate spatial reasoning or localization capabilities, such as source azimuth, elevation, or distance prediction. Also, the results and discussion do not establish how the reported epiplexity measurements relate to spatial reasoning. Could the authors clarify the intended connection between the spatial-reasoning motivation and the experimental findings?

1 comment

Thank you for the response! We discussed spatial audio to motivate the work [and without saying much, lay groundwork for future work we plan to do). However, we see that we may have overemphasized it. In truth, we have no intention of replicating the SELD experiments for those datasets in this paper. We will edit the manuscript to make this point clearer and to deemphasize spatial audio in the introduction.

Now some quick small notes:

  1. We will edit the manuscript to justify the classification choices. In brief, conditional epiplexity is about a classification task, not the original research question in those datasets. We believe it's fair to derive new classification questions from the dataset as we do not compare to previous results on that dataset to answer our research questions about epiplexity and generalization performance.
  2. Your concerns about dataset selection are valid, and brought up by the other reviewers as well. We believe that working with Reviewer oXoF’s points should help with this concern as well.
Reviewer_L9b1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduced epiplexity, a metric for structural information that can be learned from data by models with computational constraints, into the audio domain, and attempted a systematic verification of the relationship between the structural characteristics of acoustic signal datasets measured with epiplexity and model generalization performance. Inspired by prior work in the text and image domains showing that data with higher epiplexity yields better generalization performance, this paper tested its effectiveness in the acoustic signal domain. The results using more than 40 audio datasets and 3 tokenizers showed that while unconditional epiplexity, measured on a general-purpose task, has a weak relationship with performance on unseen or out-of-domain data. On the other hand, it showed that conditional epiplexity, measured with classification tasks is more strongly correlated with task-specific generalization performance.

Critical strengths

Epiplexity is an information-theoretic concept proposed in recent years as a method for quantifying the amount of structural information that AI models with computational constraints can practically learn. It will be a useful measure, because it is expected to be utilized for efficient dataset construction, curriculum learning design, synthetic training data generation, and efficient improvements in generalization performance. As the authors argue, there are no known previous examples of epiplexity being applied to acoustic signal datasets to my knowledge. Therefore, this paper is considered technically novel. Furthermore, the study includes a fairly large-scale experimental analysis, providing valuable insights to researchers in the related fields.

Critical weaknesses

Although this paper attempted to introduce epiplexity into the domain of acoustic signals, it is difficult to say that it presents entirely positive results. The fundamental goal of epiplexity is to determine how much structural information can be extracted, while assuming realistic model classes and computational constraints, without being heavily dependent on specific models or tasks. As referenced in the paper, prior work reported that data with high unconditional epiplexity improves generalization performance across downstream tasks in the text, images and video domains. However, in this paper, such results were not obtained for the unconditional epiplexity. The paper then introduced conditional epiplexity and concluded that it better correlates with the accuracy of downstream classification tasks. However, introducing a specific task makes the process closer to training a proxy model for that task and measuring performance against it, just as in conventional approaches. It might be considered that this represents a step backward from the original concept and purpose of epiplexity.

Questions for authors

Would it be possible to provide a more direct explanation regarding why unconditional epiplexity does not correlate as strongly with the generalization performance of downstream tasks as text or images do?

1 comment

Thank you for the review! There will be some more additional work to try to strengthen these weaknesses a bit. Namely:

  1. You are right about conditional epiplexity. We are planning to include a follow up experiment in the revised draft by Friday that includes work comparing two related but slightly different domains in terms of conditional epiplexity to test generalization explicitly.
  2. To answer your question, we will improve the writing to clarify this point. In short Finzi et al found that text datasets had more epiplexity than other modalities such as video and images. That also found separately that datasets with higher epiplexity had better generalization. They inferred based on these two points that epiplexity in text can explain why text models have been so successful in the last few years. To better address this point, we will add an experiment if we have time estimating the scaling laws for the amount of compute for epiplexity, similarly to the paper. We had the experiment prepared but did not have time to fully collect results. We aim to have those experiments done by Monday at the latest, with partial results ready by Friday.
Reviewer_oXoFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Epiplexity is a recently proposed from Finzi et al. that measures the amount of structural information that can be learned by some computationally bounded model. This paper applies the notion of epiplexity to audio dataset generalization. This is a compelling application, as audio datasets can vary a lot in how much randomness they have inherently and how regular patterns can be (songs versus speech transcripts). Unlike typical information-theoretic measures (ex: shannon entropy), epiplexity quantifies the ability for a model to actually learn regularities under computational constraints. I found this really interesting because with such a framework, I would assume some sort of temporal modeling (maybe like TPP), but the epiplexity measure seems to naturally account for this pattern-capturing.

  • The authors perform two main experiments in showing how epiplexity can measure audio dataset generalization. The first set of experiments measure unconditional epiplexity with 40+ datasets with three neural audio tokenizers. They use a prequential coding estimator based on autoregressive transformer training. The main metric is whether the epiplexity correlates with next token prediction performance, both zero-shot and after fine-tuning. Second, they estimate conditional epiplexity, where they measure the structure learned when predicting labels in a classification setup (using 24 labeled datasets).
  • The empirical results are a bit mixed. Unconditional epiplexity does not consistently predict better cross-domain performance, while conditional epiplexity exhibits modest positive correlations with classification accuracy/F1-score. The authors also observe substantial differences across audio tokenizers. The findings motivate a distinction between the amount of structure available to learn from a distribution and the relevance of that structure to a particular prediction task.
Critical strengths
  1. Interesting research question and application of epiplexity. I think the central research question is compelling, particularly because understanding what makes a dataset useful for generalization is still quite difficult. Audio seems like a natural setting to investigate this, given how different the underlying structure can be across speech, music, environmental sounds, etc. I especially appreciate that the authors are trying to use a theoretically motivated measure to understand properties of training data, rather than relying only on empirical downstream performance. Even if epiplexity ultimately does not predict generalization, understanding when and why this happens seems like a useful contribution.

  2. Broad empirical investigation across datasets and tokenizers. The authors put considerable effort into evaluating epiplexity across a large collection of audio datasets and different tokenizers. I think this is valuable given how expensive the estimation procedure is. I also found the tokenizer comparisons interesting because they suggest that the amount of learnable structure depends not only on the underlying data but also on how we represent it. This raises a broader question about whether some representations make certain kinds of structure more computationally accessible than others.

  3. Exploring conditional epiplexity is a promising direction. I liked the authors' decision to investigate conditional epiplexity rather than only measuring structure in the audio distribution itself. A recording can contain many regularities that are learnable but not necessarily useful for a particular task. Measuring structure relative to labels seems like a natural way to start thinking about which information is relevant for learning. I also appreciate that the authors discuss their mixed findings rather than presenting epiplexity as a universally useful measure. In particular, their suggestion of investigating shared structure between datasets seems like a promising direction that follows naturally from the results.

Critical weaknesses
  1. I am not entirely convinced that epiplexity should be expected to correlate with generalization in the first place. The motivating hypothesis is that datasets with more learnable structure should produce models that generalize better. However, I think there is an important distinction between how much structure a model can learn and how much of that structure is useful for another task. Two datasets could have equally high epiplexity while containing completely different regularities. Similarly, a dataset with relatively low epiplexity could contain exactly the information needed for a downstream task.

This makes me wonder whether the lack of correlation in Experiment 1 is actually surprising, given what epiplexity measures. I would have liked a more precise argument for why unconditional epiplexity should predict performance across different audio distributions.

The authors briefly suggest investigating joint epiplexity, which I think is interesting. One possible direction would be to ask how much pretraining on one distribution reduces the prequential coding cost of learning another. This seems closer to measuring the amount of reusable structure rather than just the total structure in either dataset.

Relatedly, the two experiments are not evaluating the same notion of generalization. Experiment 1 studies cross-domain next-token prediction, while Experiment 2 correlates conditional epiplexity with classification performance on held-out examples from the same dataset. I think the latter is still useful, but it does not directly establish that conditional epiplexity predicts transfer across distributions. I would encourage the authors to make this distinction clearer in their conclusions.

  1. It is unclear whether conditional epiplexity captures something beyond how learnable the classification task already is. The conditional epiplexity estimator measures accumulated improvements in label-prediction loss during training. However, classification accuracy and F1-score also depend on how successfully the model learns the relationship between audio and labels. This makes me wonder whether the positive correlations reflect structural complexity or simply the fact that some classification problems are easier for the chosen model to learn (label noise could also just be a possibility?).

For example, if the labels are independent of the audio, a model may have low estimated conditional epiplexity because it cannot learn meaningful predictive structure, and it would also achieve poor classification performance. Conversely, a dataset with a learnable input-label relationship may have both higher estimated epiplexity and better classification performance. I do not think this necessarily tells us that epiplexity provides additional information about generalization. I also noticed that the positive correlations seem quite sensitive to the datasets included?

I think it would be useful to compare conditional epiplexity against simpler measures such as label entropy, number of classes, and training-loss reduction. This would help establish whether epiplexity tells us something beyond how difficult or learnable a task is. A leave-one-dataset-family-out analysis would also make the main empirical claim more convincing.

  1. I would like to better understand how reliable the empirical epiplexity estimates are. The authors use the prequential coding approximation from Finzi et al., but there are several additional approximations in the implementation that I think deserve more attention. In particular, they approximate the final model's loss on previous training examples using held-out evaluation loss. This introduces a dependence on the generalization gap: if the model performs substantially better on its training data than on evaluation data, the estimated epiplexity could differ systematically from the original prequential estimate.

This seems especially relevant because the datasets being compared have very different sizes and underlying distributions. I wonder whether some of the variation in epiplexity could be explained by differences in model generalization rather than structural complexity.

The authors also mention discarding nonpositive epiplexity estimates caused by fluctuations in training loss. While I understand why this is necessary for their estimator, I would like to know how often this occurs and whether the resulting rankings are stable across random seeds or different choices in the estimation procedure.

I think a relatively simple stability analysis would strengthen the paper considerably, particularly because the main empirical claims depend on comparing these estimates across datasets.

  1. I am concerned that the tokenization choices make some of the epiplexity comparisons difficult to interpret. The three tokenizers have very different token rates and numbers of codebooks. This means that training on a fixed number of tokens does not correspond to seeing the same amount of audio across tokenizers. Additionally, flattening multiple codebooks into an autoregressive sequence introduces dependencies between tokens representing the same audio frame. I wonder how much of the measured epiplexity reflects these representation-specific dependencies rather than structure in the underlying audio.

The authors attempt to address token-rate differences by normalizing epiplexity by tokens per second. However, I am not sure this normalization is theoretically justified, since epiplexity represents the description length of a learned program rather than a per-token entropy measure.

I think comparing tokenizers under a fixed audio-duration budget, or examining how epiplexity changes as a function of token rate, would help clarify these findings. More generally, because computational constraints are central to the definition of epiplexity, I would appreciate a clearer discussion of how the training budgets are made comparable across datasets and representations.

Overall, I think this paper asks an interesting question and represents a substantial empirical effort. My main concern is that the relationship between epiplexity and generalization is not yet sufficiently isolated from simpler explanations involving task learnability, dataset composition, and representation choices. I would be particularly interested in seeing whether epiplexity can explain variation in downstream performance beyond these factors.

I also think the negative results are worth engaging with more deeply. Rather than necessarily showing that epiplexity is not useful for audio, they may suggest that the amount of learnable structure is less important than how much of that structure is shared between learning problems. I think developing this distinction would make the paper's contribution much more compelling.

Questions for authors

Thank you for the interesting paper! I would be happy to reconsider and potentially raise my score if the authors can address the following questions, particularly Questions 1–4, which are most important to my assessment.

  1. What is the theoretical motivation for expecting epiplexity to predict corss-domain generalization? Could the authors clarify why the total amount of learnable structure in a source distribution should correlate with performance on a different target distribution, rather than the amount of structure shared between them?

  2. Does conditional epiplexity provide information beyond simpler measures of task learnability? Could the authors compare its predictive value against label entropy, class count, training-loss reduction, or label noise, and investigate whether the reported correlations persist across different dataset families?

  3. How sensitive are the epiplexity estimates to the approximations used in prequential coding? In particular, could the authors quantify the effect of using held-out evaluation loss in place of the final model's training loss, and report how frequently nonpositive estimates occur and whether rankings are stable across random seeds?

  4. How should epiplexity be compared across tokenizers with substantially different token rates and codebook structures? Could the authors justify the normalization by tokens per second and investigate whether the relative rankings persist when controlling for the amount of underlying audio and computational budget?

  5. How should we interpret the different notions of generalization evaluated in the two experiments? Since Experiment 1 considers cross-domain prediction while Experiment 2 evaluates held-out classification within the same dataset, could the authors clarify what their results establish about transferable representations?

  6. Could the authors further explore what the negative results reveal about the relationship between structural information and transfer? I found the proposed direction of joint epiplexity interesting and wonder whether measuring the reduction in target-domain prequential coding cost after source-domain pretraining might provide a more direct characterization of reusable structure.

2 comments

Thank you for the detailed feedback! We are going to work on addressing these points for resubmission by early next week but wanted to answer your questions promptly before that.

Q1) Finzi et al show that epiplexity resolves a number of paradoxes that traditional information-theoretic notions do not. For example, it’s empirically known that deterministic data transformations can improve model performance. However, such transformations cannot increase Shannon information, but can increase epiplexity. We will mention this in our updated manuscript. To address your point about OOD generalization specifically, empirical studies done in Section 6 from Finzi et al found that epiplexity was a better predictor than training loss for performance on unseen data.

Q2) We have added these metrics to Experiment 2 and will look into rerunning the minimal parts of the experiment relevant to this task. This was good feedback!

Q3) We can provide statistics on non-positive estimates using our experiment logs. The training loss of the final model on prior tokens is not recorded in our experiments or in the experiments of Finzi et al, as this is too costly to compute. We would expect the discrepancy to be small when the distribution of the evaluation data is similar to that of the training data. We can, however, quantify the difference between our current method of using training losses averaged between evaluation steps and using held-out evaluation loss.

Q4) Computational budget is dependent upon the number of tokens. Since the tokenizers have different token rates, the computational budget and the amount of underlying audio cannot be fixed simultaneously unless one uses budgets high enough to encompass all audio clips with all tokenizers. This is prohibitively slow. You are correct that epiplexity is defined to be the length of a program. Conceptually, however, it measures structural information that can be observed from data within a fixed computational budget. The token-rate normalized epiplexity can thus be thought of as a measurement of information density (bits of information per second of audio).

Q5) Experiment 2 does not address generalization well as you have mentioned. We are working on a follow up experiment looking at cross domain generalization to better address your point that looks at two domains (primarily looking at domain shift in bioacoustics between focal and soundscape recordings, see https://huggingface.co/datasets/DBD-research-group/BirdSet).

Q6) We believe that the negative results show the structure in audio datasets is sufficiently different from each other creating issues with generalization. As we have said before, measuring joint epiplexity, which is a future research direction, can potentially address this point. If we have time, we will attempt to implement something for the rebuttal. However the joint epiplexity work may be too incomplete for this submission.

Thank you for the thoughtful responses and for the additional experiments in the revised manuscript! I appreciate the authors' efforts to engage with my feedback, particularly given the computational expense of these experiments. I think the revisions improve the paper, although I still have some concerns about the central conclusions.

Q1) I appreciate the additional dicsussion of Finzi et al., which provides useful motivation for investigating epiplexity. However, I think there is still an important distinction between the total learnable structure of a source distribution and the structure that is useful for generalization to another distribution. I would have liked a clearer explanation of why the former should be expected to predict the latter, particularly given the results of Experiment 1.

Q2) The stronger correlations with structural density in Figure 5 are interesting. However, I am still unsure whether these correlations capture something beyond task learnability. Since structural density incorporates conditional entropy through the total code length, I think a direct comparison against simpler measures such as label entropy, class count, and training-loss reduction would help establish the additional information provided by epiplexity.

Q3) Thank you for providing additional statistics on nonpositive estimates and comparing approximation strategies. I found these additions useful, although the observation that different approximations produce estimates differing by approximately an order of magnitude raises further questions about estimator reliability. In particular, I would be interested in understanding whether the relative dataset rankings and reported correlations remain stable across these choices.

Q4) I understand the computational constraints involved in controlling for both token count and audio duration, and I appreciate the clarification. However, I am still unsure how to interpret comparisons across tokenizers with substantially different representations and computational budgets. The proposed interpretation of token-rate-normalized epiplexity as information density is intuitive, but I think it would benefit from further empirical or theoretical justification.

Q5) I particularly appreciate the addition of the BirdSet experiment, as it directly addresses my concern about evaluating cross-domain generalization. However, the relationship between structural density and cross-domain performance remains weak and statistically nonsignificant (ρ = 0.29, p = 0.354), whereas source-domain test loss shows a stronger correlation (ρ = −0.58, p = 0.048). I think these findings make it difficult to conclude that epiplexity offers additional predictive value for transfer.

Q6) I agree that differences in domain-specific structure could help explain the negative results. I also understand that implementing the joint epiplexity likely require substantial additional work. My remaining concern is that the current experiments do not directly establish whether differences in shared structure explain the observed generalization behavior. I think this distinction could provide a valuable direction for future research.

Overall, I appreciate the authors' effort in addressing my questions, and I continue to find the research direction compelling. I also think the negative findings are valuable and worth investigating further. However, my main concern remains that the current results do not sufficiently establish what epiplexity tells us about audio generalization beyond simpler explanations involving task learnability, predictive uncertainty, and dataset differences. While the revisions have helped clarify several aspects of the work, I am not yet sufficiently convinced to change my original assessment. I encourage the authors to continue pursuing this direction, particularly the question of how shared structural information relates to transfer across distributions.

Reviewer_hsrfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores epiplexity as a predictor for model performance applied to audio domain. The paper considers 3 tokenizers - EnCodec, X-Codec and DAC to measure various correlations between the epiplexity / model description length for various datasets and the downstream model performance.

Critical strengths
  1. The paper applies the concept of epiplexity to the audio domain which has previously been unexplored.
  2. They cover several different datasets for their investigations of cross-domain generalization involving speech, spatial audio, etc. with 3 different tokenizers. The breadth of their dataset coverage is appreciated.
  3. The paper does not overstate its claim that epiplexity can be used as a reliable measure of model performance. The paper is honest in reporting weak relationship between their measures and the results observed.
Critical weaknesses
  1. The MDL definition is an approximation, and therefore cannot be a reliable indicator of epiplexity - especially eliminating non-positive MDL observations and only using a subset of evaluation data.
  2. The paper's implementation of epiplexity is a function of the model family, evaluation dataset subsampling, the parameter grid sweep values chosen which implies that it is very sensitive to these selections. No easy generalization can be made from this.
  3. The correlations between epiplexity and performance is generally pretty weak and very sensitive to the tokenizer family used. Which leads me to not be very confident in the major take-aways (if any) from this paper. Similarly for cross-domain generalization - that signal is also weak.

Based off of these comments, I feel that the experiments conducted do not definitively establish epiplexity as a metric for guiding where model performance could be improved. Although the study is well motivated and constructed over various datasets and tokenizers which help build credibility for the results and forms an interesting exploration of using epiplexity, the results themselves only establish weak correlations with the metric which are also quite dependent on the model family, tokenizers used, tasks being handled (w.r.t conditional epiplexity). Therefore I recommend weak reject.

Questions for authors
  1. When discarding MDL points, how many such data points were discarded? How much did it affect the estimations?
2 comments

Thank you for your review!

You are correct that many of these methods are approximations of epiplexity. This is because finding an exact measure of MDL for epiplexity depends on searching all possible programs. So, as done by Finzi et al. We constrain the problem to a model family and grid search over the size of the model, to make this work feasible. In the future we are interested in testing new methods for estimating this to be more efficient.

That being said, we are going to do some work towards improving the paper based on your feedback. Specifically, we're working on

measuring the correlation of conditional epiplexity to some simpler metrics (such as number of classes) which are not dependent on classification performance. running a follow up experiment to measure the relationship of conditional epiplexity and model generalization using pairs of distinct datasets built for the same classification task (i.e. two machine anomalous sound detection datasets). Measuring issues in the non-positive MDL measurements.

Thanks for the comment and the revisions! The additional discussion around the MDL approximation and discarded points improves the transparency of the method, but still does not show how discarding these points changes dataset rankings or correlations. Also, the sensitivity of the estimates to the model family, parameter sweep, and evaluation subsampling remains unclear. I believe the paper is improved, but these revisions do not fully address my main concerns, so I will maintain my current rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of our Inital Reponse and Timeline

Thank you again to all the reviewers for your feedback. We have started work implementing and rewriting parts of the manuscript to address your points by the end of this week. We have provided each of you with personalized responses to your concerns.

Holistically, we are going to focus on the following

  1. Adding a third smaller experiment for testing generalization in unconditional epiplexity
  2. Improved analysis of the results in unconditional epiplexity

That should be ready by Friday AOE. Hopefully by Monday AOE we will also have an experiment done for replicating some scaling law work that will help compare audio epiplexity to that of other modalities to address some questions about how audio relates to other modalities and previous work.

Please let us know if you have any concerns or questions, happy to discuss them with you.

AuthorsOpenReview ↗

Revised Draft Submitted

Hello!

We ran some more experiments and have updated the paper. We have also included more results looking at structural information density, or epiplexity divided by the two part code length, to estimate the impact of structure to noise. This idea came about when trying to respond to the reviewer comments.

  • We highlighted in blue significant changes made in the manuscript (the highlighting will be removed in the final submission). These changes include new results and remarks designed to answer your thoughtful questions: Notes on the impact of MDL calculation (see Section 3.1.1)
  • An experiment to study conditional epiplexity in the context of domain shift and generalization using the case study of the focal to soundscape domain shift in bioacoustics (see Section 4)
  • Revised scaling law experiments to using a token budget of 1 billion instead of 31 million and comparing audio to text and images for both epiplexity and the structural information density in the appendix (see Appendix F)
  • Minor edits to wording throughout the paper for clarity
  • Corrections to some erroneous data points, none of which seriously impacted the results.

We are interested in your thoughts! We thank you all again for your time and help in our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.