- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
“Open goals” are defined as parts of a sequence that have not yet occurred but can be confidently predicted (and expected) by a human, given the context (for example, a chord progression going back to the root chord).
The paper asks whether explicitly accounting for open goals improves the modeling of human expectation.
The method consists of using a probabilistic grammar on top of a frozen sequence model (the surface model) to reweigh its sampled continuations, which combines two pre-existing techniques in a new way.
The resulting predictions are more aligned with human surprise ratings, even without improving the prediction accuracy.
Critical strengths
The paper addresses an interesting question and does so in a pretty rigorous and scientific way. The three hypotheses are well defined, the method builds on sensible baselines, and the experiments are well designed and properly executed.
Critical weaknesses
I find the paper interesting and overall well written, but unfortunately I believe that its contribution is not relevant or impactful enough for the ICLR community. The technique of reweighting a sequence model’s sampled continuations with an external scoring function is, as the authors point out, well known, and the same holds for using a grammar’s probabilities on top of a frozen sequence model (the baseline defined as “one-step”). The author’s addition to score sampled futures in terms of the open goals inferred from the prefix seems to be, unfortunately, of no use for modern neural sequence models, and I don’t see a clear insight that would advance the understanding of the community on the topics that are of interest for ICLR.
Reading the discussion, I seem to understand that most of what the method did was to increase the overall probability assigned to the root chord (which is almost always a non-surprising continuation). It is therefore pretty unsurprising that this would result in sequences less probable (-10%) but also less surprising to the listener.
Even though I think it is strongly out of scope for ICLR, I believe that this paper constitutes good research and I would encourage the authors to resubmit to a more appropriate conference, possibly in the fields of music or neural sciences.
Questions for authors
Are there any considerations that I did not understand that made me undervalue the impact and/or relevance of this result?
I can see some extension of this work feeding into the topic of making LMs generate more human-like text (hinted at in lines 439-444), but I think that was not elaborated enough to be considered a contribution of the paper.
2 comments
Thank you for your reply. This doesn't substantially change my assessment towards acceptance, and I still believe that the applications and the scope of this work are too limited for a conference such as ICLR.
Thank you for recognizing the rigor of the study and the value of the research question. And thank you for your thoughtful question about relevance. We see this work as using deep learning to make a cognitive theory of anticipation computationally testable, rather than proposing a generative rescoring method. This places the work within ICLR's scope of applications to neuroscience and cognitive science. The question is how a learned sequence model's predictions interact with expectations about structures that remain to be completed.
For example, a listener may anticipate a return to the home chord without expecting it immediately. An intervening chord can feel appropriate because of the possible paths it opens toward that return. Our framework makes this account testable: the sequence model supplies plausible futures, while the grammar identifies open goals and evaluates their completion. Keeping the trained model fixed allows us to investigate how its learned knowledge is used to model human expectation.
This perspective also matters for interpreting the baselines. One-step already integrates possible completions of open goals through the grammar's next-event probabilities; its success therefore does not negate structural anticipation. Full condition tests the additional role of the sequence model's own futures. That contribution is model-dependent, but detectable for IDyOM and in the LSTM whole-song analysis. The improvement is reduced but remains positive after adding tonal controls in the auxiliary LSTM whole-song analysis.
The broader contribution is to turn a theory of structural anticipation into a computational question that can be tested using learned sequence models, while distinguishing what comes from explicit structural goals and what comes from the models' own possible futures.