acceptodds
Under review as a conference paper at ICLR 2027

Measuring Context Use in Music Generation

Abstract

How can we measure whether a musical continuation carries forward one player’s part and relates to the other players? We introduce Predictive Musical Information (PMus), a framework grounded in information theory for evaluating a continuation’s relationship to its shared context and individual players. PMus uses an autoregressive model as an evaluator to compare a fixed continuation’s predictability when parts of its past context are provided, withheld, or replaced. It accommodates the single mixed output common to music generators, without requiring separate tracks for each player. With an external evaluator, PMus can assess human performances and outputs from open and closed source generators without access to the generating model’s parameters or probabilities. We evaluate five measures across four open music generation models through controlled experiments and a listening study with 85 musically trained participants. The context contrasts agree with musicians’ choices on 62–67% of judgments, with the shared-context measure PMus-Duo highest at 66–67%, whereas ranking by predictability under the true past alone agrees below chance on the same judgments. The measures also respond differently to controlled changes in what the generator was given. Because listeners answered the two questions alike, the perceptual separation of self-continuation from partner response remains open. PMus provides a practical foundation for evaluating how generated music relates to its context.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
music generation, music information retrieval, audio generation, machine listening, musical interaction, co-playing, context-conditioned generation, generative model evaluation, context attribution, source attribution, information-theoretic measures, information theory, human perceptual evaluation, musical cognition, autoregressive models, listening study, black-box evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PW5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PMus to evaluate how a music continuation relates to its past. It changes the context while scoring the same continuation with an external model. It evaluates five measures through controlled generation experiments and a listening study with 85 musically trained participants.

Critical strengths
  1. The paper tackles a real gap in music-generation evaluation. Whether the continuation fits well to what came before.
  2. Its fixed-continuation comparisons are well designed, and the controlled experiments and listening study are very detailed.
Critical weaknesses
  1. My main concern is that the Self and Partner scores have not been shown to perceptually distinguish continuing one’s own part from responding to another player. PMus-Duo has higher point-estimate agreement on both questions, with no clear advantage for the designated player-specific scores.
  2. The practical case is also still limited. As the agreement depends strongly on the evaluator, and the study does not establish an advantage over generator-level preferences or usefulness for choosing between outputs from the same generator.
Questions for authors

Appendix C.1 shows that dropping only the first 0.2 s of the target lowers listener agreement from 0.640 to 0.609 for PMus-Self and from 0.634 to 0.602 for PMus-Partner. How much of the reported predictive gain comes from acoustic continuity at the codec splice rather than from musical structure, and can the authors quantify this?

Reviewer_BkPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PMus, an information-theoretic protocol for evaluating how a generated music continuation relates to its past context in a duo setting. Given a five-second shared past of two players (target A and partner B) and a fixed five-second continuation, an external autoregressive model (mainly MusicGen-large) scores the continuation's mean token surprisal under different contexts: the true mix, no context, A alone, B alone, or with unrelated decoy recordings substituted. Five measures are defined as surprisal differences. PMus-Ctx and PMus-Duo measure the gain from the shared past, PMus-Self measures what A's past adds beyond B's, and PMus-Partner and PMus-Partner+ measure what B's past adds beyond A's. The method works on a single mixed output and does not need the generator's own probabilities. Validation uses 44 duos (URMP, Slakh, MedleyDB, and separated MusicGen output) and four generators (MusicGen, InspireMusic, Magenta RT, ACE-Step). In controlled experiments, giving the generator the true partner raises the partner measures more than the self measure. In a listening study with 85 musicians (940 pairwise judgments), the measures agree with listener choices on 62–67% of judgments, versus 41–44% for ranking by likelihood under the true past. The authors note that listeners answered the "continue A" and "listen to B" questions alike, so the perceptual separation of self and partner is not established, and that agreement depends on the evaluator.

Critical strengths
  1. A useful, model-agnostic way to frame context use. Measuring how a fixed continuation's predictability changes when parts of the past are withheld or replaced is a simple idea, and it fills a real gap. Most music generation metrics judge the output in isolation or only check stem compatibility. Because PMus works on a single mixed output with an external evaluator, it can in principle score closed-source generators and human performances under one protocol. This is the main reason I think the direction is worth pursuing.

  2. Careful and transparent evaluation. The controlled generation conditions (true partner / no partner / unrelated partner) give a sensible sanity check, and the partner measures respond as intended (dz ≈ 0.8–0.9). The listening study is reasonably sized for this kind of work, and the statistics are thorough: crossed bootstrap, multiplicity correction, matched-generator contrasts, and an audit of the tokenizer boundary issue. The authors also openly report negative results, such as self/partner not being perceptually separated and Magenta failing as an evaluator, which makes the paper's claims easy to calibrate.

Critical weaknesses
  1. The listener evidence does not show that PMus measures context use rather than generator quality. All judged pairs compare outputs from different generators, and a baseline that only knows which generator produced each clip already reaches 61% agreement (Table 20). None of the PMus measures is significantly better than this baseline. So the 62–67% agreement could largely reflect which models generally sound better, not how well a specific continuation uses its context. The comparison against direct likelihood (41–44%) does not help much here, because that baseline is below chance and therefore a weak reference. Agreement also drops to chance with Magenta RT as the evaluator. The main evaluator, MusicGen, is one of the generators being scored, and once its own outputs are removed PMus-Partner+ is no longer above chance. There is also no comparison with existing context-aware metrics such as COCOLA, Stem-JEPA, or APA, which the paper cites. Since the paper's practical claim is that PMus is a useful signal of musical fit, this gap matters most.

  2. The per-player measures, which are the main motivation, are not validated. The paper is framed around separating self-continuation from partner response, but the results do not support this. Listeners answered both questions almost identically (cross-minus-same agreement 0.005). The self and partner measures are not better than the shared-context measures on their intended questions (Table 17). And where self and partner scores disagree, listeners show no question-specific preference (Table 18). What remains is essentially a PMI/PCMI-style shared-context score applied to audio tokens. This is close to prior information-dynamics work (e.g., Total Information Flow) and is tested on a small set: 44 duos and 22 tracks in the listening study. The effects are also very small (around 0.01–0.02 nats/token), and the median PMus-Ctx of generated continuations is negative. Together, this makes the contribution feel preliminary for ICLR.

Questions for authors
  1. PMus does not beat the generator-identity baseline (61%). Can you show agreement on within-generator pairs (same model, different seeds), or after conditioning on generator identity, so that it is clear PMus measures more than which model sounds better?

  2. How does PMus compare with existing context-aware metrics (COCOLA, Stem-JEPA, APA) on the same listener judgments?

  3. Direct likelihood −h(X|C) agrees with listeners below chance. Do you have an explanation, e.g., the evaluator preferring repetitive or low-complexity outputs? This affects how much the 20-point gain over that baseline means.

  4. Generated continuations have negative median PMus-Ctx, i.e., the true context makes them less predictable. How should this be interpreted, given that the measure is meant to reflect context use?

  5. A targeted listening test with pairs chosen to differ in self vs. partner scores (as you outline in Section 7) would be needed to support the per-player measures.

Reviewer_YrmVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PMus to measure the predictive gain from musical context. It changes the context while keeping the continuation fixed. Controlled generation experiments and a listening study examine whether these scores reflect how well the continuation fits its context.

Critical strengths

This paper works on a important and valuable problem. The method does not require generator probabilities or separate tracks in the generated output.

Critical weaknesses

The listening study does not clearly show a perceptual distinction between self-continuation and partner response. This limits the interpretation of the player-specific scores. Generator identity is already controlled in additional analyses, but the prediction gain over the generator-preference baseline is still unclear. Selection among outputs from the same generator also remains untested. These are limits of the current validation. I really hope the work could be more solid.

Questions for authors

Does the adjusted association in Table 21 give a stable prediction gain on held-out data? What evidence supports the player-specific scores beyond the shared-context score?

1 comment
AuthorsOpenReview ↗

Response to Reviewer YrmV

Thank you for the review and the two precise questions, they were bothering us as well. We ran an experiment to test the distinction between self-continuation and partner response, and an analysis to better understand the relationship between the prediction gain over the generator-preference baseline. Whether PMus can predict which output listeners prefer among different seeds of the same generator remains open. We will report the pilot and baseline analysis in the revision and explain how a follow-up listening study would address that question.

No perceptual distinction between self-continuation and partner response. We built the targeted test (General Response, Parts 1 and 2). When the same listener answers both questions, the answers agree with PMus-Self on the continue-A question in 28 of 45, and with PMus-Partner on the listen-to-B question in 36 of 45.

Question 1: Does the adjusted association in Table 21 give a stable prediction gain on held-out data? No, and the revision says so. All six paired held-out accuracy changes in Table 21 include zero. With instrument reports, adding PMus-Partner moves held-out accuracy from 0.675 to 0.700 (change 0.025, 95% interval −0.022 to 0.075). Without them, it moves from 0.700 to 0.672. The coefficient (0.350, 95% interval 0.094 to 0.607, Holm-corrected p = 0.0224) is an adjusted association, not a prediction gain. A broader result does hold: within generator pairs, all 45 fixed-effect coefficients are positive (General Response, Part 3).

Gain over the generator-preference baseline. The within-pair models show that the context contrasts carry information beyond which generator produced a clip. A resolved accuracy gain over the generator-preference baseline is not shown, and the revision no longer claims one (General Response, Part 3).

Selection among one generator's outputs. This remains untested. A same-model, different-seed listening study is our next experiment (General Response, Part 3).

Question 2: What evidence supports the player-specific scores beyond the shared-context score? Three pieces, two of them new:

  • Listeners (new): the answers follow PMus-Self on one question and PMus-Partner on the other (General Response, Part 1). A shared-context score cannot predict this. PMus-Duo prefers the same continuation (XAX_A) on all 10 examples, so on the listen-to-B question it predicts the opposite of PMus-Partner.
  • Construction (new): on 46 examples built without PMus, PMus-Self and PMus-Partner rank the two continuations in opposite directions, as constructed, in 40 (25% expected by chance). PMus-Duo and PMus-Ctx mostly prefer the A-continuation. Single similarity features reach 9 to 19 of 46, and a combined similarity score 32.
  • Controlled generation (Section 5.1): giving a generator its true partner raises PMus-Partner and PMus-Partner+ more than PMus-Self (Table 9). The paired effect sizes dzd_z are 0.88 and 0.80 for the partner measures, and 0.79 and 0.81 for the paired differences. Both conditioned generators receive the past mix as audio, and MusicGen is also the evaluator. Part of this effect may therefore be acoustic carry-over of B, and the revision says so.

On the submitted study's judgments, the player-specific scores do not beat PMus-Duo (Table 17). The planned analysis of the new test will evaluate it.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response, Part 1

We thank the reviewers for careful and thoughtful feedback. All three identified the same central gap: the paper does not show that listeners hear the difference between self-continuation and partner response that PMus-Self and PMus-Partner are meant to capture (Reviewer BkPB, Question 5; Reviewer PW5U, main concern; Reviewer YrmV, first weakness).

During the discussion period we built the targeted listening test that Reviewer BkPB describes. Its first results indicate that listeners can hear the difference between what PMus-Self and PMus-Partner measure.

We created an anonymous demo page, both because we are proud of these examples and because no words can convey how they sound as clearly as listening to them:

https://anonymous.4open.science/w/duo-continuation-demo-7631/#examples

The test. Each example is a 10-second duo past of players A and B, followed by two 5-second continuations of A: XAX_A develops A's own line, XBX_B takes up what B is doing. Each listener answers both questions, five examples each, in random order: "Which continuation continues player A better?" and "Which continuation shows that A listens and responds to player B?", with the instruction not to judge which track sounds better overall. A listener who hears the distinction chooses XAX_A for the first question and XBX_B for the second. All listeners have at least 3 years of musical training.

First results (9 listeners). On every example PMus-Self prefers XAX_A and PMus-Partner prefers XBX_B, so each answer either agrees with the score its question refers to or does not. Listeners' answers agreed with PMus-Self on the continue-A question in 28 of 45 answers and with PMus-Partner on the listen-to-B question in 36 of 45: 64 of 90 answers (71%) overall. The same listeners chose different continuations for the two questions, in the directions the two scores predict.

participantagreed with PMus-Self (continue-A questions)agreed with PMus-Partner (listen-to-B questions)overall agreementmean time per example
Participant 15/55/510/1075 s
Participant 25/55/510/1055 s
Participant 34/54/58/10104 s
Participant 44/53/57/1084 s
Participant 52/55/57/1095 s
Participant 62/55/57/1097 s
Participant 71/54/55/10103 s
Participant 81/50/51/1083 s
Participant 94/55/59/1075 s
All 928/4536/4564/9086 s

All participants have at least 3 years of musical training. numbering as on the demo page. Per-example results are on the demo page.

  • The experiment ended with 2 open questions about how did the participant made their choice. We excluded two participants whose free-text answers showed that they chose the continuation that sounded better overall rather than the relation asked about.
AuthorsOpenReview ↗

General Response, Part 2

The examples were chosen for large differences (PMus-Self 0.090–0.253, PMus-Partner 0.104–0.355 nats/token, an order of magnitude above the 0.01–0.02 condition effects in the paper). A combined similarity score also ranks 9–10 of them, so the pilot tests whether listeners hear the difference PMus predicts.

To make them, we made a list of potential musical missmatched scenarios, and asked claude Opus 5.5 to generate 100 examples using hard thinking. We then listened to the 100 options and manually chose those who made "musical sense" and had a big contrast between "continue A" and "listen to B". The difficulty is that a convincing response to B is often also a convincing continuation of A. B’s cue must therefore arrive near the end of the context, where A’s own material still creates a strong expectation.

In the test used musically plausible cues, each introduced by B near the end of the context. In every case, one continuation of A favors A’s own material and the other favors B’s cue.

  • Key change: B changes key just before the end. One continuation finishes A’s phrase in the original scale; the other adopts the new scale and drops the phrase.
  • Cadence: B resolves to the tonic with a strong cadence while A is on the dominant. One continuation sustains the dominant tension; the other resolves or ends abruptly.
  • Rhythm: B shifts rhythm or meter, for example from 4/4 to 3/4, so that the beats no longer coincide. One continuation keeps A’s own pattern; the other switches to B’s rhythm.
  • Tonal versus modal: B introduces G natural (A Aeolian) while A’s phrase implies G sharp resolving to A. One continuation completes the tonal resolution; the other adopts G natural and the modal direction.
  • Register: B drops to a low register while A ascends toward a peak. One continuation reaches the peak; the other breaks off and answers B in the low register.
  • Texture: B turns sparse while A plays a dense line. One continuation carries on with the dense line; the other thins out and trades gestures with B.
  • Dynamics: B drops suddenly in dynamics during A’s crescendo. One continuation reaches the climax; the other pulls back to a quiet gesture.
AuthorsOpenReview ↗

General Response, Part 3

Do PMus scores track listener preferences beyond generator identity? (BkPB, Q1; YrmV; PW5U)

Yes, modestly. After accounting for listeners’ overall preference between the generators in each matchup, we found that a clip’s odds of being chosen rose as its PMus score advantage grew.

We included four generators to evaluate PMus across different ways of producing music. For each judgment, we modeled the listener’s choice using the difference between the clips’ PMus scores and a separate baseline for each of the six generator pairings. Across five scores, three question sets, and three analysis variants, all 45 coefficients were positive and 34 confidence intervals excluded zero.

For PMus-Partner on Listen to B, a one-standard-deviation increase in score advantage corresponded to about 1.7 times the odds of being chosen (Holm-adjusted p = 0.004). The association held after dropping the first 0.5 seconds to address the splice concern (p = 0.002). It fell short of the adjusted threshold after we controlled for context-free likelihood (p = 0.066).

The largest coefficient was for PMus-Self on Listen to B, leaving open whether PMus captures preferences specific to each question. We also found no clear gain in prediction accuracy over the generator-only baseline. The estimated gain for PMus-Duo across both questions was 5.6 percentage points (95% CI [−2.6, +14.3], Table 20). The held-out analysis also showed no clear gain (Table 21). We will report these results and their limits in the revision.

Can PMus identify which output listeners prefer from the same generator?

We don’t know yet. Our screened clips include 25 pairs generated with different seeds of the same model. Their median PMus score differences were roughly half those of the judged pairs, and we have no listener judgments for those 25 pairs. During the discussion period, we prioritized the two-question pilot to test the player-specific interpretation. Our next listening study will test whether listeners prefer the higher-scoring output from the same generator. We will report the result either way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.