acceptodds
Under review as a conference paper at ICLR 2027

Voice Acting Arena: Evaluating Expressive Speech Across Acting Challenges

Abstract

Expressive text-to-speech models increasingly follow instructions, yet authentic voice acting remains difficult to compare: saying the right words is not the same as performing a scene. We introduce the Voice Acting Arena, a human-preference evaluation for complex acting directions. Its challenges span 40 fine-grained emotion targets, selected vocal-delivery directions, and multi-step scenes across ten acting domains. A language-model director translates each challenge into the controls supported by a particular speech system; listeners then compare anonymized takes on overall acting, instruction fit, authenticity, and, when requested, a vocal reaction. In a first round of 560 listener votes on eleven model entries, Gemini 3.1 Flash TTS leads on every question. Our Humaneness Voice baseline, released with commercially usable open weights, sits in the middle: with ten-candidate selection its rating cannot be separated from OpenAI GPT Audio 1.5, DramaBox, or Cartesia Sonic 3.5. To curate its training data and rank candidate takes, we train a Genuineness Score for spontaneous, authentic-sounding speech and a Vocal Burst Blending Score for how naturally laughs, sighs, and similar bursts fit the surrounding speech. A first listener check supports the genuineness score only weakly and does not yet support the blending score, so neither is treated as a measure of acting. We release the challenge design, evaluation protocol, bilingual data views, and baseline components, and keep live descriptive results separate from a frozen benchmark.

Primary area:
datasets and benchmarks
Keywords:
voice acting arena, voice acting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YjMbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Voice Acting Arena, a protocol for evaluating speech synthesis under fixed scripts and complex acting directions. Its tasks cover fine-grained emotions, vocal delivery, and multi-stage scenes. A language-model director adapts shared acting briefs to each system’s supported interface, and listeners compare anonymized outputs on overall acting, instruction adherence, authenticity, and requested vocal reactions. The paper reports an initial snapshot of 560 comparisons across 11 system entries. It also provides the Humaneness Voice open-weight baseline, a candidate-selection procedure, bilingual annotated data views, and public acting briefs. The authors distinguish these live results from a frozen benchmark and explicitly discuss the limitations of the evaluation and perceptual scorers.

Critical strengths
  1. The evaluation addresses a meaningful gap between synthesizing natural speech and convincingly performing a scene. Fixed scripts, emotional transitions, delivery constraints, and requested vocal reactions provide concrete tests of expressive control. Separating overall acting, instruction adherence, and authenticity makes the protocol more informative than a single preference or naturalness score.

  2. The baseline and accompanying resources provide a concrete foundation for further research. The analysis also examines sensitivity to listener composition, distinguishes uncertain middle ranks from the more stable leading systems, and acknowledges the limited human validation of the perceptual scorers. These analyses help readers interpret the results without overstating what the current evaluation establishes.

Critical weaknesses
  1. The connection between the documented task design and the reported evaluation is insufficiently established. Only 46 of the 560 votes concern 35 tasks from the 374-task registry; the remaining 514 use an extended pool whose generation procedure and relationship to the public acting briefs are not documented in the evaluation export. Twenty registry tasks were also used during director development, and the task-to-audio mapping is not yet frozen (pp. 4 and 9; Appendix A). Although the authors disclose these limitations, the uncertainty about task sampling, separation from development, and generation configurations limits reproducibility and the evidence supporting the proposed task design.

  2. The reliability of the four judgment dimensions across listeners remains only partially characterized. Four listeners supplied 318 of the 560 votes, and each specific audio pair received one judgment, so item-level inter-rater agreement cannot be estimated. The listener-bootstrap and subgroup analyses provide useful evidence, including the stability of the top three systems after removing the most active listeners. However, authenticity ratings have a rank correlation of only 0.48 between paid and community listeners, and several middle ranks change substantially. The current evidence therefore leaves uncertainty about which distinctions would persist with a different listener sample.

Questions for authors
  1. Can you clarify the generation, sampling, and model-pairing rules for the extended task pool, together with its relationship to the public briefs and director development? Using the existing records, can you identify a subset with documented generation configurations and no task reuse from development, and show whether the principal ranking conclusions persist?

  2. Can you provide independent repeated judgments on a modest, representative subset of the same audio pairs, either from data collected after this snapshot or through a limited additional check? Agreement and uncertainty for each dimension would help establish whether the protocol yields reproducible distinctions, particularly for authenticity.

4 comments
AuthorsOpenReview ↗

Answer to YjMb (1/3): extended task pool, pairing, and development reuse

Thank you for the review. We answer in three comments: the task pool here, listener reliability in the second, and the ethics concern in the third.

Generation. Gemini 3.7 Flash wrote the 2,000 extended tasks in two runs of 1,000 on 12 and 13 September, at temperature 0.9 and in batches of eight, with recorded seeds and code versions. Each batch was prompted with six exemplars of the same track, drawn from 84 registry tasks. The target emotion or delivery and the requested reactions came from those exemplars. Scripts, directions and contexts were newly written. 472 of the 2,000 extended tasks received the 514 votes. A Gemini language model wrote these tasks and a Gemini speech model leads the board, so the task style may favour that system. The votes on registry tasks are too few to test this, and the revision says so.

Relationship to the public briefs. The public Acting Challenge briefs are a separate resource. The extended tasks and their exemplars are distributed with the demo, so they are not held out, and Sections 1 and 3 of the revision no longer describe them that way.

Sampling and pairing. The sampler serves the least-assigned clip pair first, then the least-compared system pair, then the least-judged task, with random tie-breaking and random A/B order. This rule gives new pairs priority over repeats, and in the first round no audio pair was judged twice. This describes vote assignment; the earlier choice of tasks for audio generation is not yet recovered.

Director development. The two director-prompt searches used 20 registry tasks. We reconstructed them from the saved sampling script and its seed. The original run files are not recovered. Six votes on four of these tasks are in the export, all among the 46 votes on registry tasks.

A subset without development reuse (your first question). Without the 46 registry votes, the other 514 votes keep the same three entries on top, and Gemini is first in 97% of listener resamples. But in the current exemplar file, seven of the 20 reconstructed development tasks are also generator exemplars, all in the vocal-delivery track, so extended tasks of that track may inherit target deliveries from development tasks. The subset that excludes this as well is the extended tasks of the emotion and acting-scene tracks: 373 votes from 58 listeners on 342 tasks. They have a documented generator, none of these votes is on a development task, and no exemplar of these two tracks is a development task.

On this subset the same three entries lead in the same order, and Gemini is first in more than 99% of listener resamples. Rank correlations with the full board are 0.99 overall, 0.97 for instruction fit and 0.92 for authenticity. The subset holds 67% of all votes, so high values are expected. Second and third place are less certain than first: the same three entries lead in 57% of resamples, against 86% with all votes. The remaining 141 extended votes are in the vocal-delivery track. On them Breeze is ahead of Gemini, and Gemini is first in 36% of resamples. With 141 votes we cannot tell whether this is noise, an effect of the development exemplars, or a difference between tracks. Both subsets go into the appendix.

The conclusions that persist are therefore the identity of the three leading entries and Gemini's first place on the pooled board. Two limits remain. The reconstruction of the development tasks is not confirmed by the original run files, and similarity between tasks beyond shared exemplars is not tested.

Generation configurations. One manifest links all 560 votes to task and director hashes, model identifiers, request settings and audio hashes, and all 1,084 audio files match their recorded hashes. Hashes do not reconstruct inputs that are not in the records, and hosted APIs may no longer return the same audio.

AuthorsOpenReview ↗

Answer to YjMb (2/3): listener reliability, repeated judgments, and presentation

Listener concentration. With at most 10 to 30 votes per listener, Gemini stays first in 98% to 100% of subsampling draws. With at most 5 it does so in 64%, and with equal weight per listener it drops to third behind Breeze and ElevenLabs. The leader therefore depends on the listener sample and on how listeners are weighted, and the revision reports this next to the leaderboard.

Paid and community listeners. We agree that the correlation of 0.48 between the authenticity boards of the two groups is a concern. A board fitted without one listener predicts 69% of the non-tied authenticity votes of paid listeners and 61% of those of community listeners. The pooled authenticity board, which paid votes dominate, therefore describes paid listeners better. We do not claim that the authenticity ranking would persist with a different listener sample. You also note that several middle ranks change. The first round does not resolve the order of the four middle entries, and the revision says so.

Split halves. The split-half result depends on how the halves are formed. Splits of the listeners into two halves with balanced vote counts give median rank correlations between the halves of 0.86 overall and 0.84 for authenticity. Across 1,000 uniform random splits into two groups of 35 listeners, the medians are 0.79 overall (interquartile range 0.73 to 0.85) and 0.75 for authenticity (0.65 to 0.83). Vote balancing always places the four heaviest listeners two against two, while random splitting lets them share a half. The revision reports both. Both are checks of the system ranking, and neither measures agreement on identical clips.

Repeated judgments (your second question). The study is running on 120 existing clip pairs, 40 per track and spread over small, medium and large rating gaps, with a target of three new judgments per pair from listeners who did not judge that pair before. For each question, and for authenticity in particular, we will report agreement and Krippendorff's alpha with intervals and the numbers of judgments and listeners. The pairs exclude the tasks of our second running study, which compares the selected take of our baseline with its first take, so they are not a random sample of the first round. We will state this limit with the results, and we will post what we have before the discussion closes.

Presentation. You note that parts of the paper take effort to read. The revision adds a "Protocol at a glance" box to Section 3 and describes there who wrote the extended tasks and why each pair has one vote. It reorganises Section 9 into limitations, discussion and conclusion, and it collects the additional analyses in one new appendix section with their tables.

The third comment answers the ethics concern.

AuthorsOpenReview ↗

Answer to YjMb (3/3): rights in the gated research branch

The data were collected and the models trained at a non-profit research organisation in the European Union. For copying and analysing speech recordings it relies on the research exception for text and data mining in EU copyright law (Directive (EU) 2019/790) and on the national law that implements it. These rules allow research organisations to copy works to which they have lawful access in order to analyse them automatically for scientific research, and to keep the copies, stored securely, for research and for checking results. Lawful access includes content that is freely available online. The podcast and broadcast recordings were made freely downloadable by their producers, without login or download restrictions, when we collected them.

The same national law allows a research organisation with non-commercial purposes to make such copies available to a defined group of people for their joint scientific research, and to individual third parties who check the quality of research. We rely on this provision for the gated branch. It gives processed audio, codec representations and annotations to researchers whom we approve one by one, under non-commercial research terms. It is not an unrestricted audio release, and access will end when the research project is completed. The ungated datasets described in the paper, including material from existing licensed datasets such as Emilia and Multilingual LibriSpeech and synthetically generated speech, will remain publicly available under their respective terms. Podcast Links itself contains links and metadata, no audio.

This basis is a research exception, not a licence for the recordings, and two limits follow.

Early checkpoints in the lineage of the released weights were trained with podcast snippets (see our answer to reviewer MscJ). We do not claim that the research exception clears the weights for commercial use, and the revised abstract no longer calls them commercially usable. The licence declared for the weights is a statement about the weights, not about the recordings behind them.

It does not replace data-protection law. Recordings of identifiable speakers are personal data. We process these recordings for scientific research. The podcast and broadcast research corpus described above remains behind the manual gate. We will add a contact address and a takedown procedure to every gated release before the camera-ready version.

The revised paper describes the gated branch as manually restricted research access, not as an unrestricted audio release.

The listener-consent correction is separate. The Arena and the scorer check used different participation agreements, and the revised ethics statement describes both.

AuthorsOpenReview ↗

Which preferences carry over to a new listener sample?

You asked for repeated judgments on the same recordings, particularly for authenticity. Thirteen new Prolific listeners, using the same instructions as the first round, gave 360 judgments on 120 first-round pairs, with 40 pairs per track and 40 per third of the overall system-rating gap. Of the 120 pairs, 106 received three new judgments, seven received two and seven received four. We excluded tasks reserved for the selector study, leaving only six recordings from our two entries.

QuestionAlpha among new listenersNew choice matches originalNew choice matches higher-rated system
Overall acting0.12 (0.03 to 0.21)196/274, 71.5% (66 to 77)182/301, 60.5% (54 to 67)
Instruction fit0.08 (-0.01 to 0.15)151/215, 70.2% (64 to 76)167/279, 59.9% (53 to 66)
Authenticity0.11 (0.02 to 0.19)163/245, 66.5% (60 to 73)169/289, 58.5% (52 to 65)
Vocal reaction (34 eligible pairs)0.13 (-0.05 to 0.30)47/65, 72.3% (59 to 84)Not estimated

Krippendorff's alpha counts ties. Before the study we expected about 0.10 on the overall question, because two thirds of the pairs compare entries whose ratings predict at most a 70 to 30 preference. Matching the original requires both listeners to pick a clip. The final column requires only the new listener to pick one. For that post hoc check we fitted system ratings on the 440 first-round votes outside the repeated pairs. Parentheses give 95% intervals from resampling clip pairs.

Authenticity. Two new listeners who both chose a clip agreed on average on 58.9% of comparisons within pairs (51% to 67%). The new panel's most frequent answer matched the original on 54 of 70 pairs where both picked a clip. New listeners matched paid and volunteer originals in 66% and 68% of decisive judgments, respectively. Individual choices carry over to some extent, even though the paid and volunteer system rankings correlated only 0.48.

Which ranks persist? We fitted ratings to the new judgments and compared the nine entries other than our two sparsely represented baselines. Unlike the first round, no listener dominates this sample: 11 of the 13 listeners gave 30 judgments each and none gave more. Gemini ranked first overall in 99.5% of pair resamples and 98.7% of listener resamples. On authenticity the figures were 94.9% and 96.7%. The same three entries led together in only 52% and 47% of resamples overall, and 40% and 38% for authenticity. This post hoc check supports Gemini's lead among those nine entries in this sample. The order below it remains uncertain.

At export, 290 judgments had passed quality review and 70 were pending. On accepted judgments alone, overall alpha was 0.07 (-0.03 to 0.17), the match with the original was 69.5%, and the refitted system ratings predicted 61.6% of new choices. Table 6 and Appendix A.3 give both analyses and the sampling limits. Does this answer your question about which distinctions carry over to another listener sample?

Reviewer_2JeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Voice Acting Arena, a human evaluation framework for expressive text-to-speech. It tests whether speech models can perform complex acting directions, including subtle emotions, emotional transitions, and nonverbal reactions such as laughter or sighs. Listeners compare two recordings on overall acting, instruction following, authenticity, and vocal reactions.

The authors also present Humaneness Voice, an open-weight speech system, along with annotated speech datasets and public acting prompts. In an initial evaluation with 560 comparisons across 11 system entries, Gemini 3.1 Flash TTS ranks highest, while Humaneness Voice achieves middle-ranking performance when selecting from ten candidates.

Critical strengths
  1. The task design and data resources cover a wide range of emotions, vocal delivery styles, domains, and acting scenarios. The documented construction, filtering, and auditing procedures provide useful quality controls, although many annotations remain model-generated.
  2. The authors provide a fine-tuned open-weight speech system, corresponding SFT data, and supporting tools for directing and selecting generated speech. These resources offer a practical starting point for further research, with preliminary human evaluation of the complete system.
  3. A unified protocol for evaluating vocal performance. The framework evaluates scene-level acting through blinded pairwise comparisons, going beyond pronunciation accuracy and basic emotion recognition to consider instruction following, authenticity, and nonverbal reactions.
Critical weaknesses
  1. The evaluation rubric is broad and potentially overlapping, while rater expertise and calibration are insufficiently documented. Clearer examples and repeated judgments would help establish whether different listeners apply the criteria consistently.
  2. The two automatic scorers show limited alignment with human preferences: genuineness agreement is fragile, and blending agreement is near chance. Comparing them directly with Gemini 3.1 Pro, which supplied their training labels, would help explain these limitations.
  3. The work mainly integrates existing components into a system and evaluation framework. Its methodological novelty appears limited, and stronger human validation and controlled ablations are needed to establish the benefits of the proposed design.
Questions for authors
  1. Have the authors considered including existing evaluators, such as DRAME-Eval from Speech-DRAME, to provide complementary metrics and compare their agreement with human judgments against the proposed scorers?
5 comments
AuthorsOpenReview ↗

Answer to 2JeG (1/2): rubric overlap, listeners, and the two scorers

Thank you for the review. This comment covers the rubric, the listeners and our two scorers. The second covers existing evaluators and the contribution.

Rubric overlap. At the level of systems you are right, and the paper overstated this. In single votes the three main questions get different answers in 34% of cases (pairwise kappa 0.59 to 0.67), so listeners do not treat them as one question. But we tested for each of the eleven entries whether its rating differs between two questions, and no difference survives a Holm correction over the 33 tests (11 entries, 3 pairs of questions). The three task tracks behave the same way. Their boards agree at a mean rank correlation of 0.69, and three boards drawn from one common ranking would agree at 0.74 with these vote counts (p = 0.28). The revision presents the questions and tracks as descriptions of what was asked and states that an entry's rating does not differ reliably between questions.

Who the listeners were. Seven paid listeners gave 352 votes and 63 volunteers gave 208. The paid listeners were native-English US undergraduates aged 18 to 24, recruited through an online student community. The sign-up form recorded age, country, native language, education, occupation and familiarity with speech technology. Our export of these answers has no question on acting or audio training, so we cannot report expertise beyond this. Of the 63 volunteers, 55 gave no age or country. Every listener saw the rubric on screen and the task with key phrases of the scene and direction highlighted. There was no training or calibration phase. For quality review the server queues each listener's first three votes, every vote with a quality flag or a low-confidence mark, and a random 10% sample. 206 of the 560 votes were read by hand, and no paid vote was rejected. Paid listeners received 15 cents per comparison plus automatic increments for listening time and written feedback, 25 cents on average. The revision adds recruitment, review and pay to Appendix A and a "Protocol at a glance" box to Section 3.

Examples and consistency. The appendix shows the listener interface and twelve example tasks. Listeners were not given example judgments for the four questions, and we agree that such anchors would help. Whether the questions are applied consistently without them is what the repeated judgments measure. The first round gives two indirect checks. A board fitted without one listener predicts 65.2% of that listener's non-tied overall votes (322 of 494). If every listener voted according to the fitted ratings, the figure would be 66.5%, because many entries are close. And no listener with ten or more decisive votes agrees with the others significantly below chance. Neither check measures agreement on the same pair, which you and reviewer YjMb ask for. That study is running: 120 existing clip pairs, stratified by track and by the rating gap between the two systems, with a target of three new judgments per pair from listeners who did not judge that pair before. We will report agreement and Krippendorff's alpha per question, with intervals and completed counts.

The two scorers. We agree that the genuineness and blending scorers are not validated measures of acting. The genuineness scorer agrees with 39 of 61 decisive listener choices and the blending scorer with 31 of 67. With equal weight per rater the rates are 55% and 45%. In the baseline they serve curation and the choice among ten takes. Whether that choice helps is now tested with listeners: on 132 tasks, the take picked by the selector is compared blind with the first take of the same ten. Comparing the scorers on the same listener-judged pairs with Gemini 3.1 Pro, which labelled their training data, is the right diagnostic. We have not completed it and report no result.

AuthorsOpenReview ↗

Answer to 2JeG (2/2): existing evaluators and the contribution

DRAME-Eval. The revision cites Speech-DRAME (arXiv:2511.01261). We could not run its judge on our clips. The DRAME-Eval weights have not been released, and the authors' repository lists them as upcoming (checked on 16 November). The public code covers prompting of general audio models and fine-tuning of a new judge. We therefore make no comparison with DRAME-Eval and do not claim superiority to it. Once the weights are available, the comparison should give the judge the scene context it requires and report its agreement with each listener question.

An existing measure that can be run. We scored all 1,084 voted clips with UTMOSv2, a public predictor of speech naturalness. On the overall question, the clip with the higher predicted naturalness is the listener's choice in 39% of the non-tied votes (191 of 493, 95% interval over listeners 36% to 42%). Chance is 50%, and the other three questions give 36% to 38%. Most of the effect lies between systems. Gemini and Breeze, first and third for listeners, receive the two lowest mean scores. GPT-4o mini TTS and Fish, ninth and eleventh for listeners, receive the two highest. Across the eleven entries the rank correlation between mean predicted naturalness and listener rating is negative on all four questions (-0.42 to -0.67, where only the authenticity value has p below 0.05). Once each system's mean score is subtracted, the remaining score agrees with listeners in 47% of votes on the overall question (42% to 50%) and in 43% to 48% on the others, which is close to chance.

UTMOSv2 hears the audio without the scene and was built to predict naturalness. This is therefore no test of an acting judge, the analysis is post hoc, and the predictor may react to voice or recording character. What it does show is that an existing automatic quality measure cannot replace listener comparisons on these tasks. The entries that listeners prefer include the ones it rates least natural. Our own two scorers have not been run on the Arena clips. Their agreement figures come from a separate check on other pairs and cannot be set against the 39% here. We add the UTMOSv2 comparison to the appendix.

What is new. We do not claim new components. A language-model director, pairwise listening tests, Bradley-Terry ratings and reranking all exist. The submission is in the datasets and benchmarks area, and what it proposes is an evaluation design: a fixed script with a scene and an acting direction per task, one protocol for systems with different control interfaces, separate listener questions, and a vote-level export. The closest existing evaluations differ in what they hold fixed and in who judges. EmergentTTS-Eval (NeurIPS 2025) and InstructTTSEval (arXiv:2506.16381) score the output for single test prompts with a language model as the judge. Speech-DRAME evaluates role-play with human annotation and a trained judge. Our protocol centres on blind human comparisons of two systems under the same fixed script and scene. The revision cites these works.

What the first round establishes. The protocol runs end to end on eleven entries from ten systems. Gemini stays first under listener resampling, under caps of 10 or more votes per listener, and on the 373 votes without reconstructed development reuse (general comment). With equal weight per listener the same three entries lead in a different order. The first round does not establish that the questions or tracks separate systems, that the scorers measure acting, or what the model-specific prompt translation contributes. The last point needs new audio from every system and is named in the limitations. The controlled comparison that is running tests the complete selector against the first take (first comment). An ablation of the two scorers needs new audio as well and has not started.

AuthorsOpenReview ↗

Human agreement and automatic evaluators on identical choices

Existing evaluators. You asked for a direct comparison with our scorers using the same human decisions. In the blinded selection study, both takes had archived UTMOSv2 scores on 23 tasks. On the 20 with a decisive overall vote and distinct scores from all three predictors, genuineness picked the listener's choice 11 times, blending 8 times and UTMOSv2 8 times. This post hoc subset is too small to rank the predictors. DRAME-Eval's authors still listed its trained weights as forthcoming when we checked on 17 November. Our separate reply of 18 November reports the comparison with the Gemini 3.1 Pro teacher.

The repeated-listening study gives a larger comparison for UTMOSv2. On the 274 overall judgments where the original and a new listener both picked a clip, the original listener agreed with the new listener 196 times (71.5%), system ratings refitted without the repeated votes agreed 167 times (60.9%), and UTMOSv2 agreed 108 times (39.4%). The paired difference between the original vote and UTMOSv2 was 32.1 percentage points (95% interval 22.8 to 41.5, resampling pairs). UTMOSv2 hears the recording alone, while the listener also sees the scene and direction. This post hoc result measures how well an audio-only naturalness score predicts that scene-conditioned choice.

How do listeners use the rubric questions? The listener instructions were the same in the first round and in both follow-up studies. Listeners read the scene, the direction and the exact script, and must hear both takes completely before they can submit. They answer overall preference first and then judge each remaining question on its own. A rubric page describes strong and weak evidence for every question. For authenticity, strong evidence is believable timing, prosody, breath and hesitation, and weak evidence is robotic rhythm, forced emotion or implausible pauses. A tie is for cases where neither advantage is meaningful. The instructions contain no worked example judgments. With them, the thirteen new listeners agreed only modestly, with alpha of 0.12 for overall acting, 0.08 for instruction fit and 0.11 for authenticity. For the overall question we expected about 0.10 before the study, because most pairs compare entries with similar ratings. The plan also predicted how agreement should change with the rating gap. Two new listeners who both chose a clip agreed on average on 51% of comparisons within pairs in the closest third, near chance as predicted, and on 64% (95% interval 52% to 77%) in the widest third, against 67% predicted. When both fit and authenticity answers were decisive, new listeners chose different clips in 7 of 260 judgments (2.7%). The original listeners did so in 7 of 86 (8.1%) on those 120 pairs and in 41 of 407 (10.1%) across the full first round. The overlap supports your concern. A worked example could make the distinction clearer. A take can place a requested laugh after the right word and still sound forced. A listener can credit its timing and prefer the other take for authenticity. We have yet to test whether examples change judgments.

What the protocol adds. Holding the scene, script and candidate set fixed let us test the apparent leaderboard gain from our own selector. The selected take won 53 times and the first take won 54, with 18 ties. The gain did not reproduce. Section 8, Table 6 and Appendix A.3 report this controlled comparison alongside the evidence on how listeners use the rubric. Do these two listening studies address the human validation you wanted to see for this design?

Reviewer_MscJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Voice Acting Arena evaluates expressive TTS on fixed scripts with scene-specific acting directions. Its archived design covers 374 emotion, vocal-delivery, and integrated-scene tasks. An LLM translates each brief into model-specific controls, and listeners compare anonymized takes on overall acting, instruction fit, authenticity, and requested vocal reactions. The first leaderboard contains 560 votes across eleven entries, with Gemini 3.1 Flash TTS leading. The paper also presents a MOSS-derived Humaneness Voice baseline with a best-of-N selector, annotated English/German data views, and public acting briefs. Listener results and perceptual-score validation are preliminary.

Critical strengths
  • I like the scene-level TTS eval design. Fixed words, subtext, local emotional transitions, and vocal reactions test something that transcript accuracy and coarse emotion labels miss in standard TTS benchmarks. Keeping instruction fit separate from overall acting is especially useful when systems expose different controls.
  • The paper is transparent about uncertainty. It reports listener-resampled ranks, paid/community differences, sensitivity to heavy raters, weak scorer validation, and negative system experiments. The baseline and data resources could enable useful follow-up work. Gemini's lead also survives removing the four most active listeners.
  • I like that the paper goes beyond simply propose a benchmark. The authors build an end-to-end pipeline connecting acting challenges, data curation, model training, candidate selection, and human evaluation. They also make an effort to understand where the improvements come from. While the ablations are not yet conclusive, I think this makes the benchmark more actionable and practically useful.
Critical weaknesses
  • My main concern is that the results do not yet validate the proposed benchmark. Only 46 of 560 votes cover 35 tasks from the archived 374-task registry; the other 514 use an extended pool whose generation is not documented in the export. The task-to-audio mapping is not frozen, and 20 registry tasks were used for director development. Most tasks received one vote, while four listeners supplied 57% of votes, so item-level agreement is unknown. The top result looks more robust, but I would not interpret the middle rankings have enough evidence as a controlled benchmark comparison.
  • The perceptual selector needs stronger human evidence. Genuineness agrees with listeners on 39/61 decisive pairs, dropping to 17/33 without one influential rater; burst blending agrees on 31/67 (Table A.3). Appendix L provides useful detector-based comparisons, but these do not establish human preference gains from either term. Pass@10 and pass@1 share only 25 tasks and face different opponents. The direct 10–2 result is encouraging, but neither comparison isolates selection at a fixed generation budget.
Questions for authors
  1. Can the authors document the extended task generator and the corresponding settings, and freeze the task–prompt–audio–vote mapping? Which reported votes overlap the 20 director-development tasks, and how do results change after excluding them (Appendix A.5)?
  2. On identical candidate sets, how do random selection, the deployed ranker, and ablations of genuineness/blending compare in blinded human preferences at equal generation budget?
  3. Which upstream speech and reference-voice sources entered the released checkpoints?
6 comments
AuthorsOpenReview ↗

Answer to MscJ (1/3): task provenance, development overlap, and listeners

Thank you for the review. We answer in three comments: task provenance and listeners here, selection at equal budget in the second, checkpoint sources in the third.

Extended task generator (your first question). Gemini 3.7 Flash wrote the 2,000 extended tasks in two runs of 1,000 on 12 and 13 September, at temperature 0.9 and in batches of eight, with recorded seeds and code versions. Each batch was prompted with six exemplars of the same track, drawn from 84 registry tasks. The target emotion or delivery and the requested reactions came from the exemplars. Context, direction and script were newly written. Automatic checks enforced length, reaction placement and unique scripts. A Gemini language model wrote these tasks and a Gemini speech model leads the board, so the task style may favour that system. The votes on registry tasks are too few to test this, and the revision says so. The extended tasks are separate from the public Acting Challenge briefs. They are distributed with the demo, so they are not held out, and Sections 1 and 3 of the revision no longer describe them that way.

Pairing. The sampler serves the least-assigned eligible clip pair first, then the least-compared system pair, then the least-judged task, with random tie-breaking and random A/B order. Paid listeners first get pairs that no paid listener has started. This is why each audio pair has one vote.

Frozen mapping. One manifest now records, for each of the 560 votes, the task and director hashes, model identifiers, request settings and audio hashes. All 1,084 audio files match their recorded hashes. The records can be audited. They were assembled after the fact, hashes cannot reconstruct the prompts, and hosted models may no longer return the same audio.

Votes on development tasks. We reconstructed the 20 director-development tasks from the saved sampling script and its seed. The original run files are not recovered. Six votes on four of these tasks are in the export, all among the 46 votes on registry tasks. Without those 46, the other 514 votes keep the same three entries on top, and Gemini is first in 97% of listener resamples.

In the current exemplar file, seven of the 20 reconstructed development tasks are also generator exemplars, all in the vocal-delivery track. The strictest subset we can form is therefore the extended tasks of the emotion and acting-scene tracks: 373 votes from 58 listeners on 342 tasks. They have a documented generator, none of these votes is on a development task, and no exemplar of these two tracks is a development task. On this subset the same three entries lead in the same order, and Gemini is first in more than 99% of listener resamples. Second and third place are less certain than first: the same three entries lead in 57% of resamples, against 86% with all votes. The remaining 141 extended votes are in the vocal-delivery track. On them Breeze is ahead of Gemini, and Gemini is first in 36% of resamples. With 141 votes we cannot tell whether this is noise, an effect of the development exemplars, or a difference between tracks. Both subsets go into the appendix. Similarity between tasks beyond shared exemplars is not tested.

Listener concentration and middle ranks. With at most 10, 20 or 30 votes per listener, Gemini leads in 98% to 100% of subsampling draws. With at most 5 it leads in 64%, and with equal total weight per listener Breeze and ElevenLabs are ahead of it. The leader therefore depends on how listeners are weighted, and the revision reports this next to the leaderboard. We agree with you on the middle ranks. For none of the six pairs among Cartesia, DramaBox, GPT Audio 1.5 and Humaneness Voice pass@10 is the fitted order kept in at least 97.5% of listener resamples. The revision reports rank intervals and states that the order of these four entries is not resolved.

Agreement on identical pairs cannot be computed from the first round. A repeated-judgment study is running on 120 existing pairs, with a target of three new judgments per pair from listeners who did not judge that pair before. We will post agreement and Krippendorff's alpha per question.

AuthorsOpenReview ↗

Answer to MscJ (2/3): selection at equal generation budget

What the first round shows. The two Humaneness Voice entries use the same ten takes per task: pass@1 is the first take, pass@10 the take picked by the selector. We agree that the first round does not isolate selection. Only 25 tasks received votes for both entries. Over all tasks the gap is +82 Elo with a 95% task-bootstrap interval of -11 to 172, which includes zero. On the 25 shared tasks it is +257 Elo (87 to 491), and +268 (35 to 622) without the 13 direct votes. These figures hold the ratings of the other entries fixed, and the two entries were still judged by different listeners against different opponents. We treat the shared-task result as exploratory. It is the reason for the controlled study below.

The two scorers. Your reading of the scorer check is right, and equal weight per rater makes it weaker: the genuineness scorer then agrees with 55% of decisive choices and the blending scorer with 45%. The revision adds these figures and the split by interface version. We do not present either scorer as a validated measure of acting. We also agree that the detector-based comparisons in Appendix L do not establish a gain in listener preference.

The comparison you ask for. A blinded study at equal budget is collecting votes on 132 tasks. These are the tasks of the main generation run that received first-round votes and where the selector did not pick the first take. Listeners compare the selected take with the first take of the same ten, as two anonymous takes in the Arena interface. The first take is the unselected baseline. It equals random selection if the order of generation carries no information about quality, which we assume. On ten further voted tasks the selector picked the first take, so the two takes are identical and nothing can be compared. We will report wins, losses and ties, the number of listeners, and an interval that accounts for repeated votes by one listener.

This study tests the complete selector. It does not separate the genuineness and blending scorers. Offline, on 159 logged ten-take sets, our reconstruction of the selector reproduces every recorded pick. Removing the genuineness score changes the pick in 30% of sets, removing the blending score in 31%, and removing both in 50%. Each score therefore changes which take is chosen. Whether those changes are improvements needs the listener ablation you describe. That ablation needs new ten-take sets with all takes saved, because the first round kept only the two takes that were entered. It has not started, and we do not expect to finish it within the discussion period. We will not present the first study as an answer to that question.

Correction. The selector of the main run includes a requested-reaction term that Equation 1 omitted. The revision states the deployed rule next to Equation 1. The earlier pilot run did not use the term.

Sources. The third comment lists the sources of the released checkpoints. The documentation of early checkpoints names podcast snippets, and some training targets are under a non-commercial licence. The revised abstract therefore says "open weights" and no longer "commercially usable open weights".

AuthorsOpenReview ↗

Answer to MscJ (3/3): sources of the released checkpoints

Thank you for asking us to separate the final fine-tuning subset from the full checkpoint lineage. SFT3 is a derivative of MOSS-TTS-Local-Transformer-v1.5. The recovered sequence is early voice-acting adaptation, v2, SFT1, a full-parameter preference update exported at step 123, SFT2, and SFT3. Optional later adapters are separate artifacts.

The materialized SFT1 corpus contains 3,147,802 rows: 1,947,272 real-speech and 1,200,530 generated profile rows. Of the total, 3,063 are validation rows. The real-speech selector uses Emolia, Kartoffelphon and Multilingual LibriSpeech (MLS); it excludes EuroSpeech. The profile targets come from original corrected generated takes, rather than the later repaired or converted views.

The surviving SFT2/SFT3 selection contains 398,282 training rows, comprising 331,902 core and 66,380 blend examples. Its real targets are Emolia 170,956, Kartoffelphon 42,994 and MLS 4,715. Its generated profile targets are conditioned on the Emolia, Mediathek, designed-character, mixed-reference and anime-derived families, with 61,507, 49,852, 52,001, 10,187 and 6,070 targets respectively. This is 218,665 real and 179,617 generated targets. The generated targets also depend on the earlier synthesis checkpoint and expressive adapters used to create them.

The 500-profile catalogue contains 219 Emolia, 124 Mediathek, 115 designed-character, 27 mixed-reference and 15 anime-derived profiles. Original profile adapters attach to v2; the later SFT3 voice, emotion and vocal-dimension specialists are separate banks. The Arena used the Mira profile, emolia_c1699. Its reference maps to EN_B00032_S05043_W000010, an original-Emilia-format recording. The reference WAV distributed with the original profile adapter matches the generation reference byte for byte. A generated training target and a real-derived conditioning reference must therefore be documented separately.

Earlier v1 documentation explicitly names podcast snippets, and v2 inherits that checkpoint. We therefore cannot claim that the complete SFT3 lineage excludes podcast-derived material. The final selected corpus has no separate podcast, AudioSnippets or reference-snippet pool, but that narrower observation does not exclude inherited or reference-mediated contributions. Earlier expressive teacher adapters also name DACVAE and mitermix sources; their complete recording-level joins remain incomplete. We distinguish this uncertainty from a verified exclusion.

The Emolia name also needs qualification: its downstream CC-BY declaration does not resolve the official upstream distinction between original Emilia (CC BY-NC 4.0) and Emilia-YODAS (CC BY 4.0). Selected real targets and reference voices include original-Emilia-format IDs. We will replace blanket permissive-source wording with a stage-by-stage source and reference table and distinguish the declared weight license from the conditions on upstream recordings. Exact early v2 source revisions and some provider/reference permissions remain unresolved. The later Scaling Ladder is a separate resource and was not the final SFT3 training corpus.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers: what is new since submission

Thank you for the careful reviews. The paper contributes a scene-based evaluation protocol. Scripts are fixed, each scene has an acting direction that is translated into a prompt for each model, and listeners judge acting, instruction fit, authenticity and requested vocal reactions separately. The reviews ask where the evaluated tasks come from, whether listeners agree with each other, and what automatic measures and our selector add. This comment lists what we have added since submission. The replies to each reviewer give the details.

Task provenance. The generator of the extended task pool and the pairing rule are now documented. One manifest links all 560 votes to their tasks, generation records and audio hashes, and a second check re-hashed all 1,084 audio files and found the recorded hashes.

Development reuse. We reconstructed the 20 director-development tasks. 373 of the 560 votes are on extended tasks of the two tracks in which no generator exemplar is a development task, and none of these votes is on a development task. On these votes the same three entries lead in the same order, and Gemini is first in more than 99% of listener resamples.

Existing automatic measures. DRAME-Eval could not be run, because its weights have not been released. We ran UTMOSv2, a public predictor of speech naturalness, on all 1,084 voted clips. On the overall question it picks the listener's choice in 39% of the non-tied votes, where chance is 50%. Two of the three entries that listeners prefer most receive its two lowest mean scores. On these tasks predicted naturalness does not stand in for listener judgments of acting.

What the first round does not show. Each audio pair was judged once, so agreement on identical clips is unknown. The order of the four middle entries is not resolved. With equal weight per listener, Gemini drops from first to third. The two scorers in our baseline are not validated measures of acting. The revision states each of these points.

Review resources. We found identifying content in the anonymous resource mirrors and have withheld those links from the revised PDF pending corrected mirrors. This currently limits direct review access.

Running studies. Two blinded listening studies are collecting votes. One repeats judgments on 120 existing audio pairs. The other compares the take picked by our selector with the first take of the same ten, on 132 tasks. We will post their results with counts and uncertainty before the discussion closes.

Sources and rights. Checkpoint sources and the rights basis are answered in separate comments to reviewers MscJ and YjMb. Early checkpoints in the lineage of the released weights were trained with podcast snippets. Some training targets and the Arena reference voice come from recordings under a non-commercial licence. We rely on the research exception for text and data mining in EU copyright law and its national implementation. This is not a licence for the recordings. We do not claim that the weights are cleared for commercial use, and the revised abstract no longer calls them commercially usable. Gated access ends when the research project is completed. The described ungated datasets remain available under their respective terms. We will add a contact address and takedown procedure to every gated release before camera-ready.

AuthorsOpenReview ↗

What the two follow-up listening studies show

Does a listener's preference between two performances carry over to other listeners, and did choosing among ten takes improve our baseline? We ran the two blinded listening studies announced in our first response. Section 8, Table 6 and Appendix A.3 report the results.

Repeating the same recordings. Thirteen new listeners gave 360 judgments on 120 first-round pairs. In the planned comparison, the original listener and the new panel's most frequent answer picked the same clip on 65 of the 81 pairs where both picked a clip (80%). On authenticity this was 54 of 70 (77%). Agreement between individual new listeners was modest. For overall acting, Krippendorff's alpha, counting ties, was 0.12 (95% interval 0.03 to 0.21). Before the study we expected about 0.10: two thirds of the pairs compare entries whose ratings predict at most a 70 to 30 preference, so two listeners who both follow the ratings would often disagree. On the 290 judgments already accepted in quality review, alpha was 0.07 (-0.03 to 0.17).

What predicts a new listener's choice? On the 274 overall judgments where the original and the new listener both picked a clip, the original listener picked the new listener's choice 196 times (71.5%), the higher-rated system did so 167 times (60.9%), and the clip with the higher UTMOSv2 naturalness score did so 108 times (39.4%). This comparison was post hoc. We refitted the system ratings on the other 440 first-round votes. The original listener's advantage over those ratings was 10.6 percentage points (95% interval 4.0 to 17.5, resampling clip pairs). Knowing which performance someone preferred carried information beyond the pooled system ratings.

Which entries keep their place? In the first round Gemini led the pooled board but was third when every listener had equal weight. In the new sample, 11 of the 13 listeners gave 30 judgments each and none gave more. A post hoc fit to these judgments alone puts Gemini first among the nine entries other than our two baselines, in 99.5% of pair resamples and 98.7% of listener resamples. The order below Gemini remains uncertain.

Selection at equal budget. On 125 of the 132 planned tasks, listeners compared the selected take with the first take of the same ten. They preferred the selected take 53 times and the first take 54 times, with 18 ties. The selected take's share of decisive votes was 49.5% (exact 95% interval 39.7% to 59.4%), which excludes a 60 to 40 preference or stronger for the selected take among decisive votes in this analysis. The apparent advantage in the first leaderboard did not reproduce under this controlled comparison. The revision reports best-of-ten as our deployed recipe and leaves the contribution of its individual terms open. It claims neither an order of the middle entries nor a gain from selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.