- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Voice Acting Arena, a protocol for evaluating speech synthesis under fixed scripts and complex acting directions. Its tasks cover fine-grained emotions, vocal delivery, and multi-stage scenes. A language-model director adapts shared acting briefs to each system’s supported interface, and listeners compare anonymized outputs on overall acting, instruction adherence, authenticity, and requested vocal reactions. The paper reports an initial snapshot of 560 comparisons across 11 system entries. It also provides the Humaneness Voice open-weight baseline, a candidate-selection procedure, bilingual annotated data views, and public acting briefs. The authors distinguish these live results from a frozen benchmark and explicitly discuss the limitations of the evaluation and perceptual scorers.
Critical strengths
-
The evaluation addresses a meaningful gap between synthesizing natural speech and convincingly performing a scene. Fixed scripts, emotional transitions, delivery constraints, and requested vocal reactions provide concrete tests of expressive control. Separating overall acting, instruction adherence, and authenticity makes the protocol more informative than a single preference or naturalness score.
-
The baseline and accompanying resources provide a concrete foundation for further research. The analysis also examines sensitivity to listener composition, distinguishes uncertain middle ranks from the more stable leading systems, and acknowledges the limited human validation of the perceptual scorers. These analyses help readers interpret the results without overstating what the current evaluation establishes.
Critical weaknesses
-
The connection between the documented task design and the reported evaluation is insufficiently established. Only 46 of the 560 votes concern 35 tasks from the 374-task registry; the remaining 514 use an extended pool whose generation procedure and relationship to the public acting briefs are not documented in the evaluation export. Twenty registry tasks were also used during director development, and the task-to-audio mapping is not yet frozen (pp. 4 and 9; Appendix A). Although the authors disclose these limitations, the uncertainty about task sampling, separation from development, and generation configurations limits reproducibility and the evidence supporting the proposed task design.
-
The reliability of the four judgment dimensions across listeners remains only partially characterized. Four listeners supplied 318 of the 560 votes, and each specific audio pair received one judgment, so item-level inter-rater agreement cannot be estimated. The listener-bootstrap and subgroup analyses provide useful evidence, including the stability of the top three systems after removing the most active listeners. However, authenticity ratings have a rank correlation of only 0.48 between paid and community listeners, and several middle ranks change substantially. The current evidence therefore leaves uncertainty about which distinctions would persist with a different listener sample.
Questions for authors
-
Can you clarify the generation, sampling, and model-pairing rules for the extended task pool, together with its relationship to the public briefs and director development? Using the existing records, can you identify a subset with documented generation configurations and no task reuse from development, and show whether the principal ranking conclusions persist?
-
Can you provide independent repeated judgments on a modest, representative subset of the same audio pairs, either from data collected after this snapshot or through a limited additional check? Agreement and uncertainty for each dimension would help establish whether the protocol yields reproducible distinctions, particularly for authenticity.
4 comments
Answer to YjMb (2/3): listener reliability, repeated judgments, and presentation
Listener concentration. With at most 10 to 30 votes per listener, Gemini stays first in 98% to 100% of subsampling draws. With at most 5 it does so in 64%, and with equal weight per listener it drops to third behind Breeze and ElevenLabs. The leader therefore depends on the listener sample and on how listeners are weighted, and the revision reports this next to the leaderboard.
Paid and community listeners. We agree that the correlation of 0.48 between the authenticity boards of the two groups is a concern. A board fitted without one listener predicts 69% of the non-tied authenticity votes of paid listeners and 61% of those of community listeners. The pooled authenticity board, which paid votes dominate, therefore describes paid listeners better. We do not claim that the authenticity ranking would persist with a different listener sample. You also note that several middle ranks change. The first round does not resolve the order of the four middle entries, and the revision says so.
Split halves. The split-half result depends on how the halves are formed. Splits of the listeners into two halves with balanced vote counts give median rank correlations between the halves of 0.86 overall and 0.84 for authenticity. Across 1,000 uniform random splits into two groups of 35 listeners, the medians are 0.79 overall (interquartile range 0.73 to 0.85) and 0.75 for authenticity (0.65 to 0.83). Vote balancing always places the four heaviest listeners two against two, while random splitting lets them share a half. The revision reports both. Both are checks of the system ranking, and neither measures agreement on identical clips.
Repeated judgments (your second question). The study is running on 120 existing clip pairs, 40 per track and spread over small, medium and large rating gaps, with a target of three new judgments per pair from listeners who did not judge that pair before. For each question, and for authenticity in particular, we will report agreement and Krippendorff's alpha with intervals and the numbers of judgments and listeners. The pairs exclude the tasks of our second running study, which compares the selected take of our baseline with its first take, so they are not a random sample of the first round. We will state this limit with the results, and we will post what we have before the discussion closes.
Presentation. You note that parts of the paper take effort to read. The revision adds a "Protocol at a glance" box to Section 3 and describes there who wrote the extended tasks and why each pair has one vote. It reorganises Section 9 into limitations, discussion and conclusion, and it collects the additional analyses in one new appendix section with their tables.
The third comment answers the ethics concern.
Answer to YjMb (3/3): rights in the gated research branch
The data were collected and the models trained at a non-profit research organisation in the European Union. For copying and analysing speech recordings it relies on the research exception for text and data mining in EU copyright law (Directive (EU) 2019/790) and on the national law that implements it. These rules allow research organisations to copy works to which they have lawful access in order to analyse them automatically for scientific research, and to keep the copies, stored securely, for research and for checking results. Lawful access includes content that is freely available online. The podcast and broadcast recordings were made freely downloadable by their producers, without login or download restrictions, when we collected them.
The same national law allows a research organisation with non-commercial purposes to make such copies available to a defined group of people for their joint scientific research, and to individual third parties who check the quality of research. We rely on this provision for the gated branch. It gives processed audio, codec representations and annotations to researchers whom we approve one by one, under non-commercial research terms. It is not an unrestricted audio release, and access will end when the research project is completed. The ungated datasets described in the paper, including material from existing licensed datasets such as Emilia and Multilingual LibriSpeech and synthetically generated speech, will remain publicly available under their respective terms. Podcast Links itself contains links and metadata, no audio.
This basis is a research exception, not a licence for the recordings, and two limits follow.
Early checkpoints in the lineage of the released weights were trained with podcast snippets (see our answer to reviewer MscJ). We do not claim that the research exception clears the weights for commercial use, and the revised abstract no longer calls them commercially usable. The licence declared for the weights is a statement about the weights, not about the recordings behind them.
It does not replace data-protection law. Recordings of identifiable speakers are personal data. We process these recordings for scientific research. The podcast and broadcast research corpus described above remains behind the manual gate. We will add a contact address and a takedown procedure to every gated release before the camera-ready version.
The revised paper describes the gated branch as manually restricted research access, not as an unrestricted audio release.
The listener-consent correction is separate. The Arena and the scorer check used different participation agreements, and the revised ethics statement describes both.
Which preferences carry over to a new listener sample?
You asked for repeated judgments on the same recordings, particularly for authenticity. Thirteen new Prolific listeners, using the same instructions as the first round, gave 360 judgments on 120 first-round pairs, with 40 pairs per track and 40 per third of the overall system-rating gap. Of the 120 pairs, 106 received three new judgments, seven received two and seven received four. We excluded tasks reserved for the selector study, leaving only six recordings from our two entries.
| Question | Alpha among new listeners | New choice matches original | New choice matches higher-rated system |
|---|---|---|---|
| Overall acting | 0.12 (0.03 to 0.21) | 196/274, 71.5% (66 to 77) | 182/301, 60.5% (54 to 67) |
| Instruction fit | 0.08 (-0.01 to 0.15) | 151/215, 70.2% (64 to 76) | 167/279, 59.9% (53 to 66) |
| Authenticity | 0.11 (0.02 to 0.19) | 163/245, 66.5% (60 to 73) | 169/289, 58.5% (52 to 65) |
| Vocal reaction (34 eligible pairs) | 0.13 (-0.05 to 0.30) | 47/65, 72.3% (59 to 84) | Not estimated |
Krippendorff's alpha counts ties. Before the study we expected about 0.10 on the overall question, because two thirds of the pairs compare entries whose ratings predict at most a 70 to 30 preference. Matching the original requires both listeners to pick a clip. The final column requires only the new listener to pick one. For that post hoc check we fitted system ratings on the 440 first-round votes outside the repeated pairs. Parentheses give 95% intervals from resampling clip pairs.
Authenticity. Two new listeners who both chose a clip agreed on average on 58.9% of comparisons within pairs (51% to 67%). The new panel's most frequent answer matched the original on 54 of 70 pairs where both picked a clip. New listeners matched paid and volunteer originals in 66% and 68% of decisive judgments, respectively. Individual choices carry over to some extent, even though the paid and volunteer system rankings correlated only 0.48.
Which ranks persist? We fitted ratings to the new judgments and compared the nine entries other than our two sparsely represented baselines. Unlike the first round, no listener dominates this sample: 11 of the 13 listeners gave 30 judgments each and none gave more. Gemini ranked first overall in 99.5% of pair resamples and 98.7% of listener resamples. On authenticity the figures were 94.9% and 96.7%. The same three entries led together in only 52% and 47% of resamples overall, and 40% and 38% for authenticity. This post hoc check supports Gemini's lead among those nine entries in this sample. The order below it remains uncertain.
At export, 290 judgments had passed quality review and 70 were pending. On accepted judgments alone, overall alpha was 0.07 (-0.03 to 0.17), the match with the original was 69.5%, and the refitted system ratings predicted 61.6% of new choices. Table 6 and Appendix A.3 give both analyses and the sampling limits. Does this answer your question about which distinctions carry over to another listener sample?
Answer to YjMb (1/3): extended task pool, pairing, and development reuse
Thank you for the review. We answer in three comments: the task pool here, listener reliability in the second, and the ethics concern in the third.
Generation. Gemini 3.7 Flash wrote the 2,000 extended tasks in two runs of 1,000 on 12 and 13 September, at temperature 0.9 and in batches of eight, with recorded seeds and code versions. Each batch was prompted with six exemplars of the same track, drawn from 84 registry tasks. The target emotion or delivery and the requested reactions came from those exemplars. Scripts, directions and contexts were newly written. 472 of the 2,000 extended tasks received the 514 votes. A Gemini language model wrote these tasks and a Gemini speech model leads the board, so the task style may favour that system. The votes on registry tasks are too few to test this, and the revision says so.
Relationship to the public briefs. The public Acting Challenge briefs are a separate resource. The extended tasks and their exemplars are distributed with the demo, so they are not held out, and Sections 1 and 3 of the revision no longer describe them that way.
Sampling and pairing. The sampler serves the least-assigned clip pair first, then the least-compared system pair, then the least-judged task, with random tie-breaking and random A/B order. This rule gives new pairs priority over repeats, and in the first round no audio pair was judged twice. This describes vote assignment; the earlier choice of tasks for audio generation is not yet recovered.
Director development. The two director-prompt searches used 20 registry tasks. We reconstructed them from the saved sampling script and its seed. The original run files are not recovered. Six votes on four of these tasks are in the export, all among the 46 votes on registry tasks.
A subset without development reuse (your first question). Without the 46 registry votes, the other 514 votes keep the same three entries on top, and Gemini is first in 97% of listener resamples. But in the current exemplar file, seven of the 20 reconstructed development tasks are also generator exemplars, all in the vocal-delivery track, so extended tasks of that track may inherit target deliveries from development tasks. The subset that excludes this as well is the extended tasks of the emotion and acting-scene tracks: 373 votes from 58 listeners on 342 tasks. They have a documented generator, none of these votes is on a development task, and no exemplar of these two tracks is a development task.
On this subset the same three entries lead in the same order, and Gemini is first in more than 99% of listener resamples. Rank correlations with the full board are 0.99 overall, 0.97 for instruction fit and 0.92 for authenticity. The subset holds 67% of all votes, so high values are expected. Second and third place are less certain than first: the same three entries lead in 57% of resamples, against 86% with all votes. The remaining 141 extended votes are in the vocal-delivery track. On them Breeze is ahead of Gemini, and Gemini is first in 36% of resamples. With 141 votes we cannot tell whether this is noise, an effect of the development exemplars, or a difference between tracks. Both subsets go into the appendix.
The conclusions that persist are therefore the identity of the three leading entries and Gemini's first place on the pooled board. Two limits remain. The reconstruction of the development tasks is not confirmed by the original run files, and similarity between tasks beyond shared exemplars is not tested.
Generation configurations. One manifest links all 560 votes to task and director hashes, model identifiers, request settings and audio hashes, and all 1,084 audio files match their recorded hashes. Hashes do not reconstruct inputs that are not in the records, and hosted APIs may no longer return the same audio.