acceptodds
Under review as a conference paper at ICLR 2027

CARE-Voice: Constraint-Aware Reliable Evaluation via Selective Decisions for Voice Design

Abstract

An open-ended voice-design brief may specify attributes to change, properties to preserve, permitted exceptions, and conflicting goals, so no single scalar score can carry the evaluation. We introduce CARE-Voice (Constraint-Aware Reliable Evaluation via Selective Decisions for Voice Design), a constraint-aware framework that compiles a brief into a Voice Constraint Graph (VCG), records typed evidence and provenance, audits tool health, and delegates the final Pass/Fail/Abstain decision to a frozen selective scorer. The evaluator may acquire evidence adaptively, but it cannot rewrite the decision contract after observing a candidate. The evidence we assemble supports a reliability claim, not a human-level perceptual claim. CARE-Voice is evaluated on complementary evidence tiers: 96 controlled counterfactual conditions with exact labels, 40 parallel semantic pairs, 90 single-attribute and 52 multi-constraint outputs from a deployed service, and existing blinded-listening studies; no new human ratings are collected, and the underpowered multi-constraint study serves only as exploratory failure analysis. Three findings summarize the evidence. First, global RMS, a common shorthand for vocal energy, agrees with the human majority on only 25% of semantic pairs, whereas typed evidence with selective refusal reaches 100% selective agreement at 42.5% coverage. Second, against strict human majorities on single-attribute pairs, CARE-Voice retains comparable agreement to a strong direct judge (84.0% selective at 75.8% coverage versus 81.8% forced), with overlapping uncertainty and therefore no superiority claim. Third, abstentions and position-inconsistency checks flag half of the pairs on which listeners fail to reach a majority, exposing ambiguity that forced labels conceal. Constraint structure prevents unsupported penalties, evidence authority blocks weak proxies from deciding alone, and selective refusal turns judge instability, identity drift, and tool failures into recorded, auditable events.

Primary area:
datasets and benchmarks
Keywords:
voice design evaluation, constraint-aware evaluation, selective prediction, typed evidence, tool-health auditing, text-to-speech, reliability benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uxPCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CARE-VOICE, a method for judging audios by compiling a brief into a structured constraint graph, which an agentic judge then uses, along with a set of tools, to make a judgement or an abstention. CARE-VOICE is benchmarked on multiple real and synthetically-augmented audio datasets, with synthetic, dataset-derived, and human-derived ground-truth labels.

Critical strengths
  1. The idea of compiling a brief into a structured graph is interesting.

  2. The ability to abstain on ambiguous cases is important, and often overlooked in this area, so I’m glad that this paper tries to tackle it.

Critical weaknesses
  1. The paper is written in a very dense, indirect style that makes it very hard to understand. The abstract uses several non-obvious terms (e.g. “Voice Constraint Graph”, “tool health”) without defining them and lists a bunch of observations without explaining how they fit into an overall story. This persists throughout the paper; as just one example, consider this sentence in Section 4.3: “For a weak-proxy conflict in the real-service pilot, primary perceptual evidence may override the proxy only if the judge demonstrates access to both files, returns the same canonical direction after swapping A/B order, reaches a frozen confidence of 0.80, and has no high-severity evidence-audit error.” There are several terms (“real-service pilot”, “judge access”, “frozen confidence”, “high-severity evidence-audit error”) that have not been defined at this point, and even ones that have (e.g. “weak-proxy”) aren’t clear because no examples have been given. Likewise at the beginning of Section 5: “Controlled transformations provide exact labels for evaluator mechanics.” It is not clear what this sentence means based on the previous parts of the paper. I would strongly suggest the authors rewrite the paper to make the story and prose much clearer. Be sure to include illustrative examples of utterances, briefs, graphs, evidence sets, and judgements in the main paper so that the reader can get a better intuition for these. Be sure to explain the purpose of each test and what each result shows. The prose reminds me of AI-generated writing, so if that’s how this paper was generated then I would suggest the authors rewrite it in their own words.

  2. It is not clear from Table 1 and Table 2 if abstention is useful. In Table 1, Robust DSP already gets 100% overall accuracy, so CARE-VOICE must be abstaining on decideable cases. In Table 2, pitch-only already gets 90% overall accuracy, while CARE-VOICE abstains on 67.5% of cases, suggesting that at least 84% of the abstained cases must’ve been decideable. To be fair, Table 3 does demonstrate some possible benefit from abstentions.

  3. It is not clear if all the relevant baselines have been covered. There ought to be some comparison to ALM-only approaches such as AudioJudge and existing tool-based approaches such as TRACE. It is not clear what baselines are being considered in Table 3 (does “admitted through frozen gate” mean that Gemini 3.5 Flash worked on its own as an ALM-only judge, or something else?).

Questions for authors

See weaknesses. In order for this paper to be acceptable, its presentation would have to be much clearer throughout, and it would need to cover all relevant baselines. I would suggest the authors revise and resubmit this paper.

Reviewer_uZQNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

this paper presents CARE-Voice, which combines a Voice Constraint Graph, typed evidence, tool-health checks, and frozen PASS/FAIL/ABSTAIN rules for voice-design evaluation. the evaluation includes controlled audio transformations, semantic pairs, generated speech, and existing listener studies

Critical strengths
  1. the problem itself is meaningful and well-motivated. Separating evidence acquisition from decision authority makes unsupported judgments, tool failures, and abstentions explicit.
  2. the experimental findings, though were grounded on limited experiments, are generally useful diagnostic evidence, which could be informative
Critical weaknesses
  1. the biggest problem of this paper is that the exprimental part is not convincing. the size of human comparison is too small and compares different coverage levels. There's no well-established baseline covers the whole domains tested
  2. the presentation can be somehow confusing, figure 1 is hard to read as the font size is just too small, and I've never seen any paper with multi-paragraph abstract. Appendix K is even more confusing with bash commands point to nowhere. This is apparently AI slop in paper writing, so I have to pick this out as a separate point
Questions for authors
  1. are the 30 structural graphs generated automatically from raw briefs or supplied as gold inputs? Which reported results actually measure parsing errors then
  2. i'm curious how does direct judging perform on CARE-Voice's accepted pairs, and how does a simpler selective judge compare at the same coverage
Reviewer_VPKQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents CARE-VOICE, a method for evaluating voice-design systems using explicit constraints, different types of evidence, and the option to abstain when results are uncertain. The framework performs well on controlled tests, but its effectiveness on realistic voice-design tasks remains less clear.

Critical strengths
  • The paper addresses an important problem in voice-design evaluation and avoids relying on a single audio metric.
  • The constraint graph clearly separates requested changes, preservation requirements, and unconstrained attributes.
  • The framework checks judge consistency, tool failures, and uncertainty instead of forcing unreliable decisions.
Critical weaknesses
  • Largely written by AI , sloppy text in most places. Figures seem very poorly done via claude.
  • The human evaluation is small and does not establish an improvement over direct judging.
  • The multi-constraint results are exploratory and show substantial inconsistency in model judgments.
  • Generalization across different audio-language models and the claimed cost benefits remain unclear.
Questions for authors
  • Can the authors validate CARE-VOICE on a larger human-listening benchmark?
  • How does the framework perform when voice attributes change but speaker identity must be preserved?
  • Can the authors demonstrate reliable multi-constraint evaluation across different judge models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.