- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes CARE-VOICE, a method for judging audios by compiling a brief into a structured constraint graph, which an agentic judge then uses, along with a set of tools, to make a judgement or an abstention. CARE-VOICE is benchmarked on multiple real and synthetically-augmented audio datasets, with synthetic, dataset-derived, and human-derived ground-truth labels.
Critical strengths
-
The idea of compiling a brief into a structured graph is interesting.
-
The ability to abstain on ambiguous cases is important, and often overlooked in this area, so I’m glad that this paper tries to tackle it.
Critical weaknesses
-
The paper is written in a very dense, indirect style that makes it very hard to understand. The abstract uses several non-obvious terms (e.g. “Voice Constraint Graph”, “tool health”) without defining them and lists a bunch of observations without explaining how they fit into an overall story. This persists throughout the paper; as just one example, consider this sentence in Section 4.3: “For a weak-proxy conflict in the real-service pilot, primary perceptual evidence may override the proxy only if the judge demonstrates access to both files, returns the same canonical direction after swapping A/B order, reaches a frozen confidence of 0.80, and has no high-severity evidence-audit error.” There are several terms (“real-service pilot”, “judge access”, “frozen confidence”, “high-severity evidence-audit error”) that have not been defined at this point, and even ones that have (e.g. “weak-proxy”) aren’t clear because no examples have been given. Likewise at the beginning of Section 5: “Controlled transformations provide exact labels for evaluator mechanics.” It is not clear what this sentence means based on the previous parts of the paper. I would strongly suggest the authors rewrite the paper to make the story and prose much clearer. Be sure to include illustrative examples of utterances, briefs, graphs, evidence sets, and judgements in the main paper so that the reader can get a better intuition for these. Be sure to explain the purpose of each test and what each result shows. The prose reminds me of AI-generated writing, so if that’s how this paper was generated then I would suggest the authors rewrite it in their own words.
-
It is not clear from Table 1 and Table 2 if abstention is useful. In Table 1, Robust DSP already gets 100% overall accuracy, so CARE-VOICE must be abstaining on decideable cases. In Table 2, pitch-only already gets 90% overall accuracy, while CARE-VOICE abstains on 67.5% of cases, suggesting that at least 84% of the abstained cases must’ve been decideable. To be fair, Table 3 does demonstrate some possible benefit from abstentions.
-
It is not clear if all the relevant baselines have been covered. There ought to be some comparison to ALM-only approaches such as AudioJudge and existing tool-based approaches such as TRACE. It is not clear what baselines are being considered in Table 3 (does “admitted through frozen gate” mean that Gemini 3.5 Flash worked on its own as an ALM-only judge, or something else?).
Questions for authors
See weaknesses. In order for this paper to be acceptable, its presentation would have to be much clearer throughout, and it would need to cover all relevant baselines. I would suggest the authors revise and resubmit this paper.