acceptodds
Withdrawn from ICLR 2027

Towards Reliable General-Purpose Speech Editing Agents: Formulation, Design, and Empirical Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

General-purpose speech editing should satisfy requested changes while preserving content and acoustic properties that should remain unchanged. Specialized speech models provide complementary capabilities, but reliably composing them requires accurate observation, robust interpretation of complex editing intent, executable tool composition, and assessment of realized outcomes. We formulate agentic speech editing as a partially observable Markov decision process (POMDP), separating latent editing states, observations, belief, actions, environment transitions, reward, and policy revision. Guided by this formulation, we introduce X·Edit, a training-free speech editing agent that combines active evidence acquisition, optional belief materialization, structured tool planning, outcome evaluation, and feedback-guided contextual policy revision. On all 426 MMAE speech tasks, X·Edit achieves 54.30% instruction following, 93.05% consistency, and 20.89% exact match with up to three policy revision, performing as the state-of-the-art current speech editing system. Requiring both complete delivery and passing all native checks yields 20.19% delivered success. Controlled studies further show that acquired conditioning evidence improves fine-grained observation fidelity, explicit belief helps retain complex interacting requirements, and higher-level action representations reduce the burden of composing fragmented speech tools. Feedback-guided recovery substantially improves delivery, while gains in complete fulfillment remain smaller, highlighting reliable preservation, reward estimation, and candidate selection as remaining challenges. Together, our formulation, system design, and empirical analysis provide a principled framework and practical guidance for building reliable general-purpose speech editing agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech editing, multimodal agent, partial observation Markov decision process, speech synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3HbkOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces an agent to edit voice.

Critical strengths

The topic is useful.

Critical weaknesses

The EMR is too low. Which mean 80% of the task cannot fullfill quality checks?

The paper needs to compare with more agent-based systems.

The paper cites results from other paper directly, which means the evaluation may differ.

The paper needs ablation study.

Questions for authors

Comparing with other agent-based methods, what is the novelty of this paper?

Reviewer_DELhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for general-purpose speech editing based on the partially observable Markov decision process (POMDP) formulation. The paper demonstrate the effectiveness of the proposed framework on MMAE benchmark and further studies several sources of failures.

Critical strengths

Given the lack of general-purpose speech editing systems, even though there are many specialized models, the idea of adopting an agentic framework is novel because it is training-free and shows potential of combining the strength of several specialist models. From this perspective, the paper has sufficient novelty.

Critical weaknesses
  1. The paper is a little hard to follow. Several important details, such as tool set, observation metrics in RQ2, etc., are left in the appendix, which makes the main text not sufficiently self-contained. In addition, the evaluation metrics, such as IFR and CR, are only briefly mentioned but not rigorously defined. Even when these metrics are from existing benchmarks, the author should still provide formal definition and how to measure these quantities in the paper. Hence, I believe the paper still require substantial improvement in terms of clarity.
  2. While the instruction-following rate (IFR) is not clearly defined in the paper (and also the original MMAE paper), for an agentic system, which uses a strong LLM as the central moderator, the instruction-following rate of only 54.30% seems unsatisfactory. The author should provide a more rigorous study on this phenonmenon to distinguish the fundamental limitations of the agentic framework and the potential issues in the implementation. The current manuscript lack a deeper analysis.
  3. It will be helpful if the paper can include a study of computational cost of the proposed agentic system. This can help the reader understand the potential efficiency-effectiveness trade-off between using agentic framework and a single model like the baselines compared in the paper.
  4. The paper lack an ablation study on each component. Comparing using different models for each component can make the paper more comprehensive. In addition, the reason of adopting the formulation of POMDP seems not empirically justified. Whether this is a good setting? Or if there is another possibility? I will suggest the author to provide a study, or at least a detailed discussion, to make this point clearer.
Questions for authors

I have listed the weakness. Please refer to the weakness part. One formatting issue is that Table 1 goes beyond the page margin. Please refine it to fit the template.

Reviewer_KJJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper created a general-purpose speech editing tool, which is a training-free multi-module agent. It uses expert tools such as ASR alignment, diarization, and reasoning to find and plan the sections to be edited. Then, it uses a guider and evaluator to assess the final performance. According to the authors, the final success rate increased significantly.

Critical strengths

Overall, this is an interesting and relevant problem to solve: having a general-purpose speech editing tool.Instead of making a complex, single-tool solution, the author changed to an agentic approach to break down and solve problems in separate pieces.The application of diagnostic studies is pretty detailed, and findings are reported honestly.

Critical weaknesses

The biggest issue with this paper is that the agent and the grader are the same model, so there is a risk of hacking from the agent just to please the grader instead of truly improving ability.Second, all the conventional speech editing metrics (WER, CER, speaker similarity, etc.) were not reported. The human study is also very limited, with only a few listeners and too few samples.The overall improvement is too small as well. The title indicates this is a Markov chain decision, but overall speaking, I don't see any Markov inference. This is basically stitching several tools together as more of a flowchart; the innovation of this paper is rather limited.Finally, there is not enough of a baseline for existing agent frameworks doing speech editing. The author did not offer a comparison with those baselines.

Questions for authors
  1. Can you report WER/CER, speaker similarity, UTMOS and preservation of unedited regions?
  2. How does a simple ReAct loop with the same models and tools perform?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.