- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
People who use a dictation app often fix the text afterwards. The paper investigate the users' corrections can be useful as a training signal, e.g., if we can treat every fix as a correction and use the fixing log as training data to improve the ASR. This paper checks whether that’s a good or plausible idea. The paper dissects this into two questions: can you tell why a user edited the text (so that they can exclude user editing that does not involve ASR correction), and does learning from edits actually make the system better?
Critical strengths
-
The most exciting part of the paper is its idea, using the user correction edits as paired training data, and they go further, dissecting into different stages to see if it is possible to filter out noisy training data.
-
The 900 bilingual/code-switch and multimodal data sets are very valuable, as every edit has its audio recording. This metadata can be used in many other future directions.
Critical weaknesses
-
The paper is built on the idea that knowing why a user edited can be used to identify useful correction data, but this idea has never been tested, at the end of this paper, that using the filtered data to train a new model. The paper doesn’t test this. It is like a validation of a hypothesis for a hypothesis, and the current results show a mixed answer.
-
E1 is try to answer the big question but actually, the test set contains only utterances that users edited. In real use, most outputs are accepted without changes. Adding vocabulary terms or “don’t do this” examples could easily break some of those, and E1 would never notice.
Questions for authors
-
"unintelligible edits were excluded". what is exactly "unintelligible edits" means, the recording is hard to tell, or the text edits do not make sense. What is the portion?
-
Why routing never stored a “don’t do this” example. The paper presents this as something the audit found. But the audio rule has no route to OVER-REFINEMENT: its three questions only lead to style, mishearing or change of intent. No human annotator used that label either. So S3 with the rule, S4, S3+ and S4+ could never store a negative example; it was built into the design. The paper should say this plainly, because it changes how you read “routing by cause did not help.”