acceptodds
Under review as a conference paper at ICLR 2027

LEARNING FROM USER EDITS IN DICTATION SYSTEMS: ATTRIBUTION AND ADAPTATION

Abstract

Users of deployed multi-stage systems edit what the system produced, and it is tempting to learn from every edit as if it were a correction. We test this assumption on a production dictation pipeline (speech recognition followed by LLM post-processing) with 900 edits from 45 users. In a non-random human-labeled subset of 359 edits, 47.9% added or removed content rather than correcting a mishearing. The cause of an edit can be estimated: an LLM that sees the logs and an independent transcript of the recording assigns the same category as a human annotator for 81.0% of held-out edits, against 50.2% for always predicting the most common category, although most labels were assigned with a transcript-based suggestion visible. In an adaptation study scored against proxy references, vocabulary learned from edits reduced errors on corrected utterances by 2.2–3.0 points, whereas negative examples learned from every edit increased errors, an increase a small hand-transcribed subset did not confirm. Routing updates by the estimated cause never beat learning vocabulary from every edit. Updates learned from edits need their own validation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
user edits, implicit feedback, feedback attribution, automatic speech recognition, LLM post-processing, contextual biasing, non-parametric adaptation, human-in-the-loop learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yjTpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

People who use a dictation app often fix the text afterwards. The paper investigate the users' corrections can be useful as a training signal, e.g., if we can treat every fix as a correction and use the fixing log as training data to improve the ASR. This paper checks whether that’s a good or plausible idea. The paper dissects this into two questions: can you tell why a user edited the text (so that they can exclude user editing that does not involve ASR correction), and does learning from edits actually make the system better?

Critical strengths
  • The most exciting part of the paper is its idea, using the user correction edits as paired training data, and they go further, dissecting into different stages to see if it is possible to filter out noisy training data.

  • The 900 bilingual/code-switch and multimodal data sets are very valuable, as every edit has its audio recording. This metadata can be used in many other future directions.

Critical weaknesses
  • The paper is built on the idea that knowing why a user edited can be used to identify useful correction data, but this idea has never been tested, at the end of this paper, that using the filtered data to train a new model. The paper doesn’t test this. It is like a validation of a hypothesis for a hypothesis, and the current results show a mixed answer.

  • E1 is try to answer the big question but actually, the test set contains only utterances that users edited. In real use, most outputs are accepted without changes. Adding vocabulary terms or “don’t do this” examples could easily break some of those, and E1 would never notice.

Questions for authors
  • "unintelligible edits were excluded". what is exactly "unintelligible edits" means, the recording is hard to tell, or the text edits do not make sense. What is the portion?

  • Why routing never stored a “don’t do this” example. The paper presents this as something the audit found. But the audio rule has no route to OVER-REFINEMENT: its three questions only lead to style, mishearing or change of intent. No human annotator used that label either. So S3 with the rule, S4, S3+ and S4+ could never store a negative example; it was built into the design. The paper should say this plainly, because it changes how you read “routing by cause did not help.”

Reviewer_92e2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper evaluates whether user edits in a dictation pipeline (speech recognizer with LLM post-processing) function as transcription corrections. Analyzing 900 edits from 45 users (359 human-labeled), about half modified content rather than fixing mishearings. The best LLM classifier prompt matched human labels on 81% of held-out edits across five cause categories. A replay sim showed that learning vocab from edits reduces error on corrected statements, using all edits as neg ex increases error against proxy refs, and routing updates by inferred cause does not outperform simple vocab learning. The authors conclude that edit-derived updates require independent validation.

Critical strengths

The paper provides valuable real deployment data with recordings and shows strong transparency in acknowledging its limitations.

Critical weaknesses
  1. The evidence leaves the practical benefit of cause-based routing unresolved:
  • Attribution eval is anchored: 200 of 205 held-out labels were assigned with audio rule's suggestion visible. The reported 81% reflects agreement with suggested-anchored labels, rather than accuracy to the true cause.
  • No routed strategy learned negative examples from correction-labeled edits, leaving the benefit of this targeted update untested.
  • The deployment measurement is thin. Only 20 labeled edits drawn at random, the intent-change share is 40-64% across labeling sheets, and one annotator labeled everything...
  1. Remaining findings overlap with existing literature
  • Zhou et al. (ASRU 2023) already note that dictation edits may be revisions of intent rather than corrections, and filter for likely corrections before learning from them.
  • Validating learned updates offline before deployment is done in prod (Ponnusamy et al., AAAI 2020, Alexa self-learning). Neither is cited, and the submission has only six references.
Questions for authors
  1. Was it known before Expt 1 that the rule- and label-routed strategies could never store neg examples, since nothing could output OVER-REFINEMENT?

  2. What is classifier agreement on fully blind annotations?

  3. How does your cause-based routing differ from the filtering in Zhou et al. (2023) and the offline validation in Ponnusamy et al. (2020)?

  4. Does the vocabulary gain hold up when users' existing production vocabularies are loaded in the simulation?

Reviewer_LRiFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines whether user edits in a production dictation pipeline (ASR followed by LLM post-processing) can be treated as corrections. Using 900 edits from 45 users, the authors define five possible causes of an edit, each mapped to a candidate non-parametric update (or to no update, for intent changes). They compare rule-based and prompted-LLM classifiers (Gemini 3.8 Flash) for inferring the cause against a human-labeled subset. They then simulate per-user adaptation to test which update strategies reduce recognition errors.

Critical strengths

A practically relevant question studied on real deployment data.

Critical weaknesses
  1. The classifiers are untrained rules or prompted large language models, and the main positive finding (learning vocabulary from every edit helps) is fairly intuitive. Related work is thin (six references), missing areas such as automatic speech recognition personalization and machine translation post-editing. The data and production prompts are not released, and the closed, versioned commercial models make replication difficult.
  2. The adaptation study covers 22 users with a median of 18.5 adaptation edits each, and the data are skewed, with the 13 most active users contributing 762 of the 900 edits. Each strategy was run only twice, yet 42 of 245 test utterances changed between the two S0 runs, so two runs give limited insight into sampling variability. The simulated pipeline also differs from production: context is given only to post-processing and users’ existing vocabularies are not loaded, so it is unclear whether the vocabulary gains carry over to a production recognizer that already uses context and a user vocabulary. Finally, the size threshold used to select edits is not specified, and the edits come from a single application on one platform. Although some of these issues are acknowledged in the limitations section, I believe they substantially weaken the paper’s conclusions rather than being minor limitations.
  3. The taxonomy does not support the multi-stage framing. The paper is motivated by identifying which stage caused an edit, but this question is largely not exercised. First-stage cleanup errors are attributed to recognition by design, and none of the 115 human-labeled edits in which post-processing changed the text was labeled over-refinement, a category for which the annotation guideline has no decision step. In practice, the task reduces to distinguishing corrections, intent changes, and style edits. In addition, each edit receives a single label, and the audio-grounded rule labels an edit as an intent change whenever it judges any span to be new content, even if it judges another span to be a misheard word.
  4. The human labels are not an independent ground truth, which limits what the attribution results show. Of the 205 held-out edits, 200 were labeled by accepting or changing the audio-grounded rule’s suggestion, and only 5 were labeled blind. The reported 81.0% agreement is therefore agreement with suggestion-assisted labels, as the authors note, rather than evidence that the classifiers recover the true cause. The best-performing prompt also reads the same Whisper transcript that produced those suggestions; the authors argue its gain is concentrated on edits where the annotator rejected the suggestion, but with so few blind labels this cannot be ruled out. Moreover, all labels come from a single annotator, and agreement with a second annotator on a separate sample is moderate (κ = 0.50, with a lower bound of 0.30), so the reference itself is noisy. Finally, the headline finding that 47.9% of edits are intent changes comes from a non-random subset, only 20 of whose 359 edits were drawn uniformly at random, and the share varies from 40% to 64% across its parts, so it does not generalize to the full set of edits.
  5. The paper is hard to follow. The text is heavily hedged, key information for interpreting the main results is deferred to the appendices, and the many strategy variants and dense tables make the results difficult to read. The manuscript also contains editing issues, such as a broken reference (“Appendix ??”).
Questions for authors

Please refer to the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.