acceptodds
Withdrawn from ICLR 2027

Word-Level Text Unmixing via Evidence-Preserving Ownership Routing with Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Text from multiple sources can become interleaved into a single sequence when attribution metadata is lost, such as overlapping speech transcripts, complex document reading flows, or concurrent agent streams. While all observed words remain present, their underlying source ownership is unknown. We formalize this challenge as Word-Level Text Unmixing: given an interleaved lexical stream and the source count K, recover the original source sequences while preserving every word occurrence and its within-source order exactly. Conventional large language model (LLM) paradigms approach this by generating separated source texts directly. However, coupling latent ownership inference with lexical regeneration frequently causes LLMs to omit, duplicate, or hallucinate words, violating the exact-reconstruction objective. To resolve this, we propose Evidence-Preserving Ownership Routing (EPoR), a framework that cleanly separates ownership prediction from sequence reconstruction. During training, EPoR adapts a causal LLM to predict canonical ownership routes conditioned on the interleaved stream and prior routing decisions, inducing dynamic, history-conditioned source contexts without lexical regeneration. At inference, EPoR pairs completion-safe constrained decoding with deterministic indexed reconstruction, guaranteeing structurally valid K-source partitions that preserve every observed token occurrence exactly once. For systematic evaluation, we introduce UnMixBench, spanning controlled synthetic mixtures, timestamp-derived speech from AMI and ICSI, layout-derived document streams from ReadingBank, and simulated concurrent digital outputs. Across five evaluation tracks, a 4B EPoR model achieves the lowest minimum-permutation word error rate (MP-WER) among finetuned baselines, reducing the five-track mean by 22.3% relative to compact source-array generation and competing favorably with zero-shot frontier LLMs. These findings indicate that when lexical evidence is fully observed, decoupling ownership routing from sequence reconstruction can provide a more reliable and structurally sound alternative to generative decoders.

Primary area:
foundation or frontier models, including LLMs
Keywords:
text unmixing, large language models, structured prediction, constrained decoding, ownership prediction

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_o9pWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates Word-Level Text Unmixing, a task of recovering the original sources from interleaved word sequences without source attribution metadata. In this task, words from K different sources are mixed together, and the model aims to separate them while preserving the original words and their order. When LLMs directly generate the separated source texts, the original lexical evidence can be modified or omitted. To address this issue, the authors propose Evidence-Preserving Ownership Routing (EPoR), a learning-based framework that predicts source labels instead of generating the source texts. EPoR fine-tunes an LLM to autoregressively predict the source label for each word. During inference, the authors apply constrained decoding to ensure valid source assignments without empty sources and reconstruct the source texts based on the predicted labels. The proposed EPoR is evaluated on UnMixBench, a benchmark introduced by the authors for Word-Level Text Unmixing, and shows generally better performance than fine-tuned generative and structured baselines, as well as zero-shot LLM baselines.

Critical strengths
  • The authors identify and formulate a novel problem, Word-Level Text Unmixing.
  • The proposed architectural design choices are well supported by thorough ablation studies.
Critical weaknesses
  • Word-Level Text Unmixing seems impractical.
    • The authors explain the scenarios that require word-level text unmixing as 1) multi-speaker dialogue without source attribution, 2) document reading-flow unmixing, and 3) dialogue unmixing in agentic systems. However, it is common to use external information such as audio for scenario 1) and document layout information (e.g., tables or figures) for scenario 2). For scenario 3), it is hard to imagine such a scenario where metadata is omitted. Therefore, the practical importance of the proposed problem is unclear to me. This concern is also reflected in their benchmark, which is partly constructed using synthetically mixed data.
  • The benchmark and baselines are not representative.
    • This concern is also related to my first concern. The paper proposes a new benchmark, UNMIXBENCH, but its representativeness or validity in simulating practical scenarios is not fully assessed. I guess this concern might be addressed if the authors resolve my first concern.
  • The novelty of the method should be validated in more detail.
    • Liu et al. [1] propose a transition-based dialogue disentanglement model that maintains and updates session states based on previous assignments. A relevant baseline would adapt this model to word-level prediction, using the same LLM backbone while retaining its session-state LSTM and build/update mechanism. The adapted model should be trained on the same data and evaluated on UnMixBench under matched input and decoding conditions, including access to the full mixed stream and the known source count. This comparison would help clarify the benefit of EPoR over explicit recurrent source-state modeling. The authors should discuss this work and clarify the novelty of their method compared to it.
    • [1] Liu et al., End-to-End Transition-Based Online Dialogue Disentanglement, IJCAI, 2020.
Questions for authors

Please see the weaknesses above.

Reviewer_1a6POpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose word level text unmixing, a task of recovering the original source sequences from a single stream in which words from K sources are interleaved. Given the mixed word stream and the number of sources K, the goal is to recover each source while using every observed word exactly once and keeping its original order within the source. To solve this task, the authors propose Evidence-Preserving Ownership Routing (EPOR). Instead of generating the separated texts directly, EPOR finetunes an LLM to predict a source label for each word, conditioned on the mixed stream and the previous labels. At inference, completion-safe constrained beam search keeps only valid label sequences, and each source is reconstructed by collecting the original words according to the predicted labels. The authors also introduce UNMIXBENCH, built from MultiNLI, AMI, ICSI, ReadingBank, and simulated concurrent streams. Experiments show that a 4B EPOR model achieves compatible with respect to other baselines with bigger size.

Critical strengths
  1. The problem formulation and method design are clear. The authors formally define word level text unmixing, and the idea of separating label prediction from reconstruction is simple but well motivated. Since each source is reconstructed directly from the input words, EPOR always preserves every observed word and always produces exactly K non-empty sources.
  2. The experiments are carefully designed. The model is trained only on SNLI and evaluated on other domains without using any target-domain data. The ablation studies examine each component, including task adaptation, label history, legality masking, and canonical labels. In addition, the paired construction set is a good attempt to check whether the model simply learns the patterns of a specific mixing procedure.
Critical weaknesses
  1. The novelty of the problem and the benchmark is unclear. Word level text unmixing seems closely related to existing tasks such as conversation disentanglement, speaker diarization with lexical information, and reading order recovery. The main difference is the setting with no metadata, error-free words, and known K, but it is unclear how often this setting happens in real scenarios. In most of the motivating examples, timestamps, audio, or layout information would usually remain, and ASR/OCR outputs would contain errors. The proposed method also mainly combines existing techniques, such as restricted-growth encoding, autoregressive label prediction, and constrained decoding, as the authors acknowledge in the related work. In addition, UNMIXBENCH is constructed by interleaving existing datasets using their original timestamps or layout information, so even the AMI, ICSI, and ReadingBank tracks are still synthetic mixtures.
  2. The experimental results are not strong enough. In Table 2, compared to the structured baselines that also preserve all words, the improvement of EPOR on the realistic tracks is small, and the largest gain comes from MultiNLI, which is the most similar to the training data. EPOR also performs worse than RGS-CRF on AG News, and the ownership accuracy on the realistic tracks is still around 0.6. Moreover, Qwen3.8-Max outperforms EPOR on four of five tracks and on the average MP-WER, so the claim that EPOR competes favorably with frontier LLMs could be stated more carefully. In Table 3, canonical labels mainly reduce training cost and legality masking has only a small effect on MP-WER. Therefore, it is unclear how much the specific design choices of EPOR contribute to the final performance.
Questions for authors

The current formulation assumes that the number of sources K is known, but K is often unknown in real scenarios. How could EPOR be extended to also predict the number of sources? For example, since the canonical labels introduce new sources one by one, could the model learn when to stop introducing new sources without the max = K constraint? It would also be interesting to see how robust the current model is when a wrong K is given.

Reviewer_Vu3uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper’s central idea is to predict which source owns each observed word, rather than regenerate the separated texts, avoiding copying errors while tackling the underlying attribution problem.

Paper and contributions

  • Problem formulation: Word-level text unmixing recovers sources from an order-preserving interleaving, given the source count (K). Only the words are available.
  • Method (EPOR): A finetuned causal LLM predicts one ownership label per word, conditioned on the full input and previous assignments. First-appearance labels remove source-naming redundancy; completion-safe beam search ensures all (K) sources appear; deterministic reconstruction gathers the original words.
  • Benchmark: UNMIXBENCH covers controlled MultiNLI mixtures, AMI/ICSI meeting transcripts, ReadingBank-derived reading flows, and simulated concurrent WikiText streams.

Validation

  • Structural guarantees: The analysis and small-case exhaustive tests establish exactly (K) nonempty outputs, one-time use of every input occurrence, and preserved input order within each output. These guarantees do not establish correct ownership.
  • Empirical results: A 4B model trained on 10,000 SNLI mixtures achieves the best five-track average among finetuned baselines: 22.3% lower MP-WER than compact source-array generation, with exact evidence preservation. It beats the stronger turn-marker generator on four tracks, with paired confidence intervals excluding zero; Digital is effectively tied.
  • Component checks: Ablations support task adaptation and assignment-specific history. Legality masking primarily ensures structure; canonical labels match permutation-invariant training quality at roughly one-quarter the measured training time. Additional tests cover alternative mixers, backbones, and domains.
Critical strengths
  • A principled separation of ownership inference from evidence-preserving reconstruction. EPOR predicts assignments over indexed word occurrences and reconstructs sources deterministically, guaranteeing exactly (K) nonempty outputs without dropping, duplicating, or altering observed words. The paper clearly distinguishes these structural guarantees from correct attribution. This is consequential because it directly addresses a fundamental mismatch between unrestricted text generation and an exact-reconstruction task, rather than relying on prompts to encourage faithful copying.
  • Evidence that assignment-conditioned routing improves recovery beyond ensuring valid outputs. EPOR achieves the lowest five-track average MP-WER among finetuned systems and outperforms the stronger turn-marker generator on four tracks, with paired confidence intervals excluding zero. More importantly, the matched history ablation raises MultiNLI MP-WER from 0.3757 to 0.4486 while preserving the same structural guarantees. This supports the value of assignment-specific history—not merely the benefit of preventing lexical errors—and makes the method a substantive modeling contribution rather than only an output-format constraint.
Critical weaknesses
  • Limited evidence of practically useful recovery in the motivating settings. The main evaluations use 16–64-word streams with known source counts and complete lexical observations; digital streams are simulated, and ReadingBank “sources” are metadata-derived increasing runs rather than human-validated semantic flows. EPOR’s MP-WER remains approximately 0.62–0.65 on the speech, spatial, and digital tracks despite perfect evidence preservation. This is consequential because preserving every word does not ensure meaningful separation: the experiments establish structural reliability more convincingly than usable recovery in realistic pipelines.
  • The comparisons do not fully isolate the reasons for the performance gains. EPOR uses beam search while generation baselines use greedy decoding, and the structured routing heads have different context and optimization budgets. The headline 22.3% improvement is against a source-array pipeline with substantial parsing and copying failures, so it combines attribution improvements with output-compliance improvements. The turn-marker comparison and matched history ablation strengthen the evidence, but do not eliminate these broader confounds. This matters because the central empirical argument would be more persuasive with tightly controlled comparisons separating the benefits of learned routing, search, and deterministic reconstruction.
Questions for authors
  • Do the structured baselines’ 512-token contexts truncate any training or evaluation inputs, compared with EPOR’s 1,024-token context? If so, how much of EPOR’s advantage remains on examples where every system sees the complete input? This would affect whether the comparison demonstrates better ownership modeling or partly reflects unequal input coverage.
  • The semantic-identifiability audit covers controlled MultiNLI examples. What evidence supports recoverability of the recorded partitions from lexical input alone on speech and ReadingBank tracks? Could materially different, linguistically defensible partitions account for substantial measured error? This would affect whether MP-WER measures recoverable attribution mistakes or penalizes unavoidable ambiguity.
Reviewer_mctDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Word-level Text Unmixing, a task that reconstructs multiple source texts from interleaved word occurrences, and proposes EPOR, a history-conditioned autoregressive ownership-routing approach with constrained decoding and deterministic reconstruction. To demonstrate EPOR, the paper make a new benchmark and compare with various baselines within the benchmark, showing strong performance of the proposed method.

Critical strengths
  • Strong empirical performance. EPOR achieves the best average performance among task-finetuned baselines and remains competitive with much larger zero-shot frontier LLMs, while guaranteeing exact preservation of input words.

  • New benchmark. The paper introduces UnMixBench, covering diverse text-interleaving scenarios across multiple domains, providing a useful benchmark for fine-grained source reconstruction.

  • Extensive analysis. The paper provides comprehensive ablation studies and additional analyses of mixing intensity, source similarity, model backbones, and domain transfer.

Critical weaknesses
  • Limited practical relevance. The task assumes that the exact number of sources (K) is known and all words are perfectly observed without additional metadata. However, real-world text often comes with source-related metadata, while speech data provides acoustic cues for speaker identification. It remains unclear how frequently such lexical-only unmixing is necessary in practice and how the proposed method would benefit real-world applications.

  • Limited methodological novelty. The core components of EPOR are closely related to existing approaches. History-conditioned autoregressive assignment resembles Neural Clustering Processes (Pakman et al., 2020), while autoregressive structured prediction with constrained decoding has been explored by He and Choi (2023). Canonical labeling also builds on restricted-growth encoding (Milne, 1982). Although the paper acknowledges several of these connections, it remains unclear what methodological advances EPOR introduces beyond adapting and integrating existing techniques. A clearer discussion of these distinctions and more direct comparisons would strengthen the contribution.

  • Limited understanding of generalization and failure cases. While the paper includes cross-backbone and domain-transfer experiments, the results reveal inconsistent performance across domains. For example, EPOR substantially underperforms RGS-CRF on AG News (Table 23). Further analysis of such failure cases and the factors affecting generalization would strengthen the empirical conclusions.


References

Pakman, A., Wang, Y., & Paninski, L. (2020). Neural Clustering Processes. International Conference on Machine Learning (ICML).

He, H., & Choi, J. D. (2023). Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing. Transactions of the Association for Computational Linguistics (TACL).

Milne, S. C. (1982). Restricted Growth Functions and Incidence Relations of the Lattice of Partitions of an n-Set. Advances in Mathematics.

Questions for authors
  • Zero-shot reasoning and prompting. The frontier LLM baselines are evaluated with thinking disabled and direct source generation. Have the authors considered enabling thinking, prompting these models to predict ownership labels instead, or using a two-pass approach to refine initial assignments? Such comparisons would help clarify the benefits of the proposed output representation and task-specific fine-tuning.

  • Mixing intensity analysis. Could the authors extend Figure 3 to include representative baselines? This would help identify the conditions under which EPOR provides the greatest advantage, rather than simply characterizing the difficulty of the task.

  • Dependence on known K. How much does EPOR rely on knowing the exact number of sources? For example, how would performance change if the model were trained and evaluated with only a possible range of source counts (e.g., (K \in {2,3,4}))? Evaluating generalization to unseen source counts would also be informative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.