- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper formulates Word-Level Text Unmixing, a task of recovering the original sources from interleaved word sequences without source attribution metadata. In this task, words from K different sources are mixed together, and the model aims to separate them while preserving the original words and their order. When LLMs directly generate the separated source texts, the original lexical evidence can be modified or omitted. To address this issue, the authors propose Evidence-Preserving Ownership Routing (EPoR), a learning-based framework that predicts source labels instead of generating the source texts. EPoR fine-tunes an LLM to autoregressively predict the source label for each word. During inference, the authors apply constrained decoding to ensure valid source assignments without empty sources and reconstruct the source texts based on the predicted labels. The proposed EPoR is evaluated on UnMixBench, a benchmark introduced by the authors for Word-Level Text Unmixing, and shows generally better performance than fine-tuned generative and structured baselines, as well as zero-shot LLM baselines.
Critical strengths
- The authors identify and formulate a novel problem, Word-Level Text Unmixing.
- The proposed architectural design choices are well supported by thorough ablation studies.
Critical weaknesses
- Word-Level Text Unmixing seems impractical.
- The authors explain the scenarios that require word-level text unmixing as 1) multi-speaker dialogue without source attribution, 2) document reading-flow unmixing, and 3) dialogue unmixing in agentic systems. However, it is common to use external information such as audio for scenario 1) and document layout information (e.g., tables or figures) for scenario 2). For scenario 3), it is hard to imagine such a scenario where metadata is omitted. Therefore, the practical importance of the proposed problem is unclear to me. This concern is also reflected in their benchmark, which is partly constructed using synthetically mixed data.
- The benchmark and baselines are not representative.
- This concern is also related to my first concern. The paper proposes a new benchmark, UNMIXBENCH, but its representativeness or validity in simulating practical scenarios is not fully assessed. I guess this concern might be addressed if the authors resolve my first concern.
- The novelty of the method should be validated in more detail.
- Liu et al. [1] propose a transition-based dialogue disentanglement model that maintains and updates session states based on previous assignments. A relevant baseline would adapt this model to word-level prediction, using the same LLM backbone while retaining its session-state LSTM and build/update mechanism. The adapted model should be trained on the same data and evaluated on UnMixBench under matched input and decoding conditions, including access to the full mixed stream and the known source count. This comparison would help clarify the benefit of EPoR over explicit recurrent source-state modeling. The authors should discuss this work and clarify the novelty of their method compared to it.
- [1] Liu et al., End-to-End Transition-Based Online Dialogue Disentanglement, IJCAI, 2020.
Questions for authors
Please see the weaknesses above.