- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies summarization and faithfulness evaluation for low-resource languages. The authors introduce ParaWeave, a framework that transforms existing sentence-level parallel translation datasets into synthetic document-level benchmarks through semantic clustering, LLM-based composition, and coherence filtering. Using this approach, the authors construct a benchmark containing 3,671 documents across ten language varieties. They evaluate the faithfulness of summaries generated from low-resource and English documents, as well as the cross-lingual reliability of LLM-as-a-Judge methods. The results reveal substantial performance gaps in summarization and evaluation across different languages, particularly for certain very low-resource languages.
Critical strengths
-
Relevant research problem. The paper addresses the shortage of document-level summarization benchmarks for low-resource languages. Repurposing existing parallel translation resources is a practical approach that could reduce the cost of multilingual benchmark construction.
-
Broad empirical evaluation. The benchmark covers ten language varieties and evaluates multiple general-purpose and language-specialized models. The experiments examine summarization faithfulness, cross-lingual judge agreement, and selected inference settings. The paper also includes pipeline analyses and limited native-speaker validation, providing useful evidence about current multilingual model limitations.
Critical weaknesses
-
The representativeness and validity of the synthetic documents remain insufficiently established. The benchmark is constructed by reorganizing originally independent translation sentences into documents, but topical similarity does not necessarily guarantee natural or semantically coherent articles. The human evaluation shows that the synthetic documents score noticeably below natural news in both coherence and human-likeness, while some Arabic documents are assembled primarily from conversational or phrasebook sentences rather than realistic long-form texts. Moreover, native-speaker validation covers only four Arabic dialect groups with seven participants, while the remaining languages lack comparable human validation. It therefore remains unclear how well the constructed benchmark represents real-world multilingual document summarization.
-
The main evaluation conclusions rely heavily on model-generated judgments. The paper treats LLM judgments against the parallel English documents as the correct reference for evaluating both summarization faithfulness and cross-lingual judge performance. However, English-reference judgments are not equivalent to human ground truth. The reported agreement between the judge and human annotators is only 70%, and the human validation is limited. More importantly, the claims selected for human validation were drawn from cases where the judge agreed across English and source-language references, meaning that this validation does not directly assess the cross-lingual disagreements central to the paper. Consequently, the current evaluation does not sufficiently rule out English-reference bias, translation differences, or judge errors as contributors to the reported performance gaps. Potential training-data overlap may also complicate some model comparisons.
Questions for authors
How representative are the synthetic documents of naturally occurring low-resource language articles, and how does the limited native-speaker validation support conclusions across all ten language varieties?
Given that English-reference LLM judgments are treated as ground truth, how can the authors establish that the observed cross-lingual performance gaps reflect genuine language-understanding limitations rather than biases or errors in the English-reference evaluation?
5 comments
Response to W2/Q2 (2/2)
W2/Q2
W2/Q2: Given that English-reference LLM judgments are treated as ground truth, how can the authors establish that the observed cross-lingual performance gaps reflect genuine language-understanding limitations rather than biases or errors in the English-reference evaluation?
We ran a new native-speaker study that includes claims where the English and source-reference judgments disagree, with 6 native speakers, 2 each for Lugbara, Acholi, and Ateso. Afterwards, each annotator saw two judgments (one from the source language judge and one from the English language judge) and provided observations about how verdicts differed given both the English article and the source article to collect additional information on sources of disagreement (e.g. translation accuracy).
1. Blinded native-speaker agreement with the English-reference and source-reference verdicts:
Below, we include results from the Nilo-Saharan native speaker evaluations. These results show significantly more human annotator agreement with the English-reference judge compared to the Nilo-Saharan-reference judge.
We additionally include results for binary agreement, where we collapse the four labels into [partially supported and supported] into one label and [contradicted and unsupported] into not supported.
| Language | Judge | Binary Supported vs. Not Supported | Four-way label |
|---|---|---|---|
| Ateso | English-reference | 80% | 54% |
| Ateso | Source-reference | 54% | 35% |
| Acholi | English-reference | 85% | 63% |
| Acholi | Source-reference | 72% | 35% |
| Lugbara | English-reference | 67% | 32% |
| Lugbara | Source-reference | 46% | 23% |
2. Judge-human agreement
Notably, the English-reference judge agrees with native speakers at comparable rates to inter-annotator agreement:
| Language | English LLM judge vs. annotator | Inter-annotator agreement |
|---|---|---|
| Arabic dialects | 0.40 | 0.50 |
| Ateso | 0.30 | -0.12 |
| Acholi | 0.54 | 0.60 |
| Lugbara | 0.28 | 0.33 |
All scores are reported using Cohen’s κ agreement on the binary scale used above. In Ateso, inter-annotator κ is negative because both annotators rated nearly every claim as supported (they agree on 18 of 23 claims), which leaves no room for agreement above chance.
3. English-reference LLM judge and human disagreement
We have added an analysis of the disagreements that occurred between the English-reference LLM judge and human annotators.
When the English-reference judge marked a claim as supported and the source-reference judge did not, annotators also marked it as supported 94% of the time (88% to 100% per annotator). This shows that annotators rarely disagree with the English-reference judge when it calls a claim supported. Instead, they disagree mostly when it rejects a claim, indicating that the judge often imposes stricter requirements than the human annotators.
For example, annotators accepted "They reminisced about old times during the day they spent together" because the article says "They shared their experiences," while the judge rejected this as unfaithful with the explanation ‘Source mentions sharing experiences but not reminiscing about old times specifically.’
4. Disagreements attributed to translation differences
In addition to finding the reasons behind judgement disagreements on top of the strictness standards, the second annotation study also reveals reasons behind disagreements, as annotators had the option to tell us why they thought the verdicts differed given both the English article and the source article (Nilo-Saharan: Lugbara, Acholi, and Ateso).
They attributed only a small share of the disagreements to translation (3%).
5. Investigating training-data contamination
We checked for overlap in both the data and the models. Only 1 of 2,000 sampled source pool sentences appears in any of four large open pretraining corpora (OLMo-mix, DCLM, Dolma, and RedPajama). When given the start of a source sentence, none of the open models we evaluate complete more than 1% of them verbatim. Naming the dataset in the prompt does not change this (Golchin & Surdeanu, 2024). Only Sunflower’s 32B training data was stated to include SALT. Its translation scores were similar across SALT splits, with 55.0 chrF on dev/test and 54.1 on training sentences.
In summary, the additional experiments we ran provide evidence that the hypothesized sources of error do not significantly affect the results presented in the paper.
Thank you for encouraging us to examine these possible confounds. Running these experiments has improved the rigor of our work. Please let us know if there is additional information we can provide to answer any remaining questions.
Follow-up
Thank you again for your thoughtful review. To follow up, we believe that our additional experiments have addressed your concerns. To summarize:
-
We included additional native-speaker experiments for Lugbara, Acholi, and Ateso, bringing our human validation to 13 native speakers across seven language varieties. We also ran additional experiments testing whether our pipeline can reconstruct existing documents from pools of parallel sentences. Verdict: The Nilo-Saharan documents scored 3.5/4 for coherence, and the reconstruction experiments provided further evidence that our pipeline recovers coherent document structure.
-
To address your concern about the validity of the LLM-as-a-Judge evaluation, we evaluated claims where the English- and source-reference judges disagreed and investigated the reasons behind these disagreements. Verdict: Native speakers agreed significantly more with the English-reference judge across all three Nilo-Saharan languages. Differences in strictness contributed to English-judge and human disagreements, while annotators attributed only 3% of cross-reference disagreements to translation differences. We also provided additional checks addressing potential training-data contamination.
We believe that the added results strengthen the soundness of our work, and we would be grateful if you reconsidered your initial assessment.
We are happy to address any remaining concerns.
Response to W1/Q1 (1/2)
Thank you for your thoughtful feedback, which has helped us improve the paper. We address your concerns below and will update the final draft accordingly.
W1/Q1
We similarly viewed this as a potential weakness in the submitted draft. As a result, we ran additional native-speaker studies for the Nilo-Saharan language settings after the submission deadline.
The human validation now covers 13 native speakers across seven language varieties.
Since submission, we annotated all three Nilo-Saharan languages (Lugbara, Ateso, and Acholi) with native speakers, and annotators rated the articles we constructed 3.5/4 for coherence on average, well above the 3.07 of the Arabic articles. As a baseline, we also asked Nilo-Saharan annotators to rate the coherence of length-matched BBC News articles. The average coherence score was 3.1! These results provide evidence that the resulting synthetic documents provide a reasonable evaluation setting.
Notably, the source data of the Arabic articles are built from MADAR, whose sentences are conversational phrasebook lines. The resulting Arabic articles are dialogues, rather than news articles, which likely contributes to their lower scores (3.07 coherence) versus full natural news (3.93 coherence).
PARAWEAVE can be applied to other parallel corpora. The higher coherence ratings for articles built from written prose suggest that future users can build more natural benchmarks by choosing source material that matches their intended genre.
On topical similarity not guaranteeing coherent articles: we agree, and this is why clustering is not what makes the articles. Instead, the composer does this as a next step. The topical clustering is explicitly into very large clusters (about 150 sentence pairs on average) to limit the input context length for the composer LLM.
Evaluating whether the pipeline can reconstruct documents from existing benchmarks.
As part of our existing evaluations, we tested whether the proposed pipeline could recover existing real-world multilingual documents from the FLORES benchmark using their parallel sentences. 92 to 94% of the articles returned by our pipeline came from a single original FLORES article, and 93 to 97% of those kept the original sentence order. These returned articles are the Persian, Dari, and Tajik articles in our benchmark, so this test also directly validates them. Since submission, we ran the same test on two other document-level parallel datasets, ALT (Lao) and AfriDocMT (Zulu): 96% and 88% of the resulting articles come from a single original document, and 86% and 89% of those keep the original sentence order.
Therefore, the pipeline is able to reconstruct the original documents given a pool of thousands of parallel sentences with the document boundaries removed. This provides further evidence of the pipeline’s effectiveness.