acceptodds
Under review as a conference paper at ICLR 2027

FINDING THE PARALLELS: A PATH TO EVALUATING SUMMARIZATION FROM LOW-RESOURCE LANGUAGES

Abstract

Across benchmarks, large language models have demonstrated impressive summarization capabilities for high-resource languages. In contrast, summarization benchmarks are largely unavailable for low-resource languages. This scarcity poses a challenge for both evaluating LLM summarization capabilities and measuring the effectiveness of LLM-as-a-Judge approaches in these languages. However, previous work in machine translation has contributed parallel datasets that pair sentences in low-resource languages with higher-resource translations. We introduce PARAWEAVE, a method for automatically transforming these existing sentence-level parallel datasets into coherent document-level benchmarks while preserving their original cross-lingual alignment. We used PARAWEAVE to construct a summarization benchmark that spans ten language varieties from three families (Afro-Asiatic, Indo-Iranian, and Nilo-Saharan). We use the resulting benchmark to evaluate summarization faithfulness and LLM-as-a-Judge performance for summaries generated from documents in these languages. Pairing low-resource language documents with high-resource translations enables studying the effect of source language on both summarization quality and LLM-as-a-judge performance. These comparisons reveal gaps in model performance and demonstrate that even current frontier models struggle with faithful low-resource language summarization and evaluation in certain languages. For example, GPT-5.5's summaries of Nilo-Saharan documents are significantly less faithful than their English document counterparts. Native-speaker annotations validate the model judgments in our construction pipeline and judge faithfulness metrics. The proposed pipeline offers a general approach for repurposing existing sentence-level parallel resources for evaluation of document-level tasks.

Primary area:
datasets and benchmarks
Keywords:
Low-resource languages, parallel corpora, cross-lingual summarization, faithfulness evaluation, LLM-as-a-judge, clustering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ozmuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies summarization and faithfulness evaluation for low-resource languages. The authors introduce ParaWeave, a framework that transforms existing sentence-level parallel translation datasets into synthetic document-level benchmarks through semantic clustering, LLM-based composition, and coherence filtering. Using this approach, the authors construct a benchmark containing 3,671 documents across ten language varieties. They evaluate the faithfulness of summaries generated from low-resource and English documents, as well as the cross-lingual reliability of LLM-as-a-Judge methods. The results reveal substantial performance gaps in summarization and evaluation across different languages, particularly for certain very low-resource languages.

Critical strengths
  1. Relevant research problem. The paper addresses the shortage of document-level summarization benchmarks for low-resource languages. Repurposing existing parallel translation resources is a practical approach that could reduce the cost of multilingual benchmark construction.

  2. Broad empirical evaluation. The benchmark covers ten language varieties and evaluates multiple general-purpose and language-specialized models. The experiments examine summarization faithfulness, cross-lingual judge agreement, and selected inference settings. The paper also includes pipeline analyses and limited native-speaker validation, providing useful evidence about current multilingual model limitations.

Critical weaknesses
  1. The representativeness and validity of the synthetic documents remain insufficiently established. The benchmark is constructed by reorganizing originally independent translation sentences into documents, but topical similarity does not necessarily guarantee natural or semantically coherent articles. The human evaluation shows that the synthetic documents score noticeably below natural news in both coherence and human-likeness, while some Arabic documents are assembled primarily from conversational or phrasebook sentences rather than realistic long-form texts. Moreover, native-speaker validation covers only four Arabic dialect groups with seven participants, while the remaining languages lack comparable human validation. It therefore remains unclear how well the constructed benchmark represents real-world multilingual document summarization.

  2. The main evaluation conclusions rely heavily on model-generated judgments. The paper treats LLM judgments against the parallel English documents as the correct reference for evaluating both summarization faithfulness and cross-lingual judge performance. However, English-reference judgments are not equivalent to human ground truth. The reported agreement between the judge and human annotators is only 70%, and the human validation is limited. More importantly, the claims selected for human validation were drawn from cases where the judge agreed across English and source-language references, meaning that this validation does not directly assess the cross-lingual disagreements central to the paper. Consequently, the current evaluation does not sufficiently rule out English-reference bias, translation differences, or judge errors as contributors to the reported performance gaps. Potential training-data overlap may also complicate some model comparisons.

Questions for authors

How representative are the synthetic documents of naturally occurring low-resource language articles, and how does the limited native-speaker validation support conclusions across all ten language varieties?

Given that English-reference LLM judgments are treated as ground truth, how can the authors establish that the observed cross-lingual performance gaps reflect genuine language-understanding limitations rather than biases or errors in the English-reference evaluation?

5 comments
AuthorsOpenReview ↗

Response to W1/Q1 (1/2)

Thank you for your thoughtful feedback, which has helped us improve the paper. We address your concerns below and will update the final draft accordingly.

W1/Q1

W1a: The representativeness and validity of the synthetic documents remain insufficiently established. … It therefore remains unclear how well the constructed benchmark represents real-world multilingual document summarization.

Q1: How representative are the synthetic documents of naturally occurring low-resource language articles, and how does the limited native-speaker validation support conclusions across all ten language varieties?

We similarly viewed this as a potential weakness in the submitted draft. As a result, we ran additional native-speaker studies for the Nilo-Saharan language settings after the submission deadline.

The human validation now covers 13 native speakers across seven language varieties.

Since submission, we annotated all three Nilo-Saharan languages (Lugbara, Ateso, and Acholi) with native speakers, and annotators rated the articles we constructed 3.5/4 for coherence on average, well above the 3.07 of the Arabic articles. As a baseline, we also asked Nilo-Saharan annotators to rate the coherence of length-matched BBC News articles. The average coherence score was 3.1! These results provide evidence that the resulting synthetic documents provide a reasonable evaluation setting.

Notably, the source data of the Arabic articles are built from MADAR, whose sentences are conversational phrasebook lines. The resulting Arabic articles are dialogues, rather than news articles, which likely contributes to their lower scores (3.07 coherence) versus full natural news (3.93 coherence).

PARAWEAVE can be applied to other parallel corpora. The higher coherence ratings for articles built from written prose suggest that future users can build more natural benchmarks by choosing source material that matches their intended genre.

W1b: The benchmark is constructed by reorganizing originally independent translation sentences into documents, but topical similarity does not necessarily guarantee natural or semantically coherent articles.

On topical similarity not guaranteeing coherent articles: we agree, and this is why clustering is not what makes the articles. Instead, the composer does this as a next step. The topical clustering is explicitly into very large clusters (about 150 sentence pairs on average) to limit the input context length for the composer LLM.

Evaluating whether the pipeline can reconstruct documents from existing benchmarks.

As part of our existing evaluations, we tested whether the proposed pipeline could recover existing real-world multilingual documents from the FLORES benchmark using their parallel sentences. 92 to 94% of the articles returned by our pipeline came from a single original FLORES article, and 93 to 97% of those kept the original sentence order. These returned articles are the Persian, Dari, and Tajik articles in our benchmark, so this test also directly validates them. Since submission, we ran the same test on two other document-level parallel datasets, ALT (Lao) and AfriDocMT (Zulu): 96% and 88% of the resulting articles come from a single original document, and 86% and 89% of those keep the original sentence order.

Therefore, the pipeline is able to reconstruct the original documents given a pool of thousands of parallel sentences with the document boundaries removed. This provides further evidence of the pipeline’s effectiveness.

AuthorsOpenReview ↗

Response to W2/Q2 (2/2)

W2/Q2

W2/Q2: Given that English-reference LLM judgments are treated as ground truth, how can the authors establish that the observed cross-lingual performance gaps reflect genuine language-understanding limitations rather than biases or errors in the English-reference evaluation?

We ran a new native-speaker study that includes claims where the English and source-reference judgments disagree, with 6 native speakers, 2 each for Lugbara, Acholi, and Ateso. Afterwards, each annotator saw two judgments (one from the source language judge and one from the English language judge) and provided observations about how verdicts differed given both the English article and the source article to collect additional information on sources of disagreement (e.g. translation accuracy).

1. Blinded native-speaker agreement with the English-reference and source-reference verdicts:

Below, we include results from the Nilo-Saharan native speaker evaluations. These results show significantly more human annotator agreement with the English-reference judge compared to the Nilo-Saharan-reference judge.

We additionally include results for binary agreement, where we collapse the four labels into [partially supported and supported] into one label and [contradicted and unsupported] into not supported.

LanguageJudgeBinary Supported vs. Not SupportedFour-way label
AtesoEnglish-reference80%54%
AtesoSource-reference54%35%
AcholiEnglish-reference85%63%
AcholiSource-reference72%35%
LugbaraEnglish-reference67%32%
LugbaraSource-reference46%23%

2. Judge-human agreement

Notably, the English-reference judge agrees with native speakers at comparable rates to inter-annotator agreement:

LanguageEnglish LLM judge vs. annotatorInter-annotator agreement
Arabic dialects0.400.50
Ateso0.30-0.12
Acholi0.540.60
Lugbara0.280.33

All scores are reported using Cohen’s κ agreement on the binary scale used above. In Ateso, inter-annotator κ is negative because both annotators rated nearly every claim as supported (they agree on 18 of 23 claims), which leaves no room for agreement above chance.

3. English-reference LLM judge and human disagreement

We have added an analysis of the disagreements that occurred between the English-reference LLM judge and human annotators.

When the English-reference judge marked a claim as supported and the source-reference judge did not, annotators also marked it as supported 94% of the time (88% to 100% per annotator). This shows that annotators rarely disagree with the English-reference judge when it calls a claim supported. Instead, they disagree mostly when it rejects a claim, indicating that the judge often imposes stricter requirements than the human annotators.

For example, annotators accepted "They reminisced about old times during the day they spent together" because the article says "They shared their experiences," while the judge rejected this as unfaithful with the explanation ‘Source mentions sharing experiences but not reminiscing about old times specifically.’

4. Disagreements attributed to translation differences

In addition to finding the reasons behind judgement disagreements on top of the strictness standards, the second annotation study also reveals reasons behind disagreements, as annotators had the option to tell us why they thought the verdicts differed given both the English article and the source article (Nilo-Saharan: Lugbara, Acholi, and Ateso).

They attributed only a small share of the disagreements to translation (3%).

5. Investigating training-data contamination

We checked for overlap in both the data and the models. Only 1 of 2,000 sampled source pool sentences appears in any of four large open pretraining corpora (OLMo-mix, DCLM, Dolma, and RedPajama). When given the start of a source sentence, none of the open models we evaluate complete more than 1% of them verbatim. Naming the dataset in the prompt does not change this (Golchin & Surdeanu, 2024). Only Sunflower’s 32B training data was stated to include SALT. Its translation scores were similar across SALT splits, with 55.0 chrF on dev/test and 54.1 on training sentences.


In summary, the additional experiments we ran provide evidence that the hypothesized sources of error do not significantly affect the results presented in the paper.

Thank you for encouraging us to examine these possible confounds. Running these experiments has improved the rigor of our work. Please let us know if there is additional information we can provide to answer any remaining questions.

Thank you again for your thoughtful review. To follow up, we believe that our additional experiments have addressed your concerns. To summarize:

  1. We included additional native-speaker experiments for Lugbara, Acholi, and Ateso, bringing our human validation to 13 native speakers across seven language varieties. We also ran additional experiments testing whether our pipeline can reconstruct existing documents from pools of parallel sentences. Verdict: The Nilo-Saharan documents scored 3.5/4 for coherence, and the reconstruction experiments provided further evidence that our pipeline recovers coherent document structure.

  2. To address your concern about the validity of the LLM-as-a-Judge evaluation, we evaluated claims where the English- and source-reference judges disagreed and investigated the reasons behind these disagreements. Verdict: Native speakers agreed significantly more with the English-reference judge across all three Nilo-Saharan languages. Differences in strictness contributed to English-judge and human disagreements, while annotators attributed only 3% of cross-reference disagreements to translation differences. We also provided additional checks addressing potential training-data contamination.

We believe that the added results strengthen the soundness of our work, and we would be grateful if you reconsidered your initial assessment.

We are happy to address any remaining concerns.

Reviewer_u5gwOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a simple framework, PARAWEAVE, to automatically construct summarization benchmark for three low-resource language families, including Afro-Asiatic, Indo-Iranian, and Nilo-Saharan. To be specific, the authors transform pair sentences in low-resource languages with higher-resource tranlations from machine translation area through clustering, LLM-based composition and coherence filtering based on the high-resource sentences and then get the summarization result on the low-resource languages. Seven native speakers are invited to validate the quality of the synthetic articles.

It reports 3,671 language-specific parallel article entries across ten language varieties. The summarization evaluation generates English summaries for either low-resource-language documents or their English counterparts, then uses Claude Sonnet 4.6 to assess the faithfulness of extracted atomic claims against the English documents. Small-sized domain knowledged models are better than general models with larger size and it requires extra efforts to teach general-domain models work on the low-resource languages. A separate experiment fixes the summaries and their claim decompositions while changing the reference-document language to investigate cross-lingual differences in judge behavior.

Critical strengths
  1. A practically useful resource-construction approach with meaningful human controls.
  2. A paired design that separately diagnoses generation-side and judging-side language differences
  3. Human evaluation is included.
Critical weaknesses

1)I do not see why we need English summarization for low-resource languages, as illustrated in Figure 1. Such a setting seems unnecessary. 2) Only seven participants were involved in conducting the human evaluation, which is insufficient. 3) The conclusion for this work seems straight-forward for people in this domain since the corpus in low-resource language is rare.

Questions for authors

See weakness section.

2 comments
AuthorsOpenReview ↗

Response to W1-W3

Thank you for your review and transparency about review confidence. We address your comments below.

W1

W1: I do not see why we need English summarization for low-resource languages, as illustrated in Figure 1. Such a setting seems unnecessary.

Summarization from low-resource languages into high-resource languages like English is very useful in many settings. The task is relevant to aid organizations responding to crises in different regions, doctors providing support to non-English-speaking patients, and intelligence analysts writing reports based on foreign language documents.

Notably, cross-lingual summarization is an established field (e.g., CrossSum (Bhattacharjee et al., 2023) and WikiLingua (Ladhak et al., 2020)). Recently, an AI chatbot provided incorrect intelligence and “almost started a war” [1]. While not low-resource summarization, this event illustrates the potential stakes for cross-lingual summarization failures in a world where humans increasingly rely on these systems.

The other reviewers also mention the importance of this area in their review: Reviewer ozmu lists “Relevant research problem” as a strength. Reviewer JY95 notes that the paper “addresses a useful evaluation gap” and describes the reference-language comparison as “a useful finding for researchers evaluating low-resource language systems.”

W2

W2: Only seven participants were involved in conducting the human evaluation, which is insufficient.

Since submission, we recruited six additional native speakers for the Nilo-Saharan languages, two each for Lugbara, Ateso, and Acholi, through Upwork. The annotations followed the same tasks as the dialectal Arabic languages. The human evaluation now covers 13 native speakers across seven language varieties: four Arabic dialects, Lugbara, Ateso, and Acholi. Additionally, using two annotators per task is a common standard used in previous work [2, 3, 4].

W3

W3: The conclusion for this work seems straight-forward for people in this domain since the corpus in low-resource language is rare.

It is often expected that models perform worse on lower-resource languages. However, without benchmarks it is very difficult to accurately gauge model capabilities and performance gaps. Notably, the evaluations in our paper reveal uneven capabilities across models and languages. GPT-5.5 excels at summarizing from several low-resource languages, including Tajik and Dari, outperforming specialized models! However, our results reveal large gaps for languages like Lugbara. These results can motivate work in labs to improve language capabilities and inform decision-making about where/how models are deployed and used.

References

[1] Katie Bo Lillis and Zachary Cohen. 2026. Exclusive: US military had close call after using AI for false intelligence report, sources say. CNN, September 18, 2026. CNN article.

[2] Mingqi Gao, Wenqing Wang, Xiaojun Wan, and Yuemei Xu. 2023. Evaluating Factuality in Cross-lingual Summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 12415–12431. DOI: 10.18653/v1/2023.findings-acl.786.

[3] Huajian Zhang and Laura Perez-Beltrachini. 2024. Leveraging Entailment Judgements in Cross-Lingual Summarisation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 14481–14497. DOI: 10.18653/v1/2024.findings-acl.862.

[4] Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4693–4703. DOI: 10.18653/v1/2021.findings-acl.413.

To follow up, we have provided context above that covers:

  1. How English summarization for low-resource languages is in fact very important (e.g., for aid organizations, medical professionals, and intelligence analysts). We also included information about the field of cross-lingual summarization.

  2. Results with a larger set of human annotators, spanning additional languages. We also provided context about how our methods align with standard native-speaker evaluation practices in this field.

  3. We also highlighted how our results go beyond showing that low-resource summarization is difficult and reveal surprising differences in model capabilities. For example, we found that GPT-5.5 is highly capable at summarizing several low-resource languages, even outperforming specialized models, but performs quite poorly on languages like Lugbara.

Given these additional clarifications, we would be grateful if you reconsidered your initial assessment.

Reviewer_JY95OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ParaWeave, a pipeline that selects and orders sentences from existing parallel corpora to build aligned documents for low-resource language evaluation. The resulting benchmark covers ten language varieties. Using English summaries and parallel reference documents, the authors study summarization faithfulness and how LLM judgments change with the reference language. They find substantial gaps in both summarization faithfulness and judge consistency for some low-resource languages.

Critical strengths
  1. The paper addresses a useful evaluation gap by reusing existing parallel sentence resources. The pipeline preserves sentence alignment, and the human coherence study, original-document recovery test, and sentence preservation checks provide useful support for the constructed data.

  2. The parallel design enables controlled comparisons across languages. In particular, keeping summary claims fixed while changing the reference language reveals large changes in judge scores and, in some cases, model rankings. This is a useful finding for researchers evaluating low-resource language systems.

Critical weaknesses
  1. The human validation does not cover the disagreements central to RQ2. Appendix A states that the checked claims were selected from cases where the English- and Arabic-reference judgments already agreed. The results therefore establish cross-language inconsistency more directly than they establish which judgments are wrong. The additional automatic checkers provide support, but independent review of disagreement cases would strengthen the conclusion that the source-reference judgments are less reliable.

  2. Summaries with no extracted claims are excluded from the averages, and the exclusion counts differ substantially across models. Appendix C reports 120 such cases for gpt-oss-20B on Lugbara, compared with at most seven for several other systems. Since rerunning the extractor recovered many initially missing claims, these exclusions may reflect extraction failures as well as summary quality. The reported means may therefore compare models on different document subsets, which complicates their interpretation.

Questions for authors
  1. Could native speakers review a small sample of claims where the English- and source-reference judgments disagree, especially for Lugbara or Ateso? If this is not feasible, please distinguish observed inconsistency from errors established against independent labels.

  2. What causes the remaining summaries with no extracted claims: empty or uninformative outputs, or extraction failures? Could you report the fraction of summaries included for each model and check whether the main comparisons hold on the same set of documents?

3 comments
AuthorsOpenReview ↗

Response to W1/Q1 (1/2)

Thank you so much for your thoughtful review. We address your concerns below.

W1/Q1

W1: The human validation does not cover the disagreements central to RQ2. … The additional automatic checkers provide support, but independent review of disagreement cases would strengthen the conclusion that the source-reference judgments are less reliable.

Q1: Could native speakers review a small sample of claims where the English- and source-reference judgments disagree, especially for Lugbara or Ateso?

Thank you for identifying a potential gap in our human validation effort.

Anticipating this critique, we ran additional native-speaker studies for the Nilo-Saharan language settings after the submission deadline. As part of these evaluations, we asked annotators to validate both judge agreements and disagreements.

Annotators first rated each summary claim against the source article alone. These claims were sampled at random and include claims where judges disagreed. Afterwards, each annotator saw two judgments (one from the source language judge and one from the English language judge) and provided observations about how verdicts differed given both the English article and the source article. We ran this task with 6 human annotators in the Nilo-Saharan languages, 2 for each of the Lugbara, Acholi, and Ateso languages.

Agreement between LLM-as-Judges and native speakers:

Below, we include results from the Nilo-Saharan native speaker evaluations. These results show significantly more human annotator agreement with the English-reference judge compared to the Nilo-Saharan-reference judge.

We additionally include results for binary agreement, where we collapse the four labels into [partially supported and supported] into one label and [contradicted and unsupported] into not supported.

LanguageLLM JudgeBinary Supported vs. Not SupportedFour-way label
AtesoEnglish-reference80%54%
AtesoSource-reference54%35%
AcholiEnglish-reference85%63%
AcholiSource-reference72%35%
LugbaraEnglish-reference67%32%
LugbaraSource-reference46%23%

We also examined specific cases where the judges disagreed. When the English-reference judge marked a claim as supported and the source-reference judge did not, native speakers also marked it as supported 94% of the time. Annotators made these judgments using only the source-language article, before seeing either judge’s verdict.

Notably, the English-reference judge also agrees with native speakers at comparable rates to inter-annotator agreement:

LanguageEnglish LLM judge vs. annotatorInter-annotator agreement
Arabic dialects0.400.50
Ateso0.30-0.12
Acholi0.540.60
Lugbara0.280.33

All scores are reported using Cohen’s κ agreement on the binary scale used above. In Ateso, inter-annotator κ is negative because both annotators rated nearly every claim as supported (they agree on 18 of 23 claims).

AuthorsOpenReview ↗

Response to W2/Q2 (2/2)

W2/Q2

W2: Summaries with no extracted claims are excluded from the averages, and the exclusion counts differ substantially across models. … The reported means may therefore compare models on different document subsets, which complicates their interpretation. *

Q2: What causes the remaining summaries with no extracted claims: empty or uninformative outputs, or extraction failures?*

This is an interesting question, and answering it strengthens our paper. We include an analysis below, which we will incorporate into our final draft.

Summaries with no claims: There are two types of no-claim output. The first is extraction failures, where the summary states facts but the claim decomposition returned nothing. These are rare: an LLM (Claude) read all 764 summaries with no extracted claims individually and found only 4 clear extraction failures, under 0.1% of the 5,760 summaries on these languages.

The second, and by far the more common, is outputs with no checkable content. Notably, this tends to happen with certain models, which produce very generic descriptions of the text, incomplete outputs, or refusals instead of a summary.

With English input, 99.7% of model-generated summaries have at least one claim. However, with Nilo-Saharan source-language input, the percentage varies: 100% for GPT-5.5 and 97-99% for Aya. It is lower for Qwen3-30B-A3B (76-92%), Qwen3-8B (47-76%), and GPT-oss-20B (40-62%). This shows a second gap between low-resource and high-resource summarization that complements the existing results in the paper: Summaries from low-resource documents are more likely to contain zero claims!

We compared three setups to account for no claim summaries: (1) no-claim summaries excluded from the faithfulness mean, as in the paper; (2) no-claim summaries scored as 0; and (3) only the documents where every system produced claims. The ranking by faithfulness when reading the source-language article is the same under (1) and (2), except that GPT-oss-20B and Qwen3-8B swap places.

Setup (3) favors the general-purpose models. The articles in which general models produce no claims are the ones they understand the least. Hence, judging without these articles favors general-purpose models. In these articles every general model scores higher, while the Sunflower specialists, which read all articles about equally well, stay the same or drop. Even so, the leader stays the same in every language except Ateso, where GPT-5.5 becomes the most performant model. See results below.

ModelLugbara (1) Excl.Lugbara (2) ZeroLugbara (3) SharedAcholi (1) Excl.Acholi (2) ZeroAcholi (3) SharedAteso (1) Excl.Ateso (2) ZeroAteso (3) Shared
GPT-5.50.470.470.530.740.740.760.620.620.68
Sunflower-32B0.610.610.600.670.670.620.650.650.62
Sunflower-9B0.500.500.500.650.650.650.660.660.63
gpt-oss-20B0.400.200.420.260.160.290.310.120.40
Qwen3-8B0.290.220.400.330.200.380.290.140.33
Qwen3-30B-A3B0.210.180.250.170.150.250.160.120.25
Fanar-1-9B0.100.100.160.090.090.170.100.100.22
Aya-23-8B0.090.080.160.060.060.110.040.030.11

Summaries with at least one claim (of 240), source-language input:

ModelLugbaraAcholiAteso
GPT-5.5240240240
Sunflower-32B239239239
Sunflower-9B240240240
gpt-oss-20B12014995
Qwen3-8B183146112
Qwen3-30B-A3B210222182
Fanar-1-9B238240239
Aya-23-8B233238232
Shared (setup 3)899146

Thank you for your constructive feedback. Addressing your comments has significantly strengthened our paper. Let us know if we can provide any additional information to improve our scores.

Thank you again for your thoughtful review. To follow up, we believe that our additional results have addressed your concerns. To summarize:

  1. We included additional native speaker evaluations for Nilo-Saharan languages that evaluated the disagreement between the source language judges and English language judges. Verdict: Native speakers agreed with the English judge significantly more than the source language judge!

  2. To address your concern about zero claim summaries, we provided the requested scores with zero claim summaries included (along with all other configurations). Verdict: no summary model rankings changed by the inclusion of zero-claim summaries in the score, with the exception of GPT-oss-20B and Qwen3-8B in Lugbara and Ateso.

The added results provide clarifications that significantly strengthen our paper. We are happy to address any additional concerns. Otherwise, we would be grateful if you considered adjusting your scores.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification

Could you clarify why you do not think low-resource work summarization is necessary? Existing work establishes that the task is useful, and our results show that it is not solved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.