- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes CoverKG, which trains a knowledge graph extractor using rewards for coverage and compactness. Coverage measures the similarity between source sentences and extracted descriptions, while compactness is computed from the coverage lost when each element is removed. The two rewards are combined to train Qwen3-8B with GRPO without QA labels. Experiments with GraphRAG, LightRAG, and HippoRAG2 show improvements in 12 of 15 QA configurations and substantial reductions in index size.
Critical strengths
-
The reward can be computed directly from source text and candidate extractions, without QA annotations or running a downstream QA pipeline during each training rollout.
-
The experiments cover three different retrieval backends and several benchmarks. The method substantially reduces index size while improving QA performance in many settings, suggesting practical value for graph construction.
Critical weaknesses
-
The compactness reward can increase after adding a redundant element, contradicting lines 244–248. It exposes an incentive inconsistent with the claimed redundancy penalty, although it does not establish that the trained model exploited this behavior. Consider three equally weighted sentences and five extracted elements. Their similarities are:
Sentence 1: (0.80, 0, 0, 0, 0)
Sentence 2: (0, 0.80, 0, 0.79, 0)
Sentence 3: (0, 0, 0.80, 0, 0.79)With tau = 0.63, coverage is 0.80. Removing each element in turn gives marginal contributions of approximately (0.27, 0.0033, 0.0033, 0, 0). Now add a sixth element with the same description embedding as the first. Coverage remains unchanged, but the first and sixth elements can replace each other, so removing either causes no loss of coverage. The marginal contributions become (0, 0.0033, 0.0033, 0, 0, 0).
Although this adds no coverage and makes the extraction more redundant, the remaining positive contributions are now equal. Their participation ratio therefore increases enough to outweigh the penalty from the larger element count: compactness rises from 0.21 to 0.33, and the reward at beta = 1 rises from 0.33 to 0.47.
-
The experiments do not sufficiently support the benefit of the proposed reward. The paper compares Base, SFT, and CoverKG, but omits directly relevant methods such as AutoGraph-R1. Moreover, the ablation studies are limited to GraphRAG, while LightRAG and HippoRAG2 backends are not evaluated.
Questions for authors
-
How do you reconcile the counterexample with lines 244–248? Does the same behavior occur with embeddings from the actual model, and how does it affect training?
-
How does CoverKG compare with AutoGraph-R1 and other relevant methods? Do the ablation results also hold for LightRAG and HippoRAG2?