- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper is about trying finding hard negative samples to improve relation extraction from text. This is a well trodden path, but the idea in this paper is the inclusion of error type awareness rather than just binary error (and other elements, which, in my opinion, are not as important). The paper shows improvement on the baseline Qwen model on strict-evaluation criterion over a handful of datasets.
Critical strengths
-
This paper discusses an interesting problem. AI can be improved at the margins by finding the most difficult cases and learning on them. This paper finds those difficult cases, notes why they are wrong and then learns on them.
-
The paper has good results and shows improvement on Qwen on several datasets.
-
This paper is well written given the ICLR constraints.
Critical weaknesses
-
The paper does not provide any technically interesting breakthrough. If we provide additional training to the model, then the model should do better on the task. This is hardly a new revelation. So the work has limited technical novelty, its contribution is mostly engineering.
-
The data augmentation and error type distribution is possibly interesting (and could provide an interesting philosophical portion to an otherwise empirical paper), but there are no details on what atomic and composite errors are. I would love to explore an error ontology and think about what it means to have an error, but this is missing from the paper. In general, I am unhappy with strict-precision baselines because a "pretty much right" answer is often marked wrong; and strict-precision metrics encourage this pathological evaluation.
The last point is my main concern. I dont think its useful to try to hyper optimize to meaninglessly strict metrics when the model is probably doing pretty well already.
Questions for authors
-
Atomic and Composite Error types are interesting. But I would like to see them explored much more. What is the set of atomic errors that you are constructing? Where do they come from?
-
Hard negative sample does not need to be a definition. Remove it.
-
How does the error collection compare with regular active learning strategies? Showing labelers the uncertain or erroneous labels is a very common strategy.
-
It's curious that GRPO provides more of a boost than the error type distribution. I did not expect that. Why do you think that is?