acceptodds
Under review as a conference paper at ICLR 2027

Leveraging Hard Negative Samples: Weakness-driven Policy Optimization for Generative Information Extraction

Abstract

While Large Language Models (LLMs) show remarkable capabilities in Generative Information Extraction, their performance remains constrained by the uncontrollable nature of pre-defined label generation. Existing research, centred on prompt engineering or supervised fine-tuning, offers limited gains due to the neglect of hard negative samples. However, leveraging these samples introduces challenges, including deficient error-type awareness, inadequate training focus, and weak dynamic adaptability. To address these issues, we propose a novel framework titled Weakness-driven Policy Optimization (WPO). It first collects error samples that reveal LLM weaknesses, followed by pre-defined atomic error types. Then, WPO uses an LLM to mine composite error types to gain an error distribution. Subsequently, the framework constructs a training set via error-centric data synthesis and error-type-based data augmentation. Finally, WPO employs a dynamic weight allocation mechanism that strategically assigns weights to original training, synthetic, and augmented samples, fine-tuning the LLM using Group Relative Policy Optimization (GRPO). Experimental results on the Qwen3 series LLMs show that WPO achieves a performance gain of up to 1.12% in F1 and 5.31% in precision on Qwen3-8B in Relation Extraction compared to the state-of-the-art (SoTA) baseline. Furthermore, extensive analysis validates the cross-task generalization and cross-scenario robustness of our method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Policy Optimization, Information Extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_siKVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is about trying finding hard negative samples to improve relation extraction from text. This is a well trodden path, but the idea in this paper is the inclusion of error type awareness rather than just binary error (and other elements, which, in my opinion, are not as important). The paper shows improvement on the baseline Qwen model on strict-evaluation criterion over a handful of datasets.

Critical strengths
  • This paper discusses an interesting problem. AI can be improved at the margins by finding the most difficult cases and learning on them. This paper finds those difficult cases, notes why they are wrong and then learns on them.

  • The paper has good results and shows improvement on Qwen on several datasets.

  • This paper is well written given the ICLR constraints.

Critical weaknesses
  • The paper does not provide any technically interesting breakthrough. If we provide additional training to the model, then the model should do better on the task. This is hardly a new revelation. So the work has limited technical novelty, its contribution is mostly engineering.

  • The data augmentation and error type distribution is possibly interesting (and could provide an interesting philosophical portion to an otherwise empirical paper), but there are no details on what atomic and composite errors are. I would love to explore an error ontology and think about what it means to have an error, but this is missing from the paper. In general, I am unhappy with strict-precision baselines because a "pretty much right" answer is often marked wrong; and strict-precision metrics encourage this pathological evaluation.

The last point is my main concern. I dont think its useful to try to hyper optimize to meaninglessly strict metrics when the model is probably doing pretty well already.

Questions for authors
  1. Atomic and Composite Error types are interesting. But I would like to see them explored much more. What is the set of atomic errors that you are constructing? Where do they come from?

  2. Hard negative sample does not need to be a definition. Remove it.

  3. How does the error collection compare with regular active learning strategies? Showing labelers the uncertain or erroneous labels is a very common strategy.

  4. It's curious that GRPO provides more of a boost than the error type distribution. I did not expect that. Why do you think that is?

Reviewer_z686OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pipeline for improving LLMs on generative information extraction using the model's own errors. In the proposed method, a model is fine-tuned on half of the training set and then evaluation on the other half. Then, the GPT-5.1 model labels the resulting errors that the model made with several pre-defined atomic types and LLM-mined composite types. Guided by the error distribution, another LLM synthesizes additional training data, including the instance-level and scenario-level samples, and the augmented samples that pair an injected wrong answer and its error type with the gold answer. In the next step, the model is trained via GRPO with a fixed weight for each data source. The experimental results report the modest F1 gains over SFT and other baselines, along with the component ablations, and weight-sensitivity studies.

Critical strengths
  • The gains in the results are consistent and practically relevant. For example, WPO improves the strict-match F1 over SFT and preference-optimization baselines across multiple backbones and public datasets. On one of the dataset, its gain over full-data SFT is larger than the gain from doubling the model size. So the results in general support the paper's efficiency motivation.
  • The experimental design includes the right controls. Specifically, the paper compares against a randomly synthetized data control, alternative error-type definitions, and an open-sourced synthesis model, and runs a weight-sensitivity analysis. Replacing the GPT-4o model with an open model also preserves most of the utility gain. These controls make the paper's claims clear.
Critical weaknesses
  1. The core claim that weakness-driven construction is the motivation behind the gains, is not established by the paper's own evidence. The extra training data is written by a strong LLM. Given the paper's ablation study, it shows that randomly synthesized data already recovers most of the benefit, and the error-distribution guided construction adds only a small margin. There are components that are supposed to make the method weakness-driven are also unsupported. For example, the mined composite error types are not listed clearly. The claimed adaptation to each model's error profile is never tested. And the reward weighting reduces to a fixed per-sample gradient scale, which would cancel under the usual std-normalized GRPO advantage.
  2. The baselines and evaluation protocol do not show a clean attribution of the performance gains. Some potential baselines are missing. The most direct ways to exploit error samples, like oversampling or unweighting them or running RL only on these samples, are not compared. The plain GRPO is listed as baseline but appears only in one ablations, so on other datasets, the effect of the data cannot be separated from the switch to on-policy RL with an F1 reward. Other prior methods (RISE and SwS) that the paper mentions are not compared. And the preference-optimization baselines lack a description of how pairs are constructed and the tuning details are not reported.

Other minor comments

  • The figure 2 has no legend or axis labels
  • The notation for the dataset subsets is inconsistent, and the term "hard negative" is used unconventionally
Questions for authors
  1. In the implementation, are GRPO advantages normalized by the group standard deviation? If so, how do the reward weights affect training?
  2. How exactly is the random-synthesis control generated? Is it generated by generator, prompt, or others?
  3. How does the proposed method compare with plain GRPO and a matched-budget random-synthesis baseline on all datasets, under both SFT and GRPO?
  4. And how about the baseline that oversampled or unweights the collected the error samples, or runs GRPO only on them?
Reviewer_GvhjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Weakness-driven Policy Optimization (WPO) to enhance generative information extraction with large language models. WPO analyzes model error types and their distribution, constructs targeted synthetic and augmented training data, and trains the model using weighted GRPO. Experiments show that WPO achieves higher F1 scores than the compared baselines on relation extraction, event extraction, and aspect-based sentiment analysis. In relation extraction, it notably increases precision, although recall decreases.

Critical strengths
  1. Training targets model weaknesses. The method combines error-type analysis, targeted data construction, and weighted GRPO to address common model errors. Ablation studies also support the contribution of each component.

  1. Experiments cover multiple tasks and models. The method achieves F1 gains on relation extraction, event extraction, and aspect-based sentiment analysis.
Critical weaknesses
  1. Limited dynamic adaptability. Although the paper emphasizes dynamic data construction and weight allocation, the error distribution is estimated only once before training. Model weaknesses are not reassessed as the model is updated.

  1. Limited performance gains and a precision–recall trade-off. On relation extraction, Qwen3-8B achieves only a 1.12 percentage point increase in F1 over KTO. While precision increases, recall drops by 2.39 percentage points, indicating that the model makes fewer incorrect predictions but also misses more correct relations.
Questions for authors
  1. Dynamic adaptability: The paper emphasizes dynamic data construction and weight allocation, but the error distribution is estimated only once before training. Could the authors clarify what “dynamic” means here? As model weaknesses may change during training, could a fixed error distribution make later training less targeted?

  1. Weight settings: The weights for original, instance-level synthetic, and scenario-level synthetic samples are set to 1.0, 0.5, and 0.1, respectively. How were these values determined? Is there any theoretical justification or hyperparameter sensitivity analysis supporting these choices? How are error-type rankings mapped to weights between 0.1 and 0.9 for augmented samples?

  1. Computational cost: How does WPO compare with the baselines in overall computational cost and runtime? Beyond SFT and GRPO training, WPO requires external LLM calls for error-type analysis, data synthesis, and augmentation. Could the authors report the cost of the full pipeline and discuss the trade-off between this additional overhead and the performance gains?

  1. Recall degradation: WPO increases precision in relation extraction but reduces recall, and the error analysis shows a substantial increase in missing triplets. Could the authors explain this behavior? Given that the reward function considers both F1 and format correctness, why does the model become more conservative after training? Have the authors tried adjusting the rewards or data weights to reduce omissions and achieve a better balance between precision and recall?
Reviewer_uU43OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes a framework to finetune LLM for generative information extraction. It first uses an agent to analyze error type, and then construct post-training dataset by synthesizing error samples or augmented existing data. After that, its use GRPO to finetune the LLM, with weights highlighting error patterns that frequently occur.

Critical strengths

The writing is generally clear, and error-driven data synthesis is an important topic.

Critical weaknesses

Limited novelty. Error-driven data synthesis has been widely explored, as acknowledged in the related work. However, the paper does not clearly explain how its formulation and use of “atomic error types” differ from existing approaches. This appears to be a central motivation for the proposed framework and deserves a more explicit discussion. The weighting scheme is also relatively simple: synthetic samples receive fixed weights whose choices are not sufficiently justified, while augmented samples receive weights based on error-type frequency rankings.

Insufficient benchmarking. Comparisons with state-of-the-art models and relevant methods are limited. The framework uses GPT-5.1 for error analysis and data construction, but the GPT baseline is limited to GPT-4o. More importantly, closely related methods explicitly discussed in the paper, including RISE and SwS, are absent from the experimental comparisons.

Questions for authors

Please clarify the definition of “hard negative samples.” Definition 6 describes their intended purpose but does not provide an operational definition. Specifically, what makes a negative sample “hard”?

Please explain how to interpret Table 4 and how it supports the claim that hard negative samples contribute the most to performance. What do these results establish about the additional benefits of error-type attribution and weighting?

There appears to be a typo in lines 403–405: “resulting in a drop of 1.39% on Qwen3-4B, 1.26% on Qwen3-4B, respectively.” Qwen3-4B is mentioned twice; please verify the intended model names.

The anonymous repository link appears to have expired. Please restore access.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.