Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling

Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) is a prominent variant of Group Relative Policy Optimization (GRPO). DAPO introduces several improvements over GRPO. Among these, Dynamic Sampling contributes the most to DAPO's accuracy improvement, which has also been adopted in training frontier models. To improve accuracy, Dynamic Sampling enhances training stability by eliminating zero policy gradients caused by zero advantages. Specifically, it avoids zero gradients by filtering out prompts where sampled responses are either correct or incorrect. However, our theoretical analysis shows that Dynamic Sampling decrease training efficiency as it cannot effectively utilize hard-to-sample correct responses on hard prompts. Formally, it asymmetrically amplifies the advantages of distinct responses to the same prompts. On hard prompts, incorrect responses undergo much greater amplification than correct ones. This leads the model to avoid generating the observed incorrect responses rather than capitalizing on the hard-to-sample correct ones on hard prompts, resulting in low training efficiency. To improve training efficiency, we propose Direct Advantage Amplification (DAA), which amplifies the advantages of hard-to-sample correct responses retained by Dynamic Sampling on hard prompts. This ensures that, these hard-to-sample responses can be effectively capitalized on when Dynamic Sampling is used, implying higher training efficiency. By integrating DAA into DAPO, we obtain Difficulty-aware Advantage Amplification Policy Optimization (DA3PO), which is implemented with fewer than 30 lines of code from DAPO. Experiments show that DA3PO significantly outperforms GRPO and other classical GRPO variants on various model families and tasks.