- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GoLongRL studies long-context RL training beyond retrieval-based QA. The authors build a 23K-sample dataset covering nine task types, with rewards tailored to each task. They also propose TMN-Reweight, which adjusts advantage scales across tasks and reweights samples based on difficulty. Training with GRPO improves performance on several long-context benchmarks at both 4B and 30B scales. TMN-Reweight brings a smaller additional gain at 4B.
Critical strengths
Research Motivation: Long-context training is not just about retrieving information from longer documents. Models also need to combine, organize, and reason over what they find. I think the paper makes a reasonable case for training these capabilities together, with rewards suited to different tasks.
Presentation and Results: I found the paper easy to read. The data construction and training method are explained well, and the experiments are fairly comprehensive. The gains are quite noticeable at both 4B and 30B. It is also good to see results on reasoning and memory tasks, beyond the usual long-context benchmarks.
Critical weaknesses
Data Construction: The dataset is the main contribution of this paper. Most samples come from existing datasets or synthetic QA generation. The nine task categories largely follow LongBench Pro. I find the novelty rather limited. Table 1 shows that T1 through T4 account for 92.5% of the samples. No ablation isolates the contribution of the other five tasks. A simpler data mixture might achieve similar results. The current experiments do not provide enough evidence for the value of the proposed capability coverage.
Algorithmic Contribution: Similar difficulty weighting has been studied in HA-DW, F-GRPO, and DIVA-GRPO. The exponential weighting based on advantage signs is particularly close to HA-DW and DIVA-GRPO. The paper does not clearly distinguish its approach from these methods. The experimental evidence is also weak. TMN-Reweight improves the 4B average by only 0.8 points. No results are reported at 30B. The effects of normalization and reweighting are not evaluated separately. I find the additional algorithmic contribution quite limited.
Questions for authors
Data Ablation: How much do the different parts of the dataset contribute? I would like to see a comparison between T1–T4 alone and the full nine-task mixture, using the same data size and training budget. It would also help to separate the effects of curated and synthetic data.
Related Methods and Ablation: How does TMN-Reweight compare with HA-DW, F-GRPO, and DIVA-GRPO under the same training setup? Also, what happens when task-level normalization and difficulty-aware reweighting are applied separately?
Generalization: Does TMN-Reweight work equally well at 30B? The improvement at 4B is fairly small, and MRCR actually drops by 2.0 points. Are these results consistent across different training seeds?
Capability Evaluation: The largest gains in Table 2 come from MRCR and CorpusQA. What about ranking, ordering, structured extraction, and summarization? These tasks receive relatively little training data, and I would like to know whether they actually improve.
8 comments
Q2 / W2. Related methods and component ablations
(1) Differences from HA-DW, F-GRPO, and DIVA-GRPO
Thank you for pointing out the connection to prior work on difficulty weighting. We describe how each method works, what adapting it to our heterogeneous long-context tasks would require, and how GoLongRL addresses those requirements.
HA-DW [1] maintains an ability anchor using historical rewards across batches and applies exponential weights based on the prompt's difficulty relative to that anchor and the sign of its advantages. In our setting, rewards such as EM, F1, and NDCG have different scoring semantics. If these rewards are pooled into a single anchor, both the task mixture and the reward distribution of each task affect its value, making it difficult to isolate prompt difficulty. GoLongRL smooths each prompt's baseline with the mean for the same task in the current batch, estimates relative difficulty from the fraction of rewards above this threshold, and uses TMN to calibrate advantage scales across tasks. This requires no difficulty history across batches.
F-GRPO [2] computes a focal coefficient from the success rate under binary rewards and applies the same scaling factor to all rollouts for a prompt. This reduces the contribution of prompts with high success rates. In our setting, which combines binary and continuous rewards, metrics such as F1, IoU, and NDCG have no common boundary between correct and incorrect responses. A nonzero reward does not necessarily indicate a correct response, so adapting F-GRPO would require redefining success. Uniform scaling also leaves the relative weights of positive and negative advantages unchanged, so it cannot amplify positive advantages while attenuating negative ones for harder prompts. GoLongRL uses the fraction of rewards exceeding a threshold defined within the task, then applies reciprocal weights to positive and negative advantages. This allows it to adjust the emphasis on reinforcing better responses and suppressing poorer ones separately.
DIVA-GRPO [3] constructs question variants of different difficulty levels that preserve the answer. It computes local and global advantages jointly and combines sign-dependent exponential weighting with reward-range-based rescaling. Applying the full pipeline to our long-document, structured extraction, ranking, and summarization tasks would require constructing additional variants and verifying that their evidence agrees with the reference answers and that their output structures can be scored. This adds generation and verification costs. GoLongRL computes task scales and relative difficulty directly from the original long-context rollouts, avoiding the cost of constructing and verifying additional question variants.
For each task, GoLongRL uses the square root of the mean within-prompt reward variance as a shared normalization scale and a smoothed reference from the same task to estimate relative difficulty. It then applies asymmetric weights to positive and negative advantages. The sign-dependent exponential weighting is closely related to that in HA-DW and DIVA-GRPO. Our method differs in how it defines the statistics for each task and combines these steps on the original long-context rollouts.
(2) Comparison under the same training setup
We have yet to complete matched retraining of HA-DW, F-GRPO, and DIVA-GRPO on the nine-task GoLongRL dataset. A controlled comparison requires matching the backbone, sampling budget, number of training steps, reward handling, and evaluation setup, along with the task adaptations described above. For DIVA-GRPO, the comparison also needs to account for the cost of constructing and verifying variants.
(3) Component ablations for normalization and reweighting
We have started additional TMN-only and Reweight-only training runs. Both runs remain incomplete, and final ablation results are pending. These runs are designed to assess the effects of task-level normalization and difficulty-adaptive reweighting separately and compare each variant with the full method. Figure 4 reports the overall gain of the full TMN-Reweight method over GRPO.
References
[1] Yang et al. (2026). Your Group-Relative Advantage Is Biased, §3.
[2] Plyusov et al. (2026). F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare, §4.
[3] Gao et al. (2026). DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage, §3.
Q3 / W2. Generalization across model scales and training seeds
(1) Scope of validation at 30B
Thank you for asking about applicability to larger models. TMN-Reweight uses scalar rewards from sampled responses and the associated task labels. Its computation is independent of model size and architectural details specific to the 4B model. It reuses existing rollouts and requires no additional reward network or generation, so it can be applied to larger models such as the 30B model.
(2) Training seeds and parameter sensitivity
We do not yet have results from full training runs across multiple random seeds. A single 4B long-context RL run requires 160 training steps on 128 H800 GPUs. At approximately 40 minutes per step, this amounts to about 107 hours, or 13,700 GPU-hours. This cost limits the feasibility of completing multiple runs during the rebuttal period.
Table 7 reports sensitivity to the smoothing coefficient with the 4B backbone, training data, and evaluation setup fixed. Standard GRPO averages 62.2 across six benchmarks. At and , the averages are 62.9 and 63.0, respectively, corresponding to gains of 0.7 and 0.8 points over GRPO. The gain thus occurs at more than one tested value of . At and , the averages are 61.3 and 61.5, respectively. These results support combining the prompt and task means in the tested setting.
(3) Average gain and MRCR decrease at 4B
Thank you for pointing out the performance drops in Figure 4. With the 4B backbone, training data, and evaluation setup fixed, TMN-Reweight raises the average across six benchmarks from 62.2 to 63.0. CorpusQA, LongBench V2, and Frames improve by 4.5, 1.6, and 0.8 points, respectively. LBV1-QA is unchanged, while MRCR and DocMath decrease by 2.0 and 0.2 points, respectively.
Let be the within-group reward standard deviation for prompt , the shared scale for its task, and the relative difficulty indicator. From the method definitions, we obtain
\underbrace{\frac{s_q+\delta}{S_t+\delta}}_{\text{scale adjustment}} \underbrace{\exp\\!\left[\left(\frac12-\widehat p_q\right) \operatorname{sgn}\\!\left(A^{\mathrm{GRPO}}_{qi}\right)\right]} _{\text{adjustment by difficulty and advantage sign}} A^{\mathrm{GRPO}}_{qi}.$$ **First, TMN changes the relative update strength across prompts within a task.** GRPO standardizes each prompt separately. TMN uses a shared denominator within each task, preserving the relative scale $s_q/S_t$. Ignoring the numerical stabilizer, the weight relative to GRPO is larger for prompts with greater within-group reward variation and smaller for prompts with less variation. **Second, Reweight changes the balance between reinforcing better responses and suppressing poorer responses within a prompt.** Because TMN centers rewards at the group mean, the sum of positive advantages equals the sum of the magnitudes of negative advantages in any group with nonzero advantages. After reweighting, their ratio becomes $$\frac{\sum_{\widetilde A_{qi} > 0}\widetilde A_{qi}} {\sum_{\widetilde A_{qi} < 0}|\widetilde A_{qi}|} =w_q^2=\exp(1-2\widehat p_q).$$ When $\widehat p_q < 0.5$, the update places more weight on reinforcing the few better trajectories. When $\widehat p_q > 0.5$, it places more weight on correcting poorer trajectories. This ratio compares the total weights on positive and negative advantages. The actual gradient also depends on the direction of each trajectory's policy gradient. These mechanisms help explain the performance trade-off. Let $c_{qi}$ be the combined weight in the first equation. For the same batch of sampled responses and with the advantage statistics held fixed, the policy gradient can be written as $$g_{\mathrm{TMN\text{-}Reweight}} =\sum_{q,i}c_{qi}\\,g^{\mathrm{GRPO}}_{qi}.$$ Changing these weights can change the direction of the combined update. Let $J_b(\theta)$ denote expected performance on benchmark $b$. For a small gradient ascent step $\theta'=\theta+\eta g_{\mathrm{TMN\text{-}Reweight}}$, the first-order change in expected performance is $$\Delta J_b\approx\eta\\,\left\langle \nabla_\theta J_b,\\, g_{\mathrm{TMN\text{-}Reweight}} \right\rangle.$$ The performance gradients of different benchmarks can align differently with the same update direction, so the average score can rise while scores on individual benchmarks fall.
Q1 / W1. Data composition and source contributions
Thank you for asking about the novelty of the dataset and the contributions of its data sources. GoLongRL turns capability-oriented task design into long-context RLVR training data, reward interfaces, and a data construction pipeline. We screen examples by task and reference answer, convert numerical answers into formats that
math_verifycan parse, and convert ranking annotations into ordered lists that NDCG can score. We ground synthetic question–answer pairs in real books, papers, and conversations, verify the pairs, and filter them in multiple stages using models with different capabilities. Finally, we apply 13-gram query-overlap filtering. The resulting dataset contains 22,965 examples spanning nine tasks. We release the dataset, mixture proportions, code for data construction, training, and evaluation, and model weights.T1–T4 account for 92.5% of the training examples. T5–T9 contain 1,717 examples, or 7.48% of the full dataset. Through their output formats and reward objectives, these five tasks extend direct supervision beyond retrieval-based question answering to structured extraction, graded relevance ranking, order reconstruction, and summarization.
We provide results for three versions evaluated during dataset construction to supplement the dataset validation experiments in the submission. All three runs used the same Qwen3-4B-Thinking-2507 backbone, standard GRPO, and evaluation setup. For each run, we randomly sampled 8,000 training examples from the corresponding data pool. The counts in parentheses give the full pool sizes.
V1 was already close to the final version on single-document understanding and basic retrieval, while cross-document reasoning and long-range context memory were relatively weak. After we broadened task coverage in V2, CorpusQA rose by 7.6 points and the average across six benchmarks rose by 2.2 points. MRCR rose by only 1.9 points. Based on these results, we added examples targeting multi-hop reasoning and context memory in V3. Relative to V2, MRCR and CorpusQA rose by 26.8 and 7.3 points, respectively, and the average across six benchmarks rose by 7.2 points. With the training sample count fixed at 8,000 across all three versions, these results support refining the data composition and adding supervision targeted at the weaknesses identified in evaluation.