acceptodds
Under review as a conference paper at ICLR 2027

GoLongRL: Capability-Oriented Long-Context Reinforcement Learning with Heterogeneous Multitask Alignment

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise for improving how large language models handle long contexts. However, existing long-context RL methods focus narrowly on retrieval-path-based data, resulting in limited task coverage and homogeneous reward formulations that fail to reflect the breadth of practical long-context requirements. We present a capability-oriented post-training framework that addresses this gap through two contributions. First, we construct an openly released dataset of 23K RLVR samples spanning 9 task types, each paired with its natural evaluation metric as the reward function. Under vanilla GRPO, training on this dataset alone outperforms comparable closed-source data recipes. Second, we propose TMN-Reweight, an advantage estimation method that combines task-level mean normalization for cross-task reward scale alignment with difficulty-adaptive weighting for more reliable gradient signals. Experiments on two model scales demonstrate that capability-oriented data is the primary driver of improvement, while TMN-Reweight yields additional gains on aggregation-intensive benchmarks. General reasoning, agentic memory, and dialogue memory capabilities are preserved or improved.

Primary area:
reinforcement learning
Keywords:
Long-context, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jKeqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GoLongRL studies long-context RL training beyond retrieval-based QA. The authors build a 23K-sample dataset covering nine task types, with rewards tailored to each task. They also propose TMN-Reweight, which adjusts advantage scales across tasks and reweights samples based on difficulty. Training with GRPO improves performance on several long-context benchmarks at both 4B and 30B scales. TMN-Reweight brings a smaller additional gain at 4B.

Critical strengths

Research Motivation: Long-context training is not just about retrieving information from longer documents. Models also need to combine, organize, and reason over what they find. I think the paper makes a reasonable case for training these capabilities together, with rewards suited to different tasks.

Presentation and Results: I found the paper easy to read. The data construction and training method are explained well, and the experiments are fairly comprehensive. The gains are quite noticeable at both 4B and 30B. It is also good to see results on reasoning and memory tasks, beyond the usual long-context benchmarks.

Critical weaknesses

Data Construction: The dataset is the main contribution of this paper. Most samples come from existing datasets or synthetic QA generation. The nine task categories largely follow LongBench Pro. I find the novelty rather limited. Table 1 shows that T1 through T4 account for 92.5% of the samples. No ablation isolates the contribution of the other five tasks. A simpler data mixture might achieve similar results. The current experiments do not provide enough evidence for the value of the proposed capability coverage.

Algorithmic Contribution: Similar difficulty weighting has been studied in HA-DW, F-GRPO, and DIVA-GRPO. The exponential weighting based on advantage signs is particularly close to HA-DW and DIVA-GRPO. The paper does not clearly distinguish its approach from these methods. The experimental evidence is also weak. TMN-Reweight improves the 4B average by only 0.8 points. No results are reported at 30B. The effects of normalization and reweighting are not evaluated separately. I find the additional algorithmic contribution quite limited.

Questions for authors

Data Ablation: How much do the different parts of the dataset contribute? I would like to see a comparison between T1–T4 alone and the full nine-task mixture, using the same data size and training budget. It would also help to separate the effects of curated and synthetic data.

Related Methods and Ablation: How does TMN-Reweight compare with HA-DW, F-GRPO, and DIVA-GRPO under the same training setup? Also, what happens when task-level normalization and difficulty-aware reweighting are applied separately?

Generalization: Does TMN-Reweight work equally well at 30B? The improvement at 4B is fairly small, and MRCR actually drops by 2.0 points. Are these results consistent across different training seeds?

Capability Evaluation: The largest gains in Table 2 come from MRCR and CorpusQA. What about ranking, ordering, structured extraction, and summarization? These tasks receive relatively little training data, and I would like to know whether they actually improve.

8 comments

Q1 / W1. Data composition and source contributions

Thank you for asking about the novelty of the dataset and the contributions of its data sources. GoLongRL turns capability-oriented task design into long-context RLVR training data, reward interfaces, and a data construction pipeline. We screen examples by task and reference answer, convert numerical answers into formats that math_verify can parse, and convert ranking annotations into ordered lists that NDCG can score. We ground synthetic question–answer pairs in real books, papers, and conversations, verify the pairs, and filter them in multiple stages using models with different capabilities. Finally, we apply 13-gram query-overlap filtering. The resulting dataset contains 22,965 examples spanning nine tasks. We release the dataset, mixture proportions, code for data construction, training, and evaluation, and model weights.

T1–T4 account for 92.5% of the training examples. T5–T9 contain 1,717 examples, or 7.48% of the full dataset. Through their output formats and reward objectives, these five tasks extend direct supervision beyond retrieval-based question answering to structured extraction, graded relevance ranking, order reconstruction, and summarization.

We provide results for three versions evaluated during dataset construction to supplement the dataset validation experiments in the submission. All three runs used the same Qwen3-4B-Thinking-2507 backbone, standard GRPO, and evaluation setup. For each run, we randomly sampled 8,000 training examples from the corresponding data pool. The counts in parentheses give the full pool sizes.

V1 was already close to the final version on single-document understanding and basic retrieval, while cross-document reasoning and long-range context memory were relatively weak. After we broadened task coverage in V2, CorpusQA rose by 7.6 points and the average across six benchmarks rose by 2.2 points. MRCR rose by only 1.9 points. Based on these results, we added examples targeting multi-hop reasoning and context memory in V3. Relative to V2, MRCR and CorpusQA rose by 26.8 and 7.3 points, respectively, and the average across six benchmarks rose by 7.2 points. With the training sample count fixed at 8,000 across all three versions, these results support refining the data composition and adding supervision targeted at the weaknesses identified in evaluation.

Version (data pool size)DocMathLBV2FramesMRCRCorpusQALBV1-QAAverage
V1 (9,630)59.842.562.138.850.263.652.8
V2 (17,729)60.043.764.440.757.863.555.0
V3 (22,965)62.545.566.667.565.165.962.2

Q2 / W2. Related methods and component ablations

(1) Differences from HA-DW, F-GRPO, and DIVA-GRPO

Thank you for pointing out the connection to prior work on difficulty weighting. We describe how each method works, what adapting it to our heterogeneous long-context tasks would require, and how GoLongRL addresses those requirements.

HA-DW [1] maintains an ability anchor using historical rewards across batches and applies exponential weights based on the prompt's difficulty relative to that anchor and the sign of its advantages. In our setting, rewards such as EM, F1, and NDCG have different scoring semantics. If these rewards are pooled into a single anchor, both the task mixture and the reward distribution of each task affect its value, making it difficult to isolate prompt difficulty. GoLongRL smooths each prompt's baseline with the mean for the same task in the current batch, estimates relative difficulty from the fraction of rewards above this threshold, and uses TMN to calibrate advantage scales across tasks. This requires no difficulty history across batches.

F-GRPO [2] computes a focal coefficient from the success rate under binary rewards and applies the same scaling factor to all rollouts for a prompt. This reduces the contribution of prompts with high success rates. In our setting, which combines binary and continuous rewards, metrics such as F1, IoU, and NDCG have no common boundary between correct and incorrect responses. A nonzero reward does not necessarily indicate a correct response, so adapting F-GRPO would require redefining success. Uniform scaling also leaves the relative weights of positive and negative advantages unchanged, so it cannot amplify positive advantages while attenuating negative ones for harder prompts. GoLongRL uses the fraction of rewards exceeding a threshold defined within the task, then applies reciprocal weights to positive and negative advantages. This allows it to adjust the emphasis on reinforcing better responses and suppressing poorer ones separately.

DIVA-GRPO [3] constructs question variants of different difficulty levels that preserve the answer. It computes local and global advantages jointly and combines sign-dependent exponential weighting with reward-range-based rescaling. Applying the full pipeline to our long-document, structured extraction, ranking, and summarization tasks would require constructing additional variants and verifying that their evidence agrees with the reference answers and that their output structures can be scored. This adds generation and verification costs. GoLongRL computes task scales and relative difficulty directly from the original long-context rollouts, avoiding the cost of constructing and verifying additional question variants.

For each task, GoLongRL uses the square root of the mean within-prompt reward variance as a shared normalization scale and a smoothed reference from the same task to estimate relative difficulty. It then applies asymmetric weights to positive and negative advantages. The sign-dependent exponential weighting is closely related to that in HA-DW and DIVA-GRPO. Our method differs in how it defines the statistics for each task and combines these steps on the original long-context rollouts.

(2) Comparison under the same training setup

We have yet to complete matched retraining of HA-DW, F-GRPO, and DIVA-GRPO on the nine-task GoLongRL dataset. A controlled comparison requires matching the backbone, sampling budget, number of training steps, reward handling, and evaluation setup, along with the task adaptations described above. For DIVA-GRPO, the comparison also needs to account for the cost of constructing and verifying variants.

(3) Component ablations for normalization and reweighting

We have started additional TMN-only and Reweight-only training runs. Both runs remain incomplete, and final ablation results are pending. These runs are designed to assess the effects of task-level normalization and difficulty-adaptive reweighting separately and compare each variant with the full method. Figure 4 reports the overall gain of the full TMN-Reweight method over GRPO.

References

[1] Yang et al. (2026). Your Group-Relative Advantage Is Biased, §3.

[2] Plyusov et al. (2026). F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare, §4.

[3] Gao et al. (2026). DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage, §3.

Q3 / W2. Generalization across model scales and training seeds

(1) Scope of validation at 30B

Thank you for asking about applicability to larger models. TMN-Reweight uses scalar rewards from sampled responses and the associated task labels. Its computation is independent of model size and architectural details specific to the 4B model. It reuses existing rollouts and requires no additional reward network or generation, so it can be applied to larger models such as the 30B model.

(2) Training seeds and parameter sensitivity

We do not yet have results from full training runs across multiple random seeds. A single 4B long-context RL run requires 160 training steps on 128 H800 GPUs. At approximately 40 minutes per step, this amounts to about 107 hours, or 13,700 GPU-hours. This cost limits the feasibility of completing multiple runs during the rebuttal period.

Table 7 reports sensitivity to the smoothing coefficient α\alpha with the 4B backbone, training data, and evaluation setup fixed. Standard GRPO averages 62.2 across six benchmarks. At α=0.5\alpha=0.5 and α=0.8\alpha=0.8, the averages are 62.9 and 63.0, respectively, corresponding to gains of 0.7 and 0.8 points over GRPO. The gain thus occurs at more than one tested value of α\alpha. At α=0\alpha=0 and α=1\alpha=1, the averages are 61.3 and 61.5, respectively. These results support combining the prompt and task means in the tested setting.

(3) Average gain and MRCR decrease at 4B

Thank you for pointing out the performance drops in Figure 4. With the 4B backbone, training data, and evaluation setup fixed, TMN-Reweight raises the average across six benchmarks from 62.2 to 63.0. CorpusQA, LongBench V2, and Frames improve by 4.5, 1.6, and 0.8 points, respectively. LBV1-QA is unchanged, while MRCR and DocMath decrease by 2.0 and 0.2 points, respectively.

Let sqs_q be the within-group reward standard deviation for prompt qq, StS_t the shared scale for its task, and p^q\widehat p_q the relative difficulty indicator. From the method definitions, we obtain

\underbrace{\frac{s_q+\delta}{S_t+\delta}}_{\text{scale adjustment}} \underbrace{\exp\\!\left[\left(\frac12-\widehat p_q\right) \operatorname{sgn}\\!\left(A^{\mathrm{GRPO}}_{qi}\right)\right]} _{\text{adjustment by difficulty and advantage sign}} A^{\mathrm{GRPO}}_{qi}.$$ **First, TMN changes the relative update strength across prompts within a task.** GRPO standardizes each prompt separately. TMN uses a shared denominator within each task, preserving the relative scale $s_q/S_t$. Ignoring the numerical stabilizer, the weight relative to GRPO is larger for prompts with greater within-group reward variation and smaller for prompts with less variation. **Second, Reweight changes the balance between reinforcing better responses and suppressing poorer responses within a prompt.** Because TMN centers rewards at the group mean, the sum of positive advantages equals the sum of the magnitudes of negative advantages in any group with nonzero advantages. After reweighting, their ratio becomes $$\frac{\sum_{\widetilde A_{qi} > 0}\widetilde A_{qi}} {\sum_{\widetilde A_{qi} < 0}|\widetilde A_{qi}|} =w_q^2=\exp(1-2\widehat p_q).$$ When $\widehat p_q < 0.5$, the update places more weight on reinforcing the few better trajectories. When $\widehat p_q > 0.5$, it places more weight on correcting poorer trajectories. This ratio compares the total weights on positive and negative advantages. The actual gradient also depends on the direction of each trajectory's policy gradient. These mechanisms help explain the performance trade-off. Let $c_{qi}$ be the combined weight in the first equation. For the same batch of sampled responses and with the advantage statistics held fixed, the policy gradient can be written as $$g_{\mathrm{TMN\text{-}Reweight}} =\sum_{q,i}c_{qi}\\,g^{\mathrm{GRPO}}_{qi}.$$ Changing these weights can change the direction of the combined update. Let $J_b(\theta)$ denote expected performance on benchmark $b$. For a small gradient ascent step $\theta'=\theta+\eta g_{\mathrm{TMN\text{-}Reweight}}$, the first-order change in expected performance is $$\Delta J_b\approx\eta\\,\left\langle \nabla_\theta J_b,\\, g_{\mathrm{TMN\text{-}Reweight}} \right\rangle.$$ The performance gradients of different benchmarks can align differently with the same update direction, so the average score can rise while scores on individual benchmarks fall.
Reviewer_puGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-context RL problems by creating a 23K-sample dataset that contains aggregation, structured extraction, ranking, ordering, and summarization, and using the TMN-Reweight on the GRPO to solve the heterogeneous task-specific rewards introduced by the new dataset. The whole framework, named GoLongRL, first curates the dataset using 9 tasks with different capabilities and different scores. The GRPO part uses task mean normalization to normalize the different-scale rewards under different tasks and difficulty-adaptive reweighting to adjust the score for different difficulty prompts to give a higher weight for hard problems with positive and lower for negative. Empirical results show improved performance on six benchmarks.

Critical strengths
  • The curated dataset provides a good training set for training the long-context RL. The ability learned from this dataset can be generalized to out-of-distribution benchmarks, showing the generalizability of the learned LLMs.

  • The method TMN-reweight proposed in this paper provides a solution for solving heterogeneous task rewards and provides additional weight on the different difficulty prompts. These methods show the effectiveness for the specific curated dataset.

Critical weaknesses
  • It is still unclear why this paper wants to propose task-level mean normalization instead of directly normalizing the score when creating the dataset. This paper lacks a motivation to use this specific method. In addition to this, this paper does not discuss how to get the task-level mean. We know that we need to calculate the mean during training. However, it is unclear in this paper how to get the task mean over the task prompts that are larger than the training batch.

  • The empirical results show that the improvement is marginal compared with Qwenlong. And the ablation study is rough. It shows that the TMN-reweight sometimes even hurts the performance in Figure 4. It is also unclear what the effectiveness of each component in TMN-reweight is. For the difficulty adaptive reweighting part, it seems to be heuristic. This paper lacks the detailed motivation for using these tricks for this specific problem.

Questions for authors
  • Can you provide an explanation for why you propose task-level mean normalization instead of directly normalizing the score when creating the dataset?

  • Can you explain how to get the task-level mean during training?

  • Can you explain why you use the difficulty adaptive reweighting tricks and how this trick is unique to this problem?

  • Can you provide more ablation results on the effectiveness of each component in the TMN reweighting?

7 comments

Q1 / W1. Task-level normalization and dataset-level reward scaling

Thank you for raising this question. During dataset construction, we specify the reward functions, reference answers, and output formats. The rewards observed during training depend on the responses generated by the current policy. Even if all rewards are mapped to [0,1][0,1], reward variances can differ across tasks and change as the policy is updated. Rescaling rewards in advance leaves the level of GRPO normalization unchanged. Under a fixed positive affine transformation r′=ar+b,a>0r'=ar+b,\\ a > 0, the factor aa cancels between the numerator and denominator of the GRPO advantage for groups with nonzero variance when the numerical stabilizer is omitted. A common reward range therefore cannot substitute for switching from prompt-level to task-level normalization.

TMN in Eq. (2) retains each prompt's group mean as the centering baseline. It replaces the denominator with the root mean square of the within-group reward standard deviations across prompts of the same task in the current batch. Writing the group rewards as r=μq+sqzr=\mu_q+s_qz, where zz has zero mean and unit sample standard deviation, and again omitting the numerical stabilizer, we obtain

\widehat A^{\mathrm{TMN}}=\frac{s_q}{S_t}z.$$ The shared denominator preserves relative advantage magnitudes across prompts within the same task. Appendix A provides the analytical motivation for this choice of scale. Figure 3 shows that the cross-task coefficient of variation of per-task mean absolute advantages decreases from 0.34 with GRPO to 0.18 with TMN, consistent with the goal of reducing differences in advantage scale across tasks. ### Q2 / W1. Computing task statistics during training Thank you for asking us to clarify which prompts are used to compute the task mean. Here, a task refers to a task type, and task statistics are computed within the current training batch. A batch can contain just a subset of the training prompts for a task. Let $\mathcal U_t$ be the set of prompts for task $t$ in the current batch, $G$ the number of responses per prompt, and $r_{qi}$ the reward for response $i$ to prompt $q$. We first compute the group mean and sample variance for each prompt, then aggregate the variances by task. $$\begin{aligned} \mu_q &= \frac{1}{G}\sum_i r_{qi},\quad s_q^2 = \frac{1}{G-1}\sum_i(r_{qi}-\mu_q)^2,\\\\ S_t &= \sqrt{\frac{1}{|\mathcal U_t|}\sum_{q\in\mathcal U_t}s_q^2},\quad \widehat A_{qi} = \frac{r_{qi}-\mu_q}{S_t+\delta}. \end{aligned}$$ The two statistics serve different purposes. TMN uses $S_t$ as the shared normalization scale within a task, while advantages remain centered at each prompt's own mean $\mu_q$. Thus, the centering baseline in task-level mean normalization remains the prompt mean. The difficulty module separately computes the task mean $m_t$ to construct a smoothed threshold. $$m_t=\frac{1}{|\mathcal U_t|}\sum_{q\in\mathcal U_t}\mu_q,\quad \tau_q=\alpha\mu_q+(1-\alpha)m_t,\quad \widehat p_q=\frac{1}{G}\sum_i\mathbf 1[r_{qi} > \tau_q].$$ $$w_q=\exp(0.5-\widehat p_q),\qquad \widetilde A_{qi}= \begin{cases} \widehat A_{qi}w_q, & \widehat A_{qi} > 0,\\\\ \widehat A_{qi}/w_q, & \widehat A_{qi}\leq 0. \end{cases}$$ Here, $\delta$ is a numerical stabilizer. The implementation first groups sampled rewards by prompt UID, then aggregates prompt variances and means by `reward_mode`. All statistics come from responses already sampled for the current update. In the submitted setup, we sample $G=16$ responses per prompt. If a task has only one prompt in the batch, $S_t$ reduces to that prompt's within-group standard deviation. Tasks absent from the current batch do not contribute to that update. When all rewards in a prompt group are equal, its centered advantages are zero. This aggregation requires no pass over the task's full dataset, additional generation, or reward network.

Q3 / W2. Motivation for difficulty-adaptive reweighting

Thank you for asking us to clarify the motivation for difficulty-adaptive reweighting. TMN calibrates advantage scales across heterogeneous tasks, but prompts within each task still call for different learning signals. For harder prompts, we aim to strengthen learning from better trajectories. For easier prompts, we aim to reduce repeated reinforcement of already good responses and put more weight on correcting poorer ones. We therefore adjust the weights of positive and negative advantages according to each prompt's relative difficulty.

EM, F1, and NDCG have different scoring semantics, making it difficult to use raw mean rewards or the fraction of nonzero rewards as a common measure of prompt difficulty. We smooth the prompt-level mean reward with the mean reward for the same task in the current batch. We use p^q\widehat p_q, the fraction of responses whose rewards exceed this threshold, to measure relative difficulty within the task. We then reweight the TMN advantages asymmetrically:

A~qi=A^qiexp⁡![(0.5−p^q)sgn⁡(A^qi)].\widetilde A_{qi}=\widehat A_{qi}\exp\\!\left[(0.5-\widehat p_q)\operatorname{sgn}(\widehat A_{qi})\right].

When p^q<0.5\widehat p_q < 0.5, positive advantages are amplified and negative advantages are attenuated. When p^q>0.5\widehat p_q > 0.5, the adjustment is reversed. Compared with uniformly scaling all trajectories for a prompt, this design directly changes the balance between reinforcing better responses and suppressing poorer ones.

The design combines cross-task scale calibration with within-task difficulty adjustment for heterogeneous rewards. All required statistics come from existing rollouts. The method requires no additional generation or reward network, making it suitable for long-context training where generation is costly.

Q4 / W2. Component ablations

Thank you for pointing out the need for component ablations. We have started two additional training runs, TMN-only and Reweight-only, for comparison with standard GRPO and the full TMN-Reweight method. Both runs remain incomplete, and final ablation results are pending.

W2. Comparison with QwenLong-L1.5 and performance trade-offs

Thank you for raising the magnitude of the gains. In Table 2, GoLongRL with standard GRPO averages 62.2 at 4B and 69.8 at 30B. These scores are 6.1 and 2.6 points above the reported QwenLong-L1.5 GRPO scores of 56.1 and 67.2, respectively. The QwenLong-L1.5 values come from the original report because the corresponding GRPO checkpoints and training data are unavailable.

Figure 4 measures the additional effect of TMN-Reweight with the 4B backbone, training data, and evaluation setup held fixed. The average across six benchmarks rises from 62.2 to 63.0 (+0.8 points). CorpusQA, LongBench V2, and Frames improve by 4.5, 1.6, and 0.8 points, respectively. LBV1-QA is unchanged, while MRCR and DocMath decrease by 2.0 and 0.2 points, respectively.

Let sqs_q be the within-group reward standard deviation for prompt qq, StS_t the shared scale for its task, and p^q\widehat p_q the relative difficulty indicator. From the method definitions, we obtain

\underbrace{\frac{s_q+\delta}{S_t+\delta}}_{\text{scale adjustment}} \underbrace{\exp\\!\left[\left(\frac12-\widehat p_q\right) \operatorname{sgn}\\!\left(A^{\mathrm{GRPO}}_{qi}\right)\right]} _{\text{adjustment by difficulty and advantage sign}} A^{\mathrm{GRPO}}_{qi}.$$ **First, TMN changes the relative update strength across prompts within a task.** GRPO standardizes each prompt separately. TMN uses a shared denominator within each task, preserving the relative scale $s_q/S_t$. Omitting the numerical stabilizer, prompts with larger within-group reward variation receive greater relative weight, while those with smaller variation receive less, compared with GRPO. **Second, Reweight changes the balance between reinforcing better responses and suppressing poorer responses within a prompt.** Because TMN centers rewards at the group mean, the sum of positive advantages equals the sum of the magnitudes of negative advantages in any group with nonzero advantages. After reweighting, their ratio becomes $$\frac{\sum_{\widetilde A_{qi} > 0}\widetilde A_{qi}} {\sum_{\widetilde A_{qi} < 0}|\widetilde A_{qi}|} =w_q^2=\exp(1-2\widehat p_q).$$ When $\widehat p_q < 0.5$, the update puts more weight on reinforcing the few better trajectories. When $\widehat p_q > 0.5$, it puts more weight on correcting poorer trajectories. This ratio measures the total advantage weight on each side. The actual gradient also depends on the direction of each trajectory's policy gradient. These mechanisms help explain the performance trade-off. Let $c_{qi}$ be the combined weight in the first equation. For the same batch of sampled responses and with the advantage statistics held fixed, the policy gradient can be written as $$g_{\mathrm{TMN\text{-}Reweight}} =\sum_{q,i}c_{qi}\\,g^{\mathrm{GRPO}}_{qi}.$$ Changing the weights can change the direction of the combined update. Let $J_b(\theta)$ be the expected performance on benchmark $b$. For a small gradient ascent step $\theta'=\theta+\eta g_{\mathrm{TMN\text{-}Reweight}}$, the first-order change in expected performance is $$\Delta J_b\approx\eta\\,\left\langle \nabla_\theta J_b,\\, g_{\mathrm{TMN\text{-}Reweight}} \right\rangle.$$ The gradients of different benchmarks can align differently with the same update direction, so the average score can rise while scores on individual benchmarks fall.
Reviewer_qLTwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GoLongRL, a capability-oriented long-context RLVR framework built on 22,965 samples spanning nine task types with task-aligned rewards. It also proposes TMN-Reweight for heterogeneous multitask optimization and evaluates transfer to reasoning, memory, and overlong-context tasks.

Critical strengths

The capability-oriented formulation is well motivated. Beyond retrieval QA, the nine tasks cover structured extraction, ranking, ordering, and summarization, each with rewards suited to its output format (Table 1). The combination of curated and real-document-grounded synthetic data is practical and potentially reusable.

The results show clear GRPO gains at both 4B and 30B scales (Table 2), with additional transfer to general reasoning and memory (Tables 3-4). The analysis of task-level advantage scales also gives a concrete motivation for the proposed optimization method.

Critical weaknesses

The independent contribution of capability diversity is not fully isolated. T1-T4 make up 92.5% of training data, while gains are uneven across benchmarks and particularly large on MRCR and CorpusQA. A size- and token-matched T1-T4-only ablation would better establish the value of T5-T9.

The comparison with QwenLong-L1.5 remains cross-paper rather than fully controlled. Table 2 uses externally reported GRPO scores because the checkpoints and data are unavailable. Appendix D provides useful protocol checks, but remaining implementation differences limit direct conclusions about superiority.

TMN-Reweight needs stronger validation. It raises the 4B average by only 0.8 points while MRCR drops by 2.0 (Figure 4). Appendix C examines the smoothing coefficient, but the two components are not separately ablated, and multi-seed or 30B results are not reported.

Questions for authors
  1. What is the performance of size- and token-matched T1-T4-only GRPO training on the same 4B backbone?

  2. What evaluation differences remain in the QwenLong-L1.5 comparison after the protocol checks in Appendix D?

  3. What are the results of TMN normalization alone and difficulty-adaptive reweighting alone?

  4. Does the TMN-Reweight improvement persist across random seeds and at the 30B scale?

4 comments

Q1 / W1. Size- and token-matched T1–T4 ablation

Thank you for suggesting a direct test of the value of capability coverage.

T5–T9 contain 1,717 examples, accounting for 7.48% of the full dataset. Despite their small share, these tasks directly supervise structured extraction, ranking, order reconstruction, and summarization, extending supervision beyond retrieval-based question answering.

To supplement the dataset validation results in the submission, we report experiments on three versions evaluated during dataset construction. All three runs used the same Qwen3-4B-Thinking-2507 backbone, standard GRPO, and evaluation setup. Each run used 8,000 training examples drawn at random from the corresponding data pool. The counts in parentheses indicate the full pool sizes.

V1 was already close to the final version on single-document understanding and basic retrieval, while cross-document reasoning and long-range context memory were relatively weak. After we broadened task coverage in V2, CorpusQA rose by 7.6 points and the six-benchmark average rose by 2.2 points. MRCR rose by only 1.9 points. Based on this result, we added examples targeting multi-hop reasoning and context memory in V3. Relative to V2, MRCR and CorpusQA rose by 26.8 and 7.3 points, respectively, and the six-benchmark average rose by 7.2 points. With the training sample count fixed at 8,000 across all three versions, these results support refining the data composition and adding supervision targeted at the weaknesses identified in evaluation.

Version (data pool size)DocMathLBV2FramesMRCRCorpusQALBV1-QAAverage
V1 (9,630)59.842.562.138.850.263.652.8
V2 (17,729)60.043.764.440.757.863.555.0
V3 (22,965)62.545.566.667.565.165.962.2

Q2 / W2. Evaluation comparability with QwenLong-L1.5

Thank you for raising the question of comparison fairness. We use the same six long-context benchmarks and evaluation protocol as QwenLong-L1.5, with context lengths up to 128K. In our local evaluations, we use identical generation settings for the base models, GoLongRL, and the publicly released QwenLong-L1.5-30B-A3B. To check evaluation consistency, Appendix D (Table 8) compares our re-evaluation scores for the released model with the originally reported scores on each benchmark.

On five of the six benchmarks, our scores differ from the reported scores by at most 0.5 points. The difference on LBV1-QA is 2.7 points. This close agreement supports the performance comparisons in Table 2.

The remaining limitations mainly concern access to checkpoints and training data. QwenLong's final 4B checkpoint, intermediate GRPO checkpoints at both 4B and 30B, and training data have not been released, so we rely on the original report for the corresponding scores. We re-evaluated the publicly released QwenLong-L1.5-30B-A3B checkpoint. The tables label reported and locally evaluated scores separately so readers can verify the basis for both comparisons across sources and local comparisons before and after training.

Q3 / W3. TMN-only and Reweight-only ablations

Thank you for asking about the individual contributions of the two components. We have started the TMN-only and Reweight-only training runs, and the final ablation results are pending.

GRPO normalizes advantages using the within-group reward standard deviation for each prompt. TMN retains each prompt's group mean as the centering baseline and uses a shared denominator for prompts from the same task in the current batch. This denominator is the root mean square of their within-group reward standard deviations, which preserves relative advantage magnitudes across prompts within each task. Figure 3 shows that the coefficient of variation across tasks for mean absolute advantage drops from 0.34 to 0.18. Drawing on HA-DW, Reweight adjusts advantage weights based on relative difficulty within each task. It amplifies positive advantages and attenuates negative advantages for harder prompts, and reverses the adjustment for easier prompts.

With the same 4B backbone, training data, and evaluation setup, the full TMN-Reweight method raises the average across six benchmarks from 62.2 to 63.0 (+0.8 points). The gains come primarily from CorpusQA, LongBench V2, and Frames.

Q4 / W3. Robustness across random seeds and model scales

(1) Training seeds and sensitivity to the smoothing coefficient

Thank you for asking about the robustness of our results. A full long-context RL run is computationally expensive. One 4B run takes 160 training steps on 128 H800 GPUs. At about 40 minutes per step, this amounts to roughly 107 hours, or 13,700 GPU-hours. Given this cost, we do not yet have results from full training runs across multiple random seeds.

Table 7 reports sensitivity to the smoothing coefficient. At α=0.5\alpha=0.5 and α=0.8\alpha=0.8, the average scores across six benchmarks are 62.9 and 63.0, respectively. These are 0.7 and 0.8 points above GRPO's average of 62.2 under the same setup, showing gains at both tested values. The averages at α=0\alpha=0 and α=1\alpha=1 are 61.3 and 61.5, respectively. These results favor combining the prompt and task means over using either mean alone.

(2) Applicability at 30B

Thank you for asking about applicability to larger models. TMN-Reweight uses scalar rewards from sampled responses and the associated task labels.

We first evaluated the algorithm on a dense 4B model. With the backbone, training data, and evaluation setup fixed, TMN-Reweight raises the average score across six benchmarks from 62.2 with GRPO to 63.0. The current 30B experiments evaluate the data recipe. With standard GRPO, the average across six benchmarks rises from 60.1 to 69.8.

TMN-Reweight is independent of model size and of architectural details specific to the 4B model. It reuses existing rollouts and requires no additional reward network or generation, so it can be applied to larger models such as the 30B model.

TMN-Only and Reweight-Only Ablations (4B, Step 45)

Thank you for asking about the individual contributions of TMN and difficulty-adaptive reweighting. Given the compute available during rebuttal, we evaluated all three 4B configurations after 45 training steps. Each row reports results from that configuration’s intermediate checkpoint at step 45.

4B configurationAvg.DocMathLBV2FramesMRCRCorpusQALBV1-QA
Reweight-only56.8861.7546.1262.3852.3455.3263.40
TMN-only57.0363.1344.3362.8651.5256.5363.80
TMN + Reweight57.1762.3844.8362.9252.9856.2363.65

Effect of TMN. Figure 3 directly measures how well advantage scales align across the nine training tasks. The cross-task coefficient of variation in mean absolute advantage falls from 0.34 under GRPO to 0.18 under TMN. The new ablation complements this observation with downstream results. Adding TMN to reweighting improved five of the six benchmarks at step 45, with the largest gain on CorpusQA (+0.91), and raised the average from 56.88 to 57.17.

Effect of difficulty-adaptive reweighting. Adding reweighting to TMN raised the average from 57.03 to 57.17. The largest gains were on MRCR (+1.46) and LongBench V2 (+0.50), while DocMath decreased by 0.75. Table 7 separately tests our choice of difficulty threshold. Combining prompt- and task-level reward means gave averages of 62.9 and 63.0 at α = 0.5 and 0.8, respectively, compared with 61.3 using the task mean alone and 61.5 using the prompt mean alone. This provides empirical support for using task context when estimating prompt difficulty.

The combined method had the highest six-benchmark average at step 45. The margins are modest, but both component comparisons favor retaining the two together, complementing the comparison with vanilla GRPO in Figure 4. We will include this step-45 ablation and its per-benchmark results in the camera-ready version.

AuthorsOpenReview ↗

Request for Follow-up Before the Discussion Closes

Dear reviewer qLTw , with the rebuttal period ending soon, we'd like to kindly request you to read our rebuttal and reconsider your evaluation for our paper. We hope our responses have addressed your concerns

Thanks very much for your effort and valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.