- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Multi-Session Agentic Reinforcement Learning (MOSAIC), a new setting for training agents whose trajectories span multiple sessions due to context compaction or subagent forking. The authors propose a session-count-aware trajectory-level advantage with a session-level shaping signal based on each session’s marginal contribution to answer confidence. They evaluate MOSAIC on BrowseComp using Qwen3.5-4B, showing that it outperforms a naive broadcasting baseline in both compaction and subagent-forking scenarios while using fewer sessions on average.
Critical strengths
- Novel problem formulation. The paper addresses an important gap by extending GRPO-style methods beyond single-session trajectories to settings with context compaction and subagent forking.
- Clear presentation. The paper explains its technical ideas progressively, uses clear figures, and provides detailed implementation information in the appendix to support reproducibility.
- Thorough ablations. RQ2–RQ4 examine each component’s contribution, validate the intended advantage ordering (Fig. 5a/b), and link performance gains to session quality and allocation rather than session count alone (Table 2).
Critical weaknesses
-
Evaluation limited to a single benchmark and a single model. All experiments are conducted on BrowseComp alone, using only Qwen3.5-4B. There is no evidence that MOSAIC’s gains hold on other agentic benchmarks or at different model scales. This narrow scope is a meaningful limitation for a paper proposing a general new RL setting (“Multi-Session Agentic RL”) and method. Comparable recent work in this space (e.g., IGPO) evaluates across 7 datasets and 2 model scales, which better supports claims of generality. I’d recommend the authors add at least one additional (suitable, long-horizon) benchmark or model size to strengthen the evidence that MOSAIC’s advantages are not specific to this single task/model combination.
-
No sensitivity analysis for newly introduced hyperparameters. MOSAIC introduces four new hyperparameters: , , , and , all fixed at single values (, , , ) with no ablation sweep reported. Since these parameters directly control how much the session-aware and session-level signals can influence the final advantage, it is unclear how sensitive MOSAIC's performance is to these choices, or how robust the selected values would be if transferred to a different task, benchmark, or model scale.
-
Subagent-forking results. In Table 1, single-session-trained GRPO evaluated under test-time multi-session (subagent) inference reaches 31.91 accuracy using only 2.09 sessions on average, nearly matching MOSAIC's 32.46 at 3.77 sessions. This suggests that in the subagent scenario, the benefit of dedicated multi-session training is marginally weaker than in the compact scenario. The introduction "10.1% relative improvement" mention should be specific to the relevant setting.
I would be happy to raise my score if the authors can adequately address weaknesses 1 and 2.
Questions for authors
- How sensitive is MOSAIC's performance to the choice of , , , and ? Were these values tuned specifically on BrowseComp, and do the authors expect them to transfer to other multi-session tasks or model scales without re-tuning?
8 comments
Response to Reviewer sTzq [part 2/3]
(Continued from previous part)
Third, the smaller and alternative backbones we tested did not achieve sufficient initial task success for effective RL training in this setting. We explored Qwen3-4B, Gemma-4-E2B, and Qwen3.5-2B. Although these models avoided the memory issues above, successful trajectories were very sparse within rollout groups, and even the basic GRPO baseline did not show clear learning progress. For instance, when training Qwen3.5-2B with single-session GRPO, the mean reward over the first 30 steps was 0.047060, with no upward trend. Under such situation, an SFT warm-up is a possible way to address this difficulty and has been adopted in prior work. However, we chose not to introduce an additional SFT stage in order to keep the experimental setup controlled and avoid adding another source of variation. With sufficient computational resources, stronger backbones could offer a more viable path: we expect them to generate successful trajectories more frequently and thereby provide a better starting point for RL training.
So we hope these details clarify the considerations behind our original experimental scope. Thorough empirical validation of our method remains a priority, and we have made substantial efforts to address these challenges and broaden the evaluation through the additional benchmark results reported above.
Hyperparameter sensitivity analysis (W2 & Q1)
Thank you for raising this concern. We first address your question about how the default hyperparameters were selected. Mosaic’s hyperparameters were not tuned specifically for BrowseComp. Their initial values were primarily chosen based on the numerical scales of group-relative advantages under binary rewards and of normalized marginal information gain. We also incorporated gating mechanisms to control potentially disruptive advantage perturbations. These choices were intended to support stable optimization.
Nevertheless, without sufficient empirical validation, we cannot guarantee that the default values will transfer directly to other multi-session tasks or models without adjustment. We agree that sensitivity analysis is important for understanding a method’s robustness and transferability. Although retraining for each configuration is time-consuming, we began conducting additional sensitivity experiments after the submission deadline.
The tested values are as follows:
| Parameter | Lower value | Current value | Higher value | Additional setting |
|---|---|---|---|---|
| 0.1 | 0.2 | 0.4 | - | |
| (0.05, 0.1) | (0.1, 0.2) | (0.2, 0.4) | (0.2, 0.2) | |
| 0.05 | 0.1 | 0.2 | - |
We use a one-factor-at-a-time design, treating as a paired factor. Except for the parameter or parameter pair being varied, all values remain at their defaults.
- Rationale for the paired settings. Our default design uses . For all-failure groups, we assign a more conservative weight to the heuristic that fewer sessions may indicate insufficient exploration. For all-success groups, we apply a stronger preference for fewer sessions: once the task has been solved, preferring a trajectory that uses fewer sessions is more directly justified. In contrast, failure can have causes other than insufficient exploration, making session count a less certain guide in unsuccessful trajectories. The paired sweep preserves this asymmetry while varying the overall scale. We additionally evaluate to examine the effect of removing the asymmetry.
- Experimental setting. We conduct the experiments in a fixed multi-session context-compaction setting using Qwen3.5-4B. All other hyperparameters and settings follow Section 4.1, including the training-data order. We consistently evaluate the final checkpoint from each run, using only the multi-session (compact) inference mode on BrowseComp.
The results are:
| Configuration | Acc. | Acc. vs. Default | Avg. sessions |
|---|---|---|---|
| Default () | 38.86 | - | 2.35 |
| 38.31 | -0.55 | 3.29 | |
| 37.84 | -1.02 | 2.04 | |
| 39.34 | +0.48 | 2.35 | |
| 38.47 | -0.39 | 2.31 | |
| 38.31 | -0.55 | 2.32 | |
| 40.21 | +1.35 | 2.17 | |
| 36.81 | -2.05 | 2.20 |
We draw several main observations from these results:
- has a noticeable effect on session usage. Increasing from 0.1 to 0.4 reduces the average session count from 3.29 to 2.04, while the tested settings remain within 1.02 of the default accuracy.
- Performance changes relatively little under paired scaling of . Halving or doubling both coefficients changes accuracy by +0.48 and −0.39, respectively. The equal-weight configuration changes accuracy by −0.55.
(Continued on next part)
Response to Reviewer sTzq [part 3/3]
(Continued from previous part)
In addition to these new experiments, we examined the original training logs to better characterize how the hyperparameters affect the training signal.
- For , the modulation is bounded and preserves the sign of the original trajectory advantage in mixed-outcome groups. For example, at , the multiplicative factor lies in [0.8,1.2], limiting the magnitude of the perturbation. We also observe that the RMS deviation of this factor from 1 decreases over training, indicating that the effective modulation becomes weaker. This may be associated with reduced session usage and less variation in session counts later in training, and could help stabilize the later stages of optimization.
- For and , their effective contributions depend on how frequently the corresponding groups occur. All-failure groups are substantially more frequent than all-success groups. In the Mosaic compact training run, the fallback was active across 58 training steps, with the corresponding groups accounting for 35.59% of all groups. The fallback was active across only 14 steps, with the corresponding groups accounting for 3.07%. Consequently, despite , contributes a larger aggregate absolute advantage. Nevertheless, the combined absolute contribution of the two fallback terms remains smaller than that of the mixed-outcome groups, suggesting that they introduce a relatively limited perturbation to the overall training signal.
Overall, accuracy ranges from 36.81 to 40.21 across the tested configurations, and all configurations remain above the broadcasting baseline, which support robustness within the tested range.
Result description (W3)
Thank you for pointing this out. We agree that the magnitude of improvement differs across settings, and that our introduction should state this more precisely. The 10.1% relative improvement specifically refers to the context compaction setting compared with Broadcasting, where accuracy increases from 35.31% to 38.86%. In the subagent-forking setting, Mosaic’s accuracy gain over GRPO is more modest.
We will therefore revise the sentence on line 76 in the Introduction as follows:
“Moreover, in the context compaction setting, Mosaic achieves a 10.1% relative improvement in accuracy over the broadcasting baseline, while using fewer sessions per trajectory on average (3.17 → 2.35).”
We will carefully revise the introduction to explicitly identify the setting associated with this result and avoid overstating the scope of the improvement.
Thank you again for the time and effort you have devoted to reviewing our submission. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of these clarifications. If you have any follow-up questions, please feel free to raise them, and we would be happy to discuss them further. Thank you again for your consideration!
Response to Reviewer sTzq [part 1/3]
Dear Reviewer sTzq,
We sincerely thank you for your valuable feedback. We have carefully considered your comments and conducted additional experiments and analyses to address them. Below, we provide a point-by-point response, which we hope will resolve your concerns.
Additional benchmark evaluation experiments (W1)
Thank you for highlighting this limitation. We have conducted additional evaluations on three more benchmarks: GAIA, HLE, and xbench-DeepSearch. We considered a range of benchmarks used in related work, including the seven evaluated in IGPO, such as HotpotQA. However, most were insufficiently challenging under our experimental settings to elicit multi-session trajectories frequently enough, limiting their suitability for evaluating multi-session optimization.
The selected three benchmarks are particularly well suited to this evaluation: they provide complementary task coverage to BrowseComp while remaining sufficiently challenging to naturally elicit multi-session trajectories. The details of the three benchmarks are as follows:
We follow the same evaluation settings as in Table 1 of the paper. Accuracy is reported as a percentage, and session counts are averaged over trajectories.
GAIA:
HLE:
xbench-DeepSearch:
Across the completed comparisons, Mosaic achieves better performance cross all three additional benchmarks. These results strengthen the evidence that Mosaic’s benefits extend beyond BrowseComp.
Moreover, we would also like to explain the practical considerations behind the original submission’s focus on one benchmark and one backbone model.
First, obtaining sufficiently rich multi-session trajectories places substantial demands on task difficulty and training-data availability. We use a per-session context budget of 128K tokens to approximate practical long-context usage. Under this relatively large budget, sufficiently challenging, long-horizon tasks are important for producing trajectories that meaningfully involve multiple sessions. We therefore sought both a suitable evaluation benchmark and high-quality, publicly available training data of the corresponding task type. Based on our preliminary investigations and experiments, BrowseComp and REDSearcher_RL_1K met these requirements. In fact, we also explored the benchmarks used by IGPO during our initial experiments, but found that they did not generate multi-session trajectories frequently enough under our settings. For example, in a preliminary experiment with Qwen3-4B on HotpotQA, the average session count was only 1.0039, while accuracy had already reached approximately 60.
Second, extending the experiments to larger backbones was constrained by computational resources. We planned to repeat the experiments using Qwen3.5-9B. However, our available hardware consists of eight GPUs, each with 141 GB of memory. Although this configuration can support substantial single-session agentic RL workloads, multi-session training increases the number of training segments per rollout, substantially raising memory requirements and training time. Despite multiple attempts to optimize the configuration, our Qwen3.5-9B training runs repeatedly encountered OOM issues within the first few steps.
(Continued on next part)