acceptodds
Under review as a conference paper at ICLR 2027

Mosaic: Multi-Session Agentic Reinforcement Learning

Abstract

Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session’s advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested that is a promising target for future advances in multi-session agentic RL.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Language Agents, Reinforcement Learning, Long-horizon

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sTzqOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Multi-Session Agentic Reinforcement Learning (MOSAIC), a new setting for training agents whose trajectories span multiple sessions due to context compaction or subagent forking. The authors propose a session-count-aware trajectory-level advantage with a session-level shaping signal based on each session’s marginal contribution to answer confidence. They evaluate MOSAIC on BrowseComp using Qwen3.5-4B, showing that it outperforms a naive broadcasting baseline in both compaction and subagent-forking scenarios while using fewer sessions on average.

Critical strengths
  1. Novel problem formulation. The paper addresses an important gap by extending GRPO-style methods beyond single-session trajectories to settings with context compaction and subagent forking.
  2. Clear presentation. The paper explains its technical ideas progressively, uses clear figures, and provides detailed implementation information in the appendix to support reproducibility.
  3. Thorough ablations. RQ2–RQ4 examine each component’s contribution, validate the intended advantage ordering (Fig. 5a/b), and link performance gains to session quality and allocation rather than session count alone (Table 2).
Critical weaknesses
  1. Evaluation limited to a single benchmark and a single model. All experiments are conducted on BrowseComp alone, using only Qwen3.5-4B. There is no evidence that MOSAIC’s gains hold on other agentic benchmarks or at different model scales. This narrow scope is a meaningful limitation for a paper proposing a general new RL setting (“Multi-Session Agentic RL”) and method. Comparable recent work in this space (e.g., IGPO) evaluates across 7 datasets and 2 model scales, which better supports claims of generality. I’d recommend the authors add at least one additional (suitable, long-horizon) benchmark or model size to strengthen the evidence that MOSAIC’s advantages are not specific to this single task/model combination.

  2. No sensitivity analysis for newly introduced hyperparameters. MOSAIC introduces four new hyperparameters: ρ\rho, β0\beta_0, β1\beta_1, and λr\lambda_r, all fixed at single values (0.20.2, 0.10.1, 0.20.2, 0.10.1) with no ablation sweep reported. Since these parameters directly control how much the session-aware and session-level signals can influence the final advantage, it is unclear how sensitive MOSAIC's performance is to these choices, or how robust the selected values would be if transferred to a different task, benchmark, or model scale.

  3. Subagent-forking results. In Table 1, single-session-trained GRPO evaluated under test-time multi-session (subagent) inference reaches 31.91 accuracy using only 2.09 sessions on average, nearly matching MOSAIC's 32.46 at 3.77 sessions. This suggests that in the subagent scenario, the benefit of dedicated multi-session training is marginally weaker than in the compact scenario. The introduction "10.1% relative improvement" mention should be specific to the relevant setting.

I would be happy to raise my score if the authors can adequately address weaknesses 1 and 2.

Questions for authors
  1. How sensitive is MOSAIC's performance to the choice of ρ\rho, β0\beta_0, β1\beta_1, and λr\lambda_r? Were these values tuned specifically on BrowseComp, and do the authors expect them to transfer to other multi-session tasks or model scales without re-tuning?
8 comments
AuthorsOpenReview ↗

Response to Reviewer sTzq [part 1/3]

Dear Reviewer sTzq,

We sincerely thank you for your valuable feedback. We have carefully considered your comments and conducted additional experiments and analyses to address them. Below, we provide a point-by-point response, which we hope will resolve your concerns.


Additional benchmark evaluation experiments (W1)

Thank you for highlighting this limitation. We have conducted additional evaluations on three more benchmarks: GAIA, HLE, and xbench-DeepSearch. We considered a range of benchmarks used in related work, including the seven evaluated in IGPO, such as HotpotQA. However, most were insufficiently challenging under our experimental settings to elicit multi-session trajectories frequently enough, limiting their suitability for evaluating multi-session optimization.

The selected three benchmarks are particularly well suited to this evaluation: they provide complementary task coverage to BrowseComp while remaining sufficiently challenging to naturally elicit multi-session trajectories. The details of the three benchmarks are as follows:

  • GAIA: The 103 text-only questions from the GAIA validation set.
  • HLE: The 200-question subset specified in the list released by BATS.
  • xbench-DeepSearch: The 100 deep-search questions from the 2510 version.

We follow the same evaluation settings as in Table 1 of the paper. Accuracy is reported as a percentage, and session counts are averaged over trajectories.

GAIA:

MethodSingle Acc.Single Avg. sessionsCompact Acc.Compact Avg. sessionsSubagent Acc.Subagent Avg. sessions
Qwen3.5-4B54.371.0063.111.4356.311.91
GRPO70.871.0073.791.4977.671.37
IGPO66.021.0068.932.2457.281.39
Broadcasting--73.792.2476.701.44
Mosaic--75.731.8781.551.39

HLE:

MethodSingle Acc.Single Avg. sessionsCompact Acc.Compact Avg. sessionsSubagent Acc.Subagent Avg. sessions
Qwen3.5-4B21.501.0024.001.7120.001.90
GRPO28.501.0031.001.6030.501.45
IGPO26.501.0029.503.1530.001.80
Broadcasting--33.503.6934.001.60
Mosaic--36.002.8334.501.47

xbench-DeepSearch:

MethodSingle Acc.Single Avg. sessionsCompact Acc.Compact Avg. sessionsSubagent Acc.Subagent Avg. sessions
Qwen3.5-4B24.001.0024.002.0221.001.64
GRPO32.001.0027.001.4827.001.46
IGPO27.001.0031.003.6627.001.30
Broadcasting--27.003.3034.001.91
Mosaic--35.001.0439.001.66

Across the completed comparisons, Mosaic achieves better performance cross all three additional benchmarks. These results strengthen the evidence that Mosaic’s benefits extend beyond BrowseComp.

Moreover, we would also like to explain the practical considerations behind the original submission’s focus on one benchmark and one backbone model.

First, obtaining sufficiently rich multi-session trajectories places substantial demands on task difficulty and training-data availability. We use a per-session context budget of 128K tokens to approximate practical long-context usage. Under this relatively large budget, sufficiently challenging, long-horizon tasks are important for producing trajectories that meaningfully involve multiple sessions. We therefore sought both a suitable evaluation benchmark and high-quality, publicly available training data of the corresponding task type. Based on our preliminary investigations and experiments, BrowseComp and REDSearcher_RL_1K met these requirements. In fact, we also explored the benchmarks used by IGPO during our initial experiments, but found that they did not generate multi-session trajectories frequently enough under our settings. For example, in a preliminary experiment with Qwen3-4B on HotpotQA, the average session count was only 1.0039, while accuracy had already reached approximately 60.

Second, extending the experiments to larger backbones was constrained by computational resources. We planned to repeat the experiments using Qwen3.5-9B. However, our available hardware consists of eight GPUs, each with 141 GB of memory. Although this configuration can support substantial single-session agentic RL workloads, multi-session training increases the number of training segments per rollout, substantially raising memory requirements and training time. Despite multiple attempts to optimize the configuration, our Qwen3.5-9B training runs repeatedly encountered OOM issues within the first few steps.

(Continued on next part)

AuthorsOpenReview ↗

Response to Reviewer sTzq [part 2/3]

(Continued from previous part)

Third, the smaller and alternative backbones we tested did not achieve sufficient initial task success for effective RL training in this setting. We explored Qwen3-4B, Gemma-4-E2B, and Qwen3.5-2B. Although these models avoided the memory issues above, successful trajectories were very sparse within rollout groups, and even the basic GRPO baseline did not show clear learning progress. For instance, when training Qwen3.5-2B with single-session GRPO, the mean reward over the first 30 steps was 0.047060, with no upward trend. Under such situation, an SFT warm-up is a possible way to address this difficulty and has been adopted in prior work. However, we chose not to introduce an additional SFT stage in order to keep the experimental setup controlled and avoid adding another source of variation. With sufficient computational resources, stronger backbones could offer a more viable path: we expect them to generate successful trajectories more frequently and thereby provide a better starting point for RL training.

So we hope these details clarify the considerations behind our original experimental scope. Thorough empirical validation of our method remains a priority, and we have made substantial efforts to address these challenges and broaden the evaluation through the additional benchmark results reported above.


Hyperparameter sensitivity analysis (W2 & Q1)

Thank you for raising this concern. We first address your question about how the default hyperparameters were selected. Mosaic’s hyperparameters were not tuned specifically for BrowseComp. Their initial values were primarily chosen based on the numerical scales of group-relative advantages under binary rewards and of normalized marginal information gain. We also incorporated gating mechanisms to control potentially disruptive advantage perturbations. These choices were intended to support stable optimization.

Nevertheless, without sufficient empirical validation, we cannot guarantee that the default values will transfer directly to other multi-session tasks or models without adjustment. We agree that sensitivity analysis is important for understanding a method’s robustness and transferability. Although retraining for each configuration is time-consuming, we began conducting additional sensitivity experiments after the submission deadline.

The tested values are as follows:

ParameterLower valueCurrent valueHigher valueAdditional setting
ρ\rho0.10.20.4-
(β0,β1)(\beta_0,\beta_1)(0.05, 0.1)(0.1, 0.2)(0.2, 0.4)(0.2, 0.2)
λr\lambda_r0.050.10.2-

We use a one-factor-at-a-time design, treating (β0,β1)(\beta_0,\beta_1) as a paired factor. Except for the parameter or parameter pair being varied, all values remain at their defaults.

  • Rationale for the paired β\beta settings. Our default design uses β0<β1\beta_0<\beta_1. For all-failure groups, we assign a more conservative weight to the heuristic that fewer sessions may indicate insufficient exploration. For all-success groups, we apply a stronger preference for fewer sessions: once the task has been solved, preferring a trajectory that uses fewer sessions is more directly justified. In contrast, failure can have causes other than insufficient exploration, making session count a less certain guide in unsuccessful trajectories. The paired sweep preserves this asymmetry while varying the overall scale. We additionally evaluate β0=β1=0.2\beta_0=\beta_1=0.2 to examine the effect of removing the asymmetry.
  • Experimental setting. We conduct the experiments in a fixed multi-session context-compaction setting using Qwen3.5-4B. All other hyperparameters and settings follow Section 4.1, including the training-data order. We consistently evaluate the final checkpoint from each run, using only the multi-session (compact) inference mode on BrowseComp.

The results are:

ConfigurationAcc.Δ\Delta Acc. vs. DefaultAvg. sessions
Default (ρ=0.2,(β0,β1)=(0.1,0.2),λr=0.1\rho=0.2,(\beta_0,\beta_1)=(0.1,0.2),\lambda_r=0.1)38.86-2.35
ρ=0.1\rho=0.138.31-0.553.29
ρ=0.4\rho=0.437.84-1.022.04
(β0,β1)=(0.05,0.1)(\beta_0,\beta_1)=(0.05,0.1)39.34+0.482.35
(β0,β1)=(0.2,0.4)(\beta_0,\beta_1)=(0.2,0.4)38.47-0.392.31
(β0,β1)=(0.2,0.2)(\beta_0,\beta_1)=(0.2,0.2)38.31-0.552.32
λr=0.05\lambda_r=0.0540.21+1.352.17
λr=0.2\lambda_r=0.236.81-2.052.20

We draw several main observations from these results:

  • ρ\rho has a noticeable effect on session usage. Increasing ρ\rho from 0.1 to 0.4 reduces the average session count from 3.29 to 2.04, while the tested ρ\rho settings remain within 1.02 of the default accuracy.
  • Performance changes relatively little under paired scaling of (β0,β1)(\beta_0,\beta_1). Halving or doubling both coefficients changes accuracy by +0.48 and −0.39, respectively. The equal-weight configuration changes accuracy by −0.55.

(Continued on next part)

AuthorsOpenReview ↗

Response to Reviewer sTzq [part 3/3]

(Continued from previous part)

In addition to these new experiments, we examined the original training logs to better characterize how the hyperparameters affect the training signal.

  • For ρ\rho, the modulation is bounded and preserves the sign of the original trajectory advantage in mixed-outcome groups. For example, at ρ=0.2\rho=0.2, the multiplicative factor lies in [0.8,1.2], limiting the magnitude of the perturbation. We also observe that the RMS deviation of this factor from 1 decreases over training, indicating that the effective modulation becomes weaker. This may be associated with reduced session usage and less variation in session counts later in training, and could help stabilize the later stages of optimization.
  • For β0\beta_0 and β1\beta_1, their effective contributions depend on how frequently the corresponding groups occur. All-failure groups are substantially more frequent than all-success groups. In the Mosaic compact training run, the β0\beta_0 fallback was active across 58 training steps, with the corresponding groups accounting for 35.59% of all groups. The β1\beta_1 fallback was active across only 14 steps, with the corresponding groups accounting for 3.07%. Consequently, despite β0<β1\beta_0<\beta_1, β0\beta_0 contributes a larger aggregate absolute advantage. Nevertheless, the combined absolute contribution of the two fallback terms remains smaller than that of the mixed-outcome groups, suggesting that they introduce a relatively limited perturbation to the overall training signal.

Overall, accuracy ranges from 36.81 to 40.21 across the tested configurations, and all configurations remain above the broadcasting baseline, which support robustness within the tested range.


Result description (W3)

Thank you for pointing this out. We agree that the magnitude of improvement differs across settings, and that our introduction should state this more precisely. The 10.1% relative improvement specifically refers to the context compaction setting compared with Broadcasting, where accuracy increases from 35.31% to 38.86%. In the subagent-forking setting, Mosaic’s accuracy gain over GRPO is more modest.

We will therefore revise the sentence on line 76 in the Introduction as follows:

“Moreover, in the context compaction setting, Mosaic achieves a 10.1% relative improvement in accuracy over the broadcasting baseline, while using fewer sessions per trajectory on average (3.17 → 2.35).”

We will carefully revise the introduction to explicitly identify the setting associated with this result and avoid overstating the scope of the improvement.


Thank you again for the time and effort you have devoted to reviewing our submission. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of these clarifications. If you have any follow-up questions, please feel free to raise them, and we would be happy to discuss them further. Thank you again for your consideration!

Reviewer_9UnpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MOSAIC, a critic-free hierarchical optimization method for multi-session agentic reinforcement learning. It addresses the mismatch between single-session RL training and multi-session inference workflows (e.g., context compaction, subagent forking). MOSAIC consists of two components: (1) Session-aware trajectory-level advantage estimation, which modulates the standard GRPO outcome advantage using a bounded rank of session counts, encouraging efficient execution for successful trajectories and sufficient exploration for failed ones. (2) Session-level advantage shaping, which computes a dense reward based on the marginal information gain (change in the model's likelihood of generating the ground-truth answer) for each session transition, applying type-conditioned normalization. Validation is performed on BrowseComp using Qwen3.5-4B, demonstrating that MOSAIC outperforms single-session GRPO, test-time scaling, and the broadcasting baseline under both compaction and subagent scenarios, while using fewer sessions on average.

Critical strengths

Elegant Trajectory-Level Modulation: The formulation for session-aware trajectory advantage is highly principled. By using multiplicative modulation and centered ranks, the method rigorously guarantees that the outcome reward remains the dominant signal (preserving the sign of the advantage) while seamlessly injecting an efficiency/exploration bias based on session count. The fallback mechanisms for homogeneous groups (all-success/all-failure) are also well-conceived to maintain sample efficiency.

Effective and Computationally Efficient Dense Reward: Using the marginal information gain of the ground-truth answer to shape session-level advantages is a clever way to solve credit assignment without learning a critic. The paper shows it avoids the heavy computational cost of turn-level IG (as in IGPO) by operating at the session level, while the type-conditioned normalization successfully prevents systematic biases against specific session types (like compaction).

Critical weaknesses

Dependency on Ground-Truth Answer Tokenization: The session-level advantage shaping relies heavily on computing the log-likelihood of the ground-truth answer. This design limits the method's generalizability to tasks where the reward is purely outcome-based but not backed by a specific tokenizable ground truth (e.g., complex coding environments, interactive web navigation where the goal is a state rather than a string, or RLHF scenarios).

Complexity of Hyperparameter Tuning: The hierarchical optimization introduces several critical hyperparameters. While the default values are provided, the ablation on sensitivity is missing. It is unclear how brittle the training is if these are not carefully tuned for a new environment or base model.

Questions for authors

How does the method perform on tasks where the final outcome is evaluated by an environment state (e.g., executed code passing unit tests) rather than a textual ground-truth answer? Can the marginal information gain still be computed in such settings?

In the subagent scenario, the paper notes that the DAG structure makes immediate marginal gain less aligned with actual contribution. Could a graph-based credit assignment (e.g., propagating IG backwards through the subagent tree) further improve the session-level shaping in this specific scenario?

How sensitive is MOSAIC to the choice of ρ and λ? What is the computational overhead of the two forward passes required for IG computation compared to standard GRPO?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Unp [part 1/3]

Dear Reviewer 9Unp,

We sincerely appreciate your positive assessment on our submission! We aim to address the weaknesses and questions you raised, and we hope that this can alleviate your concerns.


Generalization of Mosaic (W1 & Q1)

Thank you for raising this point. Our current session-level advantage shaping indeed relies on the log-likelihood of the ground-truth answer. This choice is to match the style of the ground-truth answers of the training data used in our experiments. In this setting, marginal information gain offers a computationally economical local reward: it requires at most two forward propagations per session, without additional judge calls or repeated continuation rollouts, and has shown promising performance for fine-grained credit assignment [1].

However, this does not imply that Mosaic is restricted to a specific task format, as it is designed with extensibility and generality in mind. For different form of tasks, suitable session-level signals can be constructed using different approaches and directly integrated into Eqs. (17)–(18), indicating that they need not conform to the structure of ground-truth answer log-likelihood.

For instance, for tasks with complex outputs, such as agentic coding or deep research, we could construct task-specific rubrics and use an LLM-as-Judge to score the session’s trajectory or state. Let Ci={ci,k}k=1Ki\mathcal{C}_i=\{c_{i,k}\}_{k=1}^{K_i} denote the rubrics for task xix_i, and we use zi,j−z_{i,j}^{-} and zi,j+z_{i,j}^{+} to denote the states at the session’s input and output boundaries, respectively. We define $J(x_i,z;\mathcal C_i)\in\mathbb R $ denote the scalar score assigned by the judge to state zz. Then we can still calculate the session-level marginal gain as follows:

J\left(x_i,z_{i,j}^{-};\mathcal C_i\right).$$ Here, $r_{i,j}^{\mathrm{Rubric}}$ measures the change in rubric-assessed task progress across session $s_{i,j}$. It can replace the original likelihood-based local reward $r_{i,j}^{\Delta}$ in Eq. (17), after which the normalized shaping signal is incorporated into the session-level advantage through Eq. (18). Alternatively, for a replayable state $z$, including both the agent context and the relevant environment state, downstream success could also be estimated through repeated continuations under a fixed policy and continuation protocol: $$\widehat V^\pi(z)=\frac{1}{M} \sum_{m=1}^{M} R_i\left(\tau^{(m)}\right), \qquad \tau^{(m)}\sim\pi_{\theta}(\cdot\mid z).$$ Here, $R_i(\tau^{(m)})$ is the final task reward obtained after executing continuation from state $z$. For example, it may equal 1 if the resulting code passes the required unit tests or the environment reaches the target state, and 0 otherwise. Thus, $\widehat V^\pi(z)$ estimates the expected final task reward from $z$. So, the corresponding session-level local reward would be: $$r^{\mathrm{Replay}}_{i,j}=\widehat V^\pi(z_{i,j}^+)-\widehat V^\pi(z_{i,j}^-)$$ This also provides a feasible signal in principle, using environment-derived rewards without requiring a separately trained critic. However, it would require restoring boundary states and sampling additional continuations, making it substantially more expensive than the current two-forward-pass likelihood calculation. Due to resource and time constraints, we are unable to conduct more extensive experiments on these extensions at this stage, so our discussion remains conceptual. We plan to extend Mosaic to a broader range of tasks in future work, and we welcome your further thoughts and suggestions on this direction. [1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents [ICLR 2026]
AuthorsOpenReview ↗

Response to Reviewer 9Unp [part 2/3]

Hyperparameter sensitivity analysis (W2 & Q3)

Thank you for raising this concern. We agree that sensitivity analysis is important for understanding a method’s robustness and transferability. Although retraining for each configuration is time-consuming, we began conducting additional sensitivity experiments after the submission deadline.

The tested values are as follows:

ParameterLower valueCurrent valueHigher valueAdditional setting
ρ\rho0.10.20.4-
(β0,β1)(\beta_0,\beta_1)(0.05, 0.1)(0.1, 0.2)(0.2, 0.4)(0.2, 0.2)
λr\lambda_r0.050.10.2-

We use a one-factor-at-a-time design, treating (β0,β1)(\beta_0,\beta_1) as a paired factor. Except for the parameter or parameter pair being varied, all values remain at their defaults.

  • Rationale for the paired β\beta settings. Our default design uses β0<β1\beta_0<\beta_1. For all-failure groups, we assign a more conservative weight to the heuristic that fewer sessions may indicate insufficient exploration. For all-success groups, we apply a stronger preference for fewer sessions: once the task has been solved, preferring a trajectory that uses fewer sessions is more directly justified. In contrast, failure can have causes other than insufficient exploration, making session count a less certain guide in unsuccessful trajectories. The paired sweep preserves this asymmetry while varying the overall scale. We additionally evaluate β0=β1=0.2\beta_0=\beta_1=0.2 to examine the effect of removing the asymmetry.
  • Experimental setting. We conduct the experiments in a fixed multi-session context-compaction setting using Qwen3.5-4B. All other hyperparameters and settings follow Section 4.1, including the training-data order. We consistently evaluate the final checkpoint from each run, using only the multi-session (compact) inference mode on BrowseComp.

The results are:

ConfigurationAcc.Δ\Delta Acc. vs. DefaultAvg. sessions
Default (ρ=0.2,(β0,β1)=(0.1,0.2),λr=0.1\rho=0.2,(\beta_0,\beta_1)=(0.1,0.2),\lambda_r=0.1)38.86-2.35
ρ=0.1\rho=0.138.31-0.553.29
ρ=0.4\rho=0.437.84-1.022.04
(β0,β1)=(0.05,0.1)(\beta_0,\beta_1)=(0.05,0.1)39.34+0.482.35
(β0,β1)=(0.2,0.4)(\beta_0,\beta_1)=(0.2,0.4)38.47-0.392.31
(β0,β1)=(0.2,0.2)(\beta_0,\beta_1)=(0.2,0.2)38.31-0.552.32
λr=0.05\lambda_r=0.0540.21+1.352.17
λr=0.2\lambda_r=0.236.81-2.052.20

We draw several main observations from these results:

  • ρ\rho has a noticeable effect on session usage. Increasing ρ\rho from 0.1 to 0.4 reduces the average session count from 3.29 to 2.04, while the tested ρ\rho settings remain within 1.02 of the default accuracy.
  • Performance changes relatively little under paired scaling of (β0,β1)(\beta_0,\beta_1). Halving or doubling both coefficients changes accuracy by +0.48 and −0.39, respectively. The equal-weight configuration changes accuracy by −0.55.

In addition to these new experiments, we examined the original training logs to better characterize how the hyperparameters affect the training signal.

  • For ρ\rho, the modulation is bounded and preserves the sign of the original trajectory advantage in mixed-outcome groups. For example, at ρ=0.2\rho=0.2, the multiplicative factor lies in [0.8,1.2], limiting the magnitude of the perturbation. We also observe that the RMS deviation of this factor from 1 decreases over training, indicating that the effective modulation becomes weaker. This may be associated with reduced session usage and less variation in session counts later in training, and could help stabilize the later stages of optimization.
  • For β0\beta_0 and β1\beta_1, their effective contributions depend on how frequently the corresponding groups occur. All-failure groups are substantially more frequent than all-success groups. In the Mosaic compact training run, the β0\beta_0 fallback was active across 58 training steps, with the corresponding groups accounting for 35.59% of all groups. The β1\beta_1 fallback was active across only 14 steps, with the corresponding groups accounting for 3.07%. Consequently, despite β0<β1\beta_0<\beta_1, β0\beta_0 contributes a larger aggregate absolute advantage. Nevertheless, the combined absolute contribution of the two fallback terms remains smaller than that of the mixed-outcome groups, suggesting that they introduce a relatively limited perturbation to the overall training signal.

Overall, accuracy ranges from 36.81 to 40.21 across the tested configurations, and all configurations remain above the broadcasting baseline, which support robustness within the tested range.

AuthorsOpenReview ↗

Response to Reviewer 9Unp [part 3/3]

Computational overhead (Q3)

To address your question about the computational overhead of IG relative to standard GRPO, we collected the following training-time statistics.

First, we report the average total training-step time for each method:

MethodAverage step time (s/step)Relative to GRPO
GRPO2,034.51.00×
IGPO2,695.61.32×
Broadcasting8,893.54.37×
MOSAIC3,772.91.85×

However, this is highly dependent on the specific resource environment (such as the GPUs). As no fair comparison has been conducted, these figures are of limited reference value.

To characterize the IG overhead more directly, we separately report its forward-pass computation time and its share of both training-stage time (i.e., the Megatron part) and total step time:

MethodIG forward-pass time (s/step)Share of training-stage timeShare of total step time
GRPONo IG scoring——
IGPO1,244.262.8%46.2%
§3.2-only124.29.7%3.3%

In the measured runs, session-level IG (§3.2-only) requires substantially less forward-pass computation time than turn-level IG (IGPO). Together with the accuracy results, these observations support session-level shaping as a practical approach that improves task performance while limiting the computational cost of IG scoring.


Discussion of non-linear credit assignment (Q2)

Thank you for this insightful question. We find this topic very interesting and welcome the opportunity to discuss it with you.

Although both context compaction and subagent forking are multi-session settings, the former produces a linear sequence of sessions, whereas the latter introduces branching and merging. A DAG (Directed Acyclic Graph) provides a natural representation of this structure. For example, a fork turn launches one or more subagent sessions, whose outputs are subsequently returned as observations at a later turn of the main agent. Representing the main-agent’s successive execution stages as separate nodes makes these dependencies explicit.

The session-level advantage shaping signal in Section 3.2 is local and does not explicitly account for downstream interactions. Graph-based or DAG-based credit assignment could help capture contributions that become apparent only through such interactions. For example, a subagent may return a useful search lead whose immediate likelihood gain is small, while enabling the main-agent to discover decisive evidence later. Similarly, sibling outputs may provide complementary or redundant information, which is not fully captured by evaluating each output separately against the pre-fork context.

Building credit assignment on this DAG could therefore help account for these dependencies and produce more reliable session-level signals. However, developing an effective method would require careful theory design and extensive experimentation. For instance, how should credit be aggregated and distributed at nodes with multiple incoming or outgoing edges? Uniform propagation would not distinguish siblings sharing the same downstream computation, while copying the full gain to multiple upstream nodes could repeatedly count the same progress. These questions require an explicit attribution rule beyond the graph structure itself.

We view this as a promising direction for future work. DAG-based credit assignment could, in principle, provide more accurate attribution, but this would not necessarily translate into better empirical performance. So establishing its benefits would require systematic investigation and validation. Nevertheless, a DAG provides a more general representation of cross-session dependencies, and developing credit-assignment methods around this structure could advance multi-session optimization and open up valuable research opportunities.


Thank you once again for your valuable and insightful feedback. We will incorporate all the changes described above into the revised manuscript. We hope our response addresses your concerns and improves your assessment of our submission. Please feel free to leave a comment if you have any following questions.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear Reviewer 9Unp,

Thank you again for the time and effort you have devoted to reviewing our submission. Following your comments and our earlier responses, we have incorporated the additional experiments, analyses, and corresponding revisions into the manuscript. The revised PDF has now been uploaded, with all changes highlighted in red for your review.

As the author-reviewer discussion period draws to a close, we would appreciate your feedback on whether our responses and revisions have addressed your concerns. If you have any follow-up questions or comments, please feel free to share them before the discussion deadline, and we will be happy to respond.

Thank you again for your review and feedback.

Reviewer_NLZXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MOSAIC addresses reinforcement learning for agents whose trajectories span multiple context or subagent sessions. It combines outcome-conditioned session-count ranking with session-level reward shaping based on changes in ground-truth answer likelihood. On BrowseComp with Qwen3.5-4B, the method improves accuracy and reduces average session counts relative to advantage broadcasting in both context-compaction and subagent-forking settings. The paper also includes component ablations and session-utility analyses.

Critical strengths

S1. Treating compaction and subagent sessions as separate training units makes sense for these workflows. The two levels of training provide a clear alternative to giving every session the same final reward signal.

S2. The compaction setting improves accuracy while reducing average sessions relative to broadcasting. Figure 3(c)(d) also compares the two optimization levels separately, providing an initial check of their contributions.

Critical weaknesses

W1. If I have understood the fork-and-join setup correctly, all child results are returned together in Appendix C.2. I am not sure how each child then gets separate credit under Appendix B.1. The utility check in Table 2 also uses the same likelihood-gain proxy, which makes it hard to judge its link to actual task benefit.

W2. The inference-efficiency claim needs more support than session counts alone. In the subagent setting, MOSAIC uses more sessions than GRPO, although fewer than broadcasting. All policy comparisons also use Qwen3.5-4B, leaving the deployment trade-off against a stronger ready-to-use browsing agent unclear. Full inference costs would help establish the practical value of training the smaller agent.

W3. The evaluation uses one benchmark and one 4B model under a single set of session settings. Figure 3(c)(d) provides accuracy ablations, but not the corresponding session counts and inference costs for each component. Table 2 also omits the GRPO/IGPO references used in Table 1. The training curves do not fill these gaps in the accuracy–cost and session-allocation evidence.

Questions for authors

Q1. For a fork containing multiple concurrent children, what exact before/after contexts are used for each child's information gain? Do siblings receive distinct rewards, and how is the terminal main session identified and excluded? A worked example with one useful and one redundant child would resolve a central ambiguity in the credit-assignment mechanism.

Q2. How well does the likelihood-gain ranking agree with removing or withholding a session's returned information and measuring downstream task success? Even a controlled diagnostic on existing saved trajectories would help determine whether the proxy identifies useful sessions beyond predicting its own optimized score.

Q3. Could you add a representative strong API model as an inference-only reference in the same search/visit loop, with matched context and tool budgets? Reporting accuracy, monetary cost, tool calls, and latency alongside GRPO and broadcasting would clarify the deployment trade-off. Please keep this practical comparison separate from the same-backbone training controls.

Q4. Could Tables 1–2 include the §3.1-only and §3.2-only variants with accuracy, session counts, and inference cost, and include the existing GRPO/IGPO references in the session-utility comparison? This would clarify each component's accuracy–cost effect and whether the proposed allocation pattern distinguishes multi-session training from the other tested methods. Please also clarify whether compact and subagent policies are trained separately.

Q5. How does the method behave when the per-session context limit, compaction frequency, or available fork budget changes under a fixed total inference budget? Even a targeted sensitivity study would help establish that the gains reflect useful multi-session optimization rather than adaptation to one particular context-management configuration.

11 comments
AuthorsOpenReview ↗

Response to Reviewer NLZX [part 1/6]

Dear Reviewer NLZX,

We sincerely appreciate your time and effort in reviewing our submission. In this response, we aim to address the weaknesses and questions raised in your review, and provide additional experimental results, which we hope will resolve your concerns.


The mechanism of credit assignment in the subagent forking scenario (W1 & Q1)

You are correct that our implementation uses synchronous fork-and-join: children (subagent) may execute concurrently, but the main agent resumes only after all children in the current fork have terminated and their results have been collected. This joint return during rollout is separate from the per-child context construction used for information-gain scoring. We will describe this through formulation and examples.

Per-child before/after contexts. Let HtH_t denote the main-agent prefix immediately before a fork turn, and let R1,…,RKR_1,\ldots,R_K denote the children’s returned results. KK means the number of subagents created by this fork turn. During rollout, the main agent receives one aggregated observation containing all KK results. For scoring child kk, however, we use:

h_k^+ = H_t \oplus F_t \oplus O_t^{(k)},$$ where $F_t$ denotes the main-agent content of the fork turn preceding its observation, and $O_t^{(k)}$ denotes the fork observation with only child $k$’s returned information retained; the other siblings’ returned information is excluded from the scoring context. This exclusion applies only to the information gain evaluation. The actual main-agent continuation receives all returned results. Thus, siblings share the same before-context but have individually constructed after-contexts. Following Appendix B.1, we compute $\Phi_{\bar\theta}(h_k^+,y^*)$ and $\Phi_{\bar\theta}(h_k^-,y^*)$. Each child therefore receives an individually information gain value, as the foundation of distinct session-level advantage shaping signal. Although equal scores remain possible, the calculation process is child-specific. **Illustrative example**. Following your suggestion, we use a simplified example to further explain this mechanism. Consider a task in BrowseComp that searches for a film title in which both major actors have received honorary doctorates. After several turns of execution, the main agent knows that “Amitabh Bachchan” received an honorary doctorate but has not identified the film. Then a subsequent fork turn launches three children: | Child (Subagent) | Assigned subtask | Returned information | Information added to the main agent | | --- | --- | --- | --- | | A | Search for a family drama released in 1999–2009 that matches the adoption-related plot clues. | Identifies *Baghban* (2003) and relevant adoption-related plot evidence. | Supplies a candidate answer and supporting evidence. | | B | Investigate biographical clues concerning actresses who may appear in the target film. | Returns biographical facts about other actresses without establishing the film’s identity. | Adds information whose relevance remains uncertain. | | C | Investigate honorary doctorates of potential lead actors and identify a film whose two leads both satisfy this clue. | Reconfirms Bachchan’s honorary doctorate without identifying another lead or the film. | Largely repeats an already established fact. | When scoring A, neither B’s nor C’s returned information is visible; the same construction applies independently to B and C. In practice, within these three subagent sessions, A has the largest information gain value, while C has the smallest one. Because subagent C’s result provides only limited marginal contribution to the task, making C a “redundant” child as you mentioned. In contrast, subagent A’s result contains the correct final answer, so its session provides high utility. **This illustrates how a useful child and a redundant child can receive different local session-level signals despite their results being returned together**. **Terminal main session.** As specified in Appendix C.2, each rollout contains one persistent main-agent session and separate child sessions. The persistent main session produces the task-level final response and is identified by its role in the rollout, independently of child completion order. It is excluded from IG computation and receives zero local shaping, while being trained with the trajectory-level advantage. A child’s final turn is an subtask result returned to the main agent, so completing a child does not make it the terminal session.
AuthorsOpenReview ↗

Response to Reviewer NLZX [part 2/6]

The effectiveness of the session-level IG (W1 & Q2)

We agree that evaluating session utility using proxies beyond the information gain itself, would provide evidence independent of the shaping metric. Following your suggestion, we therefore conducted an additional leave-one-out (LOO) experiment.

The experimental setup is as follows:

  • Inference setting: We use the subagent-forking setting, where sibling outputs at the same fork provide clearly identifiable and separable units of returned information.
  • Model: The Qwen3.5-4B checkpoint trained with Mosaic in the subagent setting.
  • Data source: BrowseComp evaluation trajectories from the evaluation reported in the paper.
  • Sampling: We first randomly sample 300 trajectories corresponding to distinct queries, then retain those containing at least one fork turn with at least two siblings returning non-empty results.
  • Prefix replay: For each retained trajectory, we randomly select one eligible fork turn and evaluate (1) a Full branch retaining all sibling outputs and (2) a separate LOO branch for each child.
  • LOO intervention: We preserve the fork call, child identifiers, and task assignments, replacing only the target child’s result body with the fixed placeholder [Result withheld]. All other content at the replay boundary remains unchanged.
  • Replay repetitions: We independently sample three continuations for each branch.
  • Outcome evaluation: We evaluate the resulting trajectories using the same LLM-as-Judge procedure as in the paper.

This procedure yields 143 eligible trajectories, with one selected fork per trajectory: 24 forks contain two siblings, and 119 contain three. In total, we run 1,644 continuation rollouts. The Full branches are also replayed; we do not reuse their original saved trajectories.

For child kk at fork ff, we define its estimated LOO contribution as:

Δ^f,k=p^fFull−p^f,−k,\widehat{\Delta}_{f,k}=\widehat p_f^{\mathrm{Full}}-\widehat p_{f,-k},

where p^fFull\widehat p_f^{\mathrm{Full}} and p^f,−k\widehat p_{f,-k} are the empirical success rates across the three continuations of the Full branch and the branch withholding child kk’s result, respectively. A larger Δ^f,k\widehat{\Delta}_{f,k} indicates a greater loss in downstream success when that child’s information is withheld.

We then compare the following four conditions to examine whether the original session-level IG identifies children whose removal causes a larger performance drop:

BaselineAvg. Accuracy (%)Drop from Full (pp)
Full25.64—
Remove lowest-IG child24.011.63
Remove a random child23.272.37
Remove highest-IG child19.586.06

A key diagnostic in this table is the difference between removing the lowest-IG child and removing the highest-IG child:

Γ=1N∑f=1N(p^f,−kfmin⁡−p^f,−kfmax⁡),\Gamma=\frac{1}{N}\sum_{f=1}^{N}\left(\widehat p_{f,-k_f^{\min}}-\widehat p_{f,-k_f^{\max}}\right),

where N=143N=143, and kfmin⁡k_f^{\min} and kfmax⁡k_f^{\max} denote the children with the lowest and highest IG at fork ff, respectively. Equivalently, Γ\Gamma measures how much larger the average performance drop is when withholding the highest-IG child rather than the lowest-IG child. Finally, we obtain Γ=4.43\Gamma=4.43 percentage points, with a 95% bootstrap confidence interval of [2.56,6.06][2.56,6.06] percentage points, based on 5,000 resamples. The interval lies entirely above zero, supporting the finding that withholding higher-IG children causes a larger average loss in downstream task success.

Additionally, we also measure ranking agreement within each fork. For a sibling pair (a,b)(a,b) satisfying IGf,a>IGf,bIG_{f,a}>IG_{f,b}, we classify the pair as:

  • Concordant if Δ^f,a>Δ^f,b\widehat{\Delta}_{f,a}>\widehat{\Delta}_{f,b}: removing the higher-IG child causes a larger performance drop.
  • Discordant if Δ^f,a<Δ^f,b\widehat{\Delta}_{f,a}<\widehat{\Delta}_{f,b}.
  • Tied if Δ^f,a=Δ^f,b\widehat{\Delta}_{f,a}=\widehat{\Delta}_{f,b}.

Agreement is computed over pairs with non-tied LOO contributions:

Agreement=concordant pairsconcordant pairs+discordant pairs.\mathrm{Agreement}=\frac{\text{concordant pairs}}{\text{concordant pairs}+\text{discordant pairs}}.

The results are as follows:

Ranking diagnosticResult
Number of evaluated forks / children143 / 405
Number of sibling pairs with unequal IG381
Pairwise agreement among non-tied LOO pairs80.00% (95% CI: [69.55%, 84.44%])
Fraction of LOO-tied pairs among pairs with unequal IG31.76%

These results provide further evidence that session-level IG is informative about relative session utility. From a computational perspective, session-level IG is substantially cheaper to obtain: it requires two forward-only likelihood evaluations per eligible session, without sampling additional continuation rollouts. Taken together, these diagnostics provide outcome-based evidence, independent of the likelihood-gain metric, that IG can identify useful sessions, supporting its use as a practical signal for session-level credit assignment.

AuthorsOpenReview ↗

Response to Reviewer NLZX [part 3/6]

The inference-efficiency claim (W2)

Thank you for raising this point. We would first like to clarify that we do not claim a general inference-efficiency advantage over all baselines. In particular, we do not claim lower inference cost than single-session-trained baselines such as GRPO.

Table 1 reports accuracy and average session counts. The associated phrase “lower execution cost” in §4.2 refers specifically to the comparison with Broadcasting. Both Mosaic and Broadcasting optimize policies over multi-session trajectories, and their session-count comparison is intended to illustrate Mosaic’s design objective: allowing sufficient exploration or execution when needed while discouraging unnecessary session usage.

We also agree that session count alone does not establish lower execution cost, so we did not intend this statement to every baseline. Our primary objective is to improve task performance, and one of Mosaic’s motivations is precisely that sufficiently difficult tasks may require more sessions to solve. Accordingly, the observed reduction in session count relative to Broadcasting is interpreted within that specific comparison.


The comparison with a strong API model (W2 & Q3)

To provide an inference-only reference using a strong API model, we additionally evaluated DeepSeek-v4.1-flash-1017 on BrowseComp. The results are shown below:

Inference modeAccuracyAvg. session countTotal search callsTotal visit calls
Single67.61%1.0034,9877,046
Compact84.12%4.0189,02425,121
Subagent70.62%2.99106,43632,758

These results demonstrate the strong performance of this API model and substantial benefits from multi-session inference, particularly context compaction. This observation is consistent with one of Mosaic’s motivations: difficult tasks may require additional sessions to support sufficient exploration and execution.

To be honest, however, the substantial capability gap between this API model and our 4B backbone makes their performance difference difficult to interpret as a measure of Mosaic’s algorithmic contribution. Our original submission focused on multi-session optimization, using comparisons with the same backbone to isolate improvements attributable to the training method. Our objective was to study this optimization problem, rather than to achieve SOTA performance on a particular benchmark. We therefore view the strong API model as a useful deployment reference, separate from the controlled comparisons that evaluate Mosaic itself.

We also observed an issue that warrants care when interpreting these results: because search and visit tools access the live web, models may encounter publicly available benchmark questions and answers. In our supplementary evaluation of DeepSeek-v4.1-flash-1017, we identified cases of direct benchmark-answer retrieval. We share one example below:

  • Query: “Between 1990 and 1994 (Inclusive), what teams played in a soccer match with a Brazilian referee had four yellow cards, two for each team where three of the total four were not issued during the first half, and four substitutions, one of which was for an injury in the first 25 minutes of the match.”
  • Reference answer: Ireland v Romania.
  • Model’s final answer: Republic of Ireland v Romania.
  • Judge result: Correct, with a reward of 1.0.
  • Observed hack behavior: Starting at turn 6, the model shifted toward searching for the original question itself. At turn 11, it explicitly located the answer in the BrowseComp dataset. During turns 13–15, it used the retrieved answer to verify supporting details before producing its final response.

We also identified multiple similar cases. Such behavior can inflate the benchmark score as a measure of the intended research capability, since a successful answer may result from retrieving exposed benchmark answers. Therefore, we report the API-model results as an inference-only reference while retaining the same-backbone comparisons as the primary evidence for Mosaic’s algorithmic contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.