acceptodds
Withdrawn from ICLR 2027

Post-Training with Communication

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study how to post-train language models on tasks whose evidence no single model holds. We propose post-training with communication, a training paradigm in which identical copies of one model, the agents, each hold a different shard of the context, exchange free-text messages on a shared board, and then answer. No agent has a role, and no agent holds all the evidence, so the team earns full reward only by communicating. Training uses only the verifier of single-model RLVR, with no reward model and no hand-written collaboration reward. To further improve performance, we add two counterfactual credit terms, which the same verifier computes. We find that the trained team scales zero-shot. Trained with only four agents, it keeps most of its accuracy in teams of 32 and remains effective with 64, on inputs several times longer than the model's context window. At these scales, on long-context memory and multi-hop question answering, the team matches or outperforms the same model with a YaRN-extended context window or run as map-reduce. The team agent also stays a generalist: it has no fixed role, and communication training causes no measurable loss of general reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM post-training, GRPO, Multi-agent collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yFe5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the scenario in which the evidence required to solve a task is distributed across multiple context shards, such that no single model instance observes all relevant information. The paper proposes post-training with communication, where identical, parameter-shared, role-free agents each access one shard, exchange free-text messages through a shared board, and then generate answers using both their local context and the board. The method is trained with verifier-based RL and introduces two credit-assignment terms: answer credit, which encourages agents to benefit from the board, and message credit, which aims to reward messages that improve other agents’ answers. Experiments on reduction-max, HotpotQA and LongMemEval-mid show that a team trained at a small team size can be evaluated at larger team sizes.

Critical strengths
  • The paper targets a relatively underexplored setting in which the evidence required for answering a question is distributed across agents. This setting is different from existing setting, such as multi-agent debate, in which each agent observes the full context.
  • The proposed method is clear: identical, parameter-shared, role-free agents each observe only a private context shard and communicate through a shared free-text board. This design makes it clearer whether the performance gain comes from agents sharing their different pieces of information, rather than from manually assigned roles or using different agent models.
Critical weaknesses
  • It seems that the message-credit proxy does not isolate the contribution of an individual message. Instead of comparing peers' performance before and after removing only the sender's message, it compares different rollouts where all agents' messages may differ. The credit may therefore reflect other messages or overall rollout quality, rather than the sender's causal contribution.
  • Experiments are limited in scope. All main results use Qwen3-8B-Base and are evaluated on the same task constructions used for training. For example, the model trained on HotpotQA is evaluated only on HotpotQA, so it remains unclear whether the method learns a transferable ability to communicate and pool distributed evidence, rather than adapting to a specific task format. For example, training on HotpotQA should be also evaluated on some factual QA benchmark, such as TriviaQA, NQ, SimpleQA, ChineseSimpleQA.
  • The evaluated application scenarios are limited. Although the proposed framework could potentially apply to many settings with distributed information, the experiments cover only synthetic aggregation, multi-hop QA, and one long-context memory setting. In particular, for the long-context application, the paper compares mainly with context extension and map-reduce, but not with retrieval- or memory-based systems such as MemAgent. These methods are particularly relevant to the LongMemEval setting and represent a different approach to handling long-context information.
Questions for authors
  • Could the authors provide additional ablations or analyses to validate the cross-rollout message-credit proxy?
  • For the long-context memory setting, could the authors discuss or compare against retrieval- or memory-based approaches?
  • Could the authors improve the paper's presentation and layout? For example, Page 8 contains a substantial amount of unused space. Using the space more effectively could improve readability.
Reviewer_PCNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains identical, role-free copies of a language model to solve tasks from distributed context through a shared message board. Training uses verifier-based GRPO with additional message and answer credit. Experiments on numeric aggregation, HotpotQA, and LongMemEval-mid report zero-shot transfer to larger teams and no measurable degradation on selected general-reasoning benchmarks.

Critical strengths
  1. The shared-policy, split-context formulation provides a clear setting for studying evidence sharing rather than collaboration among agents that already see the full input.

  2. Evaluating four-agent-trained policies with up to 64 agents is an interesting generalization test. Including YaRN and map-reduce makes the comparison more informative than relying only on a truncated reader.

  3. Three-seed credit ablations reveal task-dependent effects, and the oracle-board control usefully investigates the remaining communication bottleneck. The paper openly reports instability and negative results.

Critical weaknesses
  1. The credit-assignment contribution is not sufficiently isolated. Vanilla team GRPO uses each agent’s own reward, whereas the message term introduces peer rewards through comparisons across entirely different boards. Although the paper acknowledges that this proxy does not estimate message removal, the ablation cannot establish whether gains require counterfactual credit or simply rewarding messages for team success. A normalized shared-team-reward baseline is needed.

  2. The long-context baselines reuse a full-information adapter without training it for YaRN or map-reduce. These are useful deployment comparisons, but they do not isolate the benefit of the proposed protocol from training–deployment alignment. A map-reduce baseline trained on the same split-context tasks would strengthen the central comparison.

  3. The main scaling results use only seed 0, which the paper identifies as one of the successful redmax runs despite substantial training instability. The LongMemEval-mid advantages also rely on only 71 questions without paired confidence intervals. Consequently, the robustness of the headline scaling result remains uncertain.

Questions for authors
  1. How does the message proxy compare with normalized shared-team reward and regenerated-peer counterfactual credit under the same training recipe?

  2. Could map-reduce be trained at G=4G=4 with the same data and verifier, then evaluated zero-shot at larger sizes? Please report total training and inference costs.

  3. Do scaling gains persist across training seeds, with strictly disjoint redmax evaluation lists and randomized HotpotQA paragraph positions? Appendix G.1 reports eight overlapping redmax lists and gold-first paragraph ordering.

  4. How much do trained teams lose when the board is removed or its messages are shuffled, particularly on HotpotQA? This would more directly measure reliance on communication.

Reviewer_rUY8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the following setup. They consider G identical copies of a model, give each of them a slice of the text, let them communicate with a bulletin board, and train them all with RLVR. The reward given to each agent is how much its message helped the rest of the agents score higher. Experiments show that a policy trained on 4 agents actually scales zero-shot up to 64 agents. It also beats single-model baselines.

Critical strengths
  • This is an interesting conceptual idea which if successful, can reduce the need for gigantic long-context models and instead use smaller models that communicate. The zero shot scaling is a nice result as well.
Critical weaknesses
  • The main concern I have is inference time. Although the goal was the bypass context window limitations, inference time with 64 agents requires potentially even larger computational costs and latency. This issue is not addressed in the paper and without some sense of the method's practicality, I don't think it is easy to judge this paper. (My hunch is that this can be even more expensive in terms of $ than a single long context model).
  • The paper uses a cross rollout proxy to compute the reward. The ideal goal is to understand how much Agent k's teammates improved due to Agent k's message, compared to no message? The authors use a baseline of the other agents to compute this. I don't believe this gives the correct reward since it is not computing the counterfactual. Do the authors have any experiments on this in an idealized setting where you do have the true counterfactual?
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.