acceptodds
Under review as a conference paper at ICLR 2027

Collaborative Learning and Reasoning over Model Slices

Abstract

Large language models (LLMs) excel across diverse tasks, benefiting from strong knowledge priors, flexible in-context reasoning, and scalable test-time computation. Despite these strengths, single-model reasoning remains vulnerable to error propagation, as later steps depend on the model's earlier outputs, leaving reasoning blind spots difficult to independently reassess. To address this limitation, we develop *collaborative learning*, encompassing the construction of heterogeneous collaborators and the learning of effective cross-model collaboration. Specifically, *model slicing* replaces a large model with smaller, heterogeneous models under a comparable parameter budget, enabling prior reasoning to be reassessed across distinct model parameterizations. *Pyramid supervision* combines protocol distillation with step-level supervision to refine reasoning and interaction decisions throughout collaboration. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. The trained heterogeneous models achieve comparable or better task performance than the original large model and established multi-agent system and multi-agent reinforcement learning baselines, while achieving higher inference efficiency than the evaluated multi-agent frameworks and exhibiting differentiated collaboration patterns across tasks. These findings show that learned collaboration turns model heterogeneity into complementary reasoning capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Collaborative Reasoning, Multi-Agent Systems, Model Slicing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DmpaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a collaborative learning framework that replaces a single large language model with several smaller, heterogeneous models under a comparable parameter budget. The method combines protocol distillation and step-level supervision to train models to reason collaboratively, decide when to hand off tasks, and revise previous outputs. Experiments across five benchmarks show improvements over single-model and multi-agent baselines.

Critical strengths
  1. The experimental results are promising, with consistent improvements across multiple tasks and comparisons against various multi-agent baselines.

  2. The ablation studies and analysis of collaboration patterns provide useful evidence for the effectiveness of the proposed training framework.

Critical weaknesses
  1. The technical novelty is limited. The proposed framework mainly combines existing techniques, including heterogeneous multi-agent collaboration, knowledge distillation, and step-level reward optimization. It is unclear what fundamentally new methodological contribution is introduced beyond their integration.

  2. The efficiency evaluation relies mainly on model invocations and parameter-weighted token counts. Actual inference latency and computational costs are not sufficiently evaluated.

  3. The experiments do not fully isolate the benefits of model heterogeneity from differences in model capacity and training. Stronger controlled comparisons would help establish whether heterogeneous collaboration is essential.

Questions for authors

How does the proposed framework compare with a single large model under the same inference compute budget?

1 comment
AuthorsOpenReview ↗

Thank You for Your Feedback — We Look Forward to Further Discussion

Dear Reviewer Dmpa,

Thank you again for your thoughtful comments. We wanted to follow up on your questions about what the models learn through collaboration and how CL compares with a single large model at comparable inference cost.

Methodological contribution (Q1). Under the shared protocol, each collaborator learns both how to advance the reasoning and when to hand off or stop. After training, the 4B model more often revises and hands off answers on MuSiQue, while it more often provides the final answer on GSM-Hard.

Model capacity and heterogeneity (Q3). We added a homogeneous three-8B control trained under the same settings as CL. With the largest collaborator matched in size, CL scores higher on eight of ten metrics while using 42.92% fewer total parameters.

Inference cost and single-model comparison (Q2, Q4). We reported TFLOPs for both input processing and output generation. On GSM-Hard, CL and the SE-RL 14B baseline use 63.67 and 64.85 TFLOPs, respectively. At this comparable cost, CL achieves 3.79 percentage points higher accuracy and 1.48 points higher F1.

We would be grateful if you could consider these results in your final assessment. If any aspect of the method or the compute comparison merits further clarification, we would welcome the opportunity to discuss it.

Warm regards,
The Authors

Reviewer_43BNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a pool of Qwen3-1.7B, 4B, and 8B models, totaling 13.7B parameters, to collaborate through continue, handoff, and terminate actions. "Model slicing" means replacing a 14B model with independently pretrained smaller checkpoints, rather than partitioning its weights. Protocol distillation learns from trajectories generated by three 8B copies, with model-slot identities permuted across demonstrations. A 14B inspector then scores reasoning and interaction decisions, and each slice is fine-tuned on fixed offline trajectories with a signed reward combining step feedback and final task performance. Evaluation covers MuSiQue, GSM-Hard, MATH, eight-language MultiPL-E, and Conifer. The system improves all ten reported metrics over the unadapted 14B model, wins eight over SE-RL, and uses fewer calls than the tested multi-agent configurations. Ablations and trajectory analyses examine correction, answer retention, redundancy, and task-dependent interaction patterns.

Critical strengths
  1. Consistent system-level gains at a comparable inference-time parameter budget. The 13.7B pool outperforms the unadapted Qwen3-14B on all ten metrics (mean +2.38 points; e.g., accuracy 42.37 vs 39.55 on MuSiQue, 78.40 vs 74.60 on MATH) and SE-RL on eight of ten, though SE-RL leads on MultiPL-E. Disjoint train–evaluation splits support the held-out comparison. This is the paper's core empirical claim, and it appears credible for the tested setup.
  2. Trajectory analyses indicate that training changes how interactions are used. On MuSiQue, removing protocol distillation raises first-answer EM (21.02% →\rightarrow 25.24%) yet cuts initial errors corrected by the final answer from 594 to 171, consistent with a benefit from downstream coordination rather than first-answer quality. An illustrative C++ case shows a recipient repairing a failing candidate by adding an omitted validity check. This makes learned coordination a plausible contribution, though heterogeneity's role is not isolated.
Critical weaknesses
  1. Heterogeneity is not cleanly isolated. CL pools 1.7B/4B/8B models while Homo. CL uses three 4B models, so the 6.02-point gap may reflect the stronger top collaborator rather than model diversity alone.
  2. Step-level supervision is hard to separate from gold-answer feedback. The inspector sees the gold answers, and on GSM-Hard/MATH incorrect candidates cannot receive positive scores, so step scores may largely encode answer correctness. The "w/o SLS" ablation shows the scores help, but not that they capture process quality beyond answer-conditioned scoring.
  3. Applicability to interactive agent settings is untested. Although the method learns handoff and termination, all benchmarks are static, single-turn tasks (QA, math, code, constrained generation). Nothing involves tool use or stateful environments, where coordination must account for action consequences and external feedback, so claims about general collaboration should be scoped accordingly.
Questions for authors
  1. Have you evaluated CL on agent benchmarks involving tool use or stateful interaction? Do its gains persist when collaboration must handle external feedback and action consequences?
  2. How does learned model collaboration differ from, and complement, harness-level self-improvement? What advantages does updating the collaborators offer over evolving prompts, routing, or workflows around frozen models?
1 comment
AuthorsOpenReview ↗

Thank You for Your Careful Review — We Welcome Further Discussion

Dear Reviewer 43BN,

Thank you again for your careful reading. Your questions about the largest collaborator’s capacity and the information captured by step-level scores prompted two additional analyses.

Model heterogeneity (Q1). We trained a homogeneous system of three Qwen3-8B models under the same configuration as CL. With the largest collaborator fixed at 8B, CL scores higher on eight of ten metrics across four of five datasets, using 13.7B total parameters compared with 24B for the homogeneous system.

Step-level supervision (Q2). We analyzed Inspector scores separately for correct and incorrect candidate answers. Scores vary within both groups on all five datasets. This also holds for incorrect candidates on GSM-Hard and MATH, where the scoring rule caps their scores at zero.

Evaluation scope and harness-level improvement (Q3, Q4). We clarified why our evaluation focuses on collaborative reasoning at the model level and how training the collaborators can complement external workflow optimization. In our direct comparison with AFlow, CL scores higher on nine of ten metrics.

The full results are in our responses above. We would particularly welcome your thoughts on whether the three-8B control and the conditional score analysis address the questions you raised.

Warm regards,
The Authors

Reviewer_2TcgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a collaborative learning framework in which multiple smaller, heterogeneous models jointly perform reasoning tasks under a comparable parameter budget to a single large model. The framework comprises two components: (1) model slicing, which replaces a single large model (e.g., 14B) with several smaller models (e.g., 1.7B + 4B + 8B); and (2) pyramid supervision, which combines protocol distillation with step-level reinforcement learning to train cross-model scheduling and interaction decisions. The approach is evaluated on four task categories, and reports performance comparable to or better than the original large model and established multi-agent baselines, while claiming higher inference efficiency.

Critical strengths

The paper presents a well-motivated and effective training strategy that encourages genuine inter-model cooperation rather than simple model ensemble.

The reported results, showing that a heterogeneous ensemble can match or exceed a single large model, are encouraging.

Critical weaknesses

The inference-efficiency comparison is not well supported. Comparing models by total parameter count alone is insufficient, as inter-model context passing introduces substantial overhead that depends heavily on KV-cache management and cross-model communication. Frequent state transfer between separate model instances is likely far more costly than equivalent computation within a single model, which could undermine the claimed efficiency advantage. The authors should provide a more careful cost analysis that accounts for these factors.

The term "model slicing" is potentially misleading. Slicing typically implies partitioning a single model into sub-networks that are recombined, whereas the proposed method trains and operates on independently parameterized models. The terminology should be revised or clarified to avoid confusion.

The training signal is derived from the largest collaborating model (e.g., 8B), which itself is weaker than the 14B model it is meant to replace. It is therefore unclear whether this signal is strong enough to guide effective scheduling. Moreover, the cooperation pattern observed among homogeneous 8B models may differ substantially from that among heterogeneous models of varying sizes, raising questions about the transferability of the training signal to the target ensemble. The authors should discuss or empirically validate this concern.

Several baselines lack clear comparability. In particular, it is not specified whether SE-RL or the other baselines were trained under the same data and computational budget. Without matching training conditions, the comparison may be confounded by differences in resources rather than in method.

Questions for authors

The paper would benefit from an analysis of the actual division of labor among the collaborating models. For instance, what functional role does the smallest model (e.g., 1.7B) play in the pipeline? Providing per-model contributions would help substantiate the claim that the models exhibit differentiated and complementary behaviors.

1 comment
AuthorsOpenReview ↗

Thank You for Your Careful Review — We Welcome Further Discussion

Dear Reviewer 2Tcg,

Thank you again for your thoughtful comments. Your questions about the cost of passing context between models and the role of the smallest collaborator motivated several additions to our response.

Inference cost and training comparability (Q1, Q4). Our TFLOPs analysis includes the computation each receiving model performs when it processes handoff context. Relative to the lowest-cost of the four multi-agent baselines on each benchmark, CL uses 25.05% fewer TFLOPs on average across the five benchmarks. We also clarified which baselines use the same training splits and number of policy-optimization epochs as CL.

Model slicing and protocol transfer (Q2, Q3). We clarified that “model slicing” refers to replacing a large model with independently parameterized smaller models. We also explained how homogeneous 8B demonstrations initialize the shared protocol. In the target heterogeneous system, removing protocol distillation lowers performance by 8.85 percentage points on average across the ten metrics.

Individual model contributions (Q5). The 1.7B model provides 53.9% of the final answers on Conifer. Appendix G.4 also illustrates its role in verification: after the 4B model introduces an arithmetic error, the 1.7B model rechecks the problem conditions and restores the correct answer.

The full comparisons and trajectory examples are in our responses above. We would appreciate hearing whether these additions address your concerns, and we would be glad to discuss any remaining questions.

Warm regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Additional Evidence

Dear Reviewers and Area Chair,

Thank you for your careful assessment and valuable suggestions. We have responded to each concern and provided additional experiments and analyses. As the discussion period draws to a close, we would like to summarize the paper’s main contributions and the additional evidence for your consideration in the final assessment.


1. Learning collaborative reasoning and a task-dependent division of labor through model training. Our framework jointly learns heterogeneous models’ reasoning and collaboration decisions under a shared protocol. All models share the same prompt and action space, without predefined functional roles. Through protocol distillation and step-level supervision, they learn to build on preceding reasoning, revise answers, hand off tasks, and terminate collaboration. This allows the division of labor to emerge according to the task and reasoning progress.


2. Improving performance under a comparable parameter budget, with a stronger homogeneous control supporting the heterogeneous configuration. With 13.7B total parameters, CL outperforms the unadapted Qwen3-14B on all ten metrics by an average of 2.38 percentage points, and exceeds SE-RL on eight metrics. To address the concern about model capacity, we additionally trained a homogeneous system of three Qwen3-8B models under the same training configuration. With both systems having an 8B model as their largest collaborator, CL achieves higher scores on eight of ten metrics across four of five datasets, with an average improvement of 1.63 percentage points, while using 42.92% fewer total parameters. These results further support the effectiveness of the heterogeneous configuration.


3. Ablations and trajectory analyses support the role of collaborative training. Removing protocol distillation and step-level supervision reduces performance by an average of 8.85 and 5.41 percentage points, respectively, across the ten metrics. Per-model analyses also reveal task-dependent specialization: the 4B model increasingly revises answers and hands off on MuSiQue, while more often providing final answers on GSM-Hard. The 1.7B model also contributes to verification and final-answer generation, with a case study showing it correcting an error introduced by a larger collaborator. Together with analyses of error correction, reduced repetition, and correct-answer retention, these findings provide concrete evidence of complementary behaviors among the models.


4. Extending the computational-cost analysis to include handoff-context processing. The supplementary TFLOPs statistics cover both input processing and output generation, including the computation required for receiving models to process handoff context. Relative to the lowest-cost baseline among MAD, AgentVerse, GPTSwarm, and AFlow on each dataset, CL achieves an average relative TFLOPs reduction of 25.05% across all five datasets. This is consistent with the efficiency trends relative to multi-agent baselines reported in the paper.


The complete tables, analyses, and corresponding manuscript references are provided in our responses above. We hope this summary helps clarify the methodological contributions and empirical findings, and respectfully ask that the additional evidence be considered in the final assessment.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.