acceptodds
Under review as a conference paper at ICLR 2027

TACIT: Internalizing Collaboration via Training-Time Experience Transfer

Abstract

Multi-agent reasoning with large language models (LLMs) has unleashed strong potential for complex tasks. However, existing paradigms, such as multi-round debate, rely heavily on verbose and transient inference-time communication, incurring substantial token overhead and sometimes redundant interactions that undermine performance. Consequently, communication remains a recurring runtime patch rather than internalized competence among agents. To this end, we introduce **TACIT** (**T**raining-time multi-**A**gent **C**ollaboration via **I**mplicit experience **T**ransfer), a framework that enables LLMs to *learn from the exemplary when confronted with the flawed* through two-stage preference optimization. Specifically, LLM-based agents first develop *self-verification* by directly assessing their own answers during generation. This endogenous confidence signal then *gates implicit collaboration* in the second stage: uncertain agents acquire superior peer trajectories as confirmed experience, thereby constructing agent-specific preference data for co-training. At inference, TACIT requires *no explicit communication*, relying only on lightweight confidence-driven routing and answer aggregation. Experiments across diverse reasoning benchmarks show that TACIT elicits reliable and general self-verification for genuine collaboration, surpassing representative baselines in task performance. Instead of test-time indiscriminate dialogue, targeted experience transfer makes TACIT a truly *tacit* paradigm — *collaboration, once learned, need not be spoken*.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent reasoning, training-time collaboration, self-verification, communication-free inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hTbMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TACIT relocates multi-agent collaboration from inference-time dialogue to training-time experience transfer. Stage I uses preference optimization to teach agents to append confirmation or uncertainty markers. Stage II constructs agent-specific preference pairs by combining an incorrect or insufficiently confident attempt with a correct, self-confirmed peer trajectory. At inference, selected agents generate independently, optionally revise within the same generation, and contribute to confidence-weighted voting. The manuscript evaluates three backbones on mathematical reasoning and studies learned self-verification and its transfer to code and symbolic reasoning tasks. Tables 1–2 report the highest accuracy in five of eight settings and the second highest in the remaining three. The results are promising, but ambiguities in the implemented method and evaluation protocol, together with incomplete isolation of the gating mechanism, limit the strength of the conclusions.

Critical strengths
  1. Practical training-time collaboration. TACIT transfers peer experience during training, reducing the need for costly inference-time communication while targeting agents' individual weaknesses.
  2. Promising empirical results. TACIT achieves the best performance in five of eight settings across three backbones. Self-verification experiments further demonstrate learnable confidence signals and partial cross-domain generalization.
Critical weaknesses

1.Implementation and reproducibility. The router differs between Section 3.4 and Appendix A.2, while Stage II preference construction differs between the main text, Algorithm 2, and code. The relationship between single-generation inference and multi-sample evaluation also remains unclear. 2. Insufficient causal evidence. Table 3 does not isolate confidence gating, since removing Stage I also affects initialization, routing, and voting. Moreover, the claimed efficiency gains lack measured token costs and latency.

Questions for authors
  1. Can the authors clarify the actual router, preference construction, and evaluation protocols used to obtain the reported results?
  2. Can the authors isolate the effect of gating while holding other components fixed, and provide measured token costs and latency?
3 comments

We apologize for the delayed response, as we have devoted time to conducting the suggested experiments as thoroughly as possible within the available period and computing resources.

> Re W1 & Q1: Implementation and reproducibility.

We thank the reviewer for highlighting these implementation ambiguities. We have clarified the following details to ensure consistency between the methodology, pseudocode, and experimental configuration.

  1. Router specification.

The router is implemented as multi-label classification, consistent with Section 3.4. Each agent receives an independent binary competence label, and the router is trained using binary cross-entropy, allowing multiple agents to be competent on the same question. The multi-class descriptions in Appendix A.2 and Algorithm 2 were inadvertent documentation errors. We have corrected the corresponding label construction, loss function, and pseudocode to accurately reflect the intended implementation.

  1. Preference construction.

Section 3.3 specifies two distinct cases:

  • Incorrect responses: The rejected sequence is ri⊕[confirm]r_i \oplus [\mathrm{confirm}], discouraging unjustified confidence.
  • Correct but low-confidence responses: The rejected sequence is ri⊕[uncertain]r_i \oplus [\mathrm{uncertain}], avoiding penalization of a correct answer.

Algorithm 2 inadvertently omitted this conditional distinction, causing an inconsistency with Section 3.3. We have corrected the pseudocode to explicitly distinguish these two cases and ensure consistency with the actual preference construction.

  1. Single-generation inference vs. multi-sample evaluation.

These refer to different levels of the evaluation procedure. At inference, TACIT routes each question to the top-3 agents among five, with each selected agent independently generating one response per system invocation, followed by confidence-weighted aggregation. No inter-agent communication occurs. In contrast, multi-sample evaluation refers to generating multiple independent samples for benchmark-level performance estimation or sampling-based baselines, rather than additional communication rounds within a single TACIT invocation.

  1. Evaluation protocols.

We have clarified the benchmark-specific evaluation settings in the Appendix, including model-specific tokenizer chat templates, prompting, decoding parameters, generation budgets, and answer-verification procedures. Native thinking mode is disabled unless otherwise specified.

We perform zero-shot chain-of-thought evaluation using the backbone’s chat template. GSM8K and MATH-500 use a maximum generation length of 2,048, both pass@1 and majority-vote (self-consistency) accuracy over eight trajectories are recorded. AIME 2024 and AIME 2025 use the same sampling parameters with a maximum of 6144 new tokens and 32 trajectories per problem. MBPP and BBH are evaluated greedily with one trajectory and a maximum length of 2,048. We report execution-based pass@1 for MBPP and macro-averaged accuracy across BBH subtasks.

Table 1. Task Evaluation Configurations.

BenchmarkTemp.Top-pMax tokensSamples
GSM8K0.70.92,0488
MATH-5000.70.92,0488
AIME 20240.70.96,14432
AIME 20250.70.96,14432
MBPPGreedy–2,0481
BBHGreedy–2,0481

> Re W2 & Q2: The claimed efficiency gains lack measured token costs and latency.

We agree that efficiency claims should be supported by explicit computational measurements. To address this concern, we conducted additional evaluations of inference token consumption, end-to-end latency, and throughput across all seven methods on MATH-500 (500 questions) with Qwen3-4B checkpoints trained on MATH. TACIT employs top-3 agent routing without explicit inter-agent communication at inference.

As shown in Table 2, TACIT achieves the highest accuracy (68.28%) while substantially reducing inference costs compared with explicit multi-agent collaboration methods. Specifically, compared with Multi-Agent Debate, TACIT achieves 3.56 percentage points higher accuracy with 58.5% fewer tokens and 43.8% shorter execution time, while improving throughput by 1.78×. Compared with iMAD, TACIT improves accuracy by 2.97 percentage points, reduces token consumption by 21.3% and execution time by 16.9%, and achieves 1.21× higher throughput.

We acknowledge that lightweight single-agent methods incur lower inference costs, but TACIT delivers substantially stronger reasoning performance. These results demonstrate a favorable accuracy–efficiency trade-off, particularly against inference-time multi-agent collaboration, supporting our central motivation: internalizing collaborative experience during training can preserve the benefits of multi-agent reasoning while substantially reducing recurring communication overhead at inference.

Table 2. Inference efficiency comparison on MATH-500 with Qwen3-4B.

MethodAccuracy (%) ↑Total Tokens / Q ↓Latency (s) ↓Throughput (Q/s) ↑
Chain-of-Thought [1]52.40643.718223.4282.238
Self-Consistency [2]56.051281.293246.8052.026
Multi-Agent Debate [3]64.725200.4732032.7950.246
iMAD [4]65.312742.5681375.5240.363
ReVISE [5]57.981231.898868.4290.576
S2R [6]63.681629.7671022.4590.489
TACIT (Ours)68.282157.8421142.5730.438

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.

[2] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023.

[3] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, 2024.

[4] Wei Fan, JinYi Yoon, and Bo Ji. imad: intelligent multi-agent debate for efficient and accurate llm inference. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence, pp. 29403–29411. AAAI Press, 2026. ISBN 978-1-57735-906-7.

[5] Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. Revise: Learning to refine at test-time via intrinsic self-verification. In International Conference on Machine Learning, pp. 33616–33634. PMLR, 2025.

[6] Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching llms to self-verify and self-correct via reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22632–22654, 2025.

> Re W2 & Q2: Gating isolation.

We thank the reviewer for this insightful observation. We clarify that the original Table 3 presents a stage-level factorial ablation, intended to assess the contributions of both training stages to the overall framework, rather than isolate the effect of confidence gating alone. We agree that removing Stage I also affects policy initialization and downstream routing and aggregation, making a more fine-grained analysis necessary.

To address this concern, we conducted an additional controlled gating ablation on MATH-500 with Qwen3-4B. Specifically, we use identical Stage I checkpoints and vary only the Stage II training sample selection strategy (with vs. without confidence gating), while keeping the remaining training configurations fixed. We further evaluate both variants under two identical inference settings: (i) uniform routing with majority voting and (ii) competence-based routing with confidence-weighted voting.

ConfigurationUniform Routing + Majority VotingCompetence Routing + Confidence-Weighted Voting
No Gating64.1365.95
Confidence Gating65.7468.28

The results consistently demonstrate the effectiveness of confidence-guided transfer. Confidence gating improves accuracy by 1.61 percentage points under uniform routing and by 2.33 percentage points under competence-based routing, confirming its contribution independently of Stage I initialization and the choice of inference aggregation strategy.

Importantly, confidence gating also achieves a 13.8% training acceleration, demonstrating benefits beyond task accuracy. These findings support our central hypothesis that learned self-verification signals help identify more valuable transfer opportunities, enabling agents to focus on informative peer experiences rather than applying supervision indiscriminately.

Overall, this controlled ablation provides more direct evidence that self-verification is not merely an auxiliary capability, but an effective mechanism for improving both the effectiveness and efficiency of cross-agent experience transfer.

Reviewer_AdQ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TACIT, a two-stage DPO framework for internalizing multi-agent experience into separately trained LLM agents. Stage I trains self-verification markers, while Stage II uses confidence and correctness to select peer trajectories for agent-specific correction training. At inference, a router selects several agents whose independent outputs are combined through confidence-weighted voting without inter-agent communication.

Critical strengths
  1. The self-verification signal is clean and lightweight. Stage I preference pairs share an identical reasoning trajectory and differ only in the terminal marker, isolating verification from reasoning-style changes. Reusing reserved in-vocabulary tokens also avoids vocabulary expansion and preserves reference-policy support under DPO.
  2. The empirical analysis covers several relevant dimensions. The paper evaluates three backbones, multiple mathematical benchmarks, cross-domain self-verification, and a factorial ablation of the two training stages, with generally strong reported performance
Critical weaknesses
  1. The evidence does not adequately support the paper’s framing of collaboration, co-evolution, and efficiency. TACIT’s agents generate independently, while “collaboration” consists of offline, ground-truth-filtered peer-trajectory construction followed by separate training, making the method closer to modified rejection-sampling distillation than interactive MAS co-evolution. The paper explicitly claims improved efficiency, yet the experiments do not report the actual computation or cost relative to the baselines. Separately, the evaluation lacks a basic compute-matched rejection-sampling fine-tuning baseline. Moreover, Table 3 shows that random-gated Stage II already improves accuracy from 52.40% to 63.00%, suggesting that much of the gain comes from injecting correct peer trajectories rather than the emphasized self-verification gating.
  2. The experimental validity and method specification contain major unresolved discrepancies. Qwen3-4B achieves 84.8% on MATH-500 in the official non-thinking evaluation, whereas the paper reports only 52.40% for CoT and 68.28% for TACIT. Although the evaluation protocols differ, this unusually large gap requires explanation. In addition, the normalized confidence in Eq. (2) is exactly a sigmoid transformation of the logit gap in Eq. (3), so the two quantities must induce identical rankings and AUROC, contradicting the claimed “markedly sharper ordering”. The router is also specified as multi-label in Section 3.4 but multiclass in Appendix A.2 and Algorithm 2, while the preference construction for correct-but-low-confidence responses differs between Section 3.3 and Algorithm 2. These inconsistencies make the reported results difficult to interpret and reproduce.
Questions for authors
  1. Can the authors report the actual inference computation cost relative to at least main baselines, and add a compute-matched rejection-sampling SFT baseline using the same verified peer trajectories, to determine whether TACIT’s gains arise from confidence-gated collaboration rather than additional correct supervision?
7 comments

We apologize for the delayed response, as we have devoted time to conducting the suggested experiments as thoroughly as possible within the available period.

> Re W1 & Q1: The evidence does not adequately support the paper’s framing.

  1. Framing of co-evolution and collaboration.

We agree that TACIT does not involve explicit interactive communication or joint policy optimization. However, multi-agent collaboration is not inherently restricted to interactive dialogue or shared parameter updates, as emphasized in our manuscript. Although agents are optimized independently, their learning processes are interdependent: each agent's improvement is guided by its own confidence signals and reasoning trajectories of its peers, establishing a form of cross-policy dependency during training. This is precisely the motivation behind TACIT: internalizing multi-agent collaborative behaviors through post-training, rather than repeatedly relying on explicit communication at inference. Agents reciprocally contribute to one another's improvement through their complementary strengths and weaknesses, which we characterize as co-evolution through mutual experience transfer.

Importantly, TACIT is designed to imitate a type of social collaborative behavior: knowing when to trust one's own judgment and when to seek external assistance. By learning self-verification signals and selectively incorporating peer experiences, agents acquire the ability to distinguish reliable reasoning from uncertain attempts. We therefore view TACIT as a concrete exploration of training-time implicit collaboration.

  1. Beyond rejection-sampling distillation.

While TACIT indeed leverages correct peer trajectories, its learning mechanism goes beyond conventional rejection-sampling distillation. Specifically, TACIT performs confidence-conditioned, agent-specific cross-policy experience transfer that internalizes peer corrections into independently deployable reasoning policies. By virtue of preference optimization guided by self-verification markers, the training objective is not merely to imitate correct solutions, but also to understand the act of generating the correct answer (confirmed or revised). In this way, self-verification and reasoning improvement are jointly reinforced, enabling agents to better regulate their own outputs and contribute to more reliable collective decision-making.

This distinction is particularly meaningful when ground-truth feedback is unavailable in real-world scenarios, making reliable self-verification essential for decision-making. Furthermore, drawing complementary trajectories from multiple agents can expose each learner to more diverse reasoning strategies than a single-policy source, potentially boosting reasoning capabilities.

We acknowledge the importance of disentangling the gains of correct-trajectory supervision from those of our method. To this end, we additionally evaluated a compute-matched rejection-sampling SFT (RS-SFT) baseline.

  • RS-SFT: Starting from the same Stage I checkpoints as TACIT, each agent undergoes a SFT stage using verified peer trajectories selected from the same candidate pool as TACIT. Final results are aggregated via majority voting.

Table 1. Accuracy (%) on mathematical benchmarks with Qwen3-4B. Results are reported as mean ± standard deviation.

MethodGSM8KMATH-500AIME 2024AIME 2025
Single-Agent GRPO91.96 ± 0.1864.20 ± 0.3321.66 ± 0.4919.27 ± 0.58
Multi-Agent GRPO92.49 ± 0.3265.32 ± 0.3623.06 ± 0.5221.05 ± 0.54
Vanilla SFT91.88 ± 0.2658.75 ± 0.2921.42 ± 0.4117.94 ± 0.36
Rejection-Sampling SFT92.42 ± 0.2960.87 ± 0.2421.48 ± 0.3718.56 ± 0.34
TACIT (Ours)95.97 ± 0.2167.36 ± 0.2922.89 ± 0.3320.92 ± 0.45

This comparison directly assesses whether TACIT simply benefits from supervision or truly from collaboration. Positive results support our claim: gains of TACIT arise from confidence-gated collaboration rather than additional correct supervision.

Although Multi-Agent GRPO achieves marginally higher accuracy on AIME 2024 and AIME 2025, these results suggest that TACIT achieves competitive reasoning accuracy while providing intrinsic self-verification capabilities at substantially lower training cost.

  • The small sizes of the AIME evaluation sets introduce substantial sampling variability.
  • Beyond achieving comparable performance, TACIT additionally learns reliable self-verification signals (73.9–91.2% AUROC), enabling agents to assess their own answers.
  • GRPO requires computationally expensive online rollouts, whereas TACIT employs offline DPO with pre-collected trajectories. In our experiments, multi-agent GRPO incurs approximately twice the training time of TACIT under identical GPU configurations.

> Re W2: Experimental validity and methodological consistency.

We appreciate the reviewer's careful assessment of TACIT. We address the four concerns as follows.

  1. Qwen3-4B evaluation discrepancy.

The observed gap between the official MATH-500 result and our reported results can be largely explained by the following causes,

  • Generation budget: The official evaluation allows up to 32,768 output tokens, substantially exceeding our maximum of 2,048 tokens.
  • Evaluation protocol: The official report does not fully specify the MATH-500 prompting and answer-verification procedures, leaving potential differences in prompt formulation, answer extraction, and mathematical equivalence checking.

The sensitivity of Qwen3 to evaluation protocols has also been independently documented (Sang et al., 2026). For instance, the reported MATH-500 accuracy drops from 81.0% under a boxed-answer evaluation setting to 39.1% under a DAPO-style answer format and corresponding extraction rules, illustrating how evaluation choices can substantially affect measured performance.

Importantly, our experiments were not intended to reproduce the official Qwen3 benchmark scores. Instead, all methods in our study are evaluated under a consistent protocol, with matched prompting, decoding budgets, and answer-verification rules. This controlled setting supports a fair assessment of relative improvements, under which TACIT improves MATH-500 accuracy from 52.40% (CoT) to 68.28% (+15.88 percentage points). We will explicitly clarify these differences and the scope of our comparisons in the revised manuscript.

  1. The inaccurate “markedly sharper ordering” claim.

We thank the reviewer for identifying this algebraic inaccuracy. Indeed, the two metrics induce identical rankings and AUROC in exact arithmetic. Our motivation for using the logit gap is its unbounded, unsquashed representation of the confidence margin, which offers numerical convenience and stability. We have corrected this explanation in the revised manuscript without changing the confidence computation or reported AUROC.

  1. Router specification.

The router is implemented as multi-label classification, consistent with Section 3.4. Each agent receives an independent binary competence label, and the router is trained using binary cross-entropy. The multi-class descriptions in Appendix A.2 and Algorithm 2 were inadvertent documentation errors, which we have corrected to match the actual implementation. We apologize for the confusion.

  1. Preference construction.

Section 3.3 specifies two distinct cases:

  • Incorrect responses: The rejected sequence is ri⊕[confirm]r_i \oplus [\mathrm{confirm}], discouraging unjustified confidence.
  • Correct but low-confidence responses: The rejected sequence is ri⊕[uncertain]r_i \oplus [\mathrm{uncertain}], avoiding penalization of a correct answer.

Algorithm 2 inadvertently omitted this conditional distinction, causing an inconsistency with Section 3.3. We have corrected the pseudocode to explicitly distinguish these two cases and ensure consistency with the actual preference construction.

[1] Sang, H., Xu, Y., Zhou, Z., He, R., & Wang, Z. (2026). Answer Format Sensitivity in Qwen3 Math Reasoning Evaluation. Hejian Blog. Link

> Re W1 & Q1: Contribution of self-verification gating.

We agree that transferring correct peer trajectories is inherently beneficial, as reflected by the random-gated improvement from 52.40% to 63.00% in Table 3. Both stages contribute to the final gains, although Stage II yields a larger improvement, which is expected since it directly internalizes complementary peer experiences to enhance reasoning capabilities, whereas Stage I primarily establishes self-verification rather than directly optimizing task performance.

However, Stage II is fundamentally built upon the self-verification capability acquired in Stage I. The learned confidence signals determine when peer experience should be transferred, enabling targeted learning from individual weaknesses rather than indiscriminate supervision. Moreover, Stage II further reinforces self-verification through marker-conditioned preference optimization, jointly improving reasoning and verification capacity. Under the same transferred-experience budget, the full TACIT framework further improves accuracy from 63.00% to 68.28%, highlighting the additional value of incorporating self-verification into experience transfer.

Importantly, the newly added rejection-sampling SFT (RS-SFT) baseline achieves only 60.87% on MATH-500, compared with TACIT's 67.36%, further demonstrating that simply imitating ground-truth-filtered correct trajectories is insufficient to reproduce TACIT's gains.

Therefore, our central claim is not that self-verification gating alone drives the improvements, but that it serves as the foundation connecting both training stages, enabling more targeted and effective internalization of complementary peer experiences.

Reviewer_AJVgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an approach based on self-verification and direct-preference optimization to train agents and then uses a gating mechanism at inference time to perform an informed collaboration with saving communication bandwith. Authors test their approach on two LM families across math datasets and also show transfer to two more benchmarks.

Critical strengths
  • The idea is simple and straightforward to understand.
Critical weaknesses
  • Main weakness is experiments: 1- The method needs ground truth during training, if so authors need to have RL and SFT baselines. For example, a baseline would be post-training the model with GRPO etc. 2- The experiments seems to be on single seed? I didn't find details on this. if so, they should report on 3 seeds. 3- The models are limited to Llama and Qwen and the impact on other families is not obvious. 4- there are no performance comparisons agains baselines in terms of mean tokens, throughput etc. to show the impact of router compared to for example self-consistency. 5- paper is missing baselines such as https://arxiv.org/abs/2509.15172 and https://arxiv.org/abs/2411.04109
Questions for authors

see weaknesses

5 comments

We apologize for the delayed response, as we have devoted time to conducting the suggested experiments as thoroughly as possible within the available period.

> Re W1: Ground-truth supervision and post-training baselines.

We thank the reviewer for this valuable suggestion. We acknowledge that TACIT utilizes ground-truth correctness during both training stages. However, our method differs from conventional correctness-driven post-training in how ground-truth supervision is incorporated into preference learning. Rather than simply preferring correct solutions over incorrect ones, TACIT uses correctness as an intermediate supervisory signal to develop self-verification and guide targeted cross-agent experience transfer.

Comparisons with standard post-training methods are essential to establish whether TACIT offers benefits beyond direct correctness supervision. To this end, we have conducted four additional baseline experiments covering both SFT and GRPO:

1. Supervised Fine-Tuning (SFT).

  • Vanilla SFT: Each agent is independently fine-tuned on correctness-filtered solutions in a single training stage, without self-verification learning or cross-agent experience transfer. This baseline evaluates the effectiveness of direct correctness-based supervision without TACIT's two-stage modeling.
  • Rejection Sampling SFT (RS-SFT): Starting from the same Stage I checkpoints as TACIT, each agent undergoes an additional SFT stage using verified peer trajectories selected from the same candidate pool as TACIT. This comparison is particularly important for disentangling the benefits of Stage II preference learning from those of rejection-sampling distillation of correct peer trajectories.

2. Group Relative Policy Optimization (GRPO).

  • Single-Agent GRPO: A single agent is directly optimized using GRPO with ground-truth correctness as the reward signal.
  • Multi-Agent GRPO: Multiple agents are independently optimized using GRPO with the same correctness-based reward formulation. Compared with Single-Agent GRPO, this baseline additionally accounts for the performance gains from multi-agent ensembling, allowing us to distinguish the benefits of TACIT's training-time collaboration from those of independently trained agent populations.

For all multi-agent baselines, we adopt majority voting rather than TACIT's confidence-weighted aggregation, avoiding dependence on the self-verification mechanism that constitutes part of our proposed method.

Importantly, TACIT does not treat ground-truth correctness as the sole optimization objective, but rather as a supervisory bridge toward acquiring reusable reasoning capabilities. Our central research question is whether the same correctness supervision can be leveraged more effectively by learning when an agent should trust its own reasoning and when it should acquire complementary experience from peers.

Accordingly, TACIT aims to improve not only downstream task accuracy but also endogenous self-verification, which provides a foundation for more adaptive multi-agent collaboration. Within our framework, these capabilities further support confidence-guided routing and communication-free inference, reducing the need for repeated inter-agent interactions at deployment.

Table 1. Accuracy (%) on mathematical benchmarks with Qwen3-4B. The best in each column is highlighted in bold.

MethodGSM8KMATH-500AIME 2024AIME 2025
Single-Agent GRPO91.96 ± 0.1864.20 ± 0.3321.66 ± 0.4919.27 ± 0.58
Multi-Agent GRPO92.49 ± 0.3265.32 ± 0.3623.06 ± 0.5221.05 ± 0.54
Vanilla SFT91.88 ± 0.2658.75 ± 0.2921.42 ± 0.4117.94 ± 0.36
Rejection-Sampling SFT92.42 ± 0.2960.87 ± 0.2421.48 ± 0.3718.56 ± 0.34
TACIT (Ours)95.97 ± 0.2167.36 ± 0.2922.89 ± 0.3320.92 ± 0.45

Although Multi-Agent GRPO achieves marginally higher accuracy on AIME 2024&2025, these results suggest that TACIT achieves competitive reasoning accuracy while providing intrinsic self-verification capabilities at substantially lower training cost.

  • The small sizes of the AIME evaluation sets introduce substantial sampling variability.
  • Beyond achieving comparable performance, TACIT additionally learns reliable self-verification signals (73.9–91.2% AUROC), enabling agents to assess their own answers.
  • GRPO requires computationally expensive online rollouts, whereas TACIT employs offline DPO with pre-collected trajectories. In our experiments, multi-agent GRPO incurs approximately twice the training time of TACIT under identical GPU configurations.

We hope the above experiments better substantiate TACIT's central premise: ground-truth supervision serves not merely to identify which to imitate or reward, but also to enable them to learn when and how to improve through self-verification and targeted experience transfer.

> Re W2: Random seeds and experimental reproducibility.

Thank you for raising this important point. We clarify that the results in Tables 1 and 2 are obtained from three independent training runs with different random seeds, each evaluated using five independent inference runs. For each training run, we first average the accuracy across the five inference runs, and then report the mean and sample standard deviation across the three independently trained runs in Tables 1 and 2. This protocol reduces inference sampling noise while capturing performance variability across independent training runs. All baselines follow the same training and evaluation protocol, ensuring a consistent and fair comparison.

Within each independent training run, we further assign distinct random seeds to individual agents for data construction and optimization, as detailed in Appendix A. Specifically, agent-specific bootstrap seeds are used to sample training questions with replacement, exposing agents to different combinations of training examples. This design encourages complementary learning experiences without imposing artificial correctness priors, thereby establishing the diversity necessary for effective cross-agent experience transfer in Stage II.

We acknowledge that the distinction between independent training runs, agent-specific seeds, and inference seeds was not sufficiently explicit in the original manuscript. We have clarified the protocol and random seed configuration in Section 4.1, with further implementation details provided in Appendix A.

> Re W4: Inference efficiency against other baselines.

We agree that efficiency claims should be supported by explicit computational measurements. To address this concern, we conducted additional evaluations of inference token consumption, end-to-end latency, and throughput across all seven methods on MATH-500 (500 questions) with Qwen3-4B checkpoints trained on MATH. TACIT employs top-3 agent routing without explicit inter-agent communication at inference.

As shown in Table 2, TACIT achieves the highest accuracy (68.28%) while substantially reducing inference costs compared with explicit multi-agent collaboration methods. Specifically, compared with Multi-Agent Debate, TACIT achieves 3.56 percentage points higher accuracy with 58.5% fewer tokens and 43.8% shorter execution time, while improving throughput by 1.78×. Compared with iMAD, TACIT improves accuracy by 2.97 percentage points, reduces token consumption by 21.3% and execution time by 16.9%, and achieves 1.21× higher throughput.

We acknowledge that lightweight single-agent methods incur lower inference costs, but TACIT delivers substantially stronger reasoning performance. These results demonstrate a favorable accuracy–efficiency trade-off, particularly against inference-time multi-agent collaboration, supporting our central motivation: internalizing collaborative experience during training can preserve the benefits of multi-agent reasoning while substantially reducing recurring communication overhead at inference.

Table 2. Inference efficiency comparison on MATH-500 with Qwen3-4B.

MethodAccuracy (%) ↑Total Tokens / Q ↓Latency (s) ↓Throughput (Q/s) ↑
Chain-of-Thought [1]52.40643.718223.4282.238
Self-Consistency [2]56.051281.293246.8052.026
Multi-Agent Debate [3]64.725200.4732032.7950.246
iMAD [4]65.312742.5681375.5240.363
ReVISE [5]57.981231.898868.4290.576
S2R [6]63.681629.7671022.4590.489
TACIT (Ours)68.282157.8421142.5730.438

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.

[2] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023.

[3] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, 2024.

[4] Wei Fan, JinYi Yoon, and Bo Ji. imad: intelligent multi-agent debate for efficient and accurate llm inference. In Proceedings of the Fortieth AAAI Conference on Artificial Intelligence, pp. 29403–29411. AAAI Press, 2026. ISBN 978-1-57735-906-7.

[5] Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. Revise: Learning to refine at test-time via intrinsic self-verification. In International Conference on Machine Learning, pp. 33616–33634. PMLR, 2025.

[6] Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching llms to self-verify and self-correct via reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22632–22654, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.