acceptodds
Under review as a conference paper at ICLR 2027

Cochain: Balancing Insufficient and Excessive Collaboration in LLM Agent Workflows

Abstract

Large Language Models (LLMs) have demonstrated impressive performance in executing complex reasoning tasks. Chain-of-thought effectively enhances reasoning capabilities by unlocking the potential of large models, while multi-agent systems provide more comprehensive solutions by integrating the collective intelligence of multiple agents. However, both approaches face significant limitations. Single-agent with chain-of-thought, due to the inherent complexity of designing cross-domain prompts, faces collaboration challenges. Meanwhile, multi-agent systems consume substantial tokens and inevitably dilute the primary problem, which is particularly problematic in business workflow tasks. To address these challenges, we propose Cochain, a collaboration prompting framework that effectively solves the business workflow collaboration problem by combining knowledge and prompts at a reduced cost. Specifically, we construct an integrated knowledge graph that incorporates knowledge from multiple stages. Furthermore, by maintaining and retrieving a prompts tree, we can obtain prompt information relevant to other stages of the business workflow. We perform extensive evaluations of Cochain across multiple datasets, demonstrating that Cochain outperforms all baselines in both prompt engineering and multi-agent LLMs. Additionally, expert evaluation results indicate that the use of a small model in combination with Cochain outperforms GPT-4. Our code is available at https://anonymous.4open.science/r/Cochain-F871.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent Collaboration, LLM Agents, Business Workflow Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7RUgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Cochain, a collaboration prompting framework for balancing insufficient and excessive collaboration in LLM agent workflows. Single-agent approaches can miss cross-stage constraints, while multi-agent systems can consume substantial tokens and dilute the main problem. Cochain combines knowledge and prompts to address these issues at a reduced cost. It constructs a collaborative knowledge graph containing explicit and tacit knowledge from multiple workflow stages, organizes retrieved knowledge into causal chains, and retrieves prompt chains from a prompts tree to guide reasoning. The paper evaluates Cochain across multiple datasets and LLM backbones, reporting improvements over prompt engineering and multi-agent baselines.

Critical strengths

The problem is meaningful. Bringing in useful knowledge from other stages while avoiding unnecessary discussion could improve both answer quality and efficiency.

Critical weaknesses
  • Several key steps are not clearly explained, including knowledge extraction, the meaning and use of Eq. 3, and prompt-chain retrieval. It is also unclear why connected paths in the knowledge graph should be called causal chains.

  • The experiments do not clearly show how Cochain balances collaboration for different queries. The paper includes prompt-budget experiments and examples, but does not directly show which necessary constraints are recovered or which irrelevant details are removed.

  • The Auto-SLURP transfer setup and component ablations need clearer explanations. I would also like to see results with newer models, such as GPT-5.x or Claude 4.x, and better justification of the GPT-4 judging setup.

Questions for authors
  • How are the triples in Eq. 1 extracted from the input-output pairs? What determines the entities and relation types?

  • Why should counterfactual prompting reveal valid hidden dependencies? The appendix gives examples, but how are these outputs checked against reliable references? What counts as an ideal collaboration output?

  • What does Eq. 3 contribute to the actual method? How are its latent variables used in knowledge extraction?

  • Why are the paths in Eq. 7 causal chains? Connecting nodes through a shared entity does not seem sufficient to establish causality.

  • Is there one prompts tree for all Q&A pairs, or a separate tree for each seed? Why is a tree needed instead of directly storing prompt chains? For a new query, how is the matching path selected? The mention of DFS does not explain this selection.

  • What exactly is removed in each ablation? If the knowledge graph is removed, how are causal chains still constructed? Is only one part of the final prompt being removed?

  • What is transferred to Auto-SLURP, and what is rebuilt using its data? Without these details, it is difficult to tell what kind of generalization this experiment tests.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7RUg (Part 1/3)

We answer each question with the exact procedure used in our implementation.

[Q1] How are the triples in Eq. 1 extracted? What determines entities and relations?

Each (X, Y) pair is processed by an LLM with a fixed few-shot extraction prompt (released in our repository): the head entity comes from the keywords of X, the tail from Y, and the relation is an open-vocabulary phrase for the method or dependency linking them, e.g., (fuel economy, improvement depends on, lightweight body design). Surface variants are merged by entity normalization (Fig. 2), and each node carries the stage label of its dataset; nodes with ≥2 stage labels become bridge entities. Entities and relations are thus determined by the workflow data, not by a fixed schema (Automotive: 414,878 explicit triple statements from 72,598 Q&A pairs, plus tacit triples from 41,383 counterfactual pairs).


[Q2] Why does counterfactual prompting reveal valid dependencies? How are outputs checked? What is an ideal collaboration output?

A factual question ("how to improve fuel economy?") elicits the default answer, where feasibility constraints stay implicit; perturbing one condition while preserving intent ("…if the existing stamping line and supplier base cannot change?") forces the agent to state what must change downstream. Validity is checked at three levels: (i) every answer passes an evaluator loop (reasonable / ambiguous / unreasonable, with revision feedback) before extraction; (ii) a blind expert audit of 2,000 sampled triples, half counterfactual (Table 15: correctness 4.99/5, relevance 4.76/5, depth 4.31/5); (iii) removing only the tacit triples lowers performance (Pharmaceutical, Qwen2-7B: 78.48/33.98/35.03 → 77.17/31.26/31.99). An ideal collaboration output satisfies the cross-stage constraints experts regard as necessary while staying focused on the decision. We now measure both: against constraints extracted from expert references, Cochain covers 40.6% vs. ≤23.4% for every other system (51.6% vs. ≤34.2% with Qwen3.8-Max).


[Q3] What does Eq. 3 contribute?

Eq. 3 is a modelling device, not a computational step. Tacit knowledge is by definition not written in the data, so Eq. 3 makes it a precise object: a latent variable θ that shapes the agent's internal state h, which shapes its answer to a counterfactual input. This explains why counterfactual probing reveals it: θ and h are unobservable, but changing one condition while keeping the intent changes the answer through exactly the considerations that condition activates, and these are what we extract as tacit triples from (X̃, Ỹ). The latent variables are not estimated; they define tacit knowledge, explain why elicitation can surface it, and motivate the design that Algorithm 1 implements (one perturbation at a time, intent preserved, evaluator-checked answers).


[Q4] Why are the paths in Eq. 7 "causal" chains?

Standard retrieval returns discrete facts and leaves the model to work out how they relate. A causal chain connects them into a path whose edges are directed cause–effect or dependency relations (extracted with causal cues such as "depends on", "requires", "increases the risk of"). For "How can ergonomics be used to reduce traffic-accident injuries?", keyword retrieval finds facts about ergonomics and about injuries but nothing relating them, whereas multi-hop expansion yields Ergonomics → applied to → Cockpit Design → influences → Driver Fatigue → increases risk of → Vehicle Collision → results in → Injuries. Across stages, bridge entities link a design choice to a production constraint, e.g., Carbon fiber → requires → complex molding → requires → new curing equipment → must be compatible with → the existing stamping line. We call these paths causal chains because every link states how one element causes, requires or affects the next; the term does not claim statistical causal identification, and a shared entity alone does not make a chain.

This cross-stage linking is what makes the paths useful: a standard multi-hop Graph-RAG over the same graph (2-hop from the 3 most similar entities, no stage-aware bridging, same budget) scores 72.49/19.58/23.39 vs. 74.16/25.04/26.00 for Cochain (Automotive, Qwen2-7B, 3 seeds, p < 1e-22), below even flat retrieval of the same triples (72.73/20.07/23.66), because unconstrained expansion drifts within one stage instead of reaching supply-chain and production constraints.

AuthorsOpenReview ↗

Response to Reviewer 7RUg (Part 2/3)

[Q5] One Prompts Tree or one per seed? Why a tree? How is the path selected?

One tree per domain: a virtual "user need" root whose children are the seed user needs (505 for Automotive), each expanding into design → supply chain → production → quality-inspection prompts (33,769 nodes). A tree is needed because prompts are conditional on upstream decisions: the 33,769 nodes contain only 1,420 distinct (stage, prompt) strings, so a prompt's meaning is given by its path, and siblings under the same upstream decision can be merged at retrieval. Flattening the same prompts into a repository is the weakest information-matched variant (72.16/18.82/22.99 vs. 74.16/25.04/26.00). Retrieval: (1) match the query to the most similar seed user need; (2) enumerate the root-to-leaf paths below it depth-first; (3) merge paths sharing the same design → supply → production prefix; (4) insert the top-k merged chains (k = 3 by default; Fig. 5) in stage order.


[Q6] What is removed in each ablation?

Yes, each ablation removes one part of the final prompt. Cochain first selects one-hop CKG facts, composes cross-stage causal chains and retrieves stage-ordered prompt chains, then assembles them as three blocks: [Knowledge], [Causal Chains] and [Workflow Prompts]. Each ablation deletes exactly one block and keeps everything else fixed; causal chains therefore remain in the w/o CKG row, which isolates the one-hop facts. Whether the organization itself matters is tested by information-matched baselines that give the same backbone the same material without Cochain's organization (Automotive, Qwen2-7B, 3 seeds): flat CKG triples 72.73/20.07/23.66, Graph-RAG 72.49/19.58/23.39, flat prompts 72.16/18.82/22.99 and their hybrid 72.34/21.06/24.12, all below Cochain's 74.16/25.04/26.00 (p < 1e-14).


[Q7] What is transferred to Auto-SLURP and what is rebuilt?

Only the procedure is transferred (extraction prompt, Prompts-Tree algorithm, retrieval and prompt composition). The knowledge graph (triples linking queries, intents, slots and actions) and prompt trees are rebuilt from the official 1,000-instance training split, and evaluation uses the official 100-instance test split and judge. The experiment tests whether the procedure generalizes to executable multi-step assistant workflows, not whether automotive knowledge transfers.


[W2] Which constraints are recovered and which irrelevant details are removed

We now measure this per question: two independent judges label each expert-required constraint in each answer as covered / partial / missing and estimate the share of off-question content (Qwen2-7B):

SystemConstraint coverage ↑Off-focus ↓
IO20.9%13.4%
Dense retrieval (same sources)23.4%15.4%
CoA20.2%20.2%
PMC / Debate (long)3.5% / 3.7%95.1% / 91.5%
Cochain40.6%20.5%

Example (closed-loop recycling in vehicle design, 7 expert constraints): Cochain covers modular, disassembly-friendly design and also recyclability-driven material selection, supplier-side material verification, material-utilization targets, process stability and incoming/final inspection (off-focus 10%); IO covers only the two design-side constraints (off-focus 30%). PMC and Debate answers are over 90% off the question, the dilution Cochain avoids.

AuthorsOpenReview ↗

Response to Reviewer 7RUg (Part 3/3)

[W3] Newer models; GPT-4 judging

We ran every information-matched variant on three current backbones with the same sources and budget (Automotive; GPT-5.4-mini with reasoning off, Qwen3.8-Max with thinking off; BS-F / GLEU / ROUGE-L):

ContextGPT-5.4-miniClaude-Sonnet-4.5Qwen3.8-Max
IO (question only)66.46 / 10.30 / 13.5365.23 / 9.77 / 12.1864.98 / 7.94 / 11.51
Role prompt only66.24 / 10.06 / 12.6865.02 / 6.18 / 9.0066.41 / 9.08 / 13.03
Dense retrieval over training Q&A66.76 / 10.63 / 13.5066.80 / 9.49 / 12.3567.42 / 9.83 / 13.72
Flat retrieval of the same CKG triples67.37 / 11.15 / 13.9066.88 / 9.62 / 12.2168.12 / 10.37 / 14.19
Graph-RAG over the same CKG67.11 / 10.93 / 13.7267.07 / 9.55 / 12.41—
Flat repository of the same prompts66.83 / 10.72 / 13.3566.82 / 8.93 / 11.9568.07 / 10.26 / 14.04
Hybrid of the three flat sources66.98 / 10.87 / 13.5666.96 / 9.78 / 12.6367.96 / 10.27 / 14.15
Cochain67.83 / 12.03 / 14.3067.25 / 10.09 / 12.7469.10 / 12.10 / 15.37

Cochain is best on every metric with all three backbones (GPT-5.4-mini: p ≤ 0.003 against every row).

On judging: in the submission, GPT-4 scoring complements the blind expert study and applies the same criterion: an answer must first answer the question correctly and is then judged on how well it accounts for collaboration across the workflow. GPT-4 scores only Qwen2-7B outputs under four conditions (raw and fine-tuned, each with and without Cochain; Fig. 3b), five times each with the scores averaged, so it never scores its own answers, and its ranking agrees with the expert study (Fig. 3a).

To address the concern directly, we added a stricter judge-based evaluation: two judges from families different from every backbone (DeepSeek-V4-Pro, GLM-5.3) score each answer blindly against the cross-stage constraints extracted from the expert reference, with a fixed rubric, constraint-level labels and position-swapped pairwise comparisons (inter-judge ρ = 0.84). The advantage holds on the current models:

GPT-5.4-miniClaude-Sonnet-4.5Qwen3.8-Max
Constraint coverage: Cochain / best matched variant / IO31.6% / 24.6% / 23.6%54.3% / 37.9% / 26.9%51.6% / 34.2% / 23.5%
Pairwise, Cochain win / loss vs. IO53.8% / 2.2%68.0% / 0.5%69.5% / 1.5%
Pairwise, Cochain loss vs. any matched variant≤ 9.8%≤ 4.5%≤ 5.5%

In the revised PDF, the small-vs-large claim is stated in terms of the expert study.

Reviewer_Gu59OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Cochain organizes domain knowledge into a graph and a tree of reusable workflow prompts. For each query, it retrieves relevant facts, connections across stages and workflow guidance to help an LLM generate its answer. Experiments cover automotive, pharmaceutical and e-commerce tasks, showing improvements on most reported quality metrics and lower online costs than the compared multi-agent systems.

Critical strengths
  1. Many workflow questions require information from several stages, and decisions at one stage can affect later steps. Combining domain knowledge with guidance across stages is a useful way to address these dependencies.

  2. The experiments cover several model families and three domains. The reported reductions in online cost and latency are encouraging, especially for applications that repeatedly use the same domain knowledge.

Critical weaknesses
  1. Cochain has access to carefully organized domain knowledge, yet the paper does not compare it with standard RAG or keyword-based retrieval over the same knowledge sources. A comparison with matched information access and input budgets would help establish how much the graph and prompt tree contribute beyond retrieving relevant material.

  2. The paper does not clearly establish when collaboration becomes excessive or when additional communication starts to hurt performance. The compared systems differ in knowledge access, prompting and execution, making it difficult to connect their performance differences to the amount of collaboration.

  3. An answer can resemble a reference and still propose an incorrect or impractical workflow. BERTScore, GLEU and ROUGE-L provide limited evidence about whether the generated advice satisfies domain constraints and solves the task. The human evaluation needs clearer criteria for assessing these properties, while the gains on Auto-SLURP are small.

  4. The cost analysis leaves unclear whether knowledge extraction, counterfactual generation, repeated checking and stage-agent training are fully included. These steps require substantial preparation, so the overall savings depend on how often the resulting knowledge and prompts are reused. Reporting the complete setup cost and the number of queries needed to recover it would make the efficiency claim easier to assess.

Questions for authors
  1. What knowledge sources were available to each baseline? Do you have a comparison with standard RAG or keyword-based retrieval using the same sources and a similar input budget?

  2. How do you define excessive collaboration? Which existing results show that increasing communication hurts performance when information access and other settings are held fixed?

  3. What criteria did the human evaluators use to assess correctness, feasibility and consistency across stages? Were system identities hidden, and did the compared systems receive the same domain knowledge?

  4. Which preparation steps are included in the reported costs? With the full setup cost included, approximately how many queries are needed for Cochain to become cheaper than the compared methods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Gu59 (Part 1/2)

[W1 & Q1] Knowledge sources per baseline; standard RAG / keyword retrieval with the same sources and budget

All multi-agent baselines in Table 1 run on the same fine-tuned stage agents as Cochain, so they share its training-derived knowledge but not its artifacts. We added keyword (BM25), dense-RAG, flat-KG, multi-hop Graph-RAG and flat-prompt baselines over the same sources Cochain is built from, with the same encoder, role prompt and per-query budget (Automotive, Qwen2-7B, 3 seeds, all SE ≤ 0.17):

Context given to the same backboneBS-FGLEUROUGE-L
IO (question only)71.8221.7524.32
Role prompt only72.1720.2923.74
Keyword retrieval (BM25) over training Q&A71.7021.5023.90
Standard dense RAG over training Q&A71.6621.3023.77
Flat retrieval of the same CKG triples72.7320.0723.66
Graph-RAG over the same CKG (2-hop)72.4919.5823.39
Flat repository of the same prompts72.1618.8222.99
Hybrid of the three flat sources72.3421.0624.12
Cochain74.1625.0426.00

Cochain beats every variant on every metric (p < 1e-14), with the same ordering for the base model (74.50/24.70/25.61 vs. ≤72.87/20.55/23.76) and Qwen3.8-Max (69.10/12.10/15.37 vs. ≤68.12/10.37/14.19); flat insertion even falls below IO on GLEU. Giving PMC and Debate the same triples does not close the gap (Pharmaceutical: PMC 66.97 → 67.51 BS-F, Debate 70.62 → 70.29, Cochain 78.50). The gain comes from how Cochain selects and composes cross-stage evidence, not from more material.


[W2 & Q2] Definition of excessive collaboration; evidence with information held fixed

Definition: with backbone and information access fixed, collaboration is excessive when additional inter-agent communication increases cost without improving, or while degrading, the satisfaction of necessary cross-stage constraints and the focus of the decision. To test this, four stage-expert agents (design, supply chain, production, quality) debate for 1–4 rounds, either with no extra context or with the identical Cochain context for every agent, so only communication varies (Automotive, Qwen3.8-Max). Constraint coverage and off-focus share are judged against the constraints extracted from the expert reference:

InformationRoundBS-FGLEUROUGE-LConstraint coverageOff-focusTokens / query
none166.399.0813.3625.2%9.8%4.3k
none464.588.6912.7536.2%16.2%67.0k
same Cochain context168.6512.2815.5147.2%10.1%4.9k
same Cochain context267.0011.2314.2945.7%10.8%26.1k
same Cochain context366.4810.9414.0544.4%12.8%47.7k
same Cochain context466.1810.8613.9544.3%13.9%69.2k
Cochain (single agent)—69.1012.1015.3748.9%13.2%≈1.2k

Without shared knowledge, communication recovers constraints (insufficient collaboration is real), but at 15× the tokens and with rising off-focus content (rounds 2–3: 34.9%/35.0% coverage). Once the information is shared, every further round lowers all metrics and coverage while off-focus content grows: this is excessive collaboration, isolated from knowledge access, prompting and execution. Cochain supplies the constraints without the communication and attains the highest coverage.

AuthorsOpenReview ↗

Response to Reviewer Gu59 (Part 2/2)

[W3 & Q3] Beyond text similarity; human-evaluation criteria; blinding; equal knowledge

(1) Criteria. Five engineers of a major automobile manufacturer compared two answers per question (A better / B better / similar) under one criterion: which answer, while answering the question, better accounts for collaboration across the workflow. An answer that does not answer the question cannot be preferred (correctness); accounting for collaboration means that what is proposed for one stage respects the requirements and limits of the others, e.g., a design change must be sourceable, manufacturable on the existing line and testable (feasibility and cross-stage consistency).

(2) Blinding. Yes: engineers saw only "Answer A" and "Answer B", with the assignment randomized per question.

(3) Equal knowledge. The expert study asks whether a 7B model with Cochain can match a much larger model used directly, so GPT-4 answers without Cochain's artifacts. Equal knowledge is tested separately by the retrieval baselines above and the judge-based evaluation below.

Constraint satisfaction rather than resemblance (W3). From each expert reference we extract the cross-stage constraints the decision must respect (6.2 per question); two non-Qwen judges (DeepSeek-V4-Pro, GLM-5.3) label each as covered (1), partial (0.5) or missing, estimate off-focus content and score cross-stage consistency (inter-judge ρ = 0.84); pairwise comparisons show the reference and constraints, in both orders (Qwen2-7B):

SystemConstraint coverage ↑Off-focus ↓Consistency (1–5) ↑Cochain win / tie / loss (%)
IO20.9%13.4%2.2175.0 / 22.0 / 3.0
Dense retrieval (same sources and budget)23.4%15.4%2.4663.2 / 28.7 / 8.0
Flat-KG (same triples)21.9%23.1%2.1976.0 / 20.8 / 3.2
Hybrid-flat22.1%21.6%2.2274.0 / 21.8 / 4.2
CoA20.2%20.2%2.2180.5 / 16.8 / 2.8
PMC / Debate (long)3.5% / 3.7%95.1% / 91.5%1.71 / 1.38—
Cochain40.6%20.5%2.82—

Cochain covers about twice as many expert-required constraints as any system, including those given the same material, and loses at most 8% of comparisons; with Qwen3.8-Max, coverage is 51.6% vs. ≤34.2% with losses ≤5.5%.

Auto-SLURP. The absolute margin is small, so we added two checks (official 100-instance split and judge; 95% Wilson CI for Cochain, exact McNemar p vs. Cochain). First, a third backbone, GPT-5.4-mini (last row):

BackboneCochainCoAPMCMedAgents
DeepSeek-V3.10.33 [0.25, 0.43]0.27 (p=0.39)0.12 (p<0.001)0.07 (p<0.001)
Claude-3.5-haiku0.38 [0.29, 0.48]0.38 (p=1.0)0.29 (p=0.18)0.26 (p=0.07)
GPT-5.4-mini0.33 [0.25, 0.43]0.27 (p=0.33)0.20 (p=0.01)0.19 (p=0.01)

Cochain is again best, 6 points above CoA and significantly better than PMC and MedAgents. Second, a single stricter judge (DeepSeek-V3.1-Terminus) lowers all accuracies but keeps the ranking (Cochain 0.17, CoA 0.10, PMC 0.05, MedAgents 0.03), widening Cochain's relative lead over CoA from 1.2× to 1.7×.


[W4 & Q4] Included preparation steps; full setup cost; break-even

Figure 7 included only the Prompts Tree. The full one-time setup (Automotive, measured tokens, Claude-3.5-Haiku prices as in Table 7) and the number of queries after which Cochain's total cost, setup included, falls below each baseline:

StepCallsTokensCost
Explicit triple extraction72,59864.5M$84.8
Counterfactual question generation41,3836.4M$9.0
Stage-agent answers (local model)41,38317.4MGPU only
Evaluator loop (repeated checking)41,38326.9M$32.1
Tacit triple extraction41,38332.5M$44.9
Prompts Tree—0.62M$2.2
Total (API)≈131M≈$173
Optional stage-agent LoRA3.9 GPU-h per stage
Break-even (queries)PMCMedAgentsDebate (short)Debate (long)CoA
Automotive5,19712,0188,7857,8666,115
Pharmaceutical17,65913,21112,27410,55211,935
E-commerce5,40812,3617,4595,9276,248

The cost is paid once; the same knowledge graph and Prompts Tree then serve every backbone in the paper and every later query, whereas interaction-heavy baselines pay on every query and for every backbone. Past break-even each query is ≈10× cheaper (0.0026 vs. 0.0246–0.0359 USD per query on Automotive). The largest term, explicit extraction, is the standard cost of any KG-based RAG system, and construction is offline, parallel and incremental.

Area_Chair_f4FEOpenReview ↗

Does authors' response address your concerns?

Thank you for your review! Could you check the authors’ response and clarify whether it addresses your original concerns? Many thanks!

AC

Reviewer_Gu59OpenReview ↗

Response

Thank you for the response. The matched-information retrieval baselines and the controlled debate experiment address my concerns about the baseline comparison and excessive collaboration. My remaining concern is whether the gains translate into more feasible workflows, since the systems in the expert evaluation had access to different information. I’m keeping my rating at 2.

Reviewer_coo8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cochain, a framework intended to balance insufficient and excessive collaboration in LLM-based business workflows. Cochain moves part of the collaboration process offline by constructing a collaborative knowledge graph and a tree of reusable, stage-specific prompts. The knowledge graph combines triples extracted from training data with additional knowledge elicited through counterfactual questioning of specialized agents. At inference time, the framework retrieves relevant knowledge, cross-stage graph paths, and prompt chains to guide response generation. Experiments cover several business domains and model backbones, with additional ablations, expert evaluations, and efficiency analyses.

Critical strengths
  1. A practically relevant problem and a sensible approach to reusing collaboration. Repeated multi-agent discussions can be expensive, particularly when similar questions recur within a relatively stable business workflow. Distilling stage-specific knowledge and guidance into reusable artifacts is a reasonable way to reduce repeated inference costs while retaining information from other stages.

  2. Positive empirical evidence across multiple settings. The paper evaluates several domains and backbones and provides component ablations, human evaluation, and an additional execution-oriented benchmark. Some improvements are substantial: for example, on the e-commerce dataset with DeepSeek-R1-7B, Cochain improves ROUGE-L from 42.03 for CoA to 47.07. Removing the prompt tree also generally reduces performance, suggesting that the retrieved guidance contributes meaningfully to answer quality.

Critical weaknesses
  1. The source of improvement is not fully isolated. Cochain benefits from additional knowledge and distilled prompts, but comparisons against information-matched retrieval baselines are missing. A single agent retrieving the same Q&A data or a flat repository of the same prompts would help establish the contribution of the proposed structure.
  2. Some claims exceed the available evidence. Text-similarity metrics do not directly establish workflow feasibility or constraint satisfaction. Auto-SLURP gains over the strongest baseline are only one percentage point, without uncertainty estimates. Claims about balancing collaboration and improving reliability should therefore be stated more cautiously.
Questions for authors
  1. Do all baselines have equivalent access to training-derived knowledge? How does Cochain compare with simpler retrieval methods using the same information?
  2. What are the Auto-SLURP evaluation size and human-evaluation protocol?
  3. Does the cost analysis include all offline construction steps? Please also reconcile the apparent 300-versus-500-query inconsistency in Figure 7.
3 comments
AuthorsOpenReview ↗

Response to Reviewer coo8 (Part 1/2)

Thank you for the positive assessment and the precise questions.

[W1 & Q1] Equivalent access to the training-derived knowledge; simpler retrieval over the same information

All baselines in Table 1 already run on the same fine-tuned stage agents as Cochain, so they share its parametric knowledge; they lack only Cochain's artifacts. To isolate the effect of structure, we added information-matched baselines that draw on the same sources Cochain is built from, each with the same role prompt and a per-query budget equal to Cochain's injected context (Automotive, Qwen2-7B design agent, 3 seeds, all SE ≤ 0.17):

Context given to the same backboneBS-FGLEUROUGE-L
IO (question only)71.8221.7524.32
Role prompt only72.1720.2923.74
BM25 over training Q&A71.7021.5023.90
Dense retrieval over training Q&A71.6621.3023.77
Flat retrieval of the same CKG triples72.7320.0723.66
Graph-RAG over the same CKG (2-hop, no stage-aware bridging)72.4919.5823.39
Flat repository of the same Prompts-Tree prompts72.1618.8222.99
Hybrid of the three flat sources72.3421.0624.12
Cochain74.1625.0426.00

Cochain wins every comparison (paired t-tests, p < 1e-14). The ordering is the same for the base Qwen2-7B (Cochain 74.50/24.70/25.61 vs. ≤72.87/20.55/23.76) and for Qwen3.8-Max (69.10/12.10/15.37 vs. ≤68.12/10.37/14.19). Flat insertion of the same knowledge even falls below IO on GLEU, and the flat prompt repository is the weakest variant: in the tree the same prompt appears under many upstream decisions (33,769 nodes, only 1,420 distinct prompts), so its meaning depends on its path, which a flat repository loses.

Giving PMC and Debate the same CKG triples does not close the gap either (Pharmaceutical, Qwen2-7B):

SystemBS-FGLEUROUGE-L
PMC → + same CKG triples66.97 → 67.5116.25 → 16.4324.75 → 25.11
Debate → + same CKG triples70.62 → 70.2916.04 → 16.3724.12 → 24.28
Cochain78.5034.4835.23

[W2] Claims vs. evidence (constraint satisfaction; Auto-SLURP margin)

We now measure constraint satisfaction directly. From each expert reference we extract the cross-stage constraints the decision must respect (6.2 per question); two judges from non-Qwen families (DeepSeek-V4-Pro, GLM-5.3) blindly label each as covered (1), partial (0.5) or missing and estimate off-focus content (inter-judge Spearman ρ = 0.84). Pairwise comparisons show the judge the reference and constraint list, in both orders (Qwen2-7B backbone):

SystemConstraint coverage ↑Off-focus ↓Cross-stage consistency (1–5) ↑Cochain win / tie / loss (%)
IO20.9%13.4%2.2175.0 / 22.0 / 3.0
Dense retrieval (same sources and budget)23.4%15.4%2.4663.2 / 28.7 / 8.0
Flat-KG (same triples)21.9%23.1%2.1976.0 / 20.8 / 3.2
Hybrid-flat22.1%21.6%2.2274.0 / 21.8 / 4.2
CoA20.2%20.2%2.2180.5 / 16.8 / 2.8
Cochain40.6%20.5%2.82—

Cochain covers about twice as many expert-required constraints as any other system and loses at most 8% of comparisons.

On Auto-SLURP the absolute margin over CoA is small, so we added two checks (official 100-instance test split and judge; 95% Wilson CI for Cochain, exact McNemar p vs. Cochain). First, a third backbone, GPT-5.4-mini, with the same judge (last row):

BackboneCochainCoAPMCMedAgents
DeepSeek-V3.10.33 [0.25, 0.43]0.27 (p=0.39)0.12 (p<0.001)0.07 (p<0.001)
Claude-3.5-haiku0.38 [0.29, 0.48]0.38 (p=1.0)0.29 (p=0.18)0.26 (p=0.07)
GPT-5.4-mini0.33 [0.25, 0.43]0.27 (p=0.33)0.20 (p=0.01)0.19 (p=0.01)

Cochain is again best with GPT-5.4-mini (6 points above CoA, as with DeepSeek-V3.1) and significantly better than PMC and MedAgents. Second, re-judging all DeepSeek-V3.1 runs with a single stricter judge (DeepSeek-V3.1-Terminus) lowers every accuracy but keeps the ranking (Cochain 0.17, CoA 0.10, PMC 0.05, MedAgents 0.03); Cochain's relative lead over CoA grows from 1.2× to 1.7×, so it does not depend on a lenient judge. Overall, Cochain is best or tied-best on all three backbones.

Following your suggestion, the revised PDF states every claim in terms of this evidence: "balancing collaboration" is supported by the constraint-coverage and controlled-communication results, the Auto-SLURP result is described as best-or-tied accuracy, and unsupported wording ("robust", "reducing hallucinations") is removed.

AuthorsOpenReview ↗

Response to Reviewer coo8 (Part 2/2)

[Q2] Auto-SLURP size and human-evaluation protocol

Auto-SLURP: the official 100-instance test split and judge; Cochain's knowledge graph and prompt trees are rebuilt from the official 1,000-instance training split with the same procedure, and no automotive knowledge is transferred.

Human evaluation: a blind questionnaire to five engineers of a major automobile manufacturer. For each question they saw answers A and B (system identities hidden, assignment randomized per question) and chose A better / B better / similar by one criterion: which answer, while answering the question, better accounts for collaboration across design, supply chain, production and quality inspection. GPT-4 answers without Cochain's artifacts, since the study asks whether a small model with Cochain can match a much larger model; the equal-knowledge comparisons are given above.


[Q3] Are all offline steps in the cost? 300 vs. 500 in Figure 7

The left group in Figure 7 is computed for 300 queries; "500 Eval Q" is a typo. Each bar is 300 × the per-query values in Table 7 (e.g., CoA: 8,944 tokens, 0.0182 USD and 253.18 s per query → 2.68M tokens, 5.46 USD and 75,954 s), and the right group is 1,000 × the same values. Figure 7's setup bar covers only the Prompts Tree; the full one-time setup (Automotive, measured tokens, Claude-3.5-Haiku prices as in Table 7) is:

StepCallsTokensCost
Explicit triple extraction72,59864.5M$84.8
Counterfactual question generation41,3836.4M$9.0
Stage-agent answers (local model)41,38317.4MGPU only
Evaluator loop41,38326.9M$32.1
Tacit triple extraction41,38332.5M$44.9
Prompts Tree—0.62M$2.2
Total (API)≈131M≈$173

The optional stage-agent LoRA takes 3.9 GPU-h per stage. This cost is paid once and the artifacts are reused by every backbone and query, whereas interaction-heavy baselines pay per query and per backbone: the setup is recovered after 5.2k–7.9k queries on Automotive (10.6k–17.7k Pharmaceutical, 5.4k–6.2k E-commerce), after which each query is ≈10× cheaper.

Area_Chair_f4FEOpenReview ↗

Does authors' response address your concerns?

Thank you for your review! Could you check the authors’ response and clarify whether it addresses your original concerns? Many thanks!

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for engaging closely with Cochain. All three reviewers agree the problem is practically relevant (coo8, Gu59, 7RUg), that reusing distilled collaboration instead of repeated multi-agent dialogue is sensible (coo8, Gu59), and that the empirical coverage is broad (coo8, Gu59). The shared questions concern (i) whether the gains come from Cochain's structure or simply from extra material, (ii) evidence beyond reference-overlap metrics, (iii) a controlled demonstration of excessive collaboration, and (iv) the full setup cost. New experiments answer each directly:

  1. The gain comes from structure, not from more information. Information-matched baselines use the same sources (BM25 and dense retrieval over the same training Q&A, flat and multi-hop Graph-RAG over the same knowledge graph, a flat repository of the same prompts, and a hybrid) with the same budget and role prompt. Cochain beats the best of them by +1.4 BS-F / +3.3 GLEU / +1.7 ROUGE-L (p < 1e-14), and it remains best on every metric with GPT-5.4-mini, Claude-Sonnet-4.5 and Qwen3.8-Max.
  2. Constraint satisfaction and excessive collaboration. Judged against the cross-stage constraints extracted from expert references, Cochain covers 40.6% of them vs. ≤23.4% for every other system, losing at most 8% of pairwise comparisons. With information access held fixed, each extra round of multi-agent debate lowers every metric while tokens grow from 4.9k to 69k per query.
  3. Full cost and generalization. The complete one-time setup costs ≈131M tokens (≈$173), is reused by every backbone and query, and pays for itself after ≈5.2k–17.7k queries. Auto-SLURP is reported with confidence intervals and paired tests.

We will upload a revised PDF with all new results and clarified method descriptions, stating every claim in terms of this evidence, and correcting Figure 7 (axis label), Table 3 and the Figure 10 caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.