- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Cochain, a collaboration prompting framework for balancing insufficient and excessive collaboration in LLM agent workflows. Single-agent approaches can miss cross-stage constraints, while multi-agent systems can consume substantial tokens and dilute the main problem. Cochain combines knowledge and prompts to address these issues at a reduced cost. It constructs a collaborative knowledge graph containing explicit and tacit knowledge from multiple workflow stages, organizes retrieved knowledge into causal chains, and retrieves prompt chains from a prompts tree to guide reasoning. The paper evaluates Cochain across multiple datasets and LLM backbones, reporting improvements over prompt engineering and multi-agent baselines.
Critical strengths
The problem is meaningful. Bringing in useful knowledge from other stages while avoiding unnecessary discussion could improve both answer quality and efficiency.
Critical weaknesses
-
Several key steps are not clearly explained, including knowledge extraction, the meaning and use of Eq. 3, and prompt-chain retrieval. It is also unclear why connected paths in the knowledge graph should be called causal chains.
-
The experiments do not clearly show how Cochain balances collaboration for different queries. The paper includes prompt-budget experiments and examples, but does not directly show which necessary constraints are recovered or which irrelevant details are removed.
-
The Auto-SLURP transfer setup and component ablations need clearer explanations. I would also like to see results with newer models, such as GPT-5.x or Claude 4.x, and better justification of the GPT-4 judging setup.
Questions for authors
-
How are the triples in Eq. 1 extracted from the input-output pairs? What determines the entities and relation types?
-
Why should counterfactual prompting reveal valid hidden dependencies? The appendix gives examples, but how are these outputs checked against reliable references? What counts as an ideal collaboration output?
-
What does Eq. 3 contribute to the actual method? How are its latent variables used in knowledge extraction?
-
Why are the paths in Eq. 7 causal chains? Connecting nodes through a shared entity does not seem sufficient to establish causality.
-
Is there one prompts tree for all Q&A pairs, or a separate tree for each seed? Why is a tree needed instead of directly storing prompt chains? For a new query, how is the matching path selected? The mention of DFS does not explain this selection.
-
What exactly is removed in each ablation? If the knowledge graph is removed, how are causal chains still constructed? Is only one part of the final prompt being removed?
-
What is transferred to Auto-SLURP, and what is rebuilt using its data? Without these details, it is difficult to tell what kind of generalization this experiment tests.
5 comments
Response to Reviewer 7RUg (Part 2/3)
[Q5] One Prompts Tree or one per seed? Why a tree? How is the path selected?
One tree per domain: a virtual "user need" root whose children are the seed user needs (505 for Automotive), each expanding into design → supply chain → production → quality-inspection prompts (33,769 nodes). A tree is needed because prompts are conditional on upstream decisions: the 33,769 nodes contain only 1,420 distinct (stage, prompt) strings, so a prompt's meaning is given by its path, and siblings under the same upstream decision can be merged at retrieval. Flattening the same prompts into a repository is the weakest information-matched variant (72.16/18.82/22.99 vs. 74.16/25.04/26.00). Retrieval: (1) match the query to the most similar seed user need; (2) enumerate the root-to-leaf paths below it depth-first; (3) merge paths sharing the same design → supply → production prefix; (4) insert the top-k merged chains (k = 3 by default; Fig. 5) in stage order.
[Q6] What is removed in each ablation?
Yes, each ablation removes one part of the final prompt. Cochain first selects one-hop CKG facts, composes cross-stage causal chains and retrieves stage-ordered prompt chains, then assembles them as three blocks: [Knowledge], [Causal Chains] and [Workflow Prompts]. Each ablation deletes exactly one block and keeps everything else fixed; causal chains therefore remain in the w/o CKG row, which isolates the one-hop facts. Whether the organization itself matters is tested by information-matched baselines that give the same backbone the same material without Cochain's organization (Automotive, Qwen2-7B, 3 seeds): flat CKG triples 72.73/20.07/23.66, Graph-RAG 72.49/19.58/23.39, flat prompts 72.16/18.82/22.99 and their hybrid 72.34/21.06/24.12, all below Cochain's 74.16/25.04/26.00 (p < 1e-14).
[Q7] What is transferred to Auto-SLURP and what is rebuilt?
Only the procedure is transferred (extraction prompt, Prompts-Tree algorithm, retrieval and prompt composition). The knowledge graph (triples linking queries, intents, slots and actions) and prompt trees are rebuilt from the official 1,000-instance training split, and evaluation uses the official 100-instance test split and judge. The experiment tests whether the procedure generalizes to executable multi-step assistant workflows, not whether automotive knowledge transfers.
[W2] Which constraints are recovered and which irrelevant details are removed
We now measure this per question: two independent judges label each expert-required constraint in each answer as covered / partial / missing and estimate the share of off-question content (Qwen2-7B):
| System | Constraint coverage ↑ | Off-focus ↓ |
|---|---|---|
| IO | 20.9% | 13.4% |
| Dense retrieval (same sources) | 23.4% | 15.4% |
| CoA | 20.2% | 20.2% |
| PMC / Debate (long) | 3.5% / 3.7% | 95.1% / 91.5% |
| Cochain | 40.6% | 20.5% |
Example (closed-loop recycling in vehicle design, 7 expert constraints): Cochain covers modular, disassembly-friendly design and also recyclability-driven material selection, supplier-side material verification, material-utilization targets, process stability and incoming/final inspection (off-focus 10%); IO covers only the two design-side constraints (off-focus 30%). PMC and Debate answers are over 90% off the question, the dilution Cochain avoids.
Response to Reviewer 7RUg (Part 3/3)
[W3] Newer models; GPT-4 judging
We ran every information-matched variant on three current backbones with the same sources and budget (Automotive; GPT-5.4-mini with reasoning off, Qwen3.8-Max with thinking off; BS-F / GLEU / ROUGE-L):
| Context | GPT-5.4-mini | Claude-Sonnet-4.5 | Qwen3.8-Max |
|---|---|---|---|
| IO (question only) | 66.46 / 10.30 / 13.53 | 65.23 / 9.77 / 12.18 | 64.98 / 7.94 / 11.51 |
| Role prompt only | 66.24 / 10.06 / 12.68 | 65.02 / 6.18 / 9.00 | 66.41 / 9.08 / 13.03 |
| Dense retrieval over training Q&A | 66.76 / 10.63 / 13.50 | 66.80 / 9.49 / 12.35 | 67.42 / 9.83 / 13.72 |
| Flat retrieval of the same CKG triples | 67.37 / 11.15 / 13.90 | 66.88 / 9.62 / 12.21 | 68.12 / 10.37 / 14.19 |
| Graph-RAG over the same CKG | 67.11 / 10.93 / 13.72 | 67.07 / 9.55 / 12.41 | — |
| Flat repository of the same prompts | 66.83 / 10.72 / 13.35 | 66.82 / 8.93 / 11.95 | 68.07 / 10.26 / 14.04 |
| Hybrid of the three flat sources | 66.98 / 10.87 / 13.56 | 66.96 / 9.78 / 12.63 | 67.96 / 10.27 / 14.15 |
| Cochain | 67.83 / 12.03 / 14.30 | 67.25 / 10.09 / 12.74 | 69.10 / 12.10 / 15.37 |
Cochain is best on every metric with all three backbones (GPT-5.4-mini: p ≤ 0.003 against every row).
On judging: in the submission, GPT-4 scoring complements the blind expert study and applies the same criterion: an answer must first answer the question correctly and is then judged on how well it accounts for collaboration across the workflow. GPT-4 scores only Qwen2-7B outputs under four conditions (raw and fine-tuned, each with and without Cochain; Fig. 3b), five times each with the scores averaged, so it never scores its own answers, and its ranking agrees with the expert study (Fig. 3a).
To address the concern directly, we added a stricter judge-based evaluation: two judges from families different from every backbone (DeepSeek-V4-Pro, GLM-5.3) score each answer blindly against the cross-stage constraints extracted from the expert reference, with a fixed rubric, constraint-level labels and position-swapped pairwise comparisons (inter-judge ρ = 0.84). The advantage holds on the current models:
| GPT-5.4-mini | Claude-Sonnet-4.5 | Qwen3.8-Max | |
|---|---|---|---|
| Constraint coverage: Cochain / best matched variant / IO | 31.6% / 24.6% / 23.6% | 54.3% / 37.9% / 26.9% | 51.6% / 34.2% / 23.5% |
| Pairwise, Cochain win / loss vs. IO | 53.8% / 2.2% | 68.0% / 0.5% | 69.5% / 1.5% |
| Pairwise, Cochain loss vs. any matched variant | ≤ 9.8% | ≤ 4.5% | ≤ 5.5% |
In the revised PDF, the small-vs-large claim is stated in terms of the expert study.
Response to Reviewer 7RUg (Part 1/3)
We answer each question with the exact procedure used in our implementation.
[Q1] How are the triples in Eq. 1 extracted? What determines entities and relations?
Each (X, Y) pair is processed by an LLM with a fixed few-shot extraction prompt (released in our repository): the head entity comes from the keywords of X, the tail from Y, and the relation is an open-vocabulary phrase for the method or dependency linking them, e.g., (fuel economy, improvement depends on, lightweight body design). Surface variants are merged by entity normalization (Fig. 2), and each node carries the stage label of its dataset; nodes with ≥2 stage labels become bridge entities. Entities and relations are thus determined by the workflow data, not by a fixed schema (Automotive: 414,878 explicit triple statements from 72,598 Q&A pairs, plus tacit triples from 41,383 counterfactual pairs).
[Q2] Why does counterfactual prompting reveal valid dependencies? How are outputs checked? What is an ideal collaboration output?
A factual question ("how to improve fuel economy?") elicits the default answer, where feasibility constraints stay implicit; perturbing one condition while preserving intent ("…if the existing stamping line and supplier base cannot change?") forces the agent to state what must change downstream. Validity is checked at three levels: (i) every answer passes an evaluator loop (reasonable / ambiguous / unreasonable, with revision feedback) before extraction; (ii) a blind expert audit of 2,000 sampled triples, half counterfactual (Table 15: correctness 4.99/5, relevance 4.76/5, depth 4.31/5); (iii) removing only the tacit triples lowers performance (Pharmaceutical, Qwen2-7B: 78.48/33.98/35.03 → 77.17/31.26/31.99). An ideal collaboration output satisfies the cross-stage constraints experts regard as necessary while staying focused on the decision. We now measure both: against constraints extracted from expert references, Cochain covers 40.6% vs. ≤23.4% for every other system (51.6% vs. ≤34.2% with Qwen3.8-Max).
[Q3] What does Eq. 3 contribute?
Eq. 3 is a modelling device, not a computational step. Tacit knowledge is by definition not written in the data, so Eq. 3 makes it a precise object: a latent variable θ that shapes the agent's internal state h, which shapes its answer to a counterfactual input. This explains why counterfactual probing reveals it: θ and h are unobservable, but changing one condition while keeping the intent changes the answer through exactly the considerations that condition activates, and these are what we extract as tacit triples from (X̃, Ỹ). The latent variables are not estimated; they define tacit knowledge, explain why elicitation can surface it, and motivate the design that Algorithm 1 implements (one perturbation at a time, intent preserved, evaluator-checked answers).
[Q4] Why are the paths in Eq. 7 "causal" chains?
Standard retrieval returns discrete facts and leaves the model to work out how they relate. A causal chain connects them into a path whose edges are directed cause–effect or dependency relations (extracted with causal cues such as "depends on", "requires", "increases the risk of"). For "How can ergonomics be used to reduce traffic-accident injuries?", keyword retrieval finds facts about ergonomics and about injuries but nothing relating them, whereas multi-hop expansion yields Ergonomics → applied to → Cockpit Design → influences → Driver Fatigue → increases risk of → Vehicle Collision → results in → Injuries. Across stages, bridge entities link a design choice to a production constraint, e.g., Carbon fiber → requires → complex molding → requires → new curing equipment → must be compatible with → the existing stamping line. We call these paths causal chains because every link states how one element causes, requires or affects the next; the term does not claim statistical causal identification, and a shared entity alone does not make a chain.
This cross-stage linking is what makes the paths useful: a standard multi-hop Graph-RAG over the same graph (2-hop from the 3 most similar entities, no stage-aware bridging, same budget) scores 72.49/19.58/23.39 vs. 74.16/25.04/26.00 for Cochain (Automotive, Qwen2-7B, 3 seeds, p < 1e-22), below even flat retrieval of the same triples (72.73/20.07/23.66), because unconstrained expansion drifts within one stage instead of reaching supply-chain and production constraints.