acceptodds
Withdrawn from ICLR 2027

ProtocolBench-Collab: Renaming a Role Changes What a Collaborative Language Model Answers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Collaborative systems built from language models, such as actor–critic pipelines, are evaluated through the role interface their designers wrote, and that interface is rarely varied. We show that their answers depend on it. PROTOCOLBENCH-COLLAB holds each question fixed and changes only the role interface, across nine renderings that range from a neutral change of role to a collaborator who prefers a named wrong option. All eight untrained models we evaluate, from four families, change their answers with the interface, and the strongest remains inconsistent on a fifth of items. We then train a shared actor–critic adapter on preference pairs each presented under four interfaces. This narrows the accuracy spread across interfaces from ten points to one and a half and raises the share of items answered identically under all nine from 65% to 73%. Presenting each pair four times also quadruples the training steps, and a control with the same number of steps and a single interface does not improve agreement (+0.6, t=1.23), which identifies interface variety as the source of the gain in agreement. Enlarging the adapter improves robustness further, and the two effects combine.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent collaboration, prompt sensitivity, role interfaces, preference optimization, protocol robustness, collaborative language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HLLiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper measures how much an LLM's answer in an actor-critic setup changes when only the role wording of the prompt changes. It uses 2000 multiple-choice items under nine prompt variants, and finds that eight instruction-tuned models (0.5B to 14B) are all sensitive.

Critical strengths

The question is relevant, since role prompts in multi-agent pipelines are rarely ablated.

Critical weaknesses

W1. The paper is very hard to follow. The task is never properly introduced, background is assumed, and the text relies on a large set of coined terms that are defined late, loosely, or not at all. Some examples:

  • The paper never states clearly what a "collaborative system" does at evaluation time. The abstract and introduction talk about actor-critic pipelines. Only in the "Endpoint" paragraph of Sec. 3 does the reader learn that every system is scored by a single forward pass on a single prompt.
  • Three near-synonyms are used for the prompt wording around a question. "Rendering" (Sec. 3) refers to the nine evaluation prompts, "frame" (Sec. 4) to the four training prompts, and "role interface" is used for both. The reader has to work out that these are the same kind of object.
  • Further terms are introduced with little or no definition: "endpoint", "anchor", "harmful revision", "trap families" and "answer contract", "mean pairwise view dissimilarity", "Protocol-Mix".
  • The pair-mining procedure is described only by reference ("we follow the trajectory construction of ACC-Collab"). The "guided versions", "steered toward y and away from it", are never explained. It is not stated how the steering is done.
  • Figure 2 packs three separate diagrams into one and is hard to parse without first reading Sec. 4. The rest of this review reflects what I could reconstruct, but I think the paper needs a substantial rewrite before it can be properly assessed. I would be glad to review a revised version.

W2. he paper relies on what the authors call "a fixed answer cue" and that in the field is known as "structured response". This is known to affect the response of the models, and alter them[1], which I believe to unfotunately invalidate the results obtained by the authors.

W3. The method is trained and analysed on one model, Qwen2.5-3B, with pairs mined from its own trajectories. The cross-model evaluation stops at 14B and includes no reasoning models. Because the endpoint allows no generation, the benchmark cannot evaluate reasoning models at all, even though current actor-critic systems are mostly built on them. The evaluation is also is limited to only 2000 samples, from a dubious mixture of benchmarks.

W4. The base model's accuracy on the six neutral prompts lies between 69.3 and 70.1 (Table 6). The 10-point spread therefore comes almost entirely from the "overconfident" prompt. The volume control alone, with one interface and four times the steps, already brings the spread to 2.13 and worst-case accuracy to 68.74, against 1.49 and 69.68 for the full method. Variety adds +0.6 average accuracy and +4.1 agreement, of which 1.07 is due to prompt length (App. L). It seems that the gains are rather small, and maybe not coming from the proposed method. More testing would be necessary to quantify the improvement.


[1] Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7407–7416, Bangkok, Thailand. Association for Computational Linguistics.

Questions for authors

See above

Reviewer_v4iGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a practical reliability issue in collaborative LLM systems: the same question may receive different answers when only the role interface changes. The authors introduce ProtocolBench-Collab, which evaluates 2,000 multiple-choice questions under nine role settings, including neutral role changes and misleading collaborator preferences. They also propose Protocol-Mix CoLoRA, which trains a shared actor–critic LoRA adapter using preference pairs presented in four different role frames. The experiments show that this training reduces answer variation across interfaces and improves worst-case performance on the tested benchmarks. The paper argues that collaborative LLM systems should be evaluated not only by average accuracy, but also by consistency under reasonable protocol variations.

Critical strengths
  1. The paper highlights a source of instability that is easy to overlook: developers choose the role prompts, but those prompts can affect an LLM’s answer. The benchmark tests this directly. It keeps each question fixed and changes the role wording. Requiring a single-letter answer also avoids scoring errors caused by inconsistent output formats. The authors test neutral, adversarial, and misleading role settings.
  2. The experiments help separate the effect of prompt variety from the effect of additional training. They also test different prompt lengths. The evaluation goes beyond average accuracy: it includes worst-case accuracy, answer consistency, harmful revisions, and resistance to misleading prompts. Harmful revision is especially useful because it tracks cases where a prompt change turns a correct answer into a wrong one.
  3. The paper provides fixed prompt templates, deterministic item selection, per-item predictions, and training records. It also includes checks on the evaluation metrics. These details make the results easier to inspect and reproduce.
Critical weaknesses
  1. The benchmark covers a limited range of tasks. All experiments use multiple-choice questions and require a single-letter answer. This makes scoring straightforward, but it leaves open whether the same prompt effects occur in open-ended reasoning, coding, planning, tool use, or multi-turn agent tasks. The role prompts are also designed by the authors. The nine versions are a useful starting point, though they represent only a small set of possible real-world protocols.
  2. The proposed training method is tested in a narrow setting. Protocol-Mix CoLoRA is evaluated only with Qwen2.5-3B-Instruct and actor–critic preference data. The paper shows that several models are sensitive to role changes, but it does not establish that this training method works with other model families, larger models, or different agent setups. The held-out prompt test is useful, but it remains close to the original task.
  3. The gains are clearer for consistency than for reasoning quality. The method reduces answer variation and improves worst-case accuracy. Accuracy under neutral prompts, however, changes only modestly. Greater agreement is not always an improvement if the model consistently gives the wrong answer.
Questions for authors
  1. Does ProtocolBench-Collab show similar results beyond multiple-choice questions?
  2. How closely do the nine role interfaces reflect real collaborative systems? Could you also test prompts from existing agent frameworks or real applications?
  3. Does Protocol-Mix CoLoRA work with other models and agent setups, such as testing another model family, or a setup?
Reviewer_eZwtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the design of collaborative systems, where language models take turns in different roles to come to a final result (e.g. answer through actor/critic roles). This work aims to show that the final answer of a collaborative system depends on the role. They do this by constructing a benchmark based on questions from standard multiple choice benchmarks (such as MMLU, ARC-Challenge, HellaSwag, Winogrande) that allows to test this hypothesis. They further introduce a training method that reduces interface variance when roles are varied.

Critical strengths

None

Critical weaknesses

Completely outdated setup. For better or for worse, AI research is a fast-moving field and this paper is severely outdated, and I do not think any current work could directly build on it. To start, at a high level this paper only cites work from 2024 or older (with a single 2025 exception, and even that one was first published in 2024), for ICLR 2027. I do not find it believable that only a single piece of related work was published since 2024, and none in 2026. In a similar spirit, the model that is fine-tuned is also no longer in wide-spread use, and from 2024. Most importantly, many of the benchmarks they use as the basis of their benchmark are fully saturated at this point (e.g. MMLU) and no longer in use by most of the community. Thus, the proposed benchmark itself is very unlikely to provide any meaningful insights in current models. I don't see any value in publishing this paper at ICLR 2027. I encourage the authors to completely update this work if they are considering a re-submission to another conference, as it stands I consider this submission a waste of reviewer time.

Questions for authors
  1. Would you be able to demonstrate that this work provides meaningful insight into modern models?
  2. Can you put your work into context of more recent related work, e.g. arxiv.org/abs/2503.13657, arxiv.org/abs/2510.27062, arxiv.org/abs/2508.17536

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.