acceptodds
Under review as a conference paper at ICLR 2027

ConCord: Distilling and Coordinating Persona Skills from Multi-User Work Records

Abstract

Persona skills distill working practices and preferences from human work records into reusable instructions for Large Language Model (LLM) agents. Recent methods combine records from multiple users to provide broader knowledge, but different users may recommend incompatible actions for the same situation. Existing methods do not explicitly preserve and coordinate such conflicts, which can leave agents uncertain about which practice to follow. We present ConCord, an automated framework for constructing and coordinating persona skills from multi-user work records. ConCord first extracts context-dependent decision rules from each user's records, and then groups users with similar practices and constructs one skill for each group, sharing knowledge across users while preserving conflicting approaches in separate skills. For each task, separate LLM agents independently generate candidate responses using different skills. A merging LLM then combines the responses and resolves conflicting recommendations based on the task requirements. We evaluated ConCord on four datasets of user work records under different scenarios that cover workplace communication, software issue handling, and computer-science question answering. ConCord achieves an average score improvement of over responses generated without skills, compared with for the strongest baseline. These results show that preserving conflicting practices during skill construction and resolving conflicts for each task improves multi-user persona skill coordination. We release the replication package at the anonymous link: https://anonymous.4open.science/r/ConCord-F060.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Persona Skill, Multi-User Skill Coordination, Conflicting Working Practices

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ha6XOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ConCord learns reusable instructions from several users’ work records. It groups users with similar working practices and builds one skill for each group. Each skill drafts an answer, and an LLM combines the drafts for the task. The paper tests this approach on four datasets with four base models.

Critical strengths
  • The paper addresses a useful problem. Users may prefer different actions in the same situation. ConCord keeps these differences in separate skills and combines their answers based on the task. The paper gives concrete examples of why simply putting conflicting rules into one skill can fail.
  • The evaluation covers several settings. The paper tests four datasets and four base models, compares five baselines and several conflict-handling methods, and reports computational cost. The full prompts help readers understand the method. The results also show that keeping more information does not always lead to better answers.
Critical weaknesses
  • Extra computation may explain part of the improvement. Table 3 shows that ConCord uses four calls and 7,400 tokens per answer, while each baseline uses one call and fewer tokens. A shared-skill baseline with the same drafting and merging steps and a similar token budget would help show how much user grouping adds.

  • The LLM judge has not been checked against human scores. Quality, resolution rate (RR), and conflict-exposed information loss (CIL) all use Claude Haiku 4.5. A human check of a sample, along with a second judge, would help show whether the reported rankings are reliable.

  • Some counts and scores need explanation. Appendix C allows at most six content elements per task, but Table 7 lists 384 elements for 42 Enron tasks. Repeated items across user sets may explain this, but the counting rule is unclear. Table 2 and Figure 4 also report different Enron quality scores for ConCord, 0.78 and 0.83. The paper should explain which runs or settings produced these values.

  • The results do not show uncertainty. The paper reports no confidence intervals or variation across repeated runs. Since some gains are small and several steps use random sampling, these checks would help readers judge which improvements are stable.

Questions for authors
  1. Does ConCord still improve over a shared skill when both use the same drafting and merging steps and similar token budgets?
  2. Do human reviewers or another LLM judge agree with the reported quality, RR, and CIL scores?
  3. What counting rules and run settings explain the Table 7 totals and the score differences between Table 2 and Figure 4?
  4. How much do the main results vary across repeated runs?
5 comments

Q1 & W1. Does ConCord still improve over a shared skill with the same drafting and merging steps and a similar token budget? Thank you for this suggestion. Yes: at the same number of calls (4) and a similar token cost (7,124 vs. 7,400), ConCord outperforms the shared skill by 0.13 on 20-MAD and 0.09 on Enron. To show this, we added four controls with GPT-4.1-mini on 20-MAD and Enron.

  • ConCord w/o grouping (k = 1, 3 drafts + merge): the shared-skill baseline the reviewer asks for. ConCord builds a single skill from all users, generates 3 drafts with it, and merges them using ConCord's merging step. This matches ConCord's number of calls (4) and token cost (7,124 vs. 7,400) and removes only the grouping.
  • ConCord w/o grouping (k = 1, single call): the same shared skill generates one answer.
  • Nvwa-Ens (3 drafts + merge): Nvwa's skill generates 3 drafts that are then merged, matching ConCord's number of calls with a similar token cost (6,881).
  • Nvwa-Ens (4 drafts + merge): the same with 4 drafts, using more tokens than ConCord (9,027).
MethodCallsTokens/task20-MAD Quality ScoreEnron Quality Score
Nvwa (single call)11,444+0.62+0.24
Nvwa-Ens (3 drafts + merge)46,881+0.56+0.55
Nvwa-Ens (4 drafts + merge)59,027+0.51+0.61
ConCord w/o grouping (k = 1, single call)11,528+0.84+0.67
ConCord w/o grouping (k = 1, 3 drafts + merge)47,124+0.80+0.69
ConCord (ours)47,400+0.93+0.78

Grouping vs. a shared skill. With the pipeline and drafting budget fixed, removing only the grouping lowers the score from 0.93 to 0.80 on 20-MAD and from 0.78 to 0.69 on Enron.

Extra drafts alone do not explain the gain. Adding drafts without grouping does not consistently help, for either the shared skill (0.84 → 0.80 on 20-MAD and 0.67 → 0.69 on Enron) or Nvwa (0.62 → 0.56 and 0.24 → 0.55). With the same number of calls and a similar token cost, ConCord outperforms Nvwa-Ens by 0.37 on 20-MAD and 0.23 on Enron, and it remains ahead even when Nvwa-Ens uses more tokens (+0.42 and +0.17).

Thus, the benefit of ConCord's additional calls comes from drafting with different group-specific skills, rather than from generating and merging more drafts.

W2 & Q2. Do human reviewers or another LLM judge agree with the reported scores? We thank the reviewer for this suggestion. We added a human evaluation and a second LLM judge (GPT-4.1). In short, on the conflict judgments behind RR and CIL, our judge agrees with humans as often as humans agree with each other. On answer quality, humans and both judges rank ConCord above the baselines, but agreement on individual pairs is low and the human margin is smaller.

Both evaluations use the same items: 200 rule pairs ("do these two rules conflict?") and 100 answer pairs sampled from Table 2 (a method's answer vs. the no-skill answer, "which is better?"; 50 from ConCord and 10 from each baseline). RR and CIL are computed from the conflict judgments, so we check these judgments directly. Score is +1 if the method's answer is better, −1 if worse, and 0 if tied, averaged as in Table 2.

(1) Human judges. Four annotators, blind to method names and to the judge's labels, labeled the items, with two annotators per item. "Judge vs. human" is the share of human labels that match the label of our judge (Haiku 4.5). "Human vs. human" is the share of items on which the two annotators agree, and serves as a reference ceiling.

ItemsJudge vs. humanHuman vs. humanScore (judge)Score (human)
Conflict judgments20066%66%––
Answer quality, all10046%56%––
Answer quality, ConCord5042%56%+0.28+0.08
Answer quality, baselines5049%56%0.00−0.05

On conflict judgments, the judge agrees with humans as often as humans agree with each other (66% vs. 66%). On answer quality, it agrees with humans less often than humans agree with each other (46% vs. 56%). Humans still rank ConCord above the baselines (+0.08 vs. −0.05), in the same direction as the judge (+0.28 vs. 0.00).

(2) A second LLM judge. We re-scored the same items with GPT-4.1, using the same prompts and settings as Haiku 4.5.

ItemsScore (Haiku 4.5)Score (GPT-4.1)
ConCord50+0.28+0.50
Colleague500.000.00
Nvwa500.00+0.10
AWM500.00−0.20
SkillClaw50+0.30+0.26
Generic50−0.30−0.20

Overall, the human evaluation and the second judge point to the same conclusions as our original judge: ConCord outperforms the baselines.

W3 & Q3. Counts and scores. Thank you for pointing this out. Every dataset is evaluated on two user sets (Appendix C), i.e., two groups of users selected with Prompt 14 whose records are used for skill distillation.

(1) Table 7 counts each element once per user set. As described in Appendix C, content elements are extracted once per task (at most six), but exposure labels are determined once per task and user set, because whether an element is conflict-exposed depends on the contested decisions of that set. Table 7 therefore counts element–user-set pairs, excluding labels that failed our format check. On Enron, the 42 tasks yield 201 distinct elements (4.8 per task, within the limit of six). All 201 have a valid label for one user set and 183 for the other, giving 201 + 183 = 384 pairs (225 exposed and 159 unexposed).

(2) Table 2 averages the two user sets, while Figure 4 uses only one of them. The alternative conflict-handling methods in Figure 4 were built on only one user set, so Figure 4 reports every method on that set. ConCord's Enron quality score is 0.83 on that set and 0.72 on the other; Table 2 reports their average (0.775, rounded to 0.78).

(3) CIL is computed on the items shared by the compared methods. ConCord's CIL also differs between Table 2 and Figure 4 (e.g., 0.62 vs. 0.71 on Enron). The number of valid checklist items differs across methods because labels that fail our format check are excluded. To compare methods on the same items, each table or figure keeps only the items that are valid for all the methods it compares and discards the rest. Table 2 and Figure 4 compare different sets of methods, so they keep different items, and the same method can receive a different CIL in the two.

Q4 & W4. Uncertainty and repeated runs. We thank the reviewer for this suggestion. We added repeated runs and bootstrap confidence intervals. In short, the averaged results are stable, with ConCord ranking first on all three metrics in every run, while settings with small gains (mainly SE-CS and per-setting RR) are not.

(1) Repeated runs. We ran all six methods three times on the four datasets with GPT-4.1-mini, rebuilding every skill and regenerating every answer (including the no-skill answers) in each run. The table reports the average over the four datasets (mean ± standard deviation over the three runs).

MethodQuality ↑RR ↑CIL ↓
AWM+0.09 ± 0.050.38 ± 0.070.83 ± 0.01
Nvwa+0.34 ± 0.050.57 ± 0.030.67 ± 0.02
Colleague+0.38 ± 0.050.58 ± 0.030.55 ± 0.02
Generic+0.30 ± 0.040.62 ± 0.030.57 ± 0.01
SkillClaw+0.21 ± 0.100.39 ± 0.060.83 ± 0.04
ConCord+0.58 ± 0.020.82 ± 0.020.42 ± 0.04

ConCord ranks first on all three averages in every run, and its margin over the best baseline (0.20 in quality, 0.20 in RR, and 0.13 in CIL) is at least three times the standard deviation of either method. Per dataset, it ranks first in 11, 10 (including one tie), and 9 of the 12 dataset–run pairs on quality, RR, and CIL. The exceptions are on SE-CS (quality once, CIL twice), 20-MAD (RR twice, computed over only 16 contested decisions), and AntiSkillBench (CIL once).

(2) Confidence intervals. For the full Table 2 (four models × four datasets), we computed 95% bootstrap intervals over test users (over contested decisions for RR). The table reports ConCord's margin over the strongest baseline on each metric, averaged over the 16 settings.

MetricStrongest baselineMargin [95% CI]
Quality ↑Nvwa+0.19 [0.16, 0.23]
RR ↑Generic+0.12 [0.06, 0.18]
CIL ↓Colleague−0.21 [−0.24, −0.18]

All three average margins are significant (their 95% intervals exclude zero). Taking the 16 settings individually, ConCord's margin over the same baseline is significant in 11 settings for quality and 13 for CIL, but in only 4 for RR, where each score is computed over only a few contested decisions.

Dear Reviewer Ha6X,

Thank you again for your time and constructive feedback on our submission.

As the rebuttal period is coming to a close today (November 18, AoE), we would like to gently follow up to see if our responses and additional experiments have addressed your concerns or if there are any remaining questions we can clarify before the deadline.

We truly appreciate your time and effort in evaluating our work.

Reviewer_5fcPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConCord is a framework for constructing and coordinating persona skills from multi-user work records. It first extracts context-dependent decision rules and stance profiles from each user’s records, groups users with similar working practices, and constructs one skill per group. Each skill draws on knowledge from all users, while following its own group’s preferences when practices conflict. For a given task, separate agents independently generate candidate responses using these skills. A merging LLM then combines compatible information and selects between conflicting recommendations based on the task and each skill’s scope description.

Critical strengths
  • The motivation is interesting. Different people may have valid but conflicting preferences for handling the same task. Preserving these alternative working practices, rather than forcing them into a single shared approach, is a meaningful problem for multi-user skill distillation.
  • Experiments examine several relevant alternatives. Paper considers removing conflicting rules, retaining both rules, and specifying conditions for their use.
Critical weaknesses
  • The technical novelty appears incremental, and the benefit of the proposed grouping pipeline is not sufficiently isolated. The method mainly combines prompted rule extraction, LLM-based user grouping, group-specific skill construction, and answer merging. I am not yet convinced that the paper establishes a sufficiently distinct technical contribution beyond adding a user-grouping and multi-agent generation pipeline to persona distillation. The comparisons also do not clearly separate the benefit of preserving different working practices from the benefit of generating and merging multiple answers.
  • The justification for merging different working approaches into one response is not sufficiently convincing. The candidate responses are intentionally generated from groups with different working practices. Each scope description is limited to one sentence and the merger is instructed to choose one action without presenting alternatives or acknowledging disagreement. Paper doesn't establish that the final merged response makes the appropriate task-specific choice or preserves a coherent working approach.
Questions for authors
  1. Does conflict coordination generalize to new situations? Can you show controlled experiments in which only a relevant condition, such as urgency, verification requirements, or the cost of an error, changes? Does the merging agent switch to the appropriate working practice, including for combinations of conflicting rules not observed during skill construction? A comparison between merging, selecting one complete candidate, and retaining conditional alternatives would also help clarify the value of the coordination mechanism.
  2. Why is user-level grouping the right granularity? Please compare the current global user partition with grouping users separately for each subtask or decision dimension, or grouping decision rules directly. Keep the skill-length and inference budgets comparable. I would also like to see whether the resulting groups actually reduce internal disagreement and preserve distinct minority practices, rather than only improving aggregate response quality.
  3. Do group-specific skills reproduce the working practices of the users they represent? Can you conduct a held-out behavioral reconstruction test, using each group’s skill to predict its members’ decisions and comparing those predictions with their actual work records? Comparing against other groups’ skills would help establish whether the distilled skills preserve meaningful behavioral differences rather than merely provide generic task guidance.
5 comments

W1. Technical novelty and isolation of the grouping benefit. Thank you for this comment. We address the two concerns in turn.

(1) Separating the benefit of preserving different practices from that of generating and merging multiple answers. We added budget-matched controls with GPT-4.1-mini on 20-MAD and Enron. Nvwa-Ens merges 3 drafts from Nvwa's skill. ConCord w/o grouping builds a single skill from all users (k = 1); with 3 drafts and ConCord's merging step, it removes only the grouping.

  • Nvwa-Ens (3 drafts + merge): Nvwa's skill generates 3 drafts that are then merged, matching ConCord's number of calls (4) with a similar token cost (6,881 vs. 7,400).
  • ConCord w/o grouping (k = 1, single call): ConCord builds a single skill from all users and generates one answer.
  • ConCord w/o grouping (k = 1, 3 drafts + merge): ConCord builds the same single skill, generates 3 drafts with it, and merges them using ConCord's merging step, matching ConCord's number of calls. This keeps the pipeline and the drafting budget fixed and removes only the grouping.
MethodCallsTokens/task20-MAD Quality ScoreEnron Quality Score
Nvwa (single call)11,444+0.62+0.24
Nvwa-Ens (3 drafts + merge)46,881+0.56+0.55
ConCord w/o grouping (k = 1, single call)11,528+0.84+0.67
ConCord w/o grouping (k = 1, 3 drafts + merge)47,124+0.80+0.69
ConCord (ours)47,400+0.93+0.78

With the same number of calls and a similar token cost, removing only the grouping lowers the score from 0.93 to 0.80 on 20-MAD and from 0.78 to 0.69 on Enron. Generating and merging more answers does not explain this gap: with ConCord's single shared skill, moving from one call to 3 drafts + merge leaves the score essentially unchanged (0.84 → 0.80 and 0.67 → 0.69), and for Nvwa it lowers the score on 20-MAD (0.62 → 0.56) while raising it on Enron (0.24 → 0.55). The gain from ConCord's additional calls therefore comes from drafting with different group-specific skills. How the groups themselves are formed is examined in Q2.

(2) Technical contribution. We agree that the individual steps of ConCord are prompted LLM calls. Our contribution lies in the problem, the design principle, and the evaluation.

A failure mode that motivates the design. To our knowledge, this is the first work to study conflicts among working practices in multi-user persona skills. Section 2.2 shows that when conflicting rules share one skill, the model notices the conflict but largely follows one practice: 94% of DeepSeek-V4-Pro's reasoning traces consider both rules, yet 86% of its answers follow Rule A, and quality is 0.27 below the average of the single-rule settings (Appendix D shows the same pattern on Enron). ConCord therefore keeps conflicting practices in separate skills and resolves them per task.

Separation adds a gain that single-skill strategies do not. Against the single-skill strategies in Figure 4 (dropping conflicts, keeping both, adding conditions, and Nvwa-style splitting), ConCord achieves the highest quality on three of the four datasets, with the largest margin on Enron (0.83 vs. 0.67), and is within 0.01 of the best on 20-MAD. The control above shows the same with the pipeline and budget fixed (+0.13 and +0.09 over a shared skill).

Conflict-specific evaluation. We introduce RR and CIL, which measure whether skills give clear guidance on contested decisions and preserve conflict-related content.

W2 & Q1: Does conflict coordination generalize to new situations? Thank you for this suggestion. We added the controlled test the reviewer describes: the skills stay fixed and only one condition of the task changes. In short, the scope lines appear to help the merge switch to the appropriate practice, and merging gives the highest answer quality on both datasets. Selecting one complete draft switches more often on 20-MAD, but its answers score lower.

We selected situations in which two of ConCord's group skills take different courses on the same decision (e.g., requesting senior review before merging vs. committing directly): 50 on 20-MAD and 48 on Enron. For each, we wrote two versions that differ by one added fact, which places the situation within the scope of one skill or the other (e.g., "the change touches several components and the documentation is incomplete" vs. "the update must ship immediately to meet a hard deadline") and never names a course of action. An independent LLM (Claude Sonnet 4.5) labeled the appropriate course for each version. The skills are those of Table 2, and all coordination methods share the same drafts. "Correct pairs" is the share of pairs in which the final answer takes the appropriate course on both versions. "Quality" is the score against the no-skill answer, as in Table 2, reported for the three mechanisms the reviewer asks to compare. "Oracle" selects, for each version, a draft that takes the appropriate course whenever one exists.

CoordinationCorrect pairs (20-MAD)Correct pairs (Enron)Quality (20-MAD / Enron)
Single skill (Nvwa)0.080.38–
Merge without scope lines0.120.29–
Conditional alternatives0.080.290.81 / 0.34
Select one complete draft0.420.400.83 / 0.58
Merge (ConCord)0.240.350.91 / 0.69
Oracle (upper bound)0.520.52–

Does the merge switch to the appropriate practice? Partly. No draft takes the appropriate course in about half of the pairs (oracle: 26 of 50 and 25 of 48), so every method should be read against this ceiling. On 20-MAD, the merge is correct on 12 of 50 pairs, compared with 6 without the scope lines, 4 for the single Nvwa skill, and 21 for selecting one complete draft. On Enron, the methods are close (17 of 48 for the merge, 14 without the scope lines, 18 for Nvwa, and 19 for selection). With about 50 pairs per dataset, these differences are suggestive rather than conclusive.

Merging vs. selecting vs. conditional alternatives. Merging gives the highest answer quality on both datasets (0.91 vs. 0.83 for selection on 20-MAD; 0.69 vs. 0.58 on Enron), consistent with its keeping content from the other drafts. Selection makes the task-specific choice more often on 20-MAD (21 vs. 12 pairs) but discards that content. Presenting conditional alternatives switches rarely (0.08 and 0.29) and gives the lowest quality (0.81 and 0.34), which supports stating a single course.

Unseen combinations. On pairs whose two courses never appear together in one user's records, the merge reaches 0.21 and 0.25, compared with 0.31 and 0.46 on the remaining pairs, so coordination generalizes only partly.

Coherence and limitations. By design, the merger does not blend conflicting courses: where the drafts differ on the same point, it is instructed to choose one course and state it once (Prompt 8). It also keeps the concrete content of the other drafts, so a merged answer can contain steps from several groups. This test does not measure coherence separately, and we did not test longer scope descriptions. The main limitation is the task-specific choice (12 of 50 and 17 of 48 pairs). Improving it, for example, by selecting the course before merging the remaining content, is in line with the adaptive routing listed as future work.

Q2: Why user-level grouping? Thank you for this question. User-level grouping follows from the problem we study. ConCord belongs to an emerging line of work that distills skills from people's work records (persona skill distillation, e.g., Nvwa and Colleague Skill), where a skill describes how a person works. Grouping users keeps each person's choices across decisions together, so each skill describes how an actual set of people works. Grouping rules directly, or grouping users per decision dimension, yields skills that no longer correspond to any set of people.

We also ran the requested comparison on 20-MAD, which shows that this choice comes at no clear cost. Only the grouping changed; the number of skills, the skill length, and the answering procedure were held fixed. "Intra-group conflict" is the share of same-decision rule pairs within a group that the conflict judge marks as conflicting. "Minority practices kept" is the share of practices held by fewer than 20% of users that remain in the final skills.

GroupingIntra-group conflict ↓Minority practices kept ↑Response Quality Score ↑
None (single skill)0.180.93+0.80
Random users (3 seeds)0.160.93+0.94
Rules, by wording0.181.00+0.95
Rules, by course of action0.130.93+0.96
Users, per decision dimension0.100.93+0.91
Users, global (ConCord)0.141.00+0.93

Answer quality and minority practices. Splitting the users or the rules into several skills raises the score from 0.80 to 0.91–0.96. The differences among the groupings are small, and a random partition of users (0.94) scores as high as ConCord (0.93), so answer quality on this dataset does not single out one granularity. All groupings also keep nearly all minority practices (0.93–1.00), since every skill is written from the analyses of all users.

Internal disagreement. A global partition of users reduces intra-group conflict relative to no grouping (0.14 vs. 0.18) and to a random partition (0.16). Clustering rules by wording does not (0.18), because two conflicting rules address the same decision in similar words. Grouping rules by course of action (0.13) and grouping users per decision dimension (0.10) reduce it further.

What the alternatives give up. The per-dimension partitions barely match the global one (average ARI 0.06). A per-dimension skill, like a skill built from grouped rules, combines the choices of different sets of people, and the combination may be one that no user follows, which is the coherence concern the reviewer raises. For persona skill distillation, we regard the user as the appropriate unit: the alternatives reduce internal conflict somewhat further, but give up the correspondence between a skill and the people it describes.

Q3: Do group-specific skills reproduce their members' working practices? Thank you for this suggestion. We added a held-out behavioral reconstruction test. In short, yes on 20-MAD, where a user's own group skill matches their held-out decisions far more often than the other groups' skills, but no on Enron, where the pattern reverses.

Each skill in Table 2 was built from the first 8 records of each user; the remaining 4 were never used (48 held-out records per dataset). We rewrote each held-out record as a task and let every skill answer it. The paper's LLM judge (Claude Haiku 4.5) then checked, for each decision in the record, whether the answer takes the same course as the user; the table reports the share of decisions on which it does. Contested decisions are those on which the group skills disagree. "Other groups' skills" averages over the groups the user does not belong to.

Skill used20-MAD, all20-MAD, contestedEnron, allEnron, contested
Own group's skill0.780.760.680.34
Other groups' skills0.670.320.740.63
Single pooled skill (k = 1)0.670.330.720.55
No skill0.570.360.670.53

20-MAD. A user's own group skill predicts their decisions best (0.78 vs. 0.67 for the other groups' skills), and the gap is largest on contested decisions: 0.76, compared with 0.32 for the other groups' skills, 0.33 for the pooled skill, and 0.36 with no skill. Generic task guidance would not separate the own group's skill from the others in this way. Among the group skills, the own group's skill is the best predictor for 7 of the 12 users, compared with 4.5 expected at random.

Enron. The pattern reverses: on contested decisions, the own group's skill matches the user less often than the other groups' skills (0.34 vs. 0.63) and than no skill (0.53), so the group skills do not reproduce their members' held-out decisions on this dataset. A possible reason is group size: three of the five Enron groups have only two members, so a group's stance rests on 16 emails and may not carry over to new ones.

Reviewer_BheSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ConCord turns work records from several people into reusable instructions for an AI agent. It groups people with similar working habits, creates a skill for each group, and merges answers written using those skills. Across four datasets and four models, it reports better average answer quality and less loss of conflict-related information than the tested baselines.

Critical strengths
  • The method addresses a clear problem: useful practices from different people can conflict, and merging them into one instruction set can hide alternatives.
  • Tests cover email, software issues, computer-science questions, and synthetic assistant conversations with four base models.
  • The paper compares several ways of handling conflicts and reports both quality and computational cost. This makes the benefits and overhead easier to assess.
Critical weaknesses
  1. Evaluation depends on one language-model judge. Claude Haiku 4.5 judges answer quality, conflict resolution, and information loss. Appendix E leaves human validation to future work, so agreement with professional judgment remains unclear.
  2. Extra generation makes the source of the gain unclear. On 20-MAD, ConCord uses four calls and 7,400 tokens per answer, versus one call and 1,444 tokens for Nvwa. The reported comparisons do not isolate conflict-aware grouping from the benefit of generating and merging more drafts; an equal-budget ensemble would help.
  3. The software task tests advice rather than completed fixes. For 20-MAD, the output is a maintainer response with a decision and action plan. This supports claims about response quality, but does not show that the skills improve implemented fixes or passing tests.
  4. Gains are uneven across tasks and models. On SE-CS, ConCord scores only +0.08, +0.12, +0.07, and −0.01 against no-skill responses. With Gemini, it also trails Nvwa on three datasets, so the overall average hides settings where the added complexity provides little benefit.
Questions for authors

see weakness

4 comments

1. Evaluation depends on one language-model judge. Thanks for your suggestions. We added human-judge experiments and report the results below.

To validate the LLM judge, we organized 4 annotators, blind to both method names and the judge's labels, to label 200 rule pairs ("do these two rules conflict?") and 100 answer pairs sampled from Table 2 (a method's answer vs. the no-skill answer, "which is better?"). Each item was labeled by two annotators. Judge vs. human is the fraction of human labels that match the judge's label, and human vs. human is the fraction of items on which the two annotators agree, serving as a reference ceiling.

TaskMethodItemsLLM Judge vs. HumanHuman vs. HumanQuality Score (Judge)Quality Score (Human)
Conflict judgment–20066%66%––
Answer qualityAll10046%56%––
Answer qualityConCord (ours)5042%56%+0.28+0.08
Answer qualityAll baselines5049%56%0.00−0.05

On conflict judgment, the LLM judge agrees with humans as often as humans agree with each other (66% vs. 66%), which supports the reliability of our conflict-handling metrics (RR and CIL). On answer quality, humans rank ConCord above the baselines (+0.08 vs. −0.05), consistent with the LLM judge.

2. Extra generation makes the source of the gain unclear. Thank you for this suggestion. To separate conflict-aware grouping from the benefit of generating and merging multiple drafts, we added four controls with GPT-4.1-mini on 20-MAD and Enron.

  • Nvwa-Ens (3 drafts + merge): Nvwa's skill generates 3 drafts that are then merged, matching ConCord's number of calls (4) with a similar token cost (6,881 vs. 7,400).
  • Nvwa-Ens (4 drafts + merge): The same with 4 drafts, using more tokens than ConCord (9,027).
  • ConCord w/o grouping (k = 1, single call): ConCord builds a single skill from all users and generates one answer.
  • ConCord w/o grouping (k = 1, 3 drafts + merge): ConCord builds the same single skill, generates 3 drafts with it, and merges them using ConCord's merging step, matching ConCord's number of calls. This keeps the pipeline and the drafting budget fixed and removes only the grouping.
MethodCallsTokens/task20-MAD Quality ScoreEnron Quality Score
Nvwa (single call)11,444+0.62+0.24
Nvwa-Ens (3 drafts + merge)46,881+0.56+0.55
Nvwa-Ens (4 drafts + merge)59,027+0.51+0.61
ConCord w/o grouping (k = 1, single call)11,528+0.84+0.67
ConCord w/o grouping (k = 1, 3 drafts + merge)47,124+0.80+0.69
ConCord (ours)47,400+0.93+0.78

With the same number of calls and a similar token cost, ConCord outperforms Nvwa-Ens by 0.37 on 20-MAD and 0.23 on Enron, and it remains ahead even when Nvwa-Ens uses more tokens (+0.42 and +0.17).

To isolate conflict-aware grouping, we keep ConCord's pipeline and drafting budget fixed and remove only the grouping (k = 1, 3 drafts + merge): the score drops from 0.93 to 0.80 on 20-MAD and from 0.78 to 0.69 on Enron. In contrast, adding drafts without grouping does not consistently help, for either ConCord's pooled skill (0.84 → 0.80 on 20-MAD and 0.67 → 0.69 on Enron) or Nvwa (0.62 → 0.56 and 0.24 → 0.55). Thus, the benefit of ConCord's additional calls comes from drafting with different group-specific skills, rather than from generating and merging more drafts.

3. The software task tests advice rather than completed fixes. Thank you for this suggestion. We added an end-to-end evaluation in which the skill-guided plan is implemented and tested. Since 20-MAD provides issue discussions without executable reference tests, we run it on SWE-bench Verified.

We randomly selected 100 tasks (fixed seed, stratified by repository and difficulty) before running any method. We use the skills distilled from 20-MAD by Nvwa and ConCord without modification, so this also tests transfer to unseen repositories.

For each task, GPT-4.1-mini first writes a maintainer plan with each skill, and a coding agent (mini-swe-agent, same model and limits) then implements the fix. A task passes if the reference tests pass and no existing test breaks, following the standard SWE-bench resolved criterion.

  • Fixed the issue counts tasks whose reference tests pass (higher is better).
  • Broke existing tests counts, among patches that applied, those that broke a previously passing test (lower is better).
SettingPass RateFixed the IssueBroke Existing Tests
No skill13/10021/10026/54
Nvwa18/10025/10028/60
ConCord (ours)32/10034/10011/56

ConCord's plans raise the pass rate from 13% to 32%, compared with 18% for Nvwa (32 vs. 18: p = 0.033, Fisher's exact test). The gain comes from both more fixed issues (34 vs. 25) and fewer regressions: among applied patches, ConCord's plans break existing tests in 11/56 cases, compared with 28/60 for Nvwa (p = 0.003). Thus, the skills improve implemented fixes, not only the quality of the written response.

This evaluation covers one model and one baseline (Nvwa). We will add it to the revised paper.

4. Gains are uneven across tasks and models. We agree: ConCord's benefit depends on how much the source documents disagree, and in some settings the multi-draft pipeline adds little. ConCord outperforms Nvwa in 13 of the 16 model–dataset settings; the three exceptions are all with Gemini-2.5-Flash (by at most 0.11). Below we explain the two weaker settings and state when the added complexity is worth it.

(1) SE-CS leaves little headroom for any skill method. ConCord averages +0.065 over the four LLMs (the mean of the four scores the reviewer cites), a small gain for four calls. No method does better: all five baselines average between −0.06 and +0.02 against no-skill responses. SE-CS also has the lowest share of conflict-exposed content (53.2% vs. 58.6–89.0% elsewhere; Table 7); a likely reason is that most CS questions have a technically correct answer, so working practices matter less. ConCord's skills still remain the best on both conflict metrics (average RR 0.94 vs. at most 0.82; average CIL 0.56 vs. 0.66–0.97), but this translates into only a small gain in answer quality here.

(2) With Gemini-2.5-Flash, the gap appears to come from drafting and merging, not from the skills. ConCord trails Nvwa by 0.09, 0.03, and 0.11 on 20-MAD, SE-CS, and AntiSkillBench and leads only on Enron (0.57 vs. 0.07); its higher average (0.17 vs. 0.10) is driven by Enron alone. Its skills, however, still have the lowest CIL on all four datasets (0.60, 0.47, 0.54, 0.13 vs. 0.79, 0.57, 0.63, 0.37 for the best baseline), suggesting that the gap arises in response generation (drafting and merging) rather than skill construction. With this model, the extra calls therefore do not help on three of the four datasets.

(3) The added complexity is optional. On 20-MAD and Enron it pays off: at the same budget (4 calls, about 7k tokens), removing only the grouping lowers the score from 0.93 to 0.80 and from 0.78 to 0.69 (grouping ablation in Response 2), and the gains carry over to implemented fixes (SWE-bench Verified pass rate: 32% vs. 18% for Nvwa; Response 3). Where it adds little, as on SE-CS or with Gemini, the single-call variant (k = 1) avoids the extra calls: with GPT-4.1-mini it costs nearly the same as Nvwa (1,528 vs. 1,444 tokens per task) and still outperforms it (+0.84 vs. +0.62 on 20-MAD; +0.67 vs. +0.24 on Enron). Skill construction is a one-time cost (30 calls, 166 s on 20-MAD with GPT-4.1-mini).

Dear Reviewer BheS,

Thank you again for your time and constructive feedback on our submission.

As the rebuttal period is coming to a close today (November 18, AoE), we would like to gently follow up to see if our responses and additional experiments have addressed your concerns or if there are any remaining questions we can clarify before the deadline.

We truly appreciate your time and effort in evaluating our work.

Dear Reviewer BheS,

We apologize for following up again, and we fully understand how busy this period is. As the discussion period closes in a few hours (November 18, AoE), we would be grateful for any feedback on whether our responses have addressed your concerns.

If any questions remain, we are happy to clarify them before the deadline.

Thank you very much for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.