- Rating
- 4(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ConCord learns reusable instructions from several users’ work records. It groups users with similar working practices and builds one skill for each group. Each skill drafts an answer, and an LLM combines the drafts for the task. The paper tests this approach on four datasets with four base models.
Critical strengths
- The paper addresses a useful problem. Users may prefer different actions in the same situation. ConCord keeps these differences in separate skills and combines their answers based on the task. The paper gives concrete examples of why simply putting conflicting rules into one skill can fail.
- The evaluation covers several settings. The paper tests four datasets and four base models, compares five baselines and several conflict-handling methods, and reports computational cost. The full prompts help readers understand the method. The results also show that keeping more information does not always lead to better answers.
Critical weaknesses
-
Extra computation may explain part of the improvement. Table 3 shows that ConCord uses four calls and 7,400 tokens per answer, while each baseline uses one call and fewer tokens. A shared-skill baseline with the same drafting and merging steps and a similar token budget would help show how much user grouping adds.
-
The LLM judge has not been checked against human scores. Quality, resolution rate (RR), and conflict-exposed information loss (CIL) all use Claude Haiku 4.5. A human check of a sample, along with a second judge, would help show whether the reported rankings are reliable.
-
Some counts and scores need explanation. Appendix C allows at most six content elements per task, but Table 7 lists 384 elements for 42 Enron tasks. Repeated items across user sets may explain this, but the counting rule is unclear. Table 2 and Figure 4 also report different Enron quality scores for ConCord, 0.78 and 0.83. The paper should explain which runs or settings produced these values.
-
The results do not show uncertainty. The paper reports no confidence intervals or variation across repeated runs. Since some gains are small and several steps use random sampling, these checks would help readers judge which improvements are stable.
Questions for authors
- Does ConCord still improve over a shared skill when both use the same drafting and merging steps and similar token budgets?
- Do human reviewers or another LLM judge agree with the reported quality, RR, and CIL scores?
- What counting rules and run settings explain the Table 7 totals and the score differences between Table 2 and Figure 4?
- How much do the main results vary across repeated runs?
5 comments
W3 & Q3. Counts and scores. Thank you for pointing this out. Every dataset is evaluated on two user sets (Appendix C), i.e., two groups of users selected with Prompt 14 whose records are used for skill distillation.
(1) Table 7 counts each element once per user set. As described in Appendix C, content elements are extracted once per task (at most six), but exposure labels are determined once per task and user set, because whether an element is conflict-exposed depends on the contested decisions of that set. Table 7 therefore counts element–user-set pairs, excluding labels that failed our format check. On Enron, the 42 tasks yield 201 distinct elements (4.8 per task, within the limit of six). All 201 have a valid label for one user set and 183 for the other, giving 201 + 183 = 384 pairs (225 exposed and 159 unexposed).
(2) Table 2 averages the two user sets, while Figure 4 uses only one of them. The alternative conflict-handling methods in Figure 4 were built on only one user set, so Figure 4 reports every method on that set. ConCord's Enron quality score is 0.83 on that set and 0.72 on the other; Table 2 reports their average (0.775, rounded to 0.78).
(3) CIL is computed on the items shared by the compared methods. ConCord's CIL also differs between Table 2 and Figure 4 (e.g., 0.62 vs. 0.71 on Enron). The number of valid checklist items differs across methods because labels that fail our format check are excluded. To compare methods on the same items, each table or figure keeps only the items that are valid for all the methods it compares and discards the rest. Table 2 and Figure 4 compare different sets of methods, so they keep different items, and the same method can receive a different CIL in the two.
Q4 & W4. Uncertainty and repeated runs. We thank the reviewer for this suggestion. We added repeated runs and bootstrap confidence intervals. In short, the averaged results are stable, with ConCord ranking first on all three metrics in every run, while settings with small gains (mainly SE-CS and per-setting RR) are not.
(1) Repeated runs. We ran all six methods three times on the four datasets with GPT-4.1-mini, rebuilding every skill and regenerating every answer (including the no-skill answers) in each run. The table reports the average over the four datasets (mean ± standard deviation over the three runs).
| Method | Quality ↑ | RR ↑ | CIL ↓ |
|---|---|---|---|
| AWM | +0.09 ± 0.05 | 0.38 ± 0.07 | 0.83 ± 0.01 |
| Nvwa | +0.34 ± 0.05 | 0.57 ± 0.03 | 0.67 ± 0.02 |
| Colleague | +0.38 ± 0.05 | 0.58 ± 0.03 | 0.55 ± 0.02 |
| Generic | +0.30 ± 0.04 | 0.62 ± 0.03 | 0.57 ± 0.01 |
| SkillClaw | +0.21 ± 0.10 | 0.39 ± 0.06 | 0.83 ± 0.04 |
| ConCord | +0.58 ± 0.02 | 0.82 ± 0.02 | 0.42 ± 0.04 |
ConCord ranks first on all three averages in every run, and its margin over the best baseline (0.20 in quality, 0.20 in RR, and 0.13 in CIL) is at least three times the standard deviation of either method. Per dataset, it ranks first in 11, 10 (including one tie), and 9 of the 12 dataset–run pairs on quality, RR, and CIL. The exceptions are on SE-CS (quality once, CIL twice), 20-MAD (RR twice, computed over only 16 contested decisions), and AntiSkillBench (CIL once).
(2) Confidence intervals. For the full Table 2 (four models × four datasets), we computed 95% bootstrap intervals over test users (over contested decisions for RR). The table reports ConCord's margin over the strongest baseline on each metric, averaged over the 16 settings.
| Metric | Strongest baseline | Margin [95% CI] |
|---|---|---|
| Quality ↑ | Nvwa | +0.19 [0.16, 0.23] |
| RR ↑ | Generic | +0.12 [0.06, 0.18] |
| CIL ↓ | Colleague | −0.21 [−0.24, −0.18] |
All three average margins are significant (their 95% intervals exclude zero). Taking the 16 settings individually, ConCord's margin over the same baseline is significant in 11 settings for quality and 13 for CIL, but in only 4 for RR, where each score is computed over only a few contested decisions.
Dear Reviewer Ha6X,
Thank you again for your time and constructive feedback on our submission.
As the rebuttal period is coming to a close today (November 18, AoE), we would like to gently follow up to see if our responses and additional experiments have addressed your concerns or if there are any remaining questions we can clarify before the deadline.
We truly appreciate your time and effort in evaluating our work.
Q1 & W1. Does ConCord still improve over a shared skill with the same drafting and merging steps and a similar token budget? Thank you for this suggestion. Yes: at the same number of calls (4) and a similar token cost (7,124 vs. 7,400), ConCord outperforms the shared skill by 0.13 on 20-MAD and 0.09 on Enron. To show this, we added four controls with GPT-4.1-mini on 20-MAD and Enron.
Grouping vs. a shared skill. With the pipeline and drafting budget fixed, removing only the grouping lowers the score from 0.93 to 0.80 on 20-MAD and from 0.78 to 0.69 on Enron.
Extra drafts alone do not explain the gain. Adding drafts without grouping does not consistently help, for either the shared skill (0.84 → 0.80 on 20-MAD and 0.67 → 0.69 on Enron) or Nvwa (0.62 → 0.56 and 0.24 → 0.55). With the same number of calls and a similar token cost, ConCord outperforms Nvwa-Ens by 0.37 on 20-MAD and 0.23 on Enron, and it remains ahead even when Nvwa-Ens uses more tokens (+0.42 and +0.17).
Thus, the benefit of ConCord's additional calls comes from drafting with different group-specific skills, rather than from generating and merging more drafts.
W2 & Q2. Do human reviewers or another LLM judge agree with the reported scores? We thank the reviewer for this suggestion. We added a human evaluation and a second LLM judge (GPT-4.1). In short, on the conflict judgments behind RR and CIL, our judge agrees with humans as often as humans agree with each other. On answer quality, humans and both judges rank ConCord above the baselines, but agreement on individual pairs is low and the human margin is smaller.
Both evaluations use the same items: 200 rule pairs ("do these two rules conflict?") and 100 answer pairs sampled from Table 2 (a method's answer vs. the no-skill answer, "which is better?"; 50 from ConCord and 10 from each baseline). RR and CIL are computed from the conflict judgments, so we check these judgments directly. Score is +1 if the method's answer is better, −1 if worse, and 0 if tied, averaged as in Table 2.
(1) Human judges. Four annotators, blind to method names and to the judge's labels, labeled the items, with two annotators per item. "Judge vs. human" is the share of human labels that match the label of our judge (Haiku 4.5). "Human vs. human" is the share of items on which the two annotators agree, and serves as a reference ceiling.
On conflict judgments, the judge agrees with humans as often as humans agree with each other (66% vs. 66%). On answer quality, it agrees with humans less often than humans agree with each other (46% vs. 56%). Humans still rank ConCord above the baselines (+0.08 vs. −0.05), in the same direction as the judge (+0.28 vs. 0.00).
(2) A second LLM judge. We re-scored the same items with GPT-4.1, using the same prompts and settings as Haiku 4.5.
Overall, the human evaluation and the second judge point to the same conclusions as our original judge: ConCord outperforms the baselines.