- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper builds a benchmark for agents that have to learn hidden and changing policies from their own serving history. The stream is split into environment windows. In each window the agent first does serving tasks and gets only an outcome score, then is tested on held out tasks with frozen state and no feedback. Tasks are split into Hidden-Dependent (visible info not enough) and Fully Specified (visible info enough), so the benchmark can measure both learning and damage to behavior that was already correct. There are three domains (Retail, Banking, Pitch) and three tiers, 53 windows and 7,718 tasks. Retail and Banking are checked by a hash of the final database plus the finish answer, Pitch by an LLM judge rubric. They test five harnesses (RAG, Mem0, SkillOpt, Continual Harness, Prime) on six models, 28 pairs. Main results: Oracle gets 95.4 on Hidden cases on average vs 14.1 for Blind, and learners are far below Oracle (best Retail L3 pair 59.6, median 22.0). Learning is costly and can hurt Fully Specified cases. Answer diversity during failure correlates with final Hidden reward.
Critical strengths
-
The setup measures something that existing benchmarks mostly do not. The policy is hidden, feedback is only the outcome, and the policy changes, gets removed and comes back. Together with Blind and Oracle as lower and upper references, and the Hidden vs Fully Specified split, a result can be read as how much of the learnable gap a harness closes and what it breaks on the way. I think this is the main reason the paper is useful. The gap result (Oracle 95.4 vs the best learner per model at 55.8 to 73.4, Table 2) is large enough that it does not depend on the noise issues below, and the Fully Specified result (Figure 6, Retail Blind 98.9 vs learners 89.5 to 95.5) shows a cost that plain accuracy would hide.
-
The protocol is careful and the paper is honest about its limits. Test probes run from frozen state and are excluded from every update, validation and selection, and do not even create a visible timestamp gap (App B.3). Retail and Banking use a deterministic verifier with no LLM. The appendix also says openly that harnesses are not budget matched (B.2), each cell is one run (B.4), and the Blind Fully Specified baseline is not a pure damage measure (D.6). I also liked the extra check in D.3 that counts distinct answers only before the first correct answer, because it removes the obvious mechanical link between "trying more answers" and "finding the right one". This makes me trust the benchmark itself more than the average agent benchmark.
Critical weaknesses
The harness level conclusions are not established yet, because every cell is one run and the harnesses are not compared under matched conditions. App B.4 says "Each cell summarizes one run; no standard error or confidence interval is implied." But Table 2 has very large swings that look like noise or failure modes. For example Prime on Kimi K3 Pitch L2 is 16.9, below Blind (25.9), while CH on GLM-5.3 Flash Banking FS L1 is 63.8 vs Blind 95.7. Some patterns also look systematic and unexplained: SkillOpt is near zero on Retail L2 for four of six models (0.0, 4.8, 1.1, 0.2) but higher on Retail L3 for five of six. The ranking also changes by model (CH 69.8 vs Prime 63.7 on GLM-5.3). On top of that, B.2 says Prime runs as a code agent with a stronger runtime than the ReAct loop, SkillOpt gets window segmentation through its scheduler and about 0.85 extra replay executions per ticket, and Mem0 gets an extra post-outcome memory turn. Prime is also missing for Opus 5 and GPT-5.6 Terra (omitted for cost, l.304). B.2 itself concludes that comparisons characterize complete systems and do not isolate harness effects (l.808 to 809), but the main text (l.319, l.453 to 454) states harness rankings and attributes exploration to the harness without this caveat, and with one run I can not tell which differences are real. This matters because a benchmark paper is mostly read for these rankings.
Questions for authors
- Can you run 3 seeds on a subset, for example two open models x all five harnesses x Retail L2/L3 and Banking L2, and report the spread? Is SkillOpt's near zero on Retail L2 (0.0, 4.8, 1.1, 0.2 on four of six models, while L3 is higher) stable, or is it the acceptance gate failing?
- Since B.2 says harnesses differ in runtime and extra executions (Prime's code runtime, SkillOpt's replay, Mem0's extra memory turn), can you either report a budget-matched comparison on a subset, or scope the harness rankings in the main text (l.319, l.453 to 454) to complete systems?
- "Effective exploration is critical" (l.130) rests on correlations across harnesses, so a harness-level factor could drive both answer diversity and reward. Can you add an intervention, for example forcing a weak harness to try untried answers after a failure? Otherwise I think "associated with" is the supported claim.
- The Banking prompt warns that "actual practice may differ from the documents" (D.6), and Opus 5 Blind gets only 52.7 to 69.1 on Banking Fully Specified. Does removing this warning change Blind and learner Fully Specified accuracy? This decides whether the "adaptation hurts" finding holds outside Retail.
4 comments
Response to Reviewer dwr8 (2/2)
Q3: Can you force a weak harness to try untried answers after failure? Otherwise “associated with” is the supported claim.
A3: In order to test whether exploring an alternative and learning from its outcome improves held-out performance, we gave CH one chance to try a different decision after each failed serving task on Banking L2, with GLM-5.3 Flash as both actor and Refiner. During serving, only the first attempt counted toward that task’s score. The Refiner received both attempts and their outcomes, with the retry and its feedback used only for learning. At held-out test time, the actor had exactly one attempt per task from the frozen learned state.
The actor changed its decision on 98% of retries and answered correctly on 72%. The table summarizes held-out results from in .
| CH configuration | Held-out Hidden (%) | Held-out FS (%) |
|---|---|---|
| Original run | 47.3 | 76.6 |
| Exploratory retry during serving | 91.9 | 99.0 |
| Gain (percentage points) | +44.6 | +22.4 |
The results show that an exploratory retry with feedback can improve later performance in this CH setting. We describe the cross-system relationship as an association. For more details, see and , .
Q4: Does removing the warning that practice may differ from documents change Blind and learner Fully Specified accuracy?
A4: Thanks for flagging this. In our early experiment, we explicitly asked Opus 5 to follow the provided instructions. Its Blind accuracy on Fully Specified tasks was then comparable to that of the other models. For the reported benchmark, we restored the original prompt, including the warning, so that all models and harnesses received the same task instructions within each setting.
To understand why Opus 5’s Blind accuracy was lower on Fully Specified tasks, we examined its decision traces under the original prompt. We found that it sometimes inferred undocumented restrictions without task-specific evidence, even when the written rules were sufficient. It then acted on those assumptions and got tasks wrong that the written rules would have resolved. The early experiment therefore shows that prompt framing can affect Blind performance. The effect of deleting just the warning on Blind and learner accuracy remains unmeasured.
Thanks again for your careful review and constructive suggestions. We have incorporated the additional results and clarifications into the revised paper. We hope these changes address your concerns and would be happy to answer any further questions.
Thank you for your detailed rebuttal
First, to comply with the PC's policy reminder: I want to clarify that I used claude to help polish my review based on my personal notes but the complete decisions were made only by myself.
I read your rebuttal and satisfied with your response. Your responses addressed all my concerns. I increased my score from 3 to 4. Wish you good luck :)
Thank you for your follow-up
Thank you for reading our rebuttal and increasing your rating! We’re glad our responses addressed your concerns, and we appreciate your constructive feedback.
Response to Reviewer dwr8 (1/2)
We thank the reviewer for the detailed reading of the protocol and appendix, and for recognizing the benchmark’s diagnostic value. We also appreciate the constructive suggestions and address each point below:
A1: Thanks for the questions!
(1) Repeatability. We ran two repeats with GLM-5.3 Flash for all five harnesses on Retail L2, Retail L3, and Banking L2. The table reports held-out Hidden accuracy as mean (sample SD) across the original run and two reruns (Table 22 in Appendix D.13).
Most configurations were stable across runs, with larger variation concentrated in a few cases. SkillOpt was more variable on Retail L2 than in the other settings. CH consistently led Retail L3 and Prime consistently led Banking L2. The lead changed on Retail L2, but the reruns support the same overall findings. We plan to extend the repeat runs to more models and settings to better characterize run-to-run variability.
(2) SkillOpt on Retail L2. We audited the GLM-5.3 and Kimi K3 runs to see whether the optimizer proposed correctly scoped rules and whether the validation gate accepted them (Table 17 in Appendix D.10).
Correctly scoped candidates were rarer on L2 for both models. Gate decisions matched the logs. In Case 8 of Appendix D.10, the candidate update contained a correct gift-card rule and a separate incorrect rule. The validation batches contained no tasks that tested the gift-card rule. The candidate tied the current skill, and the gate rejected it because it required a higher validation score. Task mix also explained part of the gap. Applying the Retail L2 task mix to L3 scores gives the following results (Table 18 in Appendix D.10).
Overall, most configurations were stable across reruns. The SkillOpt audit highlights two issues on Retail L2: correctly scoped candidates were rare, and gaps in validation coverage could leave a useful rule unrewarded. For more details, see Appendix D.13, Table 22 (p. 36), for the repeats and Appendix D.10, Tables 17–18 and Case 8 (p. 33), for the SkillOpt analysis.
A2: Thanks for raising this point. We agree that matching compute budgets is important for isolating the effects of the learning mechanisms. We matched the task streams, held-out sets, scoring, and per-task tool-call and turn limits. Matching the remaining learning-side budgets by removing method-specific executions would change the original update procedures, so the comparison would evaluate modified variants. For example, SkillOpt uses roughly 0.85 validation replays per served ticket to select updates, and Mem0 uses an extra post-outcome turn to write memories.
We therefore compare complete systems on performance and actual cost, including learning overhead. The following numbers from Figure 4 aggregate serving and test results across the four open-weight models.
Prime has higher Hidden reward and higher cost than CH, illustrating the end-to-end trade-off. For more details, see Sections 4.2 and 5.3 for the scope of the rankings and exploration findings, Section 4.3 and Figure 4 for the comparison, and Appendices B.2 and D.4 for system configurations and the cost breakdown.