acceptodds
Under review as a conference paper at ICLR 2027

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

Abstract

Agent harnesses shape how language models turn instructions into completed work by governing tool use, context management, and execution. Choosing a harness therefore requires understanding what different harnesses enable the same model to deliver, and at what cost. We introduce **Harness-Bench**, a benchmark and reusable evaluation pipeline for comparing native harness configurations on realistic work tasks. The suite comprises 300 sandboxed tasks constructed through log-based and domain-based synthesis, with explicit input resources, deliverables, and operational constraints. Task-specific oracles score final artifacts, awarding partial credit for completed work modules under safety and risk gates. The pipeline fixes external task conditions and scoring rules, preserves native harness behavior, and records token usage and API expenditure. We evaluate seven harnesses with ten model backends on general tasks and seven on multimodal tasks, using paired comparisons on shared tasks. Our evaluation reveals substantial, model-dependent performance differences: no harness performs best across all ten backends, and GLM-5.3's general-task full-credit rate varies by 13.5 percentage points across harnesses. Harness choices also change with the delivery objective: configurations that lead mean score need not lead full-credit attainment. Higher expenditure does not ensure better delivery: with GPT-5.6 Sol, Nanobot incurs 2.97 times OpenClaw's API cost while achieving a lower full-credit rate. Stratified trajectory analysis and paired cases explain concrete delivery gaps: corrected tool calls can leave workflows unfinished, and mutually consistent outputs can still omit required evidence. These findings connect model-dependent harness performance to the execution steps that turn partial progress into delivered work. Harness-Bench brings task outcomes, unmet requirements, and resource costs into a common evaluation framework, providing an empirical foundation for selecting and improving agent harnesses.

Primary area:
datasets and benchmarks
Keywords:
AI agents, agent harnesses, benchmark, realistic workflows

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JSH3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors mean to investigate agentic harnesses.

To this effort, they construct a custom 300 task dataset, using user logs, public workflows, and public resources. The tasks go through human review. The construction of a custom dataset was motivated by wanting to create a judgeless task with stable starting conditions and specific delivery requirements. Though the authors promise to open source all evaluations and trajectories, and code, it is unclear if the dataset will be open sourced.

Then the authors test a matrix of Models x Harnesses on this 300 task dataset. By doing so, the authors find some insights, such as no single harness is the best, but it depends on the model. This goes for cost as well.

Critical strengths

The full model x harness gauntlet was potentially a very costly experiment that reports many different model-harness performances.

Critical weaknesses

No contribution over existing professional work / workflow benchmarks have been established.

Incremental contribution in the model-harness evaluation space is also insufficiently established. The analysis and experiments, though probably quite expensive as api costs go, primarily reports configuration-dependent performance differences with limited insights on harness mechanisms to explain those differences.

Questions for authors

n/a

2 comments

Thank you for your review. We would like to clarify the benchmark’s data contribution, the findings supported by the execution records, and the availability of the task dataset.

W1. Contribution beyond existing workflow benchmarks

We respectfully disagree with the assessment that no contribution beyond existing professional-work and workflow benchmarks has been established. Harness-Bench contributes a reusable dataset and evaluation protocol for connecting explicit workflow requirements, module-level delivery outcomes, execution behavior, and resource use across model–harness combinations.

We recognize that prior work, including HAL and DuMateBench, has established the importance of evaluating agent systems across models and execution frameworks. Our contribution builds on this foundation by examining how task requirements and delivery criteria affect harness selection, and by linking incomplete delivery to evidence from the corresponding executions. The resulting task dataset and execution corpus also provide resources for further research beyond the particular findings reported in this paper.

Dataset availability. The task definitions were already included in the submitted supplementary materials. The benchmark dataset is also included in our release, with task instructions, initial resources, environment setup, and task-specific evaluators. We will make this scope explicit in the reproducibility statement to remove the ambiguity identified in the review.

Value of the evaluation investment. The evaluation produces more than 20,000 execution records across the 300 tasks. These records support multiple analyses of the same configurations: average performance, complete delivery, cost–quality trade-offs, execution errors, and recovery behavior. Releasing the trajectories and evaluation records enables researchers to revisit these questions without repeating the original evaluation matrix. We therefore believe that the reusable dataset and execution evidence should be considered alongside the individual empirical findings when assessing the contribution.

W2. Findings and explanations beyond configuration-dependent performance

We agree that the general observation that harness performance depends on the model is already supported by prior work. Our analysis provides more specific findings about the conditions under which harness choices change and the execution problems that accompany incomplete delivery.

1. Delivery objectives and task requirements change harness preferences.

In Appendix C.1, Table 10, optimizing model-specific harness selection for full-credit attainment improves held-out R₁ by 2.53–2.73 percentage points over selecting one shared harness. When selection instead optimizes mean score, the corresponding gains range from −0.14 to +0.14 score points. Thus, the practical benefit of model-specific selection depends on whether the application prioritizes average partial completion or complete delivery.

Task requirements also matter within the same model. Table 14 shows that Gemma’s OpenCode–Goose R₁ difference changes from +8.45 percentage points on tasks carrying an error-recovery demand tag to −7.39 percentage points on the remaining tasks. These results connect harness selection to both the delivery objective and the requirements of the work.

2. Trajectory annotation reveals distinct error and recovery patterns.

We respectfully disagree that the analysis is primarily limited to reporting configuration-dependent scores. Our Luna-assisted trajectory analysis examines execution errors, whether subsequent actions recover from them, and how these observations relate to final delivery. The annotations are grounded in task requirements, execution events, tool feedback, and artifact evidence, with human validation described in Appendix D.

To make this contribution clearer, we supplement this response with aggregate statistics already reported in Appendix D. The general-task base sample contains 5,600 trajectories. Table 22 shows markedly different recovery patterns across error categories:

Error categoryAnnotated error observationsFollowed by complete recoveryRecovery rate
Tool invocation72965589.8%
State and artifacts401266.5%
Verification96270.7%

These counts concern individual error observations, and a trajectory can contain multiple labels. They describe the score-stratified sample rather than benchmark-wide prevalence.

The distinction between an execution error and an unsuccessful outcome is also important: 403 of the 1,400 full-credit runs in the general-task base sample contain an annotated error. An error count alone is therefore insufficient to characterize harness effectiveness; whether subsequent execution resolves the error matters.

Together, these findings distinguish frequently recovered tool errors from state, artifact, and verification errors that often remain unresolved in the recorded execution. They motivate examining recovery beyond successful tool invocation: whether workspace changes persist, whether the required artifacts are produced, and whether final verification covers the original task requirements.

3. Paired executions connect these patterns to concrete delivery failures.

For example, the materials-collection case in Figure 4 shows that Hermes omits a still-valid declaration, leading to an incorrect register entry and an unnecessary reminder. Its subsequent checks of the selected files and hashes do not expose the omission, whereas Goose retains the declaration. This illustrates a specific verification failure: checking the consistency of produced outputs does not establish that all source requirements have been covered.

The aggregate annotations establish that these error categories recur across executions, while paired cases explain how particular errors arise and affect delivery. These are diagnostic findings grounded in execution evidence; we do not claim that they isolate the causal effect of individual harness components.

We will foreground these results and connect the aggregate statistics, paired cases, and implications for harness design more explicitly in the main text. We ask that the benchmark dataset, reusable execution corpus, and diagnostic findings be considered together when assessing the contribution.

Thank you again for your feedback.

Reviewer_LXB9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Harness-Bench, a benchmark for comparing agent harnesses on realistic workflow tasks. They construct 300 tasks across six domains and use task-specific oracles to evaluate final artifacts, with partial credit for completed requirements and safety checks. It evaluates seven harnesses across multiple LLMs, comparing task performance, completion rates, and execution costs. The results show that harness performance depends on the underlying model, and no single harness consistently performs best. The authors also find that higher average scores do not always translate into complete task delivery, and higher execution costs do not necessarily lead to better results. Finally, trajectory analyses examine common execution issues, including tool-error recovery, context management, and incomplete deliverables.

Critical strengths

1.The benchmark covers 300 tasks across six domains, with task-specific oracles that evaluate final deliverables. The evaluation also includes a wide range of model–harness combinations.

2.The experiments provide insights into model and harness compatibility, task completion, and execution costs. The trajectory-level analysis also helps illustrate why some configurations fail to complete the required work.

Critical weaknesses

1.The paper provides limited information about the specific sources of the user logs and public resources, as well as how the tasks were generated and synthesized. More details would help assess how representative the benchmark is of real-world workflows.

2.There are exiting works, such as HAL and DuMateBench, have already studied model–harness combinations and their impact on agent performance. While this paper provides broader evaluation coverage, some of the main findings are similar to those in existing work.

Questions for authors

1.The oracles use binary pass/fail checks at the module level, but it is unclear how well these checks capture the quality of open-ended tasks, such as research and content production. Additional human evaluation would help validate the reliability of the scoring approach.

2.Each task–model–harness combination is evaluated only once. Given the potential variability in agent execution, repeated runs would help assess the stability of the results and harness rankings.

3.The paper is sometimes difficult to follow. For example, key concepts such as objectives, deliverables, modules, and oracles are introduced without sufficiently clear definitions or examples. Some sentences are also quite long and difficult to understand, making it harder to follow the methodology and experimental findings.

2 comments
AuthorsOpenReview ↗

response to reviewer LXB9

Thank you, Reviewer LXB9, for your thoughtful review and constructive suggestions. We appreciate your recognition of the benchmark’s coverage and its analysis of model–harness compatibility. We address your concerns below.

W1. Task provenance and construction

Thank you for pointing out the need for more detail. We analyzed 5,978 real user logs from our company’s internal agent product to guide Harness-Bench’s domain composition and representative task selection. We measured task-type frequencies and clustered task objectives and workflows, identifying representative tasks such as handling out-of-order SQLite migrations, reviewing travel expense claims, and cleaning sales data.

For logs with missing environmental context or unclear objectives, we used GPT-5.6-Luna with human calibration to make reasonable extrapolations and fill in missing details. Public workflows and resources supported concrete task construction: workflows supplied procedures and business rules, while documents, repositories, and datasets supplied task resources. These sources are not mutually exclusive; a log-derived objective could be instantiated using public materials and synthetic records.

During construction, all 300 tasks underwent iterative human review and refinement of their environments and oracles, checking executability and alignment between task requirements and scoring rules. We will expand Section 3.1 to explain the log source, the frequency analysis and clustering process, and the respective roles of observed user objectives, reconstructed context, and public resources. These details will make the benchmark’s empirical grounding and the scope of its representativeness clearer.

W2. Relationship to HAL and DuMateBench

We agree that HAL and DuMateBench have already established the importance of model–harness combinations. Our contribution builds on this work by examining how explicit workflow requirements and complete delivery change harness preferences within and across models.

Two results illustrate this distinction:

  • The selection objective matters. In Appendix C.1, Table 10, optimizing model-specific harness selection for full-credit attainment improves held-out R₁ by 2.53–2.73 percentage points over selecting one shared harness. When selection instead optimizes mean score, the corresponding gains range from −0.14 to +0.14 score points. Thus, the value of model-specific selection depends on whether the goal is average partial completion or complete delivery.
  • The task requirements matter within the same model. Table 14 shows that, for Gemma, OpenCode’s R₁ advantage over Goose is 8.45 percentage points on recovery-tagged tasks, but −7.39 percentage points on the remaining tasks. This connects harness preferences to requirement-defined task groups. Because these groups are observational and their tags overlap, we do not interpret this contrast as an isolated causal effect of a particular harness component.

Harness-Bench provides a reusable task suite and evaluation pipeline for studying these relationships through explicit delivery requirements, module-level outcomes, paired executions, and execution evidence. We will sharpen this positioning in the introduction and related work, while acknowledging the findings shared with prior studies.

AuthorsOpenReview ↗

response to reviewer LXB9

Q1. Oracle reliability for open-ended tasks

Thank you for raising this point. This is an important consideration in our evaluation design. To support consistent and reliable scoring, our oracles focus on key task outcomes and explicit requirements, particularly those that can be checked programmatically. The module-level checks measure whether specified parts of the requested work have been completed. Although individual modules use binary checks, their weighted aggregation preserves partial credit at the task level.

To assess alignment with human judgment within this scope, we sampled 200 trajectories covering all evaluated models and harnesses. Human reviewers scored the runs using the task specifications and final artifacts, following the stated rubric rules. The Spearman correlation between human and oracle scores was 0.7866, supporting alignment in their ranking of task fulfillment under these rubrics.

For research and content-production tasks, however, satisfying these checks does not fully establish the quality of open-ended outputs. We have not separately validated broader qualities such as depth of analysis, creativity, or writing quality against holistic human judgments. Our rubric-based human evaluation assesses agreement on the specified requirements; it does not resolve this broader question.

This reflects a trade-off between reliable verification of concrete requirements and comprehensive evaluation of open-ended quality. We will clarify this design choice and the scope of our scores in the paper.

Q2. Stability across repeated runs

Thank you for this suggestion. We conducted a supplementary experiment on a stratified sample of 50 text-only tasks and 10 multimodal tasks, with three runs per configuration. We report the subsets separately and restrict each comparison to models with complete runs.

First, for each model, we compared the complete set of best-performing harnesses in each later run with the first run, retaining ties.

Task subsetModels with complete runsLater runs matching the firstModels with the same best-harness set across all three runs
Text-only: 50 tasks912/18 (67%)3/9 (33%)
Multimodal: 10 tasks56/10 (60%)2/5 (40%)

Second, we examined the direction of larger model–harness interactions. We selected pairs whose interaction residual had an absolute magnitude of at least 3 score points in the first run and checked whether its sign was retained in later runs.

Task subsetLarge residual pairs in the first runLater runs retaining the initial signPairs retaining the initial sign in both later runs
Text-only: 50 tasks3044/60 (73%)18/30 (60%)
Multimodal: 10 tasks2640/52 (77%)16/26 (62%)

These results provide evidence of repeatability in the direction of many larger interactions, while also showing variation in the exact best-performing harness. The best-harness set remained unchanged in 60–67% of comparisons with the first run, but remained identical across all three runs for only 33–40% of models.

We will include these results and distinguish the observed model–harness interactions from claims of a stable exact ranking. Our central interpretation is that harness selection should account for the underlying model, with the observed variability reported alongside the comparisons.

Q3. Definitions and presentation

Thank you for identifying these presentation issues. We will introduce the key terms together and explain their relationship through a concrete example:

  • Objective: the practical work the user wants to accomplish.
  • Deliverable: a requested output, such as a file, an updated record, or a resulting application state.
  • Module: a scored group of checks corresponding to a specified part of the required work.
  • Oracle: the task-specific evaluator that checks the final deliverables and state, determines module outcomes, and applies the safety and risk gates.

For example, Figure 2 illustrates a sales-data cleaning task with a clean ledger, a reject ledger, a rejection summary, and a report. In the example run, three modules pass while the reporting module fails. The task therefore receives 0.79 when the safety gate passes. This example shows how a practical objective leads to deliverables, how deliverables are checked through modules, and how the oracle combines their outcomes into a score.

We will place these definitions and the example before the scoring formula, shorten long sentences, and separate experimental observations from their interpretation more clearly.

Thank you again for your constructive feedback.

Reviewer_naigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Harness-Bench, a benchmark for comparing agent harnesses (e.g., Claude Code, Codex, OpenCode, OpenClaw) on the same model backends. It has 300 sandboxed workflow tasks in six categories, built from de-identified user logs and public resources. Task-specific oracles score only the final artifacts: each task has weighted modules that earn partial credit, and safety gates can zero the score. The authors evaluate seven harnesses with ten models and report quality, full-credit rate and cost. Main findings: no harness is best on every model; leading on mean score does not mean leading on full delivery; higher cost does not bring better results. A trajectory analysis traces the failures to verification gaps, tool-error recovery, and evidence lost during context compaction.

Critical strengths
  1. Scoring uses no LLM judge. Module-level checks on artifacts plus safety gates make the scores reproducible and auditable, and partial credit still separates "mostly done" from "delivered". The R0.8 vs. R1 split (50.8% vs. 26.2% on general tasks) shows clearly that mean score hides incomplete delivery, which is a useful point for the community.
  2. Coverage is broad and the design is fully crossed: every model runs every harness on the same tasks, in the harness's native configuration, with cost tracked. That makes it possible to analyze model–harness interaction residuals, train/held-out harness selection (+2.5–2.7 pp R1) and quality–cost frontiers under different completion standards.
  3. The failure analysis is mechanistic and backed by evidence. Examples include Goose giving Qwen no read tool (69 calls to an unavailable tool), Nanobot compaction replacing tool results before the model sees them, and Hermes leaving out a still-valid declaration while its own hash checks pass. These point to concrete harness design changes rather than a generic error taxonomy.
Critical weaknesses
  1. Statistical robustness of the rankings. Every configuration runs once at temperature 1.0. The top five harnesses are within 1.0 point of each other on pooled mean score (65.3–66.3), and Tables 2–4 have no confidence intervals. The appendix has task-bootstrap CIs, but resampling tasks does not capture run-to-run randomness, which is likely the larger source of variance in agent execution. Several headline claims ("OpenCode leads mean score", "OpenClaw leads R1") may not be statistically distinguishable. Repeated runs on even a subset (say 2–3 seeds on 50 tasks) would show whether harness rankings and interaction residuals are stable.
  2. "Native harness" is a bundle of confounds. Each harness brings its own system prompt, tool set, tool names, context compaction and request pattern, and the evaluated versions are snapshots. The paper reports which harness works better with which model, but cannot say which harness component causes the difference. Some of the largest effects look more like integration mismatches than design quality: Qwen–Goose (−24.4 residual) comes from the model asking for a read tool that Goose does not expose. Ablations would make the design implications much more convincing, for example normalizing tool names or turning compaction on and off in one or two harnesses.
  3. Oracle validity and strictness. The validation shows only that one reference solution gets full credit. It does not show that other valid solutions also get full credit. "Output contract" tasks have the lowest scores across all harnesses (≈55–59), and R1 is low overall, so some failures may come from oracles that are too specific about format rather than from real non-delivery. Fig. 2 also says the oracles are "llm + human verified", which suggests they were partly LLM-generated. A human audit of a random sample of zero-credit and partial-credit runs, estimating the false-negative rate, would strengthen the claim that the scores reflect delivery quality.
  4. Task provenance and balance. The paper does not say how many tasks come from user logs versus public workflows versus synthesis, how much of each task is agent-synthesized, or how the logs were sampled. The categories are very uneven: Web Development has a single task but is presented as one of six domains, and Content Production has 16. Category-level conclusions, and the claim of representing "realistic workflows", therefore need qualification.
Questions for authors
  1. What fraction of tasks, and of individual modules, comes from user logs, public workflows and agent synthesis? How were the logs sampled, and did you check for overlap with public resources the evaluated models might have seen in training?
  2. Could you re-run a subset with multiple seeds and report how often the best harness per model, and the sign of the large interaction residuals, stay the same?
  3. Did you have humans audit runs that scored zero or partial credit to estimate how often the oracle rejects a deliverable that a person would accept? How were oracles first written: by humans, by an LLM, or both?
  4. Do the 200K context threshold and the 150-turn cap affect harnesses unequally, especially Nanobot, which issues many requests? How many runs hit the turn limit, broken down by harness?
  5. GPT-5.6 Luna is both an evaluated backend and the trajectory annotator. Did its labels differ systematically between its own runs and other models' runs? The 100-trajectory human check is reassuring, but it is small compared with the 6,864 annotated runs.
2 comments
AuthorsOpenReview ↗

response to reviewer naig

Thank naig for your thoughtful questions and constructive suggestions.

Q1&W4. Task provenance and construction

Thank you for your helpful comment on task provenance. We analyzed 5,978 real user logs from our company's internal agent product to determine HarnessBench's domain proportions and representative tasks. We measured task-type frequencies and clustered task objectives and workflows, identifying representative tasks such as handling out-of-order SQLite migrations, reviewing travel expense claims, and cleaning sales data.

For logs with missing environmental context or unclear objectives, we used GPT-5.6-Luna with human calibration to make reasonable extrapolations and fill in missing details. Public workflows and resources supported concrete task construction, so a task could draw on multiple sources.

Regarding the Web Development coverage raised in W4, we clarify that it includes 27 tasks, and we will add more general Web Development tasks in future releases.

Q2&W1. Stability across repeated runs with multiple seeds

Thank you for the suggestion. We conducted a supplementary experiment on a stratified sample of 50 text-only tasks and 10 multimodal tasks, with three runs per configuration. We report the two subsets separately.

Best-performing harness per model. For each model, we compared the complete set of best-performing harnesses in each later run with the first run, retaining ties.

Task subsetModels with complete runsLater runs matching the firstModels with the same best-harness set across all three runs
Text-only: 50 tasks912/18 (67%)3/9 (33%)
Multimodal: 10 tasks56/10 (60%)2/5 (40%)

Sign of large interaction residuals. We selected model–harness pairs whose interaction residual had an absolute magnitude of at least 3 percentage points in the first run and checked whether the sign was retained in later runs.

Task subsetLarge residual pairs in the first runLater runs retaining the initial signPairs retaining the initial sign in both later runs
Text-only: 50 tasks3044/60 (73%)18/30 (60%)
Multimodal: 10 tasks2640/52 (77%)16/26 (62%)

On the sampled tasks, the results show evidence of stability in both the best-performing harness and the direction of large interaction residuals. In particular, large residual signs were retained in 73% of text-only and 77% of multimodal comparisons, while the best-harness set remained unchanged in 67% and 60% of comparisons, respectively. These findings suggest that the observed model–harness interactions are often reproducible across runs, while some variation in exact rankings remains.

Q3&W3. Oracle construction and human alignment

Thank you for this valuable question about oracle reliability. Our oracles are LLM-generated implementations of explicit rubric rules, followed by task-by-task human validation. The rules are:

  1. Requirement-grounded verification. Check final artifacts and state against explicit task requirements and independent references.
  2. Weighted module scoring. Award a module's weight only when all its checks pass.
  3. Equivalent valid outputs. Accept semantically equivalent outputs that satisfy the stated requirements.
  4. Safety and integrity gates. Reserve score-zeroing for protected-resource tampering, sensitive-data disclosure, or prohibited actions.

Human reviewers pilot the tasks using Deepseek V4 Flash, Gemma4-31B, and GPT-5.6-Sol and review the resulting oracle scores to examine:

  1. Whether the requirements are executable.
  2. Whether the checks align with the stated rubric.
  3. Whether the checks distinguish correct, incomplete, and incorrect deliverables.

Oracles that fail these checks are revised and tried again. Thus, oracle construction combines LLM implementation with human testing and refinement.

To assess alignment with human judgment, we sampled 200 trajectories covering all evaluated models and harnesses. Human reviewers scored each run based on the task specification and the agent's final artifacts, following the rubric rules above. The Spearman correlation between human and oracle scores was 0.7866, supporting the overall alignment of oracle scoring with human judgment.

AuthorsOpenReview ↗

response to reviewer naig

Q4&W2. 200K context window and 150-step limit

Thank you for raising this important question about the evaluation budgets. We audited the two limits separately across all 300 tasks (274 general and 26 multimodal). The formal collection contains 20,444 runs out of 20,454 expected task–model–harness combinations; missing combinations are reported separately, and runs with missing scores remain in the budget-audit denominator.

The 150-limit was reached in 63/20,444 runs (0.31%). We verified 57 explicit shared-proxy budget rejections and six Goose native-limit stops. Nanobot reached the limit in 18/2,921 runs (0.62%), despite its higher request counts. Nine capped runs already received full credit. Thus, direct truncation at this limit was uncommon, including for Nanobot.

We configured a 200K-token context window for each harness while retaining its native context-management mechanism. Of 20,444 runs, 133 (0.65%) had at least one provider-reported input at or above 200K; even Nanobot, which had the highest rate, reached only 2.43%.

The results are summarized in the table below.

HarnessRunsReported input ≥200KRate150-limit hitsRate
Claude Code2,91370.24%280.96%
Codex2,92270.24%10.03%
Goose2,922341.16%60.21%
Hermes2,92220.07%70.24%
Nanobot2,921712.43%180.62%
OpenClaw2,92210.03%20.07%
OpenCode2,922110.38%10.03%
Total20,4441330.65%630.31%

Taken together, the 200K context-window setting and the 150-turn cap did not emerge as widespread bottlenecks in the observed statistics. Both reported 200K crossings and 150-limit stops were uncommon, including for Nanobot.

Q5. Potential bias in trajectory annotation

Thank you for raising the question of potential annotator bias. To examine whether annotation agreement differs by the model generating the trajectory, we extended the human check with stratified samples of 200 Luna trajectories and 200 trajectories from other models. Spearman correlations between automated and human annotations were 0.8775 and 0.8224, respectively, a difference of 0.0551. Both groups show high agreement with human annotations, with higher observed agreement on Luna trajectories.

We interpret this as evidence of agreement in both groups, rather than proof that the annotator is unbiased. These annotations support our trajectory-mechanism analysis.

Thank you again for your valuable feedback. We look forward to your response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.