acceptodds
Under review as a conference paper at ICLR 2027

FitBench: Benchmarking Human–AI Collaboration across Option-Picking-Driven Decision Styles

Abstract

Existing benchmarks primarily measure Large Language Models (LLM) or incorporate Harness's task capabilities and performance, which can accurately evaluate the agent's ability and efficiency in reasoning and thinking tasks. However, humans in reality have different decision-making personalities and preferences. Using a single evaluation criterion to assess AI cannot truly reflect the value of this performance in real-world applications, nor can it be used to measure human-AI collaboration efficiency. It is jointly determined by the task itself, AI performance, and human decision-making personality. Therefore, if it is to be evaluated, improvements need to be made to existing benchmarks by incorporating human decision-making personality variables. To this end, we propose **FitBench**, which represents human side differences as six structured Human slots under the same task conditions, and systematically evaluates human-AI collaboration performance under different decision-making personalities, while comparing the impact of different LLM and Harness configurations. FitBench models the collaboration process as three interconnected subsystems and introduces an optional Interactive Structured Assertion (iSA) interaction adapter to enhance the observability of slot differences. In the current experimental results, the range and variance of success rates across the six slots for the same actual task show significant differences; both increase in the recorded iSA control. In addition, we can select the optimal Harness configuration for each human decision-making personality slot while keeping the LLM layer fixed, thereby significantly improving the efficiency of AI collaboration with humans of different decision-making personalities. Thus, FitBench provides a unified evaluation framework for separating model performance from human decision-making differences, analyzing the conditional interaction between Human Slot and AI execution configuration, and further studying collaboration configurations tailored to different decision-making personalities.

Primary area:
datasets and benchmarks
Keywords:
Human–AI Collaboration, Structured Human Behavior, Benchmark Design, Personalized Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BrWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FitBench evaluates agent configurations under six executable user-behavior conditions, applied to two tasks: adapted TravelPlanner and miniF2F tasks. It introduces iSA index to expose option selection, preference revelation, and feedback.

Critical strengths

I like treating user behavior as an explicit evaluation factor, which I believe is a significant setting and need. The separation between task, model, execution configuration, and user slot makes the setup inspectable. The paper also correctly distinguishes outcome dispersion from collaboration quality and retains Lean verification for proof correctness, which I believe make the results more reliable.

Critical weaknesses

However, I do have several concerns that I feel are crucial:

First, I think the claimed matching benefit is not demonstrated. In Table 3, ReAct performs best for five of six slots. Direct improves only the deferential slot, from .765 to .777. Even choosing the best configuration for every slot on the reported matrix improves the equal-slot average by only about 0.2 percentage points. I believe this does not support improvement “for most slots.” A held-out comparison against the global-best configuration is needed, at least.

Meanwhile, I feel that human grounding does not establish behavioral validity. The questionnaire informs slot content, but its exact sample size, response analysis, and behavioral calibration are not reported. I see the Appendix B.7 states that "execution parameters are design choices rather than questionnaire estimates", while these rules can support a synthetic stress test, but their ability to predict collaboration with actual users remains unestablished.

And the experimental scale and uncertainty are insufficient. Each condition has one realization across only 26 cases, including related variants. Although case-clustered intervals are mentioned, they are not presented. Repeated runs and uncertainty estimates preserving case families would help separate stable slot effects from generation variability. For the current stage, the results is highly unreliable.

Questions for authors

Pls refer to the weakness.

Reviewer_zZaFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FitBench adds human decision style as a controlled variable to agent evaluation (Task × Human Slot × LLM × Harness). The human is a rule-driven simulated user from six slots (decisive, exploratory, deferential, shifting, contradictory, low_engagement), each a hand-set atom vector plus a persona-weighted utility that picks one of the LLM's candidate options, a hidden-preference reveal schedule, a feedback template and a stopping threshold; a student questionnaire informs but does not fit these (App. B.7). An optional iSA adapter forces three structured options per turn. Tasks: 18 TravelPlanner and 8 miniF2F proof-repair cases, scored by a frozen LLM judge (0.65H + 0.15E + 0.20A / 0.85L + 0.15A). Nine LLMs under ReAct (Table 2), five harnesses under Qwen-3.7-Max (Table 3) and an iSA ablation (Table 4) are offered as evidence that slots yield distinguishable outcomes and that slot-matched harness selection "significantly improves" collaboration efficiency.

Critical strengths
  • Putting the human on the evaluation axis is the right idea. Existing agent benchmarks fix or remove the user; the same model can be a different collaborator for different decision styles without any change to the task world.
  • Clean decomposition. Task world, LLM candidate generation and slot-conditioned selection are separated; the slot cannot alter the transition kernel; one factor varies per comparison.
  • Objective anchors, blinded judge, honest limits. Hard constraints / Lean carry most of the score, the judge is external and label-blind, failures stay in the denominator, and B.7 admits the questionnaire fits no parameters.
Critical weaknesses
  1. The core mechanism (slot → option choice → outcome) is asserted, not shown. The only evidence that slots matter is non-zero Δ_case / V_case, which LLM sampling, judge variance or differing reveal schedules could produce alone. No noise floor (same slot repeated; the pilot's three seeds are unreported), no random-selection control, no selection-agreement rate, no per-slot process statistics. Argmax vs. softmax, τ_p, how A/B/C/R/Q are estimated per option, and the atom-vector-to-weight mapping are all unspecified. Table 4 cuts the other way: without iSA, TravelPlanner runs 1.00 turn with zero accepted directions — selection never happens — yet Δ_case = 0.193 vs. 0.200 with full iSA; miniF2F turns are identical (1.17) while Δ_case rises. The reported dispersion cannot be attributed to decision style.

  2. No real-human alignment; "personality" framing contradicts the appendix. No real user interacts with the system; questionnaire answers are content-coded, not fitted; no simulated-vs-real comparison exists. The abstract sells "decision-making personalities"; Table 1 and App. A.6/B withdraw the term.

  3. No per-task or per-slot breakdown; cross-task stability unestablished. Tables 2–3 pool two tasks whose slot mechanisms differ entirely (TravelPlanner: the slot controls what the AI learns; miniF2F: it only phrases feedback after Lean errors). Slot rankings are never compared across tasks; B.7 concedes cross-task stability is not established; 4.2 says miniF2F supports no preference-matching conclusion, leaving 18 TravelPlanner cases (9 independent) to carry the story.

Questions for authors
  1. Argmax or softmax in the main tables? τ_p values? How are A, B, C, R, Q estimated per option?
  2. Noise floor: Δ_case for the same slot repeated on the same case; how often do the six slots pick the same option?
  3. Tables 2–3 per task and per slot — do slot rankings agree across tasks?
  4. Report Gain_matched, Regret_h and the dev/test split.
  5. Do slot gaps survive on binary hard-constraint / Lean pass rates with A removed?
Reviewer_2b48OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FitBench evaluates how six simulated decision styles interact with LLMs and agent harnesses. It introduces iSA for structured option selection and feedback, and compares nine LLMs and five harnesses on 26 adapted TravelPlanner and miniF2F cases. The supplementary material includes implementations, configurations, trajectory logs, and judge scores that allow the reported comparisons to be inspected.

Critical strengths

Evaluating agents under different user decision behaviors is a useful direction. The controlled comparisons can reveal differences hidden by an overall task score.

The supplementary release is a concrete strength: it includes executable behavior profiles, experiment logs, and scoring records. These make the results substantially easier to audit than aggregate tables alone.

Critical weaknesses

The benefit of slot-specific harness selection remains unclear. In the paper's Table 3, ReAct leads on five of six slots, and choosing the best harness per slot improves the average score by only about 0.002. The release also documents replacing the original-case H2 records with Table 2 Qwen records. Re-aggregating the supplied scores gives H2 an overall score of 0.683 before this replacement and 0.774 after it, changing the best overall harness. The selection rationale and robustness across runs need clarification, alongside a held-out matching evaluation.

The human grounding remains limited. The code makes the predefined behavior rules inspectable, but this does not establish that they reproduce real user behavior. I did not find aggregate questionnaire findings or validation against observed human interactions in the supplied material. The experiments therefore provide stronger evidence about scripted behavior conditions than about matching AI systems to real users.

Questions for authors

Why were the original-case H2 records replaced with Table 2 Qwen records? Were the configurations equivalent, and what explains the change from 0.683 to 0.774?

Does slot-specific harness selection outperform the best single harness on held-out cases? Please report the split, gain, and uncertainty.

How many valid questionnaire responses were collected, and which findings support the implemented behavior profiles? Is there evidence that these profiles predict actual user interactions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.