- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FitBench evaluates agent configurations under six executable user-behavior conditions, applied to two tasks: adapted TravelPlanner and miniF2F tasks. It introduces iSA index to expose option selection, preference revelation, and feedback.
Critical strengths
I like treating user behavior as an explicit evaluation factor, which I believe is a significant setting and need. The separation between task, model, execution configuration, and user slot makes the setup inspectable. The paper also correctly distinguishes outcome dispersion from collaboration quality and retains Lean verification for proof correctness, which I believe make the results more reliable.
Critical weaknesses
However, I do have several concerns that I feel are crucial:
First, I think the claimed matching benefit is not demonstrated. In Table 3, ReAct performs best for five of six slots. Direct improves only the deferential slot, from .765 to .777. Even choosing the best configuration for every slot on the reported matrix improves the equal-slot average by only about 0.2 percentage points. I believe this does not support improvement “for most slots.” A held-out comparison against the global-best configuration is needed, at least.
Meanwhile, I feel that human grounding does not establish behavioral validity. The questionnaire informs slot content, but its exact sample size, response analysis, and behavioral calibration are not reported. I see the Appendix B.7 states that "execution parameters are design choices rather than questionnaire estimates", while these rules can support a synthetic stress test, but their ability to predict collaboration with actual users remains unestablished.
And the experimental scale and uncertainty are insufficient. Each condition has one realization across only 26 cases, including related variants. Although case-clustered intervals are mentioned, they are not presented. Repeated runs and uncertainty estimates preserving case families would help separate stable slot effects from generation variability. For the current stage, the results is highly unreliable.
Questions for authors
Pls refer to the weakness.