- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes GAMBIT, a framework for quickly estimating the skill of new FPS players through adaptive matches against calibrated bots. It constructs a population of scripted and learned opponents with different behavioral styles, evaluates them in a round-robin tournament, and selects an examiner bank based on matchup diversity, rating coverage, and reliability. During assessment, the framework updates a Bayesian distribution over the player’s skill after each match and selects the next opponent by maximizing expected information gain, accounting for uncertainty in both player skill and examiner ratings. However, the skill-assessment demonstration uses only a single neural proxy player, and the opponent-selection comparison relies on a single replay configuration, leaving the method’s reliability and efficiency across different players insufficiently validated.
Critical strengths
A strength of the paper is its examiner-selection design following round-robin evaluation. Rather than simply retaining the strongest bots, it considers matchup diversity, rating coverage, and reliability to remove unsuitable or redundant candidates and construct a more balanced examiner bank.
Critical weaknesses
-
Insufficient experimental validation. The skill-assessment demonstration uses only a single neural proxy player, and the opponent-selection comparison relies on a single replay configuration. These limited experiments do not adequately establish the method’s reliability and efficiency across players with different skill levels and play styles.
-
Unclear benefit of EIG-based adaptive selection. Although the paper motivates strategically diverse examiners as complementary diagnostic probes, the nearest-rating baseline achieves nearly identical cumulative expected information gain to EIG-based selection in the reported comparison. The additional benefit of EIG-based selection, and the contribution of examiner diversity to assessment quality, therefore remain insufficiently demonstrated.
Questions for authors
-
Please refer to the concerns raised in the Weaknesses section.
-
How does the size of the candidate examiner pool affect assessment quality? The paper acknowledges the limited population and identifies expansion as future work. However, with only 12 candidates and ten retained, the benefit of the selection procedure remains unclear. Could the authors evaluate larger candidate pools and compare the selected bank against retaining all candidates or selecting an equally sized bank at random?
-
Why was this particular neural proxy selected, and how representative is it? Although the paper acknowledges the need for broader validation, the skill-assessment demonstration uses only one proxy player. Could the authors evaluate multiple proxies with different skill levels and play styles, using repeated trials to assess estimation accuracy and the number of encounters required?
-
Does behavioral diversity provide benefits beyond rating coverage? While diversity influences examiner-bank construction and style-switching constraints, the EIG calculation does not explicitly model player–style interactions. Could the authors compare a diverse examiner bank with a bank selected primarily for rating coverage, matching their rating ranges and calibration uncertainty, to isolate the contribution of behavioral diversity?
-
How do repeated encounters with the same examiner affect assessment validity? The implementation prevents immediate repetition but allows an examiner to reappear later. Could players learn examiner-specific counterstrategies, causing the estimated rating to reflect familiarity with a bot rather than general skill? How would the method address or evaluate this effect, particularly for human players?
2 comments
Thank you for the response and additional experiments.
However, the new results do not establish the benefits of the proposed method over simpler baselines. In particular, the contribution of examiner-style diversity to skill-estimation accuracy and the practical benefits of EIG-based selection remain insufficiently demonstrated. Neither the examiner-bank construction procedure nor the subsequent opponent-selection mechanism shows a clear predictive advantage over random selection or other basic baselines.
While I appreciate the expanded evaluation and the authors’ candid discussion of its findings, these results do not resolve my concerns about the proposed method’s added value over simpler baselines. I therefore maintain my original score.
Reply to Reviewer MKss
We extended the evaluation across 12 held-out examinees, 12 layout-disjoint rotations, and 32 assessment replay orders per policy. At six encounters GAMBIT-EIG has held-out Brier 0.2164 and rating MAE 111.5; Elo/random has 0.2074 and 74.8. The expanded experiment is a stronger robustness check, while not showing an Elo accuracy advantage.
The reviewer is correct that the original single replay was insufficient. In the expanded comparison, Bayesian random, nearest-rating, and EIG selection yielded Brier scores of 0.2149, 0.2158, and 0.2164, respectively; EIG and nearest had the same mean posterior SD (153.4) at six encounters. EIG therefore offers an explicit expected-entropy criterion, not a demonstrated gain in held-out predictive accuracy with the present scalar model.
Reply: Please see the replies above
Reply: We evaluated several bank sizes and compared our selected ten-member bank against retaining all eleven eligible opponents and the exact average over every possible ten-member subset. After six diagnostic encounters, the selected bank achieved Brier 0.2143 and rating MAE 108.1, versus 0.2138/105.8 for the full eleven and 0.2128/102.9 for equal-size random selection. (see table in Q4 below)
Selected-minus-full differences were +0.0004 Brier (95% interval −0.0048 to +0.0056) and +2.2 MAE (−11.0 to +15.4); selected-minus-random differences were +0.0015 (−0.0059 to +0.0088) and +5.2 (−16.3 to +26.7), respectively.
The intervals include zero, so a predictive advantage of selection is not established in this population. The construction remains a reproducible way to trade off rating coverage, matchup-profile diversity, and operational reliability; scaling beyond twelve candidates is untested.
Reply: The generalist was an illustrative neural proxy, selected because it was an available calibrated learned policy, not because it represented human play. The expanded benchmark instead holds out each of the 12 candidate policies in turn, excluding its matches from examiner calibration and using disjoint assessment/reference seeds. This samples multiple scripted and learned styles, while remaining a finite frozen-data simulation rather than an independent human cohort.
Reply: We agree that behavioral diversity in the bank and the scalar EIG model are distinct. For example, the generalist and peeker differ by only 7.7 rating points but have a score-profile RMSE of 0.163, showing that similar overall ratings need not imply similar matchup behavior.
We also tested bank composition directly. With ten examiners per bank, identical rating endpoints, five scripted and five learned/hybrid policies, and nearly identical mean examiner SD (108.19), GAMBIT's selected bank produced slightly higher payoff-profile diversity than rating-only selection (0.2213 versus 0.2190). (see table below)
This did not improve six-encounter accuracy: Brier was 0.2143 versus 0.2130, and rating MAE was 108.1 versus 103.4. The paired GAMBIT-minus-rating-only MAE difference was +4.7 points (95% interval −18.7 to +28.1).
The two banks were identical in five of twelve folds, which limits the contrast. Thus bank diversity is measurable and deliberately encouraged, but its independent predictive benefit remains unestablished. The present EIG criterion does not model player-by-style interactions.
Reply: The original rule prevents immediate consecutive repeats but allows the same examiner later in the session. In the new ablation, 334 of 384 frozen assessment replays (87.0%) revisited an examiner, with an average of 1.81 repeat encounters per replay.
A strict no-repeat rule eliminated these revisits. It reduced the observed six-encounter rating MAE from 108.1 to 102.9 and Brier from 0.2143 to 0.2121; however, the paired MAE difference was −5.2 points (95% interval −50.7 to +40.3), and the Brier interval also included zero.
A possible explanation is that repeated encounters with the same examiner provide less independent information about general player ability. Our strict no-repeat experiment supports investigating this possibility, the rating MAE decreased from 108.1 to 102.9, while posterior uncertainty increased slightly (153.2 to 154.3). This suggests a potential trade-off between posterior contraction and assessment accuracy.