acceptodds
Under review as a conference paper at ICLR 2027

Practice Makes Permanent: Lock-In When Assignment Creates Its Own Evidence

41 views

Abstract

Assigning each player to the position they already play best gives them more practice in that position, making the same lineup stronger tomorrow. In a five-player team game, each match consists of five players occupying the same five positions. Any lineup must stay close to the strongest combination that can currently be assembled. We prove an exact threshold on the tolerated decline in performance: below this threshold, always fielding the strongest lineup eventually makes every alternative player–position pair permanently unreachable; for five players starting at skill 0.40 on a scale where mastery is 1, the threshold is a 24% decline. When the tolerated decline is small enough, the lineup fielded in the first match becomes permanent. If skills are observed only through noisy tryouts, no rule does better than assigning a contested position to the player with the higher average tryout score. If uncertainty is too large, decision-makers who require proof that a new lineup is safe before trying it will not risk letting a player practice a new position. We tested teams of five to fifty players and found that the observed thresholds lie within 1.4% of our predictions; even if we balance how often each player gets a desired position, five of the ten desired player–position pairs, two for each player, remain permanently unreachable. Both performative prediction and safe exploration study this assignment–practice feedback loop; here, the thresholds are exact.

Primary area:
learning theory
Keywords:
learning theory, sequential decision-making, performative feedback, safe exploration, repeated matching, learning-by-doing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZMkiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic role assignment under endogenous performance constraints. It characterizes irreversible lock-in, derives feasibility and optimal qualification-cost guarantees for rotation under symmetric assumptions, and compares scheduling policies in synthetic experiments using the NR20\mathrm{NR}_{20} metric.

Critical strengths
  1. Numerical examples and intuitive explanations make the main mechanisms and mathematical results accessible. The explanations accompanying the theoretical statements are particularly helpful.

  2. The paper establishes precise theoretical results, including exact lock-in thresholds, reversible training windows, and optimal qualification costs within the stated symmetric setting. These results provide a quantitative account of how assignment decisions can permanently restrict future opportunities.

Critical weaknesses
  • W1: Qualification versus playability. Table 1 reports performance and NR20\mathrm{NR}_{20}, while playability is discussed separately in the text. Successful urgency and rotation runs can qualify all ten desired pairs but leave only five individually playable at the final state. Reporting qualification and playability side by side would clarify the practical meaning of opportunity preservation.
  • W2: Scope of the theoretical guarantees. The quantitative results rely on symmetric or tightly controlled departures from symmetric skills, and the positive rotation guarantees further require a two-matching target structure. Refer to Q1.
Questions for authors
  • Q1 — Related to W2: Which realistic role-assignment scenarios are reasonably approximated by common learning rates and nearly equal initial skills?
  • Q2: Exceeding the two-matching alternation settings, what obstacles arise for preserving options while satisfying the performance floor?
  • Q3: How conservative are the sufficient conditions in Propositions 4 and 11?
4 comments

Response to Reviewer ZMki (1/2)

Thank you for reviewing our paper!! Your W1 changed Table 1, and your W2 led to a lock-in theorem that does not require symmetry. Both comments were concrete and constructive. :D

W1. Qualification beside playability

Reporting qualification and playability side by side would clarify the practical meaning of opportunity preservation.

Table 1 now reports two numbers side by side for each policy. The first is the number of desired pairs that have not qualified in the 20-round check after the season (column "NR20"). The second also includes pairs that have qualified but cannot play (column "+ unplayable"). Four policies in the table below, including a new pair-level planner, qualify all or almost all ten pairs at tolerance 0.10, but five of the pairs still cannot play at the end. The new playability rotation lets the two lineups alternate for 70 rounds, longer than the in-season rotation. In 44 out of 50 runs, all ten pairs were qualified and playable by the end. In the remaining 6 runs, the second lineup failed to clear the floor, leaving five pairs unqualified in each of those runs and giving a 50-run mean of 0.60 in both columns.

Six policies from Table 1 (tolerance 0.10, learning rate 0.05; means over 50 runs, 10 desired pairs):

Policy (name in Table 1)Not qualified in the 20-round check (column "NR20")Not qualified or unable to play (column "+ unplayable")Team value (share of the strongest lineup's)
Always the strongest lineup (Greedy)7.827.821.000
Desired pairs not yet qualified first (Urgency rule)0.005.000.925
The alternation of Proposition 3 (In-season rotation)0.605.000.935
Looks ahead to the end of the season (Pair-level planner, new)0.005.000.926
Rotates in ten opening rounds that are exempt from the floor (Preseason rotation)0.005.000.925
Alternates the two lineups for longer, 70 rounds (Playability rotation, new)0.600.600.901

Proposition B.5 proves that with equal starting skills, equal learning rates, and no forgetting, if two lineups rotate long enough, every pair becomes both qualified and playable. In this equal-skill setting (starting skill 0.40, learning rate 0.05), at tolerance 0.10 this rotation gives up 11.0% of Greedy's cumulative value, while qualification alone requires giving up at least 6.9%. At tolerance 0.05, this policy cannot help because rotating between two lineups requires a tolerance of at least 0.070.

W2. Scope of the guarantees

The quantitative results rely on symmetric or tightly controlled departures from symmetric skills, and the positive rotation guarantees further require a two-matching target structure.

Lock-in no longer requires symmetry. The new Theorem 1' allows each pair to have its own starting skill and learning rate, with no forgetting. Under Greedy play (fielding the strongest lineup in every round), the team becomes locked in once the first lineup is fully practiced whenever the value lost by switching to any other lineup exceeds the loss the floor allows. To see how this recovers our baseline threshold, when unpracticed skills start at 0.40, the cheapest lineup change moves two players. Each player moved is 0.60 away from full skill, resulting in a total loss of 1.20. When the first lineup is close to full skill, the floor allows a loss of about five times the tolerance. Therefore, when the tolerance is below 0.24, no other lineup clears the floor, matching the lock-in threshold in the symmetric model.

The rotation guarantee still relies on a two-matching target structure and near-symmetric skills. Corollary B.2 covers teams where each player initially has the same skill advantage in their opening position, referred to as a head start in the paper, assuming no forgetting, equal learning rates, and equal starting skills across all other positions. For example, with unpracticed skill 0.40, a head start of 0.10, and a learning rate of 0.05, the team is locked in from the first round when the tolerance is below 0.080. When the tolerance is at least 0.200, every round of rotation starting from the second lineup clears the floor (Table G.4).

Response to Reviewer ZMki (2/2)

Q1. Settings close to equal starts

Which realistic role-assignment scenarios are reasonably approximated by common learning rates and nearly equal initial skills?

Two settings have nearly equal starting skills: newly formed human teams and newly initialized neural networks. Publicly available professional match records reflect teams that are already specialized (99.2% of appearances are in the player's primary position that year). The paper tests newly initialized neural networks (Section 6). In this test, five newly initialized networks correspond to five players, and five types of image tasks correspond to five positions. Dividing the measured lock-in threshold for each pair by its predicted value gives a median ratio of 0.93, with 10th to 90th percentiles from 0.85 to 1.00. Beside those some applications could be mapped to MOE and muti-agent systems. It would also be interesting to see whether a publicly available Counter-Strike 2 dataset could be used to test our model.

Q2. Beyond two lineups

Exceeding the two-matching alternation settings, what obstacles arise for preserving options while satisfying the performance floor?

As we can think of, four obstacles arise: value cost, uneven starting skills, computation, and the rising floor itself. Under equal starting skills, equal learning rates, and no forgetting, any number of pair-disjoint lineups can alternate under the same tolerance required for two lineups (Proposition B.6). However, rotating more lineups increases the value cost and makes the rotation more fragile when starting skills are slightly uneven (drawn uniformly between 0.37 and 0.43; see table below) - which is always true in real life. When the desired pairs cannot be partitioned into disjoint lineups, the paper provides no rotation guarantee. Computationally, when the number of players is part of the input, the scheduling problem is strongly NP-complete even for two lineups with equal starting skills and equal learning rates, though it is solvable in polynomial time for any fixed team size (Theorem 3). Finally, if the team stays with its first lineup instead of rotating early, the floor rises as that lineup improves: in our Appendix B example (starting skill 0.40, learning rate 0.10, tolerance 0.10), a lineup change can move five players in round 0, three in round 1, two in round 2, and none from round 3 onward.

Rotation among more lineups (starting skill 0.40, qualification bar 0.70, 100 rounds, learning rate 0.05):

Lineups in the rotation2345
Least share of team value given up to qualify every desired pair6.90%13.70%20.33%26.58%
Runs out of 50 in which the rotation falls below the floor (tolerance 0.10; starting skills drawn between 0.37 and 0.43)6163039

Q3. Propositions 4 and 11

How conservative are the sufficient conditions in Propositions 4 and 11?

Both sufficient conditions are conservative. Appendix E now includes Table E.1 comparing each condition against the exact threshold (see table below). For Proposition 4, the sufficient interval radius is 2.90 times the exact value. For Proposition E.5 (formerly Proposition 11), with confidence intervals of two standard errors, the sufficient condition requires 533 tryouts per pair, or 1.16 times the exact worst-case value (459). However, the 533-tryout bound assumes that every interval center lies within one standard error of the true skill, which holds simultaneously across all 25 pairs with probability only about 7 in 100,000. The revision therefore adds a simultaneous 95%-confidence guarantee across all 25 pairs (Table E.2).

Sufficient against exact (a ratio nearer 1 is tighter):

ConditionSufficientExactSufficient over exactEnough in simulation
Proposition 4: interval radius beyond which the planner never changes its lineup (five players starting at 0.40, tolerance 0.25, learning rate 0.10)0.500.1732.90 (2.8 to 6.8 across 36 settings)not reported
Proposition E.5 (formerly Proposition 11): tryouts per pair, intervals of two standard errors (tolerance 0.10, learning rate 0.05, tryout noise 0.05)533459 (worst case)1.16about 325 (95% of simulated tryout phases)
Simultaneous guarantee for all 25 pairs at 95% confidence (Table E.2): equal tryouts per pair1,865 (46,625 in all)least number not knownnot knownnot reported

Thank you again for your review!! We are happy to answer any further questions at any time :)

I appreciate your clear and comprehensive response and am glad to see that your paper has become more inspiring and robust. The revised paper also reads smoothly, and the argument is consistent throughout. However, given the scope of the research problem and the extensibility of the contributions, I will maintain my current rating of 3 (weak accept), as I do not think the paper’s overall quality warrants the highest rating.

Thank you so much for taking the time to read our revised paper and for your encouraging feedback! We are really glad that the updates made the paper smoother and more robust, and we appreciate your thoughtful note on its scope and future extensions :)

Reviewer_q5xQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, they study repeated role assignment within a team, where playing a role improves the corresponding skill. In every round, the planner must select an assignment whose performance is at least (1−ϵ)(1-\epsilon) times the optimal performance under the current skills. Since assignments change skills, they also change this performance floor. The authors investigate when specialization makes alternative player–role qualifications permanently unreachable under any future floor-feasible schedule. They derive an exact threshold and a reversible training window for symmetric initial skills with a common learning rate and no forgetting, together with several extensions. Finally, they conduct synthetic experiments to support their theoretical results and explore various parts of the model.

Critical strengths
  • The proposed model is interesting. It captures the fact that specialization improves the current lineup while raising the performance floor, potentially making alternative player–role qualifications permanently unreachable.
  • They provide precise theoretical results with formal proofs under various assumptions.
  • The numerical experiments test the theoretical predictions and illustrate the performance–qualification tradeoffs of different assignment policies.
  • The appendix provides meaningful supporting analysis, including perturbation bounds and robustness checks under alternative learning dynamics and performance measures.
Critical weaknesses
  • The main reward is additive. Although alternative aggregators are explored, explicit player-specific complementarities and synergy remain unmodeled.
  • The primary opportunity measure captures qualification rather than actual role access: a player may be qualified for a role while every lineup assigning them to it violates the performance floor.
  • The practical value of preserving qualifications needs stronger justification: specialization maximizes team reward in the symmetric baseline, while preserved qualifications do not necessarily provide actual role access.
  • The main lock-in threshold assumes symmetric initial skills and a common learning rate, limiting its generality. The hardness proof uses heterogeneous skills and learning rates, leaving the computational difficulty of planning in the symmetric model unresolved.
Questions for authors
  • What is the intended planner objective? Since specialization is reward-optimal in the symmetric model, should the contribution be interpreted as preserving individual preferences rather than improving long-term team performance?
  • Definition 2 measures whether a player can reach the skill threshold for a desired role. However, being qualified does not guarantee that there exists a lineup satisfying the performance floor that assigns the player to that role. Why is qualification the primary measure of opportunity? How would the conclusions and policy comparisons change if preserving an opportunity required a feasible future schedule in which the player is both qualified for the role and actually assigned to it?
  • Which assumptions are essential? What broader structural condition could replace symmetry, and which conclusions survive substantive skill heterogeneity, unequal learning rates, or skill transfer?
5 comments

Response to Reviewer q5xQ (1/2)

Thank you for your review!! The fourth weakness you pointed out led to two new results. We now have a lock-in theorem that does not require symmetry, and a hardness result in the symmetric model.

W1. Teammate synergy

The main reward is additive ... explicit player-specific complementarities and synergy remain unmodeled.

In the new experiment, we added synergy between teammates to team value and tried several forms (Table G.3). The paper refers to the team that fields the strongest lineup in each round as greedy. In every form, whenever the tolerance is below a certain threshold, the greedy team is still locked in. In three of these forms, the measured threshold ranges from 0.331 to 0.547, which is higher than the main model's 0.24. In another form, synergy only depends on which two players play together, with a threshold of 0.081.

W2. Qualification versus access

... qualified for a role while every lineup assigning them to it violates the performance floor.

We agree that being qualified and being able to play are two different things. Table 1 now reports two numbers for each policy, in the columns NR20 and + unplayable. The first number is the number of desired pairs (a player and the position they want to play in) that have not qualified in the 20-round check after the season. The second number also counts pairs that have already qualified but cannot be included in any lineup that clears the floor. Proposition B.5 proves that if two lineups rotate long enough, each pair is qualified and able to play. It assumes that skills are not forgotten, starting skills and learning rates are the same, and tolerance is large enough for both lineups to alternate. When the tolerance is 0.10 and the learning rate is 0.05, this rotation gives up 11.0% of greedy's cumulative value, while qualification alone requires giving up at least 6.9%.

W3. Value of preserving options

The practical value of preserving qualifications needs stronger justification ...

You're right. In the symmetric model, preserving options comes at a cost to the team, as stated at the beginning of Section 4. There are two situations where a team that only practices the strongest lineup also loses value. The first situation is when the team's required lineup changes during the season. In our example, each of the two lineups has a 50% chance of becoming the one the team needs. In this case, the team that practiced both lineups has an expected value 5.55% higher than the team that only practiced the first lineup. A team that is not constrained by the floor earns more value than these two teams, indicating that the loss comes from the floor. In the second situation, two players want to play in the same position, but their skill ceilings are different. If this position is given to the player with the lower ceiling in the first round, the team in the example loses 2.1% of its value over 100 rounds. Appendix I provides the general formulas for these two losses. In the publicly available records of League of Legends professional games, if a player leaves and one of the remaining starting players has played the vacant position, the vacancy is more often filled by a remaining teammate (table below).

Vacancies that a remaining teammate filled after a player left (Appendix K):

Before 2022From 2022 on
A remaining starter had played the position at least ten times35.2% (38 of 108)18.0% (18 of 100)
No remaining starter had4.1% (68 of 1,643)4.7% (64 of 1,351)

W4. Symmetry and hardness

... symmetric initial skills and a common learning rate ... planning in the symmetric model unresolved.

The new Theorem 1' no longer requires symmetry! Now each pair can have its own starting skill and learning rate. It assumes that skills are not forgotten and that the team fields the strongest lineup in every round. It provides a sufficient condition in each direction: one ensures lock-in, and the other ensures that the given pair can still become qualified. When starting skills are the same, these two conditions meet at the same threshold. The newly added part (c) of Theorem 3 proves that the season scheduling problem is strongly NP-complete. This result holds when the number of players is part of the input, even if all starting skills are the same and the learning rate is the same. When the number of players is fixed, this problem can be solved in polynomial time.

Response to Reviewer q5xQ (2/2)

Q1. Planner's objective

What is the intended planner objective? ... preserving individual preferences rather than improving long-term team performance?

Yes, in the base model, the contribution of the paper is to preserve individual options rather than improve the long-term performance of the team. The planner's goal is to maximize the value of the team, while ensuring that each round's lineup clears the floor. The paper asks a question: under such a planner, which positions can no longer be reached by the players who want to play them? Some other policies in Table 1 give up a portion of team value (see W2 for the cost) to keep these positions within reach. The two situations where the team itself also loses are shown in W3.

Q2. Qualification as measure

Why is qualification the primary measure of opportunity? How would the conclusions and policy comparisons change ...?

Under your stricter requirement, the conclusion regarding lock-in remains unchanged. This conclusion applies to pairs that have not yet qualified. Skills only improve through play, so locked pairs cannot become qualified or play. Therefore, Definition 2 still uses qualification. What has changed is the comparison between policies, as shown in the table below. At tolerance 0.10, the urgency rule, in-season rotation, and preseason rotation qualified all or almost all of the ten pairs. But in the end, five pairs were unable to play because after all pairs were qualified, these policies only repeated the same lineup. The newly added playability rotation lets the two lineups alternate for 70 rounds, with all ten pairs qualified and able to play in 44 out of 50 runs. When the tolerance is 0.05, this policy cannot help because rotating between two lineups requires a tolerance of at least 0.070.

Table 1, desired pairs out of ten (learning rate 0.05; 50-run means; lower is better):

PolicyNR20, tolerance 0.05+ unplayable, tolerance 0.05NR20, tolerance 0.10+ unplayable, tolerance 0.10
Greedy: always fields the strongest lineup7.827.827.827.82
Count-maximin: first the players with the fewest desired positions so far4.984.985.005.00
Pair-count maximin: the same for each desired pair5.045.041.401.40
Urgency rule: desired pairs not yet qualified first4.405.000.005.00
In-season rotation: the alternation of Proposition 34.805.000.605.00
Playability rotation (new): alternates for longer (70 rounds)4.805.000.600.60
Option planner: looks one round ahead3.946.944.647.62
Pair-level planner (new): looks ahead to the end of the season3.685.560.005.00
Preseason rotation: first ten rounds exempt from the floor0.005.000.005.00

Q3. Essential assumptions

Which assumptions are essential? What broader structural condition could replace symmetry, and which conclusions survive ...?

The first necessary assumption is that the floor increases as the team becomes stronger. The second is that skills in unpracticed positions do not rise along with practiced ones, so the gap between them widens over time: if practicing one position pulls unpracticed skills up toward the ceiling as well, the gap closes and lock-in disappears. But with the addition of 3% forgetting in each round, lock-in returns: the threshold in the example is 0.243, which is almost the same as the main model's 0.24 (Appendix I). Symmetry is not necessary. Theorem 1' proves that the team will be locked in, provided that after the first lineup is fully practiced, the value lost by switching to any other lineup exceeds the loss the floor allows. Unequal learning rates do not change the threshold for lock-in, as long as the starting skills are the same (Proposition B.1). The rotation guarantee still requires no skill forgetting and equal learning rates. It also requires either the same starting skills (Proposition 3) or the same head start for each player in their opening position (Corollary B.2).

Thank you again for your thoughtful comments! They were very helpful, and several of them sparked deeper discussions on these topics. We are happy to answer any further questions.

Thank you again for your thoughtful review!! As the discussion period wraps up today, we wanted to briefly check in and see if our revision and response addressed your questions. To recap the main updates for your points: Theorem 1' removes symmetry for lock-in, Theorem 3(c) proves strong NP-completeness even under equal starting skills and learning rates, Table 1 now reports playability side by side with qualification (along with the new playability rotation), and Propositions I.5–I.7 and Table G.3 cover team-value losses from lock-in and teammate synergy. We are happy to clarify anything or answer any remaining questions before the window closes!

Reviewer_sJn3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper begins by motivating how repeated role assignments in teams improve practiced skills over time while making alternative assignments look comparatively risky.

The paper formalizes this problem as sequential one-to-one matching of team members to roles under learning-by-doing and an endogenous performance floor that requires each lineup to stay within a tolerance of the current best lineup. Based on this model, the paper derives exact thresholds for permanent role lock-in, characterizes a finite reversible window for early cross-training, proves complexity results for qualification scheduling, and shows how conservative confidence bounds can starve exploration even when options remain feasible. A number of synthetic experiments across deterministic, noisy, and skill-forgetting settings compare 12 assignment and learning policies in terms of cumulative team performance and preserved role options.

Critical strengths

The paper presents an original and mathematically clean formulation of endogenous-floor lock-in, deriving exact thresholds for greedy lock-in, early rotation feasibility, and cautious certification starvation that are well supported by the proofs and boundary sweeps.

The work highlights an insightful and consequential distinction between per-round outcome fairness and future option preservation, showing that balancing desired roles each round achieves 99.8% efficiency while still permanently locking out half of the team's desired role options.

Critical weaknesses

The approach is only validated on synthetic near-symmetric initial skills, and most of the proposed floor-respecting remedies (such as preseason and in-season rotation) break down once modest skill forgetting (δ > 0) is introduced, which brings out the question of how well these remedies transfer to realistic teams with prior specialization and skill decay.

The learning baselines in Table 1 treat all n! = 120 matchings as unstructured bandit arms without pair factorization or multi-step state planning, and deferring all empirical data plots (Figures 3–8), Table 2, Algorithm 1, and Theorem 3 to the appendix makes the main paper much less self-contained.

Questions for authors

Section 6 mentions publicly available per-match League of Legends role-assignment data (Oracle's Elixir). Would it be possible to include a real-data calibration or a heterogeneous initial-skill experiment (where members already start with strong primary-role specialization outside κ_0) to see when floor-respecting policies can still preserve secondary role options?

In Table 1, the RL and bandit baselines treat all n! matchings as independent arms. It would be useful to include a pair-factored, state-aware baseline (for example, combining the n^2 pair posteriors from Section 5 with multi-step lookahead or safe-set expansion) to see whether structured learners still suffer the same lock-in.

In Proposition 11, the m = 533 tryout calculation assumes a 1-sigma event holds simultaneously across all 25 pairs (joint probability around 7e-5). What sample size m is needed under a valid union-bounded confidence guarantee, and how much does m drop if preseason tryouts are allocated adaptively to the 2n relevant pairs?

The clarity of the main paper could be improved significantly by moving key empirical figures (such as Figures 3, 5, and 6) and the formal statement of Theorem 3 from the appendix into the main text, along with discussing potential remedies when skill forgetting (δ > 0) causes preseason rotation to erode.

3 comments

Thank you so much for the review!! Following your comments, we have added a planner baseline and corrected the tryout guarantee.

W1. Validation and remedies

... only validated on synthetic near-symmetric initial skills ... realistic teams with prior specialization and skill decay.

Both of your points are correct! At the two tolerances in Table 1, once there is forgetting, none of the tested policies that respect the floor in each round kept all ten desired pairs (see Q4). For teams that are already specialized, we conducted experiments with unequal starting skills as you suggested (see Q1). The new Theorem 1' no longer requires symmetry. Now each pair can have its own starting skill and learning rate. It assumes that skills are not forgotten and that the team fields the strongest lineup in every round. In addition to synthetic skills, we also conducted a test using neural networks: five newly initialized networks correspond to five players, and five types of image tasks correspond to five positions (Section 6). The median ratio of the measured lock-in threshold to the predicted value is 0.93.

W2. Structured baselines, self-contained text

... treat all n! = 120 matchings as unstructured bandit arms ... much less self-contained.

We added a planner to Table 1. This planner records skills by pair. In each round, it simulates the rest of the season for several candidate lineups and selects the one with the fewest unreached desired pairs at the end of the season (see Q2 for results). Theorem 3 and its newly added part (c) are now included in the main text. The simulations of the five team sizes in Figure 6 of the submitted paper are now also shown in Figure 4(b) of the main text.

Q1. Real data and unequal starts

... a real-data calibration or a heterogeneous initial-skill experiment ... outside κ_0 ...?

We conducted an experiment with unequal starting skills: each player initially had the same skill advantage in their own position. The paper refers to this advantage as the head start (Figure 3). It ranges from 0.05 to 0.30, outside of the submitted paper's κ_0. Each player wants to play in two positions, so the ten desired pairs are divided into two lineups. The first two rows of the table below are the two tolerances predicted by Corollary B.2. The third row contains the values measured in Table G.4. It illustrates the cost of a head start: to keep all ten pairs, a tolerance of 0.10 is sufficient without a head start, and 0.20 is required when the head start is 0.10. When the head start is 0.15 or greater, the third row can be smaller than the second row. The best policy there is the urgency rule, which only swaps the positions of two players at a time. We did not calibrate the model with match records. In 1,019,291 appearances in League of Legends professional games, 99.2% were in the player's main position that year, so these records rarely tell us about a player's skill in a second position.

Tolerances for each head start (no forgetting, learning rate 0.05, all other starting skills about 0.40):

Head start00.050.100.150.200.30
Predicted (Corollary B.2): below this tolerance the team is locked in from the first round0.0000.0440.0800.1090.1330.171
Predicted (Corollary B.2): from this tolerance the two lineups can alternate, starting with the second lineup0.0700.1110.2000.2730.3330.429
Measured (Table G.4, row "Any of these rules": the five rules of that table that respect the floor in every round; base skills drawn between 0.385 and 0.415; tolerances tested in steps of 0.025, for the pair-level planner in steps of 0.05): smallest tolerance at and above which one of these rules kept all ten pairs qualified and able to play in all 50 runs0.1000.1500.2000.2000.1750.200

Q2. A pair-factored, state-aware baseline

... include a pair-factored, state-aware baseline ... to see whether structured learners still suffer the same lock-in.

We added this baseline, which is the planner described in W2. This planner and the two pair-level learners (Thompson sampling and UCB) still leave desired pairs that are not qualified or unable to play. Our simulation has four settings: tolerance is 0.05 or 0.10, and learning rate is 0.05 or 0.15. The following table shows the setting with a learning rate of 0.05 and a tolerance of 0.10: the planner qualified all ten pairs, but five of them were unable to play in the end. In the other three settings, it still leaves an average of 3.22 to 5.04 desired pairs unreached. When planning with estimated skills, it falls below the floor calculated from true skills in some rounds (table below). We did not test a learner with safe-set expansion.

Tolerance 0.10, learning rate 0.05:

BaselinePairs not reached in the 20-round check (of ten)Rounds per season below the true floor
Always the strongest lineup, true skills (Greedy)7.820.00
Planner that looks ahead to the end of the season, true skills (Pair-level planner)0.00 (5.00 with the pairs that are qualified but unable to play)none: it respects the floor
The same planner on estimated skills, tryout noise 0.020.521.38
The same planner on estimated skills, tryout noise 0.051.664.14
Thompson sampling, learns pair by pair, tryout noise 0.057.920.08
Upper confidence bound (UCB), learns pair by pair, tryout noise 0.058.000.08

Q3. The 533-tryout budget

What sample size m is needed under a valid union-bounded confidence guarantee ... allocated adaptively to the 2n relevant pairs?

Thank you for pointing this out!!! A valid guarantee needs to cover all 25 pairs simultaneously, with a confidence level of 95%. With an even split, 1,865 tryouts per pair are sufficient (table below). Following your suggestion, we focused the tryouts on the ten pairs used for rotation, reducing the total number to 20,265 (Table E.2). This guarantee allows a maximum error probability of 5%. The remaining fifteen pairs get only 95 tryouts each, and they use a small portion of this 5%. So these ten pairs each get 1,884 tryouts, slightly more than 1,865. Both of these splits were fixed before the start of the season. We did not calculate an adaptive split, which is a split adjusted based on earlier tryout results.

Tryouts before the season (tolerance 0.10, learning rate 0.05, tryout noise 0.05):

Basis of the countTryouts per pairTryouts in all
Enough in 95% of simulated tryout phases, intervals of two standard deviationsabout 325
Exact worst case for these intervals459
The budget of the submitted paper (its premise holds for all 25 pairs at once with a probability of about 7 in 100,000)53313,325
All 25 pairs at once with 95% confidence, the same number of tryouts for each pair1,86546,625
The same guarantee, tryouts concentrated on the ten pairs used for rotation1,884 for each of these ten pairs, 95 for each of the other fifteen20,265

Q4. Main text and forgetting

... moving key empirical figures ... and the formal statement of Theorem 3 ... into the main text ... remedies when skill forgetting ...

Theorem 3 is now in the main text. The key figures are now in the main text too: the right panel of the submitted Figure 3 and the content of the submitted Figure 5 are the two panels of the new Figure 5, and the five team sizes of the submitted Figure 6 are in Figure 4(b). When there is forgetting (3% per round), across the four settings, none of the tested policies that respect the floor in each round kept all ten pairs. We also tried this policy: the first ten rounds are not counted towards the floor, and thereafter, as long as the floor allows, the two lineups alternate. This method is effective in the setting of tolerance 0.10 and learning rate 0.15: in all 50 runs, all ten pairs are qualified and able to play, and the team receives 81.1% of greedy's cumulative value (Appendix G). In the other three settings, it did not keep all ten pairs in any run. In the runs with both a head start and forgetting (Figure 10), when the learning rate is 0.10 or 0.15, as long as the tolerance is large enough, the policy of rotating throughout the season can keep all ten pairs. When the learning rate is 0.05, none of the tested policies that respect the floor in each round can achieve this. The remedies under forgetting are now discussed in Section 6 of the main text, next to Table 1.

Thank you again for your review!!! We are happy to answer any further questions you have!

Thank you again for your helpful review!! As the discussion period wraps up today, we wanted to briefly check in and see if our revision and response resolved your questions. In particular, following your suggestions, we have updated the preseason tryout bound to a simultaneous 95%-confidence guarantee across all 25 pairs (Section 5 and Table E.2). We also added the pair-level planner to Table 1, ran head-start and neural network experiments (Figure 3 and Section 6), and moved Theorem 3 and the key empirical plots into the main text alongside a discussion of forgetting remedies! Please let us know if you have any further questions before the window closes :)

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time, close reading, and insightful feedback.

We have made the following updates:

Theory

  • The new Theorem 1' provides sufficient conditions for lock-in: as long as skills are not forgotten and the team fields the strongest lineup in every round, each pair can have its own starting skill and learning rate. (Section 3, page 5) [q5xQ W4, Q3; sJn3 W1; ZMki W2, Q1]

  • Theorem 3 is now in the main text. The newly added part (c) proves that the scheduling problem is strongly NP-complete. This result holds when the number of players is part of the input, even if all starting skills are the same and the learning rate is the same. (Section 4, page 6) [q5xQ W4; sJn3 W2, Q4; ZMki Q2]

  • We have changed the tryout guarantee. The new guarantee holds for all 25 pairs simultaneously, with a confidence level of 95%. 1,865 tryouts per pair are sufficient, while the submitted paper stated 533. (Section 5, page 7; Table E.2, page 39) [sJn3 Q3; ZMki Q3]

  • The new Corollary B.2 and Figure 3 deal with teams where each player starts out stronger in their own position. For example, with a head start of 0.10 and a learning rate of 0.05, the team is locked in from the first round when the tolerance is below 0.080. When the tolerance is at least 0.200, every round of rotation starting from the second lineup clears the floor. (Section 4, page 6; Appendix B, page 22; Table G.4, page 46) [sJn3 W1, Q1; ZMki W2]

  • The new Proposition I.5 and Proposition I.7 indicate that lock-in can also harm the team itself. When two players have different skill ceilings and the position is given to the one with the lower ceiling at the beginning, the team in the example loses 2.1% of its value in 100 rounds. If the team's required lineup changes during the season, and the chances of both lineups being needed are equal, then the team that has practiced both lineups in advance has an expected value 5.55% higher than the team that has only practiced one. (Section 4, page 5; Appendix I, pages 51 and 52; proofs in the supplementary material) [q5xQ W3]

  • The new Proposition B.6 proves that any number of lineups that do not share pairs can alternate, requiring the same tolerance as when using two lineups. (Appendix B, page 29; proof in the supplementary material) [ZMki W2, Q2]

  • The new Table E.1 shows how conservative Proposition 4 and Proposition E.5 are, where the sufficient conditions are 2.90 times and 1.16 times the exact values in the paper's example, respectively. (Appendix E, page 38) [ZMki Q3]

Experiments and data

  • Table 1 now reports two numbers side by side for each policy: unqualified desired pairs and desired pairs that are unqualified or unable to play. Table 1 also includes two new policies: a pair-level planner and the playability rotation. At a tolerance of 0.10 and a learning rate of 0.05, the playability rotation qualified all ten pairs and allowed them to play in 44 out of 50 runs. (Section 6, page 9) [ZMki W1; q5xQ W2, Q2; sJn3 W2, Q2 (the planner)]

  • We added a test training neural networks. The median ratio of the measured lock-in threshold to the predicted value is 0.93. The paper does not report a rotation threshold for neural networks. (Section 6, page 10; Appendix J, pages 52–55) [ZMki Q1; sJn3 W1]

  • We added a description of public League of Legends match records: 99.2% of appearances were in the player's main position that year. These records are only used for description and not for calibrating the model. (Appendix K, page 55) [sJn3 Q1; ZMki Q1]

  • The submitted Figure 5 and the right panel of the submitted Figure 3 are now Figure 5 in the main text. The five team sizes of the submitted Figure 6 are in Figure 4(b). (Section 6, pages 8 and 10) [sJn3 W2, Q4]

  • We added an experiment with teammate synergy: in every form tested, the team is still locked in below a certain threshold. (Table G.3, page 45) [q5xQ W1]

The revised PDF

  • The submitted Proposition 11 is now Proposition E.5 because the statements in the appendix are now numbered by section.

  • In the appendix, the submitted Figure 6 is now Figure 8 and the left panel of the submitted Figure 3 is Figure 6.

  • "Role" in the submitted paper is now called "position", and "assessment" is now called "tryout".

  • Blue mainly marks new statements, captions of new figures and tables, new table entries, new results and changed numbers. We also improved the wording and language overall so it's easier for the reviewers to go through everything.

Thanks again for all the reviewers. We are happy to answer further questions!!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.