- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies dynamic role assignment under endogenous performance constraints. It characterizes irreversible lock-in, derives feasibility and optimal qualification-cost guarantees for rotation under symmetric assumptions, and compares scheduling policies in synthetic experiments using the metric.
Critical strengths
-
Numerical examples and intuitive explanations make the main mechanisms and mathematical results accessible. The explanations accompanying the theoretical statements are particularly helpful.
-
The paper establishes precise theoretical results, including exact lock-in thresholds, reversible training windows, and optimal qualification costs within the stated symmetric setting. These results provide a quantitative account of how assignment decisions can permanently restrict future opportunities.
Critical weaknesses
- W1: Qualification versus playability. Table 1 reports performance and , while playability is discussed separately in the text. Successful urgency and rotation runs can qualify all ten desired pairs but leave only five individually playable at the final state. Reporting qualification and playability side by side would clarify the practical meaning of opportunity preservation.
- W2: Scope of the theoretical guarantees. The quantitative results rely on symmetric or tightly controlled departures from symmetric skills, and the positive rotation guarantees further require a two-matching target structure. Refer to Q1.
Questions for authors
- Q1 — Related to W2: Which realistic role-assignment scenarios are reasonably approximated by common learning rates and nearly equal initial skills?
- Q2: Exceeding the two-matching alternation settings, what obstacles arise for preserving options while satisfying the performance floor?
- Q3: How conservative are the sufficient conditions in Propositions 4 and 11?
4 comments
Response to Reviewer ZMki (2/2)
Q1. Settings close to equal starts
Which realistic role-assignment scenarios are reasonably approximated by common learning rates and nearly equal initial skills?
Two settings have nearly equal starting skills: newly formed human teams and newly initialized neural networks. Publicly available professional match records reflect teams that are already specialized (99.2% of appearances are in the player's primary position that year). The paper tests newly initialized neural networks (Section 6). In this test, five newly initialized networks correspond to five players, and five types of image tasks correspond to five positions. Dividing the measured lock-in threshold for each pair by its predicted value gives a median ratio of 0.93, with 10th to 90th percentiles from 0.85 to 1.00. Beside those some applications could be mapped to MOE and muti-agent systems. It would also be interesting to see whether a publicly available Counter-Strike 2 dataset could be used to test our model.
Q2. Beyond two lineups
Exceeding the two-matching alternation settings, what obstacles arise for preserving options while satisfying the performance floor?
As we can think of, four obstacles arise: value cost, uneven starting skills, computation, and the rising floor itself. Under equal starting skills, equal learning rates, and no forgetting, any number of pair-disjoint lineups can alternate under the same tolerance required for two lineups (Proposition B.6). However, rotating more lineups increases the value cost and makes the rotation more fragile when starting skills are slightly uneven (drawn uniformly between 0.37 and 0.43; see table below) - which is always true in real life. When the desired pairs cannot be partitioned into disjoint lineups, the paper provides no rotation guarantee. Computationally, when the number of players is part of the input, the scheduling problem is strongly NP-complete even for two lineups with equal starting skills and equal learning rates, though it is solvable in polynomial time for any fixed team size (Theorem 3). Finally, if the team stays with its first lineup instead of rotating early, the floor rises as that lineup improves: in our Appendix B example (starting skill 0.40, learning rate 0.10, tolerance 0.10), a lineup change can move five players in round 0, three in round 1, two in round 2, and none from round 3 onward.
Rotation among more lineups (starting skill 0.40, qualification bar 0.70, 100 rounds, learning rate 0.05):
| Lineups in the rotation | 2 | 3 | 4 | 5 |
|---|---|---|---|---|
| Least share of team value given up to qualify every desired pair | 6.90% | 13.70% | 20.33% | 26.58% |
| Runs out of 50 in which the rotation falls below the floor (tolerance 0.10; starting skills drawn between 0.37 and 0.43) | 6 | 16 | 30 | 39 |
Q3. Propositions 4 and 11
How conservative are the sufficient conditions in Propositions 4 and 11?
Both sufficient conditions are conservative. Appendix E now includes Table E.1 comparing each condition against the exact threshold (see table below). For Proposition 4, the sufficient interval radius is 2.90 times the exact value. For Proposition E.5 (formerly Proposition 11), with confidence intervals of two standard errors, the sufficient condition requires 533 tryouts per pair, or 1.16 times the exact worst-case value (459). However, the 533-tryout bound assumes that every interval center lies within one standard error of the true skill, which holds simultaneously across all 25 pairs with probability only about 7 in 100,000. The revision therefore adds a simultaneous 95%-confidence guarantee across all 25 pairs (Table E.2).
Sufficient against exact (a ratio nearer 1 is tighter):
| Condition | Sufficient | Exact | Sufficient over exact | Enough in simulation |
|---|---|---|---|---|
| Proposition 4: interval radius beyond which the planner never changes its lineup (five players starting at 0.40, tolerance 0.25, learning rate 0.10) | 0.50 | 0.173 | 2.90 (2.8 to 6.8 across 36 settings) | not reported |
| Proposition E.5 (formerly Proposition 11): tryouts per pair, intervals of two standard errors (tolerance 0.10, learning rate 0.05, tryout noise 0.05) | 533 | 459 (worst case) | 1.16 | about 325 (95% of simulated tryout phases) |
| Simultaneous guarantee for all 25 pairs at 95% confidence (Table E.2): equal tryouts per pair | 1,865 (46,625 in all) | least number not known | not known | not reported |
Thank you again for your review!! We are happy to answer any further questions at any time :)
I appreciate your clear and comprehensive response and am glad to see that your paper has become more inspiring and robust. The revised paper also reads smoothly, and the argument is consistent throughout. However, given the scope of the research problem and the extensibility of the contributions, I will maintain my current rating of 3 (weak accept), as I do not think the paper’s overall quality warrants the highest rating.
Thank you so much for taking the time to read our revised paper and for your encouraging feedback! We are really glad that the updates made the paper smoother and more robust, and we appreciate your thoughtful note on its scope and future extensions :)
Response to Reviewer ZMki (1/2)
Thank you for reviewing our paper!! Your W1 changed Table 1, and your W2 led to a lock-in theorem that does not require symmetry. Both comments were concrete and constructive. :D
W1. Qualification beside playability
Table 1 now reports two numbers side by side for each policy. The first is the number of desired pairs that have not qualified in the 20-round check after the season (column "NR20"). The second also includes pairs that have qualified but cannot play (column "+ unplayable"). Four policies in the table below, including a new pair-level planner, qualify all or almost all ten pairs at tolerance 0.10, but five of the pairs still cannot play at the end. The new playability rotation lets the two lineups alternate for 70 rounds, longer than the in-season rotation. In 44 out of 50 runs, all ten pairs were qualified and playable by the end. In the remaining 6 runs, the second lineup failed to clear the floor, leaving five pairs unqualified in each of those runs and giving a 50-run mean of 0.60 in both columns.
Six policies from Table 1 (tolerance 0.10, learning rate 0.05; means over 50 runs, 10 desired pairs):
Proposition B.5 proves that with equal starting skills, equal learning rates, and no forgetting, if two lineups rotate long enough, every pair becomes both qualified and playable. In this equal-skill setting (starting skill 0.40, learning rate 0.05), at tolerance 0.10 this rotation gives up 11.0% of Greedy's cumulative value, while qualification alone requires giving up at least 6.9%. At tolerance 0.05, this policy cannot help because rotating between two lineups requires a tolerance of at least 0.070.
W2. Scope of the guarantees
Lock-in no longer requires symmetry. The new Theorem 1' allows each pair to have its own starting skill and learning rate, with no forgetting. Under Greedy play (fielding the strongest lineup in every round), the team becomes locked in once the first lineup is fully practiced whenever the value lost by switching to any other lineup exceeds the loss the floor allows. To see how this recovers our baseline threshold, when unpracticed skills start at 0.40, the cheapest lineup change moves two players. Each player moved is 0.60 away from full skill, resulting in a total loss of 1.20. When the first lineup is close to full skill, the floor allows a loss of about five times the tolerance. Therefore, when the tolerance is below 0.24, no other lineup clears the floor, matching the lock-in threshold in the symmetric model.
The rotation guarantee still relies on a two-matching target structure and near-symmetric skills. Corollary B.2 covers teams where each player initially has the same skill advantage in their opening position, referred to as a head start in the paper, assuming no forgetting, equal learning rates, and equal starting skills across all other positions. For example, with unpracticed skill 0.40, a head start of 0.10, and a learning rate of 0.05, the team is locked in from the first round when the tolerance is below 0.080. When the tolerance is at least 0.200, every round of rotation starting from the second lineup clears the floor (Table G.4).