- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the problem of extracting knowledge that lies outside the pretraining data from a given context C and converting it into natural-language skills. To this end, the authors propose Ctx2Skill, which divides the roles among five LLM agents (Challenger, Reasoner, Judge, Proposer, and Generator) and updates the target skill set in an adversarial manner. The experimental results show that Ctx2Skill consistently improves the problem-solving ability of LLMs compared to the baselines.
Critical strengths
-
The role of each agent is described in detail, which helps the reader understand the proposed scheme.
-
The experiments are extensive. The authors report results on various models and baselines.
-
The experimental results are consistent. The extensive results suggest that Ctx2Skill improves performance in a consistent way.
-
The ablation study is thorough. Questions that came up while reading the paper (e.g., what happens if the Generator and the Proposer are merged into a single model, and why using both the passed and the failed sets to update both the Challenger and the Reasoner skills is harmful) were answered, which was satisfying.
Critical weaknesses
-
It is doubtful whether the iterative self-play actually contributes much. For GPT-4.1, the performance with the Iter-1 skills is 15.9%, while the performance with Cross-Time Replay is 16.5%. Figure 4 also gives the impression that selecting the final skill set only among early iterations might be more cost-efficient.
-
There is too much repetition in the writing. For example, the second-to-last paragraph of the Introduction (starting with "In this paper") overlaps considerably with Sections 3.3 and 3.4, which gives the feeling of reading the same text twice and weakens the delivery of the paper.
-
The cost of model usage, the number of tokens, and the time required for self-play are not stated in the main text. Since five agents are run, the cost should not be negligible, and it is a pity that this point is omitted from the main text. The same holds for the time required for self-play.
-
Figure 1 does not carry enough information. If it is meant to illustrate Ctx2Skill, the part where the agents perform self-play should be explained, but this is lacking.
-
It is not stated in the main text what the initial skill sets S_0^C and S_0^R are.
Questions for authors
Included in weakness part.
2 comments
Gentle Reminder on the Discussion Period
Dear Reviewer,
We sincerely thank you for your time and effort in reviewing our paper and for your valuable comments. As the discussion period is coming to an end, we would greatly appreciate it if you could take a moment to check whether our responses have addressed your concerns. If they have, we would be grateful if you would kindly consider updating your score. If anything remains unclear, we would be happy to provide further clarification before the deadline.
Thank you again for your time and consideration.
Best regards, The Authors
Response to Reviewer 6eD7
Thank you for the careful review and the positive assessment of our experiments and ablations.
A1: Iter-1 already contains one self-play round (Prompting 12.3% → Iter-1 15.9%), and 65.8–79.0% of contexts select a later iteration, accounting for 76% of the net gain on GPT-4.1 (General Response C1). Replay selects Iter-4/5 for 27.6% (GPT-4.1) and 36.0% (GPT-5.1) of contexts, so stopping early would remove these candidates. Moreover, extra compute spent on single-pass skills does not substitute for self-play: compute-matched Best-of-12 reaches only 13.3% (General Response C2). We agree that the cost trade-off deserves explicit treatment, and we will discuss lighter settings (smaller N) for budget-constrained use.
A2: We will condense the second-to-last paragraph of the Introduction into a short overview and leave the details to §3.3–
A3: Ctx2Skill makes at most 175 calls per context (≤90 read the context, roughly 1M input tokens), paid once, with one call per query at deployment (General Response C2). We will move this to the main text and report wall-clock time; self-play runs in parallel across contexts.
A4: We have detailed the self-play part in Figure 2. We will redesign Figure 1 to show the loop: Challenger → Reasoner → Judge, the routing of failed and solved cases to the two sides, and Cross-Time Replay.
A5: Both are empty, S0C=S0R=∅ (Algorithm 1, line 1); in Iter-1, both agents rely only on their prompts and the context. We will state this in §3.3.
We thank the reviewer again for the constructive feedback and the kind recognition of our work.