- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Taking some concepts from cognitive linguistic representation, the paper introduces the first structured representation for agent skill artifacts that disentangles skill-level scheduling signals, scene-level execution structure, and logic-level action/resource-use evidence. They call it Scheduling-Structural-Logical (SSL) representation.
In SSL, a skill artifact is converted by a normalizer into three layers: a scheduling record for skill-level signals, a structural graph of execution scenes, and a logical graph of atomic actions and resource evidence. It is basically a harness engineering effort to breakdown a single skill artefact into a more detailed representation.
Critical strengths
The paper's approach is inspired by some cognitive science work, which has been validated to some extent from previous literature.
Critical weaknesses
- The representation of the skills in 3 layers (a scheduling record for skill-level signals, a structural graph of execution scenes, and a logical graph of atomic actions and resource evidence) uses complex phrases like "scenes", "transition" which is not commonly found in previous harness work. I am not sure if it can be used off-the-shelf for any LLM on the market, or we require some further fine-tuning. Furthermore, I do not know if there is any practical value to implement such heuristic-driven changes to how skills are crafted. Perhaps, the bitter lesson has taught us that as models become better, we might not need to devote so much work in this area.
Questions for authors
- What is the performance difference if applied different ablation changes to the SSL representation? in Table 12, I see "The endpoint was chosen post hoc and all cells are exploratory. " What does this mean?