- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how representation updates and action-head updates change a PPO policy. It combines old and new representations with old and new action heads to measure their effects. The authors then propose MART to limit large action-distribution changes caused by representation updates under the old action head. The experiments show smaller policy changes and better task results in some settings. Although the authors have made good efforts, some issues still need to be addressed.
Critical strengths
-
The paper provides a clear way to examine changes in different parts of the policy network. The four-policy comparison shows how representation and action-head changes may offset each other. This helps explain changes that are not visible from the complete policy alone.
-
The experiments cover several control tasks and PPO-family methods, using five paired seeds. The authors report both policy changes and task results. They also show cases where smaller changes do not improve performance, which helps readers understand the limits of MART.
Critical weaknesses
-
It is not clear why the representation-induced change needs to be limited. A large change under the old head does not by itself show a problem. The representation and action head may simply adjust together while the final policy remains similar. The paper needs more evidence that this change is linked to later training problems.
-
It is not clear whether the gains come from this specific constraint or from smaller updates in general. MART reduces several measures together, including representation-induced change, complete-policy change, and action probability ratio changes. The current results do not fully separate these effects.
Questions for authors
-
In Sections 3.2 and 4.3, the authors should check whether larger representation-induced changes are followed by worse training results when complete-policy changes are similar. This would help explain why the additional constraint is needed.
-
In Sections 4.5–4.7, the authors should better explain why MART helps in some settings but not others. It would be useful to compare cases where the two components offset each other without harm and cases where this is followed by a drop in performance.
-
The authors should compare MART with a smaller learning rate for the representation network, at least in the main traffic setting. Both methods should use the same training budget. This would help show whether simply reducing the update size gives similar gains.
-
In Section 4.6, the authors should add a penalty on the complete-policy TV or KL change. They should compare task results when the complete-policy changes are similar. This would help show what extra benefit comes from limiting the representation path.