- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper is concerned with synthetic time series generation and how generation can be edited post-hoc to enforce certain characteristics while changing others. It proposes 2 ideas: 1. A three question protocol that separates plausibility (How close is the generator before post-hoc editing to the true distribution?) correctness (Is the requested characteristic properly enforced while others are preserved?) and adoption (Does it actually improve the target metric) aiming to improve the protocol that is typically used to evaluate post-hoc approaches. 2. It proposes a method (KRPC) that enforces exact specific preservation with a refusal semantic. The paper then demonstrates applications of the framework in 4 studies from which 2 use KRPC.
Critical strengths
The paper focuses on improving research protocols and battling the general issue of overclaiming experimental results. In my opinion, work in this direction is quite important, especially given the current submission numbers.
The conceptual framework is intuitive and seems to tackle a real evaluation problem for post-hoc time-series generator editing.
The paper flags its own gaps very consistently which is transparent and intellectually honest.
Critical weaknesses
I have a number of concerns that I am listing below (not ordered by importance):
I believe the abstract and the introduction need some revisions as they are failing to provide a clear entry into this work. I base this judgement on the following points:
- The abstract is oddly specific about the results of the proposed method which, at this point, are not particularly insightful.
- Certain terms like "base law", "correct action" "justified" are comprehensible given the full manuscript but are not clearly defined in the introduction.
- Figure 1 is referenced directly on the first page but is in itself not comprehensible without Formula 3 which is only introduced much later.
- The literature review is extremely dense and does not really provide a good categorization of the work as it is not really clear how KRPC differs from other post-hoc methods and whether other works proposed similar protocols for post-hoc editing.
I would recommend to include some actual problematic examples of post-hoc edits and corresponding judgements here that motivate the proposed protocol.
As far as i understand KRPC is a concrete (but not a unique) executor that allows the framework to be implemented in practice. However, given the results in the appendix that show that the method is not necessarily better than other established approaches, I wonder why it takes such a prominent role in the manuscript. Making both the method and the framework equal contributions seems odd.
The conclusion seems rushed and is not particularly informative by itself.
Foundation models (e.g. Sundial as mentioned in the introduction) seem to be the strongest practical target for post-hoc methods. It seems like all experimental results are based on generators that are generally retrainable.
Questions for authors
It seems like only Question 2 (correctness) can be fully verified. How does this affect the usability of the proposed framework in practice?
How does one choose which properties of a generated time-series should be protected? I assume, in practice, it is often unclear what should actually be preserved exactly?
Given that we have a generator that we can actually retrain or change. When is the proposed framework still preferable?
Would it be possible to include an example of a post-hoc method in combination with a foundation model?
I thank the reviewers for their effort to address my concerns. With the state of the current manuscript I chose to keep my scoring.